Computation and Language · Cross-listed from cs.CV
Textual Supervision Enhances Geospatial Representations in Vision-Language Models
Marcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon et al. (+5 more)
Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial
reasoning. In this work, we analyze the geospatial representations acquired by three model families:
vision-only architectures (e.g., ViT),
vision-
language models (e.g., CLIP), and large-scale
multimodal foundation models (e.g., LLaVA, Qwen, and Gemma). By evaluating across image clusters, including people, landmarks, and everyday objects, grouped based on the degree of localizability, we reveal systematic gaps in spatial accuracy and show that textual supervision enhances the learning of geospatial representations. Our findings suggest the role of
language as an effective complementary modality for encoding spatial context and
multimodal learning as a key direction for advancing geospatial AI.
Read the full abstract ▾
All authors Marcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon, Luiz Felipe Vecchietti, Bryan Nathanael Wijaya, Cheng Yaw Low, Virgilio Almeida, Meeyoung Cha
Computation and Language
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding
Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal et al. (+3 more)
Meaningful
multilingual evaluation must test models in the target
language and educational context. Urdu, spoken by more than 230 million people, lacks a broad MMLU-style
benchmark built from native educational sources. We introduce UrduMMLU, a
benchmark of 26,389 Urdu MCQs across 26 subjects and five domains, collected from native Urdu MCQ banks and public examination PDFs. Unlike translation-based
benchmarks, UrduMMLU combines academic subjects with content specific to Urdu and regional education. We label the exam-derived portion through dual human annotation with strict consensus filtering. We evaluate 30 LLMs under English and Urdu
prompts, yielding 60 zero-shot evaluations, and further evaluate four open-source LLMs under multiple few-shot settings across both
prompt languages. Gemini-3.5-Flash performs best, reaching 90.23% and 90.45% accuracy, while no other model exceeds 85%. The strongest open-source model trails by 7.78 and 9.12 points, and many models lose 25 to 40 points on Urdu-centered Humanities subjects compared with STEM. Few-shot prompting yields only modest gains. Results on UrduMMLU show that current LLMs have uneven Urdu knowledge, particularly for content grounded in the regional context.
Read the full abstract ▾
All authors Ahmer Tabassum, Sarfraz Ahmad, Hasan Iqbal, Owais Aijaz, Momina Ahsan, Preslav Nakov
Computation and Language
Explicit Evidence Grounding via Structured Inline Citation Generation
Anar Yeginbergen, Amelie Wührl, Anna Rogers et al. (+1 more)
As AI systems become more widely adopted, the demand for factual and faithful generation grows. Properly attributing information through citations becomes, therefore, crucial. This work introduces FullCite, a framework that, in contrast to most previous works, generates structured inline citations linking each claim to both its source document and supporting evidence. FullCite proposes three strategies to inline citation generation:
prompt-based generation, constrained decoding over a citation grammar, and posthoc span
alignment. Using three question answering
benchmarks, namely, ASQA, BioASQ, and ExpertQA, we assess citation quality and faithfulness along three dimensions: document-level correctness, evidence span identification, and claim-citation faithfulness. Our evaluation shows that while LLMs are generally effective at identifying relevant documents, they struggle to identify the precise supporting spans within them. This gap suggests that achieving faithful attributed QA will require research to place greater emphasis on precise evidence span identification.
Read the full abstract ▾
All authors Anar Yeginbergen, Amelie Wührl, Anna Rogers, Rodrigo Agerri
Computation and Language
Learning Perspectivist Social Meaning via Demographic-Conditioned Fusion Embeddings
Amanda Cercas Curry, Lucio La Cava, Luca Maria Aiello et al. (+1 more)
Social meaning in
language is inherently perspectival, varying across annotator backgrounds, demographics, and ideological positions. However, most NLP systems collapse this variation into a single ground-truth label, ignoring the diversity of interpretations. In this work, we model social dimensions along a perspectivist spectrum, capturing how interpretations vary across demographic groups on a
dataset consisting of 28k human annotations. We
benchmark multiple modeling paradigms, including zero-shot, few-shot, and fine-tuned approaches, and propose fusion
embeddings that integrate textual and demographic representations. Our fusion models yield consistent and statistically significant improvements over text-only baselines across all fusion strategies (+5.9-6.5% relative macro PR-AUC), with shuffle ablations confirming that demographic profiles carry genuine predictive signal rather than spurious correlations.
Read the full abstract ▾
All authors Amanda Cercas Curry, Lucio La Cava, Luca Maria Aiello, Gianmarco De Francisci Morales
Computation and Language · Cross-listed from cs.LG
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang et al. (+2 more)
Low-bit quantization has been widely adopted to accelerate the inference of large
language models (LLMs) by significantly reducing computational cost and
memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and
benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.
Read the full abstract ▾
All authors Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang, Mathieu Salzmann, Lukas Cavigelli
Computation and Language
Style or Content? Evaluating Style Classifiers with Controlled Content Overlap
Zhuo Liu, Haozheng Du, Xiangxiang Xu et al. (+1 more)
Style classifiers can use content cues that correlate with style labels in naturally collected data, yet we lack a systematic way to measure this reliance. We study this problem with a controlled content overlap setup built on parallel Bible translations. Specifically, we define the overlap parameter $\alpha$ as the normalized residual of mutual information between content identity and style label, so that it measures how much content is shared across style classes: from no shared content ($\alpha=0$) to fully shared content ($\alpha=1$). Cross-overlap evaluation of RoBERTa-based classifiers shows that low-overlap models degrade when content cues are removed, while high-overlap models transfer more robustly. A cross-style content
retrieval probe further shows that content becomes less recoverable as $\alpha$ increases, with training dynamics showing this removal occurs gradually. Together, these results suggest that controlled overlap provides a simple diagnostic for separating style learning from content shortcuts.
Read the full abstract ▾
All authors Zhuo Liu, Haozheng Du, Xiangxiang Xu, Hangfeng He
Computation and Language
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Ernests Lavrinovics, Marco Letizia, Roy Janco et al. (+3 more)
We present SigmaScale, a method for learning auxiliary scaling
matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large
Language Model (LLM) compression. Instead of deriving scaling
matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight
matrices, as reflected by reductions in effective-rank
entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot
benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.
Read the full abstract ▾
All authors Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini
Computation and Language
mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
Yerzhan Sapenov and Jaromir Savelka
We introduce mmPISA-bench, a compact high-quality
multilingual reasoning benchmark derived from the OECD Programme for International Student Assessment (PISA). The
benchmark consists of 25 multiple-choice questions that require
reasoning in order to be answered correctly. Each question is provided in official human translations to 43
languages and complemented with machine-translated counterparts (i.e., 2,150 data points in total). We evaluate two mainstream proprietary LLMs across
languages,
reasoning effort levels, and translation types in terms of their ability to answer the questions correctly. Our results show that modern LLMs can reason effectively across all evaluated
languages, achieve accuracy comparable to human test-takers, with some performance variations across covered
languages. We further find that machine-translated questions do not degrade accuracy relative to official human translations which suggests that high-quality machine translation (synthetic data) might often be adequate for large-scale
multilingual reasoning evaluations where official translations are not available. Finally, we analyze
token usage and related inference cost and find that LLMs usage in some
languages is simultaneously more expensive and less accurate.
Read the full abstract ▾
Computation and Language
Modeling semantic association in self-paced reading with language model embeddings
Sara Møller Østergaard, Kenneth Enevoldsen, Afra Alishahi et al. (+1 more)
Semantic association between a word and its context has been identified as an important component of reading comprehension, even when word predictability is accounted for. Recent research has highlighted the potential of
language model ( LM)
embeddings to quantify semantic association. Yet,
embedding-based semantic association have been operationalized in a myriad of ways. In this study, we use
embeddings from LMs to estimate semantic association on a corpus of joint electroencephalography (
EEG) and self-paced reading of natural, Dutch texts. Semantic association is calculated in ten different implementations that vary the
embedding model and context lengths. The effects of semantic association across the different implementations on the N400 and self-paced reading times are examined using
Bayesian hierarchical models and Bayes factor. The results show that the choice of
embedding model can alter the estimated effect of semantic association on both the N400 and self-paced reading times. Furthermore, the results demonstrate a promising potential of sentence
embeddings for capturing semantic association, as only implementations relying on sentence
embeddings indicate reliable results of semantic association beyond word predictability on both neural and behavioral measures. Together, these findings highlight the importance of methodological choices in quantifying semantic association.
Read the full abstract ▾
All authors Sara Møller Østergaard, Kenneth Enevoldsen, Afra Alishahi, Bruno Nicenboim
Computation and Language · Cross-listed from cs.IR
Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation
Shamira Venturini and Steffen Kinkel
Evaluating the quality of automatically generated keyphrases remains a complex challenge. Traditional metrics either rely on exact lexical matching or consider semantic similarity while ignoring prediction ranking, both of which misalign with how humans judge informativeness and relevance. We introduce Semantic R-Precision (SemR-p), a novel evaluation metric that integrates semantic similarity into the rank-aware R-Precision framework. Designed from a human-centric perspective and inspired by Information
Retrieval metrics, SemR-p rewards semantically relevant keyphrases that appear early in the output list. We conducted extensive analyses to assess its semantic sensitivity, ranking awareness, and discriminative power across models and
datasets. The results suggest that SemR-p offers a complementary lens for evaluating keyphrase predictions, helping to better reflect user-centred notions of relevance alongside traditional lexical and semantic matching metrics.
Read the full abstract ▾
Computation and Language
TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents
Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli et al. (+7 more)
Autonomous LLM
agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring. Existing approaches either evaluate complete trajectories in a single pass or partition them into independently scored windows, limiting their ability to connect evidence across temporally distant actions. We propose TRACE, a monitoring framework for long-horizon LLM
agent trajectories. TRACE operates through a TIJ (Triage-Inspect-Judge) loop that identifies high-signal regions, performs targeted inspection while maintaining accumulated evidence across
reasoning steps, and synthesizes a trajectory-level verdict. We evaluate TRACE on ten task domains from SHADE-Arena against state-of-the-art baselines. TRACE achieves an aggregate F1 of 0.713 and recall of 0.844, with the largest gains on tasks requiring long-range evidence linking.
Read the full abstract ▾
All authors Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli, Snigdha Ansu, Mohammadreza Teymoorianfard, Franck Dernoncourt, Hongjie Chen, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed
Computation and Language
Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Xing Yue, Linjuan Wu, Daoxin Zhang et al. (+2 more)
Open-ended reward modeling requires judges that can follow subtle, domain-specific preferences when verifiable answers are unavailable. Existing rubric-based methods often address this by generating criteria online for each query, but the extra generation step can add inference overhead and produce rigid or misaligned guidance. We introduce Eval-Skill, an exploration-guided method that synthesizes reusable evaluation skills for reward modeling and reframes reward guidance as context
evolution rather than parameter training or per-query rubric generation. Using only 100 cases per domain for skill
evolution, Eval-Skill synthesizes reusable domain-level evaluation skills through two progressive stages, workflow generation followed by principle generation, with exploration and selection interleaved across both stages. Once generated, a skill is directly injected into the judge context. Across multiple RM
benchmarks, Eval-Skill consistently improves diverse judge backbones; on RewardBench 2, it yields significant gains over vanilla judging for each main backbone (+13.44% for Qwen3-8B, and 18.51% for DeepSeek-V4-Flash). Further analyses of
evolution-time scaling, generalizability, and transferability show that compact evaluation skills offer an efficient new paradigm for LLM-based evaluation. Code is available at https://github.com/xing-stellus-yue/Eval-Skill.
Read the full abstract ▾
All authors Xing Yue, Linjuan Wu, Daoxin Zhang, Yongliang Shen, Weiming Lu
Computation and Language · Cross-listed from cs.SD
Phonetic Error Analysis of Raw Waveform Acoustic Models
Erfan Loweimi, Zhengjun Yue, Andrea Carmantini et al. (+3 more)
We analyse error patterns of raw waveform acoustic models on TIMIT phone recognition beyond the overall phone error rate (PER). PER is decomposed across three broad phonetic class (BPC) categorisations, and confusion
matrices are constructed from substitution errors. Our models combine parametric (SincNet, Sinc2Net) or non-parametric CNNs with Bidirectional LSTMs, achieving 13.9%/15.3% PER on Dev/Test, the best reported results for raw waveform models on TIMIT. Transfer learning from WSJ reduces PER to 11.3%/12.3%, surpassing the Filterbank baseline. Per-BPC analysis reveals that BLSTM layers benefit transition-dependent classes most, while WSJ transfer learning improves consonants roughly three times more than vowels. Confusion patterns are consistent across raw waveform and Filterbank systems, indicating that the dominant confusions reflect inherent phonetic similarities.
Read the full abstract ▾
All authors Erfan Loweimi, Zhengjun Yue, Andrea Carmantini, Zoran Cvetkovic, Steve Renals, Peter Bell
Computation and Language
MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Yilun Liu, Miao Zhang, Shimin Tao et al. (+9 more)
Multilingual and multicultural
benchmarks now cover dozens of
languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained
multilingual post-evaluation diagnosis. However, single LLMs and open-ended
agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a
Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection,
multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-
language diagnostic set, evaluated on top of a large-scale
multilingual evaluation substrate (33 model families, 11
benchmarks, 26
languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human
multilingual experts in 87.9% of pairwise comparisons. Applied with
multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning
benchmark score tables into model-selection and remediation guidance.
Read the full abstract ▾
All authors Yilun Liu, Miao Zhang, Shimin Tao, Minggui He, Chunguang Zhao, Chenxin Liu, Li Zhang, Chen Liu, Cheng Qian, Liqun Deng, Xiaojun Meng, Daimeng Wei
Computation and Language · Cross-listed from cs.AI
The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
Xiaoou Liu, Tiejin Chen, Weibo Li et al. (+2 more)
Foundation model
agents are increasingly deployed for real-world decision-making, but suffer from the sim-to-real gap. While robotics and classical control have mature frameworks to address this gap, the foundation model community is treating
agent robustness as an entirely novel phenomenon. Our paper proposes formalizing the foundation model
agent evaluation and training gap as a classical sim-to-real problem structured entirely around the four elements of a Markov Decision Process, including Observation, Action, Transition, and Reward. In this paper, we set a comprehensive research agenda that translates classical discrepancies into the foundation model domain and advocates for adopting established solutions like domain randomization. We provide concrete examples, such as a
multilingual tool calling to demonstrate how severe observation space gaps lead to operationally invalid actions despite correct semantic intent. Ultimately, this agenda aims to drive a paradigm shift, yielding a unified vocabulary and standardized stress test
benchmarks to foster a new generation of highly trustworthy
agents for reliable real-world applications.
Read the full abstract ▾
All authors Xiaoou Liu, Tiejin Chen, Weibo Li, Xiyang Hu, Hua Wei
Computation and Language · Cross-listed from cs.LG
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
Yongliang Miao, Fengyuan Liu, Wei Shi et al. (+4 more)
Supervised fine-tuning (SFT) is a prevailing method for adapting large
language models to
reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However,
reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model's own
reasoning distribution. We propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware SFT framework that calibrates expert supervision according to problem-level solvability estimated from verified on-policy rollouts. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self-generated trajectories when the model already exhibits reliable
reasoning behavior. To preserve useful
reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift. Experiments across multiple models on six mathematical
reasoning benchmarks and two code
reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods. The code is available at https://github.com/zjd1sq/RASFT.
Read the full abstract ▾
All authors Yongliang Miao, Fengyuan Liu, Wei Shi, Yanguang Liu, Fei Sun, Na Zou, Mengnan Du
Computation and Language
Principles of Concept Representation in Sentence Encoders
Isabelle Mohr, John Dujany, Jonathan Souquet et al. (+1 more)
What makes a sentence encoder produce good concept representations? We approach this through the lens of representational compositionality: an encoder supports a concept family only when its latent space admits a low-distortion realization of the corresponding semantic operator. This framing predicts both where current encoders succeed and where they are structurally mismatched to their supervision. Through a controlled ablation over encoder conditions trained on 3.3 million synonym and definition pairs from WordNet and Wiktionary, evaluated on three decontaminated splits and a modifier-labeled noun-phrase
benchmark, we identify four principles. Fine-tuning recalibrates the latent geometry rather than expanding it (P1). Semantic signal concentrates in the final
transformer layer before concept-specific training begins, making cross-layer pooling redundant (P2). Hard negatives improve discrimination and stress-test robustness without improving
retrieval ranking, showing that calibration and ranking are independently addressable (P3). Finally, the effectiveness of supervision depends on the composition type of the target concept. Extensional training helps intersective and subsective families while degrading relational and intensional ones, exposing a structural limitation of current training paradigms (P4). We release two new evaluation
datasets: a DBpedia semantic-gap
benchmark and a modifier-labeled NP paraphrase suite.
Read the full abstract ▾
All authors Isabelle Mohr, John Dujany, Jonathan Souquet, Andre Freitas
Computation and Language
Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition
Tung X. Nguyen, Hieu Minh Truong, Giang Son Nguyen et al. (+3 more)
Code-switching (CS), the alternation between multiple
languages within a single utterance, remains challenging for Automatic
Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large
language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-
entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.
Read the full abstract ▾
All authors Tung X. Nguyen, Hieu Minh Truong, Giang Son Nguyen, Nhu Vo, Wray Buntine, Dung D. Le
Computation and Language
FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewards
Yining Zhu, Zihao Deng, Leiming Wang et al. (+5 more)
Large
language model (LLM)
agents increasingly rely on external experience to continually adapt to changing environments without modifying their underlying models. Recent experience mechanisms have demonstrated promising results across diverse tasks. However, their effectiveness is typically evaluated within individual
benchmark settings, and how experience mechanisms generalize across different scenarios remains insufficiently explored. In this work, we present a scenario-oriented analysis of experience mechanisms for LLM
agents. We characterize existing evaluation scenarios along four dimensions: outcome observability, credit assignment complexity, environmental dynamics, and experience reusability. Our analysis shows that existing
benchmarks often evaluate experience mechanisms under scenarios where at least one dimension is comparatively favorable, leaving more challenging combinations of scenario properties underexplored. To address this gap, we introduce FinEvolveBench, a reproducible
benchmark built on a chronological stream of rich financial news and
market data that enables systematic evaluation of experience-based self-
evolution under challenging experience regimes characterized by noisy feedback, ambiguous credit assignment, environmental non-stationarity, and limited experience reusability. Experiments show that existing approaches exhibit substantially reduced or inconsistent gains in this setting, highlighting the scenario-dependent nature of experience mechanisms and the challenge of maintaining valid experience under changing environments.
Read the full abstract ▾
All authors Yining Zhu, Zihao Deng, Leiming Wang, Jingfei Lu, Junbo Wang, Yuke Li, Jikun Shen, Chuncheng Ran
Computation and Language
OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios
Xinyi Li, Zhen Fang, Yongxin Deng et al. (+12 more)
Hallucination detection is essential for the reliable deployment of large
language models (LLMs). However, existing evaluations face two core challenges: inconsistent inference configuration and evaluation, and limited coverage of downstream domains and tasks. Consequently, reported detector performance is often difficult to compare, reproduce, and generalize beyond specific experimental settings. We introduce OpenHalDet, a unified
benchmark for
hallucination detection across diverse generation scenarios. OpenHalDet standardizes the evaluation pipeline, from
prompt construction and response generation to truthfulness annotation, detector scoring, and metric computation. It supports heterogeneous detector families under different access settings, including black-box methods that use only generated outputs, gray-box methods that rely on probability-based signals, and white-box methods that exploit internal model signals. By bringing diverse tasks, models, and detectors into a shared framework, OpenHalDet enables controlled comparison and provides a systematic view of how different detection paradigms behave in LLM applications. We release OpenHalDet as an open and extensible codebase to facilitate reproducible evaluation and future development of
hallucination detection methods. The code and
datasets are available at https://github.com/Nellie179/
Hallucination-Detection.
Read the full abstract ▾
All authors Xinyi Li, Zhen Fang, Yongxin Deng, Jinyuan Luo, Hongnan Ma, Changdae Oh, Zijing Shi, Shanshan Ye, Hanchen Wang, Shu-Lin Chen, Yadan Luo, Mengyue Yang, Sean Du, Sharon Li, Ling Chen
Computation and Language
Auditing Training Data in Domain-adapted LLMs: LoRA-MINT
Gonzalo Mancera, Daniel DeAlcala, Aythami Morales et al. (+3 more)
We present LoRA-MINT, a new methodology for Membership Inference Test (MINT) applied to recent Large
Language Models (LLMs) fine-tuned for specific Natural
Language Processing (NLP) tasks through Low-Rank Adaptation (LoRA). The primary goal is to assess whether individual samples were part of the training data of these adapted models, providing a useful
auditing tool for the management of intellectual property and sensitive data. Our analysis explores the relationship between model perplexity and membership status, providing a systematic framework for estimating data exposure in fine-tuned LLMs. We conducted experiments on four models and three
benchmark datasets, obtaining precision values in determining if given data were used for training ranging from 0.77 to 0.92, which outperform state-of-the-art baselines and demonstrate the robustness and generality of the proposed method. In general, our findings underscore the potential of LoRA-MINT as an effective and scalable framework for
auditing LLMs, improving transparency, and fostering the ethical and responsible deployment of AI and NLP technologies. For the sake of concreteness and current relevance, our discussion and experiments are centered on LoRAadjusted LLMs, but note that most of the presented methodology is easily applicable for
auditing training data given any other technique for adapting LLMs or, more generally, any other domain-adapted AI models.
Read the full abstract ▾
All authors Gonzalo Mancera, Daniel DeAlcala, Aythami Morales, Julian Fierrez, Ruben Tolosana, Francisco Jurado
Computation and Language
Didact: A Cross-Domain Capability Discovery System for Defence
Aarya Bodhankar, Aditya Joshi, Bao Gia Doan et al. (+3 more)
Policymakers in defence and defence-aligned sectors must monitor rapidly evolving research alongside sector priorities relevant to operational and strategic needs. In practice, these sources are fragmented across heterogeneous formats, disjoint repositories, and siloed update streams, making capability discovery slow and difficult to audit. We present Didact, a prototype that integrates publicly available defence reports and policy documents from Australia with a purpose-built knowledge graph derived from Australian research publications. Didact provides natural
language conversations for policy-oriented workflows, and leverages a composite
retrieval-augmented generation (RAG) pipeline. A key feature of Didact is an interactive Evidence Rail that visualises retrieved evidence and source relationships. Our evaluation of the output quality and runtime of Didact highlights its utility. While Didact has been co-developed as an academia-industry project for the Australian context, it is adaptable to other domains where knowledge is similarly fragmented. A demonstration
video is available here:
Read the full abstract ▾
All authors Aarya Bodhankar, Aditya Joshi, Bao Gia Doan, Thomas Marchant, Oscar Leslie, Flora Salim
Computation and Language
ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning
Vladislav Smirnov, Chieu Nguyen, Sergey Senichev et al. (+14 more)
Test-time compute (TTC) scaling has emerged as a powerful paradigm for improving large
language model (LLM)
reasoning by allocating additional compute during inference, e.g., via multi-sample generation and verifier-based reranking. Existing TTC scaling strategies and
reasoning scorers remain fragmented, evaluated under inconsistent protocols, and are rarely analyzed through the lens of quality-cost trade-offs. We introduce ThinkBooster, a unified framework for seamless test-time compute scaling of LLM
reasoning, which consists of (i) a modular Python library implementing state-of-the-art TTC scaling strategy and scorer families, (ii) a
benchmark that jointly evaluates performance and computational efficiency, and (iii) a deployable OpenAI-compatible proxy service that enables drop-in integration of adaptive
reasoning into real-world applications. We further provide a demo visual debugger for inspecting the
reasoning trajectories, intermediate selection decisions, and alternative
reasoning paths. Empirical results on mathematical and coding tasks reveal the performance-compute trade-offs of TTC scaling strategies and scoring methods and demonstrate that ThinkBooster provides practical gains in real-world tasks. The code is available online under an MIT license.
Read the full abstract ▾
All authors Vladislav Smirnov, Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov
Computation and Language
EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering
Xiaopeng Yuan, Zebin Wang, Suwen Wang et al. (+3 more)
Long-context question answering (QA) remains challenging for smaller
language models even when answer-bearing evidence is already present in the input. Existing within-context
retrieval methods localize and expose candidate evidence chunks for the question, but they stop at input-level evidence exposure rather than adapting the query-side
attention parameters that control how the model allocates
attention over full-context positions. In contrast, lightweight test-time adaptation methods, such as query-only test-time training (qTTT), leave evidence localization unresolved because their generic span-level self-supervised objectives do not identify which context positions support the current answer. In this paper, we propose Evidence-Aligned SElective Test-Time Training (EASE-TTT), a within-context
retrieval-augmented test-time training framework that converts selected evidence chunks into a soft
attention supervision target over their
token positions. Instead of replacing the full context with retrieved chunks, EASE-TTT uses the resulting
attention target to guide query-side adaptation, with the adapted model generating the final answer from the original full context. Experiments on six LongBench QA tasks and three small decoder-only
language models show that EASE-TTT achieves the strongest macro-average performance among full-context inference,
retrieval-only baselines, and qTTT, supporting evidence-aligned test-time adaptation in long-context QA.
Read the full abstract ▾
All authors Xiaopeng Yuan, Zebin Wang, Suwen Wang, Zongxin Yang, Haohan Wang, Yushun Dong
Computation and Language
An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection
Carl Lochstampfor and Ayan Roy
Our prior work introduced COVA, a synthetically generated multi-turn conversational smishing
dataset of 3,201 labeled conversations, establishing baseline detection
benchmarks across eight models. While XGBoost with TF-IDF features achieved the best performance, with 72.5% accuracy and 0.691 macro F1,
transformer models underperformed, which was attributed to input truncation and insufficient training data. We present COVA-X, an expanded
dataset of 10,985 conversations spanning eight elder-targeted scam categories, produced by an improved generation pipeline addressing contamination, label mismatch, stage-direction bleed, and
prompt-design failures from the first iteration. Retraining all classifiers on the expanded
dataset yields the central finding of this work: Longformer now surpasses XGBoost on all evaluation metrics, achieving 79.71% accuracy and 0.7786 macro F1 compared with 78.43% and 0.7563 for XGBoost. This directly confirms that
transformer models require larger conversational corpora to realize their contextual advantages. We additionally document a quality life-cycle including a 12.7$\times$ improvement in label correction rate, from 49.8% to 3.9%, an architectural intervention reducing virtual-kidnapping artifact rates from 67.1% to 46.5%, and a per-scam-type outcome analysis showing that scam categories modulate results in mechanism-consistent ways. A pre/post-cleanup sensitivity analysis confirms that
dataset refinement recovers genuine label-relevant signal across all three classifier architectures.
Read the full abstract ▾
Computation and Language
Are Large Language Models Suitable for Graph Computation? Progress and Prospects
Yuting Zhang, Yi Han, Kai Wang et al. (+3 more)
Large
language models (LLMs) have been increasingly explored for graph computation, where tasks require
reasoning over structured relationships and algorithmic operations. Yet, it remains unclear when LLMs can reliably support such computation and how they should be incorporated into graph-solving pipelines. Existing surveys at the intersection of LLMs and graphs primarily focus on graph learning, text-attributed graphs, or graph-
language modeling. To bridge this gap, we provide a comprehensive review of LLMs for graph computation through a role-based taxonomy. Specifically, we identify two major paradigms: i) LLMs as executors, where models directly solve graph tasks from graph descriptions and instructions; and ii) LLMs as planners, where models formulate problems, decompose
reasoning steps, and invoke external tools or
agents for execution. Based on this taxonomy, we analyze the strengths and limitations of current methods. Our review indicates that LLMs are promising for simple, small-scale tasks, but remain unreliable for large-scale and exactness-demanding tasks. Finally, we summarize available
datasets and suggest four future directions.
Read the full abstract ▾
All authors Yuting Zhang, Yi Han, Kai Wang, Wei Ni, Angela Bonifati, Wenjie Zhang
Computation and Language
Interpreting Brain Responses to Language with Sparse Features from Language Models
Michael A. Lepori, Kendrick Kay and Greta Tuckute
A central goal of cognitive neuroscience is to characterize the features that are represented by human
language cortex. Artificial
language models (LMs) have emerged as a powerful tool to address this challenge, but studies relating biological and artificial representations are often criticized as relating one black box to another. The present work introduces Augmented Sparse Encoding Models, an encoding framework that replaces dense LM hidden states with hierarchically-organized sparse autoencoder (SAE) features, while explicitly including surprisal as a predictor. Using this approach, we (i) produce interpretations of neural responses and (ii) test whether model-
brain alignment reflects primary or idiosyncratic variation in LM representations. Using a high-field 7T fMRI
dataset of eight participants listening to 200 linguistically diverse sentences, we first validate our modeling framework by recovering previous interpretations of voxel populations tuned to processing difficulty and meaning abstractness. We then interpret a previously-uncharacterized (but reliable) voxel population and find that it is tuned to people-related content. Next, we show that the fronto-temporal human
language network is predicted by a common set of features across its constituent regions, but find that frontal regions are relatively well-explained by surprisal alone, even in the absence of LM-based features. Finally, we show that
brain responses during
language processing are not merely predictable from an arbitrary set of LM features. Rather,
brain responses are best explained by the features that tend to capture the most general information encoded in LM representations, suggesting a nontrivial correspondence between
brain and LM
language representation.
Read the full abstract ▾
Computation and Language
CRAFT: A Unified Counterfactual Reasoning Framework for Tabular Question Answering and Fact Verification
Chenshuo Pan, Yu Zhao, Jie Zhang et al. (+7 more)
Table
reasoning remains challenging for large
language models (LLMs), particularly in tasks that require multi-step inference over long and structured tables. Existing approaches predominantly rely on single-direction
reasoning, which limits their ability to explore alternative hypotheses across tasks. In this work, we propose CRAFT, a unified Counterfactual
Reasoning Framework that reformulates Tabular question answering and fact verification into a general bidirectional verification process. Our method explicitly constructs both declarative statements and their counterfactual variants. Evidence is then extracted from
reasoning along both the original and counterfactual paths, and integrated via a weighted mechanism to arrive at the final answer. Experimental results show that our approach consistently surpasses representative baselines on table
reasoning datasets such as WikiTQ and TabFact, achieving especially large improvements on complex question answering. Our framework also significantly mitigates performance gaps between different backbone LLMs. This indicates that counterfactual
reasoning effectively overcomes the limitations of single-direction inference, guiding LLMs toward more discerning
reasoning and establishing a more principled paradigm for structured
reasoning tasks. Our code will be made publicly available upon acceptance.
Read the full abstract ▾
All authors Chenshuo Pan, Yu Zhao, Jie Zhang, Changzai Pan, Zhenhe Wu, Jiayi Liang, Yujie Mao, Shuangyong Song, Yongxiang Li, Zhongjiang He
Computation and Language
Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces
Debjyoti Saha Roy
Reasoning-trained
language models can select, zero-shot, a small set of relevant labels from thousands to hundreds of thousands of candidates. We ask how they do it mechanistically, and whether the mechanism can be distilled. We treat each decision as a
token-level event scored by the model's own decision margin: the
token that picks a coarse region of the label space, the
tokens that pick a label within it, and the
token where the output departs from a near-miss named earlier in the
reasoning. Attribution, exact mean-ablation, knock-in into another example's context and a null calibration give individual
attention heads causal standing. On clinical coding of MIMIC-IV discharge summaries, with all 5,651 candidate codes in context, a small, global, phase-structured set of heads is necessary and sufficient on essentially every summary; distinct head families read the candidate region and the earlier near-miss, and blocking what they read removes their effect; for mentions elicitable in the
reasoning, a disjoint mid-layer set makes the region elicitable several
tokens before the code. The structure recurs in a second reasoner. We introduce MISTILL, which, unlike chain-of-thought
distillation, also supervises the student's pooled
attention at these decision events. Read on heads found after training, it nearly doubles the causal recovery of the contrastive decision in a cross-family student and adds a small, seed-stable gain in one that already carries most of it, with no detected task difference at matched precision; under matched selection it exceeds a frozen random-head pool on both recovery and task score. A pre-registered evidence-flip test detected no improvement in evidence-dependent decisions through the tested routes.
Read the full abstract ▾
Computation and Language
Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning
Pratik Jayarao, Chaitanya Dwivedi, Himanshu Gupta et al. (+5 more)
The performance gap across
languages in LLMs is well documented, and closing it natively requires pretraining or fine-tuning on corpora that, for most
languages, are quite limited. Translation offers an alternative: converting an input into the model's dominant
language unlocks its full capabilities at once. But translating every input is wasteful for
languages the model already handles, while leaving the choice to the model fails too, as LLMs are overconfident and skip the tool even when they cannot understand the input. Prior work resolves this with
language-specific rules, heuristics, or external routers, each requiring manual engineering. We instead learn a single policy that decides when to translate from reward alone, developing
language- and domain-adaptive introspection that invokes translation only when it cannot solve a task natively. Using data from our answer-preserving translation pipeline, we continue RL on the post-trained Qwen3-4B across 22
languages in 3 resource tiers (High, Low, XLow) and 5 domains, and introduce confidence-gated GSPO for cost-sensitive tool use. The gated policy lifts reward over the untrained baseline by +4.6 on High, +23.5 on Low, and +17.5 on XLow, and extends zero-shot to 9 held-out
languages. Against an unconstrained policy that almost always translates (the reward upper bound), it preserves full reward at 66% of the cost, outperforming other cost penalties by up to +24.3 on the low-resource tiers while matching the free policy's reward on high-resource tiers at a fraction of the tool use. On 2 synthetic
languages with zero prior exposure it correctly learns to always translate, improving +18.7 over the overconfident baseline.
Read the full abstract ▾
All authors Pratik Jayarao, Chaitanya Dwivedi, Himanshu Gupta, Neeraj Varshney, Adithya M Devraj, Meet Vadera, Priyanka Nigam, Bing Yin