Computation and Language
Translation as a Scalable Proxy for Multilingual Evaluation
Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu et al. (+6 more)
The rapid proliferation of LLMs has created a critical evaluation paradox: while LLMs claim
multilingual proficiency, comprehensive non-machine-translated
benchmarks exist for fewer than 30
languages, leaving >98% of the world's 7,000
languages in an empirical void. Traditional
benchmark construction faces scaling challenges such as cost, scarcity of domain experts, and data contamination. We evaluate the validity of a simpler alternative: can translation quality alone indicate a model's broader
multilingual capabilities? Through systematic evaluation of 14 models (1B-72B parameters) across 9 diverse
benchmarks and 7 translation metrics, we find that translation performance is a good indicator of downstream task success (e.g., Phi-4, median Pearson r: MetricX = 0.89, xCOMET = 0.91, SSA-COMET = 0.87). These results suggest that the representational abilities supporting faithful translation overlap with those required for
multilingual understanding. Translation quality, thus emerges as a strong, inexpensive first-pass proxy of
multilingual performance, enabling a translation-first screening with targeted follow-up for specific tasks.
Read the full abstract ▾
All authors Sheriff Issaka, Erick Rosas Gonzalez, Lieqi Liu, Evans Kofi Agyei, Lucas Bandarkar, Nanyun Peng, David Ifeoluwa Adelani, Francisco Guzmán, Saadia Gabriel
Computation and Language
Cleansing the Artificial Mind: A Self-Reflective Detoxification Framework for Large Language Models
Kaituo Zhang, Zhimeng Jiang and Na Zou
Recent breakthroughs in Large
Language Models (LLMs) have revealed remarkable generative capabilities and emerging self-regulatory mechanisms, including self-correction and self-rewarding. However, current detoxification techniques rarely exploit these built-in abilities; instead, they rely on external modules, labor-intensive data annotation, or human intervention –factors that hinder scalability and consistency. In this paper, we introduce a fully self-reflective detoxification framework that harnesses the inherent capacities of LLMs to detect, correct toxic content, and refine LLMs without external modules and data annotation. Specifically, we propose a Toxic Signal Detector –an internal self-identification mechanism, coupled with a systematic intervention process to transform toxic text into its non-toxic counterpart. This iterative procedure yields a contrastive detoxification
dataset used to fine-tune the model, enhancing its ability for safe and coherent text generation. Experiments on
benchmark datasets such as DetoxLLM and ParaDetox show that our method achieves better detoxification performance than state-of-the-art methods while preserving semantic fidelity. By obviating the need for human intervention or external components, this paper reveals the intrinsic self-detoxification ability of LLMs, offering a consistent and effective approach for mitigating harmful content generation. Ultimately, our findings underscore the potential for truly self-regulated
language models, paving the way for more responsible and ethically guided text generation systems.
Read the full abstract ▾
Computation and Language
Industry-Aligned Granular Topic Modeling
Sae Young Moon, Myeongjun Erik Jang, Haoyan Luo et al. (+3 more)
Topic modeling has extensive applications in text mining and data analysis across various industrial sectors. Although the concept of granularity holds significant value for business applications by providing deeper insights, the capability of topic modeling methods to produce granular topics has not been thoroughly explored. In this context, this paper introduces a framework called TIDE, which primarily provides a novel granular topic modeling method based on large
language models (LLMs) as a core feature, along with other useful functionalities for business applications, such as summarizing long documents, topic parenting, and
distillation. Through extensive experiments on a variety of public and real-world business
datasets, we demonstrate that TIDE's topic modeling approach outperforms modern topic modeling methods, and our auxiliary components provide valuable support for dealing with industrial business scenarios. The TIDE framework is currently undergoing the process of being open sourced.
Read the full abstract ▾
All authors Sae Young Moon, Myeongjun Erik Jang, Haoyan Luo, Chunyang Xiao, Antonios Georgiadis, Fran Silavong
Computation and Language
Early Linguistic Pattern of Anxiety from Social Media Using Interpretable Linguistic Features: A Multi-Faceted Validation Study with Author-Disjoint Evaluation
Arnab Das Utsa
Anxiety affects hundreds of millions of individuals globally, yet large-scale screening remains limited. Social media
language provides an opportunity for scalable detection, but current models often lack interpretability, keyword-robustness validation, and rigorous user-level data integrity. This work presents a transparent approach to social media-based anxiety detection through linguistically interpretable feature-grounded modeling and cross-domain validation. Using a substantial
dataset of Reddit posts, we trained a logistic regression classifier on carefully curated subreddits for training, validation, and test splits. Comprehensive evaluation included feature ablation, keyword masking experiments, and varying-density difference analyses comparing anxious and control groups, along with external validation using clinically interviewed participants with diagnosed anxiety disorders. The model achieved strong performance while maintaining high accuracy even after sentiment removal or keyword masking. Early detection using minimal post history significantly outperformed random classification, and cross-domain analysis demonstrated strong consistency with clinical interview data. Results indicate that transparent linguistic features can support reliable, generalizable, and keyword-robust anxiety detection. The proposed framework provides a reproducible baseline for interpretable mental health screening across diverse online contexts.
Read the full abstract ▾
Computation and Language
LIME-LLM: Probing Models with Fluent Counterfactuals, Not Broken Text
George Mihaila, Suleyman Olcay Polat, Poli Nemkova et al. (+3 more)
Local explanation methods such as LIME (Ribeiro et al., 2016) remain fundamental to trustworthy AI, yet their application to NLP is limited by a reliance on random
token masking. These heuristic perturbations frequently generate semantically invalid, out-of-distribution inputs that weaken the fidelity of local surrogate models. While recent generative approaches such as LLiMe (Angiulli et al., 2025b) attempt to mitigate this by employing Large
Language Models for neighborhood generation, they rely on unconstrained paraphrasing that introduces confounding variables, making it difficult to isolate specific feature contributions. We introduce LIME-LLM, a framework that replaces random noise with hypothesis-driven, controlled perturbations. By enforcing a strict "Single Mask-Single Sample" protocol and employing distinct neutral infill and boundary infill strategies, LIME-LLM constructs fluent, on-
manifold neighborhoods that rigorously isolate feature effects. We evaluate our method against established baselines (LIME, SHAP, Integrated Gradients) and the generative LLiMe baseline across three diverse
benchmarks: CoLA, SST-2, and HateXplain using human-annotated rationales as ground truth. Empirical results demonstrate that LIME-LLM establishes a new
benchmark for black-box NLP explainability, achieving significant improvements in local explanation fidelity compared to both traditional perturbation-based methods and recent generative alternatives.
Read the full abstract ▾
All authors George Mihaila, Suleyman Olcay Polat, Poli Nemkova, Himanshu Sharma, Namratha V. Urs, Mark V. Albert
Computation and Language
Bridging Human Interpretation and Machine Representation: A Landscape of Qualitative Data Analysis in the LLM Era
Xinyu Pi, Qisen Yang, Chuong Nguyen et al. (+1 more)
LLMs are increasingly used to support qualitative research, yet existing systems produce outputs that vary widely–from trace-faithful summaries to theory-mediated explanations and system models. To make these differences explicit, we introduce a 4$\times$4 landscape crossing four levels of meaning-making (descriptive, categorical, interpretive, theoretical) with four levels of modeling (static structure, stages/timelines, causal pathways, feedback dynamics). Applying the landscape to prior LLM-based automation highlights a strong skew toward low-level meaning and low-commitment representations, with few reliable attempts at interpretive/theoretical inference or dynamical modeling. Based on the revealed gap, we outline an agenda for applying and building LLM-systems that make their interpretive and modeling commitments explicit, selectable, and governable.
Read the full abstract ▾
All authors Xinyu Pi, Qisen Yang, Chuong Nguyen, Hua Shen
Computation and Language
RAC: Retrieval-Augmented Clarification for Faithful Conversational Search
Ahmed Rayane Kebir, Vincent Guigue, Lynda Said Lhadj et al. (+1 more)
Clarification questions help conversational search systems resolve ambiguous or underspecified user queries. While prior work has focused on fluency and
alignment with user intent, especially through facet extraction, much less
attention has been paid to grounding clarifications in the underlying corpus. Without such grounding, systems risk asking questions that cannot be answered from the available documents. We introduce RAC (
Retrieval-Augmented Clarification), a framework for generating corpus-faithful clarification questions. After comparing several indexing strategies for
retrieval, we fine-tune a large
language model to make optimal use of research context and to encourage the generation of evidence-based question. We then apply contrastive preference optimization to favor questions supported by retrieved passages over ungrounded alternatives. Evaluated on four
benchmarks, RAC demonstrate significant improvements over baselines. In addition to LLM-as-Judge assessments, we introduce novel metrics derived from NLI and data-to-text to assess how well questions are anchored in the context, and we demonstrate that our approach consistently enhances faithfulness.
Read the full abstract ▾
All authors Ahmed Rayane Kebir, Vincent Guigue, Lynda Said Lhadj, Laure Soulier
Computation and Language
How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers
Jonathan Roberts, Kai Han and Samuel Albanie
Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the
token. In general,
tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of
token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to
tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about
token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs.
Read the full abstract ▾
Computation and Language
Do explanations generalize across large reasoning models?
Koyena Pal, David Bau and Chandan Singh
Large
reasoning models (LRMs) produce a textual chain of thought (CoT) in the process of solving a problem, which serves as a potentially powerful tool to understand the problem by surfacing a human-readable, natural-
language explanation. However, it is unclear whether these explanations generalize, i.e. whether they capture general patterns about the underlying problem rather than patterns which are esoteric to the LRM. This is a crucial question in understanding or discovering new concepts, e.g. in AI for science. We study this generalization question by evaluating a specific notion of generalizability: whether explanations produced by one LRM induce the same behavior when given to other LRMs. We find that CoT explanations often exhibit this form of generalization (i.e. they increase consistency between LRMs) and that this increased generalization is correlated with human preference rankings and post-training with reinforcement learning. We further analyze the conditions under which explanations yield consistent answers and propose a straightforward, sentence-level ensembling strategy that improves consistency. Taken together, these results prescribe caution when using LRM explanations to yield new insights and outline a framework for characterizing LRM explanation generalization.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Building Production-Ready Probes For Gemini
János Kramár, Joshua Engels, Zheng Wang et al. (+4 more)
Frontier
language model capabilities are improving rapidly. We thus need stronger mitigations against bad actors misusing increasingly powerful systems. Prior work has shown that activation probes may be a promising misuse mitigation technique, but we identify a key remaining challenge: probes fail to generalize under important production distribution shifts. In particular, we find that the shift from short-context to long-context inputs is difficult for existing probe architectures. We propose several new probe architectures that handle this long-context distribution shift. We evaluate these probes in the cyber-offensive domain, testing their robustness against various production-relevant distribution shifts, including multi-turn conversations, long context
prompts, and adaptive red teaming. Our results demonstrate that while our novel architectures address context length, a combination of architecture choice and training on diverse distributions is required for broad generalization. Additionally, we show that pairing probes with prompted classifiers achieves optimal accuracy at a low cost due to the computational efficiency of probes. These findings have informed the successful deployment of misuse mitigation probes in user-facing instances of Gemini, Google's frontier
language model. Finally, we find early positive results using AlphaEvolve to automate improvements in both probe architecture search and adaptive red teaming, showing that automating some AI safety research is already possible.
Read the full abstract ▾
All authors János Kramár, Joshua Engels, Zheng Wang, Bilal Chughtai, Rohin Shah, Neel Nanda, Arthur Conmy
Computation and Language · Cross-listed from cs.GT
Sequential LLM Release Facilitates Manipulation in Regulated Markets
Eilam Shapira, Moshe Tennenholtz and Roi Reichart
AI
agents increasingly mediate bargaining, negotiation and persuasion for people and firms. Such
markets extend software-mediated commerce, but add a governance problem: independent model releases change delegates available to participants. Game theory shows that expanding a strategy set can harm
equilibrium outcomes, but mostly through constructed examples. Deployed AI-
agent logs are scarce, proprietary and
privacy-sensitive, and lack counterfactuals and payoff labels. We therefore use GLEE, an independently collected
benchmark of 587K strategic decisions by 13 large
language models across 1,320 matched bargaining, negotiation and persuasion configurations, to study model release as strategy expansion. Across more than 50{,}000 release comparisons, many releases move payoffs in opposite directions: one
agent gains while the other loses. We identify the Poisoned Apple effect: a released model that no
agent adopts in
equilibrium nevertheless shifts payoffs in opposite directions and changes the regulator's
market design. Up to roughly three in ten opposing shifts arise this way, and technology restrictions can amplify the effect.
Read the full abstract ▾
Computation and Language
CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation
Vanshali Sharma, Andrea Mia Bejar, Gorkem Durak et al. (+1 more)
In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.
Read the full abstract ▾
All authors Vanshali Sharma, Andrea Mia Bejar, Gorkem Durak, Ulas Bagci
Computation and Language · Cross-listed from cs.CV
MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models
Xiaoran Fan, Zhichao Sun, Tao Ji et al. (+2 more)
As
vision-
language models (VLMs) tackle increasingly complex and
multimodal tasks, the rapid growth of Key-Value (KV) cache imposes significant
memory and computational bottlenecks during inference. While Multi-Head Latent
Attention (MLA) offers an effective means to compress the KV cache and accelerate inference, adapting existing VLMs to the MLA architecture without costly pretraining remains largely unexplored. In this work, we present MHA2MLA-VLM, a parameter-efficient and
multimodal-aware framework for converting off-the-shelf VLMs to MLA. Our approach features two core techniques: (1) a modality-adaptive partial-RoPE strategy that supports both traditional and
multimodal settings by selectively masking nonessential dimensions, and (2) a modality-decoupled low-rank approximation method that independently compresses the visual and textual KV spaces. Furthermore, we introduce parameter-efficient fine-tuning to minimize adaptation cost and demonstrate that minimizing output activation error, rather than parameter distance, substantially reduces performance loss. Extensive experiments on three representative VLMs show that MHA2MLA-VLM restores original model performance with minimal supervised data, significantly reduces KV cache footprint, and integrates seamlessly with KV quantization.
Read the full abstract ▾
All authors Xiaoran Fan, Zhichao Sun, Tao Ji, Lixing Shen, Tao Gui
Computation and Language · Cross-listed from cs.HC
Interactive Narrative Analytics: Bridging Computational Narrative Extraction and Human Sensemaking
Brian Keith
Information overload and misinformation create significant challenges in extracting meaningful narratives from large news collections. This paper defines the nascent field of Interactive Narrative Analytics (INA), which combines computational narrative extraction with interactive visual analytics to support sensemaking. INA approaches enable the interactive exploration of narrative structures through computational methods and visual interfaces that facilitate human interpretation. The field faces challenges in scalability, interactivity, knowledge integration, and evaluation standardization, yet offers promising opportunities across news analysis, intelligence, scientific literature exploration, and social media analysis. Through the combination of computational and human insight, INA addresses complex challenges in narrative sensemaking.
Read the full abstract ▾
Computation and Language
Predict the Retrieval! Test time adaptation for Retrieval Augmented Generation
Xin Sun, Zhongqi Chen, Qiang Liu et al. (+5 more)
Retrieval-Augmented Generation (RAG) has emerged as a powerful approach for enhancing large
language models' question-answering capabilities through the integration of external knowledge. However, when adapting RAG systems to specialized domains, challenges arise from distribution shifts, resulting in suboptimal generalization performance. In this work, we propose TTARAG, a test-time adaptation method that dynamically updates the
language model's parameters during inference to improve RAG system performance in specialized domains. Our method introduces a simple yet effective approach where the model learns to predict retrieved content, enabling automatic parameter adjustment to the target domain. Through extensive experiments across six specialized domains, we demonstrate that TTARAG achieves substantial performance improvements over baseline RAG systems. Code available at https://github.com/sunxin000/TTARAG.
Read the full abstract ▾
All authors Xin Sun, Zhongqi Chen, Qiang Liu, Shu Wu, Bowen Song, Weiqiang Wang, Zilei Wang, Liang Wang
Computation and Language
Hierarchical Orthogonal Residual Spread for Precise Massive Editing in Large Language Models
Xiaojie Gu, Guangxu Chen, Yuheng Yang et al. (+2 more)
Large
language models (LLMs) exhibit exceptional performance across various domains, yet they face critical safety concerns. Model editing has emerged as an effective approach to mitigate these issues. Existing model editing methods often focus on optimizing an information
matrix that blends new and old knowledge. While effective, these approaches can be computationally expensive and may cause conflicts. In contrast, we shift our
attention to Hierarchical Orthogonal Residual SprEad of the information
matrix, which reduces noisy gradients and enables more stable edits from a different perspective. We demonstrate the effectiveness of our method HORSE through a clear theoretical comparison with several popular methods and extensive experiments conducted on two
datasets across multiple LLMs. The results show that HORSE maintains precise massive editing across diverse scenarios. The code is available at https://github.com/XiaojieGu/HORSE
Read the full abstract ▾
All authors Xiaojie Gu, Guangxu Chen, Yuheng Yang, Jingxin Han, Andi Zhang
Computation and Language
The unreasonable effectiveness of pattern matching
Gary Lupyan and Blaise Agüera y Arcas
We report on an astonishing ability of large
language models (LLMs) to make sense of "Jabberwocky"
language in which most or all content words have been randomly replaced by nonsense strings, e.g., translating "He dwushed a ghanc zawk" to "He dragged a spare chair". This result addresses ongoing controversies regarding how to best think of what LLMs are doing: are they a
language mimic, a database, a blurry version of the Web? The ability of LLMs to recover meaning from structural patterns speaks to the unreasonable effectiveness of pattern-matching. Pattern-matching is not an alternative to "real" intelligence, but rather a key ingredient.
Read the full abstract ▾
Computation and Language
Relational Linearity is a Predictor of Hallucinations
Yuetian Lu, Yihong Liu, Sebastian Gerstner et al. (+3 more)
Hallucination is a central failure mode of
language models (LMs). We focus on
hallucinations in response to questions like: "Which instrument did Glenn Gould play?", but we ask these questions for synthetic entities designed to be unknown to the model. We find that LMs like Gemma-7B-IT frequently hallucinate, i.e., they have difficulty recognizing that the hallucinated fact is not part of their knowledge. Based on the idea of linear relational
embeddings, we put forward the following hypothesis. (i) Due to the abstract scheme that is used to represent them, LMs can easily produce plausible objects for non-existing subjects of linear relations, which can lead to
hallucinations. (ii) For nonlinear relations, this mechanism for producing an object is not available and so a
hallucination is easier to avoid. To test this hypothesis, we create SynthHal, a synthetic unknown-entity
benchmark for 15 relations. We find that across four instruction-tuned models, relational linearity is a strong predictor of models hallucinating an object for an unknown subject vs refusing to give an answer, with correlations $r \in [.58, .84]$. While this is not direct evidence for the hypothesized causal mechanism, it is suggestive and opens up a new line of inquiry into understanding LM
hallucinations.
Read the full abstract ▾
All authors Yuetian Lu, Yihong Liu, Sebastian Gerstner, Lea Hirlimann, Jonas Rohweder, Hinrich Schütze
Computation and Language · Cross-listed from cs.IR
Isotropy-Optimized Contrastive Learning for Semantic Course Recommendation
Ali Khreis, Anthony Nasr and Yusuf Hilal
This paper presents a semantic course recommendation system for students using a self-supervised contrastive learning approach built upon BERT (Bidirectional Encoder Representations from
Transformers). Traditional BERT
embeddings suffer from anisotropic representation spaces, where course descriptions exhibit high cosine similarities regardless of semantic relevance. To address this limitation, we propose a contrastive learning framework with data augmentation and isotropy regularization that produces more discriminative
embeddings. Our system processes student text queries and recommends Top-N relevant courses from a curated
dataset of over 500 engineering courses across multiple faculties. Experimental results demonstrate that our fine-tuned model achieves improved
embedding separation and more accurate course recommendations compared to vanilla BERT baselines.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
PubMed-OCR: PMC Open Access OCR Annotations
Hunter Heidenreich, Yosheb Getachew, Olivia Dinica et al. (+1 more)
PubMed-OCR is an OCR-centric corpus of scientific articles derived from PubMed Central Open Access PDFs. Each page image is annotated with Google Cloud
Vision and released in a compact JSON schema with word-, line-, and paragraph-level bounding boxes. The corpus spans 209.5K articles (1.5M pages; ~1.3B words) and supports layout-aware modeling, coordinate-grounded QA, and evaluation of OCR-dependent pipelines. We analyze corpus characteristics (e.g., journal coverage and detected layout features) and discuss limitations, including reliance on a single OCR engine and heuristic line reconstruction. We release the data and schema to facilitate downstream research and invite extensions.
Read the full abstract ▾
All authors Hunter Heidenreich, Yosheb Getachew, Olivia Dinica, Ben Elliott
Computation and Language
Evaluating LLM Behavior in Hiring: Implicit Weights, Fairness Across Groups, and Alignment with Human Preferences
Morgane Hoffmann, Emma Jouffroy, Warren Jouanneau et al. (+2 more)
General-purpose Large
Language Models (LLMs) show significant potential in recruitment applications, where decisions require
reasoning over unstructured text, balancing multiple criteria, and inferring fit and competence from indirect productivity signals. Yet, it is still uncertain how LLMs assign importance to each attribute and whether such assignments are in line with economic principles, recruiter preferences or broader societal norms. We propose a framework to evaluate an LLM's decision logic in recruitment, by drawing on established economic methodologies for analyzing human hiring behavior. We build synthetic
datasets from real freelancer profiles and project descriptions from a major European online freelance marketplace and apply a full factorial design to estimate how a LLM weighs different match-relevant criteria when evaluating freelancer-project fit. We identify which attributes the LLM prioritizes and analyze how these weights vary across project contexts and demographic subgroups. Finally, we explain how a comparable experimental setup could be implemented with human recruiters to assess
alignment between model and human decisions. Our findings reveal that the LLM weighs core productivity signals, such as skills and experience, but interprets certain features beyond their explicit matching value. While showing minimal average discrimination against minority groups, intersectional effects reveal that productivity signals carry different weights between demographic groups.
Read the full abstract ▾
All authors Morgane Hoffmann, Emma Jouffroy, Warren Jouanneau, Marc Palyart, Charles Pebereau
Computation and Language
Reward Modeling for Scientific Writing Evaluation
Furkan Şahinuç, Subhabrata Dutta and Iryna Gurevych
Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and
reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose
benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines
reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks
Weiyi Wang, Xinchi Chen, Jingjing Gong et al. (+2 more)
Recent LLM-for-Space systems address mission planning, scheduling, operations support, simulator control, and autonomy, but their evaluations use different task contracts, control settings, simulators, and success criteria. We introduce AstroAgentBench, a seven-family
benchmark for executable space mission planning in the domains of scheduling, observation planning, constellation design, and relay support. For each case, an
agent submits a planning artifact that is checked by an external verifier for schema, timing, geometry, resources, and mission value. Results report validity and normalized scores, with comparisons to task-specific solver references. Across five LLM
agent systems and 35 held-out cases, the strongest systems approach or exceed solver-reference scores on several families, while weaker systems often fail to produce high-value valid plans and even strong systems lose quality on
geometric, product-level, or design-heavy tasks. Trace analyses separate two failure points: task-contract misformulation and weak solution construction. Successful runs instead calibrate
agent-written implementations against verifier feedback and adapt search to case-specific structure. Ablations show that procedure injection and
memory accumulation help selectively, when they supply the missing formulation, calibration, or search support.
Read the full abstract ▾
All authors Weiyi Wang, Xinchi Chen, Jingjing Gong, Xuanjing Huang, Xipeng Qiu
Computation and Language
How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting
Parker Seegmiller, Joseph Gatto, Sarah E. Greer et al. (+4 more)
Large
language models (LLMs) show promise in drafting responses to patient portal messages, yet their integration into clinical workflows raises various concerns, including whether they would actually save clinicians time and effort in their portal workload. We investigate LLM
alignment with individual clinicians through a comprehensive evaluation of the patient message response drafting task. We develop a novel taxonomy of thematic elements in clinician responses and propose a novel evaluation framework for assessing clinician editing load of LLM-drafted responses at both content and theme levels. We release an expert-annotated
dataset and conduct large-scale evaluations of local and commercial LLMs using various adaptation techniques including thematic prompting,
retrieval-augmented generation, supervised fine-tuning, and direct preference optimization. Our results reveal substantial epistemic uncertainty in aligning LLM drafts with clinician responses. While LLMs demonstrate capability in drafting certain thematic elements, they struggle with clinician-aligned generation in other themes, particularly question asking to elicit further information from patients. Theme-driven adaptation strategies yield improvements across most themes. Our findings underscore the necessity of adapting LLMs to individual clinician preferences to enable reliable and responsible use in patient-clinician communication workflows.
Read the full abstract ▾
All authors Parker Seegmiller, Joseph Gatto, Sarah E. Greer, Ganza Belise Isingizwe, Rohan Ray, Timothy E. Burdick, Sarah Masud Preum
Computation and Language · Cross-listed from cs.LG
Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models
Chuanyue Yu, Jiahui Wang, Yuhan Li et al. (+6 more)
Diffusion Language Models (DLMs) have recently demonstrated remarkable capabilities in natural
language processing tasks. However, the potential of
Retrieval-Augmented Generation (RAG), which shows great successes for enhancing large
language models (LLMs), has not been well explored, due to the fundamental difference between LLM and DLM decoding. To fill this critical gap, we systematically test the performance of DLMs within the RAG framework. Our findings reveal that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision. We identify a key underlying issue: Response Semantic Drift (RSD), where the generated answer progressively deviates from the query's original semantics, leading to low precision content. We trace this problem to the denoising strategies in DLMs, which fail to maintain semantic
alignment with the query throughout the iterative denoising process. To address this, we propose Semantic-Preserving
REtrieval-Augmented
Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy. By actively guiding the denoising trajectory, SPREAD ensures the generation remains anchored to the query's semantics and effectively suppresses drift. Experimental results demonstrate that SPREAD significantly enhances the precision and effectively mitigates RSD of generated answers within the RAG framework.
Read the full abstract ▾
All authors Chuanyue Yu, Jiahui Wang, Yuhan Li, Heng Chang, Ge Lan, Qingyun Sun, Jia Li, Jianxin Li, Ziwei Zhang
Computation and Language
Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models
Guoming Ling, Zhongzhan Huang, Yupei Lin et al. (+4 more)
Chain-of-Thought
reasoning has significantly enhanced the problem-solving capabilities of Large
Language Models. Unfortunately, current models generate
reasoning steps sequentially without foresight, often becoming trapped in suboptimal
reasoning paths with redundant steps. In contrast, we introduce Neural Chain-of-Thought Search (NCoTS), a framework that reformulates
reasoning as a dynamic search for the optimal thinking strategy. By quantitatively characterizing the solution space, we reveal the existence of sparse superior
reasoning paths that are simultaneously more accurate and concise than standard outputs. Our method actively navigates towards these paths by evaluating candidate
reasoning operators using a dual-factor heuristic that optimizes for both correctness and computational cost. Consequently, NCoTS achieves a Pareto improvement across diverse
reasoning benchmarks, boosting accuracy by over 3.5% while reducing generation length by over 22%. Our code and data are available at https://github.com/MilkThink-Lab/Neural-CoT-Search.
Read the full abstract ▾
All authors Guoming Ling, Zhongzhan Huang, Yupei Lin, Junxin Li, Shanshan Zhong, Hefeng Wu, Liang Lin
Computation and Language
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming
Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi et al. (+3 more)
Large
Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic
reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-
language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete
algorithms. Beyond pass/fail metrics, we diagnose
reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a
dataset of 83 ICPC-style problems with gold editorials and full test suites, and evaluate 19 LLMs, arguing that future
benchmarks should explicitly separate problem solving from implementation.
Read the full abstract ▾
All authors Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi, Jan Christian Blaise Cruz, Steven Halim, Alham Fikri Aji
Computation and Language
F-Actor: Controllable Conversational Behaviour in Full-Duplex Models
Maike Züfle, Ondrej Klejch, Nicholas Sanders et al. (+3 more)
Spoken conversational systems require more than accurate
speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational
speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the
language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code is released to enable reproducible research on controllable full-duplex
speech systems.
Read the full abstract ▾
All authors Maike Züfle, Ondrej Klejch, Nicholas Sanders, Jan Niehues, Alexandra Birch, Tsz Kin Lam
Computation and Language
Membership Inference on LLMs in the Wild
Jiatong Yi and Yanyang Li
Membership Inference Attacks (MIAs) act as a crucial
auditing tool for the opaque training data of Large
Language Models (LLMs). However, existing techniques predominantly rely on inaccessible model internals (e.g., logits) or suffer from poor generalization across domains in strict black-box settings where only generated text is available. In this work, we propose SimMIA, a robust MIA framework tailored for this text-only regime by leveraging an advanced sampling strategy and scoring mechanism. Furthermore, we present WikiMIA-25, a new
benchmark curated to evaluate MIA performance on modern proprietary LLMs. Experiments demonstrate that SimMIA achieves state-of-the-art results in the black-box setting, rivaling baselines that exploit internal model information.
Read the full abstract ▾
Computation and Language
One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking
Malin Astrid Larsson, Harald Fosen Grunnaleite and Vinay Setty
Large
language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines rather than isolated components. While large proprietary models achieve strong performance, their closed weights, complexity, and high costs limit sustainability. Fine-tuning smaller open weight models for individual AFC tasks can help but requires multiple specialized models resulting in high costs. We propose multi-task learning (MTL) as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly. Using small decoder-only LLMs (e.g., Qwen3-4b), we explore three MTL strategies: classification heads, causal
language modeling heads, and instruction-tuning, and evaluate them across model sizes, task orders, and standard non-LLM baselines. While multitask models do not universally surpass single-task baselines, they yield substantial improvements, achieving up to 44%, 54%, and 31% relative gains for claim detection, evidence re-ranking, and stance detection, respectively, over zero-/few-shot settings. Finally, we also provide practical, empirically grounded guidelines to help practitioners apply MTL with LLMs for automated fact-checking.
Read the full abstract ▾