Computation and Language
Quantum Compositional NLP for Arabic: Grammar, Morphology, and Word Sense in Circuit Topology
Wajahath Mohammed
We present the first application of pregroup grammar-based
quantum compositional natural
language processing (QNLP) to Arabic; a morphologically rich, free-word-order
language whose structural complexity provides a uniquely demanding testbed for theories of meaning composition in
quantum circuits. Our system converts Arabic sentences into
quantum circuits whose
topology mirrors grammatical structure: subjects, verbs, and objects become
quantum gates, and the typed dependencies between them (the pregroup grammar) determine how those gates are wired together. We conduct three controlled experiments spanning word order, morphological tense, and verb sense disambiguation, comparing
quantum circuit methods against classical baselines including AraVec (Arabic word
embeddings) and AraBERT (a pre-trained Arabic
transformer).
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
Deep Pankajbhai Mehta
Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a
token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reasoning emerges from constrained inference manifolds in large language models
Yanbiao Ma, Fei Luo, Linfeng Zhang et al. (+10 more)
Reasoning in large
language models is predominantly evaluated through labeled
benchmarks, conflating task performance with the quality of internal inference. Here we study
reasoning as an intrinsic dynamical process by examining the
evolution of internal representations during inference. We find that inference-time dynamics consistently self-organize into low-dimensional
manifolds embedded within high-dimensional representation spaces. we find that such
geometric compression, although pervasive, is not sufficient for stable or reliable
reasoning. Instead, effective
reasoning dynamics emerge within a constrained structural regime characterized by three conditions: adequate representational expressivity, spontaneous
manifold compression, and preservation of non-degenerate information volume within the compressed subspace. Models outside this regime exhibit characteristic pathological inference dynamics. Based on these insights, we introduce a unified, label-free diagnostic computed solely from internal dynamics. These findings suggest that
reasoning in LLMs is fundamentally governed by
geometric and informational constraints, offering a complementary framework to
benchmark-centric assessment.
Read the full abstract ▾
All authors Yanbiao Ma, Fei Luo, Linfeng Zhang, Chuangxin Zhao, Mingxuan Wang, Yinan Wu, Zhe Qian, Yang Lu, Long Chen, Zhao Cao, Xiaoshuai Hao, Ji-Rong Wen, Jungong Han
Computation and Language · Cross-listed from cs.AI
State Representation and Termination for Recursive Reasoning Systems
Debashis Guha, Amritendu Mukherjee, Sanjay Kukreja et al. (+1 more)
Recursive
reasoning systems alternate between acquiring new evidence and refining an accumulated understanding. Two design choices are typically left implicit: how to represent the evolving
reasoning state, and when to stop iterating. This paper addresses both. We represent the
reasoning state as an epistemic state graph encoding extracted claims, evidential relations, open questions, and confidence weights. We define the order-gap as the distance between the states reached by expand-then-consolidate versus consolidate-then-expand; a small order-gap suggests that the two orderings agree and further iteration is unlikely to help. Our main result gives a necessary and sufficient condition for the linearised order-gap to be non-degenerate near the fixed point, showing when the criterion is informative rather than algebraically vacuous. This is a local condition, not a global convergence guarantee. We apply the framework to recursive
reasoning systems and sketch its application to
agent loops, tree-of-thought
reasoning,
theorem proving, and continual learning.
Read the full abstract ▾
All authors Debashis Guha, Amritendu Mukherjee, Sanjay Kukreja, Tarun Kumar
Computation and Language · Cross-listed from cs.CR
Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection
Xulin Hu, Che Wang, Wei Yang Bryan Lim et al. (+2 more)
Representation Engineering analyses often characterize refusal using static directions extracted from terminal or pooled representations. We ask whether this view misses how refusal is constructed across layer-
token positions. Using causal tracing, we identify a Refusal Trajectory: a sparse upstream activation pattern that often persists even when attacks such as GCG suppress terminal refusal signals. Based on this observation, we propose SALO (Sparse Activation Localization Operator), a lightweight white-box detector that operates on raw hidden-state volumes from a selected layer window. Across Qwen, Llama, and Mistral models, SALO improves jailbreak detection on several attack families under a fixed XSTest-calibrated operating point. We further analyze static RepE-style baselines, ROI sensitivity, adaptive GCG attacks, and encoded-input boundary cases, clarifying both the promise and limitations of refusal-trajectory monitoring.
Read the full abstract ▾
All authors Xulin Hu, Che Wang, Wei Yang Bryan Lim, Jianbo Gao, Zhong Chen
Computation and Language · Cross-listed from cs.LG
Prescriptive Scaling Laws for Data Constrained Training
Justin Lovelace, Christian Belardi, Srivatsa Kundurthy et al. (+2 more)
Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training
token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($\lambda=1.0$) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.
Read the full abstract ▾
All authors Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger
Computation and Language
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida et al. (+6 more)
Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering removes this sensitivity: decomposing the judgement matters more than the judge model itself. To support this claim, we introduce Prosa, the first real user multi-turn Brazilian Portuguese chat
benchmark: 1,000 WildChat conversations scored by three judges from three model families on 16 models. Under filtered rubric scoring the three judges agree on every one of the 16 ranks, whereas under holistic scoring they agree on only 7 of 16. Additionally, the rubric filtering pipeline increases the average score gap between neighbouring models by 47%, thereby improving Prosa's discriminative power. Evaluating a new model on Prosa costs approximately $2.1 when using Gemini 3 Flash as the judge. We release the
benchmark and the filtering code to ensure that future models can be assessed under identical conditions. These artifacts also make our rubric-based scoring method reusable beyond Prosa, supporting other open-ended evaluation settings.
Read the full abstract ▾
All authors Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida, Hugo Abonizio, Thiago Laitz, Ramon Pires, Marcos Piau, Celio Larcher, Rodrigo Nogueira
Computation and Language
Where Do Prompt Perturbations Break Generation? A Segment-Level View of Robustness in LoRA-Tuned Language Models
Zhuoyun Li, Boxuan Wang, Jinwei Hu et al. (+6 more)
Large
language models are sensitive to minor
prompt perturbations, yet existing robustness methods usually enforce consistency at the whole-sequence level. This holistic view can hide an important failure mode: a perturbed response may remain globally similar to the clean one while drifting on a critical entity, relation, or conclusion. We introduce S$^2$R$^2$, a segment-level framework for robust LoRA fine-tuning. S$^2$R$^2$ decomposes clean and perturbed generations into semantic segments, aligns them with an optimal-transport objective, and penalises the segments with the largest meaning drift. To connect this output-side objective with model adaptation, we add an adapter-stability regulariser motivated by segment-level
attention reallocation, using LoRA norm control as a tractable proxy for limiting perturbation-amplified evidence shifts. A PAC-
Bayesian complexity view further explains why controlling adapter growth may support transfer beyond observed perturbations. Experiments on summarisation
benchmarks show that S$^2$R$^2$ improves robustness under typographical noise, deletion, synonym replacement, and paraphrasing, while maintaining competitive clean performance and stronger cross-
dataset transfer than consistency-based baselines.
Read the full abstract ▾
All authors Zhuoyun Li, Boxuan Wang, Jinwei Hu, Zhenglin Huang, Qisong He, Xinmiao Huang, Guangliang Cheng, Xiaowei Huang, Yi Dong
Computation and Language
Fine-Tuning Pre-Trained Code Models for AI-Generated Code Detection
Jany-Gabriel Ispas and Sergiu Nisioi
This paper describes the system submitted by team Archaeology to SemEval-2026 Task~13 on AI-generated code detection. The shared task consists of three subtasks; we participate in Subtask-A (binary classification: human-written vs.\ AI-generated code) and Subtask-B (11-class attribution of the generating model). Starting from a TF-IDF and Logistic Regression baseline, we fine-tune four pre-trained code models (CodeBERT, GraphCodeBERT, UniXcoder, and CodeT5+) with separate strategies for each subtask. For Subtask-A, we use leave-one-
language-out cross-validation, code augmentation, chunked inference with trimmed-mean aggregation, and threshold calibration on a difficult
dataset. For Subtask-B, we use sandwich
token packing, class-balanced loss, and multi-seed ensembling with test-time augmentation. Our best submissions obtain macro-F1 scores of 0.737 on Subtask-A (6th/81 teams) and 0.422 on Subtask-B (7th/34 teams).
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Led to Mislead: Adversarial Content Injection for Attacks on Neural Ranking Models
Amin Bigdeli, Amir Khosrojerdi, Radin Hamidi Rad et al. (+3 more)
Neural Ranking Models (NRMs) are central to modern information
retrieval but remain highly vulnerable to
adversarial manipulation. Existing attacks often rely on heuristics or surrogate models, limiting effectiveness and transferability. We propose CRAFT, a supervised framework for black-box
adversarial rank attacks powered by large
language models (LLMs). CRAFT operates in three stages:
adversarial dataset generation via
retrieval-augmented generation and self-refinement, supervised fine-tuning on curated
adversarial examples, and preference-guided optimization to align generations with rank-promotion objectives. Extensive experiments on the MS MARCO passage
dataset, TREC Deep Learning 2019, and TREC Deep Learning 2020
benchmarks show that CRAFT significantly outperforms state-of-the-art baselines, achieving higher promotion rates and rank boosts while preserving fluency and semantic fidelity. Moreover, CRAFT transfers effectively across diverse ranking architectures, including cross-encoder,
embedding-based, and LLM-based rankers, underscoring vulnerabilities in real-world
retrieval systems. This work provides a principled framework for studying
adversarial threats in NRMs, underscores the risks of generative AI in rank manipulation, and provides a foundation for developing more robust
retrieval systems. To support reproducibility, we publicly release our source code, trained models, and
prompt templates.
Read the full abstract ▾
All authors Amin Bigdeli, Amir Khosrojerdi, Radin Hamidi Rad, Morteza Zihayat, Charles L. A. Clarke, Ebrahim Bagheri
Computation and Language · Cross-listed from cs.SE
Feedback-Normalized Developer Memory for Reinforcement-Learning Coding Agents: A Safety-Gated MCP Architecture
Mehmet Iscan
Large
language model (LLM) coding
agents increasingly operate over repositories, terminals, tests, and execution traces across long software-engineering episodes. Persistent
memory is useful, but static vector stores or generic
retrieval-augmented generation (RAG) are insufficient for reinforcement-learning (RL) code development, where small details can alter Bellman targets, terminal masks, gradient flow, or validation claims. This paper presents RL Developer
Memory, a local-first, Model Context Protocol (MCP)-native developer-
memory architecture for RL coding
agents. It treats
memory selection as a logged contextual decision process: issue_match ranks candidates and records telemetry, issue_feedback maps raw labels to bounded rewards, and issue_record_resolution links verified resolutions to earlier
retrieval events. A deterministic ranker remains deployed, while a contextual-bandit residual policy runs in shadow mode and can affect canary behavior only through conservative off-policy-evaluation (OPE) gates. RL/control
memories require theory-to-code metadata and review-gated governance. The system is evaluated on a deterministic 200-case
benchmark with RL
algorithm bugs, hard negatives, review-gated RL/control cases, and low-risk failures. In the same-commit comparison, deterministic control and full shadow/OPE both achieve 80.0% expected-decision accuracy and 100.0% hard-negative suppression; the full configuration adds learning telemetry rather than accuracy gain. Static validation passed 11/11 checks; dynamic integration passed 10/10 cases. The evidence reports limits: active learned-policy deployment and official-client MCP interoperability are unsupported, live full-configuration latency regresses, and 40 residual non-RL failures remain. The contribution is an auditable
memory-control architecture with explicit claim boundaries, not a universal coding-
agent improvement claim.
Read the full abstract ▾
Computation and Language
Automated Interpretability and Feature Discovery in Language Models with Agents
Arnau Marin-Llobet and Javier Ferrando
We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large
language models. The system runs two coupled loops: (1) explanation refinement, where an
agent proposes competing hypotheses and iteratively tests them with targeted
prompt controls and a multi-metric evaluation; and (2) feature discovery, where an
agent generates
prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP
neurons in weight-sparse
transformers, our
agent improves over one-shot auto-interpretations, discovers
language-specific and safety-relevant features, and produces auditable explanation traces, showing that
agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.
Read the full abstract ▾
Computation and Language
The grip of grammar on meaning uncertainty: cross-linguistic evidence, neural correlates, and clinical relevance
Rui He, Claudio Palominos, Samuele Vallisa et al. (+16 more)
Isolated word meanings are inherently uncertain. This uncertainty reduces when they are combined and anchored in context. We propose that grammar compresses meaning uncertainty cross-linguistically, which is reflected in
brain and selectively disrupted in disorders. Compression was operationalized as the relative difference between non-contextual surprisal estimated from lexical frequency, and contextual surprisal from grammar-sensitive models. In narratives from 20
languages, contextual surprisal reduced frequency-based surprisal. This reduction closely tracked the surprisal cost of reversing word order, and scaled with richer, non-redundant lexis as organized by more complex but optimal dependency structure. During fMRI, surprisal and its reduction explained BOLD activity for comprehension and production in overlapping but distinct regions. Uncertainty reduction was significantly attenuated in aphasia, dementia, and schizophrenia, but remained intact where primary deficit is not
language. These findings position uncertainty reduction via grammar as a foundational concept that illuminates principles,
brain basis, and disruptions of
language.
Read the full abstract ▾
All authors Rui He, Claudio Palominos, Samuele Vallisa, Ni Yang, Han Zhang, Miguel Ángel Santos Santos, Neguine Rezaii, Sergi Valero, Yonghua Huang, Huan Li, Hong Jiang, Yongjun Peng, Maria Francisca Alonso-Sánchez, Frederike Stein, Tilo Kircher, Philipp Homan, Iris Sommer, Lena Palaniyappan, Wolfram Hinzen
Computation and Language · Cross-listed from cs.CV
MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
Yin Zhang, Jiaxuan Zhao, Zonghan Wu et al. (+5 more)
Vision-
Language Models (VLMs) frequently suffer from visual perception errors and
hallucinations that compromise answer accuracy in complex
reasoning tasks. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals. Despite their effectiveness, prevailing RLVR methods face two critical limitations. First, much of the sampling budget is wasted on trajectories doomed to fail due to early visual description errors. Second, sparse rewards cannot distinguish whether failures stem from visual perception or
reasoning stages. We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal. This enables intelligent budget allocation toward high-potential trajectories via forking, while decoupled training provides independent MI-based rewards for visual perception optimization, resolving reward blindness. Experiments on six
vision-
language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy and successfully surpasses the performance of sampling 16 complete trajectories using only 10 pre-samples with top-6 selection (25% fewer complete trajectories). Our code is available at: https://anonymous.4open.science/r/mirl-main/.
Read the full abstract ▾
All authors Yin Zhang, Jiaxuan Zhao, Zonghan Wu, Zengxiang Li, Junfeng Fang, Kun Wang, Qingsong Wen, Yilei Shao
Computation and Language
FT-RAG: A Fine-grained Retrieval-Augmented Generation Framework for Complex Table Reasoning
Zebin Guo, Weidong Geng and Ruichen Mao
Retrieval-Augmented Generation (RAG) enhances Large
Language Models (LLMs) by grounding responses in external knowledge during inference. However, conventiona RAG systems under-perform on structured tabular data, largely due to coarse
retrieval granularity and insufficient table semantic comprehension. To address these limitations, we introduce FT-RAG, a fine-grained framework that employs knowledge association by decomposing tables into entry-level semantic units to construct a structured graph. FT-RAG employs a structural neighbor expansion mechanism to find semantically connected entities during graph
retrieval, followed by multi-modal fusion to consolidate the context of table
retrieval results. Further, to address the scarcity of specialized
datasets in this domain, we introduce Multi-Table-RAG-Lib, a
benchmark comprising 9870 QA pairs with high complexity and difficulty, curated to demand multi-table integration and text-table information fusion for
reasoning. FT-RAG surpasses top-performing baselines across all metrics, achieving a 23.5% and 59.2% improvement in table-level and
cell-level Hit Rates, respectively. Generation performance also sees a remarkable 62.2% increase in exact value accuracy recall. These metrics verify the framework's effectiveness in factual grounding across both pure tabular and heterogeneous table-text contexts. Therefore, our method establishes a new state-of-the-art performance for complex
reasoning over mixed-modality documents.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning
Tianshi Zheng, Rui Wang, Xiyun Li et al. (+5 more)
Frontier scientific
reasoning is rapidly emerging as a key foundation for advancing AI
agents in automated scientific discovery. Deep research
agents offer a promising approach to this challenge. These models develop robust problem-solving capabilities through post-training on information-seeking tasks, which are typically curated via knowledge graph construction or iterative web browsing. However, these strategies face inherent limitations in frontier science, where domain-specific knowledge is scattered across sparse and heterogeneous academic sources, and problem solving requires sophisticated computation and
reasoning far beyond factual recall. To bridge this gap, we introduce SciResearcher, a fully automated agentic framework for frontier-science data construction. SciResearcher synthesizes diverse conceptual and computational tasks grounded in academic evidence, while eliciting information acquisition, tool-integrated
reasoning, and long-horizon capabilities. Leveraging the curated data for supervised fine-tuning and agentic reinforcement learning, we develop SciResearcher-8B, an
agent foundation model that achieves 19.46% on the HLE-Bio/Chem-Gold
benchmark, establishing a new state of the art at its parameter scale and surpassing several larger proprietary
agents. It further achieves 13-15% absolute gains on SuperGPQA-Hard-Biology and TRQA-Literature
benchmarks. Overall, SciResearcher introduces a new paradigm for automated data construction for frontier scientific
reasoning and offers a scalable path toward future scientific
agents.
Read the full abstract ▾
All authors Tianshi Zheng, Rui Wang, Xiyun Li, Kelvin Kiu Wai Tam, Newt Nguyen Kim Hue Nam, Wei Fan, Yangqiu Song, Tianqing Fang
Computation and Language
ReMedi: Reasoner for Medical Clinical Prediction
Yushi Cao, Yiming Chen, Hongchao Jiang et al. (+2 more)
Predicting future clinical outcomes from electronic health records (EHR) remains challenging due to the complexity and heterogeneity of patient data. LLMs have shown strong potential for such predictive tasks, yet existing approaches mainly focus on enhancing medical knowledge through
distillation or RAG while relying on the model's internal ability to interpret contextual information. In this work, we present ReMedi (Reasoner for Medical Clinical Prediction), a framework for improving clinical outcome prediction from EHR. ReMedi generates rationale-answer pairs using a challenging sample regeneration mechanism for complex clinical questions, which leverages ground-truth answers as hints to enhance
reasoning for further fine-tuning and preference tuning. ReMedi integrates ground-truth outcome guidance into the preference data construction loop, regenerating rationale-answer variants. By tuning on these rationale-answer pairs, the model improves its predictive performance. Experiments on multiple EHR prediction tasks demonstrate substantial gains of up to 19.9 percent over state-of-the-art baselines in terms of F1 score, underscoring ReMedi's effectiveness in real-world clinical prediction.
Read the full abstract ▾
All authors Yushi Cao, Yiming Chen, Hongchao Jiang, Hung-yi Lee, Robby T. Tan
Computation and Language
Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models
William Guey, Wei Zhang, Pierrick Bougault et al. (+4 more)
Large
language models (LLMs) are rapidly being integrated into high-stakes public safety systems, including emergency call triage and dispatch decision support, yet their demographic
fairness in this context remains largely untested. Here we introduce a cross-lingual audit framework that operationalizes the Police Priority Dispatch System as a five-level ordinal classification task and applies a controlled minimal-pair design to isolate the effect of demographic cues. Across 19,800 model outputs spanning 11 frontier models, 15 scenario pairs, three demographic categories (religious appearance, gender, and race), and two
languages (English and Mandarin Chinese), we find that demographic bias emerges systematically when incident severity is ambiguous but largely disappears when the operational priority is clearly determined by call content. Bias magnitude varies by demographic axis, with the largest effects observed for religious appearance, followed by gender and race. Critically, bias does not transfer consistently across
languages: gender bias is substantially amplified in Mandarin Chinese, whereas race bias is more pronounced in English, revealing cross-lingual asymmetries that aggregate analyses obscure. In several scenarios, demographic cues produce counter-directional effects, challenging simple stereotype-amplification accounts of model behavior. These findings suggest that bias in LLM-based dispatch is not a fixed property of models alone, but arises from the interaction between demographic signals, contextual ambiguity, and
language. Beyond these empirical results, the proposed framework provides a scalable audit infrastructure that enables deploying agencies to evaluate candidate models on jurisdiction-relevant scenarios prior to real-world adoption.
Read the full abstract ▾
All authors William Guey, Wei Zhang, Pierrick Bougault, Yi Wang, Bertan Ucar, Vitor D. de Moura, José O. Gomes
Computation and Language
Artificial intelligence language technologies in multilingual healthcare: Grand challenges ahead
Vicent Briva-Iglesias
AI
language technologies (AILTs), increasingly enabled by large
language models (LLMs), are becoming embedded in
multilingual healthcare workflows for translation, rewriting, documentation, interpreting, and messaging in
language-discordant settings. Yet fluent output is not the same as clinically safe or equitable communication: performance varies across
languages, accents, tasks, and workflows, and efficiency gains can hide errors, reduce traceability, and shift responsibility across clinicians, translators, interpreters, and health systems. This narrative review synthesises recent peer-reviewed evidence across written communication, spoken communication, and emerging agentic workflows. Using the Human-Centered AI
Language Technology (HCAILT) lens, it examines capabilities, evaluation practices, implementation patterns, and recurrent errors through reliability, safety culture, and trustworthiness. We identify key convergences and contradictions in the literature and propose seven grand challenges for the next phase of research and deployment. Progress, we argue, requires not only better models but also accountable sociotechnical design, calibrated human oversight, and stronger collaboration across MT/NLP, translation studies, HCI, clinical practice, implementation science, and policy.
Read the full abstract ▾
Computation and Language
Hallucinations Undermine Trust; Metacognition is a Way Forward
Gal Yona, Mor Geva and Yossi Matias
Despite significant strides in factual reliability, errors – often termed
hallucinations – remain a major concern for generative AI, especially as LLMs are increasingly expected to be helpful in more complex or nuanced setups. Yet even in the simplest setting – factoid question-answering with clear ground truth-frontier models without external tools continue to hallucinate. We argue that most factuality gains in this domain have come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). We
conjecture that the latter is inherently difficult: models may lack the discriminative power to perfectly separate truths from errors, creating an unavoidable tradeoff between eliminating
hallucinations and preserving utility. This tradeoff dissolves under a different framing. If we understand
hallucinations as confident errors – incorrect information delivered without appropriate qualification – a third path emerges beyond the answer-or-abstain dichotomy: expressing uncertainty. We propose faithful uncertainty: aligning linguistic uncertainty with intrinsic uncertainty. This is one facet of metacognition – the ability to be aware of one's own uncertainty and to act on it. For direct interaction, acting on uncertainty means communicating it honestly; for agentic systems, it becomes the control layer governing when to search and what to trust. Metacognition is thus essential for LLMs to be both trustworthy and capable; we conclude by highlighting open problems for progress towards this objective.
Read the full abstract ▾
Computation and Language
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer et al. (+32 more)
Evaluating large
language models (LLMs) for medical applications remains challenging due to
benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted
datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30
benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical
reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier
reasoning models (Gemini 3 Pro Preview, GPT-5.1, & GPT-5.2) achieve the highest performance across both
benchmarks, most frontier proprietary models are significantly more
token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). A subset of our evals (Medmarks-T) can be directly used as reinforcement learning environments to post-train LLMs for medical
reasoning. Code is available at https://github.com/MedARC-AI/Medmarks
Read the full abstract ▾
All authors Benjamin Warner, Ratna Sagari Grandhi, Max Kieffer, Aymane Ouraq, Saurav Panigrahi, Geetu Ambwani, Kunal Bagga, Nikhil Khandekar, Arya Hariharan, Nishant Mishra, Manish Ram, Shamus Sim Zi Yang, Ahmed Essouaied, Adepoju Jeremiah Moyondafoluwa, Robert Scholz, Bofeng Huang, Molly Beavers, Srishti Gureja, Anish Mahishi, Sameed Khan, Maxime Griot, Hunar Batra, Jean-Benoit Delbrouck, Siddhant Bharadwaj, Ronald Clark, Ashish Vashist, Anas Zafar, Leema Krishna Murali, Harsh Deshpande, Ameen Patel, William Brown, Johannes Hagemann, Connor Lane, Paul Steven Scotti, Tanishq Mathew Abraham
Computation and Language · Cross-listed from cs.CY
Who Decides What Is Harmful? Content Moderation Policy Through A Multi-Agent Personalised Inference Framework
Ewelina Gajewska, Michal Wawer, Katarzyna Budzynska et al. (+1 more)
The increasing scale and complexity of online platforms raises critical policy questions around harmful content, digital well-being, and user autonomy. Traditional content moderation systems rely on centralised, top-down rules, often failing to accommodate the subjective nature of harm perception. This paper proposes an LLM-based multi-
agent personalised inference framework that filters content based on unique sensitivity profiles of individual users. Our architecture combines domain-specific Expert
Agents, a Manager
Agent for orchestrating content analysis and
agent selection, and a Ghost Profile
Agent for simulating user perspectives, to inform moderation decisions. Evaluated against a range of non-personalised baselines, the system demonstrates up to a 32% improvement in accuracy, showing increased
alignment with individual user sensitivities. Beyond technical performance, our framework provides policy-relevant insights for platform governance, providing a scalable way to reconcile moderation policies with societal and individual digital rights
Read the full abstract ▾
All authors Ewelina Gajewska, Michal Wawer, Katarzyna Budzynska, Jaroslaw A. Chudziak
Computation and Language · Cross-listed from cs.IR
The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
Hiun Kim, Tae Kwan Lee and Taeryun Won
Masked
Language Modeling (MLM) pre-training is one of the primary ways to initialize Neural Information
Retrieval (IR) models prior to
retrieval fine-tuning. However, studies show that MLM pre-trained models have limited readiness and transfer learning issues for fine-tuning them into Neural Bi-Encoder models. This paper studies the effect of different pre-training
datasets and pre-training options on the MLM pre-trained models for
retrieval fine-tuning. The study focuses on the SPLADE-style model, which uses the MLM layer also at fine-tuning time. More specifically, we experimented with Expanded-SPLADE (ESPLADE) models, a specific instance of SPLADE models, and in-house web document titles are used as
datasets. Pre-training, fine-tuning, and evaluation with optional test-time
pruning of sparse vectors are conducted. Our observations are three-fold: First, fine-tuned models of higher
retrieval effectiveness at both unpruned and most strict pruned settings are mostly pre-trained on a general corpus, and pre-trained with a higher learning rate, showing lower MLM accuracies. Second, in the most strict pruned setting, those models show higher-level
retrieval cost and a higher variance in the length of the individual postings list. Third, the repetition of the general pre-training
dataset does not have much effect on
retrieval effectiveness. The experimentation empirically identifies the potential limitations for aligning MLM pre-training to ESPLADE fine-tuning. Also, the experimentation provides an empirical observation that, at most strict pruned settings, the
retrieval effectiveness is better maintained by the higher-level
retrieval cost, showing the trade-off relationship between the two in our setting.
Read the full abstract ▾
Computation and Language
Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regression
Yao Du, Shanshan Song and Xiaomeng Li
Multimodal large
language models (MLLMs) struggle with numerical regression under long-tailed target distributions.
Token-level supervised fine-tuning (SFT) and point-wise regression rewards bias learning toward high-density regions, leading to regression-to-the-mean behavior and poor tail performance. We identify the lack of cross-sample relational supervision as a key limitation of existing MLLM training paradigms. To address it, we propose a distribution-aware reinforcement learning framework based on Group Relative Policy Optimization, which introduces batch-level comparison-based supervision via the Concordance Correlation Coefficient-based reward to align predicted and ground-truth distributions in terms of correlation, scale, and mean. The framework is plug-and-play, requiring no architectural modification. Experiments on a unified suite of long-tailed regression
benchmarks show consistent improvements over SFT and existing MLLM regression methods, with particularly strong gains in medium- and few-shot regimes.
Read the full abstract ▾
Computation and Language
Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning
Sangkwon Park, Donghun Kang, Jisoo Mok et al. (+1 more)
The conventional
Retrieval-Augmented Generation (RAG) paradigm of injecting raw retrieved texts into the Large
Language Model (LLM)'s context often results in suboptimal integration of retrieved information. This paper proposes to bridge
retrieval results and the LLM's
reasoning ability through Verbal Annotations, analytic narratives that explicitly articulate the logical connection between a search query and retrieved contexts. Our empirical investigation reveals the potential of Verbal Annotations to substantially enhance the LLM's ability to generate accurate, contextually-grounded responses. Motivated by this finding, we introduce Verbal-R3, a novel agentic RAG framework that consists of a Generator and a Verbal Reranker. The Generator performs iterative
retrieval and
reasoning, while the Verbal Reranker returns relevance scores and Verbal Annotations to guide the
reasoning and answering process of the Generator. The inference process of Verbal-R3 is further refined through relevance-guided test-time scaling, which efficiently allocates test-time compute for effective trajectory expansion. Verbal-R3 achieves state-of-the-art performance on complex Question Answering
benchmarks, validating the effectiveness of the proposed framework.
Read the full abstract ▾
All authors Sangkwon Park, Donghun Kang, Jisoo Mok, Sungroh Yoon