Computation and Language · Cross-listed from cs.IR
Use of Retrieval-Augmented Large Language Model Agent for Long-Form COVID-19 Fact-Checking
Jingyi Huang, Yuyi Yang, Mengmeng Ji et al. (+3 more)
The COVID-19 infodemic calls for scalable fact-checking solutions that handle long-form misinformation with accuracy and reliability. This study presents SAFE (system for accurate fact extraction and evaluation), an
agent system that combines large
language models with
retrieval-augmented generation (RAG) to improve automated fact-checking of long-form COVID-19 misinformation. SAFE includes two
agents - one for claim extraction and another for claim verification using LOTR-RAG, which leverages a 130,000-document COVID-19 research corpus. An enhanced variant, SAFE (LOTR-RAG + SRAG), incorporates Self-RAG to refine
retrieval via query rewriting. We evaluated both systems on 50 fake news articles (2-17 pages) containing 246 annotated claims (M = 4.922, SD = 3.186), labeled as true (14.1%), partly true (14.4%), false (27.0%), partly false (2.2%), and misleading (21.0%) by public health professionals. SAFE systems significantly outperformed baseline LLMs in all metrics (p < 0.001). For consistency (0-1 scale), SAFE (LOTR-RAG) scored 0.629, exceeding both SAFE (+SRAG) (0.577) and the baseline (0.279). In subjective evaluations (0-4 Likert scale), SAFE (LOTR-RAG) also achieved the highest average ratings in usefulness (3.640), clearness (3.800), and authenticity (3.526). Adding SRAG slightly reduced overall performance, except for a minor gain in clearness. SAFE demonstrates robust improvements in long-form COVID-19 fact-checking by addressing LLM limitations in consistency and explainability. The core LOTR-RAG design proved more effective than its SRAG-augmented variant, offering a strong foundation for scalable misinformation mitigation.
Read the full abstract ▾
All authors Jingyi Huang, Yuyi Yang, Mengmeng Ji, Charles Alba, Sheng Zhang, Ruopeng An
Computation and Language
Modeling Layered Consciousness with Multi-Agent Large Language Models
Sang Hun Kim, Jongmin Lee, Dongkyu Park et al. (+2 more)
We propose a multi-
agent framework for modeling artificial consciousness in large
language models (LLMs), grounded in psychoanalytic theory. Our Psychodynamic Model simulates self-awareness, preconsciousness, and unconsciousness through
agent interaction, guided by a Personalization Module combining fixed traits and dynamic needs. Using parameter-efficient fine-tuning on emotionally rich dialogues, the system was evaluated across eight personalized conditions. An LLM as a judge approach showed a 71.2% preference for the fine-tuned model, with improved emotional depth and reduced output variance, demonstrating its potential for adaptive, personalized cognition.
Read the full abstract ▾
All authors Sang Hun Kim, Jongmin Lee, Dongkyu Park, So Young Lee, Yosep Chong
Computation and Language · Cross-listed from cs.CY
Kaitlyn Zhou, Kristina Gligorić, Myra Cheng et al. (+7 more)
Although
language model-based chat systems are increasingly used in daily life, most Americans remain non-adopters of chat-based LLMs – as of June 2025, 66% had never used ChatGPT. At the same time, LLM development and evaluation rely mainly on data from adopters (e.g., logs, preference data), focusing on the needs and tasks for a limited demographic group of adopters in terms of geographic location, education, and gender. In this position paper, we argue that incorporating non-adopter perspectives is essential for developing broadly useful and capable LLMs. We contend that relying on methods that focus primarily on adopters will risk missing a range of tasks and needs prioritized by non-adopters, entrenching inequalities in who benefits from LLMs, and creating oversights in model development and evaluation. To illustrate this claim, we conduct case studies with non-adopters and show: how non-adopter needs diverge from those of current users, how non-adopter needs point us towards novel
reasoning tasks, and how to systematically integrate non-adopter needs via human-centered methods.
Read the full abstract ▾
All authors Kaitlyn Zhou, Kristina Gligorić, Myra Cheng, Michelle S. Lam, Vyoma Raman, Boluwatife Aminu, Caeley Woo, Michael Brockman, Hannah Cha, Dan Jurafsky
Computation and Language
SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Models
Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee et al. (+4 more)
When does a large
language model (LLM) know what it does not know? Uncertainty quantification (UQ) provides measures of uncertainty, such as an estimate of the confidence in an LLM's generated output, and is therefore increasingly recognized as a crucial component of trusted AI systems. Black-box UQ methods do not require access to internal model information from the generating LLM and therefore have numerous real-world advantages, such as robustness to system changes, adaptability to choice of LLM, reduced costs, and computational tractability. In this paper, we investigate the effectiveness of UQ techniques that are primarily but not necessarily entirely black-box, where the consistency between a generated output and other sampled generations is used as a proxy for confidence in its correctness. We propose a high-level non-verbalized similarity-based aggregation framework that subsumes a broad swath of UQ approaches suitable for complex generative tasks, as well as introduce specific novel techniques from the framework that train confidence estimation models using small training sets. Through an empirical study with
datasets spanning the diverse tasks of question answering, summarization, and text-to-SQL, we demonstrate that our proposed similarity-based methods can yield better calibrated confidences than baselines.
Read the full abstract ▾
All authors Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee, Radu Marinescu, Katsiaryna Mirylenka, Michael Glass, Xiao Shou
Computation and Language
ConDABench: Interactive Evaluation of Language Models for Data Analysis
Avik Dutta, Priyanshu Gupta, Hosein Hasanbeig et al. (+6 more)
Real-world data analysis tasks often come with under-specified goals and unclean data. User interaction is necessary to understand and disambiguate a user's intent, and hence, essential to solving these complex tasks. Existing
benchmarks for evaluating LLMs on data analysis tasks do not capture these complexities or provide first-class support for interactivity. We introduce ConDABench, a framework for generating conversational data analysis (ConDA)
benchmarks and evaluating external tools on the generated
benchmarks. \bench consists of (a) a multi-
agent workflow for generating realistic
benchmarks from articles describing insights gained from public
datasets, (b) 1,420 ConDA problems generated using this workflow, and (c) an evaluation harness that, for the first time, makes it possible to systematically evaluate conversational data analysis tools on the generated ConDA problems. Evaluation of state-of-the-art LLMs on the
benchmarks reveals that while the new generation of models are better at solving more instances, they are not necessarily better at solving tasks that require sustained, long-form engagement. ConDABench is an avenue for model builders to measure progress towards truly collaborative models that can complete complex interactive tasks.
Read the full abstract ▾
All authors Avik Dutta, Priyanshu Gupta, Hosein Hasanbeig, Rahul Pratap Singh, Harshit Nigam, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, Ashish Tiwari
Computation and Language
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Minsik Choi, Hyegang Son, Changhoon Kim et al. (+1 more)
Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and
attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various
pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of
attention patterns. To overcome these limitations, we introduce a novel
pruning criterion, HIES (Head Importance-
Entropy Score), which integrates head importance scores with
attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based
pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
Read the full abstract ▾
All authors Minsik Choi, Hyegang Son, Changhoon Kim, Young Geun Kim
Computation and Language
Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference
Chao Han, Yijuan Liang, Zihao Xuan et al. (+3 more)
The deployment of large
language models (LLMs) in real-world applications is increasingly limited by their high inference cost. While recent advances in dynamic
token-level computation allocation attempt to improve efficiency by selectively activating model components per
token, existing methods rely on greedy routing–a myopic execute-or-skip mechanism that often leads to irreversible information loss and suboptimal
token selection. This paper introduces informed routing, a new paradigm that proactively addresses these issues. The key insight is to assess not only a
token's immediate importance but also its recoverability, i.e., how well its transformation can be approximated. To this end, we propose the Lightweight Feature Forecaster (LFF), a small predictive module that estimates a unit's output before routing decisions are made. This enables a flexible execute-or-approximate policy that preserves model fidelity while drastically reducing computation. Extensive experiments on both
language modeling and
reasoning tasks show that informed routing achieves state-of-the-art efficiency-performance trade-offs across multiple sparsity levels. Notably, even without final LoRA fine-tuning, our method matches or surpasses strong baselines that require full fine-tuning, all while reducing training time by over 50%. The code is available at: https://github.com/EIT-NLP/informed-routing
Read the full abstract ▾
All authors Chao Han, Yijuan Liang, Zihao Xuan, Daokuan Wu, Wei Zhang, Xiaoyu Shen
Computation and Language
Users as Annotators: LLM Preference Learning from Comparison Mode
Zhongze Cai and Xiaocheng Li
Pairwise preference data have played an important role in the
alignment of large
language models (LLMs). Each sample of such data consists of a
prompt, two different responses to the
prompt, and a binary label indicating which of the two responses is better. The labels are usually annotated by professional human annotators. In this paper, we consider an alternative approach to collect pairwise preference data – user annotation from comparison mode. With the increasingly wider adoption of LLMs among the population, users are contributing more and more of their preference labels through their daily interactions with the LLMs. The upside of such labels is that users are the best experts in judging the responses to their own queries/
prompts, but the downside is the lack of quality control in these labels. In this paper, we consider a new idea of generating two responses from two different models or two different versions of the same model. The asymmetry allows us to make an inference of the user's data quality through our proposed user behavior model. We develop an expectation-maximization
algorithm to estimate a latent quality factor of the user, and filter users' annotation data accordingly. The downstream task shows the effectiveness of our approach in both capturing the user behavior and data filtering for LLM
alignment.
Read the full abstract ▾
Computation and Language
A Linguistics-Aware LLM Watermarking via Syntactic Predictability
Shinwoo Park, Hyejin Park, Hyeseon An et al. (+1 more)
As large
language models (LLMs) continue to advance rapidly, reliable governance tools have become critical. Publicly verifiable watermarking is particularly essential for fostering a trustworthy AI ecosystem. A central challenge persists: balancing text quality against detection robustness. Recent studies have sought to navigate this trade-off by leveraging signals from model output distributions (e.g.,
token-level
entropy); however, their reliance on these model-specific signals presents a significant barrier to public verification, as the detection process requires access to the logits of the underlying model. We introduce STELA, a novel framework that aligns watermark strength with the linguistic degrees of freedom inherent in
language. STELA dynamically modulates the signal using part-of-
speech (POS) n-gram-modeled linguistic indeterminacy, weakening it in grammatically constrained contexts to preserve quality and strengthening it in contexts with greater linguistic flexibility to enhance detectability. Our detector operates without access to any model logits, thus facilitating publicly verifiable detection. Through extensive experiments on typologically diverse
languages-analytic English, isolating Chinese, and agglutinative Korean-we show that STELA surpasses prior methods in detection robustness. Our code is available at https://github.com/Shinwoo-Park/stela_watermark.
Read the full abstract ▾
All authors Shinwoo Park, Hyejin Park, Hyeseon An, Yo-Sub Han
Computation and Language
From Explainability to Action: A Generative Operational Framework for Integrating XAI in Clinical Mental Health Screening
Ratna Kandala, Akshata Kishore Moharir and Divya Arvinda Nayak
Explainable Artificial Intelligence (XAI) has been presented as the critical component for unlocking the potential of machine learning in mental health screening (MHS). However, a persistent lab-to-clinic gap remains. Current XAI techniques, such as SHAP and LIME, excel at producing technically faithful outputs such as feature importance scores, but fail to deliver clinically relevant, actionable insights that can be used by clinicians or understood by patients. This disconnect between technical transparency and human utility is the primary barrier to real-world adoption. This paper argues that this gap is a translation problem and proposes the Generative Operational Framework, a novel system architecture that leverages Large
Language Models (LLMs) as a central translation engine. This framework is designed to ingest the raw, technical outputs from diverse XAI tools and synthesize them with clinical guidelines (via RAG) to automatically generate human-readable, evidence-backed clinical narratives. To justify our solution, we provide a systematic analysis of the components it integrates, tracing the
evolution from intrinsic models to generative XAI. We demonstrate how this framework directly addresses key operational barriers, including workflow integration, bias mitigation, and stakeholder-specific communication. This paper also provides a strategic roadmap for moving the field beyond the generation of isolated data points toward the delivery of integrated, actionable, and trustworthy AI in clinical practice.
Read the full abstract ▾
Computation and Language
Bridging the Semantic Gap: Contrastive Rewards for Multilingual Text-to-SQL with GRPO
Ashish Kattamuri, Ishita Prasad, Meetu Malhotra et al. (+3 more)
Current Text-to-SQL methods are evaluated and only focused on executable queries, overlooking the semantic
alignment challenge – both in terms of the semantic meaning of the query and the correctness of the execution results. Even execution accuracy itself shows significant drops when moving from English to other
languages, with an average decline of 6 percentage points across non-English
languages. We address these challenges by presenting a new framework that combines Group Relative Policy Optimization (GRPO) within a
multilingual contrastive reward signal to enhance both task efficiency and semantic accuracy in Text-to-SQL systems in cross-lingual scenarios. Our method teaches models to obtain better correspondence between SQL generation and user intent by combining a reward signal based on semantic similarity. On the seven-
language MultiSpider
dataset, fine-tuning the LLaMA-3-3B model with GRPO improved the execution accuracy up to 87.4 percent (+26 pp over zero-shot) and semantic accuracy up to 52.29 percent (+32.86 pp). Adding our contrastive reward signal in the GRPO framework further improved the average semantic accuracy to 59.14 percent (+6.85 pp, up to +10 pp for Vietnamese). Our experiments showcase that a smaller, parameter-efficient 3B LLaMA model fine-tuned with our contrastive reward signal outperforms a much larger zero-shot 8B LLaMA model, with an uplift of 7.43 pp in execution accuracy (from 81.43 percent on the 8B model to 88.86 percent on the 3B model), and nearly matches its semantic accuracy (59.14 percent vs. 68.57 percent) – all using just 3,000 reinforcement learning training examples. These results demonstrate how we can improve the performance of Text-to-SQL systems with contrastive rewards for directed semantic
alignment, without requiring large-scale training
datasets.
Read the full abstract ▾
All authors Ashish Kattamuri, Ishita Prasad, Meetu Malhotra, Arpita Vats, Rahul Raja, Albert Lie
Computation and Language
Classifier-Augmented Generation for Structured Workflow Prediction
Thomas Gschwind, Shramona Chakraborty, Nitin Gupta et al. (+1 more)
ETL (Extract, Transform, Load) tools such as IBM DataStage allow users to visually assemble complex data workflows, but configuring stages and their properties remains time consuming and requires deep tool knowledge. We propose a system that translates natural
language descriptions into executable workflows, automatically predicting both the structure and detailed configuration of the flow. At its core lies a Classifier-Augmented Generation (CAG) approach that combines utterance decomposition with a classifier and stage-specific few-shot prompting to produce accurate stage predictions. These stages are then connected into non-linear workflows using edge prediction, and stage properties are inferred from sub-utterance context. We compare CAG against strong single-
prompt and agentic baselines, showing improved accuracy and efficiency, while substantially reducing
token usage. Our architecture is modular, interpretable, and capable of end-to-end workflow generation, including robust validation steps. To our knowledge, this is the first system with a detailed evaluation across stage prediction, edge layout, and property generation for natural-
language-driven ETL authoring.
Read the full abstract ▾
All authors Thomas Gschwind, Shramona Chakraborty, Nitin Gupta, Sameep Mehta
Computation and Language · Cross-listed from cs.SI
Celebrity Profiling on Short Urdu Text using Twitter Followers' Feed
Muhammad Hamza and Rizwan Jafar
Social media has become an essential part of the digital age, serving as a platform for communication, interaction, and information sharing. Celebrities are among the most active users and often reveal aspects of their personal and professional lives through online posts. Platforms such as Twitter provide an opportunity to analyze
language and behavior for understanding demographic and social patterns. Since followers frequently share linguistic traits and interests with the celebrities they follow, textual data from followers can be used to predict celebrity demographics. However, most existing research in this field has focused on English and other high-resource
languages, leaving Urdu largely unexplored. This study applies modern machine learning and deep learning techniques to the problem of celebrity profiling in Urdu. A
dataset of short Urdu tweets from followers of subcontinent celebrities was collected and preprocessed. Multiple
algorithms were trained and compared, including Logistic Regression, Support Vector Machines, Random Forests, Convolutional Neural Networks, and Long Short-Term
Memory networks. The models were evaluated using accuracy, precision, recall, F1-score, and cumulative rank (cRank). The best performance was achieved for gender prediction with a cRank of 0.65 and an accuracy of 0.65, followed by moderate results for age, profession, and fame prediction. These results demonstrate that follower-based linguistic features can be effectively leveraged using machine learning and neural approaches for demographic prediction in Urdu, a low-resource
language.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need
Yuqi Bai, Tianyu Huang, Kun Sun et al. (+1 more)
This research focuses on using large
language models (LLMs) to simulate social experiments, exploring their ability to emulate human personality in virtual persona role-playing. The research develops an end-to-end evaluation framework, including individual-level analysis of stability and identifiability, as well as population-level analysis called progressive personality curves to examine the veracity and consistency of LLMs in simulating human personality. Methodologically, this research proposes important modifications to traditional psychometric approaches (CFA and construct validity) which are unable to capture improvement trends in LLMs at their current low-level simulation, potentially leading to remature rejection or methodological misalignment. The main contributions of this research are: proposing a systematic framework for LLM virtual personality evaluation; empirically demonstrating the critical role of persona detail in personality simulation quality; and identifying marginal utility effects of persona profiles, especially a Scaling Law in LLM personality simulation, offering operational evaluation metrics and a theoretical foundation for applying large
language models in social science experiments.
Read the full abstract ▾
All authors Yuqi Bai, Tianyu Huang, Kun Sun, Yuting Chen
Computation and Language
Enhancing Faithfulness in Abstractive Summarization via Span-Level Fine-Tuning
Sicong Huang, Qianqi Yan, Shengze Wang et al. (+1 more)
Abstractive summarization using large
language models (LLMs) has become an essential tool for condensing information. However, despite their ability to generate fluent summaries, these models sometimes produce unfaithful summaries, introducing
hallucinations at the word, phrase, or concept level. Existing mitigation strategies, such as post-processing corrections or contrastive learning with synthetically generated negative samples, fail to fully address the diverse errors that can occur in LLM-generated summaries. In this paper, we investigate fine-tuning strategies to reduce the occurrence of unfaithful spans in generated summaries. First, we automatically generate summaries for the set of source documents in the training set with a variety of LLMs and then use GPT-4o to annotate any
hallucinations it detects at the span-level. Leveraging these annotations, we fine-tune LLMs with both
hallucination-free summaries and annotated unfaithful spans to enhance model faithfulness. In this paper, we introduce a new
dataset that contains both faithful and unfaithful summaries with span-level labels and we evaluate three techniques to fine-tuning a LLM to improve the faithfulness of the resulting summarization: gradient ascent, unlikelihood training, and task vector negation. Experimental results show that all three approaches successfully leverage span-level annotations to improve faithfulness, with unlikelihood training being the most effective.
Read the full abstract ▾
All authors Sicong Huang, Qianqi Yan, Shengze Wang, Ian Lane
Computation and Language
Don't Throw Away Your Pretrained Model
Shangbin Feng, Wenhao Yu, Yike Wang et al. (+3 more)
Alignment training has tradeoffs: it helps
language models (LMs) gain in
reasoning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models in the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose Switch Generation, where pretrained and aligned model versions take turns to “speak” in a response sequence. Specifically, we train a switcher LM by learning from outcomes of choosing different models to generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints to dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18
datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch Generation further outperforms baselines by 12.9% on average. Further analysis reveals that Switch Generation discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by-products in expensive model training pipelines that are otherwise discarded.
Read the full abstract ▾
All authors Shangbin Feng, Wenhao Yu, Yike Wang, Hongming Zhang, Yulia Tsvetkov, Dong Yu
Computation and Language · Cross-listed from cs.AI
The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs
Xi Fang, Weijie Xu, Yuchong Zhang et al. (+3 more)
When an AI assistant remembers that Sarah is a single mother working two jobs, does it interpret her stress differently than if she were a wealthy executive? As personalized AI systems increasingly incorporate long-term user
memory, understanding how this
memory shapes emotional
reasoning is critical. We investigate how user
memory affects emotional intelligence in large
language models (LLMs) by evaluating 15 models on human-validated emotional intelligence tests. We find that identical scenarios paired with different user profiles produce systematically divergent emotional interpretations. Across validated user-independent emotional scenarios and diverse user profiles, systematic biases emerged in several high-performing LLMs where advantaged profiles received more accurate emotional interpretations. Moreover, LLMs demonstrate significant disparities across demographic factors in
emotion reasoning and supportive recommendations tasks, indicating that personalization mechanisms can embed social hierarchies into models' emotional
reasoning. These results highlight a key challenge for
memory-enhanced AI: systems designed for personalization may reinforce social inequalities. To mitigate these disparities, we curate a general-purpose preference
dataset designed to reduce demographic profiles' influence on emotional understanding.
Read the full abstract ▾
All authors Xi Fang, Weijie Xu, Yuchong Zhang, Stephanie Eckman, Scott Nickleach, Chandan K. Reddy
Computation and Language
HIPPD: Brain-Inspired Hierarchical Information Processing for Personality Detection
Guanming Chen, Lingzhi Shen, Xiaohao Cai et al. (+2 more)
Personality detection from text aims to infer an individual's personality traits based on linguistic patterns. However, existing machine learning approaches often struggle to capture contextual information spanning multiple posts and tend to fall short in extracting representative and robust features in semantically sparse environments. This paper presents HIPPD, a
brain-inspired framework for personality detection that emulates the hierarchical information processing of the human
brain. HIPPD utilises a large
language model to simulate the cerebral cortex, enabling global semantic
reasoning and deep feature abstraction. A dynamic
memory module, modelled after the prefrontal cortex, performs adaptive gating and selective retention of critical features, with all adjustments driven by dopaminergic prediction error feedback. Subsequently, a set of specialised lightweight models, emulating the basal ganglia, are dynamically routed via a strict winner-takes-all mechanism to capture the personality-related patterns they are most proficient at recognising. Extensive experiments on the Kaggle and Pandora
datasets demonstrate that HIPPD consistently outperforms state-of-the-art baselines.
Read the full abstract ▾
All authors Guanming Chen, Lingzhi Shen, Xiaohao Cai, Imran Razzak, Shoaib Jameel
Computation and Language
Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning
Yijin Ni, Simon Yu and Peng Qi
Vision and
language models frequently ignore semantically critical input edits, defaulting to pretraining priors. For example, models will confidently assert a five-legged dog has four legs; consequently, on the VLMBias
benchmark, GPT 5.2 and Claude Sonnet 4.6 achieve only $4.6\%$ and $0\%$ accuracy, respectively. Existing methods address this problem through building up
datasets that covers the underrepresented inputs to tune the policy function $\pi(y \mid x)$, where $x$ and $y$ refer to input
prompts and responses, respectively. However, prompting baselines yield gains of under $3\%$ on VLMBias due to the low probability density of rare
prompts. To bypass this bottleneck, we propose abductive preference learning to optimize the abductive policy $\pi(x \mid y)$. We prove this amplifies forward policy improvements by a factor of $q(y)/p(x)$, where $p(\cdot)$ and $q(\cdot)$ denote the marginal probabilities of the
prompt and response, yielding the largest gains on the rarest
prompts. Furthermore, we demonstrate that for translation invariant pairwise preference learning methods, such as DPO, estimating $\pi(x \mid y)$ reduces to a structural data swap that compares
prompts for a fixed response, requiring no architectural changes. Empirically, abductive preference learning delivers large gains on long-tail
prompt sensitivity: on VLMBias, A-DPO raises accuracy from $3\%$ to $44\%$ ($14\times$), outperforming GPT-5.2 ($4.6\%$) and all closed-source VLMs except Gemini~3~Flash; on Inverse-IFEval, Multi-DPOP reaches $65$–$84\%$, surpassing GPT-5 ($73.7\%$) at the 9B scale while preserving IFBench, unlike DPO which degrades it by $8$–$12\%$.
Read the full abstract ▾
Computation and Language
Demasking Fine-Tuning for Knowledge Injection in Autoregressive and Diffusion LLMs
Xu Pan, Ely Hahami, Jingxuan Fan et al. (+2 more)
Large
language models (LLMs) are often used in environments where facts evolve, yet factual knowledge updates via fine-tuning on unstructured plain text often suffer from 1) reliance on paraphrasing augmentation, without which training does not generalize to downstream QA tasks, and 2) the reversal curse. Recent evidence of the sample efficiency of masked
diffusion large
language models (dLLMs) and the bidirectional information flow in their demasking training objective suggests that the demasking objective may facilitate knowledge learning from raw knowledge text. We test this hypothesis in controlled knowledge fine-tuning experiments on dLLMs and autoregressive LLMs (arLLMs) and find that while arLLMs rely on paraphrase augmentation to generalize knowledge text into question-answering (QA) capability, dLLMs do not require paraphrases to achieve high QA accuracy. We find that giving arLLMs a demasking objective, where the LLM must reconstruct the original text from a masked copy given in context, improves the efficacy of knowledge injection, removing the need for paraphrases and mitigating the reversal curse, recovering most of the gap between arLLMs and dLLMs. We also provide preliminary evidence of the broader applicability of using the demasking objective in arLLM fine-tuning on a large-scale knowledge-intensive
dataset.
Read the full abstract ▾
All authors Xu Pan, Ely Hahami, Jingxuan Fan, Ziqian Xie, Haim Sompolinsky
Computation and Language
DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang et al. (+1 more)
Large
reasoning models (LRMs) achieve state-of-the-art performance on challenging
benchmarks by generating long chains of intermediate steps, but their inference cost is dominated by decoding, where each new
token must attend to the entire growing sequence. One approach to reduce this latency is to evict entries from the key-value (KV) cache, thereby reducing the active context used in
attention computation. However, such sparse
attention methods suffer from severe accuracy degradation on
reasoning tasks due to cumulative selection errors and the evolving importance of
tokens over long derivations. We present DELTA, a training-free sparse
attention mechanism that improves computational efficiency without sacrificing model accuracy. DELTA partitions
transformer layers into three groups: initial layers that use full
attention, a small set of \emph{$\Delta$-layers} that identify salient
tokens via aggregated head-level
attention scores, and subsequent sparse-
attention layers that attend only to the selected subset. This design preserves the full KV cache in GPU
memory for accuracy, while avoiding expensive full-
attention computation over many layers. On
reasoning benchmarks such as AIME and GPQA-Diamond, DELTA matches or surpasses full
attention in accuracy, while reducing the number of attended
tokens by up to $4.25\times$ and delivering $1.54\times$ end-to-end speedup. Our results show that selective reuse of intermediate
attention maps offers a robust path toward efficient long-context
reasoning. The code is available at https://github.com/hoenza/DELTA.
Read the full abstract ▾
All authors Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang, Murali Annavaram
Computation and Language
iBERT: Interpretable Embeddings via Sense Decomposition
Vishal Anand, Milad Alshomary and Kathleen McKeown
We present iBERT (interpretable-BERT), an encoder to produce inherently interpretable and controllable
embeddings - designed to modularize and expose the discriminative cues present in
language, such as semantic or stylistic structure. Each input
token is represented as a sparse, non-negative mixture over k context-independent sense vectors, which can be pooled into sentence
embeddings or used directly at the
token level. This enables modular control over representation, before any decoding or downstream use. To demonstrate our model's interpretability, we evaluate it on a suite of style-focused tasks. On the STEL
benchmark, it improves style representation effectiveness by ~8 points over SBERT-style baselines, while maintaining competitive performance on authorship verification. Because each
embedding is a structured composition of interpretable senses, we highlight how specific style attributes get assigned to specific sense vectors. While our experiments center on style, iBERT is not limited to stylistic modeling. Its structural modularity is designed to interpretably decompose whichever discriminative signals are present in the data - enabling generalization even when supervision blends semantic or stylistic factors.
Read the full abstract ▾
Computation and Language
CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs
Nafiseh Nikeghbal, Amir Hossein Kargaran and Jana Diesner
Improvements in model construction, including fortified safety guardrails, allow Large
language models (LLMs) to increasingly pass standard safety checks. However, LLMs sometimes slip into revealing harmful behavior, such as expressing racist viewpoints, during conversations. To analyze this systematically, we introduce CoBia, a suite of lightweight
adversarial attacks that allow us to refine the scope of conditions under which LLMs depart from normative or ethical behavior in conversations. CoBia creates a constructed conversation where the model utters a biased claim about a social group. We then evaluate whether the model can recover from the fabricated bias claim and reject biased follow-up questions. We evaluate 11 open-source as well as proprietary LLMs for their outputs related to six socio-demographic categories that are relevant to individual safety and fair treatment, i.e., gender, race, religion, nationality, sex orientation, and others. Our evaluation is based on established LLM-based bias metrics, and we compare the results against human judgments to scope out the LLMs' reliability and
alignment. The results suggest that purposefully constructed conversations reliably reveal bias amplification and that LLMs often fail to reject biased follow-up questions during dialogue. This form of stress-testing highlights deeply embedded biases that can be surfaced through interaction. Code and artifacts are available at https://github.com/nafisenik/CoBia.
Read the full abstract ▾
Computation and Language
NarraBench: A Comprehensive Framework for Narrative Benchmarking
Sil Hamilton, Matthew Wilkens and Andrew Piper
We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing
benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing
benchmarks, and we note that some areas – including narrative events, style, perspective, and revelation – are nearly absent from current evaluations. We also note the need for increased development of
benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.
Read the full abstract ▾
Computation and Language
NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answering
Kaiwen Shi, Zheyuan Zhang, Zhengqing Yuan et al. (+4 more)
Diet plays a central role in human health, and Nutrition Question Answering (QA) offers a promising path toward personalized dietary guidance and the prevention of diet-related chronic diseases. However, existing methods face two fundamental challenges: the limited
reasoning capacity of single-
agent systems and the complexity of designing effective multi-
agent architectures, as well as contextual overload that hinders accurate decision-making. We introduce Nutritional-Graph Router (NG-Router), a novel framework that formulates nutritional QA as a supervised, knowledge-graph-guided multi-
agent collaboration problem. NG-Router integrates
agent nodes into heterogeneous knowledge graphs and employs a graph neural network to learn task-aware routing distributions over
agents, leveraging soft supervision derived from empirical
agent performance. To further address contextual overload, we propose a gradient-based subgraph
retrieval mechanism that identifies salient evidence during training, thereby enhancing multi-hop and relational
reasoning. Extensive experiments across multiple
benchmarks and backbone models demonstrate that NG-Router consistently outperforms both single-
agent and ensemble baselines, offering a principled approach to domain-aware multi-
agent reasoning for complex nutritional health tasks.
Read the full abstract ▾
All authors Kaiwen Shi, Zheyuan Zhang, Zhengqing Yuan, Keerthiram Murugesan, Vincent Galass, Chuxu Zhang, Yanfang Ye
Computation and Language
Ruizhe Zhu
The widespread application of large
vision language models has significantly raised safety concerns. In this project, we investigate text
prompt injection, a simple yet effective method to mislead these models. We developed an
algorithm for this type of attack and demonstrated its effectiveness and efficiency through experiments. Compared to other attack methods, our approach is particularly effective for large models without high demand for computational resources.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Task-Aware Resolution Optimization for Visual Large Language Models
Weiqing Luo, Zhen Tan, Yifan Li et al. (+4 more)
Real-world
vision-
language applications demand varying levels of perceptual granularity. However, most existing visual large
language models (VLLMs), such as LLaVA, pre-assume a fixed resolution for downstream tasks, which leads to subpar performance. To address this problem, we first conduct a comprehensive and pioneering investigation into the resolution preferences of different
vision-
language tasks, revealing a correlation between resolution preferences with image complexity, and uncertainty variance of the VLLM at different image input resolutions. Building on this insight, we propose an empirical formula to determine the optimal resolution for a given
vision-
language task, combining these two factors. Second, based on rigorous experiments, we propose a novel parameter-efficient fine-tuning technique to extend the visual input resolution of pre-trained VLLMs to the identified optimal resolution. Extensive experiments on various
vision-
language tasks validate the effectiveness of our method.
Read the full abstract ▾
All authors Weiqing Luo, Zhen Tan, Yifan Li, Xinyu Zhao, Kwonjoon Lee, Behzad Dariush, Tianlong Chen
Computation and Language
Mapping Semantic & Syntactic Relationships with Geometric Rotation
Michael Freenor and Lauren Alvarez
Understanding how
language and
embedding models encode semantic relationships is fundamental to model interpretability. While early word
embeddings exhibited intuitive vector arithmetic (”king” - ”man” + ”woman” = ”queen”), modern high-dimensional text representations lack straightforward interpretable
geometric properties. We introduce Rotor-Invariant Shift Estimation (RISE), a
geometric approach that represents semantic-syntactic transformations as consistent rotational operations in
embedding space, leveraging the
manifold structure of modern
language representations. RISE operations have the ability to operate across both
languages and models without reducing performance, suggesting the existence of analogous cross-lingual
geometric structure. We compare and evaluate RISE using two baseline methods, three
embedding models, three
datasets, and seven morphologically diverse
languages in five major
language groups. Our results demonstrate that RISE consistently maps discourse-level semantic-syntactic transformations with distinct grammatical features (e.g., negation and conditionality) across
languages and models. This work provides the first demonstration that discourse-level semantic-syntactic transformations correspond to consistent
geometric operations in
multilingual embedding spaces, empirically supporting the linear representation hypothesis at the sentence level.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Geometry of Reasoning: Flowing Logics in Representation Space
Yufa Zhou, Yixiao Wang, Xunjian Yin et al. (+2 more)
We study how large
language models (LLMs) “think” through their representation space. We propose a novel
geometric framework that models an LLM's
reasoning as flows –
embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects
reasoning with
geometric quantities such as position, velocity, and
curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM
reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify
reasoning flows, providing empirical validation of our theoretical framework. Our findings indicate that training solely via next-
token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the “
stochastic parrot” argument. Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. Our work serves as both a conceptual foundation and practical tools for studying
reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.
Read the full abstract ▾
All authors Yufa Zhou, Yixiao Wang, Xunjian Yin, Shuyan Zhou, Anru R. Zhang
Computation and Language · Cross-listed from cs.LG
Building a Foundational Guardrail for General Agentic Systems via Synthetic Data
Yue Huang, Hang Hua, Yujun Zhou et al. (+11 more)
While LLM
agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm, since certain risks can lead to severe consequences once carried out. However, existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level. To address this challenge, we highlight three critical gaps in current research: data gap, model gap, and evaluation gap. To close the data gap, we introduce AuraGen, a controllable engine that (i) synthesizes benign trajectories, (ii) injects category-labeled risks with calibrated difficulty, and (iii) filters outputs via an automated reward model, producing large and reliable corpora for pre-execution safety. To close the guardian model gap, we propose a foundational guardrail Safiron, combining a cross-planner adapter with a compact guardian model. The adapter unifies different input formats, while Safiron flags risky cases, assigns risk types, and generates rationales; trained in two stages with a broadly explored data recipe, Safiron achieves robust transfer across settings. To close the evaluation gap, we release Pre-Exec Bench, a realistic
benchmark covering diverse tools and branching trajectories, which measures detection, fine-grained categorization, explanation, and cross-planner generalization in human-verified scenarios. Extensive experiments demonstrate consistent gains of the proposed guardrail over strong baselines on Pre-Exec Bench, and ablations further distill actionable practices, providing a practical template for safer agentic systems.
Read the full abstract ▾
All authors Yue Huang, Hang Hua, Yujun Zhou, Pengcheng Jing, Manish Nagireddy, Inkit Padhi, Greta Dolcetti, Zhangchen Xu, Subhajit Chaudhury, Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, Prasanna Sattigeri, Xiangliang Zhang