Computation and Language
Pretraining Exposure Explains Popularity Judgments in Large Language Models
Jamshid Mozafari, Bhawna Piryani and Adam Jatowt
Large
language models (LLMs) exhibit systematic preferences for well-known entities, a phenomenon often attributed to popularity bias. However, the extent to which these preferences reflect real-world popularity versus statistical exposure during pretraining remains unclear, largely due to the inaccessibility of most training corpora. We provide the first direct, large-scale analysis of popularity bias grounded in fully observable pretraining data. Leveraging the open OLMo models and their complete pretraining corpus, Dolma, we compute precise entity-level exposure statistics across 7.4 trillion
tokens. We analyze 2,000 entities spanning five types (Person, Location, Organization, Art, Product) and compare pretraining exposure against Wikipedia pageviews and two elicited LLM popularity signals: direct scalar estimation and pairwise comparison. Our results show that pretraining exposure strongly correlates with Wikipedia popularity, validating exposure as a meaningful proxy for real-world salience during the training period. More importantly, we find that LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons. This
alignment is strongest for larger models and persists in the long tail, where Wikipedia popularity becomes unreliable. Overall, our findings demonstrate that popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias.
Read the full abstract ▾
Computation and Language
Context Convergence Improves Answering Inferential Questions
Jamshid Mozafari, Bhawna Piryani and Adam Jatowt
While Large
Language Models (LLMs) are widely used in open-domain Question Answering (QA), their ability to handle inferential questions-where answers must be derived rather than directly retrieved-remains still underexplored. This study investigates how the structure and quality of passages influence LLM performance on such questions. We focus on convergence, a measure of how effectively sentences (hints) eliminate incorrect answers, as a criterion for constructing passages. Using subsets of the TriviaHG
dataset, we form passages by combining sentences with varying convergence levels and evaluate six LLMs of different sizes and architectures. Our results show that passages built from higher convergence sentences lead to substantially better answer accuracy than those selected by cosine similarity, indicating that convergence captures meaningful relevance for inferential
reasoning. Additionally, ordering sentences by descending convergence slightly improves performance, suggesting that LLMs tend to prioritize earlier, information-rich cues. These findings highlight convergence as a practical signal for guiding passage construction and analyzing inferential
reasoning behavior in LLMs.
Read the full abstract ▾
Computation and Language
MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
Rezarta Islamaj, Robert Leaman, Joey Chan et al. (+13 more)
Evaluating large
language models (LLMs) in the biomedical domain requires
benchmarks that can distinguish
reasoning from pattern matching and remain discriminative as model capabilities improve. Existing biomedical question answering (QA)
benchmarks are limited in this respect. Multiple-choice formats can allow models to succeed through answer elimination rather than inference, while widely circulated exam-style
datasets are increasingly vulnerable to performance saturation and training data contamination. Multi-hop
reasoning, defined as the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA
benchmarks. MedHopQA is a disease-centered multi-hop
reasoning benchmark consisting of 1,000 expert-curated question-answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format. Gold annotations are augmented with ontology-grounded synonym sets from MONDO, NCBI Gene, and NCBI Taxonomy to support both lexical and concept-level evaluation. MedHopQA was constructed through a structured process combining human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1,000 scored questions are embedded within a publicly downloadable set of 10,000 questions, with answers withheld, on a CodaBench leaderboard. MedHopQA provides both a
benchmark and a reusable framework for constructing future biomedical QA
datasets that prioritize compositional
reasoning, saturation resistance, and contamination resistance as core design constraints.
Read the full abstract ▾
All authors Rezarta Islamaj, Robert Leaman, Joey Chan, Nicholas Wan, Qiao Jin, Natalie Xie, John Wilbur, Shubo Tian, Lana Yeganova, Po-Ting Lai, Chih-Hsuan Wei, Yifan Yang, Yao Ge, Qingqing Zhu, Zhizheng Wang, Zhiyong Lu
Computation and Language
Output Composability of QLoRA PEFT Modules for Plug-and-Play Attribute-Controlled Text Generation
Michela Lorandi and Anya Belz
Parameter-efficient fine-tuning (PEFT) techniques offer task-specific fine-tuning at a fraction of the cost of full fine-tuning, but require separate fine-tuning for every new task (combination). In this paper, we explore three ways of generalising beyond single-task training/inference: (i) training on combinations of multiple, related
datasets; (ii) at inference, composing the weight
matrices of separately trained PEFT modules; and (iii) at inference, composing the outputs of separately trained PEFT modules. We test these approaches on three different LLMs, QLoRA as the PEFT technique, and three sets of controlled text generation
datasets for sentiment control, topic control, and multi-attribute control. We find that summing PEFT module outputs is a particularly strong composition method, which consistently either outperforms or matches the performance of alternative approaches. This is the case even when comparing against single-task specialised modules on the single-task test set, where three-module output composition achieves an average 2% point performance increase across all models for sentiment control.
Read the full abstract ▾
Computation and Language
A categorical error sensitivity index (ISEC): A preventive ordinal decision-support measure for irrecoverable errors in manual data entry systems
Ricardo Raúl Palma, Mauro Anibal Benetti and Fabricio Orlando Sanchez Varretti
Data entry systems remain structurally vulnerable to categorical misclassifications, particularly in small and medium sized enterprises (SMEs). When nominal categories exhibit semantic or morphological proximity, human machine interaction may produce errors that are irrecoverable ex post. In the absence of automated input controls, manual data entry frequently generates irrecoverable categorical distortions that propagate into Key Performance Indicators (KPIs), thereby misleading managerial decision making. State of the art normalization tools typically evaluate semantic and morphological dimensions in isolation and rely heavily on standard dictionaries, rendering them ineffective for SME master data rich in custom SKUs, abbreviations, and domain-specific technical jargon. This paper introduces the Categorical Error Sensitivity Index (ISEC), an ordinal composite score designed to rank category pairs according to their structural susceptibility to confusion. ISEC integrates semantic distance (via word
embeddings), custom weighted morphological transformation costs (through an adapted Damerau Levenshtein
algorithm), and empirical frequency into a unified, mathematically robust preventive framework. By leveraging vector database architectures, ISEC reduces computational complexity, achieving approximately a 195x performance improvement over brute-force methods. Validated across three heterogeneous
datasets: governmental judicial records, retail inventory, and a synthetic ISO coded metalworking catalog, ISEC provides a scalable and proactive data governance instrument that enables SMEs to detect latent structural risk embedded within their categorical data assets.
Read the full abstract ▾
Computation and Language
Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering
Rezarta Islamaj, Joey Chan, Robert Leaman et al. (+13 more)
Multi-hop question answering (QA) remains a significant challenge in the biomedical domain, requiring systems to integrate information across multiple sources to answer complex questions. To address this problem, the BioCreative IX MedHopQA shared task was designed to
benchmark in multi-hop
reasoning for large
language models (LLMs). We developed a novel
dataset of 1,000 challenging QA pairs spanning diseases, genes, and chemicals, with particular emphasis on rare diseases. Each question was constructed to require two-hop
reasoning through the integration of information from two distinct Wikipedia pages. The challenge attracted 48 submissions from 13 teams. Systems were evaluated using both surface string comparison and conceptual accuracy (MedCPT score). The results showed a substantial performance gap between baseline LLMs and enhanced systems. The top-ranked submission achieved an 89.30% F1 score on the MedCPT metric and an 87.30% exact match (EM) score, compared with 67.40% and 60.20%, respectively, for the zero-shot baseline. A central finding of the challenge was that
retrieval-augmented generation (RAG) and related
retrieval-based strategies were critical for strong performance. In addition, concept-level evaluation improved answer assessment when correct responses differed in surface form. The MedHopQA
dataset is publicly available to support continued progress in this important area. Challenge materials: https://www.ncbi.nlm.nih.gov/research/bionlp/medhopqa and
benchmark https://www.codabench.org/competitions/7609/
Read the full abstract ▾
All authors Rezarta Islamaj, Joey Chan, Robert Leaman, Jongmyung Jung, Hyeongsoon Hwang, Quoc-An Nguyen, Hoang-Quynh Le, Harikrishnan Gurushankar Saisudha, Ganesh Chandrasekar, Rustam R. Taktashov, Nadezhda Yu. Bizyukova, Sofia I. R. Conceição, Paulo R. C. Lopes, Reem Abdel Salam, Mary Adewunmi, Zhiyong Lu
Computation and Language
GKnow: Measuring the Entanglement of Gender Bias and Factual Gender
Leonor Veloso and Hinrich Schütze
Recent works have analyzed the impact of individual components of neural networks on gendered predictions, often with a focus on mitigating gender bias. However, mechanistic interpretations of gender tend to (i) focus on a very specific gender-related task, such as gendered pronoun prediction, or (ii) fail to distinguish between the production of factually gendered outputs (the correct assumption of gender given a word that carries gender as a semantic property) and gender biased outputs (based on a stereotype). To address these issues, we curate \gknow, a
benchmark to assess gender knowledge and gender bias in
language models across different types of gender-related predictions. \gknow allows us to identify and analyze
circuits and individual
neurons responsible for gendered predictions. We test the impact of
neuron ablation on
benchmarks for disentangling stereotypical and factual gender (DiFair and the test set of GKnow), as well as StereoSet. Results show that gender bias and factual gender are severely entangled on the level of both
circuits and
neurons, entailing that ablation is an unreliable debiasing method. Furthermore, we show that
benchmarks for evaluating gender bias can hide the decrease in factual gender knowledge that accompanies
neuron ablation. We curate GKnow as a contribution to the continuous development of robust gender bias
benchmarks.
Read the full abstract ▾
Computation and Language
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van et al. (+3 more)
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning
language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-
token decisions. Existing
token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover
token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce
Token-level Bregman Preference Optimization (TBPO), which posits a
token-level Bradley-Terry preference model over next-
token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the
token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization
benchmarks, TBPO improves
alignment quality and training stability and increases output diversity relative to strong sequence-level and
token-level baselines.
Read the full abstract ▾
All authors Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa Doan, Trung Le
Computation and Language
What makes a word hard to learn? Modeling L1 influence on English vocabulary difficulty
Jonas Mayer Martins, Zhuojing Huang, Aaricia Herygers et al. (+1 more)
What makes a word difficult to learn, and how does the difficulty depend on the learner's native
language? We computationally model vocabulary difficulty for English learners whose first
language is Spanish, German, or Chinese with gradient-boosted models trained on features related to a word's familiarity (e.g., frequency), meaning, surface form, and cross-linguistic transfer. Using Shapley values, we determine the importance of each feature group. Word familiarity is the dominant feature group shared by all three
languages. However, predictions for Spanish- and German-speaking learners rely additionally on orthographic transfer. This transfer mechanism is unavailable to Chinese learners, whose difficulty is shaped by a combination of familiarity and surface features alone. Our models provide interpretable, L1-tailored difficulty estimates that can be used to design vocabulary curricula.
Read the full abstract ▾
All authors Jonas Mayer Martins, Zhuojing Huang, Aaricia Herygers, Lisa Beinborn
Computation and Language · Cross-listed from cs.CR
Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Models
Sae Furukawa and Alina Oprea
Supervised Finetuning (SFT) has become one of the primary methods for adapting a large
language model (LLM) with extensive pre-trained knowledge to domain-specific, instruction-following tasks. SFT
datasets, composed of instruction-response pairs, often include user-provided information that may contain sensitive data such as personally identifiable information (PII), raising
privacy concerns. This paper studies the problem of targeted PII reconstruction from models fine- tuned on proprietary SFT data, in which an adversary attempts to recover PII associated with a specific identity. We construct multi-turn, user-centric Q&A
datasets in sensitive domains, specifically medical and legal settings, that incorporate PII to enable realistic evaluation of leakage. We then propose COVA, a coverage-aware decoding
algorithm for targeted PII reconstruction under prefix-based attacks. Using COVA, we study how the amount of information available about a target user affects the recovery of their PII from SFT models. Across
datasets, COVA consistently improves PII reconstruction over baseline decoding methods, and even limited contextual knowledge can substantially increase an adversary's success. Our findings demonstrate that small, proprietary SFT
datasets can induce meaningful
privacy leakage through the reconstruction of target-PII associations learned during fine-tuning.
Read the full abstract ▾
Computation and Language
PRISM: Pareto-Efficient Retrieval over Intent-Aware Structured Memory for Long-Horizon Agents
Jingyi Peng, Zhongwei Wan, Weiting Liu et al. (+1 more)
Long-horizon
language agents accumulate conversation history far faster than any fixed context window can hold, making
memory management critical to both answer accuracy and serving cost. Existing approaches either expand the context window without addressing what is retrieved, perform heavy ingestion-time fact extraction at substantial
token cost, or rely on heuristic graph traversal that leaves both accuracy and efficiency on the table. We present PRISM, a training-free
retrieval-side framework that treats long-horizon
memory as a joint
retrieval-and-compression problem over a graph-structured
memory. PRISM combines four orthogonal inference-time components: Hierarchical Bundle Search over typed relation paths, Query-Sensitive Edge Costing that aligns traversal with detected query intent, Evidence Compression that compresses the candidate bundle into a compact answer-side context, and Adaptive Intent Routing that routes most queries through zero-LLM tiers. By formulating
retrieval as min-cost selection over typed path templates and pairing it with an LLM-side compression step, PRISM surfaces the right evidence under a strict context budget without any fine-tuning or modification to the upstream ingestion pipeline. Experiments on the LoCoMo
benchmark show that PRISM delivers substantially higher LLM-judge accuracy than every same-protocol baseline at an order-of-magnitude smaller context budget, occupying a previously empty corner of the accuracy-context-cost frontier and demonstrating a superior balance between answer quality and
retrieval efficiency.
Read the full abstract ▾
All authors Jingyi Peng, Zhongwei Wan, Weiting Liu, Qiuzhuang Sun
Computation and Language
PreScam: A Benchmark for Predicting Scam Progression from Early Conversations
Weixiang Sun, Shang Ma, Yiyang Li et al. (+5 more)
Conversational scams, such as romance and investment scams, are emerging as a major form of online fraud. Unlike one-shot scam lures such as fake lottery or unpaid toll messages, they unfold through multi-turn conversations in which scammers gradually manipulate victims using evolving psychological techniques. However, existing research mainly focuses on static scam detection or synthetic scams, leaving open whether
language models can understand how real-world scams progress over time. We introduce PreScam, a
benchmark for modeling scam progression from early conversations. Built from user-submitted scam reports, PreScam filters and structures 177,989 raw reports into 11,573 conversational scam instances spanning 20 scam categories. Each instance is hierarchically structured according to the scam lifecycle defined by the proposed scam kill chain, and further annotated at the turn level with scammer psychological actions and victim responses. We
benchmark models on two tasks: real-time termination prediction, which estimates whether a conversation is approaching the termination stage, and scammer action prediction, which forecasts the scammer's subsequent actions. Results show a clear gap between surface-level fluency and progression modeling: supervised encoders substantially outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs. Taken together, these results show that current models can capture some scam-related cues, yet still struggle to track how risk escalates and how manipulation unfolds across turns.
Read the full abstract ▾
All authors Weixiang Sun, Shang Ma, Yiyang Li, Tianyi Ma, Zehong Wang, Colby Nelson, Xusheng Xiao, Yanfang Ye
Computation and Language
Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs
Deepak Kumar, Baban Gain and Asif Ekbal
Automatic
Speech Recognition (ASR) transcripts often contain disfluencies, such as fillers, repetitions, and false starts, which reduce readability and hinder downstream applications like chatbots and voice assistants. If left unaddressed, such disfluencies can significantly degrade the reliability of downstream systems. Most existing approaches rely on classical models that focus on identifying disfluent
tokens for removal. While this strategy is effective to some extent, it often disrupts grammatical structure and semantic coherence, leading to incomplete or unnatural sentences. Recent literature explored the use of large
language models (LLMs); however, these efforts have primarily focused on disfluency detection or data augmentation, rather than performing comprehensive correction. We propose a
multilingual correction pipeline where a sequence tagger first marks disfluent
tokens, and these signals guide instruction fine-tuning of an LLM to rewrite transcripts into fluent text. To further improve reliability, we add a contrastive learning objective that penalizes the reproduction of disfluent
tokens, encouraging the model to preserve grammar and meaning while removing disfluent artifacts. Our experiments across three Indian
languages, namely Hindi, Bengali, and Marathi show consistent improvements over strong baselines, including
multilingual sequence-to-sequence models. These results highlight that detection-only strategies are insufficient. Combining
token-level cues with instruction tuning and contrastive learning provides a practical and scalable solution for
multilingual disfluency correction in
speech-driven NLP systems. We make the codes publicly available at https://github.com/deepak-kumar-98/Mind-the-Pause.
Read the full abstract ▾
Computation and Language
A Recipe for Long-Context Reasoning in Large Language Models via On-Policy Optimization and Distillation
Miguel Moura Ramos, Duarte M. Alves and André F. T. Martins
Existing approaches to post-train models for long-context tasks face complementary limitations: (i) supervised fine-tuning (SFT) provides stable supervision but suffers from exposure bias; (ii) reinforcement learning methods such as Group Relative Policy Optimization (GRPO) train on model-generated trajectories but struggle with long-horizon credit assignment and sparse rewards; and (iii) on-policy
distillation (OPD) provides dense
token-level guidance but does not directly optimize task rewards. We study these complementary strategies for long-context
alignment and derive a recipe that combines GRPO with OPD-style teacher guidance: the student learns from its own rollouts using outcome-level rewards, while a stronger teacher provides dense
token-level regularization in place of the standard reference policy. This is especially useful when process-level supervision is difficult to obtain. To support this study, we introduce LongBlocks, a synthetic
multilingual dataset spanning multi-hop
reasoning, contextual grounding, and long-form generation. Through controlled ablations, we isolate the roles of cold-start initialization, teacher anchoring, and data mixing, showing that our recipe yields a more stable and effective path to long-context
reasoning than GRPO or OPD while preserving short-context capabilities.
Read the full abstract ▾
Computation and Language
On the Interpretability of Whisper Encodings Using Sparse Autoencoders
Dan Pluth, Zachary Nicholas Houghton, Yu Zhou et al. (+1 more)
While deep
transformer-based models have advanced rapidly, their internal mechanisms remain largely a mystery. Recent work has prioritized understanding text-based
transformer models, leaving ASR systems largely unexplored. In order to address this gap, we examine the internal representations of Whisper's encoder using a sparse autoencoder. We find diverse monosemantic features across linguistic and non-linguistic boundaries, spanning a hierarchy from phonetic to semantic representations, and conduct a causal feature-steering campaign across this hierarchy, including cross-lingual steering. We further find that steering is more reliable for higher-level features than lower-level ones, an asymmetry that may reflect redundant encoding of lower-level information. Altogether, this work demonstrates that Whisper's encoder represents a surprisingly rich hierarchy of linguistic information that extends well beyond what is strictly necessary for transcription.
Read the full abstract ▾
All authors Dan Pluth, Zachary Nicholas Houghton, Yu Zhou, Vijay K. Gurbani
Computation and Language · Cross-listed from cs.LG
Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
Arijit Sehanobish and Charles Lovering
We study the parameter placement problem: given a fixed budget of $k$ trainable entries within the B
matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream-writing projections (V, O, Down), stable across model families and scales (1.5B - 8B).
Read the full abstract ▾
Computation and Language
Mitigating Context-Memory Conflicts in LLMs through Dynamic Cognitive Reconciliation Decoding
Yigeng Zhou, Wu Li, Yifan Lu et al. (+6 more)
Large
language models accumulate extensive parametric knowledge through pre-training. However, knowledge conflicts occur when outdated or incorrect parametric knowledge conflicts with external knowledge in the context. Existing methods address knowledge conflicts through contrastive decoding, but in conflict-free scenarios, static approaches disrupt output distribution. Other dynamic decoding methods attempt to measure the degree of conflict but still struggle with complex real-world situations. In this paper, we propose a two-stage decoding method called Dynamic Cognitive Reconciliation Decoding (DCRD), to predict and mitigate context-
memory conflicts. DCRD first analyzes the
attention map to assess context fidelity and predict potential conflicts. Based on this prediction, the input is directed to one of two decoding paths: (1) greedy decoding, or (2) context fidelity-based dynamic decoding. This design enables DCRD to handle conflicts efficiently while maintaining high accuracy and decoding efficiency in conflict-free cases. Additionally, to simulate scenarios with frequent knowledge updates, we constructed ConflictKG, a knowledge conflict QA
benchmark. Experiments on four LLMs across six QA
datasets show that DCRD outperforms all baselines, achieving state-of-the-art performance.
Read the full abstract ▾
All authors Yigeng Zhou, Wu Li, Yifan Lu, Yequan Wang, Xuebo Liu, Wenya Wang, Jun Yu, Min Zhang, Jing Li
Computation and Language · Cross-listed from cs.AI
Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics
Jishnu Sethumadhavan Nair, Patrice Bechard, Rishabh Maheshwary et al. (+14 more)
World models enable
agents to anticipate the effects of their actions by internalizing environment dynamics. In enterprise systems, however, these dynamics are often defined by tenant-specific business logic that varies across deployments and evolves over time, making models trained on historical transitions brittle under deployment shift. We ask a question the world-models literature has not addressed: when the rules can be read at inference time, does an
agent still need to learn them? We argue, and demonstrate empirically, that in settings where transition dynamics are configurable and readable, runtime discovery complements offline training by grounding predictions in the active system instance. We propose enterprise discovery
agents, which recover relevant transition dynamics at runtime by reading the system's configuration rather than relying solely on internalized representations. We introduce CascadeBench, a
reasoning-focused
benchmark for enterprise cascade prediction that adopts the evaluation methodology of World of Workflows on diverse synthetic environments, and use it together with deployment-shift evaluation to show that offline-trained world models can perform well in-distribution but degrade as dynamics change, whereas discovery-based
agents are more robust under shift by grounding their predictions in the current instance. Our findings suggest that, in configurable enterprise environments,
agents should not rely solely on fixed internalized dynamics, but should incorporate mechanisms for discovering relevant transition logic at runtime.
Read the full abstract ▾
All authors Jishnu Sethumadhavan Nair, Patrice Bechard, Rishabh Maheshwary, Surajit Dasgupta, Sravan Ramachandran, Aakash Bhagat, Shruthan Radhakrishna, Pulkit Pattnaik, Johan Obando-Ceron, Shiva Krishna Reddy Malay, Sagar Davasam, Seganrasan Subramanian, Vipul Mittal, Sridhar Krishna Nemala, Christopher Pal, Srinivas Sunkara, Sai Rajeswar
Computation and Language
Correcting Selection Bias in Sparse User Feedback for Large Language Model Quality Estimation: A Multi-Agent Hierarchical Bayesian Approach
Andrea Morandi and Mahesh Viswanathan
[Abridged] Production LLM deployments receive feedback from a non-random fraction of users: thumbs sit mostly in the tails of the satisfaction distribution, and a naive average over them can land 40-50 percentage points away from true system quality. We treat this as a topic- and sentiment- stratified selection-bias problem and propose a three-
agent hierarchical
Bayesian pipeline that does not require ground-truth labels on individual interactions. A Topic Clustering
Agent partitions the stream via UMAP + HDBSCAN over text
embeddings; a Bias Modeling
Agent fits a two-stage hierarchical Beta-Binomial under NUTS, inferring per-topic selection rates $s_c$ and quality $q_c$ with partial pooling; a Synthesis
Agent reweights $q_c$ by true topic prevalence $\hat\pi_c = n_c/N$ to report a bias-corrected aggregate posterior $\bar Q = \sum_c \hat\pi_c q_c$ with credible interval, plus drift signals for online recalibration. Validation uses UltraFeedback (N=10,232 retained interactions, $C=18$ clusters, $Q^\star=0.6249$) with simulated topic- and sentiment-dependent selection biases. We compare five
Bayesian variants against Naive and IPW baselines. A mild prior on the feedback channel (typical positive-feedback rate and negative-to-positive ratio, both readable from any production dashboard without labels) keeps Hierarchical-Informed within 4-13 pp of $Q^\star$ as the bias ratio sweeps from 1:1 to 30:1, with 95% credible intervals covering $Q^\star$ in 50/50 random-seed replicates at $\kappa_{\max}=10$. Without channel-side priors, every weak-prior variant misses $Q^\star$ by 22-33 pp: the per-cluster sufficient statistics admit a one-parameter family of equally good fits, and the prior on the bias channel (not on latent quality) is what breaks the degeneracy.
Read the full abstract ▾
Computation and Language
Latent Causal Void: Explicit Missing-Context Reconstruction for Misinformation Detection
Hui Li, Zhongquan Jian, Jinsong Su et al. (+1 more)
Automatic misinformation detection performs well when deception is visible in what an article explicitly states. However, some misinformation articles remain locally coherent and only become misleading once compared with contemporaneous reports that supply background facts the article omits. We study this omission-relevant setting and observe that current omission-aware approaches typically either attach retrieved context as auxiliary evidence or infer a categorical omission signal, leaving the specific missing fact implicit. We propose Latent Causal Void (LCV), a
retrieval-guided detector that explicitly reconstructs the missing fact for each target sentence and uses it as a textual cross-source relation in graph
reasoning. Concretely, LCV retrieves temporally aligned context articles, asks a frozen instruction-tuned large
language model to generate a short missing-context description for each sentence–article pair, and feeds the resulting relation text into a heterograph over target sentences and context articles. On the bilingual
benchmark of Sheng et al., LCV improves over the strongest omission-aware baseline by $2.56$ and $2.84$ macro-F1 points on the English and Chinese splits, respectively. The results indicate that modeling the missing cross-source fact itself, rather than only attaching retrieved evidence or predicting an omission signal, is a useful representation for omission-aware misinformation detection.
Read the full abstract ▾
All authors Hui Li, Zhongquan Jian, Jinsong Su, Junfeng Yao
Computation and Language · Cross-listed from cs.CV
Design Your Ad: Personalized Advertising Image and Text Generation with Unified Autoregressive Models
Yexing Xu, Wei Feng, Shen Zhang et al. (+15 more)
Generating realistic and user-preferred advertisements is a key challenge in e-commerce. Existing approaches utilize multiple independent models driven by click-through-rate (CTR) to controllably create attractive image or text advertisements. However, their pipelines lack cross-modal perception and rely on CTR that only reflects average preferences. Therefore, we explore jointly generating personalized image-text advertisements from historical click behaviors. We first design a Unified Advertisement Generative model (Uni-AdGen) that employs a single autoregressive framework to produce both advertising images and texts. By incorporating a foreground perception module and instruction tuning, Uni-AdGen enhances the realism of the generated content. To further personalize advertisements, we equip Uni-AdGen with a coarse-to-fine preference understanding module that effectively captures user interests from noisy
multimodal historical behaviors to drive personalized generation. Additionally, we construct the first large-scale Personalized Advertising image-text
dataset (PAd1M) and introduce a Product Background Similarity (PBS) metric to facilitate training and evaluation. Extensive experiments show that our method outperforms baselines in general and personalized advertisement generation. Our project is available at https://github.com/JD-GenX/Uni-AdGen.
Read the full abstract ▾
All authors Yexing Xu, Wei Feng, Shen Zhang, Haohan Wang, Yuxin Qin, Yaoyu Li, Ao Ma, Yuhao Luo, Lu Wang, Xudong Ren, Haoran Wang, Run Ling, Zheng Zhang, Jingjing Lv, Junjie Shen, Ching Law, Longguang Wang, Yulan Guo
Computation and Language
Metaphor Is Not All Attention Needs
Olga Sorokoletova, Francesco Giarrusso, Giacomo De Luca et al. (+6 more)
Large
language models are increasingly deployed in safety-critical applications, where their ability to resist harmful instructions is essential. Although post-training aims to make models robust against many jailbreak strategies, recent evidence shows that stylistic reformulations, such as poetic transformation, can still bypass safety mechanisms with alarming effectiveness. This raises a central question: why do literary jailbreaks succeed? In this work, we investigate whether their effectiveness depends on specific poetic devices, on a failure to recognize literary formatting, or on deeper changes in how models process stylistically irregular
prompts. We address this problem through an interpretability analysis of
attention patterns. We perform input-level ablation studies to assess the contribution of individual and combinations of poetic devices; construct an interpretable vector representation of
attention maps; cluster these representations and train linear probes to predict safety outcomes and literary format. Our results show that models distinguish poetic from prose formats with high accuracy, yet struggle to predict jailbreak success within each format. Clustering further reveals clear separation by literary format, but not by safety label. These findings indicate that jailbreak success is not caused by a failure to recognize poetic formatting; rather, poetic
prompts induce distinct processing patterns that remain largely independent of harmful-content detection. Overall, literary jailbreaks appear to misalign large
language models not through any single poetic device, but through accumulated stylistic irregularities that alter
prompt processing and avoid lexical triggers considered during post-training. This suggests that robustness requires safety mechanisms that account for style-induced shifts in model behavior. We use Qwen3-14B as a representative open-weight case study.
Read the full abstract ▾
All authors Olga Sorokoletova, Francesco Giarrusso, Giacomo De Luca, Piercosma Bisconti, Matteo Prandi, Federico Pierucci, Marcello Galisai, Vincenzo Suriani, Daniele Nardi
Computation and Language
Sign Language Recognition and Translation for Low-Resource Languages: Challenges and Pathways Forward
Nigar Alishzade and Gulchin Abdullayeva
Sign
languages are natural, visual-gestural
languages used by Deaf communities worldwide. Over 300 distinct sign
languages remain severely low-resource due to limited documentation, sparse
datasets, and insufficient computational tools. This systematic review synthesizes literature on sign
language recognition and translation for under-resourced
languages, using Azerbaijan Sign
Language (AzSL) as a case study. Analysis of global initiatives extracts eight actionable lessons, including community co-design, dialectal diversity capture, and
privacy-preserving pose-based representations. Turkic sign
languages (Kazakh, Turkish, Azerbaijani) receive special
attention, as linguistic proximity enables effective transfer learning. We propose three paradigm shifts: from architecture-centric to data-centric AI, from signer-independent to signer-adaptive systems, and from reference-based to task-specific evaluation metrics. A technical roadmap for AzSL leverages lightweight MediaPipe-based architectures, community-validated annotations, and offline-first deployment. Progress requires sustained interdisciplinary collaboration centered on Deaf communities to ensure cultural authenticity, ethical governance, and practical communication benefit.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
World Action Models: The Next Frontier in Embodied AI
Siyin Wang, Junhao Shi, Zhaoyang Fu et al. (+11 more)
Vision-
Language-Action (VLA) models have achieved strong semantic generalization for
embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs):
embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning
robot teleoperation, portable human demonstrations, simulation, and internet-scale
egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.
Read the full abstract ▾
All authors Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He, Feihong Liu, Chenchen Yang, Yikang Zhou, Zhaoye Fei, Jingjing Gong, Jinlan Fu, Mike Zheng Shou, Xuanjing Huang, Xipeng Qiu, Yu-Gang Jiang
Computation and Language
Do Language Models Encode Knowledge of Linguistic Constraint Violations?
Hardy and Sebastian Padó
Large
Language Models (LLMs) achieve strong linguistic performance, yet their internal mechanisms for producing these predictions remain unclear. We investigate the hypothesis that LLMs encode representations of linguistic constraint violations within their parameters, which are selectively activated when processing ungrammatical sentences. To test this, we use sparse autoencoders to decompose polysemantic activations into sparse, monosemantic features and recover candidates for violation-related features. We introduce a sensitivity score for identifying features that are preferentially activated on constraint-violated versus well-formed inputs, enabling unsupervised detection of potential violation-specific features. We further propose a conjunctive falsification framework with three criteria evaluated jointly. Overall, the results are negative in two respects: (1) the falsification criteria are not jointly satisfied across linguistic phenomena, and (2) no features are consistently shared across all categories. While some phenomena show partial evidence of selective causal structure, the overall pattern provides limited support for a unified set of grammatical violation detectors in current LMs.
Read the full abstract ▾
Computation and Language
Is Child-Directed Language Optimized for Word Learning? A Computational Study of Verb Meaning Acquisition
Francesca Padovani, Jaap Jumelet, Yevgen Matusevych et al. (+1 more)
Is child-directed
language (CDL) optimized to support
language learning, and which aspects of linguistic development does it facilitate? We investigate this question using neural
language models trained on CDL versus adult-directed
language (ADL). We selectively remove syntactic or lexical co-occurrence information from the model training data, and evaluate the impact of these manipulations on verb meaning acquisition. While disrupting syntax impairs learning across all
datasets, models trained on CDL and spoken ADL show significantly higher resilience than those trained on written input. Tracking semantic and syntactic performance over training, we observe a semantic-first trajectory, with verb meanings emerging prior to robust syntactic proficiency, an asynchrony most pronounced in the spoken domain, especially CDL. These results suggest that the advantage for verb learning previously attributed to CDL may instead reflect broader properties of the spoken register, rather than a uniquely CDL-specific optimization.
Read the full abstract ▾
All authors Francesca Padovani, Jaap Jumelet, Yevgen Matusevych, Arianna Bisazza
Computation and Language
SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs
Xiaoyuan Li, Moxin Li, Keqin Bao et al. (+4 more)
Skill libraries enable large
language model
agents to reuse experience from past interactions, but most existing libraries store skills as isolated entries and retrieve them only by semantic similarity. This leads to two key challenges for compositional tasks. Firstly, an
agent must identify not only relevant skills but also how they depend on and build upon each other. Secondly, it also makes library maintenance difficult, since the system lacks structural cues for deciding when skills should be merged, split, or removed. We propose SKILLGRAPH, a framework that represents reusable skills as nodes in a directed graph, with typed edges encoding prerequisite, enhancement, and co-occurrence relations. Given a new task, SKILLGRAPH retrieves not just individual skills, but an ordered skill subgraph that can guide multi-step decision making. The graph is continuously updated from
agent trajectories and reinforcement learning feedback, allowing both the skill library and the
agent policy to improve together. Experiments on ALFWorld, WebShop, and seven search-augmented QA tasks show that SKILLGRAPH achieves state-of-the-art performance against
memory-augmented RL methods, with especially large gains on complex tasks that require composing multiple skills.
Read the full abstract ▾
All authors Xiaoyuan Li, Moxin Li, Keqin Bao, Yubo Ma, Wenjie Wang, Dayiheng Liu, Fuli Feng
Computation and Language
Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking
David-Maximilian Caraman and Gheorghe Cosmin Silaghi
We describe our system for SemEval-2026 Task 8 (MTRAGEval), participating in Task A (
Retrieval) across four English-
language domains. Our approach employs a three-stage pipeline: (1) query rewriting via a LoRA-fine-tuned Qwen 2.5 7B model that transforms context-dependent follow-up questions into standalone queries, (2) hybrid BM25 and dense
retrieval combined through Reciprocal Rank Fusion, and (3) cross-encoder reranking with BGE-reranker-v2-m3. On the official test set, the system achieves nDCG@5 of 0.531, ranking 8th out of 38 participating systems and 10.7% above the organizer baseline. Development comparisons reveal that domain-specific temperature tuning for query generation, where technical domains benefit from deterministic decoding and general domains from controlled randomness, provides consistent gains, while more complex strategies such as domain-aware prompting and multi-query expansion degrade performance.
Read the full abstract ▾
Computation and Language
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
Xiaoyuan Li, Yuzhe Wang, Moxin Li et al. (+6 more)
Large
Language Models (LLMs) achieve strong performance on standard knowledge evaluation
benchmarks, yet recent work shows that their knowledge capabilities remain brittle under question variants that test the same knowledge in different forms. Robustness augmentation of existing knowledge evaluation
benchmarks is therefore necessary, but current LLM-assisted generate-then-verify pipelines are costly and difficult to scale due to low-yield variant generation and unreliable variant verification. We propose SAGE (Scalable Automated Generation of Robustness
BEnchmarks), a framework for scalable robustness augmentation of knowledge evaluation
benchmarks using fine-tuned smaller models. SAGE consists of VariantQual, a rubric-based verifier trained on human-labeled seed data, and VariantGen, a variant generator initialized with supervised fine-tuning and further optimized with reinforcement learning using VariantQual as the reward model. Experiments on HellaSwag show that SAGE constructs a large-scale robustness-augmented
benchmark with quality comparable to the human-annotated HellaSwag-Pro at substantially lower cost, while the fine-tuned models further generalize to MMLU without
benchmark-specific fine-tuning.
Read the full abstract ▾
All authors Xiaoyuan Li, Yuzhe Wang, Moxin Li, Keqin Bao, Rui Men, Yichang Zhang, Dayiheng Liu, Wenjie Wang, Fuli Feng
Computation and Language · Cross-listed from cs.CR
SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
Chang Jin, An Wang, Zeming Wei et al. (+7 more)
Reusable skills are becoming a common interface for extending large
language model
agents, packaging procedural guidance with access to files, tools,
memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the
agent toward unsafe actions. We present SkillSafetyBench, a runnable
benchmark for evaluating such skill-facing safety failures. SkillSafetyBench includes 155
adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI
agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that
agent safety depends not only on model-level
alignment, but also on how
agents interpret skills, trust workflow context, and act through executable environments. The complete
benchmark is available at https://github.com/AI45Lab/skill-safety-bench.
Read the full abstract ▾
All authors Chang Jin, An Wang, Zeming Wei, Kai Wang, Biaojie Zeng, Qiaosheng Zhang, Chao Yang, Jingjing Qu, Xia Hu, Xingcheng Xu