Computation and Language · Cross-listed from cs.SE
Attention Mechanism and Heuristic Approach: Context-Aware File Ranking Using Multi-Head Self-Attention
Pradeep Kumar Sharma, Shantanu Godbole, Sarada Prasad Jena et al. (+1 more)
The identification and ranking of impacted files within software reposi-tories is a key challenge in change impact analysis. Existing deterministic approaches that combine heuristic signals, semantic similarity measures, and graph-based centrality metrics have demonstrated effectiveness in nar-rowing candidate search spaces, yet their recall plateaus. This limitation stems from the treatment of features as linearly independent contributors, ignoring contextual dependencies and relationships between metrics that characterize expert
reasoning patterns. To address this limitation, we propose the application of Multi-Head Self-
Attention as a post-deterministic scoring refinement mechanism. Our approach learns contextual weighting between features, dynamically adjust-ing importance levels per file based on relational behavior exhibited across candidate file sets. The
attention mechanism produces context-aware adjustments that are additively combined with deterministic scores, pre-serving interpretability while enabling
reasoning similar to that performed by experts when reviewing change surfaces. We focus on recall rather than precision, as false negatives (missing impacted files) are far more costly than false positives (irrelevant files that can be quickly dismissed during review). Empirical evaluation on 200 test cases demonstrates that the introduc-tion of self-
attention improves Top-50 recall from approximately 62-65% to between 78-82% depending on repository complexity and structure, achiev-ing 80% recall at Top-50 files. Expert validation yields improvement from 6.5/10 to 8.6/10 in subjective accuracy
alignment. This transformation bridges the
reasoning capability gap between deterministic automation and expert judgment, improving recall in repository-aware effort estimation.
Read the full abstract ▾
All authors Pradeep Kumar Sharma, Shantanu Godbole, Sarada Prasad Jena, Hritvik Shrivastava
Computation and Language · Cross-listed from cs.LG
MixDPO: Modeling Preference Strength for Pluralistic Alignment
Saki Imai, Pedram Heydari, Anthony Sicilia et al. (+3 more)
Preference based
alignment objectives implicitly assume that all human preferences are expressed with equal strength. In practice, however, preference strength varies across individuals and contexts – a phenomenon established in behavioral economics and discrete choice theory. This mismatch limits the ability of existing objectives to faithfully capture heterogeneous human judgments. Inspired by this literature, we introduce Mixed Logit Direct Preference Optimization (MixDPO), a generalization of Direct Preference Optimization that models variation in preference strength. MixDPO enables
alignment objectives to capture heterogeneity in how strongly preferences are expressed across training examples. We evaluate MixDPO on three preference
datasets using two open-weight
language models. Across
datasets, MixDPO improves aggregate
alignment performance (+11.2 points on Pythia-2.8B) while preserving subgroup level preferences, with the largest gains appearing in settings with higher inferred preference heterogeneity. MixDPO makes preference heterogeneity explicit through learned strength distributions. We release our code for reproducibility.
Read the full abstract ▾
All authors Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani
Computation and Language
Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
San Kim and Gary Geunbae Lee
Large
Language Models (LLMs) have greatly advanced Natural
Language Processing (NLP), particularly through instruction tuning, which enables broad task generalization without additional fine-tuning. However, their reliance on large-scale
datasets-often collected from human or web sources-makes them vulnerable to
backdoor attacks, where adversaries poison a small subset of data to implant hidden behaviors. Despite this growing risk, defenses for instruction-tuned models remain underexplored. We propose MB-Defense (Merging & Breaking Defense Framework), a novel training pipeline that immunizes instruction-tuned LLMs against diverse
backdoor threats. MB-Defense comprises two stages: (i) Defensive Poisoning, which merges attacker and defensive triggers into a unified
backdoor representation, and (ii)
Backdoor Neutralization, which breaks this representation through additional training to restore clean behavior. Extensive experiments across multiple LLMs show that MB-Defense substantially lowers attack success rates while preserving instruction-following ability. Our method offers a generalizable and data-efficient defense strategy, improving the robustness of instruction-tuned LLMs against unseen
backdoor attacks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization
Xingjian Diao, Zheyuan Liu, Chunhui Zhang et al. (+6 more)
Large
Vision-
Language Models (LVLMs) have exhibited strong
reasoning capabilities through chain-of-thought mechanisms that generate step-by-step rationales. However, such slow-thinking approaches often lead to overthinking, where models produce excessively verbose responses even for simple queries, resulting in test-time inefficiency and even degraded accuracy. Prior work has attempted to mitigate this issue via adaptive
reasoning strategies, but these methods largely overlook a fundamental bottleneck: visual perception failures. We argue that stable
reasoning critically depends on low-level visual grounding, and that
reasoning errors often originate from imperfect perception rather than insufficient deliberation. To address this limitation, we propose Gated Perception-
Reasoning Optimization (GPRO), a meta-
reasoning controller that dynamically routes computation among three decision paths at each generation step: a lightweight fast path, a slow perception path for re-examining visual inputs, and a slow
reasoning path for internal self-reflection. To learn this distinction, we derive large-scale failure attribution supervision from approximately 790k samples, using teacher models to distinguish perceptual
hallucinations from
reasoning errors. We then train the controller with multi-objective reinforcement learning to optimize the trade-off between task accuracy and computational cost under uncertainty. Experiments on five
benchmarks demonstrate that GPRO substantially improves both accuracy and efficiency, outperforming recent slow-thinking methods while generating significantly shorter responses.
Read the full abstract ▾
All authors Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Weiyi Wu, Keyi Kong, Lin Shi, Kaize Ding, Soroush Vosoughi, Jiang Gui
Computation and Language
Learning to Simulate Human Dialogue
Kanishk Gandhi, Agam Bhatia and Noah D. Goodman
To predict what someone will say is to model how they think. We study this through next-turn dialogue prediction: given a conversation, predict the next utterance produced by a person. We compare learning approaches along two dimensions: (1) whether the model is allowed to think before responding, and (2) how learning is rewarded either through an LLM-as-a-judge that scores semantic similarity and information completeness relative to the ground-truth response, or by directly maximizing the log-probability of the true human dialogue. We find that optimizing for judge-based rewards indeed increases judge scores throughout training, however it decreases the likelihood assigned to ground truth human responses and decreases the win rate when human judges choose the most human-like response among a real and synthetic option. This failure is amplified when the model is allowed to think before answering. In contrast, by directly maximizing the log-probability of observed human responses, the model learns to better predict what people actually say, improving on both log-probability and win rate evaluations. Treating chain-of-thought as a latent variable, we derive a lower bound on the log-probability. Optimizing this objective yields the best results on all our evaluations. These results suggest that thinking helps primarily when trained with a distribution-matching objective grounded in real human dialogue, and that scaling this approach to broader conversational data may produce models with a more nuanced understanding of human behavior.
Read the full abstract ▾
Computation and Language
Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
Myra Cheng, Robert D. Hawkins and Dan Jurafsky
Large
language models (LLMs) frequently fail to challenge users' harmful beliefs in domains ranging from medical advice to social
reasoning. We argue that these failures can be understood and addressed pragmatically as consequences of LLMs defaulting to accommodating users' assumptions and exhibiting insufficient epistemic vigilance. We show that social and linguistic factors known to influence accommodation in humans (at-issueness, linguistic encoding, and source reliability) similarly affect accommodation in LLMs, explaining performance differences across three safety
benchmarks that test models' ability to challenge harmful beliefs, spanning misinformation (Cancer-Myth, SAGE-Eval) and sycophancy (ELEPHANT). We further show that simple pragmatic interventions, such as adding the phrase "wait a minute", significantly improve performance on these
benchmarks while preserving low false-positive rates. Our results highlight the importance of considering pragmatics for evaluating LLM behavior and improving LLM safety.
Read the full abstract ▾
Computation and Language
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Yao Dou, Benjamin Mamut and Wei Xu
Large
language models (LLMs) now support contexts of up to 1M
tokens, but their strengths and weaknesses on complex long-context tasks remain unclear. To study this, we focus on multi-document legal case summarization, where a single case often spans many documents exceeding 100K
tokens. We systematically evaluate 12 frontier LLMs with Gavel, which consists of Gavel-Ref, a reference-based evaluation framework with checklist, residual-fact, and writing-style evaluations, and Gavel-
Agent, a reference-free
agent for evaluating factual coverage directly from source documents. Our results show that current models are more prone to omitting key information than hallucinating. They all perform well on simple checklist items, such as filing date, but struggle with rare and complex items, such as settlements. Performance also declines as case length increases. To meta-evaluate Gavel, we collect 160 hours of human annotations. Gavel-
Agent reduces
token usage by at least 36% compared to end-to-end and chunk-by-chunk methods while achieving competitive performance. Gavel-
Agent also generalizes to the medical domain, performing the best with at least 77% fewer
tokens.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards
Ali Rad, Khashayar Filom, Darioush Keivan et al. (+2 more)
Reinforcement learning with verifiable rewards (RLVR) is a simple but powerful paradigm for training LLMs: sample a completion, verify it, and update. In practice, however, the verifier is almost never clean–unit tests probe only limited corner cases; human and synthetic labels are imperfect; and LLM judges (e.g., RLAIF) are noisy and can be exploited–and this problem worsens on harder domains (especially coding) where tests are sparse and increasingly model-generated. We ask a pragmatic question: does the verification noise merely slow down the learning (rate), or can it flip the outcome (fate)? To address this, we develop an analytically tractable multi-armed bandit view of RLVR dynamics, instantiated with GRPO and validated in controlled experiments. Modeling false positives and false negatives and grouping completions into recurring
reasoning modes yields a replicator-style (natural-selection) flow on the probability simplex. The dynamics decouples into within-correct-mode competition and a one-dimensional
evolution for the mass on incorrect modes, whose drift is determined solely by Youden's index J=TPR-FPR. This yields a sharp phase transition: when J>0, the incorrect mass is driven toward extinction (learning); when J=0, the process is neutral; and when J<0, incorrect modes amplify until they dominate (anti-learning and collapse). In the learning regime J>0, noise primarily rescales convergence time ("rate, not fate"). Experiments on verifiable programming tasks under synthetic noise reproduce the predicted J=0 boundary. Beyond noise, the framework offers a general lens for analyzing RLVR stability, convergence, and algorithmic interventions.
Read the full abstract ▾
All authors Ali Rad, Khashayar Filom, Darioush Keivan, Peyman Mohajerin Esfahani, Ehsan Kamalinejad
Computation and Language
Mason Kadem and Rong Zheng
Neural networks are growing more capable on their own, but we do not understand their neural mechanisms. Understanding these mechanisms' decision-making processes, or mechanistic interpretability, enables (1) accountability and control in high-stakes domains, (2) the study of digital
brains and the emergence of cognition, and (3) discovery of new knowledge when AI systems outperform humans. This paper traces how
attention head intervention emerged as a key method for causal interpretability of
transformers. The
evolution from visualization to intervention represents a paradigm shift from observing correlations to causally validating mechanistic hypotheses through direct intervention. Head intervention studies revealed robust empirical findings while also highlighting limitations that complicate interpretation. Recent work demonstrates that mechanistic understanding now enables targeted control of model behaviour, successfully suppressing toxic outputs and manipulating semantic content through selective
attention head intervention, validating the practical utility of interpretability research for AI safety.
Read the full abstract ▾
Computation and Language
ARREST: Adversarial Resilient Regulation Enhancing Safety and Truth in Large Language Models
Sharanya Dasgupta, Arkaprabha Basu, Sujoy Nath et al. (+1 more)
Human cognition, driven by complex neurochemical processes, oscillates between imagination and reality and learns to self-correct whenever such subtle drifts lead to
hallucinations or unsafe associations. In recent years, LLMs have demonstrated remarkable performance in a wide range of tasks. However, they still lack human cognition to balance factuality and safety. Bearing the resemblance, we argue that both factual and safety failures in LLMs arise from a representational misalignment in their latent activation space, rather than addressing those as entirely separate
alignment issues. We hypothesize that an external network, trained to understand the fluctuations, can selectively intervene in the model to regulate falsehood into truthfulness and unsafe output into safe output without fine-tuning the model parameters themselves. Reflecting the hypothesis, we propose ARREST (
Adversarial Resilient Regulation Enhancing Safety and Truth), a unified framework that identifies and corrects drifted features, engaging both soft and hard refusals in addition to factual corrections. Our empirical results show that ARREST not only regulates misalignment but is also more versatile compared to the RLHF-aligned models in generating soft refusals due to
adversarial training. We make our codebase available at https://github.com/sharanya-dasgupta001/ARREST.
Read the full abstract ▾
All authors Sharanya Dasgupta, Arkaprabha Basu, Sujoy Nath, Swagatam Das
Computation and Language
Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
Iago Alves Brito, Walcy Santos Rezende Rios, Julia Soares Dollis et al. (+2 more)
Current safety evaluations of large
language models (LLMs) create a dangerous illusion of universal protection by aggregating harms under generic categories such as "Identity Hate", obscuring vulnerabilities toward specific populations. In this work, we expose the Selective Safety Trap: a systemic failure mode where models robustly defend specific populations while leaving underrepresented communities highly vulnerable to identical
adversarial attacks. To systematically audit this phenomenon, we introduce MiJaBench, a bilingual (English-Portuguese)
adversarial benchmark comprising 43,961 controlled jailbreaking
prompts across 16 minority groups. By evaluating 14 state-of-the-art LLMs on MiJaBench, we curate 615,454
prompt-response pairs that compose MiJaBench-Align, revealing that safety
alignment is not a uniform semantic capability but a demographic hierarchy, with defense rates fluctuating by up to 42% within the same model solely based on the target group. This disparity persists across architectures and
languages and is amplified by scaling, indicating that current
alignment methods learn group-specific safeguards rather than a generalized notion of harm. Through targeted direct preference optimization (DPO) on a 1B-parameter baseline, we achieve strong zero-shot safety generalizations to entirely unseen demographics and complex attack strategies. We release all
datasets and scripts to provide the community with a concrete pathway toward equitable, transferable safety
alignment.
Read the full abstract ▾
All authors Iago Alves Brito, Walcy Santos Rezende Rios, Julia Soares Dollis, Diogo Fernandes Costa Silva, Arlindo Rodrigues Galvão Filho
Computation and Language · Cross-listed from cs.AI
The Language of Bargaining: Linguistic Effects in LLM Negotiations
Stuti Sinha, Himanshu Kumar, Aryan Raju Mandapati et al. (+2 more)
Negotiation is a core component of social intelligence, requiring
agents to balance strategic
reasoning, cooperation, and social norms. Recent work shows that LLMs can engage in multi-turn negotiation, yet nearly all evaluations occur exclusively in English. Using controlled multi-
agent simulations across Ultimatum, Buy-Sell, and Resource Exchange games, we systematically isolate
language effects across English and four Indic framings (Hindi, Punjabi, Gujarati, Marwadi) by holding game rules, model parameters, and incentives constant across all conditions. We find that
language choice can shift outcomes more strongly than changing models, reversing proposer advantages and reallocating surplus. Crucially, effects are task-contingent: Indic
languages reduce stability in distributive games yet induce richer exploration in integrative settings. Our results demonstrate that evaluating LLM negotiation solely in English yields incomplete and potentially misleading conclusions. These findings caution against English-only evaluation of LLMs and suggest that culturally-aware evaluation is essential for fair deployment.
Read the full abstract ▾
All authors Stuti Sinha, Himanshu Kumar, Aryan Raju Mandapati, Rakshit Sakhuja, Dhruv Kumar
Computation and Language
Disco-RAG: Discourse-Aware Retrieval-Augmented Generation
Dongqi Liu, Hang Ding, Qiming Feng et al. (+6 more)
Retrieval-Augmented Generation (RAG) has emerged as an important means of enhancing the performance of large
language models (LLMs) in knowledge-intensive tasks. However, most existing RAG strategies treat retrieved passages in a flat and unstructured way, which prevents the model from capturing structural cues and constrains its ability to synthesize knowledge from dispersed evidence across documents. To overcome these limitations, we propose Disco-RAG, a discourse-aware framework that explicitly injects discourse signals into the generation process. Our method constructs intra-chunk discourse trees to capture local hierarchies and builds inter-chunk rhetorical graphs to model cross-passage coherence. These structures are jointly integrated into a planning blueprint that conditions the generation. Experiments on question answering and long-document summarization
benchmarks show the efficacy of our approach. Disco-RAG achieves state-of-the-art results on the
benchmarks without fine-tuning. These findings underscore the important role of discourse structure in advancing RAG systems.
Read the full abstract ▾
All authors Dongqi Liu, Hang Ding, Qiming Feng, Xurong Xie, Zhucun Xue, Chengjie Wang, Jian Li, Jiangning Zhang, Yabiao Wang
Computation and Language
Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties
Akriti Dhasmana, Aarohi Srivastava and David Chiang
We conduct an empirical study of cross-lingual transfer using spontaneous, noisy, and code-mixed
speech across a wide range of Indic dialects and
language varieties. Our results indicate that although ASR performance is generally improved with reduced phylogenetic distance between
languages, this factor alone does not fully explain performance in dialectal settings. Often, fine-tuning on smaller amounts of dialectal data yields performance comparable to fine-tuning on larger amounts of phylogenetically-related, high-resource standardized
languages. We also present a case study on Garhwali, a low-resource Pahari
language variety, and evaluate multiple contemporary ASR models. Finally, we analyze transcription errors to examine bias toward pre-training
languages, providing additional insight into challenges faced by ASR systems on dialectal and non-standardized
speech.
Read the full abstract ▾
Computation and Language · Cross-listed from physics.soc-ph
Generalization to Political Beliefs from Fine-Tuning on Sports Team Preferences
Owen Terry
Fine-tuned LLMs often exhibit unexpected behavior as a result of generalizing beyond the data they're shown. We present results in which an LLM fine-tuned to prefer either coastal sports teams or Southern sports teams adopt political beliefs that diverge significantly from those of the base model. While we hypothesized that the coastal model would become more liberal and the southern model would become more conservative, we find that their responses are usually similar to each other, without a clear-cut liberal or conservative bias. In addition to asking the models for numerical ratings of agreement with relevant political statements, we ask them to elaborate on their more radical answers, finding varying degrees of willingness to justify themselves. Further work is needed to understand the mechanisms by which fine-tuning on simple, narrow
datasets leads to seemingly unrelated changes in model behavior.
Read the full abstract ▾
Computation and Language
RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation
Joseph James, Chenghao Xiao, Yucheng Li et al. (+2 more)
Scientific rigour tends to be sidelined in favour of bold statements, leading authors to overstate claims beyond what their results support. We present RIGOURATE, a two-stage
multimodal framework that retrieves supporting evidence from a paper's body and assigns each claim an overstatement score. The framework consists of a
dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers, annotated using eight LLMs, with overstatement scores calibrated using peer-review comments and validated through human evaluation. It employes a fine-tuned reranker for evidence
retrieval and a fine-tuned model to predict overstatement scores with justification. Compared to strong baselines, RIGOURATE enables improved evidence
retrieval and overstatement detection. Overall, our work operationalises evidential proportionality and supports clearer, more transparent scientific communication.
Read the full abstract ▾
All authors Joseph James, Chenghao Xiao, Yucheng Li, Nafise Sadat Moosavi, Chenghua Lin
Computation and Language · Cross-listed from cs.LG
Quantifying the Effect of Test Set Contamination on Generative Evaluations
Rylan Schaeffer, Joshua Kazdan, Baber Abbasi et al. (+8 more)
As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabilities. While research has thoroughly investigated the impact of test set contamination on discriminative evaluations like multiple-choice question-answering, comparatively little research has studied the impact of test set contamination on generative evaluations. In this work, we quantitatively assess the effect of test set contamination on generative evaluations through the
language model lifecycle. We pretrain
language models on mixtures of web data and the MATH
benchmark, sweeping model sizes and number of test set replicas contaminating the pretraining corpus; performance improves with contamination and model size. Using scaling laws, we make a surprising discovery: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study further training: overtraining with fresh data reduces the effects of contamination, whereas supervised finetuning on the training set can either increase or decrease performance on test data, depending on the amount of pretraining contamination. Finally, at inference, we identify factors that modulate memorization: high sampling temperatures mitigate contamination effects, and longer solutions are exponentially more difficult to memorize than shorter ones, presenting a contrast with discriminative evaluations, where solutions are only a few
tokens in length. By characterizing how generation and memorization interact, we highlight a new layer of complexity for trustworthy evaluation of AI systems.
Read the full abstract ▾
All authors Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Fazl Berez, Abhay Puri, Stella Biderman, Niloofar Mireshghallah, Sanmi Koyejo
Computation and Language · Cross-listed from cs.LG
Mitigating Position-Shift Failures in Text-Based Modular Arithmetic via Position Curriculum and Template Diversity
Nikolay Yudin
Building on insights from the grokking literature, we study character-level
Transformers trained to compute modular addition from text, and focus on robustness under input-format variation rather than only in-distribution accuracy. We identify a previously under-emphasized failure mode: models that achieve high in-distribution accuracy can fail catastrophically when the same expression is shifted to different absolute character positions ("position shift") or presented under out-of-distribution natural-
language templates. Using a disjoint-pair split over all ordered pairs for p=97, we show that a baseline model reaches strong in-distribution performance yet collapses under position shift and template OOD. We then introduce a simple training recipe that combines (i) explicit expression boundary markers, (ii) position curriculum that broadens the range of absolute positions seen during training, (iii) diverse template mixtures, and (iv) consistency training across multiple variants per example. Across three seeds, this intervention substantially improves robustness to position shift and template OOD while maintaining high in-distribution accuracy, whereas an ALiBi-style ablation fails to learn the task under our setup. Our results suggest that steering procedural generalization under noisy supervision benefits from explicitly training invariances that are otherwise absent from the data distribution, and we provide a reproducible evaluation protocol and artifacts.
Read the full abstract ▾
Computation and Language
From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning
Xiaoyu Xu, Minxin Du, Zitong Li et al. (+6 more)
Although machine unlearning is essential for removing private, harmful, or copyrighted content from LLMs, current
benchmarks often fail to faithfully represent the true “forgetting scope” learned by the model. We formalize two distinct unlearning granularities, domain-level and instance-level, and propose \BiForget, an automated framework for synthesizing high-quality forget sets. Unlike prior work relying on external generators, \BiForget exploits the target model per se to elicit data that matches its internal knowledge distribution through seed-guided and
adversarial prompting. Our experiments across diverse
benchmarks show that it achieves a superior balance of relevance, diversity, and efficiency. Quantitatively, in the Harry Potter domain, it improves relevance by ${\sim}20$ and diversity by ${\sim}$0.05 while halving the total data size compared to SOTAs. Ultimately, it facilitates more robust forgetting and better utility preservation, providing a more rigorous foundation for evaluating LLM unlearning.
Read the full abstract ▾
All authors Xiaoyu Xu, Minxin Du, Zitong Li, Zi Liang, Zhibiao Guo, Shiyu Zhang, Peizhao Hu, Qingqing Ye, Haibo Hu
Computation and Language · Cross-listed from cs.CR
Shadow Unlearning: A Neuro-Semantic Approach to Fidelity-Preserving Faceless Forgetting in LLMs
Dinesh Srivasthav P, Ashok Urlana, Rahul Mishra et al. (+2 more)
Machine unlearning aims to selectively remove the influence of specific training samples to satisfy
privacy regulations such as the GDPR's 'Right to be Forgotten'. However, many existing methods require access to the data being removed, exposing it to membership inference attacks and potential misuse of Personally Identifiable Information (PII). We address this critical challenge by proposing Shadow Unlearning, a novel paradigm of approximate unlearning, that performs machine unlearning on anonymized forget data without exposing PII. We further propose a novel
privacy-preserving framework, Neuro-Semantic Projector Unlearning (NSPU) to achieve Shadow unlearning. To evaluate our method, we compile Multi-domain Fictitious Unlearning (MuFU) forget set across five diverse domains and introduce an evaluation stack to quantify the trade-off between knowledge retention and unlearning effectiveness. Experimental results on various LLMs show that NSPU achieves superior unlearning performance, preserves model utility, and enhances user
privacy. Additionally, the proposed approach is at least 10x more computationally efficient than standard unlearning approaches. Our findings foster a new direction for
privacy-aware machine unlearning that balances data protection and model fidelity.
Read the full abstract ▾
All authors Dinesh Srivasthav P, Ashok Urlana, Rahul Mishra, Bala Mallikarjunarao Garlapati, Ponnurangam Kumaraguru
Computation and Language
All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection
Yuechen Jiang, Zhiwei Liu, Yupeng Cao et al. (+10 more)
We introduce RFC Bench, a
benchmark for evaluating large
language models on financial misinformation under realistic news. RFC Bench operates at the paragraph level and captures the contextual complexity of financial news where meaning emerges from dispersed cues. The
benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis using paired original perturbed inputs. Experiments reveal a consistent pattern: performance is substantially stronger when comparative context is available, while reference free settings expose significant weaknesses, including unstable predictions and elevated invalid outputs. These results indicate that current models struggle to maintain coherent belief states without external grounding. By highlighting this gap, RFC Bench provides a structured testbed for studying reference free
reasoning and advancing more reliable financial misinformation detection in real world settings.
Read the full abstract ▾
All authors Yuechen Jiang, Zhiwei Liu, Yupeng Cao, Yueru He, Ziyang Xu, Chen Xu, Zhiyang Deng, Prayag Tiwari, Xi Chen, Alejandro Lopez-Lira, Jimin Huang, Junichi Tsujii, Sophia Ananiadou
Computation and Language
FLEx: Language Modeling with Few-shot Language Explanations
Adar Avsian, Christopher Richardson, Anirudh Sundar et al. (+1 more)
Language models have become effective at a wide range of tasks, from math problem solving to open-domain question answering. However, they still make mistakes, and these mistakes are often repeated across related queries. Natural
language explanations can help correct these errors, but collecting them at scale may be infeasible, particularly in domains where expert annotators are required. To address this issue, we introduce FLEx ($\textbf{F}$ew-shot $\textbf{L}$anguage $\textbf{Ex}$planations), a method for improving model behavior using a small number of explanatory examples. FLEx selects representative model errors using
embedding-based clustering, verifies that the associated explanations correct those errors, and summarizes them into a
prompt prefix that is prepended at inference-time. This summary guides the model to avoid similar errors on new inputs, without modifying model weights. We evaluate FLEx on CounterBench, GSM8K, and ReasonIF. We find that FLEx consistently outperforms chain-of-thought (CoT) prompting across all three
datasets and reduces up to 83% of CoT's remaining errors.
Read the full abstract ▾
All authors Adar Avsian, Christopher Richardson, Anirudh Sundar, Larry Heck
Computation and Language
LLMberjack: Guided Trimming of Debate Trees for Multi-Party Conversation Creation
Leonardo Bottona, Nicolò Penzo, Bruno Lepri et al. (+2 more)
Online debates on platforms like Reddit or Kialo are structured as reply trees: each message can receive several replies, which in turn can each receive several replies, producing a branching structure rather than a linear conversation. Turning one path (or a mix of paths) through such a tree into a natural, turn-by-turn conversation among multiple speakers (a process we call linearization) requires deciding which messages to keep, in what order, and who each message is addressed to. To this aim, we present LLMberjack, a platform for creating multi-party conversations starting from existing debates, originally structured as reply trees. The system offers an interactive interface that visualizes discussion trees and enables users to construct coherent linearized dialogue sequences while preserving participant identity and discourse relations. It integrates optional large
language model (LLM) assistance to support automatic editing of the messages and speakers' descriptions. We provide preliminary evidence about the platform's utility by showing how tree visualization facilitates the creation of coherent, meaningful conversation threads and how LLM support enhances output quality while reducing human effort. The tool is open-source and designed to promote transparent and reproducible workflows to create multi-party conversations, addressing a lack of resources of this type.
Read the full abstract ▾
All authors Leonardo Bottona, Nicolò Penzo, Bruno Lepri, Marco Guerini, Sara Tonelli
Computation and Language
ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
Nikhil Anand, Shwetha Somasundaram, Anirudh Phukan et al. (+2 more)
Large
Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA
benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.
Read the full abstract ▾
All authors Nikhil Anand, Shwetha Somasundaram, Anirudh Phukan, Apoorv Saxena, Koyel Mukherjee
Computation and Language
InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training
Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang et al. (+4 more)
GUI
agents that interact with graphical interfaces on behalf of users represent a promising direction for practical AI assistants. However, training such
agents is hindered by the scarcity of suitable environments. We present InfiniteWeb, a system that automatically generates functional web environments at scale for GUI
agent training. While LLMs perform well on generating a single webpage, building a realistic and functional website with many interconnected pages faces challenges. We address these challenges through unified specification, task-centric test-driven development, and a combination of website seed with reference design image to ensure diversity. Our system also generates verifiable task evaluators enabling dense reward signals for reinforcement learning. Experiments show that InfiniteWeb surpasses commercial coding
agents at realistic website construction, and GUI
agents trained on our generated environments achieve significant performance improvements on OSWorld and Online-Mind2Web, demonstrating the effectiveness of proposed system.
Read the full abstract ▾
All authors Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang, Zongyu Guo, Jiahao Li, Bin Li, Yan Lu
Computation and Language
Layer-wise Positional Bias in Short-Context Language Modeling
Maryam Rahimi, Mahdi Nouri and Yadollah Yaghoobzadeh
Transformer language models systematically prefer
tokens at specific input positions regardless of semantic relevance—a phenomenon known as positional bias. Prior work characterizes this bias in model behavior through performance drops in long-context tasks or in model architecture through
attention-based analyses. However, it remains unmeasured how input positions actually drive predictions layer by layer. We introduce a layer conductance framework within a sliding-window design, applied to short-context next-word prediction to isolate model-internal behavior from task and context-window pressure. The resulting layer-wise positional importance profiles are stable across diverse texts and lexical scrambling, confirming they reflect model-internal structure. Characterizing how these profiles evolve across depth, we find recency bias increases monotonically while primacy bias is subtle and diminishes. We also find that this positional bias is not uniform across word types: function words exhibit higher recency bias while content words show higher primacy bias.
Read the full abstract ▾
Computation and Language
SearchAttack: Red-Teaming LLMs against Knowledge-to-Action Threats under Online Web Search
Yu Yan, Sheng Sun, Mingfeng Li et al. (+6 more)
Recently, people have suffered from LLM
hallucination and have become increasingly aware of the reliability gap of LLMs in open and knowledge-intensive tasks. As a result, they have increasingly turned to search-augmented LLMs to mitigate this issue. However, LLM-driven search also becomes an attractive target for misuse. Once the returned content directly contains targeted, ready-to-use harmful instructions or takeaways for users, it becomes difficult to withdraw or undo such exposure. To investigate LLMs' unsafe search behavior issues, we first propose \textbf{SearchAttack} for red-teaming, which (1) rephrases harmful semantics via dense and benign knowledge to evade direct in-context decoding, thus eliciting unsafe information
retrieval, (2) stress-tests LLMs' reward-chasing bias by steering them to synthesize unsafe retrieved content. We also curate an emergent, domain-specific illicit activity
benchmark for search-based threat assessment, and introduce a fact-checking framework to ground and quantify harm in both offline and online attack settings. Extensive experiments are conducted to red-team the search-augmented LLMs for responsible vulnerability assessment. Empirically, SearchAttack demonstrates strong effectiveness in attacking these systems. We also find that LLMs without web search can still be steered into harmful content output due to their information-seeking stereotypical behaviors.
Read the full abstract ▾
All authors Yu Yan, Sheng Sun, Mingfeng Li, Zheming Yang, Chiwei Zhu, Fei Ma, Benfeng Xu, Min Liu, Qi Li
Computation and Language
KDCM: Reducing Hallucination in LLMs through Explicit Reasoning Structures
Jinbo Hao, Kai Yang, Qingzhen Su et al. (+2 more)
To mitigate
hallucinations in large
language models (LLMs), we propose a framework that focuses on errors induced by
prompts. Our method extends a chain-style knowledge
distillation approach by incorporating a programmable module that guides knowledge graph exploration. This module is embedded as executable code within the
reasoning prompt, allowing the model to leverage external structured knowledge during inference. Based on this design, we develop an enhanced
distillation-based
reasoning framework that explicitly regulates intermediate
reasoning steps, resulting in more reliable predictions. We evaluate the proposed approach on multiple public
benchmarks using GPT-4 and LLaMA-3.3. Experimental results show that code-guided
reasoning significantly improves contextual modeling and reduces
prompt-induced
hallucinations. Specifically, HIT@1, HIT@3, and HIT@5 increase by 15.64%, 13.38%, and 13.28%, respectively, with scores exceeding 95% across several evaluation settings. These findings indicate that the proposed method effectively constrains erroneous
reasoning while improving both accuracy and interpretability.
Read the full abstract ▾
All authors Jinbo Hao, Kai Yang, Qingzhen Su, Yifan Li, Chao Jiang
Computation and Language · Cross-listed from cs.CV
Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts
Zhihao Zhu, Jiafeng Liang, Shixin Jiang et al. (+5 more)
Large
Multimodal Models (LMMs) have demonstrated impressive capabilities in
video reasoning via Chain-of-Thought (CoT). However, the robustness of their
reasoning chains remains questionable. In this paper, we identify a critical failure mode termed textual inertia, where once a textual
hallucination occurs in the thinking process, models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence. To systematically investigate this, we propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the
reasoning chains of diverse LMMs spanning both native
reasoning architectures and
prompt-driven paradigms to evaluate their self-reflection capabilities. The results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation. To mitigate this, we introduce Active Visual-Context Refinement, a training-free inference paradigm which orchestrates an active visual re-grounding mechanism to enforce fine-grained verification coupled with an adaptive context refinement strategy to summarize and denoise the
reasoning history. Experiments demonstrate that our approach significantly stifles
hallucination propagation and enhances
reasoning robustness.
Read the full abstract ▾
All authors Zhihao Zhu, Jiafeng Liang, Shixin Jiang, Jinlan Fu, Ming Liu, Guanglu Sun, See-Kiong Ng, Bing Qin
Computation and Language
Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion
Yuanfeng Xu, Yuhao Chen, Liang Lin et al. (+1 more)
The bifurcation of generative modeling into autoregressive approaches for discrete data (text) and
diffusion approaches for continuous data (images) hinders the development of truly unified
multimodal systems. While Masked
Language Models (MLMs) offer efficient bidirectional context, they traditionally lack the generative fidelity of autoregressive models and the semantic continuity of
diffusion models. Furthermore, extending masked generation to
multimodal settings introduces severe
alignment challenges and training instability. In this work, we propose CoM-DAD (Coupled
Manifold Discrete Absorbing
Diffusion), a novel probabilistic framework that reformulates
multimodal generation as a hierarchical dual-process. CoM-DAD decouples high-level semantic planning from low-level
token synthesis. First, we model the semantic
manifold via a continuous latent
diffusion process; second, we treat
token generation as a discrete absorbing
diffusion process, regulated by a Variable-Rate Noise Schedule, conditioned on these evolving semantic priors. Crucially, we introduce a
Stochastic Mixed-Modal Transport strategy that aligns disparate modalities without requiring heavy contrastive dual-encoders. Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
Read the full abstract ▾
All authors Yuanfeng Xu, Yuhao Chen, Liang Lin, Guangrun Wang