Computation and Language
Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios
Autumn Toney-Wails and Ryan Wails
Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large
language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from
token logits, with derived probability and
entropy values offering insight into performance on the
prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of
token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between
token certainty and
alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten
prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the
prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2)
alignment between
token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all
prompt scenarios, their
token-level probability and
entropy values consistently diverge from the corresponding theoretical distributions.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
Eric Bigelow, Daniel Wurgaft, YingQiao Wang et al. (+4 more)
Large
language models (LLMs) can be controlled at inference time through
prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a
Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts: steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form
Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena - e.g., sigmoidal learning curves as in-context evidence accumulates - while predicting novel ones - e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of
prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.
Read the full abstract ▾
All authors Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, Ekdeep Singh Lubana
Computation and Language
SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
Jameson Sandler, Jacob K. Christopher, Thomas Hartvigsen et al. (+1 more)
Speculative decoding has become the standard approach for accelerating Large
Language Model (LLM) inference. It exploits a lossless draft-then-verify procedure to circumvent the latency of autoregressive decoding, achieving impressive speed-ups. Yet, current speculative decoding approaches remain limited by two fundamental bottlenecks: (1) the autoregressive dependency during drafting which limits parallelism, and (2) frequent rejections of draft
tokens caused by misalignment between the draft and verify models. This paper proposes SpecDiff-2, a novel framework to jointly address these two bottlenecks. It leverages discrete
diffusion as a non-autoregressive drafter to address bottleneck (1) and develops novel techniques to calibrate discrete
diffusion drafters with autoregressive verifiers, addressing bottleneck (2). Experimental results across a comprehensive
benchmark suite show that SpecDiff-2 achieves a new state-of-the-art across
reasoning, coding, and mathematical
benchmarks, improving
tokens-per-second by up to an average of +55% over previous baselines and obtaining up to 5.5x average speed-up over standard decoding, without any loss of accuracy.
Read the full abstract ▾
All authors Jameson Sandler, Jacob K. Christopher, Thomas Hartvigsen, Ferdinando Fioretto
Computation and Language
OpenSIR: Open-Ended Self-Improving Reasoner
Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis et al. (+3 more)
Recent advances in large
language model (LLM)
reasoning through reinforcement learning rely on annotated
datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math
benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on
reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general
reasoning, improving by at least +4.4 points on
reasoning models.
Read the full abstract ▾
All authors Wai-Chung Kwan, Joshua Ong Jun Leang, Pavlos Vougiouklis, Jeff Z. Pan, Marco Valentino, Pasquale Minervini
Computation and Language · Cross-listed from cs.IR
Structurally Refined Graph Transformer for Multimodal Recommendation
Ke Shi, Yan Zhang, Miao Zhang et al. (+5 more)
Multimodal recommendation systems utilize various types of information, including images and text, to enhance the effectiveness of recommendations. The key challenge is predicting user purchasing behavior from the available data. Current recommendation models prioritize extracting
multimodal information while neglecting the distinction between redundant and valuable data. They also rely heavily on a single semantic framework (e.g., local or global semantics), resulting in an incomplete or biased representation of user preferences, particularly those less expressed in prior interactions. Furthermore, these approaches fail to capture the complex interactions between users and items, limiting the model's ability to meet diverse users. To address these challenges, we present SRGFormer, a structurally optimized
multimodal recommendation model. By modifying the
transformer for better integration into our model, we capture the overall behavior patterns of users. Then, we enhance structural information by
embedding multimodal information into a hypergraph structure to aid in learning the local structures between users and items. Meanwhile, applying self-supervised tasks to user-item collaborative signals enhances the integration of
multimodal information, thereby revealing the representational features inherent to the data's modality. Extensive experiments on three public
datasets reveal that SRGFormer surpasses previous
benchmark models, achieving an average performance improvement of 4.47 percent on the Sports
dataset. The code is publicly available online.
Read the full abstract ▾
All authors Ke Shi, Yan Zhang, Miao Zhang, Lifan Chen, Jiali Yi, Kui Xiao, Xiaoju Hou, Zhifei Li
Computation and Language
FlashEVA: Accelerating LLM inference via Efficient Attention
Juan Gabriel Kostelec and Qinghai Guo
Transformer models have revolutionized natural
language processing, achieving state-of-the-art performance and demonstrating remarkable scalability. However, their
memory demands, particularly due to maintaining full context in
memory, pose significant challenges for inference. In this paper, we present FlashEVA, an efficient implementation of EVA (Efficient
Attention via Control Variates), and demonstrate how to finetune
transformers to adapt to FlashEVA
attention. Our method enables fine-tuning of
Transformer models with as few as 1.5B
tokens while preserving effectiveness across various downstream tasks. Notably, FlashEVA achieves up to 6.7x higher throughput and 5x lower peak GPU
memory usage during inference compared to standard
Transformer implementations. Despite these improvements, we observe limitations in
retrieval-focused tasks. Our implementation offers control over the trade-off between throughput and accuracy through adjustable hyperparameters, providing flexibility for diverse use cases. This work represents a significant step towards more efficient and adaptable
Transformer-based models for inference.
Read the full abstract ▾
Computation and Language
Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack
Peng Ding, Jun Kuang, Wen Sun et al. (+5 more)
Large
language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities. Investigating these weaknesses is crucial for robust safety mechanisms. Existing attacks primarily distract LLMs by introducing additional context or
adversarial tokens, leaving the core harmful intent unchanged. In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks. More specifically, we establish a taxonomy of intent transformations and leverage them to generate attacks that may be misperceived by LLMs as benign requests for information. Unlike prior methods relying on complex
tokens or lengthy context, our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless
prompts. Extensive experiments on both open-source and commercial LLMs show that ISA achieves over 70% improvement in attack success rate compared to direct harmful
prompts. More critically, fine-tuning models on only benign data reformulated with ISA templates elevates success rates to nearly 100%. For defense, we evaluate existing methods and demonstrate their inadequacy against ISA, while exploring both training-free and training-based mitigation strategies. Our findings reveal fundamental challenges in intent inference for LLMs safety and underscore the need for more effective defenses. Our code and
datasets are available at https://github.com/NJUNLP/ISA.
Read the full abstract ▾
All authors Peng Ding, Jun Kuang, Wen Sun, Zongyu Wang, Xuezhi Cao, Xunliang Cai, Jiajun Chen, Shujian Huang
Computation and Language
Multi-refined Feature Enhanced Sentiment Analysis Using Contextual Instruction
Peter Atandoh, Jie Zou, Weikang Guo et al. (+2 more)
Sentiment analysis using deep learning and pre-trained
language models (PLMs) has gained significant traction due to their ability to capture rich contextual representations. However, existing approaches often underperform in scenarios involving nuanced emotional cues, domain shifts, and imbalanced sentiment distributions. We argue that these limitations stem from inadequate semantic grounding, poor generalization to diverse linguistic patterns, and biases toward dominant sentiment classes. To overcome these challenges, we propose CISEA-MRFE, a novel PLM-based framework integrating Contextual Instruction (CI), Semantic Enhancement Augmentation (SEA), and Multi-Refined Feature Extraction (MRFE). CI injects domain-aware directives to guide sentiment disambiguation; SEA improves robustness through sentiment-consistent paraphrastic augmentation; and MRFE combines a Scale-Adaptive Depthwise Encoder (SADE) for multi-scale feature specialization with an
Emotion Evaluator Context Encoder (EECE) for affect-aware sequence modeling. Experimental results on four
benchmark datasets demonstrate that CISEA-MRFE consistently outperforms strong baselines, achieving relative improvements in accuracy of up to 4.6% on IMDb, 6.5% on Yelp, 30.3% on Twitter, and 4.1% on Amazon. These results validate the effectiveness and generalization ability of our approach for sentiment classification across varied domains.
Read the full abstract ▾
All authors Peter Atandoh, Jie Zou, Weikang Guo, Jiwei Wei, Zheng Wang
Computation and Language
Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly
Wenya Xie, Shaochen Zhong, Hoang Anh Duy Le et al. (+3 more)
Large
Reasoning Models (LRMs) are often bottlenecked by the high cost of output
tokens. We show that a significant portion of these
tokens are useless self-repetitions - what we call "word salad" - that exhaust the decoding budget without adding value. Interestingly, we observe that LRMs are self-aware when trapped in these loops: the hidden states of <\n\n>
tokens trailing each
reasoning chunk exhibit patterns that allow us to detect word salad behavior on-the-fly via a single-layer linear classifier. Once detected, a simple chop appended by a straightforward regeneration
prompt yields substantial length savings with minimal quality loss. Our work offers WordSaladChopper (WSC) - a lightweight, turnkey component for LRM that is minimally invasive to its
reasoning trajectory by only removing semantically redundant
tokens. Given its low overhead, strong savings, and the lack of semantic value of word salad
tokens, we believe it is not too far-fetched to argue that WSC - or a similar component - is a must-have for all LRM applications with user experience in mind. Our code is publicly available at https://github.com/wenyaxie023/WordSaladChopper.
Read the full abstract ▾
All authors Wenya Xie, Shaochen Zhong, Hoang Anh Duy Le, Zhaozhuo Xu, Jianwen Xie, Zirui Liu
Computation and Language · Cross-listed from cs.LG
Bao Nguyen, Hieu Trung Nguyen, Ruifeng She et al. (+2 more)
Selecting an appropriate
reasoning method for a given query remains a key challenge in
language model generation. Existing approaches typically generate multiple candidate responses and use an aggregation strategy to select the output answer, often assuming that more candidate answers yield higher accuracy. We revisit this assumption through a rigorous theoretical analysis, deriving accuracy bounds for standard aggregation methods under fixed generation distributions and candidate sizes. Building on these insights, we introduce EPIC, an Ensemble Planning with Contrastive learning framework to learn a shared representation space that captures both model
reasoning abilities and query-method compatibility. EPIC incorporates our probability bounds as a regularizer in a utility-driven optimization that balances accuracy and computational cost. Experiments on diverse mathematical
reasoning tasks show that EPIC consistently selects optimal
reasoning methods, improving accuracy while reducing computational overhead. Our code can be found at https://github.com/nguyenngocbaocmt02/EPIC.
Read the full abstract ▾
All authors Bao Nguyen, Hieu Trung Nguyen, Ruifeng She, Xiaojin Fu, Viet Anh Nguyen
Computation and Language
Exploring and Mitigating Gender Bias in Encoder-Based Transformer Models
Ariyan Hossain, Khondokar Mohammad Ahanaf Hannan, Rakinul Haque et al. (+4 more)
Gender bias in
language models has gained increasing
attention in the field of natural
language processing. Encoder-based
transformer models, which have achieved state-of-the-art performance in various
language tasks, have been shown to exhibit strong gender biases inherited from their training data. This paper investigates gender bias in contextualized word
embeddings, a crucial component of
transformer-based models. We focus on prominent architectures such as BERT, ALBERT, RoBERTa, and DistilBERT to examine their vulnerability to gender bias. To quantify the degree of bias, we introduce a novel metric, MALoR, which assesses bias based on model probabilities for filling masked
tokens. We further propose a mitigation approach involving continued pre-training on a gender-balanced
dataset generated via Counterfactual Data Augmentation. Our experiments reveal significant reductions in gender bias scores across different pronoun pairs. For instance, in BERT-base, bias scores for "he-she" dropped from 1.27 to 0.08, and "his-her" from 2.51 to 0.36 following our mitigation approach. We also observed similar improvements across other models, with "male-female" bias decreasing from 1.82 to 0.10 in BERT-large. Our approach effectively reduces gender bias without compromising model performance on downstream tasks.
Read the full abstract ▾
All authors Ariyan Hossain, Khondokar Mohammad Ahanaf Hannan, Rakinul Haque, Nowreen Tarannum Rafa, Humayra Musarrat, Shoaib Ahmed Dipu, Farig Yousuf Sadeque
Computation and Language
Fine-Tuning DialoGPT on Common Diseases in Rural Nepal for Medical Conversations
Birat Poudel, Satyam Ghimire and Er. Prakash Chandra Prasad
Conversational
agents are increasingly being explored to support healthcare delivery, particularly in resource-constrained settings such as rural Nepal. Large-scale conversational models typically rely on internet connectivity and cloud infrastructure, which may not be accessible in rural areas. In this study, we fine-tuned DialoGPT, a lightweight generative dialogue model that can operate offline, on a synthetically constructed
dataset of doctor-patient interactions covering ten common diseases prevalent in rural Nepal, including common cold, seasonal fever, diarrhea, typhoid fever, gastritis, food poisoning, malaria, dengue fever, tuberculosis, and pneumonia. Despite being trained on a limited, domain-specific
dataset, the fine-tuned model produced coherent, contextually relevant, and medically appropriate responses, demonstrating an understanding of symptoms, disease context, and empathetic communication. These results highlight the adaptability of compact, offline-capable dialogue models and the effectiveness of targeted
datasets for domain adaptation in low-resource healthcare environments, offering promising directions for future rural medical conversational AI.
Read the full abstract ▾
Computation and Language
Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge
Qi Luo, Xiaonan Li, Junqi Dai et al. (+2 more)
Retrieval-Augmented Generation has shown remarkable results to address Large
Language Models'
hallucinations, which usually uses a large external corpus to supplement knowledge to LLMs. However, with the development of LLMs, the internal knowledge of LLMs has expanded significantly, thus causing significant knowledge redundancy between the external corpus and LLMs. On the one hand, the indexing cost of dense
retrieval is highly related to the corpus size and thus significant redundant knowledge intensifies the dense
retrieval's workload. On the other hand, the redundant knowledge in the external corpus is not helpful to LLMs and our exploratory analysis shows that it instead hurts the RAG performance on those questions which the LLM can answer by itself. To address these issues, we propose Zero-RAG to tackle these challenges. Specifically, we first propose the Mastery-Score metric to identify redundant knowledge in the RAG corpus to prune it. After
pruning, answers to "mastered" questions rely primarily on internal knowledge of the LLM. To better harness the internal capacity, we propose Query Router and Noise-Tolerant Tuning to avoid the irrelevant documents' distraction and thus further improve the LLM's utilization of internal knowledge with pruned corpus. Experimental results show that Zero-RAG prunes the Wikipedia corpus by 30% and accelerates the
retrieval stage by 22%, without compromising RAG's performance.
Read the full abstract ▾
All authors Qi Luo, Xiaonan Li, Junqi Dai, Shuang Cheng, Xipeng Qiu
Computation and Language
ToM: Leveraging Tree-oriented MapReduce for Long-Context Reasoning in Large Language Models
Jiani Guo, Zuchao Li, Jie Wu et al. (+5 more)
Large
Language Models (LLMs), constrained by limited context windows, often face significant performance degradation when
reasoning over long contexts. To address this,
Retrieval-Augmented Generation (RAG) retrieves and reasons over chunks but frequently sacrifices logical coherence due to its reliance on similarity-based rankings. Similarly, divide-and-conquer frameworks (DCF) split documents into small chunks for independent
reasoning and aggregation. While effective for local
reasoning, DCF struggles to capture long-range dependencies and risks inducing conflicts by processing chunks in isolation. To overcome these limitations, we propose ToM, a novel Tree-oriented MapReduce framework for long-context
reasoning. ToM leverages the inherent hierarchical structure of long documents (e.g., main headings and subheadings) by constructing a DocTree through hierarchical semantic parsing and performing bottom-up aggregation. Using a Tree MapReduce approach, ToM enables recursive
reasoning: in the Map step, rationales are generated at child nodes; in the Reduce step, these rationales are aggregated across sibling nodes to resolve conflicts or reach consensus at parent nodes. Experimental results on 70B+ LLMs show that ToM significantly outperforms existing divide-and-conquer frameworks and
retrieval-augmented generation methods, achieving better logical coherence and long-context
reasoning. Our code is available at https://github.com/gjn12-31/ToM .
Read the full abstract ▾
All authors Jiani Guo, Zuchao Li, Jie Wu, Qianren Wang, Yun Li, Lefei Zhang, Hai Zhao, Yujiu Yang
Computation and Language · Cross-listed from cs.PL
ReMind: Understanding Deductive Code Reasoning in LLMs
Jun Gao, Yun Peng and Xiaoxue Ren
Large
Language Models (LLMs) have achieved remarkable progress in code-related tasks. Despite their advancement, empirical evidence reveals that they still struggle with deductive code
reasoning, the ability to reason about the program execution process. While prior studies have recognized this limitation, the underlying causes remain largely underexplored. In this paper, we begin by presenting a comprehensive empirical study that reveals three key challenges undermining deductive code
reasoning: (1) an intrinsic gap between generation and
reasoning abilities, (2) a consistent bias towards code sources, and (3) weak zero-shot generalization on complex
benchmarks. In light of these challenges, we propose ReMind, a multi-
agent framework composed of Mutator, Executor, and Inspector. The Mutator generates code variants to mitigate bias towards code sources, the Executor traces variable states step-by-step to expose inconsistency, and the Inspector identifies problematic
reasoning steps and provides control-flow refinement to bridge the intrinsic
reasoning gap. Through their coordinated collaboration, ReMind systematically identifies and refines
reasoning flaws, achieving outstanding performance and enabling robust zero-shot generalization. Extensive experiments on two
benchmarks with five LLMs demonstrate the superior advantages of ReMind compared to baseline approaches in deductive code
reasoning.
Read the full abstract ▾
Computation and Language
With Privacy, Size Matters: On the Importance of Dataset Size in Differentially Private Text Rewriting
Stephen Meisenbacher and Florian Matthes
Recent work in Differential
Privacy with Natural
Language Processing (DP NLP) has proposed numerous promising techniques in the form of text rewriting mechanisms. In the evaluation of these mechanisms, an often-ignored aspect is that of
dataset size, or rather, the effect of
dataset size on a mechanism's efficacy for utility and
privacy preservation. In this work, we are the first to introduce this factor in the evaluation of DP text privatization, where we design utility and
privacy tests on large-scale
datasets with dynamic split sizes. We run these tests on
datasets of varying size with up to one million texts, and we focus on quantifying the effect of increasing
dataset size on the
privacy-utility trade-off. Our findings reveal that
dataset size plays an integral part in evaluating DP text rewriting mechanisms; additionally, these findings call for more rigorous evaluation procedures in DP NLP, as well as shed light on the future of DP NLP in practice and at scale.
Read the full abstract ▾
Computation and Language
Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus
Pooja Singh, Shashwat Bhardwaj, Vaibhav Sharma et al. (+1 more)
The linguistic diversity of India poses significant machine translation challenges, especially for underrepresented tribal
languages like Bhili, which lack high-quality linguistic resources. This paper addresses the gap by introducing Bhili-Hindi-English Parallel Corpus (BHEPC), the first and largest parallel corpus worldwide comprising 110,000 meticulously curated sentences across Bhili, Hindi, and English. The corpus was created with the assistance of expert human translators. BHEPC spans critical domains such as education, administration, and news, establishing a valuable
benchmark for research in low resource machine translation. To establish a comprehensive Bhili Machine Translation
benchmark, we evaluated a wide range of proprietary and open-source
Multilingual Large
Language Models (MLLMs) on bidirectional translation tasks between English/Hindi and Bhili. Comprehensive evaluation demonstrates that the fine-tuned NLLB-200 distilled 600M variant model outperforms others, highlighting the potential of
multilingual models in low resource scenarios. Furthermore, we investigated the generative translation capabilities of
multilingual LLMs on BHEPC using in-context learning, assessing performance under cross-domain generalization and quantifying distributional divergence. This work bridges a critical resource gap and promotes inclusive natural
language processing technologies for low-resource and marginalized
languages globally.
Read the full abstract ▾
All authors Pooja Singh, Shashwat Bhardwaj, Vaibhav Sharma, Sandeep Kumar
Computation and Language
Remembering Unequally: Global and Disciplinary Bias in LLM Reconstruction of Scholarly Coauthor Lists
Ghazal Kalhor and Afra Mashhadi
Ongoing breakthroughs in large
language models (LLMs) are reshaping scholarly search and discovery interfaces. While these systems offer new possibilities for navigating scientific knowledge, they also raise concerns about
fairness and representational bias rooted in the models' memorized training data. As LLMs are increasingly used to answer queries about researchers and research communities, their ability to accurately reconstruct scholarly coauthor lists becomes an important but underexamined issue. In this study, we investigate how memorization in LLMs affects the reconstruction of coauthor lists and whether this process reflects existing inequalities across academic disciplines and world regions. We evaluate three prominent models, DeepSeek R1, Llama 4 Scout, and Mixtral 8x7B, by comparing their generated coauthor lists against bibliographic reference data. Our analysis reveals a systematic advantage for highly cited researchers, indicating that LLM memorization disproportionately favors already visible scholars. However, this pattern is not uniform: certain disciplines, such as Clinical Medicine, and some regions, including parts of Africa, exhibit more balanced reconstruction outcomes. These findings highlight both the risks and limitations of relying on LLM-generated relational knowledge in scholarly discovery contexts and emphasize the need for careful
auditing of memorization-driven biases in LLM-based systems.
Read the full abstract ▾
Computation and Language
G2: Guided Generation for Enhanced Output Diversity in LLMs
Zhiwen Ruan, Yixia Li, Yefeng Liu et al. (+5 more)
Large
Language Models (LLMs) have demonstrated exceptional performance across diverse natural
language processing tasks. However, these models exhibit a critical limitation in output diversity, often generating highly similar content across multiple attempts. This limitation significantly affects tasks requiring diverse outputs, from creative writing to
reasoning. Existing solutions, like temperature scaling, enhance diversity by modifying probability distributions but compromise output quality. We propose Guide-to-Generation (G2), a training-free plug-and-play method that enhances output diversity while preserving generation quality. G2 employs a base generator alongside dual Guides, which guide the generation process through decoding-based interventions to encourage more diverse outputs conditioned on the original query. Comprehensive experiments demonstrate that G2 effectively improves output diversity while maintaining an optimal balance between diversity and quality.
Read the full abstract ▾
All authors Zhiwen Ruan, Yixia Li, Yefeng Liu, Yun Chen, Weihua Luo, Peng Li, Yang Liu, Guanhua Chen
Computation and Language
MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts
Naoto Iwase, Hiroki Okuyama and Junichiro Iwasawa
Large
language models (LLMs) show promise in medical applications, but their ability to detect and correct errors in clinical texts remains under-evaluated, particularly beyond English. We introduce MedRECT, a bilingual
benchmark for Japanese and English that formulates medical error handling as three subtasks: error detection, error sentence extraction, and error correction. MedRECT-ja contains 663 samples derived from the Japanese Medical Licensing Examinations, while the separately sourced MedRECT-en contains 458 samples curated from MEDEC. We evaluate 11 LLMs across 17 configurations that cover proprietary and open-weight models, medical-domain specialization, and multiple
reasoning settings. Qwen3-32B scores higher in its thinking mode than in its non-thinking mode on error detection F1 and sentence extraction accuracy in both subsets, with sentence extraction accuracy higher by 24.5 percentage points on MedRECT-ja and 10.3 on MedRECT-en. Several leading general-purpose
reasoning models outperform all three evaluated medical-domain models on these two subtasks. Most models have lower point estimates on the Japanese subset, although absolute scores are not directly comparable because the subsets differ in source material and error distributions. LoRA fine-tuning yields higher sentence extraction accuracy and higher point estimates on all three reference-based correction similarity metrics in both
languages. MedRECT provides an open, reusable evaluation resource for studying medical error correction and
reasoning across Japanese and English. Our
dataset and code are available at https://github.com/pfnet-research/medrect.
Read the full abstract ▾
Computation and Language
PADBen: A Comprehensive Benchmark for Evaluating AI Text Detectors Against Paraphrase Attacks
Yiwei Zha, Rui Min and Shanu Sushmita
While AI-generated text (AIGT) detectors achieve over 90% accuracy on direct LLM outputs, they fail catastrophically against iteratively-paraphrased content. We investigate why iteratively-paraphrased text – itself AI-generated – evades detection systems designed for AIGT identification. Through intrinsic mechanism analysis, we reveal that iterative paraphrasing creates an intermediate laundering region characterized by semantic displacement with preserved generation patterns, which brings up two attack categories: paraphrasing human-authored text (authorship obfuscation) and paraphrasing LLM-generated text (plagiarism evasion). To address these vulnerabilities, we introduce PADBen, the first
benchmark systematically evaluating detector robustness against both paraphrase attack scenarios. PADBen comprises a five-type text taxonomy capturing the full trajectory from original content to deeply laundered text, and five progressive detection tasks across sentence-pair and single-sentence challenges. We evaluate 11 state-of-the-art detectors, revealing critical asymmetry: detectors successfully identify the plagiarism evasion problem but fail for the case of authorship obfuscation. Our findings demonstrate that current detection approaches cannot effectively handle the intermediate laundering region, necessitating fundamental advances in detection architectures beyond existing semantic and stylistic discrimination methods. For detailed code implementation, please see https://github.com/JonathanZha47/PadBen-Paraphrase-Attack-
Benchmark.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Jiahao Wang, Songkai Xue, Jinghui Li et al. (+1 more)
Ensuring that Large
Language Models (LLMs) align with the diverse and evolving human values across different regions and cultures remains a critical challenge in AI ethics. Current
alignment approaches often yield superficial conformity rather than genuine ethical understanding, failing to address the complex, context-dependent nature of human values. In this paper, we propose a novel ethical
reasoning paradigm for LLMs inspired by well-established ethical decision-making models, aiming at enhancing diverse human value
alignment through deliberative ethical
reasoning. Our framework consists of a structured five-step process, including contextual fact gathering, hierarchical social norm identification, option generation, multiple-lens ethical impact analysis, and reflection. This theory-grounded approach guides LLMs through an interpretable
reasoning process that enhances their ability to understand regional specificities and perform nuanced ethical analysis, which can be implemented with either
prompt engineering or supervised fine-tuning methods. We perform evaluations on the SafeWorld
benchmark that specially designed for regional value
alignment. Experimental results demonstrate our framework significantly improves LLM
alignment with diverse human values compared to baseline methods, enabling more accurate social norm identification and more culturally appropriate
reasoning. Our work provides a concrete pathway toward developing LLMs that align more effectively with the multifaceted values of global societies through interdisciplinary research.
Read the full abstract ▾
All authors Jiahao Wang, Songkai Xue, Jinghui Li, Xiaozhen Wang
Computation and Language
Reasoning Trajectories for Socratic Debugging of Student Code: From Misconceptions to Contradictions and Updated Beliefs
Erfan Al-Hossami and Razvan Bunescu
In Socratic debugging, instructors guide students towards identifying and fixing a bug on their own, instead of providing the bug fix directly. Most novice programmer bugs are caused by programming misconceptions, namely false beliefs about a programming concept. In this context, Socratic debugging can be formulated as a guided
Reasoning Trajectory (RT) leading to a statement about the program behavior that contradicts the bug-causing misconception. Upon reaching this contradiction, the ensuing cognitive dissonance is expected to lead the student to identify the false belief on their own, followed by an enduring belief update. In this paper, we introduce the task of
reasoning trajectory generation, together with a
dataset of debugging problems annotated with RTs that are manually created or LLM-generated. We then describe LLM-based solutions for generating RTs and Socratic conversations that are anchored on them. A large-scale LLM-as-judge evaluation shows that large
language and
reasoning models can generate up to 91% correct
reasoning trajectories and 98.7% valid conversation turns.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reject Only Critical Tokens: Pivot-Aware Speculative Decoding
Amir Ziashahabi, Yavuz Faruk Bakman, Duygu Nur Yaldiz et al. (+3 more)
Speculative Decoding (SD) ensures that the output matches the target model's distribution exactly. However, we argue that this distribution matching requirement is too stringent and results in unnecessarily low acceptance rates, limiting potential speedups. Instead, we advocate a reformulation of the decoding objective: the proposed decoding strategy should match the expected utility, i.e., the task-specific performance, of the target model. This perspective also aligns better with real-world use cases of LLMs, where utility (e.g., code correctness, factual accuracy) is often more important than sampling distribution. Based on this reformulation, we propose a novel decoding strategy: Pivot-Aware Speculative Decoding, which rejects only those
tokens that would lead to a utility drop in the final output. We refer to these critical
tokens as pivot
tokens. We propose a method for labeling
tokens as pivotal or non-pivotal and train a lightweight classifier to detect them. This method can be viewed as a relaxed version of standard SD, which offers much higher acceptance while preserving utility. We evaluate our method across various
datasets, demonstrating that we can achieve up to $2.5\times$ speedup with comparable utility. Source code is available at https://github.com/amir-zsh/PAD.
Read the full abstract ▾
All authors Amir Ziashahabi, Yavuz Faruk Bakman, Duygu Nur Yaldiz, Mostafa El-Khamy, Sai Praneeth Karimireddy, Salman Avestimehr
Computation and Language
LingGym: How Far Are LLMs from Thinking Like Field Linguists?
Changbing Yang, Franklin Ma, Freda Shi et al. (+1 more)
This paper introduces LingGym, a new
benchmark that evaluates LLMs' capacity for meta-linguistic
reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource
languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in
reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource
language documentation.
Read the full abstract ▾
All authors Changbing Yang, Franklin Ma, Freda Shi, Jian Zhu
Computation and Language
Reversal Invariance in Autoregressive Language Models
Mihir Sahasrabudhe
We formalize a structural property of the causal (autoregressive)
language modeling (CLM) objective: reversal invariance. Formally, the next-
token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This
symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human
language and
reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural
language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of
language while retaining standard
language modeling capacity.
Read the full abstract ▾