Computation and Language · Cross-listed from cs.CV
NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding
Mohammad H. Abbasi, Favour Nerrise, Shaurnav Ghosh et al. (+12 more)
We present NeuroQA, a large-scale
benchmark for visual question answering in 3D
brain magnetic resonance imaging (MRI), with 56,953 QA pairs from 12,977 subjects across 12
datasets. It spans ages 5-104 and five clinical domains: Alzheimer's, Parkinson's, tumors, white matter disease, and neurodevelopment. Unlike prior medical Visual Question Answering (VQA) efforts that operate on 2D slices or rely on narrow diagnostic labels, NeuroQA pairs every item with a full 3D volume. It evaluates 11 clinically grounded
reasoning skills across Yes/No, multiple-choice, and open-ended formats. Of the 203 templates, 131 are image-grounded (answerable from a 3-plane viewer) and 72 are image-informed (ground truth from quantitative volumetry or clinical instruments). To remove text-only shortcuts, we apply answer-distribution refinement, reducing closed-format text-only accuracy from $>$80% to 44.6%; image necessity is assessed separately through an image-grounding protocol released with the
benchmark. A 38-rule deterministic pipeline and two rounds of expert review verify every QA pair against FreeSurfer measurements, metadata, or radiology report fields, with zero same-subject contradictions across templates. We conduct a clinician evaluation in which two clinicians independently assess 100 frozen test items on a three-plane viewer. On closed-format (Yes/No + multiple-choice) test-public items, the best zero-shot
vision-
language model and a supervised 3D CNN baseline reach 47.5% and 43.7% accuracy respectively, both below the 49.4% text-only majority-template floor. NeuroQA adopts a two-tier release with public QA pairs for open-access
datasets and reproducible generation scripts for
datasets restricted by data use agreements (DUAs), plus subject-level splits, a held-out private test set, and an online leaderboard.
Read the full abstract ▾
All authors Mohammad H. Abbasi, Favour Nerrise, Shaurnav Ghosh, Ridvan Yesiloglu, Yuncong Mao, Bailey Trang, Mohammad Asadi, Merryn Daniel, Gustavo Chau Loo Kung, Ken Chang, Pavan Pinkesh Shah, Adam Turnbull, Kyan Younes, Seena Dehkharghani, Ehsan Adeli
Computation and Language · Cross-listed from cs.LG
Reinforcing Human Behavior Simulation via Verbal Feedback
Weiwei Sun, Xuhui Zhou, Jiarui Liu et al. (+13 more)
Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying "that was rude" or a friend explaining "here's why that hurt"). Yet, learning from feedback for LLMs has largely focused on domains like code and math, where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly used to simulate human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, DITTO receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce SOUL (Simulation gym Of hUman-Like behavior), a unified
benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. DITTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL
benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.
Read the full abstract ▾
All authors Weiwei Sun, Xuhui Zhou, Jiarui Liu, Weihua Du, Haojia Sun, Yiqing Xie, Qianou Ma, Sihao Chen, Mengting Wan, Longqi Yang, Pei Zhou, Sherry Wu, Sean Welleck, Graham Neubig, Yiming Yang, Maarten Sap
Computation and Language
Stage-Audit: Auditable Source-Frontier Discovery for Cross-Wiki Tables
Chen Shen
LLM-curated tables can appear source-grounded while containing unsupported rows: the curator may recall entries from parametric
memory and retroactively attach page-level citations that are not the actual source. We study this hazard in Seed2Frontier discovery: the task of finding complement Wikipedia pages from a seed page to assemble a structured table. Stage-Audit addresses it with disjoint curator-auditor write rights, a row-level source-citation gate, and a 12-check audit taxonomy over keys, schema, source roles, cardinality, and scope. On a curated 51-instance Seed2Frontier evaluation set spanning 15 top-level domains, Stage-Audit improves source-frontier precision over a vanilla LLM curator from 0.356 to 0.505 (+42% relative) and F1 from 0.334 to 0.451 (+35%), while maintaining explicit per-row source traceability. The vanilla-LLM-vs-Stage-Audit comparison isolates the policy contribution rather than LLM-based discovery in general.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Training Language Agents to Learn from Experience
Yuval Shalev, Zifeng Ding and Mateja Jamnik
Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in
language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system
prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the
benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving
language agents.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SI
Hidden in Plain Sight: Finding MAHA on Reddit
Sabit Ahmed, Subigya Nepal and Henry Kautz
Make America Healthy Again (MAHA) is a national health movement that encompasses a striking mix of beliefs, from broadly accepted concerns about good diet and exercise to controversial takes on organic and genetically modified food, childhood vaccination, science, and institutions. Various influencers and promoters of the MAHA movement on social media are scattered throughout the online space. Investigating the structure, discourse, and contagion of MAHA beliefs requires large-scale fine-grained digital footprints. Constructing structured data covering different MAHA themes from vast unstructured social media data is challenging. We introduce a Reddit
dataset that spans six years (2020-2025), comprising 18.5M posts from 4M users. Containing the natural and thematic context of 12 MAHA-aligned beliefs, this
dataset offers researchers from various domains the opportunity to study the dynamics of the MAHA movement, its structural and functional components, and the linguistic and behavioral patterns of its proponents.
Read the full abstract ▾
Computation and Language
Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs
Edie Pearman, Sophia Osborne, Mira Kandlikar-Bloch et al. (+3 more)
Large
language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM
benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining
benchmark-based evaluation with mechanistic interpretability techniques and
reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across
benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain
attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of
reasoning chains further suggests that these improvements stem from memorization and familiarity with the
dataset rather than genuine understanding of bias.
Read the full abstract ▾
All authors Edie Pearman, Sophia Osborne, Mira Kandlikar-Bloch, Mina Arzaghi, Florian Carichon, Golnoosh Farnadi
Computation and Language
Puzzled By ChatGPT? No more! A Jigsaw Puzzle to Promote AI Literacy and Awareness
Francesca Padovani and Malvina Nissim
The rapid adoption of Generative AI, including LLM-based chatbots like ChatGPT, has highlighted the need for accessible ways to support public understanding and AI literacy. To address this need, we introduce a game-based, interactive approach in the form of a jigsaw puzzle whose completed image is a comic-based infographic illustrating the workings, capabilities, limitations, and societal implications of these technologies. Each comic sketch also functions as a standalone informational card, providing focused explanations of specific facets of AI use, design, and impact. The visual content was created in a live collaborative session with a professional illustrator and a multidisciplinary group of experts and non experts, combining structured knowledge with informal, exploratory reflections shared during the discussion. By integrating hands-on assembly, visual storytelling, and collaborative interaction, the puzzle provides an engaging and playful tool for exploring the mechanisms, perks, and perils of AI systems in informal learning contexts.
Read the full abstract ▾
Computation and Language
Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
Carolina Camassa and Derek Shiller
Language models are trained to follow instructions, but they are also powerful pattern completers. What happens when these two objectives conflict? We construct conversations in which a user instruction to behave in a target way T (e.g., always output a specific
token, answer in a particular
language, or adopt a persona) is opposed by N hardcoded assistant turns demonstrating a competing pattern P. We then measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns. Average instruction-following rates range from 1% to 99% across models, largely uncorrelated with standard capability
benchmarks. The transition from instruction-following to pattern-following is universal but highly model-dependent. Robustness is modulated both by instruction content, with models resisting induction longer when instructions align with their trained value priors, and by output format, with diverse multi-
token responses proving substantially more resistant than single-
token outputs. Chain-of-thought
reasoning improves robustness but does not eliminate susceptibility, and can produce dissociation between correct deliberation and incorrect output. When asked to predict their behavior in this setting, models achieve 83.5% accuracy on average but systematically underestimate their own resistance to induction pressure. These results suggest that instruction-following remains brittle under induction pressure even for otherwise capable models, and that output diversity, rather than semantic engagement with the input, is the primary factor predicting robustness.
Read the full abstract ▾
Computation and Language
DEL: Digit Entropy Loss for Numerical Learning of Large Language Models
Zhaohui Zheng, Chenhang He, Shihao Wang et al. (+3 more)
Number prediction stands as a fundamental capability of large
language models (LLMs) in mathematical problem-solving and code generation. The widely adopted maximum likelihood estimation (MLE) for LLM training is not tailored to number prediction. Recently, penalty-driven approaches, e.g., Number
Token Loss and Discretized Distance Loss, introduce an inductive bias of numerical distance but induce over-sharpened and over-flattened digit distributions, respectively. In this paper, we make an in-depth analysis on LLM numerical learning, and show that existing numerical learning methods conceptually follow a criterion-distance formulation, where the criterion term represents optimization pattern and the distance term instills
geometric prior. Consequently, we present Digit
Entropy Loss (DEL) for auto-regressive numerical learning, which reformulates the conventional unsupervised
entropy optimization in three key designs: leveraging digit conditional probability and binary cross-
entropy to guide the
entropy optimization into a supervised manner; deprecating the distance term to bypass the issue of numerical distance; and generalizing the integer-based numerical learning to floating-point number optimization, enabling more accurate number prediction. Our DEL formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating-point number domain. Experiments conducted on seven mathematical
reasoning benchmarks with four representative LLMs, including CodeLlama, Mistral, DeepSeek, and Qwen-2.5, demonstrate that DEL consistently outperforms its counterparts in both overall prediction accuracy and numerical distance. Source codes are at https://github.com/PolyU-VCLab/DEL
Read the full abstract ▾
All authors Zhaohui Zheng, Chenhang He, Shihao Wang, Yuxuan Li, Ming-Ming Cheng, Lei Zhang
Computation and Language
When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
Jinlong Liu, Mohammed Bahja and Mark Lee
Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a
dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this
dataset, we study whether reviewer-style
reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to $85.9$–$90.9\%$. Failure analysis shows that $88.6\%$ of failed
reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story–criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit
reasoning traces can introduce generation instability that outweighs their overall benefit.
Read the full abstract ▾
Computation and Language
Synchronization and Turn-Taking in Full-Duplex Speech Dialogue Models
Pablo Riera, Pablo Brusco, Cristina Kuo et al. (+2 more)
Full-duplex spoken dialogue models (SDMs) can listen and speak simultaneously, enabling interaction dynamics closer to human conversation than turn-based systems. Inspired by neural coupling in human communication, we study how such models coordinate their internal representations during interaction. We simulate full-duplex dialogues between two instances of the pretrained Moshi model under controlled conditions, manipulating channel noise and decoding bias. Synchronization is measured using Centered Kernel
Alignment (CKA) across temporal lags, while anticipatory turn-taking cues are probed from delayed internal activations using causal LSTM models, from both speaker and listener perspectives. We find strong representational synchronization under no noise conditions, peaking near zero lag and degrading with noise, and we show that internal states encode anticipatory information that supports turn-taking prediction ahead of time.
Read the full abstract ▾
All authors Pablo Riera, Pablo Brusco, Cristina Kuo, Marcelo Sancinetti, S.R.K. Branavan
Computation and Language
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
Haiquan Lu, Zigeng Chen, Gongfan Fang et al. (+2 more)
LLM
agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use,
memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM
agents. Extensive experiments across long-context and agentic
benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
Read the full abstract ▾
All authors Haiquan Lu, Zigeng Chen, Gongfan Fang, Xinyin Ma, Xinchao Wang
Computation and Language
TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
Zhiben Chen, Youpeng Zhao, Yang Sui et al. (+2 more)
Diffusion Large
Language Models (dLLMs) have emerged as a competitive alternative to autoregressive (AR) models, offering better hardware utilization and bidirectional context through parallel block-level decoding. However, as dLLMs continue to scale up with mixture-of-experts (MoE) architectures, their deployment on resource-constrained devices remains an open challenge. Existing AR-based methods often incur either prohibitive I/O overhead or significant compute bottlenecks. In this work, we propose TIDE, a novel resource-efficient inference system that leverages the temporal stability of expert activations during the
diffusion process within the block. Specifically, we leverage the temporal stability of expert activations during the
diffusion process within the block and introduce an interval-based expert refresh strategy that updates the expert placement in an I/O-aware fashion. To ensure optimal performance, we formulate the inference scheduling as a mathematical programming problem, solving for the optimal interval that minimizes I/O traffic and CPU computation. Most importantly, TIDE is a lossless optimization that requires no model training, providing a "free lunch" acceleration for dLLM inference. In a single GPU-CPU system, we demonstrate that TIDE achieves up to 1.4$\times$ and 1.5$\times$ throughput improvements over prior baselines on LLaDA2.0-mini and LLaDA2.0-flash models, respectively.
Read the full abstract ▾
All authors Zhiben Chen, Youpeng Zhao, Yang Sui, Jun Wang, Yuzhang Shang
Computation and Language
From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
Juncheng Wu, Hardy Chen, Haoqin Tu et al. (+6 more)
Recent advances in
vision-
language models (VLMs) emphasize long chain-of-thought
reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to
reasoning itself. In this work, we systematically study the interplay between perception and
reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual
reasoning, and textual
reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual
reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and
reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher
reasoning accuracy with 20.8% shorter
reasoning traces, suggesting that superior perception reduces the need for excessive
reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.
Read the full abstract ▾
All authors Juncheng Wu, Hardy Chen, Haoqin Tu, Xianfeng Tang, Freda Shi, Hui Liu, Hanqing Lu, Cihang Xie, Yuyin Zhou
Computation and Language
ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning
Juncheng Wu, Letian Zhang, Yuhan Wang et al. (+9 more)
Large
language models (LLMs) and agentic systems have shown promise for clinical decision support, but existing works largely assume that evidence has already been curated and handed to the model. Real-world clinical workflows instead require
agents to actively seek, iteratively plan, and synthesize
multimodal evidence from heterogeneous sources. In this paper, we introduce ClinSeekAgent, an automated agentic framework for dynamic
multimodal evidence seeking that shifts the paradigm from passive evidence consumption to active evidence acquisition. Given a clinical task and access to raw patient data, ClinSeekAgent queries longitudinal EHR records and invokes imaging analysis tools when relevant; refines its hypotheses as new information emerges; and integrates the collected evidence into grounded clinical decisions. We validate ClinSeekAgent in two roles: as an inference-time
agent for frontier LLMs and as a training pipeline for open-source models. For inference-time validation, we construct ClinSeek-Bench, which covers 13 text-only EHR task categories and 6
multimodal EHR–chest X-ray task categories, and pairs each question with both a Curated Input of pre-selected evidence and the raw patient data for automated evidence seeking. Without any
benchmark-specific curation, ClinSeekAgent significantly outperforms curated input for both Claude Opus 5 and GPT-5.6 Sol, improving overall F1 from 66.5 to 68.8 and from 60.0 to 62.5 on
multimodal tasks. The ablations show that the performance gain come from multi-round evidence seeking rather than from access to more of the patient record. For training-time validation, we fine-tune Qwen3.5-35B-A3B on ClinSeekAgent trajectories generated by GPT-5.6 Sol. The resulting ClinSeek-35B-A3B reaches 35.6 average F1 on AgentEHR-Bench, outperforming all evaluated open-source models, including ones more than ten times larger, as well as Claude Opus 5.
Read the full abstract ▾
All authors Juncheng Wu, Letian Zhang, Yuhan Wang, Haoqin Tu, Hardy Chen, Zijun Wang, Cihang Xie, Yufan He, Can Zhao, Daguang Xu, Yucheng Tang, Yuyin Zhou
Computation and Language
KoRe: Compact Knowledge Representations for Large Language Models
Davide Cavicchini, Fausto Giunchiglia and Jacopo Staiano
Modern Large
Language Models (LLMs) have shown impressive performances in user-facing tasks such as question answering, as well as consistent improvements in
reasoning capabilities. Still, the way these models encode knowledge seems inherently flawed: by design, LLMs encode world-knowledge within their parameters. This way of representing knowledge is inherently opaque, difficult to debug and update, and prone to
hallucinations. On the other hand, Knowledge Graphs can provide human-readable and easily editable world knowledge representations, and their application in knowledge-intensive tasks has consistently proven beneficial to downstream performance. Nonetheless, current integration techniques require extensive retraining or finetuning. To overcome this issue, we introduce KoRe, a methodology to encode 1-hop sub-graphs into compact discrete knowledge
tokens and inject them into a LLM backbone. We test the proposed approach on three established
benchmarks, and report competitive performances coupled with a significant reduction (up to 10x) in
token usage. Our results show that compact discrete KG representations can efficiently and effectively be used to ground modern LLMs.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
Guangzhi Xiong, Qiao Jin, Sanchit Sinha et al. (+2 more)
Large
Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model
reasoning are typically unavailable. We address this question for chest X-ray (CXR)
reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step
reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and
token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.
Read the full abstract ▾
All authors Guangzhi Xiong, Qiao Jin, Sanchit Sinha, Zhiyong Lu, Aidong Zhang
Computation and Language
Less Back-and-Forth: A Comparative Study of Structured Prompting
Saurav Ghosh
Large
language models (LLMs) are widely used for open-ended tasks, but underspecified
prompts can lead to low-quality answers and additional interaction. This paper studies whether structured
prompt design improves response quality while reducing user effort. We compare three
prompt conditions: a raw
prompt, a checklist-improved
prompt, and a clarifying-question
prompt. We evaluate these conditions across four task types–summarization, planning, explanation, and coding–using three LLM systems: ChatGPT, Claude, and Grok. Each output is scored with a unified rubric covering task completion, correctness, compliance, and clarity. Checklist-improved
prompts achieved the highest mean rubric score, 7.50 out of 8, compared with 5.67 for raw
prompts and 6.67 for clarifying-question
prompts. Checklist
prompts also produced the best quality-effort tradeoff, using fewer average
tokens than both raw and clarifying
prompts. These results suggest that a simple
prompt checklist can improve LLM responses while reducing unnecessary interaction.
Read the full abstract ▾
Computation and Language
MixRea: Benchmarking Explicit-Implicit Reasoning in Large Language Models
Yuanqing Cai, Ziyi Huang, Minhao Liu et al. (+3 more)
Large
language models (LLMs) are increasingly integrated into high-stakes decision-making. Inspired by the theory of inattentional blindness in human cognition, we investigate whether LLMs, trained on human-preferred corpora that embed attentional biases, exhibit a similar limitation: failing to attend to subtle yet important contextual cues under explicit task instructions. To evaluate this, we introduce the task of explicit-implicit
reasoning and present MixRea, a
benchmark of 2,246 multiple-choice questions across 9
reasoning types with varying distributions of explicit and implicit information. Evaluation of 21 advanced LLMs shows that even the best-performing
reasoning model (Gemini 2.5 Pro) achieves only 42.8% consistency, revealing widespread inattentional blindness. To mitigate this, we propose Potential Relation Completion Prompting (PRCP), a prompting method that improves
reasoning by recovering overlooked causal relations. Further analysis shows that this limitation persists across diverse multi-source
reasoning tasks, highlighting the need for more cognitively aligned models.
Read the full abstract ▾
All authors Yuanqing Cai, Ziyi Huang, Minhao Liu, Lixin Duan, Wen Li, Yanru Zhang
Computation and Language
ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
Chuanyang Jin, Binze Li, Haopeng Xie et al. (+6 more)
Conversational AI has now reached billions of users, yet existing
datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale
dataset that pairs real-world multi-turn human–AI conversations with users' self-reported thoughts: their reasons for sending
prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20
language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained
alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human–AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.
Read the full abstract ▾
All authors Chuanyang Jin, Binze Li, Haopeng Xie, Cathy Mengying Fang, Tianjian Li, Shayne Longpre, Hongxiang Gu, Maximillian Chen, Tianmin Shu
Computation and Language
BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation
Zijun Jia, Yuanchang Ye, Sen Jia et al. (+6 more)
Large
language models (LLMs) can enhance factuality via
retrieval-augmented generation (RAG), but applying RAG to every query is unnecessary when the model-only answer is reliable. This motivates cascaded RAG: each query is first handled by an LLM-only branch, escalated to a RAG fallback only if the primary branch is uncertain, and abstained from when neither branch is sufficiently trustworthy. However, calibrating such cascades stage by stage may be conservative, since the final utility depends on joint uncertainty thresholding of LLM-only and RAG. In this work, we develop BalanceRAG to certify threshold pairs at a target risk level. Given uncertainty scores from the two branches, BalanceRAG frames each threshold pair as an operating point on a two-dimensional
lattice and identifies safe operating points using sequential graphical testing. This enables risk-adaptive threshold calibration, controlling the system-level error rate among accepted points, while retaining more examples. Furthermore, BalanceRAG extends to multi-risk calibration, allowing
retrieval usage to be bounded together with the selection-conditioned risk. Experiments on three open-domain question answering (QA)
benchmarks across multiple LLM backbones demonstrate that BalanceRAG meets prescribed risk levels, preserves higher coverage and more accepted correct examples, and reduces unnecessary
retrieval calls compared with always-on RAG.
Read the full abstract ▾
All authors Zijun Jia, Yuanchang Ye, Sen Jia, Yiyao Qian, Haoning Wang, Baojie Chen, Diyin Tang, Jinsong Yu, Zhiyuan Wang
Computation and Language
CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning
Dachuan Shi, Hanlin Zhu, Xiangchi Yuan et al. (+4 more)
Chain-of-thought (CoT) is a standard approach for eliciting
reasoning capabilities from large
language models (LLMs). However, the common CoT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary
token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative
reasoning. In this paper, we introduce CopT, a reformulated
reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous
embeddings as inference-time contrastive verifiers. Specifically, it contrasts the model's support for the same generated
tokens under discrete-
token inputs and continuous-
embedding inputs, yielding a sequence-level reverse KL
estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer
token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, CopT performs further on-policy thinking, where a second KL
estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic
reasoning tasks, CopT improves peak accuracy by up to 23% and reduces
token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.
Read the full abstract ▾
All authors Dachuan Shi, Hanlin Zhu, Xiangchi Yuan, Wanjia Zhao, Kejing Xia, Wen Xiao, Wenke Lee
Computation and Language
Text-to-SPARQL Generation with Reinforcement Learning: A GRPO-based Approach on DBLP
Jann Pfeifer, Debayan Banerjee and Ricardo Usbeck
Knowledge graph question answering seeks to translate natural
language questions into executable queries over knowledge graphs, but existing approaches often rely on large models or full supervision in the form of gold query annotations. This study examines whether reinforcement learning with outcome-based rewards can train a small instruction-tuned
language model to perform zero-shot Text-to-SPARQL generation in the scholarly domain. Group-Relative Policy Optimization (GRPO) is applied to the Qwen3-1.7B model on DBLP-QuAD, using
prompts that combine natural
language questions with symbolic hints about entities and relations. Training relies on execution feedback, structural constraints, and answer-level rewards, with an additional variant that incorporates gold-query-based shaping. The resulting models are compared to the unmodified zero-shot baseline and to a supervised DoRA-finetuned baseline across answer-level accuracy, execution accuracy, category-wise scores, and generalization to held-out templates. GRPO substantially improves over the zero-shot baseline and exhibits competitive generalization, while supervised DoRA finetuning achieves higher overall accuracy on the same model scale. Ablation analyses indicate that execution-based rewards account for most gains, with additional shaping yielding limited additional benefit, suggesting that outcome-based reinforcement learning is a viable training strategy when gold queries are unavailable for
token-level supervision.
Read the full abstract ▾
Computation and Language
Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents
Wenjie Tang, Minne Li, Sijie Huang et al. (+2 more)
Reinforcement learning from verifiable rewards (RLVR) is a promising paradigm for improving large
language model (LLM)
agents on long-horizon interactive tasks. However, in partially observable environments, incomplete observations cause
agent beliefs to drift over time, while delayed rewards obscure the causal impact of intermediate decisions, exacerbating temporal credit assignment challenges. To address this, we propose ReBel (Reward Belief), a process-level reinforcement learning
algorithm that explicitly models structured belief states to summarize interaction history and guide subsequent policy learning. ReBel introduces belief-consistency supervision, converting discrepancies between predicted beliefs and observed feedback into dense self-supervised signals without requiring external step-wise annotations or verifiers. It also employs belief-aware grouping to compare trajectories under similar belief states, yielding more robust and lower-variance advantage estimates. We evaluate ReBel on challenging long-horizon
benchmarks, including ALFWorld and WebShop. ReBel improves task success by up to $20.4$ percentage points over the episode-level baseline GRPO and increases sample efficiency by $2.1\times$. These results suggest that belief-aware self-supervision is a promising direction for reliable long-horizon decision-making under partial observability. Code is available at: https://github.com/Fateyetian/Rebel.git.
Read the full abstract ▾
All authors Wenjie Tang, Minne Li, Sijie Huang, Liquan Xiao, Yuan Zhou
Computation and Language
PromptRad: Knowledge-Enhanced Multi-Label Prompt-Tuning for Low-Resource Radiology Report Labeling
Ying-Jia Lin, Tzu-Chin Lo, Ping-Chien Li et al. (+3 more)
Automatic report labeling facilitates the identification of clinical findings from unstructured text and enables large-scale annotation for medical imaging research. Existing rule-based labelers struggle with the diverse descriptions in clinical reports, while fine-tuning pre-trained
language models (PLMs) requires large amounts of labeled data that are often unavailable in clinical settings. In this paper, we propose PromptRad, a knowledge-enhanced multi-label
prompt-tuning approach for radiology report labeling under low-resource settings. PromptRad reformulates multi-label classification as masked
language modeling and incorporates synonyms from the UMLS Metathesaurus into a multi-word verbalizer to enrich category representations. By fine-tuning the PLM without additional classification layers, PromptRad requires substantially less labeled data than conventional fine-tuning. Experiments on liver CT (computed
tomography) reports show that PromptRad outperforms dictionary-based and fine-tuning baselines with only 32 labeled training examples, and achieves competitive performance with GPT-4 despite using a much smaller model. Further analysis demonstrates that PromptRad captures complex negation patterns more effectively than existing methods, making it a promising solution for report labeling in data-scarce clinical scenarios. Our code is available at https://github.com/ila-lab/PromptRad.
Read the full abstract ▾
All authors Ying-Jia Lin, Tzu-Chin Lo, Ping-Chien Li, Chi-Tung Cheng, Chien-Hung Liao, Hung-Yu Kao
Computation and Language
Language Mutations Sustain the Persistences of Conspiracy Theories on Social Media
Calvin Yixiang Cheng, Dorian Quelle and Scott A. Hale
This study investigates how
language mutations affect the persistent
diffusion of conspiracy theories on social media. Drawing on a three-year
dataset of conspiracy-related posts from X, and applying computational linguistic analysis alongside survival modelling, we find that conspiracy claims with greater semantic mutations have substantially longer lifespans. Mutations in psycholinguistic properties, including pronouns, social reference words, cognitive process terms, risk- and health- related vocabularies, are associated with extended lifespans. Mutations in actor, action and target (AAT) categories are associated with longer lifespans as well. Qualitative analysis identifies two predominant mutation patterns: simplification and assimilation, at both linguistic and AAT structural levels. Taken together, the results advance our understanding of how
language mutations contribute to conspiracy persistence online and shed lights on longitudinal content moderation strategies. We argue that content moderation should consider the mutability of conspiracy claims and focus on the core claims that can address their potential variations.
Read the full abstract ▾
Computation and Language
Mind Your Moras: Orthography-Aware Error Analysis of Neural Japanese Morphological Generation
Wen Zhang
We present an orthography-aware error analysis of Japanese past-tense morphological inflection, treating hiragana not merely as a transcriptional medium, but as a representational system encoding morphophonological distinctions that may influence model generalization. We evaluate two character-level sequence-to-sequence architectures on past-tense formation using
datasets formatted according to the SIGMORPHON 2020 and 2023 shared task conventions. Despite high aggregate accuracy, models exhibit systematic, linguistically interpretable errors that cluster around specific orthographic properties of hiragana. We introduce a concise error taxonomy capturing seven primary failure modes and provide both quantitative and qualitative analyses. Gemination-related errors dominate residual failures, accounting for 75-80% of errors, particularly in verbs whose stems end in the vowel e and require gemination before the past-tense suffix. Error patterns remain highly consistent across architectures and random seeds, suggesting a robust interaction between orthographic representation, morphological structure, and data frequency effects in shaping model generalization. These results underscore the necessity of orthography-aware evaluation for understanding neural generalization in morphologically complex
languages.
Read the full abstract ▾
Computation and Language
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
Yaojie Zhang, Jianuo Huang, Junlong Ke et al. (+5 more)
Speculative decoding accelerates
memory-bound LLM inference without quality degradation by using a fast drafter to propose multiple candidate
tokens and the target model to verify them in parallel. However, conventional sequential speculative decoding suffers from mutual waiting between drafting and verification, and repeated exchange of intermediate states further increases
memory access overhead. Parallel speculative decoding addresses this limitation by performing drafting and verification within a single target forward pass, allowing future drafts to be prepared while current candidates are being verified. Although effective at small batch sizes, existing parallel speculative decoding methods either require costly continual pretraining with quality degradation or suffer from low acceptance rates. More importantly, this paradigm inherently suffers from uncertainty in both the bonus
token and the accepted length, leading to draft verification mismatch and causing throughput gains to collapse at large batch sizes. To address these limitations, we introduce FlexDraft, a lossless speculative decoding framework that flexibly adapts to varying batch sizes through three key designs. (1)
Attention Tuning enables block
diffusion drafting by tuning only the
attention projectors of the final few layers on mask
tokens, while keeping the autoregressive path frozen to preserve the target distribution and produce high quality drafts with minimal trainable parameters. (2) Bonus-guided Calibration uses a lightweight MLP conditioned on the resolved bonus
token to calibrate draft logits, mitigating draft verification mismatch caused by bonus
token uncertainty. (3) Flex Decoding dynamically switches between parallel draft and verify at small batch sizes and sequential draft then verify at large batch sizes, and adjusts verification length based on draft confidence to eliminate redundant computation.
Read the full abstract ▾
All authors Yaojie Zhang, Jianuo Huang, Junlong Ke, Yuhang Han, Yongji Long, Tianchen Zhao, Biqing Qi, Linfeng Zhang
Computation and Language
Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory
Jingwei Sun, Jianing Zhu, Jiangchao Yao et al. (+2 more)
To enable reliable long-term interaction, LLM
agents require a
memory system that can faithfully store, efficiently retrieve, and deeply reason over accumulated dialogue history. Most existing methods adopt an extracted fact based paradigm: handcrafted static
prompts compress raw dialogues into atomic facts, which are then stored, matched, and injected into downstream
reasoning. Nevertheless, such fact-centric designs inevitably discard fine-grained details in original dialogues and fail to support deep
reasoning over scattered isolated facts. Moreover, static
prompts cannot maintain consistent extraction granularity across diverse dialogue styles. To address these limitations, we propose TriMem, which maintains three coexisting representation granularities, including raw dialogue segments anchored by source identifiers for storage fidelity, extracted atomic facts for efficient
memory retrieval, synthesized profiles that aggregate dispersed facts into holistic semantic understanding for deep
reasoning. We further adopt TextGrad-based
prompt optimization, which iteratively refines extraction and profiling
prompts via response quality feedback, achieving lifelong
evolution without any parameter updating. Extensive experiments on LoCoMo and PerLTQA across multiple LLM backbones demonstrate that TriMem consistently outperforms strong
memory baselines. The code is available at https://TMLR-TriMem.github.io .
Read the full abstract ▾
All authors Jingwei Sun, Jianing Zhu, Jiangchao Yao, Tongliang Liu, Bo Han
Computation and Language · Cross-listed from cs.DC
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
Sourish Wawdhane, Avinash Kumar and Poulami Das
Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per
token. MoE serving engines distribute experts across multiple GPUs and route
tokens to appropriate GPUs at inference time based on experts activated. They process
tokens in lock-step fashion, where
tokens within a batch must finish processing before proceeding to the next layer. This synchronization barrier acts as a critical bottleneck because the performance of MoE models is limited by the straggler GPU that finishes last. Stragglers emerge when too many heavily used experts are placed on the same GPU or the slowest GPU. While prior works place experts that balance
token loads across GPUs, they all overlook GPU variability and often place highly used experts on the slowest GPUs. We propose GEM, GPU-variability-aware Expert Mapping, a framework for GPU variability-aware expert to GPU mapping for MoE models. GEM exploits two insights. First, we must place experts such that each GPU receives non-uniform
token loads based on their variability and they all finish processing a layer at about the same time. Our studies show that there are two types of experts: consistent that are used most of the time and temporal that are often used together for the remaining time. Our second insight is that we must place simultaneously used consistent and temporal experts on different GPUs and avoid placing them on slower GPUs to reduce slowdown. GEM gathers the variability profile of GPUs for each model and task and uses the
token load distributions per task to map experts to GPUs. Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.
Read the full abstract ▾