Computation and Language · Cross-listed from q-bio.NC
Coherence in the brain unfolds across separable temporal regimes
Davide Staub, Finn Rabe, Akhil Misra et al. (+10 more)
To maintain coherence in
language, the
brain must satisfy key competing temporal demands: the gradual accumulation of meaning across extended context (drift) and the rapid reconfiguration of representations at event boundaries (shift). How these processes are implemented in the human
brain during naturalistic listening remains unclear. Here, we tested whether both can be captured by annotation-free drift and shift signals and whether their neural expression shows distinct regional preferences across the
brain. These signals were derived from a large
language model (LLM) processing the narrative input. To enable high-precision voxelwise encoding models with stable parameter estimates, we densely sampled one healthy adult across more than 7 hours of listening to crime stories while collecting 7 Tesla fMRI data. We then modeled the feature-informed hemodynamic response using a regularized encoding framework validated on independent stories. Drift predictions were prevalent in default-mode network hubs, whereas shift predictions were evident bilaterally in the primary auditory cortex and
language association cortex. Together, these findings show that coherence during
language comprehension is implemented through distinct but co-expressed neural regimes of slow contextual integration and rapid event-driven reconfiguration, offering a mechanistic entry point for understanding disturbances of
language coherence in psychiatric disorders.
Read the full abstract ▾
All authors Davide Staub, Finn Rabe, Akhil Misra, Yves Pauli, Roya Hüppi, Ni Yang, Nils Lang, Lars Michels, Victoria Edkins, Sascha Frühholz, Iris Sommer, Wolfram Hinzen, Philipp Homan
Computation and Language
Sentiment-Aware Extractive and Abstractive Summarization for Unstructured Text Mining
Junyi Liu and Stanley Kok
With the rapid growth of unstructured data from social media, reviews, and forums, text mining has become essential in Information Systems (IS) for extracting actionable insights. Summarization can condense fragmented,
emotion-rich posts, but existing methods-optimized for structured news-struggle with noisy, informal content. Emotional cues are critical for IS tasks such as brand monitoring and
market analysis, yet few studies integrate sentiment modeling into summarization of short user-generated texts. We propose a sentiment-aware framework extending extractive (TextRank) and abstractive (UniLM) approaches by
embedding sentiment signals into ranking and generation processes. This dual design improves the capture of emotional nuances and thematic relevance, producing concise, sentiment-enriched summaries that enhance timely interventions and strategic decision-making in dynamic online environments.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Generative Digital Twins: Vision-Language Simulation Models for Executable Industrial Systems
YuChe Hsu, AnJui Wang, TsaiChing Ni et al. (+1 more)
We propose a
Vision-
Language Simulation Model (VLSM) that unifies visual and textual understanding to synthesize executable FlexScript from layout sketches and natural-
language prompts, enabling cross-modal
reasoning for industrial simulation systems. To support this new paradigm, the study constructs the first large-scale
dataset for generative digital twins, comprising over 120,000
prompt-sketch-code triplets that enable
multimodal learning between textual descriptions, spatial structures, and simulation logic. In parallel, three novel evaluation metrics, Structural Validity Rate (SVR), Parameter Match Rate (PMR), and Execution Success Rate (ESR), are proposed specifically for this task to comprehensively evaluate structural integrity, parameter fidelity, and simulator executability. Through systematic ablation across
vision encoders, connectors, and code-pretrained
language backbones, the proposed models achieve near-perfect structural accuracy and high execution robustness. This work establishes a foundation for generative digital twins that integrate visual
reasoning and
language understanding into executable industrial simulation systems. Project page: https://danielhsu2014.github.io/GDT-VLSM-project/
Read the full abstract ▾
All authors YuChe Hsu, AnJui Wang, TsaiChing Ni, YuanFu Yang
Computation and Language
Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen's Kappa and Semantic Similarity for Qualitative Research Validation
Nilesh Jain, Hyungil Suh, Seyi Adeyinka et al. (+2 more)
Qualitative research faces a critical reliability challenge: traditional inter-rater agreement methods require multiple human coders, are time-intensive, and often yield moderate consistency. We present a multi-perspective validation framework for LLM-based thematic analysis that combines ensemble validation with dual reliability metrics: Cohen's Kappa ($\kappa$) for inter-rater agreement and cosine similarity for semantic consistency. Our framework enables configurable analysis parameters (1-6 seeds, temperature 0.0-2.0), supports custom
prompt structures with variable substitution, and provides consensus theme extraction across any JSON format. As proof-of-concept, we evaluate three leading LLMs (Gemini 2.5 Pro, GPT-4o, Claude 3.5 Sonnet) on a psychedelic art therapy interview transcript, conducting six independent runs per model. Results demonstrate Gemini achieves highest reliability ($\kappa = 0.907$, cosine=95.3%), followed by GPT-4o ($\kappa = 0.853$, cosine=92.6%) and Claude ($\kappa = 0.842$, cosine=92.1%). All three models achieve a high agreement ($\kappa > 0.80$), validating the multi-run ensemble approach. The framework successfully extracts consensus themes across runs, with Gemini identifying 6 consensus themes (50-83% consistency), GPT-4o identifying 5 themes, and Claude 4 themes. Our open-source implementation provides researchers with transparent reliability metrics, flexible configuration, and structure-agnostic consensus extraction, establishing methodological foundations for reliable AI-assisted qualitative research.
Read the full abstract ▾
All authors Nilesh Jain, Hyungil Suh, Seyi Adeyinka, Leor Roseman, Aza Allsop
Computation and Language
Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles
Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Khushnur Binte Jahangir et al. (+2 more)
Large
Language Models (LLMs) show impressive performance on many NLP
benchmarks, yet their ability to reason in figurative, culturally grounded, and low-resource settings remains underexplored. We address this gap for Bangla by introducing BanglaRiddleEval, a
benchmark of 1,244 traditional Bangla riddles instantiated across four tasks (4,976 riddle-task artifacts in total). Using an LLM-based pipeline, we generate Chain-of-Thought explanations, semantically coherent distractors, and fine-grained ambiguity annotations, and evaluate a diverse suite of open-source and closed-source models under different prompting strategies. Models achieve moderate semantic overlap on generative QA but low correctness, MCQ accuracy peaks at only about 56% versus an 83% human baseline, and ambiguity resolution ranges from roughly 26% to 68%, with high-quality explanations confined to the strongest models. These results show that current LLMs capture some cues needed for Bangla riddle
reasoning but remain far from human-level performance, establishing BanglaRiddleEval as a challenging new
benchmark for low-resource figurative
reasoning. All data, code, and evaluation scripts are available on GitHub: https://github.com/Labib1610/BanglaRiddleEval.
Read the full abstract ▾
All authors Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Khushnur Binte Jahangir, Swakkhar Shatabda, Sarah Masud Preum
Computation and Language
SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Maxime Poli, Mahi Luthra, Youssef Benchekroun et al. (+8 more)
The parallel advances in
language modeling and
speech representation learning have raised the prospect of learning
language directly from
speech without textual intermediates. This requires extracting semantic representations directly from
speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised
speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken
language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-
distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream
language modeling
benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between
speech unit quality (ABX, PNMI) and
language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr.
Read the full abstract ▾
All authors Maxime Poli, Mahi Luthra, Youssef Benchekroun, Yosuke Higuchi, Martin Gleize, Jiayi Shen, Robin Algayres, Yu-An Chung, Mido Assran, Juan Pino, Emmanuel Dupoux
Computation and Language
Patterns vs. Patients: Evaluating LLMs against Mental Health Professionals on Personality Disorder Diagnosis through First-Person Narratives
Karolina Drożdż, Kacper Dudzic, Anna Sterna et al. (+1 more)
Growing reliance on LLMs for psychiatric self-assessment raises questions about their ability to interpret qualitative patient narratives. This depth over breadth case study directly compares state-of-the-art LLMs and mental health professionals in assessing Borderline (BPD) and Narcissistic (NPD) Personality Disorders based on Polish-
language first-person autobiographical accounts. Within our sample, the overall diagnostic scores of the top-performing Gemini Pro models (65.48%) were 21.91 percentage points higher than the average scores of the human professionals (43.57%). While both models and human experts excelled at identifying BPD (F1 = 83.4 & F1 = 80.0, respectively), models severely underdiagnosed NPD (F1 = 6.7 vs. 50.0), showing a potential reluctance toward the value-laden term "narcissism." Qualitatively, models provided confident, elaborate justifications focused on patterns and formal categories, while human experts remained concise and cautious, emphasizing the patients' sense of self and temporal experience. Our findings demonstrate that while LLMs might be competent at interpreting complex first-person clinical data, their outputs still carry critical reliability and bias issues.
Read the full abstract ▾
All authors Karolina Drożdż, Kacper Dudzic, Anna Sterna, Marcin Moskalewicz
Computation and Language
AprielGuard
Jaykumar Kasundra, Anjaneya Praharaj, Sourabh Surana et al. (+11 more)
Safeguarding large
language models (LLMs) against unsafe or
adversarial behavior is critical as they are increasingly deployed in conversational and agentic settings. Existing moderation tools often treat safety risks (e.g. toxicity, bias) and
adversarial threats (e.g.
prompt injections, jailbreaks) as separate problems, limiting their robustness and generalizability. We introduce AprielGuard, an 8B parameter safeguard model that unify these dimensions within a single taxonomy and learning framework. AprielGuard is trained on a diverse mix of open and synthetic data covering standalone
prompts, multi-turn conversations, and agentic workflows, augmented with structured
reasoning traces to improve interpretability. Across multiple public and proprietary
benchmarks, AprielGuard achieves strong performance in detecting harmful content and
adversarial manipulations, outperforming existing opensource guardrails such as Llama-Guard and Granite Guardian, particularly in multi-step and
reasoning intensive scenarios. By releasing the model, we aim to advance transparent and reproducible research on reliable safeguards for LLMs.
Read the full abstract ▾
All authors Jaykumar Kasundra, Anjaneya Praharaj, Sourabh Surana, Lakshmi Sirisha Chodisetty, Sourav Sharma, Abhigya Verma, Abhishek Bhardwaj, Debasish Kanhar, Aakash Bhagat, Khalil Slimi, Seganrasan Subramanian, Sathwik Tejaswi Madhusudhan, Ranga Prasad Chenna, Srinivas Sunkara
Computation and Language
SlideTailor: Personalized Presentation Slide Generation for Scientific Papers
Wenzheng Zeng, Mingyu Ouyang, Langyuan Cui et al. (+1 more)
Automatic presentation slide generation can greatly streamline content creation. However, since preferences of each user may vary, existing under-specified formulations often lead to suboptimal results that fail to align with individual user needs. We introduce a novel task that conditions paper-to-slides generation on user-specified preferences. We propose a human behavior-inspired agentic framework, SlideTailor, that progressively generates editable slides in a user-aligned manner. Instead of requiring users to write their preferences in detailed textual form, our system only asks for a paper-slides example pair and a visual template - natural and easy-to-provide artifacts that implicitly encode rich user preferences across content and visual style. Despite the implicit and unlabeled nature of these inputs, our framework effectively distills and generalizes the preferences to guide customized slide generation. We also introduce a novel chain-of-
speech mechanism to align slide content with planned oral narration. Such a design significantly enhances the quality of generated slides and enables downstream applications like
video presentations. To support this new task, we construct a
benchmark dataset that captures diverse user preferences, with carefully designed interpretable metrics for robust evaluation. Extensive experiments demonstrate the effectiveness of our framework.
Read the full abstract ▾
All authors Wenzheng Zeng, Mingyu Ouyang, Langyuan Cui, Hwee Tou Ng
Computation and Language
Corpus of Cross-lingual Dialogues with Minutes and Detection of Misunderstandings
Marko Čechovič, Natália Komorníková, Dominik Macháček et al. (+1 more)
Speech processing and translation technology have the potential to facilitate meetings of individuals who do not share any common
language. To evaluate automatic systems for such a task, a versatile and realistic evaluation corpus is needed. Therefore, we create and present a corpus of cross-lingual dialogues between individuals without a common
language who were facilitated by automatic simultaneous
speech translation. The corpus consists of 5 hours of
speech recordings with ASR and gold transcripts in 12 original
languages and automatic and corrected translations into English. For the purposes of research into cross-lingual summarization, our corpus also includes written summaries (minutes) of the meetings. Moreover, we propose automatic detection of misunderstandings. For an overview of this task and its complexity, we attempt to quantify misunderstandings in cross-lingual meetings. We annotate misunderstandings manually and also test the ability of current large
language models to detect them automatically. The results show that the Gemini model is able to identify text spans with misunderstandings with recall of 77% and precision of 47%.
Read the full abstract ▾
All authors Marko Čechovič, Natália Komorníková, Dominik Macháček, Ondřej Bojar
Computation and Language
FaithLens: Detecting and Explaining Faithfulness Hallucination
Shuzheng Si, Qingyi Wang, Haozhe Zhao et al. (+8 more)
Recognizing whether outputs from large
language models (LLMs) contain faithfulness
hallucination is crucial for real-world applications, e.g.,
retrieval-augmented generation and summarization. In this paper, we introduce FaithLens, a cost-efficient and effective faithfulness
hallucination detection model that can jointly provide binary predictions and corresponding explanations to improve trustworthiness. To achieve this, we first synthesize training data with explanations via advanced LLMs and apply a well-defined data filtering strategy to ensure label correctness, explanation quality, and data diversity. Subsequently, we fine-tune the model on these well-curated training data as a cold start and further optimize it with rule-based reinforcement learning, using rewards for both prediction correctness and explanation quality. Results on 12 diverse tasks show that the 8B-parameter FaithLens outperforms advanced models such as GPT-5.2 and o3. Also, FaithLens can produce high-quality explanations, delivering a distinctive balance of trustworthiness, efficiency, and effectiveness.
Read the full abstract ▾
All authors Shuzheng Si, Qingyi Wang, Haozhe Zhao, Yuzhuo Bai, Guanqiao Chen, Kangyang Luo, Gang Chen, Fanchao Qi, Minjia Zhang, Baobao Chang, Maosong Sun
Computation and Language · Cross-listed from cs.CV
Hao Guo, Xugong Qin, Jun Jie Ou Yang et al. (+4 more)
Document image
retrieval (DIR) aims to retrieve document images from a gallery according to a given query. Existing DIR methods are primarily based on image queries that retrieve documents within the same coarse semantic category, e.g., newspapers or receipts. However, these methods struggle to effectively retrieve document images in real-world scenarios where textual queries with fine-grained semantics are usually provided. To bridge this gap, we introduce a new Natural
Language-based Document Image
Retrieval (NL-DIR)
benchmark with corresponding evaluation metrics. In this work, natural
language descriptions serve as semantically rich queries for the DIR task. The NL-DIR
dataset contains 41K authentic document images, each paired with five high-quality, fine-grained semantic queries generated and evaluated through large
language models in conjunction with manual verification. We perform zero-shot and fine-tuning evaluations of existing mainstream contrastive
vision-
language models and OCR-free visual document understanding (VDU) models. A two-stage
retrieval method is further investigated for performance improvement while achieving both time and space efficiency. We hope the proposed NL-DIR
benchmark can bring new opportunities and facilitate research for the VDU community.
Datasets and codes will be publicly available at huggingface.co/
datasets/nianbing/NL-DIR.
Read the full abstract ▾
All authors Hao Guo, Xugong Qin, Jun Jie Ou Yang, Peng Zhang, Gangyan Zeng, Yubo Li, Hailun Lin
Computation and Language · Cross-listed from cs.LG
Learning to Reason in LLMs by Expectation Maximization
Junghyun Lee, Branislav Kveton, Anup Rao et al. (+4 more)
Large
language models (LLMs) solve
reasoning problems by first generating a rationale and then answering. We formalize
reasoning as a latent variable model and derive a reward-based filtered expectation-maximization (FEM) objective for learning to reason. This view connects EM and modern reward-based optimization, and shows that the main challenge lies in designing a sampling distribution of rationales that justify correct answers. We instantiate and compare three sampling schemes: rejection sampling with a budget, self-taught reasoner (
STaR), and
prompt posterior sampling (PPS), which only keeps the rationalization stage of
STaR that conditions on the correct answer in the
prompt. We experiment with LLM-as-a-judge calibration and summarization from feedback tasks, where conditioning on the correct answer provides a strong guidance for generating rationales. Our experiments show the efficacy of PPS over other sampling schemes, and that the sampling scheme can have a significant impact on performance.
Read the full abstract ▾
All authors Junghyun Lee, Branislav Kveton, Anup Rao, Subhojyoti Mukherjee, Ryan A. Rossi, Sunav Choudhary, Alexa Siu
Computation and Language
AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applications
Honglin Mu, Jinghao Liu, Kaiyang Wan et al. (+4 more)
Large
Language Models (LLMs) excel at text comprehension and generation, making them ideal for automated tasks like code review and content moderation. However, our research identifies a vulnerability: LLMs can be manipulated by "
adversarial instructions" hidden in input data, such as resumes or code, causing them to deviate from their intended task. Notably, while defenses may exist for mature domains such as code review, they are often absent in other common applications such as resume screening and peer review. This paper introduces a
benchmark to assess this vulnerability in resume screening, revealing attack success rates exceeding 80% for certain attack types. We evaluate two defense mechanisms:
prompt-based defenses achieve 10.1% attack reduction with 12.5% false rejection increase, while our proposed FIDS (Foreign Instruction Detection through Separation) using LoRA adaptation achieves 15.4% attack reduction with 10.4% false rejection increase. The combined approach provides 26.3% attack reduction, demonstrating that training-time defenses outperform inference-time mitigations in both security and utility preservation.
Read the full abstract ▾
All authors Honglin Mu, Jinghao Liu, Kaiyang Wan, Rui Xing, Xiuying Chen, Timothy Baldwin, Wanxiang Che
Computation and Language
Fun-Audio-Chat Technical Report
Tongyi Fun Team, Qian Chen, Luyao Cheng et al. (+10 more)
Recent advancements in joint
speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between
speech tokens (25Hz) and text
tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio
Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution
Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via
token grouping), while the
Speech Refined Head generates high-quality
tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding,
reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on
Speech-to-Text and
Speech-to-
Speech tasks, ranking top among similar-scale models on Spoken QA
benchmarks. They also achieve competitive to superior performance on Audio Understanding,
Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo, at https://github.com/FunAudioLLM/Fun-Audio-Chat .
Read the full abstract ▾
All authors Tongyi Fun Team, Qian Chen, Luyao Cheng, Chong Deng, Xiangang Li, Jiaqing Liu, Chao-Hong Tan, Wen Wang, Junhao Xu, Jieping Ye, Qinglin Zhang, Qiquan Zhang, Jingren Zhou
Computation and Language
Retrieval-augmented Prompt Learning for Pre-trained Foundation Models
Xiang Chen, Yixin Ou, Quan Feng et al. (+8 more)
The pre-trained foundation models (PFMs) have become essential for facilitating large-scale
multimodal learning. Researchers have effectively employed the “pre-train,
prompt, and predict” paradigm through
prompt learning to induce improved few-shot performance. However,
prompt learning approaches for PFMs still follow a parametric learning paradigm. As such, the stability of generalization in memorization and rote learning can be compromised. More specifically, conventional
prompt learning might face difficulties in fully utilizing atypical instances and avoiding overfitting to shallow patterns with limited data during the process of fully-supervised training. To overcome these constraints, we present our approach, named RetroPrompt, which aims to achieve a balance between memorization and generalization by decoupling knowledge from mere memorization. Unlike traditional prompting methods, RetroPrompt leverages a publicly accessible knowledge base generated from the training data and incorporates a
retrieval mechanism throughout the input, training, and inference stages. This enables the model to actively retrieve relevant contextual information from the corpus, thereby enhancing the available cues. We conduct comprehensive experiments on a variety of
datasets across natural
language processing and computer
vision tasks to demonstrate the superior performance of our proposed approach, RetroPrompt, in both zero-shot and few-shot scenarios. Through detailed analysis of memorization patterns, we observe that RetroPrompt effectively reduces the reliance on rote memorization, leading to enhanced generalization.
Read the full abstract ▾
All authors Xiang Chen, Yixin Ou, Quan Feng, Lei Li, Piji Li, Haibo Ye, Sheng-Jun Huang, Shuofei Qiao, Shumin Deng, Huajun Chen, Ningyu Zhang
Computation and Language
Yuxin Wang, Shicheng Fang, Bo Wang et al. (+4 more)
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for Large
Language Models (LLMs) to address knowledge-intensive queries requiring domain-specific or up-to-date information. To handle complex multi-hop questions that are challenging for single-step
retrieval, iterative RAG approaches incorporating reinforcement learning have been proposed. However, existing iterative RAG systems typically plan to decompose questions without leveraging information about the available
retrieval corpus, leading to inefficient
retrieval and
reasoning chains that cascade into suboptimal performance. In this paper, we introduce Early Knowledge
Alignment (EKA), a simple but effective module that aligns LLMs with
retrieval set before planning in iterative RAG systems with contextually relevant retrieved knowledge. Extensive experiments on six standard RAG
datasets demonstrate that by establishing a stronger
reasoning foundation, EKA significantly improves
retrieval precision, reduces cascading errors, and enhances both performance and efficiency. Our analysis from an
entropy perspective demonstrate that incorporating early knowledge reduces unnecessary exploration during the
reasoning process, enabling the model to focus more effectively on relevant information subsets. Moreover, EKA proves effective as a versatile, training-free inference strategy that scales seamlessly to large models. Generalization tests across diverse
datasets and
retrieval corpora confirm the robustness of our approach. Overall, EKA advances the state-of-the-art in iterative RAG systems while illuminating the critical interplay between structured
reasoning and efficient exploration in reinforcement learning-augmented frameworks. The code is released at \href{https://github.com/yxzwang/EarlyKnowledgeAlignment}{Github}.
Read the full abstract ▾
All authors Yuxin Wang, Shicheng Fang, Bo Wang, Qi Luo, Xuanjing Huang, Yining Zheng, Xipeng Qiu
Computation and Language
M$^3$KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
Hyeongcheol Park, Jiyoung Seo, Jaewon Mun et al. (+6 more)
Retrieval-Augmented Generation (RAG) has recently been extended to
multimodal settings, connecting
multimodal large
language models (MLLMs) with vast corpora of external knowledge such as
multimodal knowledge graphs (MMKGs). Despite their recent success,
multimodal RAG in the audio-visual domain remains challenging due to 1) limited modality coverage and multi-hop connectivity of existing MMKGs, and 2)
retrieval based solely on similarity in a shared
multimodal embedding space, which fails to filter out off-topic or redundant knowledge. To address these limitations, we propose M$^3$KG-RAG, a Multi-hop
Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving
reasoning depth and answer faithfulness in MLLMs. Specifically, we devise a lightweight multi-
agent pipeline to construct multi-hop MMKG (M$^3$KG), which contains context-enriched triplets of
multimodal entities, enabling modality-wise
retrieval based on input queries. Furthermore, we introduce GRASP (Grounded
Retrieval And Selective
Pruning), which ensures precise entity grounding to the query, evaluates answer-supporting relevance, and prunes redundant context to retain only knowledge essential for response generation. Extensive experiments across diverse
multimodal benchmarks demonstrate that M$^3$KG-RAG significantly enhances MLLMs'
multimodal reasoning and grounding over existing approaches. Project website: https://kuai-lab.github.io/cvpr2026m3kgrag/
Read the full abstract ▾
All authors Hyeongcheol Park, Jiyoung Seo, Jaewon Mun, Hogun Park, Wonmin Byeon, Sung June Kim, Hyeonsoo Im, JeungSub Lee, Sangpil Kim
Computation and Language
ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction
Aly Lidayan, Jakob Bjorner, Satvik Golechha et al. (+2 more)
As the time horizons of sequential decision-making tasks grow, keeping full interaction histories in model context becomes increasingly costly. Recent work reduces context lengths by instead conditioning decision-making
agents on recursively updated natural-
language summaries, which are concise and interpretable. However, they underperform
agents with access to the full context, suggesting that they fail to generate sufficient summaries. To address this we propose ABBEL, a recursive summarization framework that isolates and directly supervises each summary's information contents in the form of explicit natural-
language belief states. First, we analyze the belief states generated by frontier models under ABBEL across five domains, and verify that performance is often degraded due to omitting or incorrectly updating information. We also discover settings where models use
memory inefficiently by retaining extraneous information. We target these limitations by fine-tuning with two RL-based methods: belief grading, which reduces update errors by rewarding belief generations based on their information content, and peak belief penalties, which encourage compressing the beliefs with the greatest
memory footprints. We demonstrate that these methods significantly reduce the performance gap with full context models, and enable ABBEL to outperform prior
memory agent work by 40% while using 67% of the
memory. Our code is available at https://github.com/jakob-bjorner/optimal-explorer-dev
Read the full abstract ▾
All authors Aly Lidayan, Jakob Bjorner, Satvik Golechha, Kartik Goyal, Alane Suhr
Computation and Language
A Novel Graph-Sequence Learning Model for Inductive Text Classification
Zuo Wang and Ye Yuan
Text classification plays an important role in various downstream text-related tasks, such as sentiment analysis, fake news detection, and public opinion analysis. Recently, text classification based on Graph Neural Networks (GNNs) has made significant progress due to their strong capabilities of structural relationship learning. However, these approaches still face two major limitations. First, these approaches fail to fully consider the diverse structural information across word pairs, e.g., co-occurrence, syntax, and semantics. Furthermore, they neglect sequence information in the text graph structure information learning module and can not classify texts with new words and relations. In this paper, we propose a Novel Graph-Sequence Learning Model for Inductive Text Classification (TextGSL) to address the previously mentioned issues. More specifically, we construct a single text-level graph for all words in each text and establish different edge types based on the diverse relationships between word pairs. Building upon this, we design an adaptive multi-edge message-passing paradigm to aggregate diverse structural information between word pairs. Additionally, sequential information among text data can be captured by the proposed TextGSL through the incorporation of
Transformer layers. Therefore, TextGSL can learn more discriminative text representations. TextGSL has been comprehensively compared with several strong baselines. The experimental results on diverse benchmarking
datasets demonstrate that TextGSL outperforms these baselines in terms of accuracy.
Read the full abstract ▾
Computation and Language
Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents
Yiming Du, Baojun Wang, Yifan Xiang et al. (+11 more)
Temporal
reasoning over long, multi-session dialogues is a critical capability for conversational
agents. However, existing works and our pilot study have shown that as dialogue histories grow in length and accumulate noise, current long-context models struggle to accurately identify temporally pertinent information, significantly impairing
reasoning performance. To address this, we introduce
Memory-T1, a framework that learns a time-aware
memory selection policy using reinforcement learning (RL). It employs a coarse-to-fine strategy, first
pruning the dialogue history into a candidate set using temporal and relevance filters, followed by an RL
agent that selects the precise evidence sessions. The RL training is guided by a multi-level reward function optimizing (i) answer accuracy, (ii) evidence grounding, and (iii) temporal consistency. In particular, the temporal consistency reward provides a dense signal by evaluating
alignment with the query time scope at both the session-level (chronological proximity) and the utterance-level (chronological fidelity), enabling the
agent to resolve subtle chronological ambiguities. On the Time-Dialog
benchmark,
Memory-T1 boosts a 7B model to an overall score of 67.0%, establishing a new state-of-the-art performance for open-source models and outperforming a 14B baseline by 10.2%. Ablation studies show temporal consistency and evidence grounding rewards jointly contribute to a 15.0% performance gain. Moreover,
Memory-T1 maintains robustness up to 128k
tokens, where baseline models collapse, proving effectiveness against noise in extensive dialogue histories. The code and
datasets are publicly available at https://github.com/Elvin-Yiming-Du/
Memory-T1/
Read the full abstract ▾
All authors Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, Kam-Fai Wong
Computation and Language · Cross-listed from cs.AI
Reason2Decide: Rationale-Driven Multi-Task Learning
H M Quamran Hasan, Housam Khalifa Bashier, Jiayi Dai et al. (+2 more)
Despite the wide adoption of Large
Language Models (LLM)s, clinical decision support systems face a critical challenge: achieving high predictive accuracy while generating explanations aligned with the predictions. Current approaches suffer from exposure bias leading to misaligned explanations. We propose Reason2Decide, a two-stage training framework that addresses key challenges in self-rationalization, including exposure bias and task separation. In Stage-1, our model is trained on rationale generation, while in Stage-2, we jointly train on label prediction and rationale generation, applying scheduled sampling to gradually transition from conditioning on gold labels to model predictions. We evaluate Reason2Decide on three medical
datasets, including a proprietary triage
dataset and public biomedical QA
datasets. Across model sizes, Reason2Decide outperforms other fine-tuning baselines and some zero-shot LLMs in prediction (F1) and rationale fidelity (BERTScore, BLEU, LLM-as-a-Judge). In triage, Reason2Decide is rationale source-robust across LLM-generated, nurse-authored, and nurse-post-processed rationales. In our experiments, while using only LLM-generated rationales in Stage-1, Reason2Decide outperforms other fine-tuning variants. This indicates that LLM-generated rationales are suitable for pretraining models, reducing reliance on human annotations. Remarkably, Reason2Decide achieves these gains with models 40x smaller than contemporary foundation models, making clinical
reasoning more accessible for resource-constrained deployments while still providing explainable decision support.
Read the full abstract ▾
All authors H M Quamran Hasan, Housam Khalifa Bashier, Jiayi Dai, Mi-Young Kim, Randy Goebel
Computation and Language
Schoenfeld's Anatomy of Mathematical Reasoning by Language Models
Ming Li, Chenrui Fan, Yize Cheng et al. (+2 more)
Large
language models increasingly expose
reasoning traces, yet their underlying cognitive structure and steps remain difficult to identify and analyze beyond surface-level statistics. We adopt Schoenfeld's Episode Theory as an inductive, intermediate-scale lens and introduce ThinkARM (Anatomy of
Reasoning in Models), a scalable framework that explicitly abstracts
reasoning traces into functional
reasoning steps such as Analysis, Explore, Implement, Verify, etc. When applied to mathematical problem solving by diverse models, this abstraction reveals reproducible thinking dynamics and structural differences between
reasoning and non-
reasoning models, which are not apparent from
token-level views. We further present two diagnostic case studies showing that exploration functions as a critical branching step associated with correctness, and that efficiency-oriented methods selectively suppress evaluative feedback steps rather than uniformly shortening responses. Together, our results demonstrate that episode-level representations make
reasoning steps explicit, enabling systematic analysis of how
reasoning is structured, stabilized, and altered in modern
language models.
Read the full abstract ▾
All authors Ming Li, Chenrui Fan, Yize Cheng, Soheil Feizi, Tianyi Zhou
Computation and Language
Bias Beneath the Tone: Empirical Characterisation of Tone Bias in LLM-Driven UX Systems
Heet Bodara, Md Masum Mushfiq and Isma Farah Siddiqui
Large
Language Models are increasingly used in conversational systems such as digital personal assistants, shaping how people interact with technology through
language. While their responses often sound fluent and natural, they can also carry subtle tone biases such as sounding overly polite, cheerful, or cautious even when neutrality is expected. These tendencies can influence how users perceive trust, empathy, and
fairness in dialogue. In this study, we explore tone bias as a hidden behavioral trait of large
language models. The novelty of this research lies in the integration of controllable large
language model based dialogue synthesis with tone classification models, enabling robust and ethical
emotion recognition in personal assistant interactions. We created two synthetic dialogue
datasets, one generated from neutral
prompts and another explicitly guided to produce positive or negative tones. Surprisingly, even the neutral set showed consistent tonal skew, suggesting that bias may stem from the model's underlying conversational style. Using weak supervision through a pretrained DistilBERT model, we labeled tones and trained several classifiers to detect these patterns. Ensemble models achieved macro F1 scores up to 0.92, showing that tone bias is systematic, measurable, and relevant to designing fair and trustworthy conversational AI.
Read the full abstract ▾