Computation and Language
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Yun Hong, Yan Zhou and Yang Feng
Empathy is essential for fostering natural interactions in spoken dialogue systems, as it enables machines to recognize the emotional tone of human
speech and deliver empathetic responses. Recent research has made significant progress in developing empathetic spoken chatbots based on large
language models (LLMs). However, several challenges still exist when training such models, including reliance on costly empathetic
speech instruction data and a lack of emotional expressiveness in the generated
speech. Finetuning LLM with cross-modal empathetic instruction data may also lead to catastrophic forgetting and a degradation of its general capability. To address these challenges, we propose FreezeEmpath, an end-to-end empathetic spoken chatbot trained in a simple and efficient manner. The entire training process relies solely on existing
speech instruction data and
speech emotion recognition (SER) data, while keeping the LLM's parameters frozen. Experiments demonstrate that FreezeEmpath is able to generate emotionally expressive
speech and outperforms other empathetic models in empathetic dialogue, SER, and SpokenQA tasks, demonstrating the effectiveness of our training strategy.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations
Yunjia Xi, Menghui Zhu, Jianghao Lin et al. (+4 more)
Recently, large
language models (LLMs) have advanced recommendation systems (RSs), and recent works have begun to explore how to integrate LLMs into industrial RSs. While most approaches deploy LLMs offline to generate and pre-cache augmented representations for RSs, high-dimensional representations from LLMs introduce substantial storage and computational costs. Thus, it is crucial to compress LLM representations effectively. However, we identify a counterintuitive phenomenon during representation compression: Mid-layer Representation Advantage (MRA), where representations from middle layers of LLMs outperform those from final layers in recommendation tasks. This degraded final layer renders existing compression methods, which typically compress on the final layer, suboptimal. We interpret this based on modularity theory that LLMs develop spontaneous internal functional modularity and force the final layer to specialize in the proxy training task. Thus, we propose \underline{M}odul\underline{a}r \underline{R}epresentation \underline{C}ompression (MARC) to explicitly control the modularity of LLMs. First, Modular Adjustment explicitly introduces compression and task adaptation modules, enabling the LLM to operate strictly as a representation-learning module. Next, to ground each module to its specific task, Modular Task Decoupling uses information constraints and different network structures to decouple tasks. Extensive experiments validate that MARC addresses MRA and produces efficient representations. Notably, MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.
Read the full abstract ▾
All authors Yunjia Xi, Menghui Zhu, Jianghao Lin, Bo Chen, Ruiming Tang, Yong Yu, Weinan Zhang
Computation and Language
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Ziyang Liu
We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only
language model trained on 5B
tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 to 1727. A simple residual-axis training-time intervention – Depth Registers with a register-magnitude hinge loss (DR+sink) – reduces this to 119 (about 14x) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 PPL. The residual ~2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers (qkv, w1, w3) and block-internal generators (o_proj, w2). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2's bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ~0.28 nats across three matched checkpoints with Delta-remove(w2) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot – adding online per-head value Hadamard plus online w2-input rotation – does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-
token, single-seed setting, and our experiments do not isolate the partition from the hinge.
Read the full abstract ▾
Computation and Language
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Weicheng Lin, Yi Zhang, Jiawei Dang et al. (+1 more)
Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large
language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA $A$
matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the $A$
matrix is frozen, and only the $B$
matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural
language understanding, commonsense
reasoning, math
reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.
Read the full abstract ▾
All authors Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang
Computation and Language
Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents
Akriti Jain, Anish Mulay, Divyansh Verma et al. (+3 more)
Decision-making is a cognitively intensive task that requires synthesizing relevant information from multiple unstructured sources, weighing competing factors, and incorporating subjective user preferences. Existing methods, including large
language models and traditional decision-support systems, fall short: they often overwhelm users with information or fail to capture nuanced preferences accurately. We present Decisive, an interactive decision-making framework that combines document-grounded
reasoning with
Bayesian preference inference. Our approach grounds decisions in an objective option-scoring
matrix extracted from source documents, while actively learning a user's latent preference vector through targeted elicitation. Users answer pairwise tradeoff questions adaptively selected to maximize information gain over the final decision. This process converges efficiently, minimizing user effort while ensuring recommendations remain transparent and personalized. Through extensive experiments, we demonstrate that our approach significantly outperforms both general-purpose LLMs and existing decision-making frameworks achieving up to 20% improvement in decision accuracy over strong baselines across domains.
Read the full abstract ▾
All authors Akriti Jain, Anish Mulay, Divyansh Verma, Aishani Pandey, Pritika Ramu, Aparna Garimella
Computation and Language
Retrieval-Augmented Multimodal Model for Fake News Detection
Yiheng Li, Weihai Lu, Hanyi Yu et al. (+1 more)
In recent years,
multimodal multidomain fake news detection has garnered increasing
attention. Nevertheless, this direction presents two significant challenges: (1) Failure to Capture Cross-Instance Narrative Consistency: existing models usually evaluate each news in isolation, fail to capture cross-instance narrative consistency, and thus struggle to address the spread of cluster based fake news driven by social media; (2) Lack of Domain Specific Knowledge for
Reasoning: conventional models, which rely solely on knowledge encoded in their parameters during training, struggle to generalize to new or data-scarce domains (e.g., emerging events or niche topics). To tackle these challenges, we introduce
Retrieval-Augmented
Multimodal Model for Fake News Detection (RAMM). First, RAMM employs a
Multimodal Large
Language Model (MLLM) as its backbone to capture cross-modal semantic information from news samples. Second, RAMM incorporates an Abstract Narrative
Alignment Module. This component adaptively extracts abstract narrative consistency from diverse instances across distinct domains, aggregates relevant knowledge, and thereby enables the modeling of high-level narrative information. Finally, RAMM introduces a Semantic Representation
Alignment Module, which aligns the model's decision-making paradigm with that of humans - specifically, it shifts the model's
reasoning process from direct inference on
multimodal features to an instance-based analogical
reasoning process. Extensive experimental results on three public
datasets validate the efficacy of our proposed approach. Our code is available at the following link: https://github.com/li-yiheng/RAMM
Read the full abstract ▾
All authors Yiheng Li, Weihai Lu, Hanyi Yu, Yue Wang
Computation and Language
Santosh Kesiraju, Bolaji Yusuf, Šimon Sedláček et al. (+2 more)
This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence
embedding spaces. We train FLiP models to recover the lexical content from
multilingual (LaBSE),
multimodal (SONAR) and API-based (Gemini) sentence
embedding spaces in several high- and mid-resource
languages. We show that FLiP can recall more than 75% of lexical content from the
embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and
language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.
Read the full abstract ▾
All authors Santosh Kesiraju, Bolaji Yusuf, Šimon Sedláček, Oldřich Plchot, Petr Schwarz
Computation and Language
Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion
Chen Zhang, Jiuheng Lin, Zhiyuan Liao et al. (+1 more)
Adapting large
language models (LLMs) to low-resource
languages (LRLs) is constrained by the scarcity of task data and computational resources. Although Proxy Tuning offers a logit-level strategy for introducing scaling effects, it often fails in LRL settings because the large model's weak LRL competence might overwhelm the knowledge of specialized smaller models. We thus propose TriMix, a test-time logit fusion framework that dynamically balances capabilities from three different sources: LRL competence from a continually pretrained small model, task competence from high-resource
language instruction tuning, and the scaling benefits of large models. It is data- and compute-efficient, requiring no LRL task annotations, and only continual pretraining on a small model. Experiments across four model families and eight LRLs show that TriMix consistently outperforms single-model baselines and Proxy Tuning. Our analysis reveals that prioritizing the small LRL-specialized model's logits is crucial for success, challenging the prevalent large-model-dominant assumption.
Read the full abstract ▾
All authors Chen Zhang, Jiuheng Lin, Zhiyuan Liao, Yansong Feng
Computation and Language · Cross-listed from eess.AS
NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
Yuan Xie, Jiaqi Song, Guang Qiu et al. (+9 more)
Integrating large
language models (LLMs) into automatic
speech recognition (ASR) has become a mainstream paradigm in recent years. Although existing LLM-based ASR models demonstrate impressive performance on public
benchmarks, their training remains predominantly data-driven, leaving key practical challenges insufficiently addressed – particularly limited downward scalability in resource-constrained deployments and
hallucinations under acoustically challenging conditions. To address these issues, we present NIM4-ASR, a production-oriented LLM-based ASR framework optimized for both efficiency and robustness. Grounded in a principled delineation of functional roles between the encoder and the LLM, we redesign the multi-stage training paradigm to align each module with its intended capability boundary. Specifically, we reformulate the pre-training architecture and objective to mitigate the modality gap and improve parameter efficiency; introduce an iterative asynchronous SFT stage to preserve acoustic fidelity and constrain representation drift; and design an ASR-specialized reinforcement learning stage to further enhance recognition quality and robustness. We additionally incorporate a suite of production-oriented optimizations, including robustness under noisy and silent conditions, real-time streaming inference, and hotword customization via
retrieval-augmented generation (RAG). Experiments show that NIM4-ASR achieves state-of-the-art performance on multiple public
benchmarks with merely 2.3B parameters, while substantially outperforming larger-scale competitors on internal
benchmarks – particularly in entity-intensive real-world scenarios. NIM4-ASR further supports million-scale hotword customization via RAG with sub-millisecond
retrieval latency, enabling efficient adaptation to emerging entities and personalized user requirements.
Read the full abstract ▾
All authors Yuan Xie, Jiaqi Song, Guang Qiu, Xianliang Wang, Kai Qiao, Junfeng Yuan, Shengqing Liu, Yi Zhang, Bowen Chen, Ming Lei, Jie Gao, Jie Wu
Computation and Language
Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts
Run Xu, Lu Li, Rongzhao Zhang et al. (+1 more)
Recent
multimodal large
language models have shown promising ability in generating humorous captions for images, yet they still lack stable control over explicit cultural context, making it difficult to jointly maintain image relevance, contextual appropriateness, and humor quality under a specified cultural background. To address this limitation, we introduce a new
multimodal generation task, culture-aware humorous captioning, which requires a model to generate a humorous caption conditioned on both an input image and a target cultural context. Captions generated under different cultural contexts are not expected to share the same surface form, but should remain grounded in similar visual situations or humorous rationales.To support this task, we establish a six-dimensional evaluation framework covering image relevance, contextual fit, semantic richness, reasonableness, humor, and creativity. We further propose a staged
alignment framework that first initializes the model with high-resource supervision under the Western cultural context, then performs multi-dimensional preference
alignment via judge-based GRPO with a Degradation-aware Prototype Repulsion Constraint to mitigate reward hacking in open-ended generation, and finally adapts the model to the Eastern cultural context with a small amount of supervision. Experimental results show that our method achieves stronger overall performance under the proposed evaluation framework, with particularly large gains in contextual fit and a better balance between image relevance and humor under cultural constraints.
Read the full abstract ▾
All authors Run Xu, Lu Li, Rongzhao Zhang, Jie Xu
Computation and Language
Mix and Match: Context Pairing for Scalable Topic-Controlled Educational Summarisation
Nathikan Yodthapa, Thanapong Intharah and Sahan Bulathwela
Topic-controlled summarisation enables users to generate summaries focused on specific aspects of source documents. This paper investigates a data augmentation strategy for training small
language models (sLMs) to perform topic-controlled summarisation. We propose a pairwise data augmentation method that combines contexts from different documents to create contrastive training examples, enabling models to learn the relationship between topics and summaries more effectively. Using the SciTLDR
dataset enriched with Wikipedia-derived topics, we systematically evaluate how augmentation scale affects model performance. Results show consistent improvements in win rate and semantic
alignment as the augmentation scale increases, while the amount of real training data remains fixed. Consequently, a T5-base model trained with our augmentation approach achieves competitive performance relative to larger models, despite using significantly fewer parameters and substantially fewer real training examples.
Read the full abstract ▾
Computation and Language
Modeling Human Perspectives with Socio-Demographic Representations
Leixin Zhang and Cagri Coltekin
Humans often hold different perspectives on the same issues. In many NLP tasks, annotation disagreement can reflect valid subjective perspectives. Modeling annotator perspectives and understanding their relationship with other human factors, such as socio-demographic attributes, have received increasing
attention. Prior work typically focuses on single demographic factors or limited combinations. However, in real-world settings, annotator perspectives are shaped by complex social contexts, and finer-grained socio-demographic attributes can better explain human perspectives. In this work, we propose Socio-Contrastive Learning, a method that jointly models annotator perspectives while learning socio-demographic representations. Our method provides an effective approach for the fusion of socio-demographic features and textual representations to predict annotator perspectives, outperforming standard concatenation-based methods. The learned representations further enable analysis and visualization of how demographic factors relate to variation in annotator perspectives. Our code is available at GitHub: https://github.com/Leixin-Zhang/Socio_Contrastive_Learning
Read the full abstract ▾
Computation and Language
JudgeMeNot: Personalizing Large Language Models to Emulate Judicial Reasoning in Hebrew
Itay Razumenko, Arnon Sturm and Nir Grinberg
Despite significant advances in large
language models, personalizing them for individual decision-makers remains an open problem. Here, we introduce a synthetic-organic supervision pipeline that transforms raw judicial decisions into instruction-tuning data, enabling parameter-efficient fine-tuning of personalized models for individual judges in low-resource settings. We compare our approach to state-of-the-art personalization techniques across three different tasks and settings. The results show that Causal
Language Modeling followed by synthetically generated instruction-tuning significantly outperforms all other baselines, providing significant improvements across lexical, stylistic, and semantic similarity. Notably, our model-generated outputs are indistinguishable from the
reasoning of human judges, highlighting the viability of efficient personalization, even in low-resource settings.
Read the full abstract ▾
Computation and Language
SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation
Muxin Pu, Xiao-Ming Wu, Mei Kuan Lim et al. (+3 more)
We present SignDPO, a novel multi-level Direct Preference Optimisation (DPO) framework designed to enhance the
alignment of skeleton-based Sign
Language Translation. While current skeleton-based models have made significant progress using Maximum Likelihood Estimation, they are primarily constrained by an imitation-based paradigm that lacks discriminative sensitivity to the fine-grained spatio-temporal nuances of sign
language, often leading to semantic drift. To address this, SignDPO shifts the optimisation goal from simple sequence mimicry to structured preference
alignment across spatial, temporal, and linguistic dimensions. Our framework involves three key designs. First, we introduce a hierarchical perturbation strategy to construct spatial and temporal non-preferred samples at both global and local granularities automatically. Second, we propose a self-guiding mechanism that leverages decoder cross-
attention scores to identify and perturb semantically salient skeletal regions, forcing the model to distinguish genuine sign signals from structural distortions. Third, we establish an automated
language-level preference generator by fine-tuning a dedicated perturbation model, capturing complex output-level failure modes without manual annotation. Extensive experiments on three widely adopted
benchmarks, CSL-Daily, How2Sign, and OpenASL, demonstrate that SignDPO consistently outperforms state-of-the-art gloss-free methods and even rivals established gloss-based ones. Our results suggest that multi-level preference
alignment is a powerful paradigm for bridging the gap between high-
entropy skeletal trajectories and discrete linguistic semantics.
Read the full abstract ▾
All authors Muxin Pu, Xiao-Ming Wu, Mei Kuan Lim, Chun Yong Chong, Wei Li, Chen Change Loy
Computation and Language
How Creative Are Large Language Models in Generating Molecules?
Wen Tao, Yiwei Wang, Peng Zhou et al. (+6 more)
Molecule generation requires satisfying multiple chemical and biological constraints while searching a large and structured chemical space. This makes it a non-binary problem, where effective models must identify non-obvious solutions under constraints while maintaining exploration to improve success by escaping local optima. From this perspective, creativity is a functional requirement in molecular generation rather than an aesthetic notion. Large
language models (LLMs) can generate molecular representations directly from natural
language prompts, but it remains unclear what type of creativity they exhibit in this setting and how it should be evaluated. In this work, we study the creative behavior of LLMs in molecular generation through a systematic empirical evaluation across physicochemical, ADMET, and biological activity tasks. We characterize creativity along two complementary dimensions, convergent creativity and divergent creativity, and analyze how different factors shape these behaviors. Our results indicate that LLMs exhibit distinct patterns of creative behavior in molecule generation, such as an increase in constraint satisfaction when additional constraints are imposed. Overall, our work is the first to reframe the abilities required for molecule generation as creativity, providing a systematic understanding of creativity in LLM-based molecular generation and clarifying the appropriate use of LLMs in molecular discovery pipelines.
Read the full abstract ▾
All authors Wen Tao, Yiwei Wang, Peng Zhou, Bryan Hooi, Wanlong Fang, Tianle Zhang, Xiao Luo, Yuansheng Liu, Alvin Chan
Computation and Language · Cross-listed from cs.SE
CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora
Shangyu Li, Juyong Jiang, Meibo Ren et al. (+7 more)
Transpilation, or code translation, aims to convert source code from one programming
language (PL) to another. It is beneficial for many downstream applications, from modernizing large legacy codebases to augmenting data for low-resource PLs. Recent large
language model (LLM)-based approaches have demonstrated immense potential for code translation. Among these approaches, training-based methods are particularly important because LLMs currently do not effectively adapt to domain-specific settings that suffer from a lack of knowledge without targeted training. This limitation is evident in transpilation tasks involving low-resource PLs. However, existing training-based approaches rely on a pairwise transpilation paradigm, making it impractical to support a diverse range of PLs. This limitation is particularly prominent for low-resource PLs due to a scarcity of training data. Furthermore, these methods suffer from suboptimal reinforcement learning (RL) reward formulations. To address these limitations, we propose CodePivot, a training framework that leverages Python as an intermediate representation (IR), augmented by a novel RL reward mechanism, Aggressive-Partial-Functional reward, to bootstrap the model's
multilingual transpilation ability without requiring parallel corpora. Experiments involving 10 PLs show that the resulting 7B model, trained on Python-to-Others tasks, consistently improves performance across both general and low-resource PL-related transpilation tasks. It outperforms substantially larger mainstream models with hundreds of billions more parameters, such as Deepseek-R1 and Qwen3-235B-A22B-Instruct-2507, on Python-to-Others tasks and Others-to-All tasks, respectively. In addition, it outperforms its counterpart trained directly on Any-to-Any tasks on general transpilation tasks. The code and data are available at https://github.com/lishangyu-hkust/CodePivot.
Read the full abstract ▾
All authors Shangyu Li, Juyong Jiang, Meibo Ren, Sizhe Zhong, Huiri Tan, Yunhao Gou, Xu Han, Chun Yong Chong, Yun Peng, Jiasi Shen
Computation and Language · Cross-listed from cs.MA
Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation
Nuo Chen, Yicheng Tong, Yuzhe Yang et al. (+5 more)
Multi-
agent systems (MAS) are increasingly used for open-ended idea generation, driven by the expectation that collective interaction will broaden the exploration diversity. However, when and why such collaboration truly expands the solution space remains unclear. We present a systematic empirical study of diversity in MAS-based ideation across three bottom-up levels: model intelligence,
agent cognition, and system dynamics. At the model level, we identify a compute efficiency paradox, where stronger, highly aligned models yield diminishing marginal diversity despite higher per-sample quality. At the cognition level, authority-driven dynamics suppress semantic diversity compared to junior-dominated groups. At the system level, group-size scaling yields diminishing returns and dense communication
topologies accelerate premature convergence. We characterize these outcomes as collective failures emerging from structural coupling, a process where interaction inadvertently contracts
agent exploration and triggers diversity collapse. Our analysis shows that this collapse arises primarily from the interaction structure rather than inherent model insufficiency, highlighting the importance of preserving independence and disagreement when designing MAS for creative tasks. Our code is available at https://github.com/Xtra-Computing/MAS_Diversity.
Read the full abstract ▾
All authors Nuo Chen, Yicheng Tong, Yuzhe Yang, Yufei He, Xueyi Zhang, Qingyun Zou, Qian Wang, Bingsheng He
Computation and Language
Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design
Xinyao Zhang, Nicole Sonne Heckmann, Manuela Del Castillo Suero et al. (+2 more)
Background: The potential of large
language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018-2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping, although LTM provided the most consistent improvements in
reasoning stability. Conclusion: Off-the-shelf general-purpose LLMs currently offer superior support for pharmacoepidemiologic design compared to biomedical LLMs.
Prompt strategy strongly influenced LLM performance.
Read the full abstract ▾
All authors Xinyao Zhang, Nicole Sonne Heckmann, Manuela Del Castillo Suero, Francesco Paolo Speca, Maurizio Sessa
Computation and Language · Cross-listed from cs.CV
Yu Zhang, Chuyang Sun, Kehai Chen et al. (+3 more)
Large
Vision-
Language Models (LVLMs) still struggle with
vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on large-scale annotated data for fine-tuning, which incurs massive computational overhead, or employ static post-hoc strategies that overlook the dynamic nature of
hallucination emergence. To address these, we introduce a new self-rewarding framework, enabling dynamic
hallucination mitigation at inference time without external supervision. On the empirical side, we reveal that visual
hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase. Drawing on these insights, we propose PSRD (\textbf{Phase-wise Self-Reward Decoding) for online
hallucination correction guided by phase-wise self-reward signals. To reduce the cost of repeated self-evaluation during decoding, we distill the
hallucination guidance signal from LVLMs into a lightweight reward model. The reward model subsequently provides on-the-fly guidance for targeted intervention during the decoding process, enabling precise
hallucination suppression. The proposed PSRD significantly reduces the
hallucination rate of LLaVA-1.5-7B by 50.0% and consistently outperforms existing post-hoc methods across five
hallucination evaluation
benchmarks for four LVLMs. Further analysis confirms that PSRD effectively mitigates
hallucination propagation and achieves a highly controllable trade-off between strong performance and inference efficiency.
Read the full abstract ▾
All authors Yu Zhang, Chuyang Sun, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang
Computation and Language
ltzGLUE: Luxembourgish General Language Understanding Evaluation
Alistair Plum, Felicia Körner, Anne-Marie Lutgen et al. (+8 more)
This paper presents ltzGLUE, the first Natural
Language Understanding (NLU)
benchmark for Luxembourgish (LTZ) based on the popular GLUE
benchmark for English. Although NLU tasks are available for many European
languages nowadays, LTZ is one of the official national
languages that is often overlooked. We construct new tasks and reuse existing ones to introduce the first official NLU
benchmark and accompanying evaluation of encoder models for the
language. Our tasks include common natural
language processing tasks in binary and multi-class classification settings, including named entity recognition, topic classification, and intent classification. We evaluate various pre-trained
language models for LTZ to present an overview of the current capabilities of these models on the LTZ
language.
Read the full abstract ▾
All authors Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cédric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
Computation and Language
Modeling Multiple Support Strategies within a Single Turn for Emotional Support Conversations
Jie Zhu, Huaixia Dou, Junhui Li et al. (+5 more)
Emotional Support Conversation (ESC) aims to assist individuals experiencing distress by generating empathetic and supportive dialogue. While prior work typically assumes that each supporter turn corresponds to a single strategy, real-world supportive communication often involves multiple strategies within a single utterance. In this paper, we revisit the ESC task by formulating it as multi-strategy utterance generation, where each utterance may contain one or more strategy-response pairs. We propose two generation methods: All-in-One, which predicts all strategy-response pairs in a single decoding step, and One-by-One, which iteratively generates strategy-response pairs until completion. Both methods are further enhanced with cognitive
reasoning guided by reinforcement learning to improve strategy selection and response composition. We evaluate our models on the ESConv
dataset under both utterance-level and dialogue-level settings. Experimental results show that our methods effectively model multi-strategy utterances and lead to improved supportive quality and dialogue success. To our knowledge, this work provides the first systematic empirical evidence that allowing multiple support strategies within a single utterance is both feasible and beneficial for emotional support conversations. All code and data will be publicly available at https://github.com/aliyun/qwen-dianjin.
Read the full abstract ▾
All authors Jie Zhu, Huaixia Dou, Junhui Li, Lifan Guo, Feng Chen, Jinsong Su, Chi Zhang, Fang Kong
Computation and Language · Cross-listed from cs.AI
From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?
Hasan Amin, Harry Yizhou Tian, Xiaoni Duan et al. (+3 more)
Although large
language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful
estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as
estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline
estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.
Read the full abstract ▾
All authors Hasan Amin, Harry Yizhou Tian, Xiaoni Duan, Chien-Ju Ho, Rajiv Khanna, Ming Yin
Computation and Language
Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards
Raffaele Pisano and Roberto Navigli
Process Reward Models (PRMs) have emerged as a powerful tool for providing step-level feedback when evaluating the
reasoning of Large
Language Models (LLMs), which frequently produce chains of thought (CoTs) containing errors even when the final answer is correct. However, existing PRM
datasets remain expensive to construct, prone to annotation errors, and predominantly limited to the mathematical domain. This work introduces a novel and scalable approach to PRM
dataset generation based on planning logical problems expressed in the Planning Domain Definition
Language (PDDL). Using this method, we generate a corpus of approximately one million
reasoning steps across various PDDL domains and use it to train PRMs. Experimental results show that augmenting widely-used PRM training
datasets with PDDL-derived data yields substantial improvements in both mathematical and non-mathematical
reasoning, as demonstrated across multiple
benchmarks. These findings indicate that planning problems constitute a scalable and effective resource for generating robust, precise, and fine-grained training data for PRMs, going beyond the classical mathematical sources that dominate this field.
Read the full abstract ▾
Computation and Language
ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering
Yindong Zhang, Wenmian Yang, Yiquan Zhang et al. (+1 more)
Developing
agents capable of navigating fragmented, multi-source information remains challenging, primarily due to the scarcity of
benchmarks reflecting hybrid workflows combining database querying with external APIs. To bridge this gap, we introduce ReCoQA, a large-scale
benchmark of 29,270 real-estate instances featuring machine-verifiable supervision for intermediate steps, including structured intent labels, SQL queries, and API calls. Complementarily, we propose HIRE-
Agent, a hierarchical framework instantiating an understand-plan-execute architecture as a strong baseline. By orchestrating a Front-end parser, a planning Supervisor, and execution Specialists, HIRE-
Agent effectively integrates heterogeneous evidence. Extensive experiments demonstrate that HIRE-
Agent constitutes a strong baseline and substantiates the necessity of hierarchical collaboration for complex, real-world
reasoning tasks.
Read the full abstract ▾
All authors Yindong Zhang, Wenmian Yang, Yiquan Zhang, Weijia Jia
Computation and Language
A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents
Bao Gia Doan, Aditya Joshi, Pantelis Elinas et al. (+4 more)
RAG-based question-answering (QA) in specialist domains faces a cold-start problem: lack of evaluative
benchmarks and absence of labeled data for post-training. We present DoRA (Domain-oriented RAG Assessment), a novel
benchmark construction and evaluation framework using only a small set of specialist domain documents. DoRA systematically generates synthetic QA training and evaluation
datasets with auditable evidence across five domain-specific intents. To mitigate same-pipeline circularity, DoRA's training and test splits use different LLM families (Claude Sonnet for training; GPT-4o for test) drawn from disjoint seed-document corpora. Instantiated on 40 defense-related documents (written in English), DoRA yields ~6.6K curated instances. Compared against 8 LLM baselines over a
benchmark of 1,259 samples, a LoRA-adapted Llama3.1-8B trained on the synthetic training set consistently improves performance over 6 coverage and faithfulness metrics, especially reducing
hallucination by more than half under the default GTE
retrieval setting, with gains persisting across alternative retrievers and prompting-based baselines. Defense-domain expertise is incorporated in three stages of our evaluation: (a) determining the quality of the synthetic QA generated by DoRA, (b) ascertaining the reliability of LLM-as-judge scores, and (c) evaluating the generalization of the QA pipeline on completely human-written QA examples. We position DoRA as a practical framework for specialist-domain RAG under domain shift, with defense as a high-stakes case study.
Read the full abstract ▾
All authors Bao Gia Doan, Aditya Joshi, Pantelis Elinas, Aarya Bodhankar, Oscar Leslie, Tom Marchant, Flora Salim
Computation and Language · Cross-listed from cs.CV
From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
Qidong Wang, Junjie Hu and Ming Jiang
Recent work has increasingly explored
neuron-level interpretation in
vision-
language models (VLMs) to identify
neurons critical to final predictions. However, existing
neuron analyses generally focus on single tasks, limiting the comparability of
neuron importance across tasks. Moreover, ranking strategies tend to score
neurons in isolation, overlooking how task-dependent information pathways shape the write-in effects of feed-forward network (FFN)
neurons. This oversight can exacerbate
neuron polysemanticity in multi-task settings, introducing noise into the identification and intervention of task-critical
neurons. In this study, we propose HONES (Head-Oriented
Neuron Explanation & Steering), a gradient-free framework for task-aware
neuron attribution and steering in multi-task VLMs. HONES ranks FFN
neurons by their causal write-in contributions conditioned on task-relevant
attention heads, and further modulates salient
neurons via lightweight scaling. Experiments on four diverse
multimodal tasks and two popular VLMs show that HONES outperforms existing methods in identifying task-critical
neurons and improves model performance after steering. Our source code is released at: https://github.com/petergit1/HONES.
Read the full abstract ▾
Computation and Language
Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?
H S V N S Kowndinya Renduchintala and Sumit Bhatia
Large
Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence: while they learn some linguistic phenomena with near-perfect mastery, they often perform below chance on others, even after training on trillions of
tokens. In this work, we investigate whether these failures stem from inherent architectural limitations or simply the scarcity of these specific grammatical constructions in web-scale corpora. We pre-train simple GPT-2 Small (124M) models on a 100M-
token random sample of the FineWeb corpus and intervene by injecting a minimal amount (1%) of synthetic data targeting specific linguistic phenomena. We find that this targeted intervention substantially improves model performance in 8 out of the 9 worst-performing BLiMP paradigms - notably the accuracy on a specific paradigm, only_npi_scope, surges from 20.9% to 69.4%. Furthermore, we observe that these interventions generally preserve or slightly improve aggregate performance. However, while we also identify a resistant phenomenon, principle_A_c_command, whose performance remains below chance even after our data augmentation, our findings do serve as an optimistic existence proof that even small
language models can substantially improve on those linguistic phenomena on which models typically perform poorly, provided the pre-training data contains sufficient exposure to them. This suggests that efforts towards human-scale
language modeling may benefit greatly by focusing on data composition. The code to reproduce our results is open-sourced at https://github.com/kowndinya-renduchintala/heterogeneity-in-formal-linguistic-competence.
Read the full abstract ▾
Computation and Language
Automatic Slide Updating with User-Defined Dynamic Templates and Natural Language Instructions
Kun Zhou, Jiakai He, Wenmian Yang et al. (+3 more)
Presentation slides are a primary medium for data-driven reporting, yet keeping complex, analytics-style decks up to date remains labor-intensive. Existing automation methods mostly follow fixed template filling and cannot support dynamic updates for diverse, user-authored slide decks. We therefore define "Dynamic Slide Update via Natural
Language Instructions on User-provided Templates" and introduce DynaSlide, a large-scale
benchmark with 20,036 real-world instruction-execution triples (source slide, user instruction, target slide) grounded in a shared external database and built from business reporting slides under bring-your-own-template (BYO-template) conditions. To tackle this task, we propose SlideAgent, an
agent-based framework that combines
multimodal slide parsing, natural
language instruction grounding, and tool-augmented
reasoning for tables, charts, and textual conclusions. SlideAgent updates content while preserving layout and style, providing a strong reference baseline on DynaSlide. We further design end-to-end and component-level evaluation protocols that reveal key challenges and opportunities for future research. The
dataset and code are available at https://github.com/XiaoZhou2024/SlideAgent.
Read the full abstract ▾
All authors Kun Zhou, Jiakai He, Wenmian Yang, Zhensheng Wang, Yiquan Zhang, Weijia Jia
Computation and Language
Latent Preference Modeling for Multi-Session Personalized Tool Calling
Yejin Yoon, Minseo Kim and Taeuk Kim
Users often omit essential details in their requests to LLM-based
agents, resulting in under-specified inputs for tool use. This poses a fundamental challenge for tool-augmented
agents, as API execution typically requires complete arguments, highlighting the need for personalized tool calling. To study this problem, we present Multi-Session Personalized Tool Calling (MPT), a
benchmark comprising 4,695 instances over 459 multi-session interaction histories that cover three challenges: Preference Recall, Induction, and Transfer. We further propose PRefine, a test-time
memory method that maintains the user's latent preference as a textual hypothesis revised through a generate–verify–refine loop. Across five LLMs, existing
memory systems often fail to improve on full-history prompting; PRefine outperforms all baselines and alone surpasses it on Preference Transfer. These results indicate that
memory for personalized
agents must abstract behavior into preferences, rather than simply archive it.
Read the full abstract ▾
Computation and Language
GraSP: Graph-Structured Skill Compositions for LLM Agents
Tianle Xia, Lingxiang Hu, Yiding Sun et al. (+5 more)
Skill ecosystems for LLM
agents have matured rapidly, yet recent
benchmarks show that providing
agents with more skills does not monotonically improve performance – focused sets of 2-3 skills outperform comprehensive documentation, and excessive skills actually hurt. The bottleneck has shifted from skill availability to skill orchestration:
agents need not more skills, but a structural mechanism to select, compose, and execute them with explicit causal dependencies. We propose GraSP, the first executable skill graph architecture that introduces a compilation layer between skill
retrieval and execution. GraSP transforms flat skill sets into typed directed acyclic graphs (DAGs) with precondition-effect edges, executes them with node-level verification, and performs locality-bounded repair through five typed operators – reducing replanning from O(N) to O(d^h). Across ALFWorld, ScienceWorld, WebShop, and InterCode with eight LLM backbones, GraSP outperforms ReAct, Reflexion, ExpeL, and flat skill baselines in every configuration, improving reward by up to +19 points over the strongest baseline while cutting environment steps by up to 41%. GraSP's advantage grows with task complexity and is robust to both skill over-
retrieval and quality degradation, confirming that structured orchestration – not larger skill libraries – is the key to reliable
agent execution.
Read the full abstract ▾
All authors Tianle Xia, Lingxiang Hu, Yiding Sun, Ming Xu, Lan Xu, Siying Wang, Wei Xu, Jie Jiang