Computation and Language
Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search
Clemencia Siro, Zahra Abbasiantaeb, Yifei Yuan et al. (+2 more)
Conversational search systems increasingly employ clarifying questions to refine user queries and improve the search experience. Previous studies have demonstrated the usefulness of text-based clarifying questions in enhancing both
retrieval performance and user experience. While images have been shown to improve
retrieval performance in various contexts, their impact on user performance when incorporated into clarifying questions remains largely unexplored. We conduct a user study with 73 participants to investigate the role of images in conversational search, specifically examining their effects on two search-related tasks: (i) answering clarifying questions and (ii) query reformulation. We compare the effect of
multimodal and text-only clarifying questions in both tasks within a conversational search context from various perspectives. Our findings reveal that while participants showed a strong preference for
multimodal questions when answering clarifying questions, preferences were more balanced in the query reformulation task. The impact of images varied with both task type and user expertise. In answering clarifying questions, images helped maintain engagement across different expertise levels, while in query reformulation they led to more precise queries and improved
retrieval performance. Interestingly, for clarifying question answering, text-only setups demonstrated better user performance as they provided more comprehensive textual information in the absence of images. These results provide valuable insights for designing effective
multimodal conversational search systems, highlighting that the benefits of visual augmentation are task-dependent and should be strategically implemented based on the specific search context and user characteristics.
Read the full abstract ▾
All authors Clemencia Siro, Zahra Abbasiantaeb, Yifei Yuan, Mohammad Aliannejadi, Maarten de Rijke
Computation and Language
Challenges in Translating Technical Lectures: Insights from the NPTEL
Basudha Raje, Sadanand Venkatraman, Nandana TP et al. (+5 more)
This study examines the practical applications and methodological implications of Machine Translation in Indian
Languages, specifically Bangla, Malayalam, and Telugu, within emerging translation workflows and in relation to existing evaluation frameworks. The choice of
languages prioritized in this study is motivated by a triangulation of linguistic diversity, which illustrates the significance of
multilingual accommodation of educational technology under NEP 2020. This is further supported by the largest MOOC portal, i.e., NPTEL, which has served as a corpus to facilitate the arguments presented in this paper. The curation of a spontaneous
speech corpora that accounts for lucid delivery of technical concepts, considering the retention of suitable register and lexical choices are crucial in a diverse country like India. The findings of this study highlight metric-specific sensitivity and the challenges of morphologically rich and semantically compact features when tested against surface overlapping metrics.
Read the full abstract ▾
All authors Basudha Raje, Sadanand Venkatraman, Nandana TP, Soumyadeepa Das, Polkam Poojitha, M. Vijaykumar, Tanima Bagchi, Hema A. Murthy
Computation and Language · Cross-listed from cs.SD
Controllable Dysarthric Speech Synthesis with Patient-Specific Conditioning for Speaker-Diverse ASR Augmentation
Haoshen Wang, Xueli Zhong, Bingbing Lin et al. (+5 more)
Dysarthric
speech recognition is limited by high speaker variability and scarce labeled data. Existing synthesis methods often couple speaker identity with dysarthric articulation, reducing control over generated
speech. We propose a controllable dysarthric
speech synthesis framework for ASR augmentation with separate
prompt-derived timbre prefixes and learnable patient-specific pathology prefixes. Built on a pre-trained neural codec
language model adapted using LoRA, the framework combines both prefixes through additive conditioning. A dual-classifier objective with gradient reversal and voice-conversion-based counterfactual augmentation promotes factor separation, allowing learned pathology conditions to be paired with different target speakers, including healthy speakers. Experiments on TORGO show that generated
speech can partially replace real dysarthric training data and provides effective speaker-diverse augmentation when combined with real data. Objective ASR, perceptual, factor-separation, and phoneme-level analyses indicate preservation of target-speaker timbre and pathology-dependent patterns consistent with real dysarthric
speech.
Read the full abstract ▾
All authors Haoshen Wang, Xueli Zhong, Bingbing Lin, Jia Huang, Xingduo Pan, Shengxiang Liang, Nizhuan Wang, Wai Ting Siok
Computation and Language
Old wine in old glasses: Comparing computational and qualitative methods in identifying incivility on Persian Twitter during the #MahsaAmini movement
Hossein Kermani, Fatemeh Oudlajani, Pardis Yarahmadi et al. (+3 more)
This paper compares three approaches to detecting incivility in Persian tweets: human qualitative coding, supervised learning with ParsBERT, and large
language models (ChatGPT). Using 47,278 tweets from the #MahsaAmini movement in Iran, we evaluate the accuracy and efficiency of each method. ParsBERT substantially outperforms seven evaluated ChatGPT models in identifying hate
speech. We also find that ChatGPT struggles not only with subtle cases but also with explicitly uncivil content, and that
prompt language (English vs. Persian) does not meaningfully affect its outputs. The study provides a detailed comparison of these approaches and clarifies their strengths and limitations for analyzing hate
speech in a low-resource
language context.
Read the full abstract ▾
All authors Hossein Kermani, Fatemeh Oudlajani, Pardis Yarahmadi, Hamideh Mahdi Soltani, Mohammad Makki, Zahra HosseiniKhoo
Computation and Language
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
Clemencia Siro, Pourya Aliannejadi and Mohammad Aliannejadi
Large
language models (LLMs) are increasingly used as evaluators for natural
language generation, applying human-defined rubrics to assess system outputs. However, human rubrics are often static and misaligned with how models internally represent
language quality. We introduce GER-Eval (Generating Evaluation Rubrics for Evaluation) to investigate whether LLMs can design and apply their own evaluation rubrics. We evaluate the semantic coherence and scoring reliability of LLM-defined criteria and their
alignment with human criteria. LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them consistently within models, but their scoring reliability degrades in factual and knowledge-intensive settings. Closed-source models such as GPT-4o achieve higher agreement and cross-model generalization than open-weight models such as Llama. Our findings position evaluation as a learned linguistic capability of LLMs, consistent within models but fragmented across them, and call for new methods that jointly model human and LLM evaluative
language to improve reliability and interpretability.
Read the full abstract ▾
Computation and Language
Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models
Mingzi Cao, Xingwei Tan, Mahmud Elahi Akhter et al. (+4 more)
Deduction, induction, and abduction are fundamental
reasoning paradigms, core for human logical thinking. Although improving Large
Language Model (LLM)
reasoning has attracted significant research efforts, the extent to which the fundamental paradigms induce generalization has yet to be systematically explored. In this study, we shed light on how the interplay between these core paradigms influences LLMs'
reasoning behavior. To this end, we first collect a new
dataset of
reasoning trajectories from symbolic tasks, each targeting one of the three fundamental paradigms, to abstract from concrete world knowledge. Then, we investigate effective ways for inducing these skills into LLMs. We experiment with a battery of methods including simple fine-tuning, and more complex approaches to increase model depth, or transform a dense model to a mixture-of-experts. We comprehensively evaluate induced models on realistic out-of-domain tasks, that are entirely formulated in natural
language and contain real-world knowledge. Our results reveal that our approach yields strong generalizability with substantial performance gains (up to $14.60$) across realistic tasks.
Read the full abstract ▾
All authors Mingzi Cao, Xingwei Tan, Mahmud Elahi Akhter, Marco Valentino, Maria Liakata, Xi Wang, Nikolaos Aletras
Computation and Language · Cross-listed from cs.CY
We Should Separate Memorization from Copyright
Adi Haviv, Niva Elkin-Koren, Uri Hacohen et al. (+2 more)
The widespread use of foundation models has introduced a new risk factor of copyright issue. This issue is leading to an active, lively and on-going debate amongst the data-science community as well as amongst legal scholars. Where claims and results across both sides are often interpreted in different ways and leading to different implications. Our position is that much of the technical literature relies on traditional reconstruction techniques that are not designed for copyright analysis. As a result, memorization and copying have been conflated across both technical and legal communities and in multiple contexts. We argue that memorization, as commonly studied in data science, should not be equated with copying and should not be used as a proxy for copyright infringement. We distinguish technical signals that meaningfully indicate infringement risk from those that instead reflect lawful generalization or high-frequency content. Based on this analysis, we advocate for an output-level, risk-based evaluation process that aligns technical assessments with established copyright standards and provides a more principled foundation for research,
auditing, and policy.
Read the full abstract ▾
All authors Adi Haviv, Niva Elkin-Koren, Uri Hacohen, Roi Livni, Shay Moran
Computation and Language
Muhammad Naufil
Multilingual large
language models (LLMs) have gained significant popularity for their ability to process and generate text across multiple
languages. However, deploying these models in production can be inefficient when only a subset of the supported
languages is of interest. There has been some research conducted on identifying whether machine translation models have
language-specific or
language-agnostic heads, however no research has been conducted for
multilingual LLMs, to the best of our knowledge, that as we know are capable of performing diverse tasks beyond just translation. This paper explores whether
multilingual LLMs have specialized
language attention heads for each
language, and investigates the possibility of removing
language-specific heads for unwanted
languages without degrading performance in the targeted
languages. Our findings could inform more efficient deployment strategies for
multilingual LLMs, enabling reduced model complexity while maintaining high accuracy for targeted
languages.
Read the full abstract ▾
Computation and Language
VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modeling
Ziyang Cheng, Yuhao Wang, Heyang Liu et al. (+4 more)
Recent
Speech Large
Language Models~(LLMs) have achieved impressive capabilities in end-to-end
speech interaction. However, the prevailing autoregressive paradigm imposes strict serial constraints, limiting generation efficiency and introducing exposure bias. In this paper, we investigate Masked
Diffusion Modeling~(MDM) as a non-autoregressive paradigm for
speech LLMs and introduce VocalNet-MDM. To adapt MDM for streaming
speech interaction, we address two critical challenges: training-inference mismatch and iterative overhead. We propose Hierarchical Block-wise Masking to align training objectives with the progressive masked states encountered during block
diffusion decoding, and Iterative Self-
Distillation to compress multi-step refinement into fewer steps for low-latency inference. Trained on a limited scale of only 6K hours of
speech data, VocalNet-MDM achieves a 3.7$\times$–10$\times$ decoding speedup and reduces first-chunk latency by 34% compared to AR baselines. It maintains competitive recognition accuracy while achieving state-of-the-art text quality and
speech naturalness, demonstrating that MDM is a promising and scalable alternative for low-latency, efficient
speech LLMs.
Read the full abstract ▾
All authors Ziyang Cheng, Yuhao Wang, Heyang Liu, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang
Computation and Language
Beyond Scalar Scores: Reinforcement Learning for Error-Aware Quality Estimation of Machine Translation
Archchana Sindhujan, Girish A. Koushik, Shenbin Qian et al. (+2 more)
Quality Estimation (QE) aims to assess the quality of machine translation (MT) outputs without relying on reference translations, making it essential for real-world, large-scale MT evaluation. Large
Language Models (LLMs) have shown significant promise in advancing the field of quality estimation of machine translation. However, most of the QE approaches solely rely on scalar quality scores, offering no explicit information about the translation errors that should drive these judgments. Moreover, for low-resource
languages where annotated QE data is limited, existing approaches struggle to achieve reliable performance. To address these challenges, we introduce the first segment-level QE
dataset for English to Malayalam, a severely resource-scarce
language pair in the QE domain, comprising human-annotated Direct Assessment (DA) scores and Translation Quality Remarks (TQR), which are short, contextual, free-form annotator comments that describe translation errors. We further introduce ALOPE-RL, a policy-based reinforcement learning framework that trains efficient adapters based on policy rewards derived from DA score and TQR. Integrating error-aware rewards with ALOPE-RL, enables LLMs to reason about translation quality beyond numeric scores. Despite being trained on a small-scale QE
dataset, ALOPE-RL achieves state-of-the-art performance on English to Malayalam QE using compact LLMs (<=4B parameters}) fine-tuned with LoRA and 4-bit quantization, outperforming both larger LLM-based baselines and leading encoder-based QE models. Our results demonstrate that error-aware, policy-based learning can deliver strong QE performance under limited data and compute budgets. We release our
dataset, code, and trained models to support future research.
Read the full abstract ▾
All authors Archchana Sindhujan, Girish A. Koushik, Shenbin Qian, Diptesh Kanojia, Constantin Orăsan
Computation and Language · Cross-listed from cs.MA
ValueFlow: Measuring the Propagation of Value Perturbations in Multi-Agent LLM Systems
Jinnuo Liu, Chuke Liu and Hua Shen
Multi-
agent large
language model (LLM) systems increasingly consist of
agents that observe and respond to one another's outputs. While value
alignment is typically evaluated for isolated models, how value perturbations propagate through
agent interactions remains poorly understood. We present ValueFlow, a perturbation-based framework that measures value drift in multi-
agent systems via a 56-value valuation
dataset derived from the Schwartz Value Survey, with
agent value orientations scored using an LLM-as-a-judge protocol. ValueFlow decomposes value drift into
agent-level response behavior and system-level structural effects, captured by two metrics: e̱ta}-susceptibility, an
agent's sensitivity to perturbed peer value signals, and system susceptibility (SS), the effect of node-level perturbations on final system outputs.Experiments span across value dimensions, backbones, personas, and
topologies, showing that susceptibility varies sharply across values and is strongly shaped by interaction structure, indicating that value
alignment in multi-
agent systems is a system-level property, not just an
agent-level one. ValueFlow thus provides a principled basis for
auditing and mitigating value propagation in deployed multi-
agent systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches
Syed Mehtab Hussain Shah, Frank Hopfgartner and Arnim Bleier
Reproducing computational research is often assumed to be as simple as rerunning the original code with provided data. In practice, missing packages, fragile file paths, version conflicts, or incomplete logic frequently cause analyses to fail, even when materials are shared. This study investigates whether large
language models and AI
agents can automate the diagnosis and repair of such failures, making computational results easier to reproduce and verify. We evaluate this using a controlled reproducibility testbed built from five fully reproducible R-based social science studies. Realistic failures were injected, ranging from simple issues to complex missing logic, and two automated repair workflows were tested in clean Docker environments. The first workflow is
prompt-based, repeatedly querying
language models with structured
prompts of varying context, while the second uses
agent-based systems that inspect files, modify code, and rerun analyses autonomously. Across
prompt-based runs, reproduction success ranged from 31-79 percent, with performance strongly influenced by
prompt context and error complexity. Complex cases benefited most from additional context.
Agent-based workflows performed substantially better, with success rates of 69-96 percent across all complexity levels. These results suggest that automated workflows, especially
agent-based systems, can significantly reduce manual effort and improve reproduction success across diverse error types. Unlike prior
benchmarks, our testbed isolates post-publication repair under controlled failure modes, allowing direct comparison of
prompt-based and
agent-based approaches.
Read the full abstract ▾
Computation and Language
How Do Language Models Understand Tables? A Mechanistic Analysis of Cell Location
Xuanliang Zhang, Dingzirui Wang, Keyan Xu et al. (+2 more)
While Large
Language Models (LLMs) are increasingly deployed for table-related tasks, the internal mechanisms enabling them to process linearized two-dimensional structured tables remain opaque. In this work, we investigate the process of table understanding by dissecting the atomic task of
cell location. Through activation patching and complementary interpretability techniques, we delineate the table understanding mechanism into a sequential three-stage pipeline: Semantic Binding, Coordinate Localization, and Information Extraction. We demonstrate that models locate the target
cell via an ordinal mechanism that counts discrete delimiters to resolve coordinates. Furthermore, column indices are encoded within a linear subspace that allows for precise steering of model focus through vector arithmetic. Finally, we reveal that models generalize to multi-
cell location tasks by multiplexing the identical
attention heads identified during atomic location. Our findings provide a comprehensive explanation of table understanding within
Transformer architectures.
Read the full abstract ▾
All authors Xuanliang Zhang, Dingzirui Wang, Keyan Xu, Qingfu Zhu, Wanxiang Che
Computation and Language
GISA: A Benchmark for General Information-Seeking Assistant
Yutao Zhu, Xingshuo Zhang, Maosen Zhang et al. (+9 more)
The advancement of large
language models (LLMs) has significantly accelerated the development of search
agents capable of autonomously gathering information through multi-turn web interactions. Various
benchmarks have been proposed to evaluate such
agents. However, existing
benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these
benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a
benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep
reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.
Read the full abstract ▾
All authors Yutao Zhu, Xingshuo Zhang, Maosen Zhang, Jiajie Jin, Liancheng Zhang, Xiaoshuai Song, Kangzhi Zhao, Wencong Zeng, Ruiming Tang, Han Li, Ji-Rong Wen, Zhicheng Dou
Computation and Language · Cross-listed from cs.CV
Learning Self-Correction in Vision-Language Models via Rollout Augmentation
Yi Ding, Ziliang Qiu, Bolian Li et al. (+1 more)
Self-correction is essential for solving complex
reasoning problems in
vision-
language models (VLMs). However, existing reinforcement learning (RL) methods struggle to learn it, as effective self-correction behaviors emerge only rarely, making learning signals extremely sparse. To address this challenge, we propose correction-specific rollouts (Octopus), an RL rollout augmentation framework that synthesizes dense self-correction examples by recombining existing rollouts. This augmentation simultaneously improves sample efficiency due to rollout reuse and stabilizes RL optimization through balanced supervision. Furthermore, we introduce a response-masking strategy that decouples self-correction from direct
reasoning, avoiding signal conflicts and enabling both behaviors to be learned effectively. Building on this, we introduce Octopus-8B, a
reasoning VLM with controllable self-correction capability. Across 7
benchmarks, it achieves SoTA performance among open-source VLMs, outperforming the best RLVR baseline by 1.0 score while requiring only $0.72\times$ training time per step.
Read the full abstract ▾
All authors Yi Ding, Ziliang Qiu, Bolian Li, Ruqi Zhang
Computation and Language
Characterizing, Evaluating, and Optimizing Complex Reasoning
Haoran Zhang, Yafu Li, Zhi Wang et al. (+4 more)
Large
Reasoning Models (LRMs) increasingly rely on
reasoning traces with complex internal structures. However, existing work lacks a unified answer to three fundamental questions: (1) what defines high-quality
reasoning, (2) how to reliably evaluate long, implicitly structured
reasoning traces, and (3) how to use such evaluation signals for
reasoning optimization. To address these challenges, we provide a unified perspective. (1) We introduce the ME$^2$ principle to characterize
reasoning quality along macro- and micro-level concerning efficiency and effectiveness. (2) Built on this principle, we model
reasoning traces as directed acyclic graphs (DAGs) and develop a DAG-based pairwise evaluation method, capturing complex
reasoning structures. (3) Based on this method, we construct the TRM-Preference
dataset and train a Thinking Reward Model (TRM) to evaluate
reasoning quality at scale. Experiments show that thinking rewards serve as an effective optimization signal. At test time, selecting better
reasoning leads to better outcomes (up to 19.3% gain), and during RL training, thinking rewards enhance
reasoning and performance (up to 3.9% gain) across diverse tasks. Code and data are available at https://github.com/Simplified-
Reasoning/TRM.
Read the full abstract ▾
All authors Haoran Zhang, Yafu Li, Zhi Wang, Zhilin Wang, Shunkai Zhang, Xiaoye Qu, Yu Cheng
Computation and Language · Cross-listed from cs.LG
Beyond Correctness: Learning Robust Reasoning via Transfer
Hyunseok Lee, Soheil Abbasloo, Jihoon Tack et al. (+1 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM
reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the
reasoning process itself. We adopt a simple philosophical view, robust
reasoning should remain useful beyond the mind that produced it, and treat
reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial
reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce
reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable
reasoning and significantly more sample efficient.
Read the full abstract ▾
All authors Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, Jinwoo Shin
Computation and Language
Large Language Models and Impossible Language Acquisition: "False Promise" or an Overturn of our Current Perspective towards AI
Ziyan Wang and Longlong Ma
In Chomsky's provocative critique "The False Promise of CHATGPT," Large
Language Models (LLMs) are characterized as mere pattern predictors that do not acquire
languages via intrinsic causal and self-correction structures like humans, therefore are not able to distinguish impossible
languages. It stands as a representative in a fundamental challenge to the intellectual foundations of AI, for it integrally synthesizes major issues in methodologies within LLMs and possesses an iconic a priori rationalist perspective. We examine this famous critique from both the perspective in pre-existing literature of linguistics and psychology as well as a research based on an experiment inquiring into the capacity of learning both possible and impossible
languages among LLMs. We constructed a set of syntactically impossible
languages by applying certain transformations to English. These include reversing whole sentences, and adding negation based on word-count parity. Two rounds of controlled experiments were each conducted on GPT-2 small models and long short-term
memory (LSTM) models. Descriptive analysis of single-run training trajectories shows that GPT-2 small models exhibit lower final loss, faster convergence, and lower perplexity on natural
language compared to impossible
language conditions, with the reversed condition showing the largest departure (loss ratios up to 2.25 * natural). LSTM models, by contrast, show minimal differences across conditions. Given the single-run nature of our experiments (n=1 per condition), we report descriptive comparisons and caution that formal statistical inference is precluded. Based on theoretical analysis and descriptive empirical findings, we propose a new
vision within Chomsky's theory towards LLMs, and a shift of theoretical paradigm outside Chomsky, from his "rationalist-romantics" paradigm to functionalism and empiricism in LLMs research.
Read the full abstract ▾
Computation and Language
Prism: Spectral-Aware Block-Sparse Attention
Xinghao Wang, Pengyu Wang, Xiaoran Liu et al. (+4 more)
Block-sparse
attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained
attention as a proxy for block importance estimation, but often resort to expensive
token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained
attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional
Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full
attention while delivering up to $\mathbf{5.1\times}$ speedup.
Read the full abstract ▾
All authors Xinghao Wang, Pengyu Wang, Xiaoran Liu, Fangxu Liu, Jason Chu, Kai Song, Xipeng Qiu
Computation and Language
TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
Linye Wei, Zixiang Luo, Pingzhi Tang et al. (+1 more)
Diffusion large
language models (dLLMs) have recently gained significant
attention due to their inherent support for parallel decoding. Building on this paradigm, Mixture-of-Experts (MoE) dLLMs with autoregressive (AR) initialization have further demonstrated strong performance competitive with mainstream AR models. However, we identify a fundamental mismatch between MoE architectures and
diffusion-based decoding. Specifically, a large number of experts are activated at each denoising step, while only a small subset of
tokens is ultimately accepted, resulting in substantial inference overhead and limiting their deployment in latency-sensitive applications. In this work, we propose TEAM, a plug-and-play framework that accelerates MoE dLLMs by enabling more accepted
tokens with fewer activated experts. TEAM is motivated by the observation that expert routing decisions exhibit strong temporal consistency across denoising levels as well as spatial consistency across
token positions. Leveraging these properties, TEAM employs three complementary expert activation and decoding strategies, conservatively selecting necessary experts for decoded and masked
tokens and simultaneously performing aggressive speculative exploration across multiple candidates. Experimental results demonstrate that TEAM achieves up to 2.2x speedup over vanilla MoE dLLM, with negligible performance degradation. Code is released at https://github.com/PKU-SEC-Lab/TEAM-MoE-dLLM.
Read the full abstract ▾
All authors Linye Wei, Zixiang Luo, Pingzhi Tang, Meng Li
Computation and Language
Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning
Zhuoen Chen, Dongfang Li, Meishan Zhang et al. (+2 more)
Large
Language Models (LLMs) face significant challenges in long-context processing, including quadratic computational costs, information forgetting, and the context fragmentation inherent in
retrieval-augmented generation (RAG). We propose a cognitively inspired framework for efficient long-context inference based on chunk-wise compression and selective
memory recall, rather than processing all raw
tokens. The framework segments long inputs into chunks and encodes each chunk into compressed
memory representations using a learned compressor. A gating module dynamically selects relevant
memory blocks, which are then iteratively processed by a
reasoning module with an evolving working
memory to solve downstream tasks. The compressor and reasoner are jointly optimized via end-to-end reinforcement learning, while the gating module is trained separately as a classifier. Experimental results show that the proposed method achieves competitive accuracy on multi-hop
reasoning benchmarks such as RULER-HQA, extrapolates context length from 7K to 1.75M
tokens, and offers a favorable accuracy-efficiency trade-off compared to strong long-context baselines. In particular, it achieves up to a 2 times reduction in peak GPU
memory usage and a 6 times inference speedup over MemAgent.
Read the full abstract ▾
All authors Zhuoen Chen, Dongfang Li, Meishan Zhang, Baotian Hu, Min Zhang
Computation and Language · Cross-listed from cs.LG
Reinforcement Learning with Backtracking Feedback
Bilgehan Sel, Vaishakh Keshava, Phillip Wallis et al. (+3 more)
Addressing the critical need for robust safety in Large
Language Models (LLMs), particularly against
adversarial attacks and in-distribution errors, we introduce Reinforcement Learning with Backtracking Feedback (RLBF). This framework advances upon prior methods, such as BSAFE, by primarily leveraging a Reinforcement Learning (RL) stage where models learn to dynamically correct their own generation errors. Through RL with critic feedback on the model's live outputs, LLMs are trained to identify and recover from their actual, emergent safety violations by emitting an efficient "backtrack by x
tokens" signal, then continuing generation autoregressively. This RL process is crucial for instilling resilience against sophisticated
adversarial strategies, including middle filling, Greedy Coordinate Gradient (GCG) attacks, and decoding parameter manipulations. To further support the acquisition of this backtracking capability, we also propose an enhanced Supervised Fine-Tuning (SFT) data generation strategy (BSAFE+). This method improves upon previous data creation techniques by injecting violations into coherent, originally safe text, providing more effective initial training for the backtracking mechanism. Comprehensive empirical evaluations demonstrate that RLBF significantly reduces attack success rates across diverse
benchmarks and model scales, achieving superior safety outcomes while critically preserving foundational model utility.
Read the full abstract ▾
All authors Bilgehan Sel, Vaishakh Keshava, Phillip Wallis, Lukas Rutishauser, Ming Jin, Dingcheng Li
Computation and Language
ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts
Hung Quang Tran, Nam Tien Pham, Son T. Luu et al. (+1 more)
Emotion classification plays a significant role in
emotion prediction and harmful content detection. Recent advancements in NLP, particularly through large
language models (LLMs), have greatly improved outcomes in this field. This study introduces ViGoEmotions – a Vietnamese
emotion corpus comprising 20,664 social media comments in which each comment is classified into 27 fine-grained distinct
emotions. To evaluate the quality of the
dataset and its impact on
emotion classification, eight pre-trained
Transformer-based models were evaluated under three preprocessing strategies: preserving original emojis with rule-based normalization, converting emojis into textual descriptions, and applying ViSoLex, a model-based lexical normalization system. Results show that converting emojis into text often improves the performance of several BERT-based baselines, while preserving emojis yields the best results for ViSoBERT and CafeBERT. In contrast, removing emojis generally leads to lower performance. ViSoBERT achieved the highest Macro F1-score of 61.50% and Weighted F1-score of 63.26%. Strong performance was also observed from CafeBERT and PhoBERT. These findings highlight that while the proposed corpus can support diverse architectures effectively, preprocessing strategies and annotation quality remain key factors influencing downstream performance.
Read the full abstract ▾
All authors Hung Quang Tran, Nam Tien Pham, Son T. Luu, Kiet Van Nguyen
Computation and Language · Cross-listed from cs.AI
MemAdapter: Fast Alignment across Agent Memory Paradigms via Generative Subgraph Retrieval
Xin Zhang, Kailai Yang, Chenyue Li et al. (+4 more)
Memory mechanism is a core component of LLM-based
agents, enabling
reasoning and knowledge discovery over long-horizon contexts. Existing
agent memory systems are typically designed within isolated paradigms (e.g., explicit, parametric, or latent
memory) with tightly coupled
retrieval methods that hinder cross-paradigm generalization and fusion. In this work, we take a first step toward unifying heterogeneous
memory paradigms within a single
memory system. We propose MemAdapter, a
memory retrieval framework that enables fast
alignment across
agent memory paradigms. MemAdapter adopts a two-stage training strategy: (1) training a generative subgraph retriever from the unified
memory space, and (2) adapting the retriever to unseen
memory paradigms by training a lightweight
alignment module through contrastive learning. This design improves the flexibility for
memory retrieval and substantially reduces
alignment cost across paradigms. Comprehensive experiments on three public evaluation
benchmarks demonstrate that the generative subgraph retriever consistently outperforms five strong
agent memory systems across three
memory paradigms and
agent model scales. Notably, MemAdapter completes cross-paradigm
alignment within 13 minutes on a single GPU, achieving superior performance over original
memory retrievers with less than 5% of training compute. Furthermore, MemAdapter enables effective zero-shot fusion across
memory paradigms, highlighting its potential as a plug-and-play solution for
agent memory systems.
Read the full abstract ▾
All authors Xin Zhang, Kailai Yang, Chenyue Li, Hao Li, Qiyu Wei, Jun'ichi Tsujii, Sophia Ananiadou
Computation and Language
WorldTravel: A Realistic Multimodal Travel-Planning Benchmark with Tightly Coupled Constraints
Zexuan Wang, Chenghao Yang, Yingqi Que et al. (+18 more)
Real-world autonomous planning requires coordinating tightly coupled constraints where a single decision dictates the feasibility of all subsequent actions. However, existing
benchmarks predominantly feature loosely coupled constraints solvable through local greedy decisions and rely on idealized data, failing to capture the complexity of extracting parameters from dynamic web environments. We introduce WorldTravel, a
benchmark comprising 150 real-world travel scenarios across 5 cities that demand navigating an average of 15+ interdependent temporal and logical constraints. To evaluate
agents in realistic deployments, we develop WorldTravel-Webscape, a multi-modal environment featuring over 2,000 rendered webpages where
agents must perceive constraint parameters directly from visual layouts to inform their planning. Our evaluation of 10 frontier models reveals a significant performance collapse: even the state-of-the-art GPT-5.2 achieves only 32.67% feasibility in text-only settings, which plummets to 19.33% in multi-modal environments. We identify a critical Perception-Action Gap and a Planning Horizon threshold at approximately 10 constraints where model
reasoning consistently fails, suggesting that perception and
reasoning remain independent bottlenecks. These findings underscore the need for next-generation
agents that unify high-fidelity visual perception with long-horizon
reasoning to handle brittle real-world logistics.
Read the full abstract ▾
All authors Zexuan Wang, Chenghao Yang, Yingqi Que, Zhenzhu Yang, Huaqing Yuan, Yiwen Wang, Zhengxuan Jiang, Shengjie Fang, Zhenhe Wu, Zhaohui Wang, Zhixin Yao, Jiashuo Liu, Jincheng Ren, Yuzhen Li, Yang Yang, Jiaheng Liu, Jian Yang, Zaiyuan Wang, Ge Zhang, Zhoufutu Wen, Wenhao Huang
Computation and Language · Cross-listed from cs.LG
ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection
Debajyoti Datta, Trishala Neeraj, Bibek Paudel et al. (+2 more)
Long-context inference is constrained by KV-cache
memory, which grows linearly with sequence length; KV-cache compression therefore hinges on reliably selecting which past
tokens to retain. Most geometry-based eviction methods score keys by cosine similarity to a global centroid, but cosine is scale-invariant and can discard magnitude cues that distinguish semantically salient
tokens. We propose ManifoldKV, a training-free scorer that ranks
tokens by Euclidean distance to the key centroid, capturing both angular and radial deviations. On the RULER
benchmark, ManifoldKV achieves 95.7% accuracy at 4K-16K contexts with 20% compression; matching the best
geometric baseline while improving robustness in two regimes where cosine scoring fails. First, on multi-key
retrieval, ManifoldKV reduces directional collisions, achieving 92.4% vs KeyDiff's 77.0% (+15.4 points) on 3-key NIAH at 50% compression. Second, to address dilution and performance collapse of global centroids at 64K context, we introduce WindowedManifoldKV, which restores accuracy to 84.3% at 25% compression, a 49-point recovery over global L2 and +3.2 points over KeyDiff. The method requires only 3 lines of code and works across 4 architectures without tuning.
Read the full abstract ▾
All authors Debajyoti Datta, Trishala Neeraj, Bibek Paudel, Vyom Sharma, Subhabrata Mukherjee
Computation and Language
Cheng Yang, Chufan Shi, Bo Shui et al. (+7 more)
Unified
multimodal models (UMMs) aim to integrate
multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent textual and visual modalities are aligned. To investigate this question, we use
reasoning-guided image generation as a diagnostic task, where models produce textual
reasoning first and then generate images. We introduce UReason, a
benchmark for evaluating
reasoning-to-generation
alignment in this paradigm, consisting of 2,000 human-curated and human-verified instances spanning five
reasoning-intensive tasks: Code, Arithmetic, Spatial, Attribute, and Text. To enable controlled analysis, we develop an evaluation framework that compares direct generation,
reasoning-guided generation, and decontextualized generation, which conditions only on the refined
prompt extracted from
reasoning. Across eight widely used open-source UMMs, while we find that
reasoning-guided generation yields improvements over direct generation, somewhat surprisingly, decontextualized generation consistently outperforms
reasoning-guided generation by a large margin. Our further analyses suggest that the intended visual semantics in textual
reasoning are not reliably reflected in the generated images, despite their unified design and training. Overall, UReason serves as a practical litmus test for
reasoning-to-generation
alignment and provides a challenging
benchmark for developing next-generation, more tightly aligned UMMs.
Read the full abstract ▾
All authors Cheng Yang, Chufan Shi, Bo Shui, Yaokang Wu, Muzi Tao, Huijuan Wang, Ivan Yee Lee, Yong Liu, Xuezhe Ma, Taylor Berg-Kirkpatrick
Computation and Language
Latent Reasoning with Supervised Thinking States
Ido Amos, Avi Caciularu, Mor Geva et al. (+4 more)
Reasoning with a chain-of-thought (CoT) enables Large
Language Models (LLMs) to solve complex tasks but incurs significant inference costs due to the generation of long rationales. We propose Thinking States, a method that performs
reasoning while the input is processing. Specifically, Thinking States generates sequences of thinking
tokens every few input
tokens, transforms the thoughts back into
embedding space, and adds them to the following input
tokens. This has two key advantages. First, it captures the recurrent nature of CoT, but where the thought
tokens are generated as input is processing. Second, since the thoughts are represented as
tokens, they can be learned from natural
language supervision, and using teacher-forcing, which is parallelizable. Empirically, Thinking States outperforms other latent
reasoning methods on multiple
reasoning tasks, narrowing the gap to CoT on math problems, and matching its performance on 2-Hop QA with improved latency. On state-tracking tasks, we show Thinking States leads to stronger
reasoning behavior than CoT, successfully extrapolating to longer sequences than seen during training.
Read the full abstract ▾
All authors Ido Amos, Avi Caciularu, Mor Geva, Amir Globerson, Jonathan Herzig, Lior Shani, Idan Szpektor
Computation and Language
A Generative Model for Joint Multiple Intent Detection and Slot Filling
Liz Li and Wei Zhu
In task-oriented dialogue systems, spoken
language understanding (SLU) is a critical component, which consists of two sub-tasks, intent detection and slot filling. Most existing methods focus on the single-intent SLU, where each utterance only has one intent. However, in real-world scenarios users usually express multiple intents in an utterance, which poses a challenge for existing dialogue systems and
datasets. In this paper, we propose a generative framework to simultaneously address multiple intent detection and slot filling. In particular, an
attention-over-
attention decoder is proposed to handle the variable number of intents and the interference between the two sub-tasks by incorporating an inductive bias into the process of multi-task learning. Besides, we construct two new multi-intent SLU
datasets based on single-intent utterances by taking advantage of the next sentence prediction (NSP) head of the BERT model. Experimental results demonstrate that our proposed
attention-over-
attention generative model achieves state-of-the-art performance on two public
datasets, MixATIS and MixSNIPS, and our constructed
datasets.
Read the full abstract ▾
Computation and Language
Improving Data and Reward Design for Scientific Reasoning in Large Language Models
Zijie Chen, Zhenghao Lin, Xiao Liu et al. (+3 more)
Solving open-ended science questions remains challenging for large
language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the data construction and reward design for scientific post-training. We develop a large-scale, systematic data processing pipeline that transforms heterogeneous open-source science data into Dr. SCI
dataset, which comprises of 1M questions across eight STEM subjects, with explicit verifiable/open-ended splits, scalable difficulty annotation, and fine-grained rubrics that operationalize evaluation for open-ended answers. Building on this
dataset, we propose the Dr. SCI post-training pipeline, which redesigns the standard SFT -> RL workflow through three components: (i) Exploration-Expanding SFT, which broadens the model's
reasoning pattern coverage prior to RL; (ii) Dynamic Difficulty Curriculum, which adapts training data to the model's evolving scientific capability; and (iii) SciRubric-Guided RL, which enables stable reinforcement learning on open-ended scientific questions via rubric-based evaluation with explicit answer correctness. Qwen3-4B-Base trained using Dr. SCI pipeline achieves 63.2 on GPQA-diamond and 32.4 on GPQA-general, consistently improves over strong post-trained baselines such as o1-mini and GPT-4o, demonstrating substantial gains in scientific
reasoning, especially in open-ended settings.
Read the full abstract ▾
All authors Zijie Chen, Zhenghao Lin, Xiao Liu, Zhenzhong Lan, Yeyun Gong, Peng Cheng