Computation and Language
PromptGuard at BLP-2025 Task 1: A Few-Shot Classification Framework Using Majority Voting and Keyword Similarity for Bengali Hate Speech Detection
Rakib Hossan and Shubhashis Roy Dipta
The BLP-2025 Task 1A requires Bengali hate
speech classification into six categories. Traditional supervised approaches need extensive labeled
datasets that are expensive for low-resource
languages. We developed PromptGuard, a few-shot framework combining chi-square statistical analysis for keyword extraction with adaptive majority voting for decision-making. We explore statistical keyword selection versus random approaches and adaptive voting mechanisms that extend classification based on consensus quality. Chi-square keywords provide consistent improvements across categories, while adaptive voting benefits ambiguous cases requiring extended classification rounds. PromptGuard achieves a micro-F1 of 67.61, outperforming n-gram baselines (60.75) and random approaches (14.65). Ablation studies confirm chi-square-based keywords show the most consistent impact across all categories.
Read the full abstract ▾
Computation and Language
GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
Adam Byerly and Daniel Khashabi
Large
language models (LLMs) exhibit pronounced position bias in long-context needle-in-haystack problems, systematically prioritizing the location of information over its relevance. While current mitigations rely on white-box access, this is effectively impossible for many state-of-the-art models. We introduce GOLD PANNING, a black-box
Bayesian framework that performs inference-time active search over long contexts by (i) reordering documents to concentrate high-belief items in highly diagnostic positions (signal anchoring) and (ii) updating beliefs over document relevance from model outputs. Unlike conventional active learning, which prioritizes uncertainty reduction, GOLD PANNING leverages anchoring – once flagged, keep it in sight – to preserve weak cues. We implement this using iterative assignment derived from the model's diagnosticity profile, which provably identifies a target among $N$ documents in $O(\log N)$ rounds, ensuring scalability to many-document settings.On needle-in-a-haystack
retrieval and long-context QA, GOLD PANNING matches Permutation Self-Consistency's target identification with $30--65%$ fewer queries and remains effective under calibration mismatch, suggesting coarse positional ordering drives performance gains. These results demonstrate that inherent model biases need not be failures, but can be used as tools for control.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Machine learning methods fail to provide cohesive atheoretical construction of personality traits from semantic embeddings
Ayoub Bouguettaya and Elizabeth M. Stuart
The lexical hypothesis posits that personality traits are encoded in
language and is foundational to models like the Big Five. We created a bottom-up personality model from a classic adjective list using machine learning and compared its descriptive utility against the Big Five by analyzing one million Reddit comments. The Big Five, particularly Agreeableness, Conscientiousness, and Neuroticism, provided a far more powerful and interpretable description of these online communities. In contrast, our machine-learning clusters provided no meaningful distinctions, failed to recover the Extraversion trait, and lacked the psychometric coherence of the Big Five. These results affirm the robustness of the Big Five and suggest personality's semantic structure is context-dependent. Our findings show that while machine learning can help check the ecological validity of established psychological theories, it may not be able to replace them.
Read the full abstract ▾
Computation and Language
Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
Steve Han, Gilberto Titericz Junior, Tom Balough et al. (+1 more)
This research introduces the Judge's Verdict
Benchmark, a novel two-step methodology to evaluate Large
Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (
Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlation analysis to comprehensive Cohen's Kappa analysis that measures actual agreement patterns. The two-step approach includes: (1) a correlation test that filters judges with strong
alignment, followed by (2) a human-likeness test using z-scores to identify two distinct judgment patterns: human-like judgment (|z| < 1) that mimics natural human variation, and super-consistent judgment (z > 1) that exceeds typical human-to-human agreement levels. This methodology reveals that 27 out of 54 tested LLMs achieve Tier 1 performance: 23 models exhibit human-like patterns that preserve the nuances of human judgment, while 4 models demonstrate super-consistent behavior, a pattern that could indicate either enhanced reliability or oversimplification of complex judgments. Testing 43 open-source models (1B-405B parameters) and 11 closed models (GPT, Gemini, Claude variants), we demonstrate that judge excellence is not solely dependent on model size but on specific training strategies. Our key contributions include: (1) establishing that correlation alone is insufficient for judge evaluation, (2) introducing a "Turing Test for judges" based on agreement patterns, and (3) providing a standardized
benchmark for classifying LLM judges into distinct performance tiers for different evaluation needs.
Read the full abstract ▾
All authors Steve Han, Gilberto Titericz Junior, Tom Balough, Wenfei Zhou
Computation and Language
VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
Yubo Sun, Chunyi Peng, Yukun Yan et al. (+5 more)
Visual
Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping
Vision-
Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual
hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable
reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and
reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual
retrieval-augmented framework for multi-image
reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs
reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with
token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and
reasoning. Experiments on multiple visual question answering
benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19%, while substantially reducing visual
hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and
reasoning reliability in multi-image settings. Codes and data are available at https://github.com/OpenBMB/VisRAG
Read the full abstract ▾
All authors Yubo Sun, Chunyi Peng, Yukun Yan, Shi Yu, Zhenghao Liu, Sen Mei, Chi Chen, Maosong Sun
Computation and Language · Cross-listed from cs.LG
It's 2025 – Narrative Learning is the new baseline to beat for explainable machine learning
Gregory D. Baker
In this paper, we introduce Narrative Learning, a methodology where models are defined entirely in natural
language and iteratively refine their classification criteria using explanatory
prompts rather than traditional numerical optimisation. We report on experiments to evaluate the accuracy and potential of this approach using 3 synthetic and 3 natural
datasets and compare them against 7 baseline explainable machine learning models. We demonstrate that on 5 out of 6 of these
datasets, Narrative Learning became more accurate than the baseline explainable models in 2025 or earlier because of improvements in
language models. We also report on trends in the lexicostatistics of these models' outputs as a proxy for the comprehensibility of the explanations.
Read the full abstract ▾
Computation and Language
Layout-Aware Parsing Meets Efficient LLMs: A Unified, Scalable Framework for Resume Information Extraction and Evaluation
Fanwei Zhu, Jinke Yu, Zulong Chen et al. (+6 more)
Automated resume information extraction is critical for scaling talent acquisition, yet its real-world deployment faces three major challenges: the extreme heterogeneity of resume layouts and content, the high cost and latency of large
language models (LLMs), and the lack of standardized
datasets and evaluation tools. In this work, we present a layout-aware and efficiency-optimized framework for automated extraction and evaluation that addresses all three challenges. Our system combines a fine-tuned layout parser to normalize diverse document formats, an inference-efficient LLM extractor based on parallel prompting and instruction tuning, and a robust two-stage automated evaluation framework supported by new
benchmark datasets. Extensive experiments show that our framework significantly outperforms strong baselines in both accuracy and efficiency. In particular, we demonstrate that a fine-tuned compact 0.6B LLM achieves top-tier accuracy while significantly reducing inference latency and computational cost. The system is fully deployed in Alibaba's intelligent HR platform, supporting real-time applications across its business units.
Read the full abstract ▾
All authors Fanwei Zhu, Jinke Yu, Zulong Chen, Ying Zhou, Junhao Ji, Zhibo Yang, Yuxue Zhang, Haoyuan Hu, Zhenghao Liu
Computation and Language · Cross-listed from cs.SE
A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
Jiale Guo, Suizhi Huang, Mei Li et al. (+8 more)
The integration of Large
Language Models (LLMs) into software engineering has driven a transition from traditional rule-based systems to autonomous agentic systems capable of solving complex problems. However, systematic progress is hindered by a lack of comprehensive understanding of how
benchmarks and solutions interconnect. This survey addresses this gap by providing the first holistic analysis of LLM-powered software engineering, offering insights into evaluation methodologies and solution paradigms. We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into
prompt-based, fine-tuning-based, and
agent-based paradigms, and (2)
Benchmarks, including tasks such as code generation, translation, and repair. Our analysis highlights the
evolution from simple
prompt engineering to sophisticated agentic systems incorporating capabilities like planning,
reasoning,
memory mechanisms, and tool augmentation. To contextualize this progress, we present a unified pipeline illustrating the workflow from task specification to deliverables, detailing how different solution paradigms address various complexity levels. Unlike prior surveys that focus narrowly on specific aspects, this work connects 50+
benchmarks to their corresponding solution strategies, enabling researchers to identify optimal approaches for diverse evaluation criteria. We also identify critical research gaps and propose future directions, including multi-
agent collaboration, self-evolving systems, and formal verification integration. This survey serves as a foundational guide for advancing LLM-driven software engineering. We maintain a GitHub repository that continuously updates the reviewed and related papers at https://github.com/lisaGuojl/LLM-
Agent-SE-Survey.
Read the full abstract ▾
All authors Jiale Guo, Suizhi Huang, Mei Li, Dong Huang, Xingsheng Chen, Regina Zhang, Zhijiang Guo, Han Yu, Siu-Ming Yiu, Pietro Lio, Kwok-Yan Lam
Computation and Language
Preference-Aware Memory Update for Long-Term LLM Agents
Haoran Sun, Zekun Zhang and Shaoning Zeng
One of the key factors influencing the
reasoning capabilities of LLM-based
agents is their ability to leverage long-term
memory. Integrating long-term
memory mechanisms allows
agents to make informed decisions grounded in historical interactions. While recent advances have significantly improved the storage and
retrieval components, by encoding
memory into dense vectors for similarity search or organizing
memory as structured knowledge graphs most existing approaches fall short in
memory updating. In particular, they lack mechanisms for dynamically refining preference
memory representations in response to evolving user behaviors and contexts. To address this gap, we propose a Preference-Aware
Memory Update Mechanism (PAMU) that enables dynamic and personalized
memory refinement. By integrating sliding window averages (SW) with exponential moving averages (EMA), PAMU constructs a fused preference-aware representation that captures both short-term fluctuations and long-term user tendencies. We conduct experiments on five task scenarios of the LoCoMo
dataset, and the results show that our mechanism can significantly improve the output quality of LLM in five baselines, validating its effectiveness in long-term conversations.
Read the full abstract ▾
Computation and Language
All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language
Shiyuan Guo, Henry Sleight and Fabien Roger
Detecting harmful AI actions is important as AI
agents gain adoption. Chain-of-thought (CoT) monitoring is one method widely used to detect
adversarial attacks and AI misalignment. However, attackers and misaligned models might evade CoT monitoring through ciphered
reasoning:
reasoning hidden in encrypted, translated, or compressed text. To assess this risk, we test whether models can perform ciphered
reasoning. For each of 28 different ciphers, we fine-tune and
prompt up to 10 models to reason in that cipher. We measure model accuracy on math problems as a proxy for
reasoning ability. Across the models we test, we find an asymmetry: model accuracy can drop significantly when
reasoning in ciphered text, even though models demonstrate comprehension of ciphered text by being able to translate it accurately to English. Even frontier models struggle with lesser-known ciphers, although they can reason accurately in well-known ciphers like rot13. We show that ciphered
reasoning capability correlates with cipher prevalence in pretraining data. We also identify scaling laws showing that ciphered
reasoning capability improves slowly with additional fine-tuning data. Our work suggests that evading CoT monitoring using ciphered
reasoning may be an ineffective tactic for current models and offers guidance on constraining the development of this capability in future frontier models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
CogniDir: Combating Cognitive Malicious Comments via Adaptive Distributional Learning for Robust Fake News Detection
Zhao Tong, Chunlin Gong, Yimeng Gu et al. (+5 more)
The proliferation of Large
Language Models (LLMs) has enabled a new class of psychologically grounded malicious comments, shifting fake news attacks from surface-level textual noise to deep cognitive and logical manipulation. This shift severely undermines existing detectors, which conventionally rely on static attack assumptions and fixed training distributions. To bridge this gap, we introduce CogniDir, an adaptive distributional learning framework that reformulates robust detection as a dynamic data mixture optimization problem for social media content safety. Grounded in cognitive psychology, we first formalize mechanism-specific cognitive
adversarial paradigms to systematically expose deep-seated detector vulnerabilities. To address the vulnerability heterogeneity, CogniDir derives an information-theoretic score coupling empirical accuracy with probabilistic confidence, which is then mapped to adaptive sampling proportions through a Dirichlet-mean parameterization. This formulation enables smooth, feedback-driven reallocation of training exposure toward the most brittle attack mechanisms. Experimental results on three
benchmarks demonstrate that CogniDir yields state-of-the-art robustness, improving F1 scores by up to 17.9% over competitive baselines under heterogeneous, AI-generated
adversarial pressures.
Read the full abstract ▾
All authors Zhao Tong, Chunlin Gong, Yimeng Gu, Haichao Shi, Qiang Liu, Shu Wu, Xingcheng Xu, Xiao-Yu Zhang
Computation and Language
ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models
Wenbin Guo, Xin Wang, Jiaoyan Chen et al. (+3 more)
Large
Language Models (LLMs) have recently emerged as a powerful paradigm for Knowledge Graph Completion (KGC), offering strong
reasoning and generalization capabilities beyond traditional
embedding-based approaches. However, existing LLM-based methods often struggle to fully exploit structured semantic representations, as the continuous
embedding space of pretrained KG models is fundamentally misaligned with the discrete
token space of LLMs. This discrepancy hinders effective semantic transfer and limits their performance. To address this challenge, we propose ReaLM, a novel and effective framework that bridges the gap between KG
embeddings and LLM tokenization through the mechanism of residual vector quantization. ReaLM discretizes pretrained KG
embeddings into compact code sequences and integrates them as learnable
tokens within the LLM vocabulary, enabling seamless fusion of symbolic and contextual knowledge. Furthermore, we incorporate ontology-guided class constraints to enforce semantic consistency, refining entity predictions based on class-level compatibility. Extensive experiments on two widely used
benchmark datasets demonstrate that ReaLM achieves state-of-the-art performance, confirming its effectiveness in aligning structured knowledge with large-scale
language models.
Read the full abstract ▾
All authors Wenbin Guo, Xin Wang, Jiaoyan Chen, Lingbing Guo, Zhao Li, Zirui Chen
Computation and Language
SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG
Xiaonan Si, Meilin Zhu, Simeng Qin et al. (+7 more)
Retrieval-augmented generation (RAG) systems enhance large
language models (LLMs) with external knowledge but are vulnerable to corpus poisoning and contamination attacks, which can compromise output integrity. Existing defenses often apply aggressive filtering, leading to unnecessary loss of valuable information and reduced reliability in generation. To address this problem, we propose a two-stage semantic filtering and conflict-free framework for trustworthy RAG. In the first stage, we perform a joint filter with semantic and cluster-based filtering which is guided by the Entity-intent-relation extractor (EIRE). EIRE extracts entities, latent objectives, and entity relations from both the user query and filtered documents, scores their semantic relevance, and selectively adds valuable documents into the clean
retrieval database. In the second stage, we proposed an EIRE-guided conflict-aware filtering module, which analyzes semantic consistency between the query, candidate answers, and retrieved knowledge before final answer generation, filtering out internal and external contradictions that could mislead the model. Through this two-stage process, SeCon-RAG effectively preserves useful knowledge while mitigating conflict contamination, achieving significant improvements in both generation robustness and output trustworthiness. Extensive experiments across various LLMs and
datasets demonstrate that the proposed SeCon-RAG markedly outperforms state-of-the-art defense methods.
Read the full abstract ▾
All authors Xiaonan Si, Meilin Zhu, Simeng Qin, Lijia Yu, Lijun Zhang, Shuaitong Liu, Xinfeng Li, Ranjie Duan, Yang Liu, Xiaojun Jia
Computation and Language
The Idola Tribus of AI: Large Language Models tend to perceive order where none exists
Shin-nosuke Ishikawa, Masato Todo, Taiki Ogihara et al. (+1 more)
We present a tendency of large
language models (LLMs) to generate absurd patterns despite their clear inappropriateness in a simple task of identifying regularities in number series. Several approaches have been proposed to apply LLMs to complex real-world tasks, such as providing knowledge through
retrieval-augmented generation and executing multi-step tasks using AI
agent frameworks. However, these approaches rely on the logical consistency and self-coherence of LLMs, making it crucial to evaluate these aspects and consider potential countermeasures. To identify cases where LLMs fail to maintain logical consistency, we conducted an experiment in which LLMs were asked to explain the patterns in various integer sequences, ranging from arithmetic sequences to randomly generated integer series. While the models successfully identified correct patterns in arithmetic and
geometric sequences, they frequently over-recognized patterns that were inconsistent with the given numbers when analyzing randomly generated series. This issue was observed even in multi-step
reasoning models, including OpenAI o3, o4-mini, and Google Gemini 2.5 Flash Preview Thinking. This tendency to perceive non-existent patterns can be interpreted as the AI model equivalent of Idola Tribus and highlights potential limitations in their capability for applied tasks requiring logical
reasoning, even when employing chain-of-thought
reasoning mechanisms.
Read the full abstract ▾
All authors Shin-nosuke Ishikawa, Masato Todo, Taiki Ogihara, Hirotsugu Ohba
Computation and Language · Cross-listed from cs.CV
StreamingVLM: Real-Time Understanding for Infinite Video Streams
Ruyi Xu, Guangxuan Xiao, Yukang Chen et al. (+3 more)
Vision-
language models (VLMs) could power real-time assistants and autonomous
agents, but they face a critical challenge: understanding near-infinite
video streams without escalating latency and
memory usage. Processing entire
videos with full
attention leads to quadratic computational costs and poor performance on long
videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of
attention sinks, a short window of recent
vision tokens, and a long window of recent text
tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full
attention on short, overlapped
video chunks, which effectively mimics the inference-time
attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new
benchmark with
videos averaging over two hours that requires dense, per-second
alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.
Read the full abstract ▾
All authors Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, Song Han
Computation and Language
Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
Sondos Mahmoud Bsharat and Zhiqiang Shen
Large
language models (LLMs) have demonstrated impressive
reasoning capabilities when provided with chain-of-thought exemplars, but curating large
reasoning datasets remains laborious and resource-intensive. In this work, we introduce Prompting Test-Time Scaling (P-TTS), a simple yet effective inference-time data augmentation strategy for enhancing LLM
reasoning through finetuning. Rather than collecting thousands or even millions of examples, P-TTS leverages a small pool of only 90 manually selected
reasoning instances and systematically varies exemplar augmentation through principled instruction prompting intensities at test time to synthesize diverse
reasoning trajectory contexts. Then we finetune the various sizes of Qwen-2.5 models on P-TTS data. Across a suite of mathematical
reasoning AIME2024 & 25, MATH500, and GPQA-Diamond, our P-TTS-7B and 32B models outperform the prior competitive baselines like S1 and S1.1 (1K-shot), achieving absolute accuracy gains of +26.66% and +30.00% on AIME'24 (7B), and +13.34% and +6.67% on AIME'25 (7B); P-TTS-32B yields gains of +23.33% and +16.63% on AIME'24, and +26.63% and +3.33% on AIME'25 (vs. S1 and S1.1, respectively), with comparable or better performance on MATH500 and GPQA-Diamond. We further show that P-TTS enhances zero-shot generalization accuracy on out-of-domain
reasoning benchmarks of Gaokao, Kaoyan, OlympiadBench, AMC23, GradeSchoolMath, and Minerva. Our analysis suggests that test-time scaling effectively explores the latent space of
reasoning patterns, amplifying LLM problem-solving with minimal annotation overhead, and further unlocking the
reasoning potential and capabilities of LLMs. Prompting Test-Time Scaling offers a practical, low-cost way to elicit LLM
reasoning in resource-constrained or rapidly evolving domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Kaijian Zou, Aaron Xiong, Yunxiang Zhang et al. (+6 more)
Competitive programming problems are increasingly used to evaluate the coding capabilities of large
language models (LLMs) due to their complexity and ease of verification. Yet, current coding
benchmarks face limitations such as a lack of exceptionally challenging problems, insufficient test case coverage, and reliance on online platform APIs that limit accessibility. To address these issues, we introduce LiveOIBench, a large-scale competitive programming
benchmark featuring 403 expert-curated problems, averaging 60 official test cases each, drawn from 72 contests across 14 Informatics Olympiads held between 2023 and 2025. LiveOIBench has four key features: (1) expert-designed tasks with detailed subtask rubrics and extensive test cases; (2) direct comparison to elite human contestants; (3) continuous updates to reduce contamination risk; and (4) a fully offline, reproducible evaluation system. Benchmarking 34 popular general-purpose and
reasoning LLMs, we find that GPT-5 achieves an 81.76th percentile, still falling short of top human contestants, while among the open-weight models, GPT-OSS-120B reaches only the 60th percentile.
Reasoning-trace analyses indicate that robust
reasoning models prioritize precise problem analysis over excessive exploration. Finally, analyses across release dates, task familiarity, and code similarity find minimal evidence of data contamination in our
benchmark. Our leaderboard, code, and data are available at: https://liveoibench.github.io/.
Read the full abstract ▾
All authors Kaijian Zou, Aaron Xiong, Yunxiang Zhang, Frederick Zhang, Yueqi Ren, Jirong Yang, Ayoung Lee, Shitanshu Bhushan, Lu Wang
Computation and Language
Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models
Donghang Wu, Haoyang Zhang, Jun Chen et al. (+8 more)
Real-time Spoken
Language Models (SLMs) struggle to leverage Chain-of-Thought (CoT)
reasoning due to the prohibitive latency of generating the entire thought process sequentially. Enabling SLMs to think while speaking, similar to humans, is attracting increasing
attention. We present, for the first time, Mind-Paced Speaking (MPS), a
brain-inspired framework that enables high-fidelity, real-time
reasoning. Similar to how humans utilize distinct
brain regions for thinking and responding, we propose a novel dual-
brain approach, employing a "Formulation
Brain" for high-level
reasoning to pace and guide a separate "Articulation
Brain" for fluent
speech generation. This division of labor eliminates mode-switching, preserving the integrity of the
reasoning process. Experiments show that MPS significantly outperforms existing think-while-speaking methods and achieves
reasoning performance comparable to models that pre-compute the full CoT before speaking, while drastically reducing latency. Under a zero-latency configuration, the proposed method achieves an accuracy of 92.8% on the mathematical
reasoning task Spoken-MQA and attains a score of 82.5 on the
speech conversation task URO-Bench. MPS is the methodology underlying our released Step-Audio R1.1 system, effectively bridging the gap between high-quality
reasoning and real-time interaction.
Read the full abstract ▾
All authors Donghang Wu, Haoyang Zhang, Jun Chen, Xiangyu Zhang, Hexin Liu, Eng Siong Chng, Fei Tian, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu
Computation and Language · Cross-listed from cs.AI
GraphMERT: Efficient and Scalable Distillation of Reliable Knowledge Graphs from Unstructured Data
Margarita Belova, Jiaxin Xiao, Shikhar Tuli et al. (+1 more)
Researchers have pursued neurosymbolic artificial intelligence (AI) applications for nearly three decades. A marriage of the neural and symbolic components can lead to rapid advancements in AI. Yet, the field has not realized this promise since most neurosymbolic AI frameworks fail to scale. In addition, the implicit representations and approximate
reasoning of purely neural approaches limit interpretability and trust. Knowledge graphs (KGs), a gold-standard representation of explicit semantic knowledge, can address the symbolic side of the problem. However, automatically deriving reliable KGs from text corpora remains an open problem. We address these challenges by introducing GraphMERT, a tiny graphical encoder-only model that distills high-quality KGs from unstructured text corpora and its own internal representations. GraphMERT and its equivalent KG form a modular neurosymbolic stack: neural learning of abstractions; symbolic KGs for verifiable
reasoning. GraphMERT + KG is the first efficient and scalable neurosymbolic model to achieve state-of-the-art
benchmark accuracy along with superior symbolic representations relative to baselines. Concretely, we target reliable domain-specific KGs that are both (1) factual (with provenance) and (2) valid (ontology-consistent relations with domain-appropriate semantics). When a large
language model (LLM), e.g., Qwen3-32B, generates domain-specific KGs, it falls short on reliability due to
prompt sensitivity, shallow domain expertise, and hallucinated relations. On text obtained from PubMed papers on diabetes, our 80M-parameter GraphMERT yields a KG with a 69.8% FActScore; a 32B-parameter baseline LLM yields a KG that achieves only 40.2% FActScore. The GraphMERT KG also attains a higher ValidityScore of 68.8%, versus 43.0% for the LLM baseline.
Read the full abstract ▾
All authors Margarita Belova, Jiaxin Xiao, Shikhar Tuli, Niraj K. Jha
Computation and Language
Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Xiao Yu, Baolin Peng, Michel Galley et al. (+6 more)
Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon, interactive tasks such as web
navigation and computer/phone-use. Inspired by literature on human cognition, we argue that current AI
agents need ”vicarious trial and error” - the capacity to mentally simulate alternative futures before acting - in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM
agents to integrate such simulation into their
reasoning. In stage 1, we introduce
Reasoning with Simulations (ReSim), which trains the
agent to generate structured
reasoning traces from expanded search trees built from real experience gathered through environment interactions. ReSim thus grounds the
agent's
reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its
reasoning. In stage 2, we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the
agent's simulation and decision-making ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic
benchmarks (Sokoban and ALFWorld) and one realistic
benchmark (AndroidWorld) demonstrate that (1) ReSim effectively infuses simulation ability into AI
agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning-intensive tasks. Together, these results highlight the central role of simulation in enabling AI
agents to reason, plan, and act more effectively in the ever more challenging environments.
Read the full abstract ▾
All authors Xiao Yu, Baolin Peng, Michel Galley, Hao Cheng, Qianhui Wu, Janardhan Kulkarni, Suman Nath, Zhou Yu, Jianfeng Gao
Computation and Language
AutoPR: Let's Automate Your Academic Promotion!
Qiguang Chen, Zheng Yan, Mingda Yang et al. (+10 more)
As the volume of peer-reviewed research surges, scholars increasingly rely on social platforms for discovery, while authors invest considerable effort in promoting their work to ensure visibility and citations. To streamline this process and reduce the reliance on human effort, we introduce Automatic Promotion (AutoPR), a novel task that transforms research papers into accurate, engaging, and timely public content. To enable rigorous evaluation, we release PRBench, a
multimodal benchmark that links 512 peer-reviewed articles to high-quality promotional posts, assessing systems along three axes: Fidelity (accuracy and tone), Engagement (audience targeting and appeal), and
Alignment (timing and channel optimization). We also introduce PRAgent, a multi-
agent framework that automates AutoPR in three stages: content extraction with
multimodal preparation, collaborative synthesis for polished outputs, and platform-specific adaptation to optimize norms, tone, and tagging for maximum reach. When compared to direct LLM pipelines on PRBench, PRAgent demonstrates substantial improvements, including a 604% increase in total watch time, a 438% rise in likes, and at least a 2.9x boost in overall engagement. Ablation studies show that platform modeling and targeted promotion contribute the most to these gains. Our results position AutoPR as a tractable, measurable research problem and provide a roadmap for scalable, impactful automated scholarly communication.
Read the full abstract ▾
All authors Qiguang Chen, Zheng Yan, Mingda Yang, Libo Qin, Yixin Yuan, Hanjing Li, Jinhao Liu, Yiyan Ji, Dengyun Peng, Jiannan Guan, Mengkang Hu, Yantao Du, Wanxiang Che
Computation and Language
WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives
Daniel Brubaker, William Sheffield, Junyi Jessy Li et al. (+1 more)
The role of world knowledge has been particularly crucial to predict the discourse connective that marks the discourse relation between two arguments, with
language models (LMs) being generally successful at this task. We flip this premise in our work, and instead study the inverse problem of understanding whether discourse connectives can inform LMs about the world. To this end, we present WUGNECTIVES, a
dataset of 8,880 stimuli that evaluates LMs' inferences about novel entities in contexts where connectives link the entities to particular attributes. On investigating 17 different LMs at various scales, and training regimens, we found that tuning an LM to show
reasoning behavior yields noteworthy improvements on most connectives. At the same time, there was a large variation in LMs' overall performance across connective type, with all models systematically struggling on connectives that express a concessive meaning. Our findings pave the way for more nuanced investigations into the functional role of
language cues as captured by LMs. We release WUGNECTIVES at https://github.com/kanishkamisra/wugnectives
Read the full abstract ▾
All authors Daniel Brubaker, William Sheffield, Junyi Jessy Li, Kanishka Misra
Computation and Language
A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages
Raoyuan Zhao, Yihong Liu, Hinrich Schütze et al. (+1 more)
Large
reasoning models (LRMs) increasingly rely on step-by-step Chain-of-Thought (CoT)
reasoning to improve task performance, particularly in high-resource
languages such as English. While recent work has examined final-answer accuracy in
multilingual settings, the thinking traces themselves, i.e., the intermediate steps that lead to the final answer, remain underexplored. In this paper, we present the first comprehensive study of
multilingual CoT
reasoning, evaluating three key dimensions: performance, consistency, and faithfulness. We begin by measuring
language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or
prompt-hacked to think in a target
language, revealing strong
language preferences and divergent performance across
languages. Next, we assess crosslingual consistency of thinking traces by interchanging them between
languages. We find that the quality and effectiveness of thinking traces vary substantially depending on the
prompt language. Finally, we adapt perturbation-based techniques – i.e., truncation and error injection – to probe the faithfulness of thinking traces across
languages, showing that models rely on traces to varying degrees. We release our code and data to support future research.
Read the full abstract ▾
All authors Raoyuan Zhao, Yihong Liu, Hinrich Schütze, Michael A. Hedderich
Computation and Language
Hierarchical Indexing with Knowledge Enrichment for Multilingual Video Corpus Retrieval
Yu Wang, Tianhao Tan and Yifei Wang
Retrieving relevant instructional
videos from
multilingual medical archives is crucial for answering complex, multi-hop questions across
language boundaries. However, existing systems either compress hour-long
videos into coarse
embeddings or incur prohibitive costs for fine-grained matching. We tackle the
Multilingual Video Corpus
Retrieval (mVCR) task in the NLPCC-2025 M4IVQA challenge with a multi-stage framework that integrates
multilingual semantics, domain terminology, and efficient long-form processing.
Video subtitles are divided into semantically coherent chunks, enriched with concise knowledge-graph (KG) facts, and organized into a hierarchical tree whose node
embeddings are generated by a
language-agnostic
multilingual encoder. At query time, the same encoder embeds the input question; a coarse-to-fine tree search prunes irrelevant branches, and only the top-ranked chunks are re-scored by a lightweight large
language model (LLM). This design avoids exhaustive cross-encoder scoring while preserving chunk-level precision. Experiments on the mVCR test set demonstrate state-of-the-art performance, and ablation studies confirm the complementary contributions of KG enrichment, hierarchical indexing, and targeted LLM re-ranking. The proposed method offers an accurate and scalable solution for
multilingual retrieval in specialized medical
video collections.
Read the full abstract ▾
Computation and Language
Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models
Qiguang Chen, Hanjing Li, Libo Qin et al. (+7 more)
Recently,
Diffusion Large
Language Models (DLLMs) have offered high throughput and effective sequential
reasoning, making them a competitive alternative to autoregressive LLMs (ALLMs). However, parallel decoding, which enables simultaneous
token updates, conflicts with the causal order often required for rigorous
reasoning. We first identify this conflict as the core Parallel-Sequential Contradiction (PSC). Behavioral analyses in both simple and complex
reasoning tasks show that DLLMs exhibit genuine parallelism only for directly decidable outputs. As task difficulty increases, they revert to autoregressive-like behavior, a limitation exacerbated by autoregressive prompting, which nearly doubles the number of decoding steps with remasking without improving quality. Moreover, PSC restricts DLLMs' self-reflection,
reasoning depth, and exploratory breadth. To further characterize PSC, we introduce three scaling dimensions for DLLMs: parallel,
diffusion, and sequential. Empirically, while parallel scaling yields consistent improvements,
diffusion and sequential scaling are constrained by PSC. Based on these findings, we propose several practical mitigations, parallel-oriented prompting,
diffusion early stopping, and parallel scaling, to reduce PSC-induced ineffectiveness and inefficiencies.
Read the full abstract ▾
All authors Qiguang Chen, Hanjing Li, Libo Qin, Dengyun Peng, Jinhao Liu, Jiangyi Wang, Chengyue Wu, Xie Chen, Yantao Du, Wanxiang Che
Computation and Language
SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
Chenyu Wang, Paria Rashidinejad, DiJia Su et al. (+9 more)
Diffusion large
language models (dLLMs) are emerging as an efficient alternative to autoregressive models due to their ability to decode multiple
tokens in parallel. However, aligning dLLMs with human preferences or task-specific rewards via reinforcement learning (RL) is challenging because their intractable log-likelihood precludes the direct application of standard policy gradient methods. While prior work uses surrogates like the evidence lower bound (ELBO), these one-sided approximations can introduce significant policy gradient bias. To address this, we propose the Sandwiched Policy Gradient (SPG) that leverages both an upper and a lower bound of the true log-likelihood. Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation. Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku.
Read the full abstract ▾
All authors Chenyu Wang, Paria Rashidinejad, DiJia Su, Song Jiang, Sid Wang, Siyan Zhao, Cai Zhou, Shannon Zejiang Shen, Feiyu Chen, Tommi Jaakkola, Yuandong Tian, Bo Liu
Computation and Language
Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
Raoyuan Zhao, Yihong Liu, Lena Altinger et al. (+2 more)
Large
language models (LLMs) are increasingly deployed in
multilingual, real-world applications with user inputs – naturally introducing typographical errors (typos). Yet most
benchmarks assume clean input, leaving the robustness of LLMs to typos across
languages largely underexplored. To address this gap, we introduce MulTypo, a
multilingual typo generation
algorithm that simulates human-like errors based on
language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning
language inference, multi-choice question answering, mathematical
reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring
reasoning – while the natural
language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe
language-dependent robustness: high-resource
languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and
multilingual robustness evaluation. We release a Python package for MulTypo and make the source code publicly available at https://github.com/cisnlp/multypo.
Read the full abstract ▾
All authors Raoyuan Zhao, Yihong Liu, Lena Altinger, Hinrich Schütze, Michael A. Hedderich
Computation and Language
Mitigating Overthinking through Reasoning Shaping
Feifan Song, Shaohang Wei, Bofei Gao et al. (+8 more)
Large
reasoning models (LRMs) boosted by Reinforcement Learning from Verifier Reward (RLVR) have shown great power in problem solving, yet they often cause overthinking: excessive, meandering
reasoning that inflates computational cost. Prior designs of penalization in RLVR manage to reduce
token consumption while often harming model performance, which arises from the oversimplicity of
token-level supervision. In this paper, we argue that the granularity of supervision plays a crucial role in balancing efficiency and accuracy, and propose Group Relative Segment Penalization (GRSP), a step-level method to regularize
reasoning. Since preliminary analyses show that
reasoning segments are strongly correlated with
token consumption and model performance, we design a length-aware weighting mechanism across segment clusters. Extensive experiments demonstrate that GRSP achieves superior
token efficiency without heavily compromising accuracy, especially the advantages with harder problems. Moreover, GRSP stabilizes RL training and scales effectively across model sizes.
Read the full abstract ▾
All authors Feifan Song, Shaohang Wei, Bofei Gao, Yejie Wang, Wen Luo, Wei Li, Linli Yao, Weimin Xiong, Liang Chen, Tianyu Liu, Houfeng Wang
Computation and Language
Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking
Mohammad Hossein Sameti, Sepehr Harfi Moridani, Ali Zarean et al. (+1 more)
Pre-trained
transformer-based models have significantly advanced automatic
speech recognition (ASR), yet they remain sensitive to accent and dialectal variations, resulting in elevated word error rates (WER) in linguistically diverse
languages such as English and Persian. To address this challenge, we propose an accent-invariant ASR framework that integrates accent and dialect classification into the recognition pipeline. Our approach involves training a spectrogram-based classifier to capture accent-specific cues, masking the regions most influential to its predictions, and using the masked spectrograms for data augmentation. This enhances the robustness of ASR models against accent variability. We evaluate the method using both English and Persian
speech. For Persian, we introduce a newly collected
dataset spanning multiple regional accents, establishing the first systematic
benchmark for accent variation in Persian ASR that fills a critical gap in
multilingual speech research and provides a foundation for future studies on low-resource, linguistically diverse
languages. Experimental results with the Whisper model demonstrate that our masking and augmentation strategy yields substantial WER reductions in both English and Persian settings, confirming the effectiveness of the approach. This research advances the development of
multilingual ASR systems that are resilient to accent and dialect diversity. Code and
dataset are publicly available at: https://github.com/MH-Sameti/Accent_invariant_ASR
Read the full abstract ▾
All authors Mohammad Hossein Sameti, Sepehr Harfi Moridani, Ali Zarean, Hossein Sameti
Computation and Language
Can We Reliably Rank Model Performance across Domains without Labeled Data?
Veronica Rammouz, Aaron Gonzalez, Carlos Cruzportillo et al. (+3 more)
Estimating model performance without labels is an important goal for understanding how NLP models generalize. While prior work has proposed measures based on
dataset similarity or predicted correctness, it remains unclear when these estimates produce reliable performance rankings across domains. In this paper, we analyze the factors that affect ranking reliability using a two-step evaluation setup with four base classifiers and several large
language models as error predictors. Experiments on the GeoOLID and Amazon Reviews
datasets, spanning 15 domains, show that large
language model-based error predictors produce stronger and more consistent rank correlations with true accuracy than drift-based or zero-shot baselines. Our analysis reveals two key findings: ranking is more reliable when performance differences across domains are larger, and when the error model's predictions align with the base model's true failure patterns. These results clarify when performance estimation methods can be trusted and provide guidance for their use in cross-domain model evaluation.
Read the full abstract ▾
All authors Veronica Rammouz, Aaron Gonzalez, Carlos Cruzportillo, Adrian Tan, Nicole Beebe, Anthony Rios