Computation and Language · Cross-listed from cs.IR
Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey
Xiantao Zhang
Visually rich documents (VRDs) challenge
retrieval-augmented generation (RAG) with layout-dependent semantics, brittle OCR, and evidence spread across complex figures and structured tables. This survey examines how
Multimodal Large
Language Models (MLLMs) are being used to make VRD
retrieval practical for RAG. We organize the literature into three roles: Modality-Unifying Captioners,
Multimodal Embedders, and End-to-End Representers. We compare these roles along
retrieval granularity, information fidelity, latency and index size, and compatibility with reranking and grounding. We also outline key trade-offs and offer some practical guidance on when to favor each role. Finally, we identify promising directions for future research, including adaptive
retrieval units, model size reduction, and the development of evaluation methods.
Read the full abstract ▾
Computation and Language
DeepResearch-Slice: Bridging the Retrieval-Utilization Gap via Explicit Text Slicing
Shuo Lu, Yinuo Xu, Jianjie Cheng et al. (+3 more)
Deep Research
agents predominantly optimize search policies to maximize
retrieval probability. However, we identify a critical bottleneck: the
retrieval-utilization gap, where models fail to use gold evidence even after it is retrieved, due to context blindness in noisy environments. To bridge this gap, we propose DeepResearch-Slice, a simple yet effective neuro-symbolic framework. Unlike implicit
attention, our approach predicts precise span indices to perform a deterministic hard filter before
reasoning. Extensive evaluations across six
benchmarks show substantial robustness gains. Applying our method to frozen backbones yields a 73 percent relative improvement, from 19.1 percent to 33.0 percent, effectively mitigating noise without requiring parameter updates to the
reasoning model. These results highlight the need for explicit grounding mechanisms in open-ended research.
Read the full abstract ▾
All authors Shuo Lu, Yinuo Xu, Jianjie Cheng, Lingxiao He, Meng Wang, Jian Liang
Computation and Language · Cross-listed from cs.CE
SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval
Chenyang Shao, Fengli Xu and Yong Li
AI
agents have seen widespread adoption in information
retrieval for scientific research, giving rise to tools such as Deep Research. However, existing
retrieval agents mainly rely on keyword- or
embedding-based methods. While effective at capturing content-level similarities, they struggle to understand complex relational networks among scientific papers, such as identifying corroborating or conflicting studies and tracing technological lineages. This fundamental limitation often results in fragmented knowledge structures, misinterpreted research sentiment, and ineffective modeling of collective scientific progress. To address this limitation, we introduce SciNet, the first Scientific Network relation-aware
dataset for information
retrieval agents. Built on a meta-database of 269 million papers across 7 disciplines and containing 8,940 carefully designed tasks, SciNet systematically captures three levels of relational understanding: ego-centric
retrieval of papers with novel knowledge structures, pairwise identification of scholarly relationships, and path-wise reconstruction of scientific
evolution. Extensive evaluation of three categories of
retrieval agents shows that their accuracy on relation-aware tasks often falls below 20%, highlighting a fundamental shortcoming of current
retrieval paradigms. Importantly, in a downstream literature review application,
agents empowered with SciNet achieve a 25.3% improvement in review quality, highlighting the critical value of relation-aware
retrieval for deepening scientific insights. We publicly release SciNet at https://github.com/tsinghua-fib-lab/SciNet to support future research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.DB
Bridging Business Intent and Data: A Benchmark for Automatic Relational Data Product Generation
Faisal Chowdhury, Sola Shirai, Sarthak Dash et al. (+2 more)
A data product is designed to address a specific business need by transforming raw data into a curated, usable asset that delivers actionable insights. Despite practical advances in related areas like text-to-SQL and ELT pipelines, there is no comprehensive
benchmark for evaluating the end-to-end process of automatically generating such data products from high-level business requests. To fill this gap, we introduce DP-Bench, a first-of-its-kind
benchmark for automatic data product creation, built by combining elements from existing ELT and text-to-SQL
datasets. We also propose baseline methods using LLMs to generate data products, providing a foundation for future research in bridging natural
language business intent and structured data representation. We make the DP-Bench
dataset and code available at: https://huggingface.co/
datasets/ibm-research/dp-bench.
Read the full abstract ▾
All authors Faisal Chowdhury, Sola Shirai, Sarthak Dash, Nandana Mihindukulasooriya, Horst Samulowitz
Computation and Language · Cross-listed from cs.CY
Explainable Ethical Assessment on Human Behaviors by Generating Conflicting Social Norms
Yuxi Sun, Wei Gao, Hongzhan Lin et al. (+2 more)
Human behaviors are often guided or constrained by social norms, which are defined as shared, commonsense rules. For example, underlying an action “report a witnessed crime" are social norms that inform our conduct, such as “It is expected to be brave to report crimes”. Current AI systems that assess valence (i.e., support or oppose) of human actions by leveraging large-scale data training not grounded on explicit norms may be difficult to explain, and thus untrustworthy. Emulating human assessors by considering social norms can help AI models better understand and predict valence. While multiple norms come into play, conflicting norms can create tension and directly influence human behavior. For example, when deciding whether to “report a witnessed crime”, one may balance bravery against self-protection. In this paper, we introduce ClarityEthic, a novel ethical assessment approach, to enhance valence prediction and explanation by generating conflicting social norms behind human actions, which strengthens the moral
reasoning capabilities of
language models by using a contrastive learning strategy. Extensive experiments demonstrate that our method outperforms strong baseline approaches, and human evaluations confirm that the generated social norms provide plausible explanations for the assessment of human behaviors.
Read the full abstract ▾
All authors Yuxi Sun, Wei Gao, Hongzhan Lin, Jing Ma, Wenxuan Zhang
Computation and Language · Cross-listed from cs.CY
A Multifaceted Analysis of Social Biases in Large Language Models
Xulang Zhang, Rui Mao and Erik Cambria
Large
language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making. However, ensuring that these models uphold
fairness across varied contexts is critical to their safe and responsible deployment. In this study, we undertake a comprehensive examination of four widely adopted LLMs, probing their underlying biases and inclinations across the dimensions of politics, ideology, alliance,
language, and gender. Through a series of carefully designed experiments, we investigate their political neutrality using news summarization, ideological biases through news stance classification, tendencies toward specific geopolitical alliances via United Nations voting patterns,
language bias in the context of
multilingual story completion, and gender-related affinities as revealed by responses to the World Values Survey. Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Study
Jhessica Silva, Diego A. B. Moreira, Gabriel O. dos Santos et al. (+4 more)
In Artificial Intelligence (AI),
language models have gained significant importance due to the widespread adoption of systems capable of simulating realistic conversations with humans through text generation. Because of their impact on society, developing and deploying these
language models must be done responsibly, with
attention to their negative impacts and possible harms. In this scenario, the number of AI Ethics Tools (AIETs) publications has recently increased. These AIETs are designed to help developers, companies, governments, and other stakeholders establish trust, transparency, and responsibility with their technologies by bringing accepted values to guide AI's design, development, and use stages. However, many AIETs lack good documentation, examples of use, and proof of their effectiveness in practice. This paper presents a methodology for evaluating AIETs in
language models. Our approach involved an extensive literature survey on 213 AIETs, and after applying inclusion and exclusion criteria, we selected four AIETs: Model Cards, ALTAI, FactSheets, and Harms Modeling. For evaluation, we applied AIETs to
language models developed for the Portuguese
language, conducting 35 hours of interviews with their developers. The evaluation considered the developers' perspective on the AIETs' use and quality in helping to identify ethical considerations about their model. The results suggest that the applied AIETs serve as a guide for formulating general ethical considerations about
language models. However, we note that they do not address unique aspects of these models, such as idiomatic expressions. Additionally, these AIETs did not help to identify potential negative impacts of models for the Portuguese
language.
Read the full abstract ▾
All authors Jhessica Silva, Diego A. B. Moreira, Gabriel O. dos Santos, Alef Ferreira, Helena Maia, Sandra Avila, Helio Pedrini
Computation and Language
Buu Phan, Ashish Khisti and Karen Ullrich
Computing next-
token likelihood ratios between two
language models (LMs) is a standard task in training paradigms such as knowledge
distillation. Since this requires both models to share the same probability space, it becomes challenging when the teacher and student LMs use different tokenizers, for instance, when edge-device deployment necessitates a smaller vocabulary size to lower
memory overhead. This work addresses this vocabulary misalignment problem by uncovering an implicit recursive structure in the commonly deployed Byte-Pair Encoding (BPE)
algorithm and utilizing it to create a probabilistic framework for cross-tokenizer likelihood scoring. Our method enables sequence likelihood evaluation for vocabularies different from the teacher model native tokenizer, addressing two specific scenarios: when the student vocabulary is a subset of the teacher vocabulary, and the general case where it is arbitrary. In the subset regime, our framework computes exact likelihoods and provides next-
token probabilities for sequential sampling with only ${O}(1)$ model evaluations per
token. When used for
distillation, this yields up to a $12\%$ reduction in
memory footprint for the Qwen2.5-1.5B model while also improving baseline performance up to $4\%$ on the evaluated tasks. For the general case, we introduce a rigorous lossless procedure that leverages BPE recursive structure, complemented by a fast approximation that keeps large-vocabulary settings practical. Applied to GSM8K mathematical
reasoning distillation, our method improves accuracy by over $2\%$ the current state of the art. Code: github.com/truongbuu/cross-tokenizer-scoring
Read the full abstract ▾
Computation and Language
Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models
George-Andrei Dima, Răzvan-Alexandru Smădu and Dumitru-Clementin Cercel
Focusing on low-resource
languages is an essential step toward democratizing generative AI. In this work, we contribute to reducing the
multimodal NLP resource gap for Romanian. We translate the widely known Flickr30K
dataset into Romanian and further extend it for visual question answering by leveraging open-source LLMs. We demonstrate the usefulness of our
datasets by fine-tuning open-source VLMs on Romanian visual question answering. We select VLMs from three widely used model families: LLaMA 3.2, LLaVA 1.6, and Qwen2. For fine-tuning, we employ the parameter-efficient LoRA method. Our models show improved Romanian capabilities in visual QA, as well as on tasks they were not trained on, such as Romanian image description generation. The seven-billion-parameter Qwen2-VL-RoVQA obtains top scores on both tasks, with improvements of +2.29% and +4.45% in BERTScore F1 on VQA and captioning, respectively, over its original version. Finally, the models show substantial reductions in grammatical errors compared to their original forms, indicating improvements not only in
language understanding but also in Romanian fluency.
Read the full abstract ▾
Computation and Language
Multiscale Aggregated Hierarchical Attention (MAHA): A Game Theoretic and Optimization Driven Approach to Efficient Contextual Modeling in Large Language Models
Caner Erden
The quadratic computational complexity of MultiHead SelfAttention (MHSA) remains a fundamental bottleneck in scaling Large
Language Models (LLMs) for longcontext tasks. While sparse and linearized
attention mechanisms attempt to mitigate this, they often compromise the representation of global dependencies or fail to capture multiscale semantic granularity effectively. In this paper, we propose Multiscale Aggregated Hierarchical
Attention (MAHA), a novel architectural framework that reformulates the
attention mechanism through hierarchical decomposition and mathematically rigorous aggregation. Unlike conventional approaches that treat
token interactions at a single resolution, MAHA dynamically partitions the input sequence into hierarchical scales via learnable downsampling operators. The core innovation lies in its aggregation strategy: we model the fusion of scalespecific
attention matrices as a resource allocation problem, solved via a convex optimization framework or a Nash equilibriumbased gametheoretic approach. This ensures a theoretically optimal balance between local nuance and global context fidelity. Implemented within a hybrid dilatedconvolutional
transformer backbone, MAHA utilizes differentiable optimization layers to enable endtoend training. Experimental evaluations demonstrate that MAHA achieves superior scalability; empirical FLOPs analysis confirms an 81% reduction in computational cost at a sequence length of 4096 compared to standard
attention. This work bridges the gap between optimization theory and sequence modeling, offering a scalable solution for nextgeneration LLMs.
Read the full abstract ▾
Computation and Language
DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline
Houman Kazemzadeh, Kiarash Mokhtari Dizaji, Seyed Reza Tavakoli et al. (+12 more)
In our study, we evaluated large
language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external knowledge integration method to improve accuracy. We benchmarked ten LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy
dataset, measuring baseline accuracy without modification. Baseline performance ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores, while smaller open-source models showed substantially lower performance. We then developed DrugRAG, a three-step
retrieval-augmented generation (RAG) pipeline that retrieves structured, evidence-based drug information and augments model
prompts with contextual pharmacological evidence, operating externally and requiring no changes to model architecture or parameters. DrugRAG increased accuracy across all five evaluated models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61.0% to 71%, Llama 3.1 8B: 46% to 67%). McNemar analyses demonstrated statistically significant paired improvements primarily in smaller and mid-sized open-source models. These findings demonstrate that integrating structured external drug knowledge via DrugRAG can improve LLM performance on pharmacy-focused question-answering tasks without modifying the underlying models, providing a practical pipeline for enhancing evidence-based pharmacy-focused AI applications.
Read the full abstract ▾
All authors Houman Kazemzadeh, Kiarash Mokhtari Dizaji, Seyed Reza Tavakoli, Farbod Davoodi, MohammadReza KarimiNejad, Parham Abed Azad, Fatemeh Latifi, Ali Sabzi, Armin Khosravi, Siavash Ahmadi, Babak Khalaj, Mohammad Hossein Rohban, Glolamali Aminian, Zohreh Amoozgar, Tahereh Javaheri
Computation and Language
Integrating Large Language Models and Knowledge Graphs to Capture Political Viewpoints in News Media
Massimiliano Fadda, Enrico Motta, Francesco Osborne et al. (+2 more)
News sources play a central role in democratic societies by shaping political and social discourse through specific topics, viewpoints and voices. Understanding these dynamics is essential for assessing whether the media landscape offers a balanced and fair account of public debate. In earlier work, we introduced a pipeline that, given a news corpus, i) uses a hybrid human-machine approach to identify the range of viewpoints expressed about a given topic, and ii) classifies relevant claims with respect to the identified viewpoints, defined as sets of semantically and ideologically congruent claims (e.g., positions arguing that immigration positively impacts the UK economy). In this paper, we improve this pipeline by i) fine-tuning Large
Language Models (LLMs) for viewpoint classification and ii) enriching claim representations with semantic descriptions of relevant actors drawn from Wikidata. We evaluate our approach against alternative solutions on a
benchmark centred on the UK immigration debate. Results show that while both mechanisms independently improve classification performance, their integration yields the best results, particularly when using LLMs capable of processing long inputs.
Read the full abstract ▾
All authors Massimiliano Fadda, Enrico Motta, Francesco Osborne, Diego Reforgiato Recupero, Angelo Salatino
Computation and Language · Cross-listed from cs.LG
Task Matrices: Linear Maps for Cross-Model Finetuning Transfer
Darrin O' Brien, Dhikshith Gajulapalli and Eric Xia
Results in interpretability suggest that large
vision and
language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task
matrix, a linear transformation from a base to finetuned
embedding state. We demonstrate that for
vision and text models and ten different
datasets, a base model augmented with a task
matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction
Advait Gosai, Tyler Vuong, Utkarsh Tyagi et al. (+8 more)
End-to-end (E2E) spoken dialogue systems are increasingly replacing cascaded pipelines for voice-based human-AI interaction, processing raw audio directly without intermediate transcription. Existing
benchmarks primarily evaluate these models on synthetic
speech and single-turn tasks, leaving realistic multi-turn conversational ability underexplored. We introduce Audio MultiChallenge, an open-source
benchmark to evaluate E2E spoken dialogue systems under natural multi-turn interaction patterns. Building on the text-based MultiChallenge framework, which evaluates Inference
Memory, Instruction Retention, and Self Coherence, we introduce a new axis Voice Editing that tests robustness to mid-utterance
speech repairs and backtracking. We further augment each axis to the audio modality, such as introducing Audio-Cue challenges for Inference
Memory that require recalling ambient sounds and paralinguistic signals beyond semantic content. We curate 452 conversations from 47 speakers with 1,712 instance-specific rubrics through a hybrid audio-native agentic and human-in-the-loop pipeline that exposes model failures at scale while preserving natural disfluencies found in unscripted human
speech. Our evaluation of proprietary and open-source models reveals that even frontier models struggle on our
benchmark, with Gemini 3 Pro Preview (Thinking), our highest-performing model achieving a 54.65% pass rate. Error analysis shows that models fail most often on our new axes and that Self Coherence degrades with longer audio context. These failures reflect difficulty of tracking edits, audio cues, and long-range context in natural spoken dialogue. Audio MultiChallenge provides a reproducible testbed to quantify them and drive improvements in audio-native multi-turn interaction capability.
Read the full abstract ▾
All authors Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, Yunzhong He
Computation and Language
T5Gemma 2: Seeing, Reading, and Understanding Longer
Biao Zhang, Paul Suganthan, Gaël Liu et al. (+17 more)
We introduce T5Gemma 2, the next generation of the T5Gemma family of lightweight open encoder-decoder models, featuring strong
multilingual,
multimodal and long-context capabilities. T5Gemma 2 follows the adaptation recipe (via UL2) in T5Gemma – adapting a pretrained decoder-only model into an encoder-decoder model, and extends it from text-only regime to
multimodal based on the Gemma 3 models. We further propose two methods to improve the efficiency: tied word
embedding that shares all
embeddings across encoder and decoder, and merged
attention that unifies decoder self- and cross-
attention into a single joint module. Experiments demonstrate the generality of the adaptation strategy over architectures and modalities as well as the unique strength of the encoder-decoder architecture on long context modeling. Similar to T5Gemma, T5Gemma 2 yields comparable or better pretraining performance and significantly improved post-training performance than its Gemma 3 counterpart. We release the pretrained models (270M-270M, 1B-1B and 4B-4B) to the community for future research.
Read the full abstract ▾
All authors Biao Zhang, Paul Suganthan, Gaël Liu, Ilya Philippov, Sahil Dua, Ben Hora, Kat Black, Gus Martins, Omar Sanseviero, Shreya Pathak, Cassidy Hardin, Francesco Visin, Jiageng Zhang, Kathleen Kenealy, Qin Yin, Xiaodan Song, Olivier Lacombe, Armand Joulin, Tris Warkentin, Adam Roberts
Computation and Language
Incentives or Ontology? A Structural Rebuttal to OpenAI's Hallucination Thesis
Richard Ackermann and Simeon Emanuilov
OpenAI has recently argued that
hallucinations in large
language models result primarily from misaligned evaluation incentives that reward confident guessing rather than epistemic humility. On this view,
hallucination is a contingent behavioral artifact, remediable through improved
benchmarks and reward structures. In this paper, we challenge that interpretation. Drawing on previous work on structural
hallucination and empirical experiments using a Licensing Oracle, we argue that
hallucination is not an optimization failure but an architectural inevitability of the
transformer model.
Transformers do not represent the world; they model statistical associations among
tokens. Their
embedding spaces form a pseudo-ontology derived from linguistic co-occurrence rather than world-referential structure. At ontological boundary conditions - regions where training data is sparse or incoherent - the model necessarily interpolates fictional continuations in order to preserve coherence. No incentive mechanism can modify this structural dependence on pattern completion. Our empirical results demonstrate that
hallucination can only be eliminated through external truth-validation and abstention modules, not through changes to incentives, prompting, or fine-tuning. The Licensing Oracle achieves perfect abstention precision across domains precisely because it supplies grounding that the
transformer lacks. We conclude that
hallucination is a structural property of generative architectures and that reliable AI requires hybrid systems that distinguish linguistic fluency from epistemic responsibility.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs
Jun Zhang, Teng Wang, Yuying Ge et al. (+4 more)
This paper does not introduce a novel method but instead establishes a straightforward, incremental, yet essential baseline for
video temporal grounding (VTG), a core capability in
video understanding. While
multimodal large
language models (MLLMs) excel at various
video understanding tasks, the recipes for optimizing them for VTG remain under-explored. In this paper, we present TimeLens, a systematic investigation into building MLLMs with strong VTG ability, along two primary dimensions: data quality and algorithmic design. We first expose critical quality issues in existing VTG
benchmarks and introduce TimeLens-Bench, comprising meticulously re-annotated versions of three popular
benchmarks with strict quality criteria. Our analysis reveals dramatic model re-rankings compared to legacy
benchmarks, confirming the unreliability of prior evaluation standards. We also address noisy training data through an automated re-annotation pipeline, yielding TimeLens-100K, a large-scale, high-quality training
dataset. Building on our data foundation, we conduct in-depth explorations of algorithmic design principles, yielding a series of meaningful insights and effective yet efficient practices. These include interleaved textual encoding for time representation, a thinking-free reinforcement learning with verifiable rewards (RLVR) approach as the training paradigm, and carefully designed recipes for RLVR training. These efforts culminate in TimeLens models, a family of MLLMs with state-of-the-art VTG performance among open-source models and even surpass proprietary models such as GPT-5 and Gemini-2.5-Flash. All codes, data, and models will be released to facilitate future research.
Read the full abstract ▾
All authors Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, Limin Wang
Computation and Language
MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
Zefan Cai, Haoyi Qiu, Tianyi Ma et al. (+13 more)
Modern
multimodal generative models can synthesize visually compelling images and
videos, but it remains unclear whether this visual fluency reflects genuine
reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative
Reasoning Benchmark and Evaluation), a
benchmark for evaluating generative
reasoning across
video, image, and
language-based systems. MMGR covers 10 tasks from three domains (Abstract
Reasoning,
Embodied Navigation, and Physical Commonsense) and probes five
reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for
video generation, process-aware chain-of-frame
reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art
video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and
reasoning correctness:
video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view
embodied navigation. Image generators often outperform
video generators on
embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger
reasoning. MMGR reframes evaluation of
multimodal generation from whether outputs look realistic to whether they solve the underlying
reasoning problem.
Read the full abstract ▾
All authors Zefan Cai, Haoyi Qiu, Tianyi Ma, Haozhe Zhao, Gengze Zhou, Tingting Liao, Xinyan Velocity Yu, Kung-Hsiang Huang, Ke Wan, Shawn Lin, Parisa Kordjamshidi, Minjia Zhang, Wen Xiao, Jiuxiang Gu, Nanyun Peng, Junjie Hu
Computation and Language
Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization
Yen-Ju Lu, Kunxiao Gao, Mingrui Liang et al. (+5 more)
Recent audio
language models can follow long conversations. However, research on
emotion-aware or spoken dialogue summarization is constrained by the lack of data that links
speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries,
emotion-rich summaries, and utterance-level labels for speaker age, gender, and
emotion. The
dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with
emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes
speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460
emotion-diverse dialogues, each paired with both a factual and an
emotion-focused summary. We release an online demo at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/, with plans to release the full
dataset in the near future. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end
speech modeling.
Read the full abstract ▾
All authors Yen-Ju Lu, Kunxiao Gao, Mingrui Liang, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba
Computation and Language
Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
Lanxiang Hu, Siqi Kou, Yichao Fu et al. (+5 more)
Multi-
token generation has emerged as a promising paradigm for accelerating
transformer-based large model inference. Recent efforts primarily explore
diffusion Large
Language Models (dLLMs) for parallel decoding to reduce inference latency. To achieve AR-level generation quality, many techniques adapt AR models into dLLMs to enable parallel decoding. However, they suffer from limited speedup compared to AR models due to a pretrain-to-posttrain mismatch. Specifically, the masked data distribution in post-training deviates significantly from the real-world data distribution seen during pretraining, and dLLMs rely on bidirectional
attention, which conflicts with the causal prior learned during pretraining and hinders the integration of exact KV cache reuse. To address this, we introduce Jacobi Forcing, a progressive
distillation paradigm where models are trained on their own generated parallel decoding trajectories, smoothly shifting AR models into efficient parallel decoders while preserving their pretrained causal inference property. The models trained under this paradigm, Jacobi Forcing Model, achieves 3.8x wall-clock speedup on coding and math
benchmarks with minimal loss in performance. Based on Jacobi Forcing Models' trajectory characteristics, we introduce multi-block decoding with rejection recycling, which enables up to 4.5x higher
token acceptance count per iteration and nearly 4.0x wall-clock speedup, effectively trading additional compute for lower inference latency. Our code is available at https://github.com/hao-ai-lab/JacobiForcing.
Read the full abstract ▾
All authors Lanxiang Hu, Siqi Kou, Yichao Fu, Samyam Rajbhandari, Tajana Rosing, Yuxiong He, Zhijie Deng, Hao Zhang
Computation and Language · Cross-listed from eess.AS
Segmental Attention Decoding With Long Form Acoustic Encodings
Pawel Swietojanski, Xinwei Li, Mingbin Xu et al. (+3 more)
We address the fundamental incompatibility of
attention-based encoder-decoder (AED) models with long-form acoustic encodings. AED models trained on segmented utterances learn to encode absolute frame positions by exploiting limited acoustic context beyond segment boundaries, but fail to generalize when decoding long-form segments where these cues vanish. The model loses ability to order acoustic encodings due to permutation invariance of keys and values in cross-
attention. We propose four modifications: (1) injecting explicit absolute positional encodings into cross-
attention for each decoded segment, (2) long-form training with extended acoustic context to eliminate implicit absolute position encoding, (3) segment concatenation to cover diverse
segmentations needed during training, and (4) semantic
segmentation to align AED-decoded segments with training segments. We show these modifications close the accuracy gap between continuous and segmented acoustic encodings, enabling auto-regressive use of the
attention decoder.
Read the full abstract ▾
All authors Pawel Swietojanski, Xinwei Li, Mingbin Xu, Takaaki Hori, Dogan Can, Xiaodan Zhuang
Computation and Language
TiME: Tiny Monolingual Encoders for Efficient NLP Pipelines
David Schulmeister, Valentin Hartmann, Lars Klein et al. (+1 more)
Today, a lot of research on
language models is focused on large, general-purpose models. However, many NLP pipelines only require models with a well-defined, small set of capabilities. While large models are capable of performing the tasks of those smaller models, they are simply not fast enough to process large amounts of data or offer real-time responses. Furthermore, they often use unnecessarily large amounts of energy, leading to sustainability concerns and problems when deploying them on battery-powered devices. In our work, we show how to train small models for such efficiency-critical applications. As opposed to many off-the-shelf NLP pipelines, our models use modern training techniques such as
distillation, and offer support for low-resource
languages. We call our models TiME (Tiny Monolingual Encoders) and comprehensively evaluate them on a range of common NLP tasks, observing an improved trade-off between
benchmark performance on one hand, and throughput, latency and energy consumption on the other. Along the way, we show that distilling monolingual models from
multilingual teachers is possible, and likewise distilling models with absolute positional
embeddings from teachers with relative positional
embeddings.
Read the full abstract ▾
All authors David Schulmeister, Valentin Hartmann, Lars Klein, Robert West
Computation and Language
JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction
Atsuyuki Miyai, Shota Onohara, Jeonghun Baek et al. (+1 more)
This paper introduces JMMMU-Pro, an image-based Japanese Multi-discipline
Multimodal Understanding
Benchmark, and Vibe
Benchmark Construction, a scalable construction method. Following the
evolution from MMMU to MMMU-Pro, JMMMU-Pro extends JMMMU by composing the question image and question text into a single image, thereby creating a
benchmark that requires integrated visual-textual understanding through visual perception. To build JMMMU-Pro, we propose Vibe
Benchmark Construction, a methodology in which an image generative model (e.g., Nano Banana Pro) produces candidate visual questions, and humans verify the outputs and, when necessary, regenerate with adjusted
prompts to ensure quality. By leveraging Nano Banana Pro's highly realistic image generation capabilities and its ability to embed clean Japanese text, we construct a high-quality
benchmark at low cost, covering a wide range of background and layout designs. Experimental results show that all open-source LMMs struggle substantially with JMMMU-Pro, underscoring JMMMU-Pro as an important
benchmark for guiding future efforts in the open-source community. We believe that JMMMU-Pro provides a more rigorous evaluation tool for assessing the Japanese capabilities of LMMs and that our Vibe
Benchmark Construction also offers an efficient guideline for future development of image-based VQA
benchmarks.
Read the full abstract ▾
All authors Atsuyuki Miyai, Shota Onohara, Jeonghun Baek, Kiyoharu Aizawa
Computation and Language
Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer
Adarsha Shrestha, Basanta Pokharel, Binit Shrestha et al. (+2 more)
Nepali, a low-resource
language spoken by over 32 million people, continues to face challenges in natural
language processing (NLP) due to its complex grammar, agglutinative morphology, and limited availability of high-quality corpora. Most efforts to date have centered on basic encoder architectures; they remain insufficient for Nepali-specific text generation. This study presents a GPT-2-based Nepali
language model trained using several training strategies inspired by GPT-3, including optimized learning rate schedules, batch scaling, and architectural refinements. A custom 16k Byte-Pair Encoding (BPE) tokenizer was trained exclusively on Nepali text to ensure more consistent
segmentation and improved input representation. The model was pretrained on a combined
dataset comprising a 10.75GB cleaned NepBERTa corpus and additional web-scraped Nepali news articles. FlashAttention was integrated to reduce
memory usage and stabilize training. After two epochs, the model achieved a training loss of 3.168177, a validation loss of 3.081982, and a final perplexity of 21.80, demonstrating its capability to generate coherent Nepali news-style text.
Read the full abstract ▾
All authors Adarsha Shrestha, Basanta Pokharel, Binit Shrestha, Smriti Adhikari, Dinesh Gothe
Computation and Language
Low-Resource, High-Impact: Building Corpora for Inclusive Language Technologies
Ekaterina Artemova, Laurie Burchell, Daryna Dementieva et al. (+3 more)
This tutorial (https://tum-nlp.github.io/low-resource-tutorial) is designed for NLP practitioners, researchers, and developers working with
multilingual and low-resource
languages who seek to create more equitable and socially impactful
language technologies. Participants will walk away with a practical toolkit for building end-to-end NLP pipelines for underrepresented
languages – from data collection and web crawling to parallel sentence mining, machine translation, and downstream applications such as text classification and
multimodal reasoning. The tutorial presents strategies for tackling the challenges of data scarcity and cultural variance, offering hands-on methods and modeling frameworks. We will focus on fair, reproducible, and community-informed development approaches, grounded in real-world scenarios. We will showcase a diverse set of use cases covering over 10
languages from different
language families and geopolitical contexts, including both digitally resource-rich and severely underrepresented
languages.
Read the full abstract ▾
All authors Ekaterina Artemova, Laurie Burchell, Daryna Dementieva, Shu Okabe, Mariya Shmatova, Pedro Ortiz Suarez
Computation and Language
Polypersona: Persona-Grounded LLM for Synthetic Survey Responses
Tejaswani Dash, Dinesh Karri, Anudeep Vurity et al. (+4 more)
This paper introduces PolyPersona, a generative framework for synthesizing persona-conditioned survey responses across multiple domains. The framework instruction-tunes compact chat models using parameter-efficient LoRA adapters with 4-bit quantization under a resource-adaptive training setup. A dialogue-based data pipeline explicitly preserves persona cues, ensuring consistent behavioral
alignment across generated responses. Using this pipeline, we construct a
dataset of 3,568 synthetic survey responses spanning ten domains and 433 distinct personas, enabling controlled instruction tuning and systematic multi-domain evaluation. We evaluate the generated responses using a multi-metric evaluation suite that combines standard text generation metrics, including BLEU, ROUGE, and BERTScore, with survey-specific metrics designed to assess structural coherence, stylistic consistency, and sentiment
alignment.Experimental results show that compact models such as TinyLlama 1.1B and Phi-2 achieve performance comparable to larger 7B to 8B baselines, with a highest BLEU score of 0.090 and ROUGE-1 of 0.429. These findings demonstrate that persona-conditioned fine-tuning enables small
language models to generate reliable and coherent synthetic survey data. The proposed framework provides an efficient and reproducible approach for survey data generation, supporting scalable evaluation while facilitating bias analysis through transparent and open protocols.
Read the full abstract ▾
All authors Tejaswani Dash, Dinesh Karri, Anudeep Vurity, Gautam Datla, Tazeem Ahmad, Saima Rafi, Rohith Tangudu
Computation and Language
Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis
Hongli Li, Che Han Chen, Kevin Fan et al. (+3 more)
Despite the growing promise of large
language models (LLMs) in automated essay scoring (AES), empirical findings regarding their reliability compared to human raters remain mixed. Following the PRISMA 2020 guidelines, we synthesized 65 published and unpublished studies from January 2022 to August 2025 that examined agreement between LLM-generated scores and human ratings. Agreement levels varied substantially both across and within studies, with reported values spanning a wide range. Overall, the findings suggest that LLM-human agreement is highly context-dependent. Implications, challenges, and directions for future research are discussed.
Read the full abstract ▾
All authors Hongli Li, Che Han Chen, Kevin Fan, Chiho Young-Johnson, Soyoung Lim, Yali Feng
Computation and Language
Benchmarking Vietnamese Legal Knowledge of Large Language Models
Nguyen Tien Dong, Minh-Anh Nguyen, Thanh Dat Hoang et al. (+6 more)
The rapid advancement of large
language models (LLMs) has enabled new possibilities for applying artificial intelligence within the legal domain. Nonetheless, the complexity, hierarchical organization, and frequent revisions of Vietnamese legislation pose considerable challenges for evaluating how well these models interpret and utilize legal knowledge. To address this gap, the Vietnamese Legal
Benchmark (VLegal-Bench) is introduced, the first comprehensive
benchmark designed to systematically assess LLMs on Vietnamese legal tasks. Informed by Bloom's cognitive taxonomy, VLegal-Bench encompasses multiple levels of legal understanding through tasks designed to reflect practical usage scenarios. The
benchmark comprises 10,450 samples generated through a rigorous annotation pipeline, where legal experts label and cross-validate each instance using our annotation system to ensure every sample is grounded in authoritative legal documents and mirrors real-world legal assistant workflows, including general legal questions and answers,
retrieval-augmented generation, multi-step
reasoning, and scenario-based problem solving tailored to Vietnamese law. By providing a standardized, transparent, and cognitively informed evaluation framework, VLegal-Bench establishes a solid foundation for assessing LLM performance in Vietnamese legal contexts and supports the development of more reliable, interpretable, and ethically aligned AI-assisted legal systems. To facilitate access and reproducibility, we provide a public landing page for this
benchmark at https://vilegalbench.cmcai.vn/.
Read the full abstract ▾
All authors Nguyen Tien Dong, Minh-Anh Nguyen, Thanh Dat Hoang, Nguyen Tuan Ngoc, Dao Xuan Quang Minh, Phan Phi Hai, Nguyen Thi Ngoc Anh, Binh Vu, Dung D. Le
Computation and Language
Dual-objective Language Models: Training Efficiency Without Overfitting
David Samuel and Lucas Georges Gabriel Charpentier
This paper combines autoregressive and masked-
diffusion training objectives without any architectural modifications, resulting in flexible
language models that outperform single-objective models. Autoregressive modeling has been a popular approach, partly because of its training efficiency; however, that comes at the cost of sensitivity to overfitting. On the other hand, masked-
diffusion models are less efficient to train while being more resilient to overfitting. In this work, we demonstrate that dual-objective training achieves the best of both worlds. To derive the optimal balance between both objectives, we train and evaluate 50
language models under varying levels of data repetition. We show that it is optimal to combine both objectives under all evaluated settings and that the optimal balance is similar whether targeting autoregressive or masked-
diffusion downstream performance.
Read the full abstract ▾
Computation and Language
VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse
Ying Nie, Kai Han, Hongguang Li et al. (+5 more)
The rapid scaling of Large
Language Models (LLMs) has achieved remarkable performance, but it also leads to prohibitive
memory costs. Existing parameter-efficient approaches such as
pruning and quantization mainly compress pretrained models without enhancing architectural capacity, thereby hitting the representational ceiling of the base model. In this work, we propose VersatileFFN, a novel feed-forward network (FFN) that enables flexible reuse of parameters in both width and depth dimensions within a fixed parameter budget. Inspired by the dual-process theory of cognition, VersatileFFN comprises two adaptive pathways: a width-versatile path that generates a mixture of sub-experts from a single shared FFN, mimicking sparse expert routing without increasing parameters, and a depth-versatile path that recursively applies the same FFN to emulate deeper processing for complex
tokens. A difficulty-aware gating dynamically balances the two pathways, steering "easy"
tokens through the efficient width-wise route and allocating deeper iterative refinement to "hard"
tokens. Crucially, both pathways reuse the same parameters, so all additional capacity comes from computation rather than
memory. Experiments across diverse
benchmarks and model scales demonstrate the effectiveness of the method. The code is available at https://github.com/huawei-noah/noah-research/tree/master/VersatileFFN.
Read the full abstract ▾
All authors Ying Nie, Kai Han, Hongguang Li, Hang Zhou, Tianyu Guo, Enhua Wu, Xinghao Chen, Yunhe Wang