Computation and Language
LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding
Zhivar Sourati, Zheng Wang, Marianne Menglin Liu et al. (+8 more)
Question answering over visually rich documents (VRDs) requires
reasoning not only over isolated content but also over documents' structural organization and cross-page dependencies. However, conventional
retrieval-augmented generation (RAG) methods encode content in isolated chunks during ingestion, losing structural and cross-page dependencies, and retrieve a fixed number of pages at inference, regardless of the specific demands of the question or context. This often results in incomplete evidence
retrieval and degraded answer quality for multi-page
reasoning tasks. To address these limitations, we propose LAD-RAG, a novel Layout-Aware Dynamic RAG framework. During ingestion, LAD-RAG constructs a symbolic document graph that captures layout structure and cross-page dependencies, adding it alongside standard neural
embeddings to yield a more holistic representation of the document. During inference, an LLM
agent dynamically interacts with the neural and symbolic indices to adaptively retrieve the necessary evidence based on the query. Experiments on MMLongBench-Doc, LongDocURL, DUDE, and MP-DocVQA demonstrate that LAD-RAG improves
retrieval, achieving over 90% perfect recall on average without any top-k tuning, and outperforming baseline retrievers by up to 20% in recall at comparable noise levels, yielding higher QA accuracy with minimal latency.
Read the full abstract ▾
All authors Zhivar Sourati, Zheng Wang, Marianne Menglin Liu, Yazhe Hu, Mengqing Guo, Sujeeth Bharadwaj, Kyu Han, Tao Sheng, Sujith Ravi, Morteza Dehghani, Dan Roth
Computation and Language
EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models
Donggyu Lee, Hyeok Yun, Meeyoung Cha et al. (+3 more)
Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes,
market conditions, time periods, or populations. This poses a challenge for large
language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes? To address this, we introduce EconCausal, a large-scale
benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering. Across 19 models, we observe source-sign over-transfer in cross-context prediction. While top models reach ~84% accuracy in single-context prediction, mean accuracy is 74.75% overall but drops to 44.27% when source and target signs disagree. On these cases, accuracy is 11.54 percentage points (pp) lower with the signed source example than with the target input alone. Mean accuracy is below 60% under corrupted source signs. Models also struggle to predict study-reported nonsignificant effects (19.16% accuracy). The
dataset and
benchmark are publicly available at https://anonymous.4open.science/r/econcausal-
benchmark-6F12.
Read the full abstract ▾
All authors Donggyu Lee, Hyeok Yun, Meeyoung Cha, Sungwon Park, Sangyoon Park, Jihee Kim
Computation and Language
Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
Ziyi Wang, Yuxuan Lu, Yimeng Zhang et al. (+2 more)
Simulating step-wise human behavior with Large
Language Models (LLMs) has become an emerging research direction, enabling applications in various practical domains. While prior methods, including prompting, supervised fine-tuning (SFT), and reinforcement learning (RL), have shown promise in modeling step-wise behavior, they primarily learn a population-level policy without conditioning on a user's persona, yielding generic rather than personalized simulations. In this work, we pose a critical question: how can LLM
agents better simulate personalized user behavior? We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments. Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals. Experiments on the OPeRA
dataset emonstrate that Customer-R1 not only significantly outperforms prompting and SFT-based baselines in next-action prediction tasks, but also better matches users' action distribution, indicating higher fidelity in personalized behavior simulation.
Read the full abstract ▾
All authors Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Jing Huang, Dakuo Wang
Computation and Language
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik et al. (+5 more)
Small
Language models (SLMs) offer an efficient and accessible alternative to Large
Language Models (LLMs), delivering strong performance while using far fewer resources. We introduce a simple and effective framework for pretraining SLMs that brings together three complementary ideas. First, we identify structurally sparse sub-network initializations that consistently outperform randomly initialized models of similar size under the same compute budget. Second, we use evolutionary search to automatically discover high-quality sub-network initializations, providing better starting points for pretraining. Third, we apply knowledge
distillation from larger teacher models to speed up training and improve generalization. Together, these components make SLM pretraining substantially more efficient: our best model, discovered using evolutionary search and initialized with LLM weights, matches the validation perplexity of a comparable Pythia SLM while requiring 5.16x and 1.26x fewer floating point operations for
token budgets of 10B and 100B, respectively. We release all code publicly, offering a practical and reproducible path toward cost-efficient small
language model development at scale.
Read the full abstract ▾
All authors Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik, Gabriela Kadlecová, Vladyslav Moroshan, Timur Carstensen, Frank Hutter, Aaron Klein
Computation and Language · Cross-listed from cs.SI
Machines in the Crowd? Measuring the Footprint of Machine-Generated Text on Reddit
Lucio La Cava, Luca Maria Aiello and Andrea Tagarelli
Generative Artificial Intelligence is reshaping online communication by enabling large-scale production of Machine-Generated Text (MGT) at low cost. While its presence is rapidly growing across the Web, little is known about how MGT integrates into social media environments. In this paper, we present the first large-scale characterization of MGT on Reddit. Using a state-of-the-art statistical method for detection of MGT, we analyze over two years of activity (2022-2024) across 51 subreddits representative of Reddit's main community types such as information seeking, social support, and discussion. We study the concentration of MGT across communities and over time, and compared MGT to human-authored text in terms of social signals it expresses and engagement it receives. Our very conservative estimate of MGT prevalence indicates that synthetic text is marginally present on Reddit, but it can reach peaks of up to 9% in some communities in some months. MGT is unevenly distributed across communities, more prevalent in subreddits focused on technical knowledge and social support, and often concentrated in the activity of a small fraction of users. MGT also conveys distinct social signals of warmth and status giving typical of
language of AI assistants. Despite these stylistic differences, MGT achieves engagement levels comparable than human-authored content and in a few cases even higher, suggesting that AI-generated text is becoming an organic component of online social discourse. This work offers the first perspective on the MGT footprint on Reddit, paving the way for new investigations involving platform governance, detection strategies, and community dynamics.
Read the full abstract ▾
Computation and Language
How much speech data is necessary for ASR in African languages? An evaluation of data scaling in Kinyarwanda and Kikuyu
Benjamin Akera, Evelyn Nafula, Patrick Walukagga et al. (+3 more)
The development of Automatic
Speech Recognition (ASR) systems for low-resource African
languages remains challenging due to limited transcribed
speech data. While recent advances in large
multilingual models like OpenAI's Whisper offer promising pathways for low-resource ASR development, critical questions persist regarding practical deployment requirements. This paper addresses two fundamental concerns for practitioners: determining the minimum data volumes needed for viable performance and characterizing the primary failure modes that emerge in production systems. We evaluate Whisper's performance through comprehensive experiments on two Bantu
languages: systematic data scaling analysis on Kinyarwanda using training sets from 1 to 1,400 hours, and detailed error characterization on Kikuyu using 270 hours of training data. Our scaling experiments demonstrate that practical ASR performance (WER < 13%) becomes achievable with as little as 50 hours of training data, with substantial improvements continuing through 200 hours (WER < 10%). Complementing these volume-focused findings, our error analysis reveals that data quality issues, particularly noisy ground truth transcriptions, account for 38.6% of high-error cases, indicating that careful data curation is as critical as data volume for robust system performance. These results provide actionable
benchmarks and deployment guidance for teams developing ASR systems across similar low-resource
language contexts. We release accompanying and models see https://github.com/SunbirdAI/kinyarwanda-whisper-eval
Read the full abstract ▾
All authors Benjamin Akera, Evelyn Nafula, Patrick Walukagga, Gilbert Yiga, John Quinn, Ernest Mwebaze
Computation and Language
Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models
Chengzhi Zhong, Fei Cheng, Qianying Liu et al. (+3 more)
Large
language models exhibit strong
multilingual capabilities despite limited exposure to non-English data. Prior studies show that English-centric large
language models map
multilingual content into English-aligned representations at intermediate layers and then project them back into target-
language token spaces in the final layer. From this observation, we hypothesize that this cross-lingual transition is governed by a small and sparse set of dimensions, which occur at consistent indices across the intermediate to final layers. Building on this insight, we introduce a simple, training-free method to identify and manipulate these dimensions, requiring only as few as 50 sentences of either parallel or monolingual data. Experiments on a
multilingual generation control task reveal the interpretability of these dimensions, demonstrating that the interventions in these dimensions can switch the output
language while preserving semantic content, and that it surpasses the performance of prior
neuron-based approaches at a substantially lower cost.
Read the full abstract ▾
All authors Chengzhi Zhong, Fei Cheng, Qianying Liu, Yugo Murawaki, Chenhui Chu, Sadao Kurohashi
Computation and Language
Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models
Benjamin Akera, Evelyn Nafula Ouma, Gilbert Yiga et al. (+11 more)
There are more than 2000 living
languages in Africa, most of which have been bypassed by advances in
language technology. Current leading LLMs exhibit strong performance on a number of the most common
languages (e.g. Swahili or Yoruba), but prioritise support for the
languages with the most speakers first, resulting in piecemeal ability across disparate
languages. We contend that a regionally focussed approach is more efficient, and present a case study for Uganda, a country with high linguistic diversity. We describe the development of Sunflower 14B and 32B, a pair of models based on Qwen 3 with state of the art comprehension in the majority of all Ugandan
languages. These models are open source and can be used to reduce
language barriers in a number of important practical applications.
Read the full abstract ▾
All authors Benjamin Akera, Evelyn Nafula Ouma, Gilbert Yiga, Patrick Walukagga, Phionah Natukunda, Trevor Saaka, Solomon Nsumba, Lilian Teddy Nabukeera, Joel Muhanguzi, Imran Sekalala, Nimpamya Janat Namara, Engineer Bainomugisha, Ernest Mwebaze, John Quinn
Computation and Language
Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible
Imry Ziv, Nur Lan and Emmanuel Chemla
Are large
language models (LLMs) sensitive to the distinction between humanly possible and impossible
languages? This question was recently used in a broader debate on whether LLMs and humans share the same innate learning biases. Previous work has answered it in the positive by comparing LLM learning curves on existing
language datasets and on "impossible"
datasets derived from them via various perturbation functions. Using the same methodology, we examine this claim on a wider set of
languages and impossible perturbations. We find that in most cases, GPT-2 learns each
language and its impossible counterpart equally easily, in contrast to previous findings. We also apply a more lenient condition by testing whether GPT-2 provides any kind of separation between the whole sets of natural vs. impossible
languages, based on cross-linguistic variance in metrics derived from the learning curves. Taken together, these perspectives show that GPT-2 provides no systematic separation between the possible and the impossible.
Read the full abstract ▾
Computation and Language
CARPAS: Towards Content-Aware Refinement of Provided Aspects for Summarization in Large Language Models
Yong-En Tian, Yu-Chien Tang, An-Zi Yen et al. (+1 more)
Aspect-based summarization has attracted significant
attention for its ability to generate more fine-grained and user-aligned summaries. While most existing approaches assume a set of predefined aspects as input, real-world scenarios often present challenges where these given aspects may be incomplete, irrelevant, or entirely missing from the document. Users frequently expect systems to adaptively refine or filter the provided aspects based on the actual content. In this paper, we initiate this novel task setting, termed Content-Aware Refinement of Provided Aspects for Summarization (CARPAS), with the aim of dynamically adjusting the provided aspects based on the document context before summarizing. We construct three new
datasets and conduct pilot experiments with four representative prompting strategies. Our analysis reveals that LLMs often over-generate aspects, resulting in excessively long and misaligned summaries. Building on this observation, we introduce a two-stage framework that derives lightweight scope guidance before aspect refinement and summarization, improving focus and reducing over-generation. Our extensive experiments show that the proposed approach significantly improves performance across all
datasets. Moreover, our deeper analyses uncover LLMs' compliance when the requested number of aspects differs from their own estimations, establishing a crucial insight for the deployment of LLMs in similar real-world applications.
Read the full abstract ▾
All authors Yong-En Tian, Yu-Chien Tang, An-Zi Yen, Wen-Chih Peng
Computation and Language
Quantifying Data Contamination in Psychometric Evaluations of LLMs
Jongwook Han, Woojung Song, Jonggeun Lee et al. (+1 more)
Recent studies apply psychometric questionnaires to Large
Language Models (LLMs) to assess high-level psychological constructs such as values, personality, moral foundations, and dark traits. Although prior work has raised concerns about possible data contamination from psychometric inventories, which may threaten the reliability of such evaluations, there has been no systematic attempt to quantify the extent of this contamination. To address this gap, we propose a framework to systematically measure data contamination in psychometric evaluations of LLMs, evaluating three aspects: (1) item memorization, (2) evaluation memorization, and (3) target score matching. Applying this framework to 21 models from major families and four widely used psychometric inventories, we provide evidence that popular inventories such as the Big Five Inventory (BFI-44) and Portrait Values Questionnaire (PVQ-40) exhibit strong contamination, where models not only memorize items but can also adjust their responses to achieve specific target scores.
Read the full abstract ▾
All authors Jongwook Han, Woojung Song, Jonggeun Lee, Yohan Jo
Computation and Language
NurseLLM: The First Specialized Language Model for Nursing
Md Tawkat Islam Khondaker, Julia Harrington and Shady Shehata
Recent advancements in large
language models (LLMs) have significantly transformed medical systems. However, their potential within specialized domains such as nursing remains largely underexplored. In this work, we introduce NurseLLM, the first nursing-specialized LLM tailored for multiple choice question-answering (MCQ) tasks. We develop a multi-stage data generation pipeline to build the first large scale nursing MCQ
dataset to train LLMs on a broad spectrum of nursing topics. We further introduce multiple nursing
benchmarks to enable rigorous evaluation. Our extensive experiments demonstrate that NurseLLM outperforms SoTA general-purpose and medical-specialized LLMs of comparable size on different
benchmarks, underscoring the importance of a specialized LLM for the nursing domain. Finally, we explore the role of
reasoning and multi-
agent collaboration systems in nursing, highlighting their promise for future research and applications.
Read the full abstract ▾
Computation and Language
More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning
Yike Zhao, Simin Guo, Ziqing Yang et al. (+3 more)
The
reasoning capabilities of Large
Language Models (LLMs) play a critical role in many downstream tasks, yet depend strongly on the quality of training data. Despite various proposed data construction methods, their practical utility in real-world pipelines remains underexplored. In this work, we conduct a comprehensive analysis of open-source
datasets and data synthesis techniques for mathematical
reasoning, evaluating them under a unified pipeline designed to mirror training and deployment scenarios. We further distill effective data selection strategies and identify practical methods suitable for industrial applications. Our findings highlight that structuring data in more interpretable formats, or distilling from stronger models often outweighs simply scaling up data volume. This study provides actionable guidance for integrating training data to enhance LLM capabilities, supporting both cost-effective data curation and scalable model enhancement. We hope this work will inspire further research on how to balance "more data" versus "better data" for real-world
reasoning tasks.
Read the full abstract ▾
All authors Yike Zhao, Simin Guo, Ziqing Yang, Shifan Han, Dahua Lin, Fei Tan
Computation and Language
Reasoning for Hierarchical Text Classification: The Case of Patents
Lekang Jiang, Wenjun Sun and Stephan Goetz
Hierarchical text classification (HTC) assigns documents to multiple levels of a pre-defined taxonomy. Automated patent subject classification represents one of the hardest HTC scenarios because of professional difficulties and extensive labels. Prior approaches only output a flat label set, which offers little insight into the reason behind predictions. Therefore, we propose
Reasoning for Hierarchical Classification (RHC), a novel framework that reformulates HTC as a step-by-step
reasoning task to sequentially deduce hierarchical labels. RHC trains large
language models (LLMs) in two stages: a cold-start stage that aligns outputs with chain-of-thought (CoT)
reasoning format and a reinforcement learning (RL) stage to enhance multi-step
reasoning ability. RHC demonstrates four advantages in our experiments. (1) Effectiveness: RHC surpasses previous baselines and outperforms the supervised fine-tuning counterparts by approximately 3% in accuracy and macro F1. (2) Explainability: RHC produces natural-
language justifications before prediction to facilitate human inspection. (3) Scalability: RHC scales favorably with model size with larger gains compared to standard fine-tuning. (4) Applicability: Beyond patents, we further demonstrate that RHC achieves state-of-the-art performance on other widely used HTC
benchmarks, which highlights its broad applicability.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
A Multi-Agent Framework for Stateful Inference-Time Search
Arshika Lalan, Rajat Ghosh, Aditya Kolsur et al. (+1 more)
Recent work explores agentic inference-time techniques to perform structured, multi-step
reasoning. However, stateless inference often struggles on multi-step tasks due to the absence of persistent state. Moreover, task-specific fine-tuning or instruction-tuning often achieve surface-level code generation but remain brittle on tasks requiring deeper
reasoning and long-horizon dependencies. To address these limitations, we propose stateful multi-
agent evolutionary search, a training-free framework that departs from prior stateless approaches by combining (i) persistent inference-time state, (ii)
adversarial mutation, and (iii) evolutionary preservation. We demonstrate its effectiveness in automated unit test generation through the generation of edge cases. We generate robust edge cases using an evolutionary search process, where specialized
agents sequentially propose, mutate, and score candidates. A controller maintains persistent state across generations, while evolutionary preservation ensures diversity and exploration across all possible cases. This yields a generalist
agent capable of discovering robust, high-coverage edge cases across unseen codebases. Experiments show our stateful multi-
agent inference framework achieves substantial gains in coverage over stateless single-step baselines, evaluated on prevalent unit-testing
benchmarks such as HumanEval and TestGenEvalMini and using three diverse LLM families - Llama, Gemma, and GPT. These results indicate that combining persistent inference-time state with evolutionary search materially improves unit-test generation.
Read the full abstract ▾
All authors Arshika Lalan, Rajat Ghosh, Aditya Kolsur, Debojyoti Dutta
Computation and Language
Comparing Human and Language Models Sentence Processing Difficulties on Complex Structures
Samuel Joseph Amouyal, Aya Meltzer-Asscher and Jonathan Berant
Large
language models (LLMs) that fluently converse with humans are a reality - but do LLMs experience human-like processing difficulties? We systematically compare human and LLM sentence comprehension across seven challenging linguistic structures. We collect sentence comprehension data from humans and five families of state-of-the-art LLMs, varying in size and training procedure in a unified experimental framework. Our results show LLMs overall struggle on the target structures, but especially on garden path (GP) sentences. Indeed, while the strongest models achieve near perfect accuracy on non-GP structures (93.7% for GPT-5), they struggle on GP structures (46.8% for GPT-5). Additionally, when ranking structures based on average performance, rank correlation between humans and models increases with parameter count. For each target structure, we also collect data for their matched baseline without the difficult structure. Comparing performance on the target vs. baseline sentences, the performance gap observed in humans holds for LLMs, with two exceptions: for models that are too weak performance is uniformly low across both sentence types, and for models that are too strong the performance is uniformly high. Together, these reveal convergence and divergence in human and LLM sentence comprehension, offering new insights into the similarity of humans and LLMs.
Read the full abstract ▾
Computation and Language
TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning
Manish Nagaraj, Sakshi Choudhary, Utkarsh Saxena et al. (+2 more)
Instruction tuning is essential for aligning large
language models (LLMs) to downstream tasks and commonly relies on large, diverse corpora. However, small, high-quality subsets, known as coresets, can deliver comparable or superior results, though curating them remains challenging. Existing methods often rely on coarse, sample-level signals like gradients, an approach that is computationally expensive and overlooks fine-grained features. To address this, we introduce TRIM (
Token Relevance via Interpretable Multi-layer
Attention), a forward-only,
token-centric framework. Instead of using gradients, TRIM operates by matching underlying representational patterns identified via
attention-based "fingerprints" from a handful of target samples. Such an approach makes TRIM highly efficient and uniquely sensitive to the structural features that define a task. Coresets selected by our method consistently outperform state-of-the-art baselines by up to 9% on downstream tasks and even surpass the performance of full-data fine-tuning in some settings. By avoiding expensive backward passes, TRIM achieves this at a fraction of the computational cost. These findings establish TRIM as a scalable and efficient alternative for building high-quality instruction-tuning
datasets.
Read the full abstract ▾
All authors Manish Nagaraj, Sakshi Choudhary, Utkarsh Saxena, Deepak Ravikumar, Kaushik Roy
Computation and Language
Opt-ICL at LeWiDi-2025: Maximizing In-Context Signal from Rater Examples via Meta-Learning
Taylor Sorensen and Yejin Choi
Many natural
language processing (NLP) tasks involve subjectivity, ambiguity, or legitimate disagreement between annotators. In this paper, we outline our system for modeling human variation. Our system leverages
language models' (LLMs) in-context learning abilities, along with a two-step meta-learning training procedure for 1) post-training on many
datasets requiring in-context learning and 2) specializing the model via in-context meta-learning to the particular data distribution of interest. We also evaluate the performance of our system submission to the Learning With Disagreements (LeWiDi) competition, where it was the overall winner on both tasks. Additionally, we perform an ablation study to measure the importance of each system component. We find that including rater examples in-context is crucial for our system's performance,
dataset-specific fine-tuning is helpful on the larger
datasets, post-training on other in-context
datasets is helpful on one of the competition
datasets, and that performance improves with model scale.
Read the full abstract ▾
Computation and Language
TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription
Guo Yutong, Wanying Wang, Yue Wu et al. (+2 more)
Table Visual Question Answering (Table VQA) is typically addressed by large
vision-
language models (VLMs). While such models can answer directly from images, they often miss fine-grained details unless scaled to very large sizes, which are computationally prohibitive, especially for mobile deployment. A lighter alternative is to have a small VLM perform OCR and then use a large
language model (LLM) to reason over structured outputs such as Markdown tables. However, these representations are not naturally optimized for LLMs and still introduce substantial errors. We propose TALENT (Table VQA via Augmented
Language-Enhanced Natural-text Transcription), a lightweight framework that leverages dual representations of tables. TALENT
prompts a small VLM to produce both OCR text and natural
language narration, then combines them with the question for
reasoning by an LLM. This reframes Table VQA as an LLM-centric
multimodal reasoning task, where the VLM serves as a perception-narration module rather than a monolithic solver. Additionally, we construct ReTabVQA, a more challenging Table VQA
dataset requiring multi-step quantitative
reasoning over table images. Experiments show that TALENT enables a small VLM-LLM combination to match or surpass a single large VLM at significantly lower computational cost on both public
datasets and ReTabVQA.
Read the full abstract ▾
All authors Guo Yutong, Wanying Wang, Yue Wu, Zichen Miao, Haoyu Wang
Computation and Language
Modeling Sarcastic Speech: Semantic and Prosodic Cues in a Speech Synthesis Framework
Zhu Li, Yuqing Zhang, Xiyuan Gao et al. (+2 more)
Sarcasm is a pragmatic phenomenon in which speakers convey meanings that diverge from literal content, relying on an interaction between semantics and prosodic expression. However, how these cues jointly contribute to the recognition of sarcasm remains poorly understood. We propose a computational framework that models sarcasm as the integration of semantic interpretation and prosodic realization. Semantic cues are derived from an LLaMA 3 model fine-tuned to capture discourse-level markers of sarcastic intent, while prosodic cues are extracted through semantically aligned utterances drawn from a database of sarcastic
speech, providing prosodic exemplars of sarcastic delivery. Using a
speech synthesis testbed, perceptual evaluations show that semantic and prosodic cues enhance perceived sarcasm, with the combined system achieving the best downstream F1 while maintaining high subjective sarcasm ratings. These findings highlight the complementary roles of semantics and prosody in pragmatic interpretation and illustrate how modeling can shed light on the mechanisms underlying sarcastic communication.
Read the full abstract ▾
All authors Zhu Li, Yuqing Zhang, Xiyuan Gao, Shekhar Nayak, Matt Coler
Computation and Language · Cross-listed from cs.AI
The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemas
Baixuan Xu, Tianshi Zheng, Zhaowei Wang et al. (+4 more)
Enabling LLMs to effectively operate long-horizon task which requires long-term planning and multiple interactions is essential for open-world autonomy. Conventional methods adopt planning with actions where a executable action list would be provided as reference. However, this action representation choice would be impractical when the environment action space is combinatorial exploded (e.g., open-ended real world). This naturally leads to a question: As environmental action space scales, what is the optimal action representation for long-horizon
agents? In this paper, we systematically study the effectiveness of two different action representations. The first one is conventional planning with actions (PwA) which is predominantly adopted for its effectiveness on existing
benchmarks. The other one is planning with schemas (PwS) which instantiate an action schema into action lists (e.g., "move [OBJ] to [OBJ]" -> "move apple to desk") to ensure concise action space and reliable scalability. This alternative is motivated by its
alignment with human cognition and its compliance with environment-imposed action format restriction. We propose cognitive bandwidth perspective as a conceptual framework to qualitatively understand the differences between these two action representations and empirically observe a representation-choice inflection point between ALFWorld (~35 actions) and SciWorld (~500 actions), which serve as evidence of the need for scalable representations. We further conduct controlled experiments to study how the location of this inflection point interacts with different model capacities: stronger planning proficiency shifts the inflection rightward, whereas better schema instantiation shifts it leftward. Finally, noting the suboptimal performance of PwS
agents, we provide an actionable guide for building more capable PwS
agents for better scalable autonomy.
Read the full abstract ▾
All authors Baixuan Xu, Tianshi Zheng, Zhaowei Wang, Hong Ting Tsang, Weiqi Wang, Tianqing Fang, Yangqiu Song
Computation and Language
All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
Miriam Wanner, Leif Azzopardi, Paul Thomas et al. (+3 more)
Existing methods for evaluating the factuality of large
language model (LLM) responses treat all claims as equally important. This results in misleading evaluations when vital information is missing or incorrect as it receives the same weight as peripheral details, raising the question: how can we reliably detect such differences when there are errors in key information? Current approaches that measure factuality tend to be insensitive to omitted or false key information. To investigate this lack of sensitivity, we construct VITALERRORS, a
benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information. Using this
dataset, we demonstrate the insensitivities of existing evaluation metrics to key information errors. To address this gap, we introduce VITAL, a set of metrics that provide greater sensitivity in measuring the factuality of responses by incorporating the relevance and importance of claims with respect to the query. Our analysis demonstrates that VITAL metrics more reliably detect errors in key information than previous methods. Our
dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.
Read the full abstract ▾
All authors Miriam Wanner, Leif Azzopardi, Paul Thomas, Soham Dan, Benjamin Van Durme, Nick Craswell
Computation and Language
Accelerating Diffusion LLM Inference via Local Determinism Propagation
Fanheng Kong, Jingyuan Zhang, Yahui Liu et al. (+4 more)
Diffusion large
language models (dLLMs) represent a significant advancement in text generation, offering parallel
token decoding capabilities. However, existing open-source implementations suffer from quality-speed trade-offs that impede their practical deployment. Conservative sampling strategies typically decode only the most confident
token per step to ensure quality (i.e., greedy decoding), at the cost of inference efficiency due to repeated redundant refinement iterations–a phenomenon we term delayed decoding. Through systematic analysis of dLLM decoding dynamics, we characterize this delayed decoding behavior and propose a training-free adaptive parallel decoding strategy, named LocalLeap, to address these inefficiencies. LocalLeap is built on two fundamental empirical principles: local determinism propagation centered on high-confidence anchors and progressive spatial consistency decay. By applying these principles, LocalLeap identifies anchors and performs localized relaxed parallel decoding within bounded neighborhoods, achieving substantial inference step reduction through early commitment of already-determined
tokens without compromising output quality. Comprehensive evaluation on various
benchmarks demonstrates that LocalLeap achieves 6.94$\times$ throughput improvements and reduces decoding steps to just 14.2% of the original requirement, achieving these gains with negligible performance impact. The source codes are available at: https://github.com/friedrichor/LocalLeap.
Read the full abstract ▾
All authors Fanheng Kong, Jingyuan Zhang, Yahui Liu, Zirui Wu, Yu Tian, Victoria W., Guorui Zhou
Computation and Language
LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish
Fred Philippy, Laura Bernardy, Siwen Guo et al. (+2 more)
Instruction tuning has become a key technique for enhancing the performance of large
language models, enabling them to better follow human
prompts. However, low-resource
languages such as Luxembourgish face severe limitations due to the lack of high-quality instruction
datasets. Traditional reliance on machine translation often introduces semantic misalignment and cultural inaccuracies. In this work, we address these challenges by creating a cross-lingual instruction tuning
dataset for Luxembourgish, without resorting to machine-generated translations into it. Instead, by leveraging aligned data from English, French, and German, we build a high-quality
dataset that preserves linguistic and cultural nuances. We provide evidence that cross-lingual instruction tuning not only improves representational
alignment across
languages but also the model's generative capabilities in Luxembourgish. This highlights how cross-lingual data curation can avoid the common pitfalls of machine-translated data and directly benefit low-resource
language development.
Read the full abstract ▾
All authors Fred Philippy, Laura Bernardy, Siwen Guo, Jacques Klein, Tegawendé F. Bissyandé
Computation and Language
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
Amir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan et al. (+1 more)
While automatic metrics drive progress in Machine Translation (MT) and Text Summarization (TS), existing metrics have been developed and validated almost exclusively for English and other high-resource
languages. This narrow focus leaves Indian
languages, spoken by over 1.5 billion people, largely overlooked, casting doubt on the universality of current evaluation practices. To address this gap, we introduce ITEM, a large-scale
benchmark that systematically evaluates the
alignment of 29 automatic metrics with human judgments across six major Indian
languages, enriched with fine-grained annotations. Our extensive evaluation, covering agreement with human judgments, sensitivity to outliers,
language-specific reliability, inter-metric correlations, and resilience to controlled perturbations reveals four central findings: (1) LLM-based evaluators show the strongest
alignment with human judgments at both segment and system levels; (2) outliers exert a significant impact on metric-human agreement; (3) In TS, metrics are more effective at capturing content fidelity, whereas in MT, they better reflect fluency; and (4) Metrics differ in their robustness and sensitivity when subjected to diverse perturbations. Collectively, these findings offer critical guidance for advancing metric design and evaluation in Indian
languages.
Read the full abstract ▾
All authors Amir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri Koto
Computation and Language
Does Local News Stay Local?: Online Content Shifts in Sinclair-Acquired Stations
Miriam Wanner, Sophia Hager and Anjalie Field
Local news stations are often considered to be reliable sources of non-politicized information, particularly local concerns that residents care about. Because these stations are trusted news sources, viewers are particularly susceptible to the information they report. The Sinclair Broadcast group is a broadcasting company that has acquired many local news stations in the last decade. We investigate the effects of local news stations being acquired by Sinclair: how does coverage change? We use computational methods to investigate changes in internet content put out by local news stations before and after being acquired by Sinclair and in comparison to national news outlets. We find that there is clear evidence that local news stations report more frequently on national news at the expense of local topics, and that their coverage of polarizing national topics increases.
Read the full abstract ▾
Computation and Language
Yuntao Gui and James Cheng
Despite their remarkable natural
language understanding capabilities, Large
Language Models (LLMs) have been underutilized for
retrieval tasks. We present Search-R3, a novel framework that addresses this limitation by adapting LLMs to generate search
embeddings as a direct output of their
reasoning process. Our approach exploits LLMs' chain-of-thought capabilities, allowing them to produce more effective
embeddings by
reasoning step-by-step through complex semantic analyses. We implement this through three complementary mechanisms. (1) a supervised learning stage enables the model's ability to produce quality
embeddings, (2) a reinforcement learning (RL) methodology that optimizes
embedding generation alongside
reasoning, and (3) a specialized RL environment that efficiently handles evolving
embedding representations without requiring complete corpus re-encoding at each training iteration. Our extensive evaluations on diverse
benchmarks demonstrate that Search-R3 significantly outperforms prior methods by unifying the
reasoning and
embedding generation processes. This integrated post-training approach represents a substantial advancement in handling complex knowledge-intensive tasks that require both sophisticated
reasoning and effective information
retrieval. Project page: https://github.com/ytgui/Search-R3
Read the full abstract ▾
Computation and Language
Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities
Rajvee Sheth, Samridhi Raj Sinha, Mahavir Patil et al. (+2 more)
Amidst the rapid advances of large
language models (LLMs), most LLMs still struggle with mixed-
language inputs, limited Codeswitching (CSW)
datasets, and evaluation biases, which hinder their deployment in
multilingual societies. This survey provides the first comprehensive analysis of CSW-aware LLM research, reviewing 327 studies spanning five research areas, 15+ NLP tasks, 30+
datasets, and 80+
languages. We categorize recent advances by architecture, training strategy, and evaluation methodology, outlining how LLMs have reshaped CSW modeling and identifying the challenges that persist. The paper concludes with a roadmap that emphasizes the need for inclusive
datasets, fair evaluation, and linguistically grounded models to achieve truly
multilingual capabilities https://github.com/lingo-iitgn/awesome-code-mixing/.
Read the full abstract ▾
All authors Rajvee Sheth, Samridhi Raj Sinha, Mahavir Patil, Himanshu Beniwal, Mayank Singh
Computation and Language
Auditing a KB Elicitation of Frontier LLM Knowledge: A Multi-dimensional Analysis of GPTKB v1.5
Shrestha Ghosh, Luca Giordano, Yujia Hu et al. (+2 more)
LLMs are remarkable artifacts that have revolutionized a range of knowledge-intensive tasks. A significant contributor is their factual knowledge, which, to date, remains poorly understood, and is usually analyzed from biased samples. In this paper, we provide a framework and the results of a multi-dimensional analysis of GPTKB v1.5 (Hu et al., 2025a), a recursively elicited Knowledge Base (KB) of 100 million facts (or beliefs) of a frontier LLM, namely, GPT-4.1. Given the scale of the elicited facts, we provide a multi-dimensional approach to qualitatively and quantitatively analyze these facts as opposed to the mainstream fact completion
benchmarks, which are prone to availability bias. We find that the models' factual knowledge differs quite significantly from established knowledge bases, and that its accuracy is significantly lower than indicated by previous
benchmarks. We also find that inconsistency, ambiguity and
hallucinations are major issues, shedding light on future research opportunities in neuro-symbolic AI concerning extraction, consolidation and verification of factual LLM knowledge.
Read the full abstract ▾
All authors Shrestha Ghosh, Luca Giordano, Yujia Hu, Tuan-Phong Nguyen, Simon Razniewski
Computation and Language
Native Hybrid Attention for Efficient Sequence Modeling
Jusen Du, Jiaxi Hu, Tao Zhang et al. (+2 more)
Transformers excel at sequence modeling but face quadratic complexity, while linear
attention offers improved efficiency but often compromises recall accuracy over long contexts. In this work, we introduce Native Hybrid
Attention (NHA), a novel hybrid architecture of linear and full
attention that integrates both intra & inter-layer hybridization into a unified layer design. NHA maintains long-term context in key-value slots updated by a linear RNN, and augments them with short-term
tokens from a sliding window. A single softmax
attention operation is then applied over all keys and values, enabling per-
token and per-head context-dependent weighting without requiring additional fusion parameters. The inter-layer behavior is controlled through a single hyperparameter, the sliding window size, which allows smooth adjustment between purely linear and full
attention while keeping all layers structurally uniform. Experimental results show that NHA surpasses
Transformers and other hybrid baselines on recall-intensive and commonsense
reasoning tasks. Furthermore, pretrained LLMs can be structurally hybridized with NHA, achieving competitive accuracy while delivering significant efficiency gains. Code is available at https://github.com/JusenD/NHA.
Read the full abstract ▾
All authors Jusen Du, Jiaxi Hu, Tao Zhang, Weigao Sun, Yu Cheng