Computation and Language
Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
Chengyu Huang, Sheng-Yen Chou, Zhengxin Zhang et al. (+1 more)
Self-play has recently emerged as a promising paradigm for post-training Large
Language Models (LLMs). In self-play, the target LLM creates the task input (e.g., a question), which it then addresses itself by producing a task output (e.g., an answer). A reward model evaluates the output, and the rewards are used to train the LLM, typically via Reinforcement Learning (RL). A key benefit of self-play for post-training LLMs is its minimal supervision costs: self-play avoids the need for high-quality input-output pairs traditionally constructed by humans or expensive proprietary models. Existing work, however, explores self-play only for verifiable tasks, such as math and coding, for which objective ground truth is available and easily checkable. In this paper, we seek to extend self-play to more realistic open-ended tasks. We propose POP, a self-play framework that uses the same LLM to synthesize evaluation rubrics along with each input-output pair. The rubric is used to evaluate outputs and train the model. Crucially, we ground the framework on a content-rich pretraining corpus to (1) enable an exploitable generation-verification gap and reduce reward hacking, and (2) prevent mode collapse. On Qwen-2.5-7B, POP increases performance of both the pretrained base model and instruction-tuned model on multiple tasks ranging from long-form healthcare QA to creative writing and instruction following.
Read the full abstract ▾
All authors Chengyu Huang, Sheng-Yen Chou, Zhengxin Zhang, Claire Cardie
Computation and Language
Culturally uneven urban perception in large language models
Rong Zhao, Wanqi Liu, Zhizhou Sha et al. (+3 more)
Large
language models (LLMs) are increasingly used to describe and evaluate cities, yet the cultural structure of their urban judgments remains understudied. Here we introduce a measurement framework for testing whether LLM-based urban perception is culturally neutral, using a globally stratified street-view image
dataset. Open-ended descriptions and structured scores generated by three frontier
multimodal models all show that the neutral baseline lies closer to regional framings associated with Europe and North America than to other cultural framings. Comparisons between AI and human urban perception further show that prompting can move AI responses closer to specific regional human descriptions, but fails to recover the variety and diversity of human responses, flattening observed demographic patterns and introducing sentiment-based self-favouring bias. These results indicate a systematic risk in treating AI as a neutral tool for urban tasks, especially when model outputs are used to compare, evaluate or represent cities across cultural contexts.
Read the full abstract ▾
All authors Rong Zhao, Wanqi Liu, Zhizhou Sha, Nanxi Su, Yecheng Zhang, Ying Long
Computation and Language
TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs
Ziyi Wang, Chen Zhang, Wenjun Peng et al. (+2 more)
Explainability for Large
Language Model (LLM)
agents is especially challenging in interactive, partially observable settings, where decisions depend on evolving beliefs and other
agents. We present TriEx, a tri-view explainability framework that instruments sequential decision making with aligned artifacts: (i) structured first-person self-
reasoning bound to an action, (ii) explicit second-person belief states about opponents updated over time, and (iii) third-person oracle audits grounded in environment-derived reference signals. This design turns explanations from free-form narratives into evidence-anchored objects that can be compared and checked across time and perspectives. Using imperfect-information strategic games as a controlled testbed, we show that TriEx enables scalable analysis of explanation faithfulness, belief dynamics, and evaluator reliability, revealing systematic mismatches between what
agents say, what they believe, and what they do. Our results highlight explainability as an interaction-dependent property and motivate multi-view, evidence-grounded evaluation for LLM
agents. Code is available at https://github.com/Einsam1819/TriEx.
Read the full abstract ▾
All authors Ziyi Wang, Chen Zhang, Wenjun Peng, Qi Wu, Xinyu Wang
Computation and Language · Cross-listed from cs.LG
MoBayes: A Modular Bayesian Framework for Separating Reasoning from Language in Conversational Clinical Decision Support
Yusuf Kesmen, Fay Elhassan, Jiayi Ma et al. (+7 more)
Large
language models (LLMs) are increasingly used for conversational clinical decision support, yet they conflate next
token prediction with probabilistic decision making. We argue that this conflation reflects an architectural limitation: such systems lack explicit posterior tracking, controllable abstention thresholds, and auditable
reasoning chains. We introduce MoBayes, a Modular
Bayesian dialogue framework that separates
reasoning from
language. The LLM acts only as a
language interface, parsing patient conversation into structured observations, while a
Bayesian module performs probabilistic inference over these observations to update posteriors, select follow-up questions via expected-information-gain and determine when to stop or defer through calibrated decision thresholds. This design enables explicit posterior tracking, controllable selective decision-making, and replaceable population-specific statistical backends without retraining the
language model. Across empirical and LLM-generated knowledge bases, MoBayes outperforms standalone frontier LLM doctors, including matched model-family comparisons where inexpensive sensor models paired with MoBayes exceed larger autonomous models at lower cost. The advantage persists under
adversarial patient communication styles and across varying diagnostic scenarios. These results suggest that reliable conversational clinical decision support systems should separate probabilistic
reasoning from
language generation rather than scaling model size alone. Code is available at https://anonymous.4open.science/r/MoBayes/
Read the full abstract ▾
All authors Yusuf Kesmen, Fay Elhassan, Jiayi Ma, Julien Stalhandske, Yena Chang, David Sasu, Alexandra Kulinkina, Akhil Arora, Lars Klein, Mary-Anne Hartley
Computation and Language · Cross-listed from cs.LG
Continuous Semantic Caching for Low-Cost LLM Serving
Baran Atalar, Xutong Liu, Jinhang Zuo et al. (+3 more)
As Large
Language Models (LLMs) become increasingly popular, caching responses so that they can be reused by users with semantically similar queries has become a vital strategy for reducing inference costs and latency. Existing caching frameworks have proposed to decide which query responses to cache by assuming a finite, known universe of discrete queries and learning their serving costs and arrival probabilities. As LLMs' pool of users and queries expands, however, such an assumption becomes increasingly untenable: real-world LLM queries reside in an infinite, continuous
embedding space. In this paper, we establish the first rigorous theoretical framework for semantic LLM response caching in continuous query space under uncertainty. To bridge the gap between discrete optimization and continuous representation spaces, we introduce dynamic $\epsilon$-net discretization coupled with Kernel Ridge Regression. This design enables the system to formally quantify estimation uncertainty and generalize partial feedback on LLM query costs across continuous semantic query neighborhoods. We develop both offline learning and online adaptive
algorithms optimized to reduce switching costs incurred by changing the cached responses. We prove that our online
algorithm achieves a sublinear regret bound against an optimal oracle, which reduces to existing bounds for discrete query models. Extensive empirical evaluations demonstrate that our framework approximates the continuous optimal cache well while also reducing computational and switching overhead compared to existing methods.
Read the full abstract ▾
All authors Baran Atalar, Xutong Liu, Jinhang Zuo, Siwei Wang, Wei Chen, Carlee Joe-Wong
Computation and Language · Cross-listed from cs.CV
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
Yiyang Du, Zhanqiu Guo, Haochen Zhang et al. (+5 more)
Vision-
Language-Action Models (VLAs) inherit their visual and linguistic capabilities from
Vision-
Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the
embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of
alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity
estimator to select the most VLA-aligned candidates from a large VLM pool, and produce EmbodiedMidtrain-1M, a VLM
dataset that we further use to mid-train the VLM before downstream VLA fine-tuning. Experiments on both simulated
robot manipulation
benchmarks and real-world tasks show consistent VLA performance gains over off-the-shelf checkpoints across VLM backbones and VLA action heads. These gains extend to downstream tasks not used to guide data selection. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both
dataset-level and sample-level
alignment signals, favoring spatial
reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research.
Read the full abstract ▾
All authors Yiyang Du, Zhanqiu Guo, Haochen Zhang, Annabella Macaluso, Xin Ye, Liu Ren, Yonatan Bisk, Chenyan Xiong
Computation and Language · Cross-listed from cs.CY
Frictionless Love: Associations Between AI Companion Roles and Behavioral Addiction
Vibhor Agarwal, Ke Zhou, Edyta Paulina Bogucka et al. (+1 more)
AI companion chatbots increasingly shape how people seek social and emotional connection, sometimes substituting for relationships with romantic partners, friends, teachers, or even therapists. When these systems adopt those metaphorical roles, they are not neutral: such roles structure people's ways of interacting, distribute perceived AI harms and benefits, and may reflect behavioral addiction signs. Yet these role-dependent risks remain poorly understood. We analyze 248,830 posts from seven prominent Reddit communities describing interactions with AI companions. We identify ten recurring metaphorical roles (for example, soulmate, philosopher, and coach) and show that each role supports distinct ways of interacting. We then extract the perceived AI harms and AI benefits associated with these role-specific interactions and link them to behavioral addiction signs, all of which has been inferred from the text in the posts. AI soulmate companions are associated with romance-centered ways of interacting, offering emotional support but also introducing emotional manipulation and distress, culminating in strong attachment. In contrast, AI coach and guardian companions are associated with practical benefits such as personal growth and task support, yet are nonetheless more frequently associated with behavioral addiction signs such as daily life disruptions and damage to offline relationships. These findings show that metaphorical roles are a central ethical design concern for responsible AI companions.
Read the full abstract ▾
All authors Vibhor Agarwal, Ke Zhou, Edyta Paulina Bogucka, Daniele Quercia
Computation and Language
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Md Nayem Uddin, Kumar Shubham, Eduardo Blanco et al. (+2 more)
Personalized
agents that interact with users over long periods must maintain persistent
memory across sessions and update it as circumstances change. However, existing
benchmarks predominantly frame long-term
memory evaluation as fact
retrieval from past conversations, providing limited insight into
agents' ability to consolidate
memory over time or handle frequent knowledge updates. We introduce Memora, a long-term
memory benchmark spanning weeks to months long user conversations. The
benchmark evaluates three
memory-grounded tasks: remembering,
reasoning, and recommending. To ensure data quality, we employ automated
memory-grounding checks and human evaluation. We further introduce Forgetting-Aware
Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated
memory when evaluating long-term
memory. Evaluations of four LLMs and six
memory agents reveal frequent reuse of invalid
memories and failures to reconcile evolving
memories.
Memory agents offer marginal improvements, exposing shortcomings in long-term
memory for personalized
agents.
Read the full abstract ▾
All authors Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, Gengyu Wang
Computation and Language · Cross-listed from cs.CY
Bias in the Tails: How Name-conditioned Evaluative Framing in Resume Summaries Destabilizes LLM-based Hiring
Huy Nghiem, Phuong-Anh Nguyen-Le, Sy-Tuyen Ho et al. (+1 more)
Research has documented LLMs' name-based bias in hiring and salary recommendations. In this paper, we instead consider a setting where LLMs generate candidate summaries for downstream assessment. In a large-scale controlled study, we analyze nearly one million resume summaries produced by 4 models under systematic race-gender name perturbations, using synthetic resumes and real-world job postings. By decomposing each summary into resume-grounded factual content and evaluative framing, we find that factual content remains largely stable, while evaluative
language exhibits subtle name-conditioned variation concentrated in the extremes of the distribution, especially in open-source models. Our hiring simulation demonstrates how evaluative summary transforms directional harm into symmetric instability that might evade conventional
fairness audit, highlighting a potential pathway for LLM-to-LLM automation bias.
Read the full abstract ▾
All authors Huy Nghiem, Phuong-Anh Nguyen-Le, Sy-Tuyen Ho, Hal Daume III
Computation and Language · Cross-listed from cs.LG
Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders
Het Patel, Tiejin Chen, Hua Wei et al. (+2 more)
Large
language models can be uncertain yet correct, or confident yet wrong, raising the question of whether their output-level uncertainty and their actual correctness are driven by the same internal mechanisms or by distinct feature populations. We introduce a 2x2 framework that partitions model predictions along correctness and confidence axes, and uses sparse autoencoders to identify features associated with each dimension independently. Applying this to Llama-3.1-8B and Gemma-2-9B, we identify three feature populations that play fundamentally different functional roles. Pure uncertainty features are functionally essential: suppressing them severely degrades accuracy. Pure incorrectness features are functionally inert: despite showing statistically significant activation differences between correct and incorrect predictions, the majority produce near-zero change in accuracy when suppressed. Confounded features that encode both signals are detrimental to output quality, and targeted suppression of them yields a 1.1% accuracy improvement and a 75%
entropy reduction, with effects transferring across the ARC-Challenge and RACE
benchmarks. The feature categories are also informationally distinct: the activations of just 3 confounded features from a single mid-network layer predict model correctness (AUROC ~0.79), enabling selective abstention that raises accuracy from 62% to 81% at 53% coverage. The results demonstrate that uncertainty and correctness are distinct internal phenomena, with implications for interpretability and targeted inference-time intervention.
Read the full abstract ▾
All authors Het Patel, Tiejin Chen, Hua Wei, Evangelos E. Papalexakis, Jia Chen
Computation and Language
Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference
Olga Kellert, Sriya Kondury, Candice Koo et al. (+2 more)
Annotation pipelines in Natural
Language Processing (NLP) commonly assume a single latent ground truth per instance and resolve disagreement through label aggregation. Perspectivist approaches challenge this view by treating disagreement as potentially informative rather than erroneous. We present a large-scale analysis of graded health-literacy annotations from 6,323 open-ended COVID-19 responses collected in Ecuador and Peru. Each response was independently labeled by multiple annotators using proportional correctness scores, reflecting the degree to which responses align with normative public-health guidelines, allowing us to analyze the full distribution of judgments rather than aggregated labels. Variance decomposition shows that question-level conceptual difficulty accounts for substantially more variance than annotator identity, indicating that disagreement is structured by the task itself rather than driven by individual raters. Agreement-stratified analyses further reveal that key social-scientific effects, including country, education, and urban-rural differences, vary in magnitude and in some cases reverse direction across levels of inter-annotator agreement. These findings suggest that graded health-literacy evaluation contains both epistemically stable and unstable components, and that aggregating across them can obscure important inferential differences. We therefore argue that strong perspectivist modeling is not only conceptually justified but statistically necessary for valid inference in graded interpretive tasks.
Read the full abstract ▾
All authors Olga Kellert, Sriya Kondury, Candice Koo, Nemika Tyagi, Steffen Eikenberry
Computation and Language
Tracing Relational Knowledge Recall in Large Language Models
Nicholas Popovič and Michael Färber
We study how large
language models recall relational knowledge during text generation, with a focus on identifying latent representations suitable for relation classification via linear probes. Prior work shows how
attention heads and MLPs interact to resolve subject, predicate, and object, but it remains unclear which representations support faithful linear relation classification and why some relation types are easier to capture linearly than others. We systematically evaluate different latent representations derived from
attention head and MLP contributions, showing that per-head
attention contributions to the residual stream are comparatively strong features for linear relation classification. Feature attribution analyses of the trained probes, as well as characteristics of the different relation types, reveal clear correlations between probe accuracy and relation specificity, entity connectedness, and how distributed the signal on which the probe relies is across
attention heads. Finally, we show how
token-level feature attribution of probe predictions can be used to reveal probe behavior in further detail.
Read the full abstract ▾
Computation and Language
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding
Zijie Wang, MohammadHossein Rezaei, Farzana Rashid et al. (+1 more)
Negation is a common and important semantic feature in natural
language, yet Large
Language Models (LLMs) struggle when negation is involved in natural
language understanding tasks. Commonsense knowledge, on the other hand, despite being a well-studied topic, lacks investigations involving negation. In this work, we show that commonsense knowledge with negation is challenging for models to understand. We present a novel approach to automatically augment existing commonsense knowledge corpora with negation, yielding two new corpora containing over 2M triples with if-then relations. In addition, pre-training LLMs on our corpora benefits negation understanding.
Read the full abstract ▾
All authors Zijie Wang, MohammadHossein Rezaei, Farzana Rashid, Eduardo Blanco
Computation and Language
Depression Risk Assessment in Social Media via Large Language Models
Giorgia Gulino and Manuel Petrucci
Depression is one of the most prevalent and debilitating mental health conditions worldwide, frequently underdiagnosed and undertreated. The proliferation of social media platforms provides a rich source of naturalistic linguistic signals for the automated monitoring of psychological well-being. In this work, we propose a system based on Large
Language Models (LLMs) for depression risk assessment in Reddit posts, through multi-label classification of eight depression-associated
emotions and the computation of a weighted severity index. The method is evaluated in a zero-shot setting on the annotated DepressionEmo
dataset (~6,000 posts) and applied in-the-wild to 469,692 comments collected from four subreddits over the period 2024-2025. Our best model, gemma3:27b, achieves micro-F1 = 0.75 and macro-F1 = 0.70, results competitive with purpose-built fine-tuned models (BART: micro-F1 = 0.80, macro-F1 = 0.76). The in-the-wild analysis reveals consistent and temporally stable risk profiles across communities, with marked differences between r/depression and r/anxiety. Our findings demonstrate the feasibility of a cost-effective, scalable approach for large-scale psychological monitoring.
Read the full abstract ▾
Computation and Language
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
Chenxi Zhou, Pengfei Cao, Jiang Li et al. (+4 more)
Post-Training Quantization (PTQ) is critical for the efficient deployment of Large
Language Models (LLMs). While 4-bit quantization is widely regarded as an optimal trade-off, reducing the precision to 2-bit usually triggers a catastrophic “performance cliff.” It remains unclear whether the underlying mechanisms differ fundamentally. Consequently, we conduct a systematic mechanistic analysis, revealing two qualitatively distinct failure modes: Signal Degradation, where the computational patterns remain intact but information precision is impaired by cumulative error; and Computation Collapse, where key components fail to function, preventing correct information processing and destroying the signal in the early layers. Guided by this diagnosis, we conduct mechanism-aware interventions, demonstrating that targeted, training-free repair can mitigate Signal Degradation, but remains ineffective for Computation Collapse. Our findings provide a systematic diagnostic framework for PTQ failures and suggest that addressing Computation Collapse requires structural reconstruction rather than mere compensation.
Read the full abstract ▾
All authors Chenxi Zhou, Pengfei Cao, Jiang Li, Bohan Yu, Jinyu Ye, Jun Zhao, Kang Liu
Computation and Language · Cross-listed from cs.LG
DR-Venus: Towards Frontier Edge-Scale Deep Research Agents with Only 10K Open Data
Venus Team, Sunhao Dai, Yong Deng et al. (+10 more)
Edge-scale deep research
agents based on small
language models are attractive for real-world deployment due to their advantages in cost, latency, and
privacy. In this work, we study how to train a strong small deep research
agent under limited open-data by improving both data quality and data utilization. We present DR-Venus, a frontier 4B deep research
agent for edge-scale deployment, built entirely on open data. Our training recipe consists of two stages. In the first stage, we use agentic supervised fine-tuning (SFT) to establish basic agentic capability, combining strict data cleaning with resampling of long-horizon trajectories to improve data quality and utilization. In the second stage, we apply agentic reinforcement learning (RL) to further improve execution reliability on long-horizon deep research tasks. To make RL effective for small
agents in this setting, we build on IGPO and design turn-level rewards based on information gain and format-aware regularization, thereby enhancing supervision density and turn-level credit assignment. Built entirely on roughly 10K open-data, DR-Venus-4B significantly outperforms prior agentic models under 9B parameters on multiple deep research
benchmarks, while also narrowing the gap to much larger 30B-class systems. Our further analysis shows that 4B
agents already possess surprisingly strong performance potential, highlighting both the deployment promise of small models and the value of test-time scaling in this setting. We release our models, code, and key recipes to support reproducible research on edge-scale deep research
agents.
Read the full abstract ▾
All authors Venus Team, Sunhao Dai, Yong Deng, Jinzhen Lin, Yusheng Song, Guoqing Wang, Xiaofeng Wu, Yuqi Zhou, Shuo Yang, Zhenzhe Ying, Zhanwei Zhang, Changhua Meng, Weiqiang Wang
Computation and Language · Cross-listed from cs.LG
Rethinking Reinforcement Fine-Tuning in LVLM: Convergence, Reward Decomposition, and Generalization
Carter Adams, Rafael Oliveira, Gabriel Almeida et al. (+1 more)
Reinforcement fine-tuning with verifiable rewards (RLVR) has emerged as a powerful paradigm for equipping large
vision-
language models (LVLMs) with agentic capabilities such as tool use and multi-step
reasoning. Despite striking empirical successes, most notably Visual Agentic Reinforcement Fine-Tuning (Visual-ARFT), the theoretical underpinnings of this paradigm remain poorly understood. In particular, two critical questions lack rigorous answers: (i)~how does the composite structure of verifiable rewards (format compliance, answer accuracy, tool executability) affect the convergence of Group Relative Policy Optimization (GRPO), and (ii)~why does training on a small set of tool-augmented tasks transfer to out-of-distribution domains? We address these gaps by introducing the Tool-Augmented Markov Decision Process (TA-MDP), a formal framework that models
multimodal agentic decision-making with bounded-depth tool calls. Within this framework, we establish three main results. First, we prove that GRPO under composite verifiable rewards converges to a first-order stationary point at rate $O(1/\sqrt{T})$ with explicit dependence on the number of reward components and group size (
Theorem~1). Second, we derive a Reward Decomposition
Theorem that bounds the sub-optimality gap between decomposed per-component optimization and joint optimization, providing a precise characterization of when reward decomposition is beneficial (
Theorem~2). Third, we establish a PAC-Bayes generalization bound for tool-augmented policies that explains the strong out-of-distribution transfer observed in Visual-ARFT (
Theorem~3).
Read the full abstract ▾
All authors Carter Adams, Rafael Oliveira, Gabriel Almeida, Sofia Torres
Computation and Language
Discovering a Shared Logical Subspace: Steering LLM Logical Reasoning via Alignment of Natural-Language and Symbolic Views
Feihao Fang, My T. Thai and Yuanyuan Lei
Large
Language Models (LLMs) still struggle with multi-step logical
reasoning. Existing approaches either purely refine the
reasoning chain in natural
language form or attach a symbolic solver as an external module. In this work, we instead ask whether LLMs contain a shared internal logical subspace that simultaneously aligns natural-
language and symbolic-
language views of the
reasoning process. Our hypothesis is that this logical subspace captures logical
reasoning capabilities in LLMs that are shared across views while remaining independent of surface forms. To verify this, we employ Canonical Correlation Analysis on the paired residual activations from natural-
language and symbolic-
language reasoning chains, learning a low-dimensional subspace with maximum cross-view correlation. Furthermore, we design a training-free approach that steers LLMs
reasoning chain along this logical subspace, thereby leveraging the complementary
reasoning signals from both views. Experiments on four logical
reasoning benchmarks demonstrate the effectiveness of our approach, improving accuracy by up to 11 percentage points and generalizing well on out-of-domain problems.
Read the full abstract ▾
Computation and Language
Epistemic orientation in parliamentary discourse is associated with deliberative democracy
Segun Aroyehun, Stephan Lewandowsky and David Garcia
The pursuit of truth is central to democratic deliberation and governance, yet political discourse reflects varying epistemic orientations, ranging from evidence-based
reasoning grounded in verifiable information to intuition-based
reasoning rooted in beliefs and subjective interpretation. We introduce a scalable approach to measure epistemic orientation using the Evidence–Minus–Intuition (EMI) score, derived from large
language model (LLM) ratings and
embedding-based semantic similarity. Applying this approach to 15 million parliamentary
speech segments spanning 1946 to 2025 across seven countries, we examine temporal patterns in discourse and its association with deliberative democracy and governance. We find that EMI is positively associated with deliberative democracy within countries over time, with consistent relationships in both contemporaneous and lagged analyses. EMI is also positively associated with the transparency and predictable implementation of laws as a dimension of governance. These findings suggest that the epistemic nature of political discourse is crucial for both the quality of democracy and governance.
Read the full abstract ▾
Computation and Language
An Answer is just the Start: Related Insight Generation for Open-Ended Document-Grounded QA
Saransh Sharma, Pritika Ramu, Aparna Garimella et al. (+1 more)
Answering open-ended questions remains challenging for AI systems because it requires synthesis, judgment, and exploration beyond factual
retrieval, and users often refine answers through multiple iterations rather than accepting a single response. Existing QA
benchmarks do not explicitly support this refinement process. To address this gap, we introduce a new task, document-grounded related insight generation, where the goal is to generate additional insights from a document collection that help improve, extend, or rethink an initial answer to an open-ended question, ultimately supporting richer user interaction and a better overall question answering experience. We curate and release SCOpE-QA (Scientific Collections for Open-Ended QA), a
dataset of 3,000 open-ended questions across 20 research collections. We present InsightGen, a two-stage approach that first constructs a thematic representation of the document collection using clustering, and then selects related context based on neighborhood selection from the thematic graph to generate diverse and relevant insights using LLMs. Extensive evaluation on 3,000 questions using two generation models and two evaluation settings shows that InsightGen consistently produces useful, relevant, and actionable insights, establishing a strong baseline for this new task.
Read the full abstract ▾
All authors Saransh Sharma, Pritika Ramu, Aparna Garimella, Koyel Mukherjee
Computation and Language
Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
Nurkhan Laiyk, Gerard I. Gállego, Javier Ferrando et al. (+1 more)
Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that
multilingual model representations can be
language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit
language-agnosticity, using machine translation as a case study. Across three decoder-only
multilingual LLMs, we find that translation FVs extracted from a single English$\to$X direction transfer to other target
languages, consistently improving the rank of correct translation
tokens across multiple unseen
languages. We further find that the highest-gain
tokens span multiple
languages and that translation FVs across directions share most of their top-ranked heads, indicating that the FV encodes a largely
language-agnostic translation signal rather than a
language-pair-specific mapping.
Read the full abstract ▾
All authors Nurkhan Laiyk, Gerard I. Gállego, Javier Ferrando, Fajri Koto
Computation and Language
Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
Yi Zhong, Buqiang Xu, Yijun Wang et al. (+4 more)
At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write
prompts for each step, and repeatedly revise the logic as requirements evolve – making development costly, time-consuming, and error-prone. To study whether large
language models can automate this multi-round interaction process, we introduce Chat2Workflow, a
benchmark for generating executable visual workflows directly from natural
language, and propose a robust agentic baseline to improve performance. The
benchmark is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art
language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially given complex and evolving requirements. Although our agentic baseline yields up to 6.05% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.
Read the full abstract ▾
All authors Yi Zhong, Buqiang Xu, Yijun Wang, Zifei Shan, Shuofei Qiao, Guozhou Zheng, Ningyu Zhang
Computation and Language
Pause or Fabricate? Training Language Models for Grounded Reasoning
Yiwen Qiu, Linjuan Wu, Yizhou Liu et al. (+9 more)
Large
language models have achieved remarkable progress on complex
reasoning tasks. However, they often implicitly fabricate information when inputs are incomplete, producing confident but unreliable conclusions – a failure mode we term ungrounded
reasoning. We argue that this issue arises not from insufficient
reasoning capability, but from the lack of inferential boundary awareness – the ability to recognize when the necessary premises for valid inference are missing. To address this issue, we propose Grounded
Reasoning via Interactive Reinforcement Learning (GRIL), a multi-turn reinforcement learning framework for grounded
reasoning under incomplete information. GRIL decomposes the
reasoning process into two stages: clarify and pause, which identifies whether the available information is sufficient, and grounded
reasoning, which performs task solving once the necessary premises are established. We design stage-specific rewards to penalize
hallucinations, enabling models to detect gaps, stop proactively, and resume
reasoning after clarification. Experiments on GSM8K-Insufficient and MetaMATH-Insufficient show that GRIL significantly improves premise detection (up to 45%), leading to a 30% increase in task success while reducing average response length by over 20%. Additional analyses confirm robustness to noisy user responses and generalization to out-of-distribution tasks.
Read the full abstract ▾
All authors Yiwen Qiu, Linjuan Wu, Yizhou Liu, Yuchen Yan, Jin Ma, Xu Tan, Yao Hu, Daoxin Zhang, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
Computation and Language
The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text
Andrew Hong, Jason Potteiger and Luis E. Zapata
An earlier paper (Hong, Potteiger, and Zapata 2026) established that an unoptimized GPT 4.1
prompt predicts fan-reported experience ratings within one point 67% of the time from open-ended survey text. This paper tests the relative impact of
prompt design and model selection on that performance. We compared four configurations on approximately 10,000 post-game surveys from five MLB teams: the original baseline
prompt and a moderately customized version, crossed with three GPT models (4.1, 4.1-mini, 5.2).
Prompt customization added roughly two percentage points of within +/-1 agreement on GPT 4.1 (from 67% to 69%). Both model swaps from that best configuration degraded performance: GPT 5.2 returned to the baseline, and GPT 4.1-mini fell six percentage points below it. Both levers combined were dwarfed by the input itself: across capable configurations, accuracy varied more than an order of magnitude more by the linguistic character of the text than by the choice of
prompt or model. The ceiling has two parts. One is a bias in how the model reads text, which
prompt design can correct. The other is a difference between what fans write about and what they actually decide, which no engineering can close because the missing information is not in the text.
Prompt customization moved the first part; model selection moved neither reliably. The result is not that "
prompt engineering helps a little" but that
prompt engineering helps in a specific and predictable way, on the part of the ceiling it can reach.
Read the full abstract ▾
Computation and Language
Micro Language Models Enable Instant Responses
Wen Cheng, Tuochao Chen, Karim Helwani et al. (+3 more)
Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter
language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro
language models ($\mu$LMs): ultra-compact models (8M-30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency. We show that useful
language generation survives at this extreme scale with our models matching several 70M-256M-class existing models. We design a collaborative generation framework that reframes the cloud model as a continuator rather than a respondent, achieving seamless mid-sentence handoffs and structured graceful recovery via three error correction methods when the local opener goes wrong. Empirical results show that $\mu$LMs can initiate responses that larger models complete seamlessly, demonstrating that orders-of-magnitude asymmetric collaboration is achievable and unlocking responsive AI for extremely resource-constrained devices. The model checkpoint and demo are available at https://github.com/Sensente/micro_
language_model_swen_project.
Read the full abstract ▾
All authors Wen Cheng, Tuochao Chen, Karim Helwani, Sriram Srinivasan, Luke Zettlemoyer, Shyamnath Gollakota
Computation and Language · Cross-listed from cs.AI
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models
Josue Torres-Fonseca, Naihao Deng, Yinpei Dai et al. (+5 more)
Multimodal Large
Language Models are increasingly adopted as autonomous
agents in interactive environments, yet their ability to proactively address safety hazards remains insufficient. We introduce SafetyALFRED, built upon the
embodied agent benchmark ALFRED, augmented with six categories of real-world kitchen hazards. While existing safety evaluations focus on hazard recognition through disembodied question answering (QA) settings, we evaluate eleven state-of-the-art models from the Qwen, Gemma, and Gemini families on not only hazard recognition, but also active risk mitigation through
embodied planning. Our experimental results reveal a significant
alignment gap: while models can accurately recognize hazards in QA settings, average mitigation success rates for these hazards are low in comparison. Our findings demonstrate that static evaluations through QA are insufficient for physical safety, thus we advocate for a paradigm shift toward
benchmarks that prioritize corrective actions in
embodied contexts. We open-source our code and
dataset under https://github.com/sled-group/SafetyALFRED.git
Read the full abstract ▾
All authors Josue Torres-Fonseca, Naihao Deng, Yinpei Dai, Shane Storks, Yichi Zhang, Rada Mihalcea, Casey Kennington, Joyce Chai
Computation and Language
The "Small World of Words" German Free-Association Norms
Samuel Aeschbach, Rui Mata, Kaidi Lõo et al. (+2 more)
Free-association norms provide essential empirical data for investigating linguistic, semantic, and cultural phenomena in the cognitive sciences. Although large-scale norms exist for
languages such as English, Dutch, Spanish, and Mandarin Chinese, no comparable resource has been available for German. To address this gap, we present free-association norms for 5,877 German cue words as part of the German version of the
multilingual Small World of Words (SWOW) project. We describe the data collection procedures, participant characteristics, and our comprehensive preprocessing pipeline before introducing the resulting SWOW-DE data set. Using data from three established psycholinguistic paradigms, we show that SWOW-DE norms robustly predict performance in lexical decision tasks, relatedness judgments, and psycholinguistic word ratings. Furthermore, we demonstrate that SWOW-DE responses compare favorably with existing German resources and provide a preliminary cross-linguistic comparison revealing both shared and
language-specific association patterns, highlighting promising directions for future research. Overall, SWOW-DE represents the largest collection of German free associations to date and offers a unique resource for linguistic, psychological, and cross-cultural research.
Read the full abstract ▾
All authors Samuel Aeschbach, Rui Mata, Kaidi Lõo, Simon De Deyne, Dirk U. Wulff
Computation and Language
Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models
Kihyuk Lee
This study compared repeated generation consistency of exercise prescription outputs across three large
language models (LLMs), specifically GPT-4.1, Claude Sonnet 4.6, and Gemini 2.5 Flash, under temperature=0 conditions. Each model generated prescriptions for six clinical scenarios 20 times, yielding 360 total outputs analyzed across four dimensions: semantic similarity, output reproducibility, FITT classification, and safety expression. Mean semantic similarity was highest for GPT-4.1 (0.955), followed by Gemini 2.5 Flash (0.950) and Claude Sonnet 4.6 (0.903), with significant inter-model differences confirmed (H = 458.41, p < .001). Critically, these scores reflected fundamentally different generative behaviors: GPT-4.1 produced entirely unique outputs (100%) with stable semantic content, while Gemini 2.5 Flash showed pronounced output repetition (27.5% unique outputs), indicating that its high similarity score derived from text duplication rather than consistent
reasoning. Identical decoding settings thus yielded fundamentally different consistency profiles, a distinction that single-output evaluations cannot capture. Safety expression reached ceiling levels across all models, confirming its limited utility as a differentiating metric. These results indicate that model selection constitutes a clinical rather than merely technical decision, and that output behavior under repeated generation conditions should be treated as a core criterion for reliable deployment of LLM-based exercise prescription systems.
Read the full abstract ▾
Computation and Language
RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian
Mircea Timpuriu, Mihaela-Claudia Cercel and Dumitru-Clementin Cercel
The importance of clear and correct text in legal documents cannot be understated, and, consequently, a grammatical error correction tool meant to assist a professional in the law must have the ability to understand the possible errors in the context of a legal environment, correcting them accordingly, and implicitly needs to be trained in the same environment, using realistic legal data. However, the manually annotated data required by such a process is in short supply for
languages such as Romanian, much less for a niche domain. The most common approach is the synthetic generation of parallel data; however, it requires a structured understanding of the Romanian grammar. In this paper, we introduce, to our knowledge, the first Romanian-
language parallel
dataset for the detection and correction of grammatical errors in the legal domain, RoLegalGEC, which aggregates 350,000 examples of errors in legal passages, along with error annotations. Moreover, we evaluate several neural network models that transform the
dataset into a valuable tool for both detecting and correcting grammatical errors, including knowledge-
distillation Transformers, sequence tagging architectures for detection, and a variety of pre-trained text-to-text
Transformer models for correction. We consider that the set of models, together with the novel RoLegalGEC
dataset, will enrich the resource base for further research on Romanian.
Read the full abstract ▾
Computation and Language
A Bolu: A Structured Dataset for the Computational Analysis of Sardinian Improvisational Poetry
Silvio Calderaro and Johanna Monti
The growing interest of Natural
Language Processing (NLP) in minority
languages has not yet bridged the gap in the preservation of oral linguistic heritage. In particular, extemporaneous poetry - a performative genre based on real-time improvisation, metrical-rhetorical competence - remains a largely unexplored area of computational linguistics. This methodological gap necessitates the creation of specific resources to document and analyse the structures of improvised poetry. This is the context in which A Bolu was created, the first structured corpus of extemporaneous poetry dedicated to cantada logudorese, a variant of the Sardinian
language. The
dataset comprises 2,835 stanzas for a total of 141,321
tokens. The study presents the architecture of the corpus and applies a multidimensional analysis combining descriptive statistical indices and computational linguistics techniques to map the characteristics of the poetic text. The results indicate that the production of Sardinian extemporaneous poets is characterised by recurring patterns that support Parry and Lord's theory of formulaicity. This evidence not only provides a new key to understanding oral creativity, but also offers a significant contribution to the development of NLP tools that are more inclusive and sensitive to the specificities of less widely spoken
languages.
Read the full abstract ▾