Computation and Language
Distinguishing Revision and Delayed Elaboration in Incremental Narrative Interpretation
Yi-Chun Chen
Both human and AI systems that process narrative or long-form content operate incrementally: input is received over time, and internal representations must be updated accordingly. Incremental interpretation, therefore, depends not only on what is represented but also on how the representational state evolves under new evidence. We distinguish two structurally different update operators that arise in narrative interpretation: revision-driven update and delayed elaboration. Revision-driven updates retract or replace previously committed structure in response to a contradiction and are therefore non-monotonic. Delayed elaboration, by contrast, refines initially underspecified elements through constraint addition without retracting prior commitments, yielding monotonic extension of the interpretive state. Although both operators may alter how earlier material is understood, they impose fundamentally different structural requirements on state transitions. Using visual narratives as a diagnostic domain, we demonstrate how a structured narrative representation can explicitly separate committed from underspecified content and support both update operators during incremental construction. Through a worked example, we show how delayed elaboration enables monotonic refinement of interpretive state, while revision requires non-monotonic correction. We discuss the broader relevance of this structural distinction for incremental
reasoning and hybrid symbolic-neural systems.
Read the full abstract ▾
Computation and Language
Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention
Zimu Zhao
Standard multi-head
attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric
Attention Heads (AAH), a head-wise context- allocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096-
token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full
attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with
Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic
Read the full abstract ▾
Computation and Language
Same Facts, Different Updates: Inference Setup Shapes LLM Behavior in Medical Allocation
Spencer Gibson, Tyler Crosse, Magnus Saebo et al. (+3 more)
Large
language models are being incorporated into sensitive and important decision-making processes across nearly all fields. While prior work studies model bias around inputs and scenario framing, models can also behave in unexpected and undesirable ways due to context accumulated over their deployment. In this work, we study a medical example in which a model is asked to assign resource-allocation probabilities to two people given brief clinical context, and then sees the same scenario with a single extra sentence containing contrasting patient information, either with or without its previous response in context. Across three of four tested models, the paired-context and independent-inference experiments have different probability shifts, often in opposite directions (in favor of Person B vs. in favor of Person A) when new information is provided. We include additional paired-context experiments to show the effect of varying attributes across scenario axes. Our findings show the context-dependent effect of patient information in a sensitive medical use case. More broadly, our work shows the importance of carefully incorporating LLM-based systems into decision-making processes, context engineering, and further model behavioral studies.
Read the full abstract ▾
All authors Spencer Gibson, Tyler Crosse, Magnus Saebo, Achyutha Menon, Eyon Jang, Diogo Cruz
Computation and Language
Institutional Prestige as Geographic Bias in Large Language Models: Evidence from Three Factorial Experiments with Bootstrap Confidence Intervals
Maikel Leyva-Vazquez and Florentin Smarandache
We investigate whether large
language models (LLMs) systematically discriminate in candidate evaluations based on applicant name ethnicity and/or institutional prestige and geographic location. Three factorial experiments are reported (4,320 API calls, four LLMs, five professional domains). Study 1 (3x4 design) finds a statistically robust institution-tier gradient of +0.297 points on a 10-point scale (95% bootstrap CI: +0.175 to +0.422), while name-origin effects are negligible and non-significant (95% CI crosses zero). Study 2 (2x2 Prestige x Country design) breaks the prestige-geography confound: the prestige effect (+0.185; 95% CI: +0.093 to +0.275) exceeds the country-of-origin effect (+0.126; 95% CI: +0.037 to +0.218) by 1.5x. Study 3 (2x2 Journal x Institution design) reveals that journal prestige (Nature vs. a peripheral open-access journal) dominates institutional prestige by 5.7x: journal effect +1.937 (95% CI: +1.811 to +2.062) vs. institution effect +0.341 (95% CI: +0.184 to +0.504). A "rescue effect" is confirmed: publishing in Nature compensates for low institutional prestige more strongly for candidates from the University of Guayaquil (+2.127) than from MIT (+1.745). Results are quantified using the Neutrosophic Bias Index NBI<T,I,F>; the I component reveals elevated evaluation inconsistency for low-prestige profiles, an epistemic disadvantage not captured by mean-only metrics. Code and data: https://github.com/mleyvaz/geo-bias-llm
Read the full abstract ▾
Computation and Language
Different Facets of Verbalised Overconfidence: an Interpretability Study
Davide Mazzaccara, Leonardo Bertolazzi and Raffaella Bernardi
Large
language models tend to overconfidence, giving assertive answers when the evidence suggests hedging or abstention. Using controlled
reasoning scenarios that manipulate logical necessity and possibility, we study this behavior in Qwen3-4B, across three ways to express uncertainty: verbal epistemic markers, abstention, and numeric confidence scores. Our results confirm this tendency toward overconfidence, particularly when the model is prompted to output a numeric confidence score. At the interpretability level, we propose a method that differentially identifies transcoder features responsible for uncertainty and certainty. Our analysis reveals Qwen3-4B's default mechanism favors certainty generation through a broad coalition of shared features, while uncertainty is implemented as a sparse override mediated by a small set of dedicated features. Intervening on these uncertainty features both causally proves this imbalance underlying overconfidence and also mitigate overconfident errors. The same set of features generalise across the three uncertainty-expression settings,
languages, and an out-of-distribution modality task.
Read the full abstract ▾
Computation and Language
StocksTalk: A Voice-Enabled Conversational Agent for Structured Query Generation over Web Data
Akshat Parmar, Vikranth Udandarao, Abhay Shakya et al. (+4 more)
StocksTalk is a voice-enabled conversational system for transforming spoken financial screening requests into executable and validated structured queries over real-world
market data. The system combines streaming
speech recognition,
retrieval-augmented constraint extraction, schema-grounded LLM-based SQL generation, rule-based validation, and human-in-the-loop verification within an interactive dashboard. Unlike traditional template-driven financial assistants, StocksTalk exposes intermediate
reasoning artifacts, including extracted constraints, normalized financial metrics, operator grounding, and generated queries, allowing users to inspect and refine each stage before execution. To evaluate the system, we curate a
benchmark of 150 spoken financial
prompts spanning multiple investment strategies and input noise conditions. Experimental results show that
retrieval grounding, constrained query generation, and interactive verification substantially improve constraint extraction accuracy, SQL executability, logical consistency, and multi-turn stability compared to baseline LLM-based approaches. StocksTalk demonstrates how transparent, voice-driven interfaces can bridge natural
language interaction and structured financial analysis, providing an effective framework for conversational stock screening and decision support.
Read the full abstract ▾
All authors Akshat Parmar, Vikranth Udandarao, Abhay Shakya, Tanmay Hire, Avinash Anand, Rajiv Ratn Shah, Daniel Wang Zhengkui
Computation and Language
DeepTCM1.0: A Multi-Expert AI Agent for Deciphering Mechanisms of Chinese Herbal Formulae Based on General Large Language Models
Wenxin Duan, Hanwei Wang, Zhongying Peng et al. (+4 more)
Background: Mechanistic elucidation of traditional Chinese medicine (TCM) compound formulas remains a central challenge in the modernization of TCM. Conventional approaches, including data mining and network pharmacology, are insufficient for achieving deep integration between classical TCM theory and modern scientific research. In addition, direct question-answering using general-purpose artificial intelligence large
language models is limited by inadequate adaptation to TCM theoretical frameworks and susceptibility to
reasoning hallucinations. Consequently, there is an urgent need to develop intelligent analytical methods aligned with the holistic principles of TCM. Objective: To establish a multi-expert intelligent
agent framework integrating classical TCM theory with modern life sciences, thereby enabling systematic and interpretable mechanistic analysis of TCM compound formulas, with Guizhi Decoction serving as a representative validation case. Methods: The DeepTCM1.0 framework was constructed based on the general-purpose large
language model DeepSeek V3.2. It adopts a three-tier collaborative architecture and a three-round iterative quality-control workflow, simulating the collaborative analytical process of 11 interdisciplinary intelligent
agents. The framework was applied to the mechanistic interpretation of Guizhi Decoction from the dual perspectives of classical traditional Chinese medicine theory and modern scientific research. Framework performance was comprehensively evaluated through double-blind five-dimensional scoring, intraclass correlation coefficient (ICC) reliability testing, Mann-Whitney U tests, and effect size analysis. The evaluation employed four independent large
language models as evaluators, each conducting five rounds of repeated scoring on five anonymized reports, resulting in a total of 100 independent scoring assessments.
Read the full abstract ▾
All authors Wenxin Duan, Hanwei Wang, Zhongying Peng, Zhonghua Lu, Jiayi An, Fan Song, Yong Liang
Computation and Language
MemoryForge: Synthesize Lifelong Memory for Human-Like LLM Agents
Bohan Tang and Yiwen Guo
Equipping Large
Language Models (LLMs) with human-like personas is crucial for agentic applications, such as role-play and user simulation. Traditional
prompt-based methods rely on descriptive conditioning by injecting static textual profiles, which often makes
agents show generic behaviors due to a lack of realistic life
memory. To fill this gap, we introduce
memory-based conditioning, a paradigm inspired by the cognitive psychology, which replaces abstract profiles with an autobiographical
memory base, enabling frozen LLMs to dynamically retrieve situation-relevant
memory to guide their behaviors. We formalize its enabling task as customized lifelong
memory synthesis and propose MemoryForge, a novel framework to synthesize such lifelong
memory from brief target personas. MemoryForge has three key components: a context generator for socio-historical grounding, a life organizer for developmental coherence toward the target identity, and a multi-resolution simulator that balances broad temporal summaries with high-fidelity episodic experiences. Experiments on PersonaGym for role-play and SimulatorArena for user-simulation, show that the synthesized
memory base by MemoryForge enables frozen LLMs to exhibit more human-like behaviors than strong descriptive conditioning baselines across multiple metrics and LLM backbones.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Structure Over Scale: Schema-Constrained Causal Graphs for RAG
Marc Saouda, Rajprakash Bale, Eren Aldis et al. (+1 more)
Graph-based
retrieval-augmented generation (GraphRAG) grounds answers in structured knowledge, but current systems extract entities and relationships exhaustively, producing graphs whose size and construction cost scale with corpus length rather than with the
reasoning a query requires. We introduce HCG-RAG (Hierarchical Causal Graph RAG), which replaces open-ended extraction with schema-constrained causal graphs: an automated pipeline distills a corpus into a fixed, typed vocabulary of causal variables and materializes a compact two-tier graph over it. Our schema-constrained graphs match entity-relation baselines on answer quality at a fraction of the cost: 3-20x fewer nodes, 8x-135x fewer build-time LLM calls than the most LLM-intensive baseline (MS-GraphRAG), and graphs compact enough for a domain expert to audit, correct, and extend. On medical and clinical
benchmarks, including a neurologist-validated epilepsy
dataset, HCG-RAG matches or exceeds the best entity-relation systems. An ablation isolates the causal graph as a structured
retrieval filter, contributing +6 percentage points (pp) over
embedding-only
retrieval. Across all domains with discoverable hierarchical causal structure, only methods imposing higher-level organization outperform flat entity-relation
retrieval, indicating that what is placed in the graph matters more than how many nodes it contains.
Read the full abstract ▾
All authors Marc Saouda, Rajprakash Bale, Eren Aldis, Cloves Almeida
Computation and Language · Cross-listed from cs.AI
DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents
Raffi Khatchadourian
When a financial
agent repeats its decision, how much does that tell us about the actions behind it? DFAH-Bench examines this question through replay experiments, a controlled test of omitted tool calls, and banking tasks with authorization checks. Across 570 eligible prospective replay episodes, decisions agreed at rates of 94-95%, while tool paths, including inputs and results, agreed at 45-52%; one condition fell one group short of the planned minimum. In a separate 48-execution diagnostic, deleting the same calls from repeated records left agreement perfect. In banking probes, repeated authorization decisions sometimes conflicted with constructed rule labels. The $\tau$-Knowledge banking study scheduled 1,080 episodes, with 1,033 known outcomes. Structural checks alone completed more primary tasks than either model-based gate-and-recovery bundle. Input limits and recovery behavior materially shape these results. The typed-choice bundle had lower mean episode cost than the generative bundle on complete task pairs, but fewer successes across the primary schedule under every assignment of unknown outcomes. Missing outcomes prevented the planned confirmatory tests, so the banking comparisons are descriptive. Together, the findings show that agreement alone cannot establish reliability: it can conceal changes in execution, systematic omissions, and unsupported decisions. Evaluation therefore needs to connect what the
agent did, the evidence and rules governing its action, and whether it completed the task.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
CANONIC: Governance Is Compilation
Dexter Hadley
We present CANONIC: governed intelligence that compiles digital artifacts into an evidence ledger at scale. Large
language models generate prose faster than anyone can check it, the failure Oxford
Languages named 'slop', its 2025 Word of the Year. CANONIC governs whether content may enter a corpus the way a compiler decides whether a program is well-formed: mechanically, by a grammar, at the boundary of admission. Governance reduces to three axioms (Triad, Inheritance, Introspection) that map one-to-one onto compiler theory's syntax, scope-resolution, and type-system layers, and admission is a decidable, linear-time check. We then ask, with a pre-registered cross-provider
benchmark across four regimes, whether structural admission keeps slop out. It does not: no prose-reading gate reliably separates reliable from unreliable content. Slop is not a property an
algorithm computes. It is a verdict of domain expertise. So a governance layer does not decide slop; it keeps the record auditable – every claim anchored to a definition, a commit, and an evidence window, reproducible and checkable end to end.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Guido Freire, Agustín Martínez-Suñé and Viviana Cotik
Large
Language Models have the potential to expand and improve the access to clinical information by enabling new ways of interacting with medical knowledge in natural
language. However, their deployment in medical question-answering settings is safety-critical, since misaligned outputs can lead to severe patient harm. AI control is an emerging approach that introduces external safeguards to mitigate unsafe behaviours in misaligned systems and has been shown to be effective in domains such as code generation. However, its applicability and effectiveness in medical settings have not been systematically studied. In this work, we present a pipeline for evaluating AI control protocols to mitigate medication-related harm. To this end, we introduce DrugBench, an AI control evaluation
benchmark which combines 3,671 multi-turn medical conversations from HealthBench with drug information from official FDA labels, covering four categories of medication-related harm: drug interactions, contraindications, dosing constraints, and patient action restrictions. Furthermore, inspired by the medical domain, we argue that safety should account for the severity of unsafe outputs, not just their probability. Under this revised definition, we show that existing control protocols can be subverted and propose severity-based monitoring to address this limitation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier
Lorenz Wolf, Connor Watts, Roger Creus Castanyer et al. (+4 more)
The limiting resource for training
agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As
reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes; solver-in-the-loop generator training is intractable. We introduce PROPEL, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from $10.1\% \rightarrow 20.0\%$ for a Qwen2.5-3B-Instruct solver and from $5.3\% \rightarrow 12.6\%$ for a Qwen2.5-7B-Instruct solver. For SWE, PROPEL increases the share of generations at the targeted solve rate from $9.8\% \rightarrow 19.6\%$ for Qwen3.5-27B on repositories not seen during training of probe and generator.
Read the full abstract ▾
All authors Lorenz Wolf, Connor Watts, Roger Creus Castanyer, Geoffrey Bradway, Maxwill Lin, Augustine N. Mavor-Parker, Matthew Daborn-Sargent
Computation and Language · Cross-listed from cs.CV
Last But Not Least: Boundary Attention CalibratiON for Multimodal KV Cache Compression
Tianhao Chen, Yuheng Wu, Kelu Yao et al. (+3 more)
Multimodal Large
Language Models (MLLMs) achieve strong
vision-
language reasoning but incur large KV caches and high decoding latency with long visual contexts. Existing compression methods rely on observation window
attention for stable
token importance estimation, yet this aggregation can dilute sparse critical evidence and discard answer-relevant
tokens under aggressive compression. We identify last query
attention as a complementary signal for recovering such evidence, though its irrelevant signals may introduce additional noise. We propose BACON, a plug-and-play method that calibrates observation window
attention with last query evidence while suppressing noise through intra-layer coherence and inter-layer persistence. Across diverse
benchmarks, models, budgets, and compression methods, BACON improves
multimodal KV-cache compression by 7.5% on average under the most aggressive budget, with gains up to 30.9%.
Read the full abstract ▾
All authors Tianhao Chen, Yuheng Wu, Kelu Yao, Xiaogang Xu, Xiaobin Hu, Dongman Lee
Computation and Language · Cross-listed from cs.AI
WorkBench Revisited: Workplace Agents Two Years On
Olly Styles and Sam Miller
The best
agent on WorkBench in March 2024, GPT-4, completed just 43% of tasks. We revisit the
benchmark in June 2026 and find that the best
agent to date, Claude Fable 5, now completes 98%. Beyond this considerable progress in frontier
agent performance, three things stand out. First, unintended harmful actions, such as emailing the wrong person, fell from 26% of tasks for GPT-4 to 1.9% for Claude Fable 5; capability and safety go together on WorkBench rather than trade off, so the models that finish the most tasks also do the least unintended damage. Second, the rise of open-weight models has drastically lowered costs for a performance level that was only accessible to proprietary models, while frontier costs have stayed stable. Third, while several classes of error have been eliminated, frontier models still make some basic mistakes that occasionally result in irreversible harm. We release an updated version of the
benchmark with data and code quality improvements, new model scores, and analysis of
agent progress on WorkBench since 2024.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Multimodal Speaker Identification in Classroom Environments
Michael L. Chrzan, Meghavarshini Krishnaswamy, Robert Gibboni et al. (+3 more)
Automated analysis of K-12 classroom dynamics faces challenges due to background noise and variable child
speech, often confounding acoustic-only models. This study evaluates a
multimodal speaker identification framework anchoring acoustic
embeddings with LLM-derived semantic context. Using a subset of the EDSI
dataset (8 math classrooms, N = 2,801 utterances), we found an acoustic baseline (ECAPA-TDNN) achieved only 39.0% accuracy. By integrating transcript-based "contextual anchoring" into a gradient boosting classifier, our
multimodal approach raised student identification to 50.3%. Performance also improved for utterances over 5 seconds, reaching 76.9% accuracy (vs. 64.9% baseline) with a 90.9% Top-3 accuracy. Additionally, the model distinguished teacher vs. student roles with 99.3% accuracy. This approach advances the feasibility of automated feedback systems capable of considering individual student participation, a crucial step for supporting equitable instruction at scale.
Read the full abstract ▾
All authors Michael L. Chrzan, Meghavarshini Krishnaswamy, Robert Gibboni, Katie Wetstone, Wei Ai, Jing Liu
Computation and Language · Cross-listed from cs.AI
Orchestra-o1: Omnimodal Agent Orchestration
Fan Zhang, Vireo Zhang, Shengju Qian et al. (+8 more)
The recent success of
agent swarms has shifted the paradigm of large
language model (LLM)-based
agents from single-
agent workflows to multi-
agent systems, highlighting the importance of
agent orchestration for task decomposition and collaboration. However, existing orchestration frameworks are limited to a narrow set of modalities and struggle to generalize to more complex settings where heterogeneous modalities coexist and interact. This limitation becomes particularly pronounced in omnimodal scenarios, where tasks require the unified understanding and coordination of diverse inputs such as text, image, audio, and
video. In this work, we propose Orchestra-o1, an omnimodal
agent orchestration framework designed to support efficient
agent collaboration across multiple modalities. Orchestra-o1 introduces a unified orchestration mechanism that enables modality-aware task decomposition, online sub-
agent specialization, and parallel sub-task execution. This scalable design allows
agent systems to effectively tackle complex real-world tasks involving heterogeneous information sources, surpassing the second-best approach by 10.3% accuracy on the OmniGAIA
benchmark. Furthermore, we introduce decision-aligned group relative policy optimization (DA-GRPO), an efficient agentic reinforcement learning approach for training Orchestra-o1-8B, which also achieves state-of-the-art performance against all existing open-source omnimodal
agents.
Read the full abstract ▾
All authors Fan Zhang, Vireo Zhang, Shengju Qian, Haoxuan Li, Hao Wu, Jinyang Wu, Donghao Zhou, Zhihong Zhu, Zheng Lian, Xin Wang, Pheng-Ann Heng
Computation and Language
LLMs Can Better Capture Human Judgments–With the Right Prompts
Danica Dillion, Chen Cecilia Liu, Baihui Wang et al. (+5 more)
Are large
language models (LLMs) bad at capturing human judgment? Two commonly stated limitations are that LLMs fail to capture full distributions of responses, and that their judgments are unstable across wording variations. We demonstrate simple prompting strategies that mitigate these limitations. Across two
datasets–a U.S.-representative set of 144 moral scenarios and 38 moral beliefs from the International Social Survey Programme's Family and Changing Gender Roles module covering 32 countries–we show how simple elicitation techniques help improve AI-human
alignment. First, prompting models to report standard deviations and response proportions recovers the full range of human responses better than common strategies. Second, ensuring scenarios are clear to human participants–as reflected in human confusion ratings–boosts model
alignment, and LLMs can track human confusion ratings. At the same time, we find that LLMs' estimates of their own error are poorly calibrated, though they can predict human variability relatively well. These results suggest that asking better questions to LLMs can yield better answers.
Read the full abstract ▾
All authors Danica Dillion, Chen Cecilia Liu, Baihui Wang, Daniele Barolo, Tanmay Rajore, Niket Tandon, Pranathi Ravikumar, Kurt Gray
Computation and Language
Agent-based models for the evolution of morphological alternation patterns
Aravinth Kulanthaivelu and Richard Sproat
Why is the past of English "go" the apparently unrelated "went"? Such alternations are frequent in
languages. They neither aid communication nor learnability, yet they can be persistent, surviving over centuries or millennia. We present a multi-
agent simulation of the emergence of morphological stem and inflection alternations. Alternate forms arise by phonological changes or, as with "go/went", from lexical alternatives associated with a subset of the population. When an
agent 'hears' another
agent use a novel form for a slot in the paradigm of a word (say, the past tense of go), they will with some probability adopt that form, possibly spreading its use to other slots in the paradigm that shared the same original form. Thus alternative forms can spread through the population and become entrenched as stem or inflectional marker alternants. Unlike many previous computational studies, our system allows for naturalistic lexical forms, realistic phonological rules, lexicons with hundreds or thousands of entries, and
agent populations in the tens or hundreds. It supports several network
topologies,
diffusion patterns and
agent adoption policies. One issue with such simulations is evaluation: how realistic is the resulting morphology compared to those of real
languages? We introduce the AI Historical Linguist, a novel Large
Language Model-driven system that models a debate between two historical linguists. We use this to compare a set of real
language morphologies, disguised morphologies, and experimentally evolved morphologies. The results suggest that among the factors that favor more plausible morphologies are scale-free social networks and random Bernoulli adoption of forms. We also present three case studies modeling attested historical changes, allowing us to test what might have happened if history had been different. All code and data are released.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
Rafal Kocielnik, Pengrui Han, Peiyang Song et al. (+5 more)
Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate
prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into
alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.
Read the full abstract ▾
All authors Rafal Kocielnik, Pengrui Han, Peiyang Song, Myrl G. Marmarelis, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez
Computation and Language
Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review
Xinyu Zhao, Rana Muhammad Shahroz Khan, Zhen Xu et al. (+2 more)
The integration of Large
Language Models (LLMs) and
Multimodal LLMs (MLLMs) into scientific peer-review workflows introduces novel and significant risks for
adversarial manipulation, especially given the
multimodal nature of scientific papers where figures, not just text, convey core evidence. This creates a significant gap: current robustness studies on AI peer-review are overwhelmingly text-only. Moreover, the problem is distinct from standard jailbreaking, as a peer-review attack seeks to induce a domain-specific, targeted failure (e.g., "inflate this score") rather than a general safety policy violation, for which no practical defenses exist. To address this, we introduce PaperGuard, the first comprehensive
benchmark designed to systematically evaluate and defend AI-generated peer-review against these domain-specific, cross-modal attacks. Our framework is built on three pillars: (1) a new
multimodal peer-review
dataset spanning multiple scientific domains; (2) a unified suite of attacks, including black-box
prompt injections and white-box perturbations, specifically designed to target both text (GCG) and figures (PGD); and (3) a practical defense, motivated by the long-context challenge of academic papers, that uses chunk-based
embedding search to efficiently localize and mitigate harmful instructions. Our extensive experiments, conducted across state-of-the-art models, confirm that AI reviewers are pervasively vulnerable. PaperGuard establishes the foundational
benchmark, protocols, and actionable defense necessary to pioneer trustworthy, attack-resilient AI-assisted scholarly reviewing.
Read the full abstract ▾
All authors Xinyu Zhao, Rana Muhammad Shahroz Khan, Zhen Xu, Zhen Tan, Tianlong Chen
Computation and Language
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Happy Buzaaba, Cheikh Mouhamadou Bamba Dione, David Ifeoluwa Adelani et al. (+15 more)
Despite their linguistic diversity and global significance, African
languages remain underrepresented in research and resources to support NLP. We aim to bridge this gap by introducing AfriSUD, the first large-scale collection of syntactically annotated treebanks for nine diverse African
languages spanning major
language families and regions across Sub-Saharan Africa. Using the Surface-Syntactic Universal Dependencies (SUD) framework, our community-led effort provides high-quality, native-speaker verified data that capture typological key features such as agglutination and tone. We evaluate a range of models on AfriSUD for part-of-
speech tagging and dependency parsing including non-
transformer baselines,
multilingual pretrained encoders, and LLMs. Our results reveal a significant syntax gap, where models still show clear limitations across the nine
languages, suggesting that existing architectures may not fully capture the structural diversity of African-
language syntax.
Read the full abstract ▾
All authors Happy Buzaaba, Cheikh Mouhamadou Bamba Dione, David Ifeoluwa Adelani, Sylvain Kahane, Kim Gerdes, Bruno Guillaume, Kevin Guan, Aremu Anuoluwapo, Naome A. Etori, Shamsuddeen Hassan Muhammad, Utitofon Inyang, Peter Nabende, David Sabiiti Bamutura, Andiswa Bukula, Chinedu Uchechukwu, Rooweither Mabuya, Idris Akinade, Christiane Fellbaum
Computation and Language
Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models
Darpan Aswal, Thomas Palmeira Ferraz, Yongxin Zhou et al. (+1 more)
Latent
reasoning models (LRMs) replace explicit chain-of-thought with continuous thoughts. Recent work treats observable latent-state patterns, such as BFS-like frontiers and decodable arithmetic computation, as evidence for internal
reasoning mechanisms. Evaluating two LRMs (Coconut and CODI) against controls lacking the proposed recurrence or curriculum, we find these patterns also appear in the controls and do not consistently affect behavior. Causal interventions reveal that latent-thought utilization is not binary but graded, scaling with a thought's causal effect on model behavior.
Geometric analyses reveal this effect concentrates in low-dimensional subspaces whose step-wise geometry grows more structured as their behavioral influence increases. Latent thoughts should therefore be treated as hidden computation, not hidden explanation: decodability,
attention, or static structure alone cannot establish LRM
reasoning mechanisms; that requires matched controls and causal tests.
Read the full abstract ▾
All authors Darpan Aswal, Thomas Palmeira Ferraz, Yongxin Zhou, Maxime Peyrard
Computation and Language
MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection
Fatimah Almalki, Areej Alhothali, Lulwah Alharigy et al. (+1 more)
Detecting mental health disorders from Arabic social media text remains challenging due to dialectal variation, informal
language, limited high-quality annotated resources, and severe class imbalance. While English mental health natural
language processing (NLP) has progressed substantially, Arabic multi-class disorder classification remains insufficiently studied. This study proposes a two-phase framework for Arabic mental health text classification. In phase 1, three Arabic pre-trained
language models, AraBERT, CAMeLBERT, and MARBERT, undergo Domain-Adaptive and Task-Adaptive Pretraining (DAPT and TAPT) using a large-scale corpus of unlabeled Arabic mental health tweets. The adapted models are evaluated under a unified protocol to identify the most effective backbone model. In phase 2, the selected model is assessed across four configurations combining single-stage and hierarchical two-stage classification architectures with full fine-tuning and Low-Rank Adaptation (LoRA). To support this study, we constructed a novel annotated Arabic mental health
dataset comprising 50,670 tweets across six categories, with strong inter annotator agreement (Krippendorff's Alpha = 0.733, average pairwise agreement = 0.797). Experimental results show that the domain-adapted MARBERT (MentalMARBERT) achieves statistically significant improvements over baseline models in both accuracy and macro-F1. The hierarchical two-stage architecture combined with full fine-tuning achieves the best overall performance, reaching a macro-F1 of 0.861 and an accuracy of 0.877. These findings demonstrate the effectiveness of domain-specific adaptive pretraining and hierarchical classification for Arabic mental health disorder detection.
Read the full abstract ▾
All authors Fatimah Almalki, Areej Alhothali, Lulwah Alharigy, Abdulrahman Aladeem
Computation and Language · Cross-listed from cs.LG
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
Tianyu Ding, Jianhong Xin and Juan Pablo De la Cruz Weinstein
Long-horizon tool-use reinforcement learning learns from outcome verification, but trajectory-level advantages are broadcast over
reasoning, API, and answer
tokens. Direct self-
distillation can supply a denser signal, but in our experiments it can also destroy tool use by rehearsing teacher behavior without identifying which actions the verifier rewards. We introduce Sibling-Guided Credit
Distillation (SGCD), which uses
distillation for bounded credit weighting rather than as a competing actor loss. Dynamic sampling produces mixed successful and failed sibling rollouts; an external LLM summarizes their contrast into a training-only credit reference; and detached teacher/student divergence reshapes GRPO
token advantages. The deployed student receives only the clean task
prompt. Across AppWorld and tau^3-airline, SGCD reports higher held-out point estimates than GRPO-family comparators: AppWorld TGC improves from 42.9 to 45.6 on test_normal and from 24.7 to 27.0 on test_challenge, and tau^3-airline held-out evaluator score improves from 0.583 to 0.602. These results support a narrow design rule for long-horizon tool-use
agents: use
distillation to guide credit assignment while keeping policy gradient in charge of the actor update.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation
Mahmoud Srewa, Praneetsai Iddamsetty, Mohammad Abdullah Al Faruque et al. (+1 more)
Closed-loop driving simulators typically populate their environments with non-ego traffic
agents that behave largely the same way, produced either by rule-based traffic managers or by learned models trained toward a single behavioral mode. Recent work introduces style variation through post-hoc labels on observational data or LLM-inferred reward weights, but these signals act as proxies for what a style should reward rather than demonstrations of humans explicitly asked to drive in that style. We introduce PersonaDrive, a pipeline that conditions a
vision-
language-action (VLA) driving
agent on retrieved demonstrations from a style-instructed human driving
dataset, in which participants drive CARLA leaderboard routes under aggressive, neutral, and conservative instructions on a driver-in-the-loop rig. The pipeline has three stages: (i) offline triplet mining over per-style human driving data using a combined image-text similarity score; (ii) training a lightweight
retrieval head that fuses frozen visual features with a small control encoder over per-style databases; and (iii) fine-tuning a single VLA backbone to treat retrieved context points as in-context behavioral demonstrations during waypoint prediction. At inference, the same backbone is conditioned on any style by swapping which per-style database the
retrieval head queries, so selecting a style requires no per-style retraining while enabling human-style, style-diverse non-ego
agents for closed-loop simulation. On Bench2Drive, PersonaDrive (no style) improves the driving score by 4.6% over SimLingo and 2.5% over HiP-AD, and under style conditioning attains the highest driving score in every style within a roughly 2% band (its weakest style surpassing the strongest baseline, DMW, by 5.4%), while average speed and acceleration rise by 18% and 25% from the conservative to the aggressive instruction.
Read the full abstract ▾
All authors Mahmoud Srewa, Praneetsai Iddamsetty, Mohammad Abdullah Al Faruque, Salma Elmalaki
Computation and Language
Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistants
Shuxian Fan, Seonwoo Min, Youna Hu et al. (+7 more)
Conversational shopping assistants now serve hundreds of millions of customers, yet no existing
benchmark jointly evaluates the open-ended multi-turn
reasoning, domain expertise, and criterion-level quality that real shopping conversations demand. Shopping
reasoning is unique among
language model applications. Unlike factual question answering or verifiable code generation, it requires balancing subjective preferences, budget constraints, and cross-product trade-offs across multi-turn dialogue, capabilities absent from previous e-commerce and general-purpose
benchmarks. We introduce the Shopping
Reasoning Bench, an expert-authored
benchmark of 525 missions (232 single-turn, 293 multi-turn) with 10863 importance-weighted binary rubrics authored by retail domain experts. These criteria are organized under a taxonomy of five
reasoning categories and fifteen subcategories covering diverse demands such as preference refinement, trade-off analysis, and compatibility assessment. An evaluation of nine models across three families (GPT, Claude, Gemini) shows that pass rates reach only 57–77% overall. On multi-turn missions, all models score 13–29 points lower on optional above-and-beyond criteria than on required ones, and performance degrades 4–18 points as conversations progress. These gaps show that current models handle basic shopping assistance but fall short of expert-level advice, making Shopping
Reasoning Bench a challenging testbed for future shopping assistant development.
Read the full abstract ▾
All authors Shuxian Fan, Seonwoo Min, Youna Hu, Botao Xia, Jayakrishnan Unnikrishnan, Rowan Musselmann, Yifan Gao, Qingyu Yin, Priyanka Nigam, Bing Yin