Computation and Language
Self-Generated Text Recognition: Quality Heuristics, Cross-Task Transfer, and Downstream Bias in LLM Evaluation
Jesse St. Amand, Callum Canavan, Sohaib Imran et al. (+5 more)
Self-Generated Text Recognition (SGTR)–the ability of an LLM to identify its own outputs–poses risks to AI safeguards that rely on LLMs as evaluators or monitors: an LLM may recognize outputs from other copies of the same model and make biased judgments or collude outright. Prior work has drawn conflicting conclusions about whether current models possess significant SGTR capabilities. We explain these disagreements by identifying key experimental design choices–which we term operationalizations–that drive divergent results. Evaluating 13-21 models across six presentation operationalizations and four task-domain operationalizations, we find that accuracy varies substantially with evaluation format (pairwise vs individual assessments of text), conversation format (presenting candidate text in user tags vs assistant tags), and the domain of the task used to generate candidate text (e.g., coding vs summarization). We corroborate previous observations that a quality heuristic–models attributing authorship to text they perceive as higher quality–is a dominant confound. We also find that improving a model's SGTR performance via supervised fine-tuning (SFT) on one operationalization can generalize to others, and can increase the model's preference for its own outputs when it acts as a judge in the AlpacaEval framework. Our results suggest that, despite confounds, some models possess practical SGTR capabilities, and that SGTR should be monitored and considered in the design of safety-critical AI applications.
Read the full abstract ▾
All authors Jesse St. Amand, Callum Canavan, Sohaib Imran, Joseph Hewson, Aaron Lutz, Shi Feng, Puria Radmard, Lennie Wells
Computation and Language
Jais 2: A Family of Arabic-Centric Open Large Language Models
Mohamed Anwar, Abed Alhakim Freihat, George Ibrahim et al. (+57 more)
Jais 2 is a family of Arabic-centric large
language models developed jointly by MBZUAI, Cerebras, and Inception, designed to advance Arabic-centric
language modeling, with strong performance across the Arabic and culturally grounded
benchmarks evaluated in this report. The family includes, to our knowledge, the largest open Arabic-centric LLM trained from scratch at 70B parameters, and a competitive 8B-parameter variant among the evaluated open models. A custom Arabic-centric vocabulary enables efficient training and inference. In addition, an optimized architecture and training recipe yield highly compute-efficient training. With a substantially smaller
token budget than comparable models, Jais 2 achieves strong Arabic performance on the
benchmarks considered in this report and competitive English results. The models obtain leading results among the evaluated open models on OALL2 and AraGen. They also perform strongly on several culturally grounded Arabic
benchmarks, including poetry, religion, cuisine, and dream interpretation, as well as in general tasks such as translation and summarization. We release the models in HuggingFace under a commercially permissive license. Jais 2 70B is also released as a chat app on the Web, iOS, and Android; it runs on Cerebras hardware, delivering up to 2,000
tokens per second, and enabling high-throughput Arabic-centric chat serving in our deployment setting. By uniting scale, linguistic diversity, cultural fidelity, openness, and speed, Jais 2 provides an open-weight foundation intended to support further research and development in Arabic-centric LLMs.
Read the full abstract ▾
All authors Mohamed Anwar, Abed Alhakim Freihat, George Ibrahim, Mostafa Awad, Abdelrahman Sadallah, Gurpreet Gosal, Gokulakrishnan Ramakrishnan, Sarath Chandran, Biswajit Mishra, Rituraj Joshi, Ahmed Frikha, Etienne Goffinet, Abhishek Maiti, Ali El Filali, Sarah AlBarri, Samujjwal Ghosh, Rahul Pal, Parvez Mullah, Awantika Shukla, Sajid siddiki, Samta Kamboj, Onkar Pandit, Sunil Kumar Sahu, AbdelRahman Elbadawy, Amr Mohamed, Ahmad Chamma, Evan Dufraisse, Abdelaziz Bounhar, Dani Bouch, Hadi Abdine, Guokan Shang, Fajri Koto, Yuxia Wang, Zhuohan Xie, Ali Mekky, Rania Elbadry, Sarfraz Ahmad, Momina Ahsan, Omar El Herraoui, Daniil Orel, Hasan Iqbal, Kareem Elzeky, Mervat Abassy, Kareem Elozeiri, Saadeldine Eletter, Farah Atif, Nurdaulet Mukhituly, Haonan Li, Xudong Han, Aaryamonvikram Singh, Zainul Abedien Ahmed Quraishi, Neha Sengupta, Larry Murray, Avraham Sheinin, Joel Hestness, Natalia Vassilieva, Hector Xuguang Ren, Zhengzhong Liu, Michalis Vazirgiannis, Preslav Nakov
Computation and Language
When Benchmarks Mislead: Shortcut Learning, Length Confounds, and the Limits of Cross-Dataset Generalization in Multilingual Fake News and Sarcasm Detection
Muhammad Abdullah Haroon
Cross-
dataset generalisation is a fundamental requirement for deploying text classifiers in real-world settings, yet systematic evaluation across corpora from different sources remains uncommon in fake news detection and virtually absent in sarcasm detection research. This paper presents a unified empirical study of zero-shot cross-
dataset transfer in three domains: Urdu fake news detection (FND), English FND, and sarcasm detection. For each domain, we fine-tune xlm-roberta-base on one corpus and evaluate it on a second corpus from a different source, comparing against TF-IDF baselines with Logistic Regression (LR) and Support Vector Machines (SVM). In Urdu FND (Ax-to-Grind vs. Notri-Fact), we identify a severe length confound in the Ax-to-Grind
dataset, fake articles average 3.4 times more words than real articles, causing catastrophic A to B transfer collapse (macro F1 = 0.005) while B to A achieves F1 = 0.771. Extension to English FND (WELFake vs. ISOT) and sarcasm detection (TweetEval Irony vs. Sarcasm Corpus V2) reveals that such failure modes extend beyond Urdu, confirming that shortcut learning from distributional artefacts is a cross-lingual, cross-domain challenge in binary text classification. We provide a reusable diagnostic methodology, combining class-conditional length analysis, bidirectional transfer asymmetry, and predicted label collapse inspection, applicable across any binary text classification setting.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Mianqiu Huang, Taofeng Xue, Chong Peng et al. (+13 more)
Computer-use
agents must solve long-horizon tasks through repeated interaction with partially observable,
multimodal desktop environments. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options. EvoCUA-1.5 extends self-evolving computer-use
agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes. Online RL in this setting requires more than directly reusing single-turn
language-RL recipes. Multi-turn interaction introduces context-managed observations, sparse terminal rewards, variable-length trajectories, and slow environment feedback. EvoCUA-1.5 addresses these challenges with Step-Level Policy Optimization (STEPO), which preserves trajectory-level advantage balance after decomposition into step-level samples; policy-aware filtering and pass-rate calibration over verifiable synthesized tasks; Dynamic Tri-Adaptive Curriculum (DTAC), which combines learnable tasks, difficult positive replay, and controlled infeasible-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini-group batching. Experiments show that these components improve training stability and downstream performance. EvoCUA-1.5 achieves 63.2% success on OSWorld-Verified, outperforming comparable 32B/35B-scale open-weight baselines and even approaching models with significantly larger parameter counts. Overall, EvoCUA-1.5 provides a practical framework for scaling online RL in multi-turn computer-use
agents.
Read the full abstract ▾
All authors Mianqiu Huang, Taofeng Xue, Chong Peng, Jinrui Ding, Jie Yang, Sicheng Fan, Jiale Hong, Yufei Gao, Xiaocheng Zhang, Linsen Guo, Xin Yang, Dengchang Zhao, Yuchen Xie, Peng Pei, Xunliang Xie, Xipeng Qiu
Computation and Language · Cross-listed from cs.AI
Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
Yaowen Ye and Jacob Steinhardt
AI
agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards. This multi-
agent competition can lead to behaviors that serve individual gains at collective cost – for instance, marketing
agents may post misleading content as a result of competing for engagement on social media. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations. Motivated by this, we study norm enforcement mechanisms for
language model
agents. We find that simple enforcement mechanisms are exploited by misaligned
agents for competitive advantage, even when they are not explicitly trained or prompted to do so. We thus turn our
attention to designing more robust mechanisms, and identify two key ingredients: estimating each
agent's reliability over time, and updating this estimate with escalating penalties for repeated misbehavior. Across three simulated environments and a variety of
agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines. Our results position norm enforcement mechanisms as scalable levers for shaping
agents' behavior, but only when designed to anticipate becoming part of the system they govern. Our code and data are available at https://yaowenye.com/norm-enforcement.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarm
Igor Itkin
A cheap swarm of unreliable
agents can be steered to a correct consensus by a few strong, expensive "oracle" correctors. We ask how much one must spend, and where to place the oracles. We model the swarm as a consensus on a graph in which each oracle pins one node toward the truth at a cost-coupled, concave strength, and measure quality by the coherence H(R)=tr M(R)^{-1}. Our first result is that H stays submodular (each added oracle helps less than the last) even when the oracles differ in strength, so a cost-benefit greedy comes within 1-1/e of the best placement at any budget. Inverting the budget gives the budget-correctness frontier B*(eps), the least spend that guarantees an eps-correct consensus: closed-form on the complete graph, and a minimal oracle count k* when oracles cost the same. Whether a budget then buys a few strong oracles or many medium onese
curvature of the cost-quality law: diminishing returns favour spreadsharply increasion. Measured onthe Qwen3 ladder (0.6-32B), the law is concave for math verificatio convex foremergent code tracing, so the verdict is genuinely task-dependent.https://github.com/YehudaItkin/budgeted-oracle-placemen
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Josip Jukić and Ivan Titov
On-policy
distillation provides dense teacher supervision on a
language model's own trajectories. In self-
distillation with privileged context, this supervision comes from the model itself, conditioned on a hint or solution trace hidden from the student. When the teacher's preferences hinge on privileged information, it can assign higher probability to continuations the student cannot infer from its own context. Matching these preferences throughout training can induce predictive drift and degrade out-of-distribution (OOD)
reasoning. We propose GeoSD, a self-
distillation method that controls this drift through two complementary
geometric terms. A Hellinger loss weights each teacher preference by the student–teacher overlap, reducing the influence of
tokens to which the student assigns low probability. Because these influences can still accumulate, a Fisher–Rao penalty regulates predictive distance from a copy of the student refreshed periodically during training. Both terms compare next-
token distributions in Fisher–Rao geometry and are jointly optimized with a preconditioner motivated by the natural gradient. Across three model families, GeoSD retains strong in-distribution gains while improving average mathematical OOD accuracy by 5.7–8.6 points over the base model. OOD gains hold across five model scales from 1.7B to 32B and transfer to code generation, where GeoSD improves code accuracy by 1.9 points on average despite distilling on mathematics alone. Our analysis of mathematical
reasoning shows that standard matching rapidly concentrates probability mass at high-
entropy states and that its samples confidently agree on incorrect answers. In contrast, GeoSD preserves alternative
token mass and reduces false consensus.
Read the full abstract ▾
Computation and Language
LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering
Huan Wu, Ali Emami, Muhammad Furquan Hassan et al. (+5 more)
African American English (AAE), a rule-governed dialect spoken by over 30 million people, is routinely misinterpreted and "corrected" by large
language models (LLMs). Across six instruction-tuned LLMs (14B to 70B), we show that state-of-the-art models systematically prefer Standard American English (SAE) continuations even when the preceding context is in AAE, effectively rewriting AAE into SAE. We present an end-to-end framework to audit and mitigate this bias. For
auditing, we introduce conditional Dialect Group Invariance (cDGI), which isolates true model bias from translator-induced artifacts, and a feature-level localization analysis that identifies which AAE markers most strongly trigger bias; we find that syntactic constructions, especially negative concord (e.g., "ain't nobody"), are universal triggers across all models. For mitigation, we introduce, to our knowledge, the first application of activation steering to dialect bias: a training-free, test-time method that extracts dialect directions via causal tracing and injects them into bias-relevant layers. Activation steering reduces bias 5 to 20 times more than prompting while preserving SAE fluency. To enable this work, we release REAL-AAE , the largest real-AAE parallel corpus to date: 17,479 AAE/SAE/ AAE_back triplets from natural tweets (2 to 6 times larger than prior real-AAE resources), validated automatically (BERTScore F1 = 0.95) and by three native AAE speakers (83.0% semantic agreement).
Read the full abstract ▾
All authors Huan Wu, Ali Emami, Muhammad Furquan Hassan, Osaretin Igbinoba, Osakpolor Idusuyi, Osamede Igbinoba, Faiza Khan Khattak, Laleh Seyyed-Kalantari
Computation and Language
Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs
Albert Zeyer, Ralf Schlüter and Hermann Ney
Speech-to-text
alignment means finding the temporal boundaries of each word in the audio. Some models provide such an
alignment directly and others do not. Connectionist temporal classification (CTC) and transducer models have an
alignment by construction, whereas
attention-based encoder-decoders (AED) and
speech large
language models (LLMs) do not, and their word timings are usually read off the
attention weights instead. All of these signals live on the encoder frame grid, which bounds their temporal precision. We study a generic gradient-based
alignment that applies to any differentiable ASR model. We take the gradient of each teacher-forced
token log probability with respect to the input, reduce it to a per-frame saliency, and decode the resulting
matrix into word boundaries with a single dynamic-programming pass. The method needs no training, no model modification and no
alignment heads, works across all model families including the
speech LLMs, and aligns on the input grid rather than on the coarser encoder grid. We evaluate it on sixteen models from four families, on read (TIMIT) and spontaneous (Buckeye)
speech, each against the model's own native or
attention-based
alignment. We find that the gradient yields a usable
alignment for every model, that it is usually somewhat behind a strong native aligner but better where the native
alignment is weak, as for the streaming models, and that its main disadvantage is the cost of one backward pass per
token.
Read the full abstract ▾
Computation and Language
Ad Headline Generation using Self-Critical Masked Language Model
Yashal Shakti Kanungo, Sumit Negi and Aruna Rajan
For any E-commerce website it is a nontrivial problem to build enduring advertisements that attract shoppers. It is hard to pass the creative quality bar of the website, especially at a large scale. We thus propose a programmatic solution to generate product advertising headlines using retail content. We propose a state of the art application of Reinforcement Learning (RL) Policy gradient methods on
Transformer based Masked
Language Models. Our method creates the advertising headline by jointly conditioning on multiple products that a seller wishes to advertise. We demonstrate that our method outperforms existing
Transformer and LSTM + RL methods in overlap metrics and quality audits. We also show that our model-generated headlines outperform human submitted headlines in terms of both grammar and creative quality as determined by audits.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Trees from Marginals: Autoregressive drafting with factorized priors
Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy et al. (+1 more)
Speculative decoding greatly increases the interactivity of autoregressive
language models by trading off computation for extra
tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-
token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed
tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification
algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.08-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 16.3%.
Read the full abstract ▾
All authors Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy, Artur Chakhvadze
Computation and Language · Cross-listed from cs.AI
When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoning
Yotam Wolf, Noam Wies and Amnon Shashua
Training large
language models (LLMs) with extended
reasoning has enabled in-context search, in which models iteratively generate, critique, and revise solution attempts. We provide a theoretical analysis of in-context search by modeling it as approximate inference over
reasoning traces, where the base model defines a prior and self-reflection provides feedback for posterior updates, and study the resulting inference-time sampling complexity - the number of sequential attempts needed to achieve high success probability. We show that when reflections reliably localize early mistakes, in-context search can yield exponential improvements over the base model, solving problems with exponentially small zero-shot pass rates using only a
polynomial number of sequential attempts, whereas when this property fails, conditioning on past attempts offers no asymptotic benefit over parallel sampling. We further show that these gains are robust and learnable: approximate posterior updates suffice, and cross-
entropy training on search rollouts recovers the required behavior with
polynomial sample complexity. Finally, we show that under a stagewise abstraction of reinforcement learning with verifiable rewards, the optimal policy extension implements the same posterior reweighting rule. We validate key qualitative predictions of the theory on real large
reasoning models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories
Hengyu Jin, Shu Yang and Di Wang
Latent
reasoning performs multi-step inference in continuous hidden states, promising more compact and efficient
reasoning. However, these opaque states raise a question of faithfulness: whether the latent
reasoning steps drive the final answer. Prior work studies this question at selected checkpoints and reports several unfaithful behaviors. This endpoint view leaves how evidence of faithfulness evolves during training unexamined. We track behavioral and activation-based evidence across training using verified counterfactual edits and interventions on the latent
reasoning states. We find that high task accuracy can coexist with low counterfactual responsiveness: as accuracy improves, responsiveness can decline, and different latent
reasoning approaches follow distinct trajectories. On ProsQA, output sensitivity to norm-noise replacement declines alongside counterfactual responsiveness, although the result depends on the replacement. Across separately trained binary-choice and open-ended GSM settings, intervention sensitivity follows opposite trajectories. These results show that evaluating only a final checkpoint can obscure both when counterfactual responsiveness changes and what the latent states contribute.
Read the full abstract ▾
Computation and Language
Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
Felix Feldman, Joshua Harris, Timothy Laurence et al. (+7 more)
Large
language models (LLMs) achieve promising results on medical question answering
benchmarks, yet their use in public health is constrained by
hallucinations and the rapid
evolution of official guidance.
Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in an explicitly maintained corpus, but end-to-end performance depends critically on
retrieval configuration and on evaluation beyond multiple-choice formats. We extend PubHealthBench, a question answering (QA)
benchmark of 7,929 questions derived from UK Government public health guidance, into a
retrieval-augmented setting and systematically evaluate
retrieval and generation choices. We compare dense, sparse, and hybrid
retrieval across multiple
embedding models and corpus variants, and show that hybrid
retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance. Providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or outperform larger models used without
retrieval, with gains primarily driven by
retrieval quality and careful context selection. To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, and validate it against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale. Overall, our results highlight
retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.
Read the full abstract ▾
All authors Felix Feldman, Joshua Harris, Timothy Laurence, Leo Loman, Ollie Higgins, Fan Grayson, Poonam Soma, Bethany Pace-Bonello, Michael Borowitz, Toby Nonnenmacher
Computation and Language
Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
Andrei-George Durdun, Victor Constantinescu and Radu Tudor Ionescu
Automatically recognizing the sentiment, positive or negative, from
speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words. Recent solutions rely on audio foundation models to solve the task, but it remains unclear if such models can take all aspects into account. To this end, we propose a
multimodal solution that integrates audio and text information via cross-modal
transformers, where text transcripts are automatically generated via an automatic
speech recognition (ASR) tool. Moreover, we create multiple text modalities by automatically translating the transcripts into multiple
languages via machine translation tools. Audio and
multilingual text features are combined via a cascaded architecture comprising cross-modal
transformer blocks that integrate modalities one by one. We further distill knowledge from the
multimodal model, called teacher, into a unimodal (audio only) model, called student. We conduct experiments on a large-scale
dataset, demonstrating that the automatically generated textual information can bring significant performance boosts in
multimodal sentiment polarity classification. Our ablation study confirms that both automatic transcripts and automatic translations are helpful. Moreover, we show that the audio-only model can be enhanced via
distillation, boosting performance without any computational overhead during inference. To reproduce the reported results, we publicly release our code at https://github.com/andreidurdun/cross-modal-audio-sentiment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
Aparna Madva, Sharath Srivatsa, Srinath Srinivasa et al. (+1 more)
As Artificial Intelligence (AI) makes inroads into different parts of the Indian subcontinent, there is significant interest in studying how AI impacts the linguistic and cultural foundations of this civilization. AI is seen as a ”double-edged sword” where on the one hand, it can enable access and inclusion for a large population, on the other, it can homogenize worldviews and exclude underrepresented
languages and worldviews. In this paper, we try to characterize this problem by addressing the extensive characteristic nature of Indian linguistics and the way they closely connect to cultural practices and worldview. We then perform a longitudinal survey of how Natural
Language Processing (NLP) techniques have evolved in this space, tracing the historical development of Indic NLP, covering key milestones, methodological shifts, and resource creation efforts. In addition, the paper also examines the structural and sociolinguistic characteristics of Indian
languages, such as rich morphology, complex scripts and grammar rules, diglossia, and large dialectal variation, and explains how these create unique challenges for building AI foundation models. We then discuss the growing role of Indic foundation models and analyze how these models address these long-standing resource and representation gaps. Finally, we propose a research direction called 'Culture Sensing', which re-imagines AI based on hermeneutic
reasoning. Culture Sensing aims to address open problems such as ensuring equitable performance across low-resource
languages and producing outputs that are culturally meaningful. By bringing together past work, current techniques, and emerging trends, this paper outlines research directions that can guide the next phase of Indic NLP and contribute to the development of more robust and inclusive Indic foundation models.
Read the full abstract ▾
All authors Aparna Madva, Sharath Srivatsa, Srinath Srinivasa, Tulika Saha
Computation and Language
On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?
Ramon Ferrer-i-Cancho, Catherine Hobaiter, Thore Bergman et al. (+1 more)
Dependency parsing consists of finding a tree representation for a sequence. Unsupervised dependency parsing aims to develop parsing methods without a gold standard during model training. In human
languages, an unsupervised parser can be evaluated because some gold standard is usually available or can be created. For other species, a gold standard is unknown. Thus one may conclude that it is impossible to determine the accuracy of an unsupervised parser and, consequently, dependency parsing is unfeasible in other species. However, here we apply recent advances in network science to demonstrate that the proportion of correct edges retrieved by a parser must be high for the sequences of vocalizations or gestures that non-human primates produce due to the fast decay of the sequence length distribution. In contrast, human
language sequences lack that property. Therefore, evaluation without a gold standard is feasible in non-human primates but a hard problem in humans.
Read the full abstract ▾
All authors Ramon Ferrer-i-Cancho, Catherine Hobaiter, Thore Bergman, Morgan Gustison
Computation and Language
Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs
Zhenyu Liu, Xuanyu Zhang, Yunxin Li et al. (+10 more)
Developing seamless, high-performance, native intelligent full-duplex Spoken
Language Models (SLMs) remains a critical challenge and long-standing goal for the
speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity – ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic
alignment channel. Extensive experiments on multiple full-duplex
benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both
speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.
Read the full abstract ▾
All authors Zhenyu Liu, Xuanyu Zhang, Yunxin Li, Qixun Teng, Shenyuan Jiang, Haolan Chen, Minjun Zhao, Fanbo Meng, Yu Xu, Yancheng He, Baotian Hu, Haizhou Li, Min Zhang
Computation and Language
Life Style Levels: Neighborhood Delineation using Geospatial Data
Srivatsa Kulkarni and Debarag Banerjee
Fine-scale socioeconomic information is often unavailable across rapidly ur-banizing regions of the developing world, like India, limiting the ability to delineate intra-urban variations in affluence and deprivation. This study pro-poses a scalable, grid-based urban delineation framework using building morphology derived from open-source satellite imagery. Urban areas across 59 Indian cities and towns are partitioned into high-resolution spatial grids and characterized using interpretable morphological indicators, which are combined into a transparent, rule-based scoring framework to delineate areas with contrasting levels of urban affluence. The resulting classifications are validated through ground-level Google Street View observations, revealing a sharp contrast between the grid classes which are consistent with the ex-pected effects of the lifestyle affluence indicators. We further investigate density-based clustering of building footprints in Mumbai to identify dense urban settlements, demonstrating that the resulting clusters exhibit substan-tial spatial overlap with known informal settlements across the city. Finally, we conduct an exploratory analysis mapping consumer loan delinquency across the derived affluence classes. By relying entirely on publicly available geospatial data, the proposed framework provides a scalable, interpretable, and cost-effective approach for granular urban affluence mapping across In-dian cities.
Read the full abstract ▾
Computation and Language
RSF-GLLM: Bridging the Semantic Gap in Multi-Hop Knowledge Graph QA via Recurrent Soft-Flow and Decoupled LLM Generation
Sambaran Bandyopadhyay and Ananth Muppidi
Multi-hop Question Answering over Knowledge Graphs faces a critical challenge: traditional retrieve-then-read pipelines break differentiability, preventing the retriever from learning to bridge the semantic gap where intermediate nodes lack lexical overlap with the query. To address this, we propose RSF-GLLM, a framework decoupling differentiable graph
reasoning from answer generation. Our Recurrent Soft-Flow (RSF) module employs a GRU-guided query updater to propagate continuous relevance scores, utilizing a dynamic gating mechanism to traverse semantically dissimilar bridge nodes via structural cues. We introduce flow sparsity regularization to theoretically guarantee convergence from soft probabilities to discrete
reasoning paths. These paths are extracted and textualized to fine-tune a Large
Language Model (LLM), ensuring generation is grounded in factual
topology. Experiments on WebQSP and CWQ demonstrate that RSF-GLLM achieves competitive performance with superior inference efficiency compared to LLM based computationally expensive approaches.
Read the full abstract ▾
Computation and Language
DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Generation
Chenyu Zhou, Yaqi Wu, Xiaolei Guo et al. (+7 more)
Multi-hop
retrieval-augmented generation (RAG) acquires evidence sequentially, with each document contributing supporting facts, bridge entities, query refinements, or sufficient evidence for answering. Evidence acquisition can involve iterative
retrieval, query reformulation, evidence assessment, and sufficiency checking. We introduce DynaKRAG, a unified evidence-action framework that learns a shared state-conditioned policy for coordinating these operations. At each step, a deterministic validity layer constructs the executable action set, a learned continuation gate selects between answer generation and further evidence acquisition, and a learned advantage scorer ranks feasible evidence operations by their predicted gain relative to immediate answer generation. The selected operation updates the shared state and may enable additional operations. Across HotpotQA, 2Wiki, and MuSiQue with Qwen2.5-7B, GPT-4o-mini, and Llama-3.1-8B, DynaKRAG ranks first among the compared methods in both EM and F1 for all nine
dataset–backbone pairs. Relative to matched-backbone baseline method, DynaKRAG improves F1 in every pair while achieving total-
token efficiency gains of 10.1–34.3% and
retrieval-call efficiency gains of 15.1–43.4%, establishing Pareto dominance under these measures. With Qwen2.5-7B, terminal evidence compression further improves answer quality across all three
datasets while reducing the context passed to final answer generation by 54.4%–71.5%. These results demonstrate that unified, state-conditioned evidence control supports strong answer quality, efficient
retrieval, and compact answer-generation contexts.
Read the full abstract ▾
All authors Chenyu Zhou, Yaqi Wu, Xiaolei Guo, Jiaqi Huang, Xianfa Zhang, Junxu Zhang, Zhuo Yu, Zhubo Shi, Jianghao Lin, Dongdong Ge
Computation and Language
Pitwall: Faithful Natural-Language Race-Strategy Briefings from a Calibrated Real-Time Monte Carlo Engine
Juan S. Santillana
Live sports commentary is grounded generation under a deadline: statements concern real, named athletes, the grounding state changes every few seconds, and no reference text exists at generation time. We present Pitwall, a production system that generates natural-
language Formula 1 strategy briefings in English, Spanish, and Portuguese, treating faithfulness as an architectural property rather than an aspiration: every published sentence is decomposed into typed factual claims (positions, gaps, tyres, pace, overtakes, race control) and each claim is verified against the probabilistic race state that prompted it. The same verifier gates the fine-tuning data: of 3,045 model-written targets, only the 81.9% whose every claim is state-supported are retained, the rest falling back to a provably faithful template, so the generator never sees an ungrounded target. Verification is meaningful because of the grounding substrate: a vectorized Monte Carlo engine (N=2,000 per-lap race continuations) calibrated on 126 races (2018-2024) and validated on fully held-out 2025-2026 seasons (winner-in-top-3 90.3% over 155 backtests; held-out Brier 0.0745). A recurring finding spans both halves of the system: virtues trade off and must be gated separately. In simulation, calibration-optimal is not decision-optimal; in generation, fine-tuning on richer targets buys vividness that collapses into
hallucination when the grounding state is sparse – a failure a four-base replication traces to base-model instruction adherence, not scale, and that sparse-context
auditing removes from the production model. End-to-end operation – live timing to verified trilingual briefings – was confirmed at two consecutive live Grands Prix (Austria and Britain, 2026); at Silverstone a timestamped probability trace, committed to disk before the outcome was known, locked onto the eventual winner ten laps before the flag.
Read the full abstract ▾
Computation and Language
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
So Hasegawa, Shailaja Keyur Sampat, Lei Liu et al. (+1 more)
Current
benchmarks for evaluating Large
Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact
retrieval from small tables and overlook the challenges of large multi-tabular
datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a
benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The
benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging
benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.
Read the full abstract ▾
All authors So Hasegawa, Shailaja Keyur Sampat, Lei Liu, Wei-Peng Chen
Computation and Language · Cross-listed from eess.AS
WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
Sihang Nie, Jinxin Ji, Xiaofen Xing et al. (+4 more)
While recent Large
Language Model (LLM)-based Text-to-
Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal
alignment, such as audiobook narration and
video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated
datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual
dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-
token mechanism within the LLM to formulate an explicit “acoustic planning” process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the
token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete
tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.
Read the full abstract ▾
All authors Sihang Nie, Jinxin Ji, Xiaofen Xing, Deyi Tuo, Chengbin Jin, Jialong Mai, Xiangmin Xu
Computation and Language
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
Taeyun Roh, Eunha Lee, Wonjune Jang et al. (+3 more)
Biomedical question answering requires not only accurate extraction of information from scientific literature but also reliable integration of evidence across multiple documents. This study presents a question-type-specific large
language model (LLM) framework for BioASQ 14b Task B, designed to improve answer robustness and evidence grounding in biomedical question answering. Rather than applying a single prompting strategy to all questions, the framework selects different inference procedures for yes/no, factoid, and list questions according to their distinct
reasoning and evaluation requirements. For yes/no questions, snippet shuffling and self-reflection are used to reduce sensitivity to evidence ordering and improve decision stability. For factoid questions, full-snippet input is combined with chain-of-thought-based in-context learning to support accurate biomedical entity identification. For list questions, a multi-
agent architecture is employed, in which evidence extraction, candidate generation, answer verification, and final aggregation are handled collaboratively. Preliminary experiments on BioASQ 13b were used to identify effective inference strategies for each question type, and the resulting framework was subsequently evaluated in the official BioASQ 14b Task B challenge. In the official evaluation, our framework showed competitive performance across multiple batches and achieved first place in the factoid subtask of Batch 4. These results demonstrate the effectiveness of combining question-type-specific inference, ensemble prediction, and
agent-based verification for reliable biomedical question answering.
Read the full abstract ▾
All authors Taeyun Roh, Eunha Lee, Wonjune Jang, Sohyun Chung, Junha Jung, Jaewoo Kang
Computation and Language · Cross-listed from cs.AI
Danus: Orchestrating Mathematical Reasoning Agents with Fact-Graph Memory
Jihao Liu, Guoxiong Gao, Zeming Sun et al. (+8 more)
Recent LLM-based mathematical
reasoning agents have begun to tackle research-level problems and, in several cases, have contributed to the resolution of open problems. However, scaling and orchestrating such
agents effectively remains challenging, due to the difficulty of coordinating parallel proof search while keeping intermediate claims organized and reliable. In this paper, we propose Danus, an orchestration system for research-level mathematical
reasoning centered on a shared fact graph as a global
memory-management mechanism. Danus consists of a main
agent that performs planning and coordination, multiple worker
agents that carry out proof search in parallel, and a stateless verifier that checks proposed mathematical claims before they are admitted into the fact graph. Each verified fact is stored together with its proof and logical dependencies, allowing the system to build long arguments incrementally while keeping the shared proof state organized. The main
agent periodically summarizes the evolving proof state, redirects workers across promising directions, and supports interaction with human mathematicians through progress reports. We evaluate Danus through six research-level case studies in algebraic geometry, singularity theory, and combinatorics, illustrating how the fact-graph
memory mechanism enables Danus to construct long, detailed mathematical proofs. Our results suggest that fact-graph-based orchestration provides an effective route toward scaling mathematical
reasoning agents for long-horizon research problems. Danus is open source at https://github.com/frenzymath/Danus.
Read the full abstract ▾
All authors Jihao Liu, Guoxiong Gao, Zeming Sun, Bin Wu, Shurui Liu, Jiedong Jiang, Haocheng Ju, Leheng Chen, Ronnie Cheng, Xiping Zhang, Bin Dong
Computation and Language · Cross-listed from cs.SE
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
Evgeny Shilov
Developers increasingly delegate real maintenance work to product-grade coding
agents, and many state tasks in their native
language, in the style of a customer request rather than a curated English issue. We introduce RuBench 1.0, a
benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify), each specified natively in Russian – written from scratch, not translated – and judged by the upstream maintainer's regression tests, which we withhold from release. All fix commits postdate the training-data cutoffs of every evaluated model. Round 1 evaluates Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 (3 independent runs each; pass@1 with task-level uncertainty); the best configuration resolves 78.7% of tasks.
Auditing full trajectories of an hors-concours configuration (Claude Code + Fable 5), we caught the product silently substituting the model on 20% of tasks via an official safeguard fallback – evidence that the deployed product, not the model, is the unit actually measured. Version 2 adds Round 2: seven further configurations on the same frozen set under a per-configuration freshness gate – the Russian-
market agents SourceCraft CLI (ds, legacy) and Koda CLI (koda-pro), Antigravity with Gemini 3.1 Pro and 3.5 Flash, and Codex CLI with GPT-5.6 Sol and Luna. SourceCraft's flagship resolves 68.1% (N=23), above GPT-5.5 and both Gemini rows. A tool-call contamination re-audit of all 437 Round-2 trajectories finds the Russian and Gemini columns clean (0/293
cells) while flagging systematic oracle-hunting in the GPT-5.6 family (8/69 and 13/75
cells), including one case of mining a prior round's artifacts from the run machine's disk; honest scores are published alongside raw ones. We release statements, metadata, trajectories, and diffs; oracles are withheld with a SHA-256 manifest.
Read the full abstract ▾
Computation and Language
Automated Compliance Mapping in Cloud Security with Domain-Adapted Sentence Transformers
John Bianchi, Luca Petrillo, Fabio Martinelli et al. (+1 more)
Mapping cloud security controls to technical metrics is currently a manual process. This paper proposes domain adaptation of Sentence
Transformer models to automate it. We build a training corpus of 3,499 semantic pairs from five European security standards and a set of technical metrics, then expand it via back-translation and LLM-based paraphrasing to up to 13,996 samples across four scenarios. We fine-tune five architectures and evaluate their performance on two independent tasks: control-to-metric and cross-standard controls association. All fine-tuned models outperform their zero-shot baselines. On the control-to-metric task, the best model gains up to 23 nDCG@10 points, while on the cross-standard control task, multi-qa-mpnet-dot-v1 under back-translation reaches 0.870 nDCG@10. The results show that in-domain training data is a primary driver of performance for the considered case studies.
Read the full abstract ▾
All authors John Bianchi, Luca Petrillo, Fabio Martinelli, Marinella Petrocchi
Computation and Language
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Andrea Alfarano, Andrea Bacciu, Saab Mansour et al. (+2 more)
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22
languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q&A
datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form
reasoning, avoiding LLM-as-a-judge and
embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource
languages substantially improves UE performance, suggesting that comprehension of low-resource
languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource
languages, demonstrating that generation
language matters more than the question
language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in
multilingual settings.
Read the full abstract ▾
All authors Andrea Alfarano, Andrea Bacciu, Saab Mansour, Amin Mantrach, Marcello Federico
Computation and Language
From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition
Lukmal Ilyas and Nevidu Jayatilleke
Dhivehi, the national
language of the Maldives, is currently under-resourced for automatic
speech recognition (ASR) and other NLP tasks. This study investigates whether cross-lingual transfer learning from Sinhala, a linguistically related, relatively well-resourced Insular Indo-Aryan
language, can improve Dhivehi ASR. We conduct seventeen experiments across five transfer learning paradigms: Dhivehi-only baselines, sequential fine-tuning,
multilingual fine-tuning, continual pre-training, and a control using Turkish as an unrelated
language. The strongest system, continual pre-training on Sinhala followed by fine-tuning on Dhivehi with KenLM, achieves 12.89% WER and 2.70% CER, outperforming the Dhivehi-only baseline by 13.50% WER and 3.02% CER. However, the adaptation strategy and decoding configuration are equally critical for a successful transfer learning experiment. We conduct seventeen controlled experiments spanning five transfer learning paradigms: Dhivehi-only baselines, sequential fine-tuning,
multilingual fine-tuning, continual pre-training, and a control experiment using Turkish as an unrelated
language. The strongest system, continual pre-training on Sinhala followed by fine-tuning on Dhivehi with KenLM, achieves 12.89% WER and 2.70% CER, outperforming the Dhivehi-only baseline by 13.50% WER and 3.02% CER. The Turkish control experiment confirms that observed improvements stem from linguistic relatedness; adaptation strategy and decoding configuration are also critical.
Read the full abstract ▾