Computation and Language · Cross-listed from cs.CR
Training a General Purpose Automated Red Teaming Model
Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea et al. (+1 more)
Automated methods for red teaming LLMs are an important tool to identify LLM vulnerabilities that may not be covered in static
benchmarks, allowing for more thorough probing. They can also adapt to each specific LLM to discover weaknesses unique to it. Most current automated red teaming methods are intended for tackling safety and content moderation. Thus, they make use of content safety models as evaluators and optimize for circumventing them, and as such, have not been tested with other
adversarial intents not typically captured by these. We propose a pipeline for training a red teaming model that can generalize to arbitrary
adversarial goals, including objectives it has not been directly trained on, and that does not depend on the existence of a pre-existing evaluator available at training time. We demonstrate that finetuning small models, such as Qwen3-8B, using this pipeline results in a substantial improvement in their ability to generate attacks for both in and out of domain
adversarial goals.
Read the full abstract ▾
All authors Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea, Christopher Parisien
Computation and Language
Implicit Framing in Obstetric Counseling Notes: A Grounded LLM Pipeline on a VBAC-Eligible Cohort
Baris Karacan, Barbara Di Eugenio, Patrick Thornton et al. (+2 more)
Clinical framing – the linguistic manner in which clinical information is presented – can influence patient understanding and decision-making, with important implications for healthcare outcomes. Obstetrics is a high-stakes domain in which physicians counsel patients on delivery mode choices such as vaginal birth after cesarean (VBAC) and repeat cesarean section (RCS), yet counseling
language remains underexplored in large-scale clinical text analysis. In this work, we analyze physician counseling
language in 2,024 obstetric history and physical narratives for a rigorously defined cohort of patients for whom both VBAC and RCS were clinically viable options. To control for confounding due to medical contraindications, we first construct a VBAC-eligible cohort using structured clinical data supplemented by a large
language model (LLM)-based extraction pipeline constrained to grounded, verbatim evidence from free-text narratives. We then apply a zero-shot LLM framework to categorize counseling segments into predefined framing categories capturing how physicians linguistically present delivery options. Our analysis reveals a significant difference in counseling framing distributions between VBAC and RCS notes; risk-focused
language accounts for a substantially larger share of counseling segments in RCS documentation than in VBAC, with category-level differences confirmed by statistical testing, highlighting the value of controlled LLM-based framing analysis in obstetric care.
Read the full abstract ▾
All authors Baris Karacan, Barbara Di Eugenio, Patrick Thornton, Joanna Tess, Subhash Kumar Kolar
Computation and Language
DeepImagine: Clinical Trial Outcome Prediction via Stepwise Local Counterfactual Imaginations
Youze Zheng, Jianyou Wang, Yuhan Chen et al. (+8 more)
Predicting the outcomes of prospective clinical trials remains a major challenge. Clinical trial outcomes result from complex interactions among experimental factors such as drug interventions, participant demographics, and protocols. Here, we introduce DeepImagine, a framework that predicts target trial outcomes through stepwise counterfactual imagination anchored on historical trials with observed results. Starting from a relevant historical trial, DeepImagine sequentially modifies one differing experimental factor at a time. With each step a large
language model (LLM) is posed a local counterfactual: how would the current imagined outcome change with this single perturbation? The updated result is carried forward as the input to the next step, until the historical configuration exactly matches the target, yielding the final prediction. Empirically, DeepImagine consistently outperforms direct one-step prediction across several off-the-shelf LLMs, with further gains when multiple imagination pathways, initiated from different historical anchors, are aggregated. We also construct natural counterfactuals augmented with synthetic
reasoning traces and train a family of specialized
language models, each dedicated to learning one factor's local counterfactual transition. Integrating these learned local operators into DeepImagine yields substantial improvements over general-purpose LLM baselines. Our findings position stepwise counterfactual imagination, distinct from both correlational prediction and explicit structural causal modeling, as a promising direction for clinical trial outcome prediction. All code, training scripts, and evaluation scripts are available at \href{https://github.com/deepimagine-counterfactual/DeepImagine}{https://github.com/deepimagine-counterfactual/DeepImagine}.
Read the full abstract ▾
All authors Youze Zheng, Jianyou Wang, Yuhan Chen, Matthew Feng, Longtian Bao, Hanyuan Zhang, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Umber Dube, Ramamohan Paturi
Computation and Language
Evaluating Temporal Consistency in Multi-Turn Language Models
Yash Kumar Atri, Steven L. Johnson and Tom Hartvigsen
Language models are increasingly deployed in interactive settings where users reason about facts over time rather than in isolation. In such scenarios, correct behavior requires models to maintain and update implicit temporal assumptions established earlier in a conversation. We study this challenge through the lens of temporal scope stability: the ability to preserve, override, or transfer time-scoped factual context across dialogue turns. We introduce ChronoScope, a large-scale diagnostic
benchmark designed to isolate temporal scope behavior in controlled multi-turn interactions, comprising over one million deterministically generated question chains grounded in Wikidata. ChronoScope evaluates whether models can correctly retain inferred temporal scope when follow-up questions omit explicit time references, spanning implicit carryover, explicit scope switching, cross-entity transfer, and longer temporal trajectories. Through extensive evaluation of state-of-the-art
language models, we find that temporal scope stability is frequently violated in controlled multi-turn settings, with models often drifting toward present-day assumptions despite correct underlying knowledge. These failures intensify with interaction length and persist even under oracle context conditions, revealing a gap between single-turn factual accuracy and coherent temporal
reasoning under sequential interaction. We make our
dataset and evaluation suite publicly available at https://github.com/yashkumaratri/ChronoScope
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Haoze He, Xingyuan Ding, Xuan Jiang et al. (+5 more)
Despite MoE models leading many
benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMixer and ESFT mitigate router collapse with dense mixing or auxiliary load-balancing losses, but these introduce noisy gradients that often degrade performance. In preliminary experiments, we systematically pruned experts and observed that while certain super experts are activated far more frequently, discarding less used experts still leads to notable performance degradation. This suggests that even rarely activated experts encode non-trivial knowledge useful for downstream tasks. Motivated by this, we propose an auxiliary-loss-free MoE SFT framework that combines bias-driven sparsification with always-active gated condenser experts. Rather than enforcing balanced activation across all experts, our method encourages task-relevant experts to remain active while pushing long-tailed experts toward inactivity. The condenser experts provide a persistent, learnable pathway that alleviates gradient starvation and facilitates consolidation of information that would otherwise remain fragmented across sparsely activated experts. Analysis further suggest that this design better preserves long-tailed expert information under sparse routing. Experiments on large-scale MoE models demonstrate that our approach outperforms state-of-the-art SFT baselines such as DenseMixer and ESFT, achieving average gain of 2.5%+ on both mathematical
reasoning and commonsenseQA
benchmarks.
Read the full abstract ▾
All authors Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller
Computation and Language
Chinese-SkillSpan: A Span-Level Dataset for ESCO-Aligned Competency Extraction from Chinese Job Ads
Guojing Li, Zichuan Fu, Junyi Li et al. (+6 more)
Job Skill Named Entity Recognition (JobSkillNER) aims to automatically extract key skill information from large-scale job posting data, which is important for improving talent-
market matching efficiency and supporting personalized employment services. To the best of our knowledge, this work presents the first Chinese JobSkillNER
dataset for recruitment texts. We propose annotation guidelines tailored to Chinese job postings and an LLM-empowered Macro-Micro collaborative annotation pipeline. The pipeline leverages the contextual understanding ability of large
language models (LLMs) for initial annotation and then refines the results through expert sentence-level adjudication. Using this pipeline, we annotate more than 20,000 instances collected from four major recruitment platforms over the period 2014-2025. Based on these efforts, we release Chinese-SkillSpan, the first Chinese JobSkillNER
dataset aligned with the ESCO occupational skill standard across four dimensions: knowledge, skill, transversal competence, and
language competence (LSKT). Experimental results show that the
dataset supports effective model training and evaluation, indicating that Chinese-SkillSpan helps fill a major gap in Chinese JobSkillNER resources and provides a useful
benchmark for intelligent recruitment research. Code and data are available at https://sites.google.com/view/cn-skillspan-resources .
Read the full abstract ▾
All authors Guojing Li, Zichuan Fu, Junyi Li, Wenxia Zhou, Xinyang Wu, Jinning Yang, Jingtong Gao, Feng Huang, Xiangyu Zhao
Computation and Language · Cross-listed from cs.AI
FormalScience: Scalable Human-in-the-Loop Autoformalisation of Science with Agentic Code Generation in Lean
Jordan Meadows, Lan Zhang and Andre Freitas
Formalising informal mathematical
reasoning into formally verifiable code is a significant challenge for large
language models. In scientific fields such as physics, domain-specific machinery (e.g. Dirac notation, vector calculus) imposes additional formalisation challenges that modern LLMs and agentic approaches have yet to tackle. To aid autoformalisation in scientific domains, we present FormalScience; a domain-agnostic human-in-the-loop agentic pipeline that enables a single domain expert (without deep formal
language experience) to produce syntactically correct and semantically aligned formal proofs of informal
reasoning for low economic cost. Applying FormalScience to physics, we construct FormalPhysics, a
dataset of 200 university-level (LaTeX) physics problems and solutions (primarily
quantum mechanics and electromagnetism), along with their Lean4 formal representations. Compared to existing formal math
benchmarks, FormalPhysics achieves perfect formal validity and exhibits greater statement complexity. We evaluate open-source models and proprietary systems on a statement autoformalisation task on our
dataset via zero-shot prompting, self-refinement with error feedback, and a novel multi-stage agentic approach, and explore autoformalisation limitations in modern LLM-based approaches. We provide the first systematic characterisation of semantic drift in physics autoformalisation in terms of concepts such as notational collapse and abstraction elevation which reveals what formal
language verifies when full semantic preservation is unattainable. We release the codebase together with an interactive UI-based FormalScience system which facilitates autoformalisation and
theorem proving in scientific domains beyond physics.https://github.com/jmeadows17/formal-science
Read the full abstract ▾
Computation and Language
Uncertainty Quantification for LLM Function-Calling
Zihuiwen Ye, Lukas Aichberger, Michael Kirchhof et al. (+5 more)
Large
Language Models (LLMs) are increasingly deployed to autonomously solve real-world tasks. A key ingredient for this is the LLM Function-Calling paradigm, a widely used approach for equipping LLMs with tool-use capabilities. However, an LLM calling functions incorrectly can have severe implications, especially when their effects are irreversible, e.g., transferring money or deleting data. Hence, it is of paramount importance to consider the LLM's confidence that a function call solves the task correctly prior to executing it. Uncertainty Quantification (UQ) methods can be used to quantify this confidence and prevent potentially incorrect function calls. In this work, we present what is, to our knowledge, the first evaluation of UQ methods for LLM Function-Calling (FC). While multi-sample UQ methods, such as Semantic
Entropy, show strong performance for natural
language Q&A tasks, we find that in the FC setting the
entropy-based multi-sample methods we evaluate offer no clear advantage over simple single-sample UQ methods. Additionally, we find that the particularities of FC outputs can be leveraged to improve the performance of existing UQ methods in this setting. Specifically, multi-sample UQ methods benefit from clustering FC outputs based on their abstract syntax tree parsing, while for single-sample UQ methods, restricting the logit-based score to semantically meaningful
tokens leaves discrimination essentially unchanged but improves calibration and, by smaller margins, the utility of the score under abstention.
Read the full abstract ▾
All authors Zihuiwen Ye, Lukas Aichberger, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal, Arno Blaas, Adam Golinski
Computation and Language · Cross-listed from cs.AI
The Power of Power Law: Asymmetry Enables Compositional Reasoning
Zixuan Wang, Xingyu Dang, Jason D. Lee et al. (+1 more)
Natural
language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency. While a common intuition suggests that reweighting or curating data towards a uniform distribution may help models better learn these long-tail skills, we find a counterintuitive result: across a wide range of compositional
reasoning tasks, such as state tracking and multi-step arithmetic, training under power-law distributions consistently outperforms training under uniform distributions. To understand this advantage, we introduce a minimalist skill-composition task and show that learning under a power-law distribution provably requires significantly less training data. Our theoretical analysis reveals that power law sampling induces a beneficial asymmetry that improves the pathological loss landscape, which enables models to first acquire high-frequency skill compositions with low data complexity, which in turn serves as a stepping stone to efficiently learn rare long-tailed skills. Our results offer an alternative perspective on what constitutes an effective data distribution for training models.
Read the full abstract ▾
All authors Zixuan Wang, Xingyu Dang, Jason D. Lee, Kaifeng Lyu
Computation and Language
Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledge
Mengyu Wang, Xiaoying Zhi, Zhiyi Li et al. (+4 more)
While the next-
token prediction (NTP) paradigm enables large
language models (LLMs) to express their intrinsic knowledge, its sequential nature constrains performance on specialized, non-generative tasks. We attribute this performance bottleneck to the LLMs' knowledge expression mechanism, rather than to deficiencies in knowledge acquisition. To address this, we propose Self-Knowledge Re-expression (SKR), a novel, task-agnostic adaptation method. SKR transforms the LLM's output from generic
token generation to highly efficient, task-specific expression. SKR is a fully local method that uses only unannotated data, requiring neither human supervision nor model
distillation. Experiments on a large financial document
dataset demonstrate substantial improvements: over 40% in Recall@1 for information
retrieval tasks, over 76% reduction in object detection latency, and over 33% increase in anomaly detection AUPRC. Our results on the MMDocRAG
dataset surpass those of leading
retrieval models by at least 12.6%.
Read the full abstract ▾
All authors Mengyu Wang, Xiaoying Zhi, Zhiyi Li, Robin Schmucker, Shay B. Cohen, Tiejun Ma, Fran Silavong
Computation and Language · Cross-listed from cond-mat.mtrl-sci
Large language model-enabled automated data extraction for concrete materials informatics
Zhanzhao Li, Kengran Yang, Qiyao He et al. (+1 more)
The promise of data-driven materials discovery remains constrained by the scarcity of large, high-quality, and accessible experimental
datasets. Here, we introduce a generalizable large
language model (LLM)-powered pipeline for automated extraction and structuring of materials data from unstructured scientific literature, using concrete materials as a representative and particularly challenging example. The pipeline exhibits robust performance across a broad range of LLMs and achieves an $F_1$ score of up to 0.98 for diverse composition–process–property attributes. Within one hour, it extracts nearly 9,000 high-quality records with over 100 attributes from a corpus screened from more than 27,000 publications, enabling the construction of the largest open laboratory database for blended cement concrete. Machine learning analyses underscore the importance of large, diverse, and information-rich
datasets for enhancing both in-distribution accuracy and out-of-distribution generalization to unseen materials. The proposed pipeline is readily adaptable to other materials domains and accelerates the development of scalable data infrastructures for materials informatics.
Read the full abstract ▾
All authors Zhanzhao Li, Kengran Yang, Qiyao He, Kai Gong
Computation and Language
AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs
Pouya Pezeshkpour and Estevam Hruschka
Verification is becoming central to both reinforcement-learning-based training and inference-time control of large
language models (LLMs). Yet current verifiers face a fundamental trade-off: LLM-based verifiers are expressive but hard to control and prone to error, while deterministic executable verifiers are reliable and interpretable but often limited in capability. We study the following question: given a development set of LLM outputs and labels for a target objective, such as correctness, can we automatically induce a minimal set of Python verifiers whose joint satisfaction closely matches that objective? We propose AutoPyVerifier, a framework that uses an LLM to synthesize candidate verifier functions and then refines them through search over a directed acyclic graph (DAG). By navigating the DAG, AutoPyVerifier systematically explores the space of deterministic executable verifiers and selects a compact verifier set whose joint satisfaction best approximates the target objective. Across mathematical
reasoning, coding, function calling, and instruction-following
benchmarks for several state-of-the-art LLMs, AutoPyVerifier improves target-objective prediction by up to 55.0 F1 points over the initial LLM-generated verifier sets. Additional analyses show that the most useful verification targets vary by
benchmark and model, and that the DAG-based search shifts the learned verifier sets toward more structural and semantically grounded checks. We further show that exposing the discovered verifier set to an LLM as an external tool improves downstream accuracy by up to 17.0 points. We release our code
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
PExA: Parallel Exploration Agent for Complex Text-to-SQL
Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang et al. (+3 more)
LLM-based
agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art
benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.
Read the full abstract ▾
All authors Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur, Srivas Prasad, Yunmo Chen
Computation and Language · Cross-listed from cs.LG
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Jinming Yang, Zheng Hu, Chuxian Qiu et al. (+3 more)
LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model
alignment, leaderboard construction, quality control, and so on. However, the scalability and trustworthiness of this approach can be substantially distorted by Self-Preference Bias (SPB), which is a directional evaluative deviation in which LLMs systematically favor or disfavor their own generated outputs during evaluation. Existing measurements rely on costly human annotations and conflate generative capability with evaluative stance, and thus are impractical for large-scale deployment in real-world systems. To address this issue, we introduce a fully automated framework to quantifying and mitigating SPB, which constructs equal-quality pairs of responses with negligible quality differences, enabling statistical disentanglement of discriminability from bias propensity without human gold standards. Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB. To mitigate this bias, we propose a structured multi-dimensional evaluation strategy grounded in cognitive load decomposition, which reduces SPB by 31.5% on average.
Read the full abstract ▾
All authors Jinming Yang, Zheng Hu, Chuxian Qiu, Zhenyu Deng, Xinshan Jiao, Tao Zhou
Computation and Language
TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction
Chengye Wang, Lin Fu, Zexi Kuang et al. (+1 more)
Existing document OCR largely targets plain text or Markdown, discarding the structural and executable properties that make LaTeX essential for scientific publishing. We study page-level reconstruction of scientific PDFs into compilable LaTeX and introduce TexOCR-Bench, a
benchmark, and TexOCR-Train, a large-scale training corpus, for this task. TexOCR-Bench features a multi-dimensional evaluation suite that jointly assesses transcription fidelity, structural faithfulness, and end-to-end compilability. Leveraging TexOCR-Train, we train a 2B-parameter model, TexOCR, using supervised fine-tuning (SFT) and reinforcement learning (RL) with verifiable rewards derived from LaTeX unit tests that directly enforce compilability and referential integrity. Experiments across 21 frontier models on TexOCR-Bench show that existing systems frequently violate key document invariants, including consistent section structure, correct float placement, and valid label-reference links, which undermines compilation reliability and downstream usability. Our analysis further reveals that RL with verifiable rewards yields consistent improvements over SFT alone, particularly on structural and compilation metrics.
Read the full abstract ▾
All authors Chengye Wang, Lin Fu, Zexi Kuang, Yilun Zhao
Computation and Language
How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
Longju Bai, Zhemin Huang, Xingyao Wang et al. (+5 more)
The wide adoption of AI
agents in complex human workflows is driving rapid growth in LLM
token consumption. When
agents are deployed on tasks that require a significant amount of
tokens, three questions naturally arise: (1) Where do AI
agents spend the
tokens? (2) Which models are more
token-efficient? and (3) Can
agents predict their
token usage before task execution? In this paper, we present the first systematic study of
token consumption patterns in agentic coding tasks. We analyze trajectories from eight frontier LLMs on SWE-bench Verified and evaluate models' ability to predict their own
token costs before task execution. We find that: (1) agentic tasks are uniquely expensive, consuming 1000x more
tokens than code
reasoning and code chat, with input
tokens rather than output
tokens driving the overall cost; (2)
token usage is highly variable and inherently
stochastic: runs on the same task can differ by up to 30x in total
tokens, and higher
token usage does not translate into higher accuracy; instead, accuracy often peaks at intermediate cost and saturates at higher costs; (3) models vary substantially in
token efficiency: on the same tasks, Kimi-K2 and Claude-Sonnet-4.5, on average, consume over 1.5 million more
tokens than GPT-5; (4) task difficulty rated by human experts only weakly aligns with actual
token costs, revealing a fundamental gap between human-perceived complexity and the computational effort
agents actually expend; and (5) frontier models fail to accurately predict their own
token usage (with weak-to-moderate correlations, up to 0.39) and systematically underestimate real
token costs. Our study offers new insights into the economics of AI
agents and can inspire future research in this direction.
Read the full abstract ▾
All authors Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
Computation and Language
Representational Harms in LLM-Generated Narratives Against Global Majority Nationalities
Ilana Nguyen, Harini Suresh, Thema Monroe-White et al. (+1 more)
Large
language models (LLMs) are increasingly used for text generation tasks from everyday use to high-stakes enterprise and government applications, including simulated interviews with asylum seekers. While many works highlight the new potential applications of LLMs, there are risks of LLMs encoding and perpetuating harmful biases about non-dominant communities across the globe. To better evaluate and mitigate such harms, more research examining how LLMs portray diverse individuals is needed. In this work, we study how national origin identities are portrayed by widely-adopted LLMs in response to open-ended narrative generation
prompts. Our findings demonstrate the presence of persistent representational harms by national origin, including harmful stereotypes, erasure, and one-dimensional portrayals of Global Majority identities. Minoritized national identities are simultaneously underrepresented in power-neutral stories and overrepresented in subordinated character portrayals, which are over fifty times more likely to appear than dominant portrayals. The degree of harm is amplified when US nationality cues (e.g., “American”) are present in input
prompts. Notably, we find that the harms we identify cannot be explained away via sycophancy, as US-centric biases persist even when replacing US nationality cues with non-US national identities in the
prompts. Based on our findings, we call for further exploration of cultural harms in LLMs through methodologies that center Global Majority perspectives and challenge the uncritical adoption of US-based LLMs for the classification, surveillance, and misrepresentation of the majority of our
planet.
Read the full abstract ▾
All authors Ilana Nguyen, Harini Suresh, Thema Monroe-White, Evan Shieh
Computation and Language · Cross-listed from cs.LG
Neural Recovery of Historical Lexical Structure in Bantu Languages from Modern Data
Hillary Mutisya and John Mugane
We investigate whether neural models trained exclusively on modern morphological data can recover cross-lingual lexical structure consistent with historical reconstruction. Using BantuMorph v7, a
transformer over Bantu morphological paradigms, we analyze 14 Eastern and Southern Bantu
languages, extract encoder
embeddings for their noun and verb lemmas, and identify 728 noun and 1,525 verb cognate candidates shared across 5+
languages. Evaluating these candidates against established historical resources-the Bantu Lexical Reconstructions database (BLR3; 4,786 reconstructed Proto-Bantu forms) and the ASJP basic vocabulary-we confirm 10 of the top 11 noun candidates (90.9%) align with previously reconstructed Proto-Bantu forms, including *-ntU 'person' (8
languages), *gombe 'cow' (9
languages), and *mUn (9
languages). Extending to verbs, 12 verb cognates align with reconstructed Proto-Bantu roots, including *-bon- 'see' and *-jIm- 'stand', each attested across wide geographic ranges. Cross-model validation using an independent translation model (NLLB-600M) confirms these patterns: both models recover cognate clusters and phylogenetic groupings consistent with established Guthrie-zone classifications (p < 0.01). Cross-lingual noun class analysis reveals that all 13 productive classes maintain >0.83 cosine similarity across
languages (within-class > between-class, p < 10^-9). Our
dataset is restricted to Eastern and Southern Bantu, so we interpret these results as recovering shared Bantu lexical structure consistent with Proto-Bantu rather than definitively distinguishing Proto-Bantu retentions from later regional innovations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Zero-Shot Morphological Discovery in Low-Resource Bantu Languages via Cross-Lingual Transfer and Unsupervised Clustering
Hillary Mutisya and John Mugane
We present a method for discovering morphological features in low-resource Bantu
languages by combining cross-lingual transfer learning with unsupervised clustering. Applied to Giriama (nyf), a
language with only 91 labeled paradigms, our pipeline discovers noun class assignments for 2,455 words and identifies two previously undocumented morphological patterns: an a- prefix variant for Class 2 (vowel coalescence - the merger of two adjacent vowels - of wa-, 95.1% consistency) and a contracted k'- prefix (98.5% consistency). External validation on 444 known Giriama verb paradigms confirms 78.2% lemmatization accuracy, while a v3 corpus expansion to 19,624 words (9,014 unique lemmas) achieves 97.3%
segmentation and 86.7% lemmatization rates across all major word classes. Our ensemble of transfer learning from Swahili and unsupervised clustering, combined via weighted voting, exploits complementary strengths: transfer excels at cognate detection (leveraging ~60% vocabulary overlap) while clustering discovers
language-specific innovations invisible to transfer. We release all code and discovered lexicons to support morphological documentation for low-resource Bantu
languages.
Read the full abstract ▾
Computation and Language
Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought
Keshav Ramji, Tahira Naseem and Ramón Fernandez Astudillo
While long, explicit chains-of-thought (CoT) have proven effective on complex
reasoning tasks, they are costly to generate during inference. Non-verbal
reasoning methods have emerged with shorter generation lengths by leveraging continuous representations, yet their performance lags behind verbalized CoT. We propose $\textbf{Abstract Chain-of-Thought}$, a discrete latent
reasoning post-training mechanism in which the
language model produces a short sequence of
tokens from a reserved vocabulary in lieu of a natural
language CoT, before generating a response. To make previously unseen ”abstract”
tokens useful, we introduce a policy iteration-style warm-up loop that alternates between (i.) bottlenecking from a verbal CoT via masking and performing supervised fine-tuning, and (ii.) self-
distillation by training the model to generate abstract
tokens from the
prompt alone via constrained decoding with the codebook. After warm-up, we optimize the generation of abstract sequences with warm-started reinforcement learning under constrained decoding. Abstract-CoT achieves up to $11.6\times$ fewer
reasoning tokens while demonstrating comparable performance across mathematical
reasoning, instruction-following, and multi-hop
reasoning, and generalizes across
language model families. We also find an emergent power law distribution over the abstract vocabulary, akin to those seen in natural
language, that evolves across the training phases. Our findings highlight the potential for post-training latent
reasoning mechanisms that enable efficient inference through a learned abstract
reasoning language.
Read the full abstract ▾
Computation and Language
CRAFT: Clustered Regression for Adaptive Filtering of Training data
Parthasarathi Panda, Asheswari Swain and Subhrakanta Panda
Selecting a small, high-quality subset from a large corpus for fine-tuning is increasingly important as corpora grow to tens of millions of datapoints, making full fine-tuning expensive and often unnecessary. We propose CRAFT (Clustered Regression for Adaptive Filtering of Training data), a vectorization-agnostic selection method for training sequence-to-sequence models. CRAFT decomposes the joint source-target distribution and performs a two-stage selection: (i) match the validation source distribution through proportional budget allocation across k-means clusters, and (ii) within each source cluster, select training pairs whose target
embeddings minimize a conditional expected distance derived from the validation target distribution. We prove that proportional cluster allocation bounds the continuous KL divergence between selected and validation distributions, with the residual controlled by cluster diameters. We evaluate CRAFT on English-Hindi translation by selecting training data from 33 million NLLB sentence pairs and fine-tuning mBART via LoRA. CRAFT achieves 43.34 BLEU, outperforming TSDS (41.21) by 2.13 points on the same candidate pool and encoder while completing selection over 40 times faster. With TF-IDF vectorization, the entire pipeline completes in under one minute on CPU. TAROT achieves 45.61 BLEU, but CRAFT completes selection in 26.86 seconds versus TAROT's 75.6 seconds, a 2.8 time speedup.
Read the full abstract ▾
Computation and Language
BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
Jinghong Chen, Jingbiao Mei, Guangyu Yang et al. (+1 more)
A common approach to question answering with
retrieval-augmented generation (RAG) is to concatenate documents into a single context and pass it to a
language model to generate an answer. While simple, this strategy can obscure the contribution of individual documents, making attribution difficult and contributing to the “lost-in-the-middle” effect, where relevant information in long contexts is overlooked. Concatenation also scales poorly: computational cost grows quadratically with context length, a problem that becomes especially severe when the context includes visual data, as in visual question answering. Attempts to mitigate these issues by limiting context length can further restrict performance by preventing models from benefiting from the improved recall offered by deeper
retrieval. We propose
Bayesian Ensemble
Retrieval-Augmented Generation (BERAG), along with
Bayesian Ensemble Fine-Tuning (BEFT), as a RAG framework in which
language models are conditioned on individual retrieved documents rather than a single combined context. BERAG treats document posterior probabilities as ensemble weights and updates them
token by
token using Bayes' rule during generation. This approach enables probabilistic re-ranking, parallel
memory usage, and clear attribution of document contribution, making it well-suited for large document collections. We evaluate BERAG and BEFT primarily on knowledge-based visual question answering tasks, where models must reason over long, imperfect
retrieval lists. The results show substantial improvements over standard RAG, including strong gains on Document Visual Question Answering and
multimodal needle-in-a-haystack
benchmarks. We also demonstrate that BERAG mitigates the “lost-in-the-middle” effect. The document posterior can be used to detect insufficient grounding and trigger deflection, while document
pruning enables faster decoding than standard RAG.
Read the full abstract ▾
All authors Jinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne
Computation and Language · Cross-listed from cs.IR
Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines
Negar Arabzadeh, Andrew Drozdov, Michael Bendersky et al. (+1 more)
Large
Language Models (LLMs) have made query reformulation ubiquitous in modern
retrieval and
Retrieval-Augmented Generation (RAG) pipelines, enabling the generation of multiple semantically equivalent query variants. However, executing the full pipeline for every reformulation is computationally expensive, motivating selective execution: can we identify the best query variant before incurring downstream
retrieval and generation costs? We investigate Query Performance Prediction (QPP) as a mechanism for variant selection across ad-hoc
retrieval and end-to-end RAG. Unlike traditional QPP, which estimates query difficulty across topics, we study intra-topic discrimination - selecting the optimal reformulation among competing variants of the same information need. Through large-scale experiments on TREC-RAG using both sparse and dense retrievers, we evaluate pre- and post-
retrieval predictors under correlation- and decision-based metrics. Our results reveal a systematic divergence between
retrieval and generation objectives: variants that maximize ranking metrics such as nDCG often fail to produce the best generated answers, exposing a "utility gap" between
retrieval relevance and generation fidelity. Nevertheless, QPP can reliably identify variants that improve end-to-end quality over the original query. Notably, lightweight pre-
retrieval predictors frequently match or outperform more expensive post-
retrieval methods, offering a latency-efficient approach to robust RAG.
Read the full abstract ▾
All authors Negar Arabzadeh, Andrew Drozdov, Michael Bendersky, Matei Zaharia
Computation and Language
Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models
Felix Herron, Solange Rossato, Alexandre Allauzen et al. (+1 more)
Modern automatic
speech recognition (ASR) systems have been observed to function better for certain speaker groups (SGs) than others, despite recent gains in overall performance. One potential impediment to progress towards fairer ASR is a more nuanced understanding of the types of modeling errors that
speech encoder models make, and in particular the difference between the structure of
embeddings for high-performance and low-performance SGs. This paper proposes a framework typifying two types of error that can occur in modeling phonemes in ASR systems: random error/high variance in phoneme
embedding, vs systematic error/
embedding bias. We find that training phoneme classification probes only on a single, typically disadvantaged SG, sometimes improves performance for that SG, which is evidence for the existence of SG-level bias in phoneme
embeddings. On the other hand, we find that speakers and SGs with higher levels of phoneme variance are the same as those with worse phoneme prediction accuracy. We conclude that both types of error are present in phoneme
embeddings and both are candidate causes for SG-level unfairness in ASR, though random error is likely a greater hindrance to
fairness than systematic error. Furthermore, we find that finetuning encoder models using a
fairness-enhancing
algorithm (domain enhancing and
adversarial training) changes neither the benefits of in-domain phoneme classification probe training, nor measured levels of random
embedding error.
Read the full abstract ▾
All authors Felix Herron, Solange Rossato, Alexandre Allauzen, François Portet
Computation and Language
From graphemic dependence to lexical structure: a Markovian perspective on Dante's Commedia
Angelo Maria Sabatini
This study investigates the structural organisation of Dante's Divina Commedia through a symbolic representation based on vowel-consonant (V/C) encoding. Modelling the resulting sequence through Markov chains provides parsimonious measures of local graphemic dependence, capturing persistence and alternation patterns. Across the poem, these measures reveal canto-level variation, with a clearer longitudinal trend from the Inferno to the Paradiso in the two-state compared to the four-state model. Trigram analysis identifies a restricted set of recurrent configurations acting as graphemic probes, allowing local dependency patterns to be examined in identifiable lexical environments. A complementary classification analysis provides an independent lexical characterisation of the three cantiche and reveals a clearer longitudinal organisation, particularly within the Purgatorio. Lexical anchoring identifies correspondences between selected graphemic probes and discriminative
token features. The results provide an interpretable framework for examining correspondences between local graphemic dependence and lexical differentiation.
Read the full abstract ▾
Computation and Language
Dharma, Data and Deception: An LLM-Powered Rhetorical Analysis of Cow-Urine Health Claims on YouTube
Sheza Munir, Ratna Kandala, Anamta Khan et al. (+2 more)
Health misinformation remains one of the most pressing challenges on social media, particularly when cultural traditions intersect with scientific-sounding claims. These dynamics are not only global but also deeply local, manifesting in culturally specific controversies that require careful analysis. Motivated by this, we examine 100 YouTube transcripts that promote or debunk cow urine (gomutra) as a health remedy, focusing on rhetorical strategies such as appeals to authority, efficacy appeals, and conspiracy framing. We employ large
language models (LLMs) including GPT-4, GPT-4o, GPT-4.1, GPT-5, Gemini 2.5 Pro, and Mistral Medium 3 to annotate transcripts using a 14-category taxonomy of persuasive tactics. Our analysis reveals that promoters predominantly rely on efficacy appeals and social proof, while debunkers emphasize authority and rebuttal. Human evaluation of a subset of annotations yielded 90.1% inter-annotator agreement, confirming the reliability of our taxonomy and validation process. This work advances computational methods for misinformation analysis and demonstrates how LLMs can support large-scale studies of cultural discourse online.
Read the full abstract ▾
All authors Sheza Munir, Ratna Kandala, Anamta Khan, Deepti, Joyojeet Pal
Computation and Language · Cross-listed from cs.AI
QuantClaw: Precision Where It Matters for OpenClaw
Manyi Zhang, Ji-Fu Li, Zhongao Sun et al. (+5 more)
Autonomous
agent systems such as OpenClaw introduce significant efficiency challenges due to long-context inputs and multi-turn
reasoning. This results in prohibitively high computational and monetary costs in real-world development. While quantization is a standard approach for reducing cost and latency, its impact on
agent performance in realistic scenarios remains unclear. In this work, we analyze quantization sensitivity across diverse complex workflows over OpenClaw, and show that precision requirements are highly task-dependent. Based on this observation, we propose QuantClaw, a plug-and-play precision routing plugin that dynamically assigns precision according to task characteristics. QuantClaw routes lightweight tasks to lower-cost configurations while preserving higher precision for demanding workloads, saving cost and accelerating inference without increasing user complexity. Experiments show that our QuantClaw maintains or improves task performance while reducing both latency and computational cost. Across a range of
agent tasks, it achieves up to 21.4% cost savings and 15.7% latency reduction on GLM-5 (FP8 baseline). These results highlight the benefit of treating precision as a dynamic resource in
agent systems.
Read the full abstract ▾
All authors Manyi Zhang, Ji-Fu Li, Zhongao Sun, Xiaohao Liu, Zhenhua Dong, Xianzhi Yu, Haoli Bai, Xiaobo Xia
Computation and Language
Learning Evidence Highlighting for Frozen LLMs
Shaoang Li, Yanhang Shi, Yufei Li et al. (+10 more)
Large
Language Models (LLMs) can reason well, yet often miss decisive evidence when it is buried in long, noisy contexts. We introduce HiLight, an Evidence Emphasis framework that decouples evidence selection from
reasoning for frozen LLM solvers. HiLight avoids compressing or rewriting the input, which can discard or distort evidence, by training a lightweight Emphasis Actor to insert minimal highlight tags around pivotal spans in the unaltered context. A frozen Solver then performs downstream
reasoning on the emphasized input. We cast highlighting as a weakly supervised decision-making problem and optimize the Actor with reinforcement learning using only the Solver's task reward, requiring no evidence labels and no access to or modification of the Solver. Across sequential recommendation and long-context question answering, HiLight consistently improves performance over strong
prompt-based and automated
prompt-optimization baselines. The learned emphasis policy transfers zero-shot to both smaller and larger unseen Solver families, including an API-based Solver, suggesting that the Actor captures genuine, reusable evidence structure rather than overfitting to a single backbone.
Read the full abstract ▾
All authors Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Sandeep Pandey, Luke Simon, Xi Liu, Jian Li
Computation and Language
Using Embedding Models to Improve Probabilistic Race Prediction
Noah Dasanaike and Kosuke Imai
Estimating racial disparity requires individual-level race data, which are often unavailable due to the sensitivity of collecting such information. To address this problem, many researchers utilize
Bayesian Improved Surname Geocoding (BISG), which have critically relied on Census surname data. Unfortunately, these data capture race-surname relationships only for common surnames, omitting approximately 10% of the US population. We show that predictive performance degrades substantially for individuals with such omitted, uncommon surnames because standard BISG implementation relies on a uninformative generic prior in these cases. To address this limitation, we propose
embedding-powered BISG (eBISG), which uses pre-trained text
embeddings to represent names as dense vectors and trains neural networks on 2020 Census surname and first-name data to estimate race probabilities for names not covered in the Census. We compare five approaches: standard BISG using only surnames, BIFSG incorporating first name probabilities, surname
embedding for unlisted names, surname and first name
embedding combining both, and a full-name
embedding trained on voter file data from Southern states that captures interactions between name components. We show that each successive eBISG approach improves race prediction, with the full-name
embedding yielding the largest gains, particularly for Hispanic and Asian voters whose surnames are absent from the Census list.
Read the full abstract ▾
Computation and Language
Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners
Haidong Yuan, Haokun Zhao, Wanshi Xu et al. (+4 more)
Large
language models (LLMs) often fail to meet the pedagogical needs of K-12 English learners in non-native contexts due to a proficiency mismatch. To address this widespread challenge, we introduce a proficiency-aligned framework that adapts LLM outputs to learner abilities, using China's national curriculum (CSE) as a representative case. Our framework enables precise control over lexical complexity through a four-tier grading system, supported by a comprehensive suite of new resources: graded vocabulary lists and a multi-turn dialogue corpus. Our core technical contribution is the DDPO
algorithm,Diversity Driven Policy Optimization, a multi-turn GRPO-based approach designed to preserve dialogue diversity while holistically optimizing dialogue quality. This method significantly outperforms conventional approaches, achieving low out-of-vocabulary rates and high diversity while enhancing conversational naturalness and pedagogical value. While grounded in the CSE, our framework is designed for flexibility and can be readily adapted to other educational standards. Our models, data, and code will all be open-sourced, providing a scalable platform for personalized English speaking practice that effectively addresses the unique challenges faced by K-12 learners in non-immersive environments.
Read the full abstract ▾
All authors Haidong Yuan, Haokun Zhao, Wanshi Xu, Songjun Cao, Qingyu Zhou, Long Ma, Hongjie Fan