Computation and Language · Cross-listed from cs.SD
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jisu Jeon, Seungyeon Jwa, Joosung Lee et al. (+6 more)
Large Audio-
Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated
speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise
benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the
benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic
speech evaluation.
Read the full abstract ▾
All authors Jisu Jeon, Seungyeon Jwa, Joosung Lee, Jinhyeon Kim, Woojin Chung, Hwiyeol Jo, Jeonghoon Kim, Jonghyun Choi, Soyoon Kim
Computation and Language
Measuring User's Mental Models of Speech Translation in Human-AI Collaboration
HyoJung Han, Nishant Balepur, Jordan Boyd-Graber et al. (+1 more)
Millions of people use machine translation (MT) tools daily, yet little is known about their perception of what systems can and cannot do. This paper studies users' mental models of
speech translation systems through a new framework based on cross-lingual question answering, where users either accept MT output or request professional re-translation to answer questions based on the information presented in a foreign
language. By analyzing user behavior and accuracy trends across varying translation qualities, we examine to what extent they can predict where the system is likely to be wrong, and how this mental model evolves. Users develop stronger mental models with practice, especially when they have some knowledge of the source
language, primarily by relying on surface-level error cues. Moreover, providing
speech transcriptions can help users develop better mental models. Our results show the promise of cross-lingual question answering as a downstream task for studying MT mental models and advancing our understanding of human-AI collaboration.
Read the full abstract ▾
All authors HyoJung Han, Nishant Balepur, Jordan Boyd-Graber, Marine Carpuat
Computation and Language
The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking
Arka Ujjal Dey and John Collomosse
Fact-checking systems built on LLMs achieve high verdict accuracy on standard
benchmarks, yet routinely output Supports labels whose cited evidence does not license the claim. Structured decomposition is the natural way to inspect those warrants, but rigid extraction protocols strip the full-claim context that facets need. We introduce SIFT – claim-conditioned re-scoring of extracted evidence spans against the full claim – paired with WSP (Warranted Supports Proportion), an automatic NLI check that the cited warrant entails the claim. We evaluate on FEVER, SciFact, 5PILS, and DP across four open-source backbones. SIFT recovers accuracy on
cells where naive decomposition costs up to 27.6 points, while raising WSP above direct prompting; WSP itself calibrates against human gold evidence at AUC 0.92 and precision 0.98.
Read the full abstract ▾
Computation and Language
Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelity
Yuanhe Zhao, Tianyu Zhang, Huafei Xing et al. (+3 more)
Retrieval-Augmented Generation enhances large
language models by incorporating external knowledge, but deploying it in sensitive scenarios risks
privacy leakage via malicious
prompts. To address this, we propose a multi-
agent framework that sanitizes retrieved content through semantic rewriting. By employing three specialized
agents for
privacy extraction, semantic analysis, and reconstruction, our approach collaboratively removes sensitive identifiers while preserving the semantic core. We evaluate the framework on the ChatDoctor and Wiki-PII
datasets across six large
language models. Experimental results demonstrate a significant reduction in
privacy leakage under targeted attacks. For instance, we reduced targeted information exposure in LLaMA-3-8B from 144 instances in the baseline to just 1. Furthermore, we maintain strong contextual fidelity with a BLEU-1 score of 0.122, outperforming the existing SAGE method's 0.117. Finally, the framework operates as an asynchronous preprocessing module, introducing no additional latency to online inference, as all rewriting is executed as a one-time offline preprocessing step. To promote reproducibility, the source code of this work is publicly available at https://github.com/foursoils/
Privacy-Preserving-RAG.
Read the full abstract ▾
All authors Yuanhe Zhao, Tianyu Zhang, Huafei Xing, Derek F. Wong, Jianbin Li, Tao Fang
Computation and Language
Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models
Jory Alshaalan, Haya Albaker, Abeer Aldayel et al. (+2 more)
The evaluation of cultural grounding context becomes complex when multiple cultures convey the same moral lesson. This challenge is particularly relevant to large
language models (LLMs), which produce narratives across a wide range of
languages and cultural contexts. However, it remains uncertain whether these models preserve culturally grounded meaning when equivalent moral lessons are conveyed through distinct cultural forms. This study introduces a
multilingual evaluation narrative framework that integrates a cross-linguistic collection of 414 proverbs spanning 15
languages and uses four LLMs to generate 13k narratives. By employing semantically equivalent proverbs as culturally grounded
prompts, the analysis assesses whether models preserve meaning across
languages, how cross-lingual conditioning influences narrative realization, and whether different model families converge on similar interpretations. Results indicate that cross-lingual prompting largely preserves proverb-level semantic meaning while systematically redistributing agency, social positioning, and narrative structure. Additionally, strong inter-model convergence is observed in both monolingual and cross-lingual settings, suggesting that
multilingual LLMs rely on shared semantic abstractions despite architectural and linguistic differences. These findings shed light on the need for more comprehensive evaluations of cultural grounding. Relying exclusively on semantic similarity in
multilingual narrative assessments may overestimate cultural preservation by neglecting culturally meaningful variations in narrative expression.
Read the full abstract ▾
All authors Jory Alshaalan, Haya Albaker, Abeer Aldayel, Aljawharah Alabdullatif, Rehab Alahmadi
Computation and Language
Qwen-AgentWorld: Language World Models for General Agents
Yuxin Zuo, Zikai Xiao, Li Sheng et al. (+30 more)
A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for
reasoning and planning. In this work, we investigate how world modeling based on
language models can further push the boundaries of general
agents. (i) We first focus on building foundation models for agentic environment simulation. We introduce Qwen-AgentWorld-35B-A3B and Qwen-AgentWorld-397B-A17B, the first
language world models capable of simulating agentic environments covering 7 domains via long chain-of-thought
reasoning. Leveraging more than 10M environment interaction trajectories of 7 domains in real-world environments, we develop Qwen-AgentWorld through a three-stage training pipeline: CPT injects general-purpose world modeling capabilities from the state transition dynamics and augmented professional corpora, SFT activates next-state-prediction
reasoning, and RL sharpens simulation fidelity through a tailored framework with hybrid rubric-and-rule rewards. To evaluate
language world models, we present AgentWorldBench, a comprehensive
benchmark constructed from real-world interactions of 5 frontier models on 9 established
benchmarks. Empirical results demonstrate that Qwen-AgentWorld significantly outperforms existing frontier models. (ii) Beyond foundation models, we further investigate two complementary paradigms through which world modeling enhances general
agents. First, as a decoupled environment simulator, Qwen-AgentWorld supports scalable and controllable simulation of thousands of real-world environments for agentic RL, yielding gains that surpass real-environment training alone. Second, as a unified
agent foundation model, world-model training acts as a highly effective warm-up that improves downstream performance across 7 agentic
benchmarks. Code: https://github.com/QwenLM/Qwen-AgentWorld
Read the full abstract ▾
All authors Yuxin Zuo, Zikai Xiao, Li Sheng, Fei Huang, Jianhong Tu, Yuxuan Liu, Tianyi Tang, Xiaomeng Hu, Yang Su, Qingfeng Lan, Yantao Liu, Qin Zhu, Yinger Zhang, Bowen Yu, Haiquan Zhao, Haiyang Xu, Jianxin Yang, Jiayang Cheng, Junyang Wang, Lianghao Deng, Mingfeng Xue, Tianyi Bai, Yang Fan, Yubo Ma, Yucheng Li, Zeyu Cui, Zhihai Wang, Zhihui Xie, Zhuorui Ye, An Yang, Dayiheng Liu, Jingren Zhou, Ning Ding
Computation and Language
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Federico Marcuzzi, Xuefei Ning, Roy Schwartz et al. (+1 more)
As Large
Language Models are increasingly deployed in critical applications, robustly evaluating their social biases is paramount. However, the current literature suffers from widespread methodological fragmentation, which yields contradictory conclusions. This stems largely from ignoring the structural framing of
benchmark-level evaluations. To resolve this, we introduce a unified and controllable framework that standardizes heterogeneous
benchmarks to systematically contrast isolated demographic assessments with forced-choice comparative settings. Crucially, this allows us to disentangle the confounding effects of Chain-of-Thought
reasoning, neutral fallback options, and other structural artifacts in social bias evaluations. Our evaluation across multiple model families reveals a massive, systematic paradigm gap: while isolated assessments limit prejudice activation, comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts. Alarmingly, CoT
reasoning exacerbates social biases under comparative settings, and this systemic bias persists as a deterministic prejudice even when models are provided neutral fallback options or claim to answer randomly. Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size. Ultimately, we offer a crucial methodological guideline: while researchers must leverage comparative settings to robustly audit hidden biases, practitioners cannot safely rely on comparative deployments in ambiguous real-world tasks.
Read the full abstract ▾
All authors Federico Marcuzzi, Xuefei Ning, Roy Schwartz, Iryna Gurevych
Computation and Language
MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recovery
Enze Ma, Yufan Zhou, Wei-Chieh Huang et al. (+7 more)
Long-term
memory promises LLM
agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that interaction forms. In practice, however, this
memory is evaluated mostly through downstream behavior, such as later answers, personalization quality, or task success, which tests that understanding only indirectly and leaves the
memory artifact itself largely unaudited. We argue that long-term
memory should instead be evaluated as an auditable post-interaction artifact: after ordinary assistance, what structured user state can be reconstructed from the
memory the
agent leaves behind? We instantiate this view in MEMPROBE, a
benchmark in which a
memory-equipped
agent assists simulated users, each carrying a hidden, taxonomy-anchored user-state bank, across a trajectory of leak-controlled tasks, after which that bank is reconstructed from the
agent's resulting
memory under both full-store and top-k access. Built on synthetic ground truth for efficient, scalable measurement, MEMPROBE spans 50 simulated users with 31 hidden dimensions each (1,550 recovery targets) and tests 5 representative
memory systems. Testing state-of-the-art
memory agents, we find that successful assistance and recoverable
memory behave as distinct capabilities. Task completion nearly saturates, even for a memoryless baseline, while category-balanced recovery stays moderate (about 0.6) and drops further under top-k
retrieval. MEMPROBE is the first
benchmark to study
memory recovery directly, reconstructing the user state a system retains and scoring it against ground truth. We see recovery as a concrete objective for future
memory agents to optimize, and MEMPROBE as a step toward an environment where
agents are trained to remember their users, growing more faithful the longer they know them.
Read the full abstract ▾
All authors Enze Ma, Yufan Zhou, Wei-Chieh Huang, Jie Yang, Huanhuan Ma, Zixuan Wang, Chengze Li, Chunyu Miao, Philip S. Yu, Zhen Wang
Computation and Language · Cross-listed from cs.AI
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
Khanak Khandelwal
Scaling
adversarial evaluation of large
language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real. We present AdversaBench, an end-to-end red-teaming pipeline that mutates seed
prompts with five structured operators, queries a target model, and confirms failures through a three-judge panel with a meta-judge tiebreaker. We report experiments on 45 seeds across three categories:
reasoning, instruction-following, and tool use. Every seed produced a confirmed failure. Four findings stand out. First, operator effectiveness varies sharply by category: inject_distractor scores 0.00 mean reward on instruction-following seeds but 0.80-0.83 on
reasoning and tool-use. Second, binary failure rate hides difficulty: instruction-following seeds required 2.4 attacker iterations on average versus 1.1 for other categories, a gap visible in survival curves. Third, pairwise judge agreement of 80-87% coexists with near-zero Cohen's kappa due to label skew; category-level disagreement rates are more informative. Fourth,
adversarial prompts generated against Llama 3.1 8B transfer zero-shot to Llama 3.3 70B, suggesting the mutations exploit general behavioral patterns rather than model-specific weaknesses. Code,
dataset, and analysis scripts are available at https://github.com/khanak0509/AdversaBench .
Read the full abstract ▾
Computation and Language
Cross-Lingual Exploration for Parametric Knowledge
Elisha Diskind, Itamar Trainin, Uri Shaham et al. (+3 more)
Parametric knowledge in Large
Language Models is not equally accessible across
languages. As a result, standard inference techniques often struggle to surface localized facts, leading to failures in cross-lingual knowledge transfer and consistency. In this work, we investigate techniques for accessing hidden factual knowledge by exploring cross-lingual prompting strategies. We identify four inherent dimensions of cross-lingual exploration that directly govern parametric knowledge
retrieval and evaluate them on
multilingual factual
benchmarks covering 17 typologically diverse
languages. Our results demonstrate that cross-lingual exploration significantly improves knowledge transfer and factual recall, representing a more efficient compute Pareto frontier than native-
language scaling. Furthermore, we observe corresponding improvements in cross-lingual consistency, exceeding what can be explained by accuracy gains alone. Overall, our work establishes
multilingual prompt exploration as a highly effective inference-time strategy for unlocking latent parametric knowledge.
Read the full abstract ▾
All authors Elisha Diskind, Itamar Trainin, Uri Shaham, Leshem Choshen, Idan Szpektor, Omri Abend
Computation and Language
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Yuru Wang, Lejun Cheng, Yuxin Zuo et al. (+14 more)
We introduce NatureBench, a cross-discipline
benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding
agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior
agent-on-research
benchmarks. Evaluating ten frontier
agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that
agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the
benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench
Read the full abstract ▾
All authors Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang
Computation and Language
AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning
Honglin Guo, Qi Zhang, Yu Zhang et al. (+7 more)
Large
language models are increasingly deployed as
agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded
reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing
benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a
benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M
tokens, far exceeding any model's context window, so
agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.
Read the full abstract ▾
All authors Honglin Guo, Qi Zhang, Yu Zhang, Weijie Li, Rui Zheng, Zhikai Lei, Qiyuan Peng, Zhiheng Xi, Tao Gui, Qi Zhang
Computation and Language
Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams
Arda Eren, Micheal Cheung, Youqian Zhang et al. (+2 more)
Scam phone calls exploit vulnerable communities worldwide, yet research on detection has focused almost exclusively on English and other high-resource
languages. In low-resource settings such as Turkish, detection is especially difficult, as annotated data is scarce and technological defenses remain limited. This research investigates how large
language models (LLMs) can support scam detection in Turkish by introducing the first public multi-modal
dataset of 100 aligned audio-transcript pairs of scam and benign conversations. We evaluate seven LLMs spanning three model families: Gemini 2.5 (Flash, Flash-Lite, Pro), GPT-4o, and Qwen (Max, Plus, Turbo), under three input conditions: raw audio, automatic
speech-to-text transcripts, and transcripts refined by a native speaker. Our results suggest that transcript-based inputs consistently outperform direct audio processing, while human-corrected and uncorrected transcripts perform comparably. By centering a low-resource
language and real world threat, this work highlights the urgent need for culturally and linguistically inclusive AI safety research and more robust multi-modal systems for fraud prevention.
Read the full abstract ▾
All authors Arda Eren, Micheal Cheung, Youqian Zhang, Grace Ngai, Eugene Yujun Fu
Computation and Language · Cross-listed from cs.AI
A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial
Haichao Chen, Songchi Zhou, Zhengyun Zhao et al. (+28 more)
Rare diseases affect millions of individuals worldwide, yet timely diagnosis remains a major public health challenge due to scarcity of specialized clinical expertise. While large
language models (LLMs) show promise to support rare disease diagnosis, current models are constrained by insufficient clinical deployability, limited clinically grounded evidence, and scarcity of training data. Here we present RaDaR (Rare Disease navigatoR), an open-source, compact
reasoning LLM (32B parameters) for rare disease diagnosis. RaDaR was trained with 49,170 publicly available free-text cases and 104,666 synthetic cases with
reasoning-enhanced training. RaDaR showed the strongest performance among evaluated open-source models, including the 671B DeepSeek-R1, across public
benchmarks and four external validation centers. In a retrospective cohort, RaDaR prioritized the final diagnosis before documented clinical suspicion in 61.06 percent of cases, corresponding to a potential lead time of 1.87 months and 50.18 percent of the within-center interval. In a randomized physician-assistance trial, RaDaR assistance improved physicians' rare-disease diagnostic accuracy by 21.44 percentage points compared with internet search alone. Synthetic-data ablations suggested that phenotype-anchored narratives provide useful training signal for long-tail rare diseases, with a monotonic scaling trend within the tested data range. Together, RaDaR and its development and validation framework provide a deployable rare-disease
reasoning model and a reproducible development framework for diagnostic AI under data scarcity.
Read the full abstract ▾
All authors Haichao Chen, Songchi Zhou, Zhengyun Zhao, Shikai Hu, Xianghong Jin, Hongwei Ji, Li He, Shuli Li, Yiming Qin, Xin Tan, Runfeng Shi, Yih Chung Tham, Jiaye Zhu, Ye Li, Ye Jin, Longhao Cao, Dawei Li, Honghan Wu, Hongqiu Gu, Guanqiao Li, Tudor Groza, Chunying Li, Dian Zeng, Weihong Yu, Gareth Baynam, Saumya Shekhar Jamuar, Min Shen, Shuyang Zhang, Bin Sheng, Sheng Yu, Tien Yin Wong
Computation and Language
UOL@IDEM at BEA 2026 Shared Task 1: Neural Fusion and Feature-Rich Modeling for L1-Aware Vocabulary Difficulty Prediction
Nouran Khallaf and Serge Sharoff
This paper describes UOL@IDEM's closed-track submission to the BEA 2026 shared task on L1-aware vocabulary difficulty prediction. We model the task as regression and train separate systems for Spanish, German, and Mandarin Chinese\footnote{Below we use Chinese for brevity.}. Our system combines
multilingual contextual representations with engineered features capturing frequency, surface form,
retrieval evidence, semantic
alignment, cognate similarity, and masked-
language-model predictability. Development results show consistent gains over the official closed-track baselines, with sentence-
embedding encoders such as BGE-M3,
multilingual E5, and LaBSE performing best. Official submissions achieve RMSE scores of 1.132, 1.037, and 0.891 for Spanish, German, and Chinese, respectively. Feature analysis identifies frequency as the most stable predictor, while contextual predictability, form similarity,
retrieval, and semantic features provide complementary L1-sensitive signals. Error analysis shows strong ranking performance but weaker calibration for the easiest items, which are often overpredicted. See https://github.com/Nouran-Khallaf/UoL-IDEM-BEA2026-Vocabulary-Difficulty-Prediction
Read the full abstract ▾
Computation and Language
The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African Languages in Frontier LLMs
Olaoye Anthony Somide
Commercial large
language models bill, scale latency, and budget context per
token. Yet tokenizers assign more subword
tokens to the same meaning in some
languages than in others, so speakers of
languages with high
token-fertility pay a structural penalty before a model is ever invoked. This penalty is documented for
multilingual settings in general, but it has not been measured systematically for African
languages at the level of enterprise deployment economics and cognitive context capacity. We measure it across 20 African
languages spanning five
language families and three scripts (Latin, Ge'ez/Ethiopic, N'Ko; 19 appear in the primary FLORES-200+ corpus, with Nigerian Pidgin measured via MAFAND-MT only), using parallel corpora so that the
language effect is isolated from content. Across 11 frontier and open tokenizers on FLORES-200+, every African
language carries a tokenization premium above English (median 1.88x on GPT-5 / o200k_base, up to 8.92x for N'Ko); the penalty is largest for Ethiopic and N'Ko scripts (reaching 7-9x) and is near-invariant across corpora (FLORES vs SIB-200 Pearson r = 0.9998). Translated into deployment terms, this results in up to 8.9x inference cost and an equivalent generation-latency multiplier (N'Ko vs English on GPT-5; 7.4x for Amharic), and as little as 11% of English's effective context window. The best currently available tokenizer for African
languages, Gemma 4, reduces the mean premium from 3.31x (cl100k_base) to 2.38x, but no tokenizer eliminates the penalty. We release an open measurement tool (afri-fertility), a public leaderboard, a results
dataset, and mitigation guidance for African builders. The penalty falls hardest on the
languages whose speakers can least afford it, a digital divide encoded directly into the subword vocabulary.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
An LLM-based Two-Stage Transformer Framework for Cross-Domain Bearing Fault Diagnosis with Limited Data
Jinghan Wang, Feng Cheng, Wentao Wu et al. (+3 more)
Bearing fault diagnosis faces critical challenges when
dataset heterogeneity, operating condition variations, and limited labeled data occur simultaneously in industrial environments. Existing approaches address these issues in isolation and rely on implicit feature
alignment, limiting effectiveness under concurrent challenges. This paper proposes a knowledge-guided two-stage transfer learning framework that employs a lightweight GPT-2-style
Transformer with causal self-
attention for hierarchical feature extraction from vibration signals, establishing explicit pathways where pre-trained encoder weights and fault prototype
embeddings serve as knowledge carriers from multi-source pre-training to target adaptation. The framework addresses the dual-shift challenge through multi-source learning for generalizable representations, prototype-based knowledge modulation for target adaptation, and taxonomy-adaptive classification for seamless transfer across heterogeneous fault categories. Experimental validation on four real-world
datasets demonstrates 92.61% average accuracy with only 10% labeled target data, outperforming state-of-the-art methods by 17.24 percentage points, establishing a practical pathway toward cost-effective predictive maintenance in Industry 4.0 applications.
Read the full abstract ▾
All authors Jinghan Wang, Feng Cheng, Wentao Wu, Hang Li, Gaoliang Peng, Tianchen Liu
Computation and Language · Cross-listed from cs.AI
Theodore Papamarkou, Vladislav Smirnov, Viktor Mazanov et al. (+4 more)
Modern coding
agents pair LLM generators with various tools, including cheap diagnostics and expensive verifiers. The tool-use decisions are typically governed by orchestrators that often use fixed rules and ignore uncertainty. We formulate orchestration as cost-sensitive sequential hypothesis testing: a
Bayesian controller maintains a belief over candidate correctness and dynamically decides whether to gather more evidence, refine the candidate, verify it, or stop. Across six generators and nine coding
benchmarks,
Bayesian control proves to be most valuable when verification is costly and critics are informative but imperfect. Beyond control, the belief state yields an interpretable correctness score that outperforms
token-probability and raw tool-success baselines for uncertainty quantification.
Read the full abstract ▾
All authors Theodore Papamarkou, Vladislav Smirnov, Viktor Mazanov, Artem Vazhentsev, Preslav Nakov, Timothy Baldwin, Artem Shelmanov
Computation and Language
Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learning
Shiding Zhu, Yudi Qi, Yajie Wang et al. (+6 more)
Experience-driven self-
evolution is critical for large
language model (LLM)
agents to improve through open-world interaction. However, existing experience learning methods mostly rely on single-
agent loops, where the same
agent executes tasks, summarizes outcomes, and determines
memory content. This setup makes
agents vulnerable to the Self-Confirmation Trap: wrong-but-self-consistent trajectories are misidentified as successful experience, leading to cumulative errors during
retrieval and reuse. To address this issue, we propose EDV, an Execute-Distill-Verify framework for reliable experience learning. In the Execute stage, multiple heterogeneous
agents explore the same task space in parallel to generate diverse candidate trajectories. In the Distill stage, a dedicated third-party
agent comparatively analyzes these trajectories to produce candidate experiences, reducing executor-centric summarization bias. In the Verify stage, the execution group validates candidates via a consensus mechanism, and only approved experiences are written into shared or private
memory. By decoupling the three stages, EDV transforms experience learning from isolated self-reflection into collaborative construction, filtering erroneous and noisy content before
memory insertion. We evaluate EDV on three challenging long-horizon
benchmarks: tau2-bench, Mind2Web and MMTB. Results show EDV consistently outperforms strong baselines, validating that reliable experience construction is essential for robust
agent self-
evolution. Our code is available at https://github.com/shidingz/EDV.
Read the full abstract ▾
All authors Shiding Zhu, Yudi Qi, Yajie Wang, Jiaze Li, Chao Song, Yaorui Shi, Yibo Miao, Hanqi Gao, Kai Zhang
Computation and Language
Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction
Nitesh Kumar
In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review.
Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
Arnaud Ricci
We introduce Age of LLM, a turn-based 1v1
benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base. Three stressors are deliberate: fog of war, full diplomacy (messages, ceasefires, ultimatums; uranium kept secret), and a reliability dimension where every turn must follow a strict JSON schema and an illegal action is silently discarded. The engine is private and each match uses a fresh random map seed and opponent, mitigating the data contamination that affects public
benchmarks. Models receive a (near) rule-only
prompt with no build-order advice (two tactical seed phrases were present during data collection; see Section 2.7). We
benchmark 15
reasoning models across 54 matches and 5,258 actions. Findings: (1) the
nuclear rush dominates (78% on the rules-coherent v0.11+ sub-corpus; 85% corpus-wide) with a sole-launcher signature that is largely mechanical under secret-simultaneous launch rules, not a cognitive deterrence failure; (2) military conquest is rare but faster (12.3 vs 18.9 turns); (3) diplomacy is prolific yet almost never consummated; (4) ~58% of illegal actions are fog/state errors, making the illegal-action rate a measure of belief-tracking; (5) – the least established, and the only one we label exploratory – a weak link associates reliability with winning. The corpus is small, unbalanced and not side-swapped, so the ranking is a preliminary descriptive view, not a contribution. Beyond ranking, the turn-by-turn traces of actions and messages make the corpus a lens on how LLMs reason under
adversarial uncertainty – their belief-tracking, spontaneous deception, and per-model cognitive "personas" – which we frame as a future research direction. We release the replay format, an isometric viewer and all replays; engine source on request.
Read the full abstract ▾
Computation and Language
AutoSpecNER: A Fine-Grained Named Entity Recognition Dataset for Vehicle Specification Extraction
Jordan Lee, Filippos Ventirozos, Abdirahman Abdullahm et al. (+3 more)
Vehicle advertisements contain rich specification information, but automotive NER resources remain limited. We introduce AutoSpecNER, an expert-annotated
dataset for fine-grained entity recognition in vehicle listings. The
dataset includes 659 advertisements from a popular car-selling website, with over 10,000 entities annotated across 15 categories, including MODEL, ENGINE_SPEC, and BATTERY_CAPACITY. Annotation quality was validated through inter-annotator agreement, achieving an average score of 91.5%. We
benchmark rule-based extraction, fine-tuned
transformer encoders, and large
language models. DeBERTa achieves the best performance with a 90% micro-F1 score, outperforming the rule-based baseline (43%) and the strongest large
language model (77.8%).
Read the full abstract ▾
All authors Jordan Lee, Filippos Ventirozos, Abdirahman Abdullahm, Ioanna Nteka, Peter Appleby, Matthew Shardlow
Computation and Language
On the Stability of Prompt Ranking in Large Language Model Evaluation
Shaoshuai Du, Penghao Liang, Yixian Shen et al. (+3 more)
Prompt-based interaction has become a dominant paradigm for using large
language models (LLMs), where multiple candidate
prompts are evaluated and the top-ranked one is selected for downstream use. This workflow implicitly assumes that
prompt rankings are stable under minor variations in evaluation conditions. In this paper, we systematically study
prompt ranking stability under common sources of variability, including random seeds and limited evaluation subsets. Across three open-weight LLMs and two
benchmark tasks, we find that while overall rank correlations are often moderate to high, the identity of the top-performing
prompt frequently changes, leading to unreliable selection decisions. To address this issue, we propose a simple stability-aware selection strategy based on a lower confidence bound, which accounts for both performance and variance. Our results show that this approach improves robustness in unstable settings while remaining competitive in more stable regimes. These findings highlight the importance of accounting for evaluation uncertainty in
prompt selection and LLM benchmarking.
Read the full abstract ▾
All authors Shaoshuai Du, Penghao Liang, Yixian Shen, Chuanqi Shi, Hang Zhang, Lun Wang
Computation and Language · Cross-listed from cs.CR
ComputeFHE: A Privacy-Preserving General-Purpose Computation Library
Faris Serdar Tasel and Efe Ciftci
Fully Homomorphic Encryption (FHE) enables computations to be performed directly on encrypted data while preserving data confidentiality. However, its practical applications remain limited by high computational costs and development complexity. This paper presents ComputeFHE, an open-source C++ library that facilitates the development of
privacy-preserving applications based on the TFHE cryptosystem. The library provides encrypted integer and fixed-point data types together with arithmetic, logical, comparison, conditional, and oblivious array-access operations which allow developers to implement
algorithms using a familiar imperative programming paradigm. ComputeFHE supports both conventional TFHE arithmetic based on standard two-input logic gates and an optimized Arithmetic Logic Unit (ALU) architecture utilizing FHE-friendly logic primitives. Experimental results demonstrate significant reductions in the number of required bootstrapping operations, achieving performance improvements of up to 3.9x for selected operations. In addition, the library includes a simulation mode that enables testing, debugging, and complexity analysis without performing actual cryptographic computations while providing
circuit complexity and bootstrapping costs. Built on top of OpenFHE, ComputeFHE offers a practical and accessible framework for developing and evaluating
privacy-preserving
algorithms and applications.
Read the full abstract ▾
Computation and Language
MorfFlex: Handling Rich Morphology
Jaroslava Hlaváčová, Marie Mikulová, Barbora Štěpánková et al. (+2 more)
We present MorfFlex, a morphological dictionary architecture suitable for
languages with extensive regularity in both inflection and derivation. As the primary example of MorfFlex in use we introduce MorfFlex CZ, a morphological dictionary of Czech. It is distributed as a simple, unstructured list of <wordform, lemma, tag> triplets, however, its manually maintained, unpublished source files and conversion scripts encode a sophisticated system of inflectional and derivational patterns. These patterns dramatically reduce the otherwise enormous size of the dictionary, which currently contains over 100 million wordforms and more than 1 million lemmas. The MorfFlex CZ dictionary serves as an essential resource for ensuring the consistency of manual morphological annotation in the Prague Dependency Treebanks and underpins state-of-the-art automatic tools such as MorphoDiTa. In this paper, we focus on: (i) presenting an effective method for managing the rich morphological system within the dictionary, and (ii) demonstrating the utility of such a
language resource for maintaining annotation consistency in corpora and supporting the development of advanced NLP applications.
Read the full abstract ▾
All authors Jaroslava Hlaváčová, Marie Mikulová, Barbora Štěpánková, Milan Straka, Jan Hajič
Computation and Language
Automatic Part-of-Speech Tagging of Arabic-English Dictionary Senses through WordNet
Diaa M. Fayed, Aly A. Fahmy, Mohsen A. Rashwan et al. (+1 more)
This paper proposed an
algorithm for part-of-
speech (POS) tagging senses of a bilingual dictionary. The
algorithm is applied on the Al-Mawrid Arabic-English dictionary. The tagging task is accomplished by transferring the POS tags of the English translation equivalences (TEs) to the dictionary senses after dis-ambiguities process. The English POS tags of senses are acquired from the Princeton WordNet. POS tagging of bilingual dictionary senses is prerequisite to link a bilingual dictionary to WordNet and/or standardizing that dictionary into WordNet-LMF format where the synset (set of synonyms), not word, is the basic brick. The registered accuracy is high though the cost is little. Building NLP/HLT tools needs linguistic experts, large investments, and long time. For statistical approach, we need large annotated corpora and for rule-based approach, we need large lexicon that contains rich linguistic and world knowledge. That motivates the appearance of what are called resource-light approaches to develop natural
language processing (NLP) tools for poor-resource
languages.
Read the full abstract ▾
All authors Diaa M. Fayed, Aly A. Fahmy, Mohsen A. Rashwan, Wafaa K. Fayed
Computation and Language · Cross-listed from cs.IR
PETRA: Transforming Web Text for Petroleum-Engineering Domain Adaptation
Kirill Dubovikov, Omar El Mansouri, Hachem Madmoun et al. (+12 more)
Petroleum-engineering search exposes a supervision gap for strong general retrievers: relevant evidence exists in public web text, but domain relevance labels are scarce. To address this gap, we propose PETRA, a large-scale Petroleum Engineering Text for
Retrieval Adaptation
dataset and pipeline that converts noisy public web data into a curated domain corpus and synthetic supervision for dense
retrieval and reranking. PETRA contains 1.36M curated chunks, approximately 2B
token equivalents, $\approx$859k,
embedding training rows from $\approx$224k anchors, and roughly 400k teacher-scored reranker candidate rows. Its construction combines high-recall energy-domain curation, an energy-domain classifier with 98.4% test accuracy, chunk-grounded query generation, LLM-written hard negatives, and
retrieval-mined candidate lists. PETRA improves first-stage in-domain Normalized Discounted Cumulative Gain (nDCG) from 0.703 to 0.763 through score fusion. Reranker adaptation improves the public Earth Science
benchmark by 44% relative and a six-task
reasoning-intensive panel by 23%. Failed training recipes show that high train-holdout accuracy on synthetic labels does not predict
retrieval gains;
retrieval-mined data helps only after being repackaged as teacher-scored candidate lists sampled from the inference-time candidate distribution.
Read the full abstract ▾
All authors Kirill Dubovikov, Omar El Mansouri, Hachem Madmoun, Yanda Li, Sandeep Kumar, Aya El Mir, Supriyo Ghosh, Writabrata Bhattacharya, Adrian Garcia-Garcia, Onkar Pandit, Sunil Kumar Sahu, Federico Castanedo, Larry Murray, Martin Takac, Salem Lahlou
Computation and Language
Meet UD_Czech-PDTC: A Large and Genre-Rich Treebank in Universal Dependencies
Marie Mikulová, Barbora Štěpánková, Daniel Zeman et al. (+3 more)
Czech has been part of Universal Dependencies since its first release in 2015. It has also been one of the best represented
languages, with the Prague Dependency Treebank being order of magnitude larger than most other UD treebanks. More recently, three other
datasets from the Prague family were added and the annotations thoroughly revisited, forming the "Prague Dependency Treebank-Consolidated" (PDT-C). In comparison to the original PDT, PDT-C is more than twice as large, but it is also much more diverse in terms of genres and domains. In this paper, we describe the conversion of the new resource to Universal Dependencies. While the two annotation schemes are relatively similar at the first sight, there are numerous small differences in
topology of the dependency structures and in granularity of the POS and relation type inventories. We demonstrate a selection of such differences on examples, discuss the diverging motivations, as well as ways to overcome the differences during conversion. We argue that while PDT is less "universal" and more tightly bound to one
language, its multi-layer annotation is rich and provides all information needed for basic UD trees, and much more.
Read the full abstract ▾
All authors Marie Mikulová, Barbora Štěpánková, Daniel Zeman, Jan Štěpánek, Milan Straka, Jan Hajič
Computation and Language
Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
Guruprakash J and Krithika L.B
Transformer-based
language models have become the default substrate for natural
language processing and the pace of new releases has made it hard for practitioners to separate durable ideas from the noise of incremental announcements. This review works at two levels. At the level of mechanism, we organise the main
transformer families into a working taxonomy, covering encoder-only, decoder-only, encoder-decoder, long-context, permutation-based, and generator-discriminator variants. We then extend the discussion to post-2023 developments that changed the picture in practice: instruction tuning, reinforcement learning from human feedback, direct preference optimisation, mixture-of-experts scaling,
retrieval augmentation and the current flagship model families from OpenAI, Anthropic, Google, Meta, Mistral and DeepSeek. At the level of use, we survey deployments across healthcare, finance, legal, education, customer service, creative writing and scientific work. Based on this we link each to the specific capabilities that make a
transformer the appropriate tool. The contribution of this paper is a critical assessment that is based on the survey. We compare architectures on four axes that matter to deployment decisions, we quantify the trade-off between parameter count and energy cost. We also discuss how
alignment methods, data provenance and
benchmark saturation change what it means to call a model "state of the art". The final section lists the research questions that we think deserve more
attention.
Read the full abstract ▾
Computation and Language
Prague Dependency Treebank – Consolidated 2.0: Enriching a Complex Annotation Scheme
Marie Mikulová, Jiří Mírovský, Milan Straka et al. (+4 more)
The Prague Dependency Treebank framework is unique in its attempt to systematically include and link different layers of
language, including a meaning representation with several types of inter-sentential phenomena, especially coreference and discourse relations. We present its second consolidated version (PDT-C 2.0), which concludes almost 30-years long project of sustained development of the resource to a uniformly and coherently annotated, genre-diversified, almost 4 million
token language resource of Czech
language, with accompanying fully compatible lexicons. In addition to continuous linguistic research, the richly linguistically annotated corpus is also widely used in international comparisons of the development of traditional and novel NLP tools as well as in conversions into other formalisms. The corpus and the trained parsers are available under the CC BY-NC-SA licence.
Read the full abstract ▾
All authors Marie Mikulová, Jiří Mírovský, Milan Straka, Pavlína Synková, Jan Štěpánek, Barbora Štěpánková, Jan Hajič