Computation and Language · Cross-listed from cs.AI
CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
Sil Hamilton, Albert Yu Sun, Oscar J. Romero et al. (+4 more)
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic
datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A
benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication
reasoning, filling a crucial gap in the benchmarking ecosystem.
Read the full abstract ▾
All authors Sil Hamilton, Albert Yu Sun, Oscar J. Romero, Carl-Leander Henneking, David Mimno, Bishan Yang, Igor Labutov
Computation and Language
D2C-Routing: Dimension-to-Composition Evidence Routing for Mixed-Origin AI-Generated Text Detection
Xin Chen, Fuwei Zhang, Yiqi Tong et al. (+3 more)
AI-generated text detection is commonly framed as a binary document-level judgment about whether a text is human-written or machine-generated. This framing breaks down for mixed-origin writing, where content origin and expression origin may differ. We cast mixed-origin detection as dimension-to-composition source attribution, inferring content origin and expression origin before composing them into four collaboration types. We propose Dimension-to-Composition Routing (D2C-Routing), which routes content-side and expression-side evidence to supervised dimension heads before a learned gated composition layer predicts the final label. On MixD2C, a reconstructed split derived from the HART mixed-origin
benchmark, our disclosed D2C-Routing-based detector system reaches 0.8603 four-way Avg TPR@1%FPR, 6.5 points above the same-split RACE-local rerun. Core ablations support the routing design, while error analysis shows that distinguishing AI-content/human-expression from fully AI-generated text remains the hardest boundary. Code is available at https://github.com/bystander563/d2c-routing-artifact.
Read the full abstract ▾
All authors Xin Chen, Fuwei Zhang, Yiqi Tong, Wei Guo, Yutian Xiao, Fuzhen Zhuang
Computation and Language
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Kairong Luo, Jiarui Cui, Yaorui Yin et al. (+8 more)
Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion
tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in
token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.
Read the full abstract ▾
All authors Kairong Luo, Jiarui Cui, Yaorui Yin, Shengqi Chen, Yiming Yang, Linxiang Gao, Yanmohan Wang, Chengxia Li, Mingzhe Zhang, Kaifeng Lyu, Wenguang Chen
Computation and Language
Your Voice Cloning System is Secretly a Voice Anonymizer
Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak et al. (+1 more)
Speaker anonymization suppresses speaker-identifying attributes from
speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a
multilingual voice cloning model trained on 27k hours of
speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances
privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European
languages across CommonVoice and
Multilingual LibriSpeech, our system achieves near-optimal
privacy (EER $\approx$ 0.49), competitive intelligibility, and substantially better
speech quality than dedicated anonymization baselines, while requiring no
language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.
Read the full abstract ▾
All authors Romolo Muletta, Felix Matthias Saaro, Mark Cieliebak, Jan Deriu
Computation and Language
RCMN: Understanding Misleadingness in Influential Public Discourse
Peiling Yi
Influential public discourse shapes public beliefs and can also mislead, not only through what is stated, but also through how information is framed, omitted, contextualised, and communicated. Yet less research has focused on how such misleadingness arises and shapes the interpretations formed by readers. To address this gap, we introduce Reader-Centric Misleadingness Understanding (RCMN), a framework that operationalises misleadingness through five dimensions: misleading mechanism, likely reader interpretation, evidence-warranted interpretation, emotional arousal, and communicative intent. Based on this framework, we construct an evidence-grounded
dataset of influential public discourse. Empirical findings show that misleadingness is diverse and extends well beyond fabrication, with unsupported inference, exaggeration, and omission among the prevalent mechanisms, and is frequently associated with heightened emotional arousal and distortive communicative intent. Moreover, we investigate whether lightweight claim-and-context representations retain sufficient cues for understanding reader-centric misleadingness without access to richer contextual, evidential, and
multimodal information. Evaluation across five recent generative foundation models shows that reader-level interpretations can often be recovered from such limited representations, whereas identifying how misleadingness is produced remains considerably more challenging. These findings highlight the potential of lightweight representations for scalable misleadingness analysis, while reliable understanding of misleading mechanisms continues to require richer contextual and evidential grounding.
Read the full abstract ▾
Computation and Language
INTENT-AS-A-TOOL Makes it Easy to Track Agentic Misalignment
Yutong Zhang, Jianshuo Dong, Peng Xu et al. (+5 more)
As large
language models (LLMs) are deployed as autonomous
agents, safety failures increasingly involve consequential actions. We study agentic misalignment, where
agents take harmful actions under goal conflicts and pressures. Using chain-of-thought (CoT) monitoring, we find that harmful execution is often preceded by intent signals in
reasoning. However, post-hoc CoT labels are too coarse to show how intent changes during generation. We introduce INTENT-AS-A-TOOL, an approach that adds intent-targeted tools to give the model a dedicated channel for expressing commitment to a target behavior. The probability of calling an intent tool provides a judge-free, fine-grained signal of the model's tendency to pursue that behavior. Our results show that INTENT-AS-A-TOOL complements CoT monitoring, expands post-hoc CoT labels into dense trajectories, and identifies critical steps for online intervention. These findings suggest that action preferences are useful for tracking agentic misalignment during
reasoning. Our code and data are accessible: https://github.com/RebeccaZhang22/intent-as-a-tool.
Read the full abstract ▾
All authors Yutong Zhang, Jianshuo Dong, Peng Xu, Long Wang, Jie Zhang, Tianwei Zhang, Xiaoping Zhang, Han Qiu
Computation and Language
Pair-Level Essay-Scale Republication and Reuse from Fragmented Historical Text Reuse: A Workflow Study on Eighteenth-Century Books and Newspapers
Ke Shu, Kira Hinderks, Eetu Mäkelä et al. (+1 more)
This paper addresses the recovery of essay-scale republication and reuse from fragmented text-reuse evidence, a setting whose central challenge is pair-level evidence consolidation and not fragment
retrieval alone. The study focuses on a candidate set centered on essays by eighteenth-century Scottish philosopher David Hume, spanning books from ECCO (Eighteenth Century Collections Online) and historical newspapers. Because the input consists of fragmented reuse hits instead of clean document pairs, and positive coverage is inherently incomplete, we formulate the task as pair-level evidence consolidation into plausible transmission relations and compare three methodological families: a staged rule-based workflow, baselines (a decision tree and two direct LLM settings), and automated rule adaptation. On labeled ECCO–ECCO slices, pair-level feature aggregation alone already reaches 0.948 F1 on the main labeled slice, while the final workflow gives the strongest overall precision-recall trade-off among the tested rule stages. On the full ECCO–ECCO candidate universe, direct LLM baselines flag up to 14,886 pairs as reprints compared to 771 for the final workflow, behaving in this direct-
prompt setup as high-recall candidate expanders rather than precision-controlled deployment classifiers. On ECCO–Newspaper, manual audit confirms all 176 predicted positives as genuine cases of republication or reuse, while issue duplication and source-side multiplicity reveal additional provenance structure. Under incomplete ground truth, auditable pair-level evidence consolidation provides a practical way to produce compact candidate spaces for historical inspection.
Read the full abstract ▾
All authors Ke Shu, Kira Hinderks, Eetu Mäkelä, Mikko Tolonen
Computation and Language · Cross-listed from cs.LG
Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting
Xinwei Qiang, Xiang Fang, Chang Chen et al. (+3 more)
Block drafters propose several
tokens in one forward pass, before earlier target
tokens are realised. Their rejection mixes two losses: missing within-block path information and imperfect modelling of observable information. Accepted length cannot distinguish them. We separate the two with an information floor, the minimum expected rejection at a specified conditioning order; rejection above this floor is the model gap. Measurements across four domains and four open-weight targets, plus floor estimates on a frontier API target, yield three findings. First, the all-parallel floor reaches $0.286$ at the final slot on Qwen3-4B, limiting even the best proposal to $71\%$ per-slot acceptance. Second, one realised
token removes $86$–$100\%$ of this floor, a locality also recovered by an independent mutual-information analysis. Third, current drafters remain far above their floors: the final-slot model gap accounts for $43$–$64\%$ of DFlash rejection and $85$–$92\%$ of DSpark's oracle-conditioned rejection. Guided by this separation of short-range conditioning from proposal quality, we replace a context-independent predecessor correction with a prefix-
attention head that reads committed context conditional on the predecessor. With supporting backbone components, the resulting Qwen3-4B drafter improves mean serving accepted length by $3.93\%$ over the released DSpark checkpoint across nine tasks, without increasing the conditioning order. Code and data: https://github.com/tie-pilot-qxw/specfloor; drafter checkpoints: https://huggingface.co/TIE-Pilot/dspark-attnconv-block7-qwen3-4b.
Read the full abstract ▾
All authors Xinwei Qiang, Xiang Fang, Chang Chen, Zaifeng Pan, Yue Guan, Yufei Ding
Computation and Language
BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks
Jeong-Yoon Kim and Tae-Yoon Lee
Operational facilities accumulate extensive local telemetry, but
agent-ready interactive evaluations with source-verifiable answers are costly to construct from such archives. Site-specific measurement histories also motivate a local conversion workflow that can operate where raw operational data must remain under organizational control. We introduce BTS-AgentBench, a deterministic compiler that transforms read-only telemetry and equipment metadata into tool-backed, multi-turn
agent tasks, phase-level targets, evidence and executable evaluation contracts. The BTS release includes 532 episodes across nine families; a second-corpus port produces 204 XAI4HEAT episodes. Frozen-input replay reproduces released artifacts byte-for-byte, and independent raw-source checks cover all 89 BTS and 41 XAI4HEAT test episodes. Replayable provenance links 1,170 trial identities to recorded execution and scorer conditions, while 1,125 completed traces retain model responses and tool results. An obligation-level comparison of 216 scored failures found that linked records exposed additional recorded failed obligations in 202 cases (93.5%), including tool arguments, stream identity, state carryover and missing interaction steps. The pipeline thereby supports source-grounded task generation, exact reconstruction and auditable diagnosis of
agent executions. Code, released
benchmarks, and reproduction instructions are available at https://anonymous.4open.science/r/telemetry-
benchmark-reproduction-3D68
Read the full abstract ▾
Computation and Language
Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Shuyi Fan, Boyuan Deng, Mengyu Xu et al. (+3 more)
LLM-judge audits assess bias by comparing ratings across matched conditions. Difference-in-differences designs compare two candidate responses within each item and then compare that contrast across a manipulated attribute, with an endpoint read off a bounded rating scale. Building on prior work on nonlinear difference-in-differences, we show in closed form that when ratings are bounded, a severity shift common to both responses produces an observed interaction whenever the scale bounds attenuate it unequally. We examine this problem in a pre-registered audit comprising 990 calls to a frozen pedagogy judge. The registered primary analysis did not detect an effect of a stated novice or advanced profile on scaffolding preference. The only nominally significant secondary interaction concerned the productive-struggle sub-score ($+0.378$ points; $p = 0.002$). A post-hoc construction with zero differential preference reproduced 79 to 85% of this interaction using the observed severity shift and the scale floor alone. The observed interaction therefore does not identify differential preference on the latent scale.
Read the full abstract ▾
All authors Shuyi Fan, Boyuan Deng, Mengyu Xu, Xinhong Xie, Chenyang Li, Hongyang Zhang
Computation and Language · Cross-listed from cs.AI
BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models
Jinghan Zhang, Fengran Mo, Zhiyu Chen et al. (+3 more)
Although Large
language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a
benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five
datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the
benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research.
Read the full abstract ▾
All authors Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han, Kunpeng Liu, Chang-Tien Lu
Computation and Language · Cross-listed from cs.AI
Naive Prompt Optimization: Rethinking the Need for Complex Prompt Search
Yuan Chang and Xiaoqi Chen
Efficiently improving autonomous
agents across diverse tasks is central to accelerating recursive self-improvement (RSI) in agentic AI, with
prompt optimization emerging as a promising approach capable of delivering performance gains comparable to those achieved by fine-tuning model weights, while reducing computational costs in both optimization and serving. However, recent developments increasingly favor unnecessarily complex
prompt optimizers. We introduce Naive
Prompt Optimization (NPO), a lightweight single-lineage method that iteratively revises
prompts using a teacher model with rollout feedback. NPO achieves comparable or better performance than GEPA with fewer rollouts, and its advantage increases with stronger teacher models, suggesting that stronger teacher
reasoning can partially substitute for optimizer-side search complexity. In interactive games, NPO remains broadly competitive with GEPA, while GRPO performs better on some tasks less amenable to
prompt optimization. We also show that NPO-optimized
prompts elicit similar performance improvements when applied verbatim to other student models, especially across models within the same family. Overall, our preliminary results show that simple, linear
prompt optimization can rival substantially more sophisticated and complex search procedures.
Read the full abstract ▾
Computation and Language
SCIT: Testing Causal Cache Carriers in Latent Chain-of-Thought Models
Yi Ding, Lijun Huang and Menglin Yang
Latent chain-of-thought models move intermediate
reasoning from emitted text into continuous states, improving compactness but hiding the causal object. We introduce SCIT, the Suffix Cache Interchange Test, a causal protocol that constructs exact source-recipient counterfactuals, patches declared cache segments, and identifies which
transformer object carries the counterfactual computation. SCIT combines sufficiency tests with K/V component splits, hidden-state controls, semantic source controls, decoded validation, and matched corruption. On CODI-GPT2 and a Sim-CoT-style GPT-2 reproduction, counterfactual arithmetic transfers primarily through value-cache suffix trajectories rather than hidden states, keys, reusable answer slots, or single-
token triggers. Complete sufficiency-and-necessity evidence for the late-value-suffix mechanism holds for the main CODI-GPT2 checkpoint; the Sim-CoT-style checkpoint shows the same sufficiency and decoded-control pattern but insufficient matched-corruption evidence for a necessity call. Beyond these local arithmetic
cells, SCIT reveals carrier-regime shifts: arithmetic-like GPT-2/1B
cells preserve latent-tail value/KV transfer, whereas competent 8B and repaired non-arithmetic
cells route through
prompt-prefix or full-cache K/V; boundary
cells receive no mechanism call. SCIT therefore contributes a cache-level diagnostic, a checkpoint-specific GPT-2 arithmetic mechanism, and a competence-gated carrier map rather than a universal latent-tail claim.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Xingshan Zeng, Zishan Xu, Boju Zhang et al. (+11 more)
LLM
agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many
agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object $(E,q,\tau,v)$, comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or
dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as
agents and environments evolve.
Read the full abstract ▾
All authors Xingshan Zeng, Zishan Xu, Boju Zhang, Yuzhou Wu, Lingzhi Wang, Jianghao Lin, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Weinan Zhang, Yong Yu, Qun Liu, Weiwen Liu
Computation and Language
BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing
Yu Yvonne Wu, Arvind Pillai, Yuliang Chen et al. (+7 more)
Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sparse snapshots of wellbeing. Wearable devices offer longitudinal behavioral and physiological signals for continuous, low-burden monitoring. Recent LLM-driven personal-health
agents enable natural
language queries over wearable signals, but mainly handle short-term,
retrieval-based lookups (e.g., highest step count over a week). They do not evaluate whether
agents can reason over long-term signals to predict wellbeing scores paired with evidence-grounded rationales. To address this gap, we introduce BALMS, the first systematic
benchmark of LLM-based agentic systems for longitudinal mental health sensing. BALMS spans 3 real-world longitudinal
datasets, 2 task families (closed-form wellbeing-score prediction and rationale generation auto-graded by an LLM-as-Judge), 3 agentic paradigms evaluated across 5 open- and closed-source LLM backbones. We find that zero-shot
agents rarely outperform a simple mean baseline, except with stronger backbones or compact, semantically meaningful features. Chain-of-thought prompting improves
reasoning-oriented backbones, but does not guarantee temporal grounding or numerical correctness. Together with more analysis on efficiency and temporal scaling, BALMS highlights the need for longitudinal mental health
agents that selectively retrieve history, ground temporal evidence, and reason over interpretable behavioral features.
Read the full abstract ▾
All authors Yu Yvonne Wu, Arvind Pillai, Yuliang Chen, Yuwei Zhang, Sudarshan Regmi, Tess Z. Griffin, Michael V. Heinz, Lisa A. Marsch, Nicholas C. Jacobson, Andrew Campbell
Computation and Language
When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue
Yen-Ju Lu, Yuzhe Wang, Yaohan Guan et al. (+8 more)
Understanding spoken dialogue requires joint
reasoning over lexical content and paralinguistic acoustic signals such as
emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in
speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and
speech-grounded interpretations agree, enabling evaluation beyond
adversarial audio dependence. This results in ContraTalk, a controlled
benchmark containing 501 questions across five discourse dimensions: interaction behavior,
emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style
reasoning framework that converts
speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the
reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving
speech-grounded
reasoning.
Read the full abstract ▾
All authors Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai, Mingrui Liang, Kaavya Chaparala, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba
Computation and Language · Cross-listed from cs.AI
Calibrated Enough to Know, Not Calibrated to Act: Fabricated Evidence Makes LLM Agents Commit to the Unknowable
Pranav Aggarwal
An LLM
agent shown a professional-looking
market panel commits to a directional call on a provably unpredictable question far more often than one asked the bare question: across 12 frontier models, commitment rises from 6.5% to 54.0% as evidence is escalated. It commits just as readily when every number on the panel is invented: fabricating the entire display, so nothing the model can see is true except the question itself, still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine
market data. What unlocks confident action is not information but the authority of its packaging. The failure is narrow and locatable. Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy. Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline. Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those. The act/don't-act gate is what fails, and the effect is concentrated in a few models rather than universal. Because the gate is separable, it can be trained. Supervised fine-tuning of a 3B model on 540 synthetic cases, predominantly dice, coins, jars and timers, drives commitment to 0.0% on the original cases and transfers to three unseen domains. It does not survive everything: the gate holds exactly when the response format leaves room to reason, and rigid formats that remove that room leave the model confident and wrong on questions it otherwise answers correctly. The gate is trainable and context-fragile, and deployment needs both halves of that sentence.
Read the full abstract ▾
Computation and Language
Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models
Himal Badu
Autoregressive large
language models (LLMs) routinely generate factually incorrect outputs with high decoding confidence, limiting their deployment in high-stakes workflows. Existing output-stage uncertainty metrics can fail when models are overconfident on false assertions, while multi-sample verification pipelines introduce substantial
memory and latency overhead. This work evaluates whether internal hidden-state transition dynamics during generation can signal factual errors without auxiliary decoding calls. We introduce Prediction of Prediction (PoP), a mechanism that captures layer-transition uncertainty by fusing intermediate hidden representations across depth during a single forward pass. Evaluated on the TruthfulQA
benchmark using autoregressive
transformer backbones, PoP achieves an area under the receiver operating characteristic curve (AUROC) of 75.5% for factual-correctness classification. The mechanism operates within the base forward pass, adding less than 1.2% runtime latency and requiring zero additional generation passes. The numerical results are reported from the author-verified experimental implementation and are bounded by the evaluation scope described below.
Read the full abstract ▾
Computation and Language
STAR : Sentence Translation Alignment Rate for Document-to-Document Machine Translation
Yichen Dong, Hao Wang, Junhui Li et al. (+3 more)
Large
Language Models (LLMs) have enabled a shift from sentence-level to document-to-document (Doc2Doc) machine translation, promising improved global coherence. However, document-to-document generation in a single pass frequently suffers from structural misalignment, manifesting as sentence omissions or
hallucinations that violate the core requirement of source-target correspondence. To address this, we introduce Sentence Translation
Alignment Rate (
STAR), an auxiliary metric that explicitly quantifies sentence-level structural fidelity. Building on this, we propose
STAR-masked Preference Optimization (StarPO), a framework that ranks document-level hypotheses by structural quality and utilizes a dynamic
alignment mask to focus optimization on misaligned segments. Experimental results across news and literary domains demonstrate that StarPO significantly enhances translation quality and structural integrity. Notably, StarPO allows compact models to surpass the performance of massive proprietary systems like GPT-4o while maintaining superior
token efficiency.
Read the full abstract ▾
All authors Yichen Dong, Hao Wang, Junhui Li, Linlong Xu, Longyue Wang, Weihua Luo
Computation and Language
Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
Basel Mousi, Fahim Dalvi, Shammur Chowdhury et al. (+2 more)
Multimodal foundation models are increasingly used in
speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs.
speech) and
language (English vs. Arabic). We introduce a
speech-augmented visually grounded contrastive triplet
benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented
reasoning from complete failure. Evaluating recent
multimodal models under text and
speech in English and Arabic, we find that modality and
language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with
speech amplifying partial failures. We make the
benchmark publicly available to the community.
Read the full abstract ▾
All authors Basel Mousi, Fahim Dalvi, Shammur Chowdhury, Firoj Alam, Nadir Durrani
Computation and Language
Small Frequency Corrections Can Change What Survives KV Cache Compression
Hong Chen, Yudong Zeng, Yongwei Huang et al. (+5 more)
Compressing a key-value cache before its next question is known requires choosing what to retain without knowing which evidence will matter. Value energy measures entry strength but does not distinguish isolated keys from those with many similar neighbors. We introduce TwinKV, a training-free method that discounts value energy by nonlocal post-RoPE key frequency. Prefix
attention allocates head capacities, while retained entries preserve their original keys and values under an exact storage budget. Across four
language models, TwinKV exceeds five evaluated compressed baselines in mean score on LongBench, LooGLE, and RULER at 50% KV removal. Component controls isolate the frequency contribution. On Llama-3.2-1B RULER at 75% removal, normalized frequency weights average 0.95, yet change 7% of nonprotected retained positions and improve value-only retention by about 5.5 points under both uniform and adapted capacities. Permuting the weights within heads weakens this gain. These results show that modest frequency corrections can change retention and answering outcomes, with effects that depend on the model and task.
Read the full abstract ▾
All authors Hong Chen, Yudong Zeng, Yongwei Huang, Zuhao Ouyang, Dongnan Zheng, Junyan Zhang, Yubo Gao, Xuming Hu
Computation and Language
Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Ej Zhou, Suchir Salhan, Catherine Arnett et al. (+1 more)
Cross-lingual
alignment in
multilingual language models is typically attributed to joint training: shared parameters, mixed-
language batches, or explicit
alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual
language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with
alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual
alignment can emerge from the structure of
language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular
multilingual systems built from monolingual components.
Read the full abstract ▾
All authors Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen
Computation and Language
DocTalkBN: A Novel Dataset of Expert Telemedicine Conversations in Bengali
Anik Saha, Fahmida Sultana Naznin, Sadatul Islam Sadi et al. (+3 more)
Reliable medical conversational AI requires authentic expert–patient interaction data, yet such
datasets remain scarce, especially for low-resource
languages such as Bengali. We present DocTalkBN, a large-scale
multimodal dataset of real-world expert telemedicine conversations in Bengali, collected from nationally broadcast telemedicine programs featuring board-certified physicians. DocTalkBN contains 557.63 hours of paired audio and text, 1,515 multi-turn patient calls, 10,274 host–doctor question–answer exchanges, totaling 1.7M
tokens, spanning 26 medical specialties. Unlike prior resources derived from medical forums, written health content, or synthetic data, our
dataset preserves the spontaneity, contextual richness, and spoken characteristics of authentic medical interactions in a low-resource setting. To support
benchmark-driven research, we further construct three downstream tasks from the corpus, medical triage classification, advice safety evaluation, and medical named entity recognition, and
benchmark a diverse set of large
language models and encoder-based baselines. Our results show that DocTalkBN is a practically useful resource, particularly for clinically grounded
reasoning tasks. We release this resource to facilitate future research on reliable medical NLP and safer, more culturally grounded healthcare systems for low-resource
languages. Our source codes and
dataset are publicly available at https://anonymous.4open.science/r/doctalk.
Read the full abstract ▾
All authors Anik Saha, Fahmida Sultana Naznin, Sadatul Islam Sadi, Ananya Shahrin Promi, Wahid Al Azad Navid, Rifat Shahriyar
Computation and Language · Cross-listed from cs.LG
Unifying Detection and Adaptation in Task-Free Continual Learning
Dezheng Han, Anbang Zhang, Zhihao Zhu et al. (+1 more)
To mitigate catastrophic forgetting in downstream continual learning (CL) for large
language models (LLMs), existing methods typically constrain parameter updates or introduce task-specific adaptation modules. However, these methods often rely on explicit task boundaries during training, limiting their applicability to realistic task-free scenarios. In this paper, we propose a Fisher-guided unified (FiUni) framework for batch-level task detection and parameter-efficient continual adaptation. FiUni is motivated by a key observation about the Fisher information
matrix (FIM) of pre-trained models: the orthogonality among the principal subspaces of its Kronecker-Factored Approximate
Curvature (K-FAC) approximation, estimated from a small number of downstream task samples, can reflect the similarity between different tasks. Based on this observation, FiUni constructs FIM-derived frozen subspaces to guide low-rank adaptation (LoRA), while matching the Fisher principal subspace of each incoming batch window with historical subspaces. This enables FiUni to adaptively determine whether to reuse existing knowledge, expand a related subspace, or create a new subspace, dynamically balancing knowledge sharing and task isolation. Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.
Read the full abstract ▾
All authors Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo
Computation and Language
Research Design Tracking and Assessment for the Social Sciences
Marco Rovera, Sergiu Burlacu, Dominique Cappelletti et al. (+5 more)
Reliable assessment of causal research designs in the social sciences is critical for evidence-based policy-making, yet has so far relied entirely on manual expert analysis. We introduce Automated Research Design Tracking and Assessment (ARDTrA), a task that involves detecting the research design used in a paper and assessing the quality of its application. We create an expert-annotated
dataset of papers covering six families of counterfactual research designs and evaluate the task using a multi-turn RAG-based conversational pipeline. Across four
retrieval strategies, four LLMs and six
embedding models, we find that passage length is the main driver of performance, explaining 52-66% of the variance. A per-research-design analysis also shows that human and machine difficulty do not align: the designs that prove hardest for the system are not those on which expert annotators disagree most, pointing to two independent sources of task difficulty.
Read the full abstract ▾
All authors Marco Rovera, Sergiu Burlacu, Dominique Cappelletti, Alessio Tomelleri, Sonia Marzadro, Martina Bazzoli, Annalisa Tassi, Jessica Gagete-Miranda
Computation and Language
Cascaded Batch Prompting
Sho Hoshino and Peinan Zhang
Although batch prompting makes large
language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredictable downstream task performance. We propose cascaded batch prompting, a two-stage approach designed to resolve the unpredictability of conventional batch prompting by disentangling complex
reasoning from symbol grounding. Experiments on multiple-choice question answering and natural
language inference demonstrate that the proposed method outperforms the standard single prompting baseline while achieving a speedup proportional to batch size, establishing a new state of the art on the Pareto frontier.
Read the full abstract ▾
Computation and Language
Reasoning about In-Context Samples for Machine-Translation
Maxime Bouthors, Josep Crego and François Yvon
Large
Language Models (LLMs) can be trained to perform chain-of-thoughts
reasoning in order to improve the reliability of their responses. In this work, we investigate how explicit
reasoning can be leveraged for LLM-Based Machine Translation (MT) with in-context samples. We introduce a novel fragment-based
reasoning framework in which the model first extracts parallel source-target fragments from retrieved similar exemplars, and uses these fragments as intermediate
reasoning traces to produce the final translation. To train our model, we distill silver fragments and drafts from a large teacher model. Our experiments with the Qwen3 model family, over 6
languages, including up to 5 domains per
language, demonstrate that fragment-based MT significantly outperforms alternative methods like standard k-shot or basic drafting.
Read the full abstract ▾
Computation and Language
Representing and Parsing Korean Constituency Structure at Different Levels of Granularity
Jungyeul Park, KyungTae Lim, Zihao Huang et al. (+3 more)
Korean constituency parsing raises a representational challenge because the terminal units of a phrase-structure tree do not straightforwardly correspond to simple surface words. Korean eojeols are morphologically complex spacing units, and existing constituency resources differ in how they represent eojeol-internal morphology and non-overt elements. This paper compares three constituency parsing representations derived from the Penn Korean Treebank: Morpheme+XPOS, Eojeol+XPOS, and Eojeol+UPOS. We construct these representations by removing null elements, aligning Penn Korean phrase structure with overt eojeol
tokens, preserving Penn Korean phrase labels where possible, and varying the terminal and preterminal layers. We then evaluate canonical non-binary transition-based constituency parsers in top-down, in-order, and bottom-up orders under a shared modeling and evaluation setup. All experiments use gold terminal
segmentation and gold preterminal labels and therefore evaluate constituency parsing conditioned on gold morphosyntactic annotation. Eojeol terminals yield shorter transition sequences, but Eojeol+UPOS parsing substantially underperforms the morphologically richer conditions. Eojeol+XPOS narrows this gap, while Morpheme+XPOS gives the strongest results even after its predictions are projected to the eojeol terminal domain. Under these gold-annotation conditions, the results show that fine-grained morphological and XPOS representations provide valuable evidence for the evaluated parsers. This empirical finding concerns the information available for parsing and does not by itself determine the linguistically preferable terminal domain. Independently, linguistic and resource-design considerations motivate eojeol as a stable and interpretable surface domain for phrase-structure annotation, with morpheme-level and XPOS information retained as aligned morphosyntactic evidence.
Read the full abstract ▾
All authors Jungyeul Park, KyungTae Lim, Zihao Huang, Eunkyul Leah Jo, Yige Chen, Chulwoo Park
Computation and Language
ITL: Interpretable Document Alignment with Structured Reference Frameworks
Raúl Giráldez, Dayrelis Mena and Jesús S. Aguilar–Ruiz
Measuring
alignment between documents and structured reference frameworks requires identifying conceptual evidence distributed throughout the text and reporting it through measures that are quantitative, interpretable, and traceable. Many commonly used
retrieval and classification approaches return either pairwise similarity scores or one or more class labels, whereas fewer methods provide concept-level scores that are directly traceable to the terminological evidence supporting them. We present Intelligent Target Locator (ITL), a domain-agnostic and
language-portable methodology that estimates the affinity between the textual units of a target document and the concepts defined in a Structured Reference Document ($SRD$). From the $SRD$, ITL induces concept-specific terminological profiles built from independent terms, bigrams, trigrams, and co-occurrences. Each term is assigned an importance weight that combines concept membership, term-type specificity and inter-concept discriminability. The output is a textual-unit–concept affinity
matrix that can be aggregated at different levels of granularity. We conduct an internal consistency assessment using the 17 Sustainable Development Goals (SDGs), evaluating each official goal statement against the $SRD$ induced from the same set of descriptors. Every statement reached its highest affinity with the corresponding concept, and the mean affinity across the remaining concepts stayed marginal relative to the mean reference affinity. This separation indicates that ITL distinguishes the conceptual profiles of the framework. ITL thus offers a general basis for quantifying document
alignment with structured frameworks while keeping each result traceable to the terminological evidence that supports it.
Read the full abstract ▾
Computation and Language
JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols
Chen Chen, Yaolin Chen, Xuehan Sun et al. (+5 more)
Large
language model (LLM) judges are increasingly used across various evaluation scenarios, making their judgment capabilities valuable intellectual property. However, black-box access exposes these capabilities to model extraction attacks. Existing extraction methods do not specifically target LLM judges and provide limited support for multiple evaluation protocols under restricted query budgets. In this study, we propose JUDGESTEALER, the first query-efficient model extraction framework for replicating judging capabilities across pointwise scoring, pairwise comparison, and listwise ranking protocols. JUDGESTEALER exploits the strong cross-protocol agreement to acquire pointwise scores and transform them into pairwise and listwise supervisions without additional victim queries. To capture informative judge patterns and improve query efficiency, JUDGESTEALER dynamically selects pointwise inputs based on semantic diversity, predictive uncertainty, and potential judge biases. It further applies score smoothing and multi-protocol review to preserve the ordinal structure of scores and mitigate catastrophic forgetting during surrogate adaptation. Extensive experiments on state-of-the-art LLM-as-a-judge and reward models show that JUDGESTEALER consistently outperforms existing extraction baselines, achieving up to 73.3%, 87.0%, and 71.6% accuracy for pointwise, pairwise, and listwise evaluation, respectively. JUDGESTEALER also remains effective across different sur- rogate model scales, adaptation strategies, and
reasoning settings. Moreover, JUDGESTEALER demonstrates robustness against representative extraction defenses.
Read the full abstract ▾
All authors Chen Chen, Yaolin Chen, Xuehan Sun, Juan Lin, Xueluan Gong, Yuhang Zheng, Qian Wang, Kwok-Yan Lam