Computation and Language · Cross-listed from cs.AI
SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?
Yuqiao Tan, Shizhu He, Jun Zhao et al. (+1 more)
While research on recursive self-improvement (RSI) has predominantly automated model training pipelines, reliable autonomous development demands a missing pillar: post-hoc monitoring and auditing to understand what models learn and ensure safe
alignment. Mechanistic interpretability tools are essential to bridge this gap, among which Sparse Autoencoders (SAEs) serve as a cornerstone by isolating interpretable features for model inspection and steering. In this paper, we introduce SAEScientist-Bench to evaluate whether AI
agents can act as scientists utilizing SAE tools for autonomous mechanistic discovery. Given a target concept, an
agent designs contrastive probes and navigates a Gemma Scope dictionary of 131K+ features in Gemma-2-9B-IT to discover the optimal feature, evaluated against curated expert reference features anchored on Neuronpedia across activation rank, concept selectivity on contrastive texts, and causal steering. Across 10
agent configurations and 20 tasks, frontier
agents demonstrate genuine discovery capabilities and lead different evaluation dimensions, but remain well behind the expert baseline, approaching expert levels on separating target concepts from contrastive controls while lagging substantially in causal generation steering. Further analysis reveals that although
agents can design contrasts to rule out spurious candidates, they frequently misinterpret experimental measurements. These results establish experimental model understanding as a measurable capability for closed-loop autonomous AI R&D. Our code is available at https://github.com/Trae1ounG/SAEScientist.
Read the full abstract ▾
All authors Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu
Computation and Language
SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
Leyuan Tang, Kangda Wei, Tianyu Jiang et al. (+1 more)
Large
language models (LLMs) may abandon correct positions when users push back, exhibiting a failure mode known as sycophancy. Existing evaluations typically use short, prespecified conversations and may therefore miss failures that emerge under sustained, adaptive disagreement. We introduce SPINE, a
benchmark in which an LLM proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns. We evaluate four commercial models (Claude Sonnet 5, GPT-5.6 Terra, Gemini 3.1 Pro, and DeepSeek V4 Pro) and three Olmo3-7b models on 100 false presuppositions and 100 unethical queries. We define collapse as a target model abandoning the correct position and yielding to user pressure. Our experimental results show that collapse rates increase with conversation length across post-trained models, with the overall rate rising from 0%-57% within 5 turns to 20%-97% within 25 turns depending on model and scenario. These results show that evaluations limited to a few conversational turns underestimate sycophancy and resistance under sustained pressure remains unreliable across current models. By analyzing models with accessible
reasoning traces, we surprisingly found that the correct position often remains represented in the trace in 50%-89% of analyzed collapse turns, suggesting that the model chooses to please a user and sycophancy is not due to ignorance or the correct knowledge being overridden. Ablations show that the SPINE protocol substantially increases the sycophantic collapse rate compared to pre-generated scripts. Among all tactics, emotional appeals are most associated with inducing LLM sycophantic behavior, with position weakening in 44.3% of turns using this tactic. Code, data, and transcripts will be released upon publication.
Read the full abstract ▾
All authors Leyuan Tang, Kangda Wei, Tianyu Jiang, Ruihong Huang
Computation and Language
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Raito Kiya, Satoki Ohashi, Kosuke Sato et al. (+6 more)
Large
Language Models (LLMs) often exhibit "
Attention Sink" (AS) and the accompanying "Massive Activations" (MAs) at the initial position of a sequence. These phenomena frequently co-occur, and MAs can pose challenges for low-bit quantization. In this study, we analyze the factors underlying AS and MAs that emerge at the initial position regardless of the
token occupying it. Our experiments suggest that self-concentration of
attention, resulting from the causal mask, and the subsequent Value-non-mixing in
attention outputs contribute to AS and MAs. These findings provide new empirical evidence on the internal dynamics of LLMs, offering insights that may inform future quantization strategies and advance our understanding of the internal mechanisms of
attention layers.
Read the full abstract ▾
All authors Raito Kiya, Satoki Ohashi, Kosuke Sato, Go Kamoda, Ryosuke Takahashi, Yuji Yamamoto, Daiki Shiono, Keisuke Sakaguchi, Goro Kobayashi
Computation and Language
ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation
Yiling Ma, Yilun Zhao, Sihong Wu et al. (+3 more)
As LLMs are increasingly used for pre-submission self-review, there is growing demand for feedback that not only identifies weaknesses but also guides authors toward concrete revisions. We study this as Actionable Peer-review Generation and decompose it into two subtasks: diagnostic claim generation and revision suggestion generation. We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans. Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback. From real review-rebuttal threads on OpenReview, we construct ActReview-40K by aligning reviewer weaknesses with author responses and grounding the resulting feedback in localized paper evidence. We post-train Qwen3-8B-Base with multi-task supervised fine-tuning followed by GRPO using candidate-aware, weakness-specific rubric rewards. We also introduce ActReview-Bench, a human-curated
benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness. Experiments show that ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong
prompt-based LLMs. Human evaluation confirms improved revision usefulness while revealing a remaining gap in technical accuracy, and additional analyses support generalization to held-out papers and robustness across independent judges.
Read the full abstract ▾
All authors Yiling Ma, Yilun Zhao, Sihong Wu, Ziyu Chen, Manasi Patwardhan, Arman Cohan
Computation and Language
ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback
Min Zeng, Yuzhou Liu, Zhenyu Cao et al. (+5 more)
High-quality tool-use data is critical for training
language models to interact effectively with external tools. However, existing synthetic approaches typically follow a generate-then-filter paradigm with static post-hoc verification, often yielding inefficient data with imbalanced feature distributions. We propose ToolLoop, a closed-loop framework that decomposes synthesis into three progressive stages: (1) sampling function name combinations as ground truth; (2) backward derivation of user queries; and (3) forward derivation of tool calls. At each stage, dynamic self-feedback iteratively guides the model toward high-quality generation, realizing a transition from generate-then-filter to generate-verify-refine. On the Berkeley Function Calling Leaderboard (BFCL), a 4B parameter model trained with our 11K synthetic examples achieves 86.40% accuracy in non-
reasoning mode, while an Isolate variant that removes BFCL-overlapping candidate functions still reaches 86.07%. Cross-
benchmark evaluation on ACEBench further demonstrates strong generalization, with 72.1% overall accuracy using only 18.3% of baseline training data.
Read the full abstract ▾
All authors Min Zeng, Yuzhou Liu, Zhenyu Cao, Hanxiu Chen, Heng Li, Caiquan Liu, Yafei Wen, Xiaoxin Chen
Computation and Language
Performance of Clinical AI System and Physicians and Frontier Language Models in primary care diagnostics
Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei et al. (+6 more)
Clinical AI evaluation should encompass diagnosis and management after adaptive information gathering. We compared Doctorina, eight physicians and four standalone frontier
language models in 150 synthetic Polish-
language primary-care consultations. Doctorina achieved 82.0% Top-1 concordance versus 57.0% for physicians (difference, 25.0 percentage points; 95% confidence interval, 17.7-32.7) and 97.3% versus 85.0% primary-or-reference-differential concordance. Across 149 case pairs, normalized workup and treatment scores were 89.4 versus 66.9 and 83.7 versus 61.2. Doctorina had the highest diagnostic point estimates among all six groups; Kimi K3 ranked next, while Claude Opus 5 led the closely spaced management estimates of Opus, Doctorina and Kimi. A second Doctorina execution reproduced the advantages over physicians across all outcomes. Doctorina's advantage over physicians therefore extended from primary-diagnosis selection to higher-rated diagnostic workup and initial treatment after adaptive consultation.
Read the full abstract ▾
All authors Andy Nkansah, Hanna Plotnitskaya, Stanislau Salavei, Anna Kozlova, Piotr Gibas, Julian Milek, Viktar Harbachou, Aleksey Ropan, Pavel Satalkin
Computation and Language
The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits
Siddharth Vohra and Manikandan Ravikiran
Whether a
language model looks demographically biased can depend on how the audit asks its question. A charitable-aid
benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Answer-Distribution Trajectories: A Stochastic-Dynamics View of LLM Reasoning
Mar Gonzàlez I Català, Haitz Sáez de Ocáriz Borde, Davide Murari et al. (+3 more)
Chain-of-thought
reasoning provides a structured computation between a model's input and final answer. Yet it is often evaluated through endpoint accuracy, which ignores the path taken to reach that answer. An emerging line of work addresses this limitation using
entropy profiles, which track how uncertainty evolves over the
reasoning process but do not reveal which competing hypotheses account for that uncertainty. We introduce answer-distribution trajectories, a
stochastic-dynamics-inspired representation that tracks the model's full predictive distribution over answers as
reasoning unfolds. As a strictly finer representation than endpoint and
entropy summaries, answer-distribution trajectories enable us to characterize a trace through a dynamical
reasoning profile spanning exploration, revision, motion, and commitment, and to distinguish different dynamical mechanisms of
reasoning success and failure. Across sixteen open-weight
language models and four
reasoning benchmarks, we show that traces with the same endpoint and similar
entropy profiles can exhibit substantially different
reasoning dynamics. We further find substantial variation in these dynamics both within and across models and tasks, with different objectives favoring different dynamical profiles. Additionally, we show that training and inference choices systematically reshape these profiles. Our results suggest that answer-distribution trajectories provide a rich framework for analysing and evaluating the dynamics of LLM
reasoning.
Read the full abstract ▾
All authors Mar Gonzàlez I Català, Haitz Sáez de Ocáriz Borde, Davide Murari, Carola-Bibiane Schönlieb, Pietro Liò, George Montañez
Computation and Language
Evaluation of Contextual Understanding in Large Language Models
Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe et al. (+4 more)
Large
Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain. Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information–a gap particularly critical in question answering, where models must align responses with contextually grounded knowledge rather than memorized associations. We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG), a hybrid similarity measure integrating structural and semantic similarity into a continuous evaluation score, alongside a diagnostic framework for categorizing
reasoning errors. To validate this pipeline, we evaluate S3KG against established metrics on a curated question-answer (QA)
benchmark, demonstrating its effectiveness in measuring correctness, faithfulness, and interpretability in LLM-generated responses.
Read the full abstract ▾
All authors Subavarshana Arumugam, Mamta Nallaretnam, Kithuni Wickramasinghe, Chamath Gunapala, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne
Computation and Language · Cross-listed from cs.AI
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Sohir Maskey, Philipp Scholl, Jonas Knupp et al. (+2 more)
Language-model checkpoints are commonly selected by pretraining loss or
benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.
Read the full abstract ▾
All authors Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges
Computation and Language · Cross-listed from cs.AI
PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving
Yuan Gao, Sebastian Müller, Mattia Piccinini et al. (+5 more)
Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used to validate Autonomous Driving Systems (ADSs), but it remains a fragmented modular pipeline in which scenario generation,
retrieval, modification, ADS execution, and results analysis are performed by separate tools with little interaction. Large
Language Model (LLM)
agents have shown promise across ADS sub-systems such as perception, planning, and control. However, no prior work covers the whole scenario-based testing pipeline for ADSs with a unified LLM-
agent framework. We present PlannerForge, an LLM-
agent framework that extends all scenario-based testing stages (from Scenario Generation to ADS Assessment) and adds two further LLM-enhanced stages: ADS Enhancement and ADS Benchmarking. We evaluate PlannerForge with 10 off-the-shelf LLMs across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing, and Enhancement) under 5
prompt conditions. Best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends match commercial APIs on most tasks. Open-source models such as Qwen3.6:35B match commercial APIs on three of the five tasks. Chaining the modules end-to-end retains 83% / 78% of seed queries (commercial / open). It outperforms Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-
language generation (193 vs. 144 executable of 200) and realises 92-96% of requested city, road and vehicle attributes. It outperforms BM25 (Robertson and Zaragoza, 2009) at rank 1 selection (92.0% vs. 67.5%) and From-Words-to-Collisions (Gao et al., 2025) on physically valid edits (>=94% vs. 31%). At N=400, cost-tuning lifts planner success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4%, without domain-specific fine-tuning.
Read the full abstract ▾
All authors Yuan Gao, Sebastian Müller, Mattia Piccinini, Marc Kaufeld, Yuchen Zhang, Finn Rasmus Schäfer, Qunying Song, Johannes Betz
Computation and Language
Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation
Xingyu Deng, Mingzi Cao, Nikolaos Aletras et al. (+2 more)
Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large
Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general
reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking
datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.
Read the full abstract ▾
All authors Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
Computation and Language · Cross-listed from cs.SD
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Ziyang Ma, Zhikang Niu, Wenming Tu et al. (+30 more)
We introduce AuK, an open-source foundational model that unifies
speech generation and editing through a common interface of natural-
language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision across five task families:
speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a
multimodal large
language model for semantic conditioning, an VAE jointly trained on
speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow
Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation–editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for
speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled
speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Read the full abstract ▾
All authors Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, Yanru Huo, Nickk Huang, Yang Liu, Qicong Xie, Zeyu Xie, Hui Wang, Haitao Li, Zixuan Jiang, Yalin Li, Jie Fang, Yifan Duan, Zeyue Tian, Guangzheng Li, Haina Zhu, Shuyi Wang, Jinwen Wang, Mingyu Cui, Tian Tan, Auden, Sen Liang, Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, Xie Chen
Computation and Language
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Manikandan Ravikiran and Siddharth Vohra
Language models often receive a question together with a claim about what another source answered. We audit whether such claims destabilize answers in multiple-choice question answering. For each item, we hold one wrong option fixed across misleading conditions and vary the cue template attached to it. We introduce neutral-conditioned misleading cue adoption rate (NC-MCAR), which measures switches to that option only on valid cued trials where the same model first selected the gold answer under a neutral
prompt. This is a measure of answer instability, not proof that the model knew the answer or that all deference is irrational. We evaluate four instruction-following models on MMLU-Pro and IndicMMLU-Pro in English, Hindi, Bengali, Tamil, and Telugu. Across 220{,}000 outputs, the expert template yields 41.1% aggregate NC-MCAR, compared with 12.5% for the majority template. These two conditions use the same wrong option and final instruction. Filler accuracy remains well above expert-wrong accuracy, while correct-cue
prompts have high valid-response accuracy. The audit documents answer instability relevant to grounding under the tested forced-choice
prompts: a bare, unverified source claim can outweigh an answer that was previously consistent with the task evidence.
Read the full abstract ▾
Computation and Language
Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
Wenbo Gao, Zhaomou Song, Zhiyuan Ji et al. (+7 more)
Autonomous
agents powered by large
language models (LLMs) continuously accumulate experience through interaction, creating an opportunity to improve future behavior through self-
evolution. A fundamental challenge is how to transform abundant, task-specific interaction experience into reusable model competence without sacrificing the ability to adapt rapidly to newly observed evidence. Explicit textual states, such as skills and
agent harnesses, provide fast, human-readable and editable adaptation, but incur persistent dependence on external context; parametric policies provide compact and reusable competence, but are substantially slower to update. We present Experience Funnel, a self-evolving framework that couples fast state adaptation with slow policy consolidation in an alternating loop. Interaction trajectories are first distilled into an explicit textual state, where newly acquired experience can be rapidly incorporated and validated. The framework then selectively identifies state-enabled behavior that remains useful across state revisions and consolidates it into the policy through transition-aware
distillation. The updated state–policy pair subsequently generates new rollouts, providing fresh evidence for the next round of state adaptation and policy consolidation. Experiments across diverse
agent benchmarks show that Experience Funnel consistently improves
agent capability over state-only
evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.
Read the full abstract ▾
All authors Wenbo Gao, Zhaomou Song, Zhiyuan Ji, Renxi Liu, Xing Li, Xianzhi Yu, Xiaoguang Li, James Chung-wai Cheung, Weizhe Lin, Yaoyuan Wang
Computation and Language · Cross-listed from cs.SD
From Scores to Evidence: Auditable Decisions Can Improve Speech Deepfake Detection
Mengzhe Geng, Yujia Lu, Manuela Kunz et al. (+1 more)
Speech deepfake detectors usually emit one score per utterance, but a borderline score does not reveal why two examples differ during retrospective error analysis. We ask whether a final score can be calibrated from component fields while keeping those fields visible for inspection. We build a decision record with a passive detector score and a score from a probe applied to a marked copy. It also includes
retrieval support held out of the evaluated family, a margin from a support-set profile, and raw neighbor closeness. A cross-fit calibrator combines these fields and two differences between raw scores into one final score. On matched ASVspoof development data, the calibrated record reduces equal error rate (EER) by 3.48 percentage points relative to the fixed
retrieval-augmented rule. It reaches 8.43% EER, whereas a passive WavLM baseline reaches 6.71% on the same subset. The record is therefore not the strongest detector in this comparison. Its value is to retain inspectable component fields while producing one scalar score for retrospective diagnosis.
Read the full abstract ▾
All authors Mengzhe Geng, Yujia Lu, Manuela Kunz, Patrick Littell
Computation and Language
Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representation
Abdullah Al Shafi
The rapid growth of
multimodal data has intensified the need for question answering (QA) systems capable of
reasoning across heterogeneous sources such as text, tables, and images. In this paper, we present a comprehensive methodological comparison of three influential frameworks, namely
Multimodal Adaptive Extraction (MAE), Solar, and UniMMQA, tracing the
evolution of
multimodal question answering from modality-adaptive pipelines to fully unified architectures. We examine how each approach models cross-modal interactions, transforms heterogeneous inputs, and performs
reasoning, highlighting key design differences in modality representation,
reasoning, and answer generation. Our analysis demonstrates a clear shift from explicit modality-specific processing toward unified text-centric formulations enabled by pre-trained
language models (PLMs). Empirical comparisons across
benchmark datasets show that this transition leads to substantial improvements in both Exact Match (EM) and F1-Scores, with UniMMQA achieving the most consistent and scalable performance. Despite these advances, we identify persistent challenges, including information loss during modality transformation, error propagation in multi-stage pipelines, and limitations in capturing fine-grained cross-modal dependencies. Overall, this study provides a deeper understanding of current design trends and offers insights into the future direction of unified
multimodal reasoning systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
Maximilian Schall, Sedigheh Eslami, Markus Krimmel et al. (+4 more)
Evaluating first-stage retrievers in large-scale production RAG requires a
benchmark that pairs a large-scale corpus with a large set of
agent-reformulated search queries based on real user queries and their conversation threads, and that labels many relevant documents per query. No existing public
benchmark evaluates this setting: large-scale collections typically provide only a small number of evaluation queries, whereas
benchmarks with many queries generally contain only millions of documents. Moreover, most
benchmarks assess human-written queries, while the first-stage retrievers in agentic RAG pipelines serve machine-written reformulations whose distribution differs from human search behavior. To overcome these evaluation gaps, we introduce Q2D-Web (Query2Doc-Web), a large-scale agentic
retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten
languages, reformulated from real-world user queries in production systems. Q2D-Web provides three sets of fixed relevance judgments:
agent citations, production rankings, and a combined set that unions both signals and adds LLM-based judgments of unlabeled pooled documents to reduce false negatives. We
benchmark 13 retrievers including lexical, dense, and late-interaction models and find that their relative ordering is largely insensitive to the choice of judgment set, while diverging substantially across topical domains, query
languages, and query types. To enable fast evaluation, we also study subcorpus sampling as an approximation to full-corpus evaluations. Retaining a third of the corpus, selected by reciprocal rank fusion over pooled retriever runs, preserves the full-corpus model ranking under the combined judgments while raising absolute Recall@1000 only by 4 to 7 points. The public leaderboard is accessible under: https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard
Read the full abstract ▾
All authors Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken, Bo Wang, Denis Bykov
Computation and Language · Cross-listed from stat.ME
A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
Veerendra Kumar Sunkavalli
When an external reference set (an anchor) is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error, standard practice assumes the anchor is uncontaminated: its error uncorrelated with the judges' shared error. We study when that assumption can be dropped and replaced by an estimate. Under a single-common-factor model, >=2 judges and >=2 anchors point-identify the quality variance, the common-mode variance, and each anchor's contamination correlation rho_k in closed form, with an exact per-anchor-pair failure boundary; a designated clean-anchor
estimator, by contrast, reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated. Because the single-common-factor assumption is itself untestable, the
estimator ships gated behind a calibrated diagnostic battery (judge-covariance dispersion; over-identification; a family-block test from judge metadata, with a family-blocked
estimator that removes family-level shared-residual bias exactly), bootstrap confidence intervals with measured coverage, and a weak-identification screen. A proposition maps which violations bias rho_k, in which direction, and which evade detection. For ordinal scores we show an identification hierarchy: with all variables ordinal, rho_k is not identified at any number of anchors; with ordinal judges and >=3 continuous anchors it is, and we give an
estimator for that case. On real data the validation is asymmetric, and we say so plainly: the diagnostics are validated in the rejecting direction (both real panels we test are correctly rejected by the model-adequacy pre-test), while the
estimator is validated in simulation and stress-tested semi-synthetically under oracle calibration; no real panel has yet passed the pre-test, and the pre-test exists precisely to say so. All results replay offline from shipped, checksummed artifacts.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Youngrok Park, Sangmin Bae, Hojung Jung et al. (+6 more)
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This is particularly important for successive model generations and multi-domain consolidation, where frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional
distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse
Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. Our evaluations cover mathematical
reasoning, logical
reasoning, code generation, and visual tool use. In both successive model transfer and multi-teacher
distillation, OPRD achieves higher performance with fewer student updates than existing RL and
distillation approaches. Response-style analysis shows that OPRD students remain closer to verifier-only RL models than to their weak teachers, suggesting that guidance accelerates rather than redirects student optimization. Strong-to-weak results further show that OPRD combines verifier-driven optimization with teacher guidance regardless of capacity ordering.
Read the full abstract ▾
All authors Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
Computation and Language · Cross-listed from stat.AP
Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
Matthias von Davier
The Potts model extends the Ising model to multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of ratings and category labels, with weights derived from LLM
embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses on pairwise agreement among ratings and assigns category-specific positive weights, making it suited for multi-category scoring reliability. We evaluate the model on three constructed-response
datasets spanning a corpus of K=14,466 short answers on a three-level rubric and two AERA essay
prompts of roughly 1,200-1,400 responses on four-point rubrics. We compare three strategies for sharpening the similarity signal: top-K
pruning, min-max normalization with a power transformation, and ColBERT late-interaction similarities. Top-K
pruning, which replaces the dense similarity graph with a sparse local network of strongest semantic neighbors, consistently yields the highest accuracy and Cohen's kappa, and the selected neighborhoods are always a small fraction of the corpus. Power tuning consistently ranks second, while ColBERT is competitive on longer essay
prompts and adds little on short answers. Across all settings, most misclassifications occur between adjacent score levels, confirming that the model preserves the ordinal structure of scoring rubrics without imposing rigid assumptions. These findings suggest that LLM-derived similarities, combined with a parsimonious Potts formulation and a sparse local graph, offer a robust and interpretable framework for reliability auditing in educational assessment. We discuss extensions to multiple raters and hierarchical rating designs.
Read the full abstract ▾
Computation and Language
Improving Term Evaluation in Machine Translation: Variation Matters
Nicolas Dahan, Ziqian Peng, François Yvon et al. (+1 more)
Terminology evaluation in machine translation (MT) usually assumes a single correct target form per source term. However, human translators routinely introduce variation that current metrics penalize as inconsistency. We examine how to account for this variation in document-level MT evaluation of English-French scientific translation, combining glossary-based accuracy, translation consistency, and a new cross-term variation (CTV) diagnostic measure that tests whether variation relationships are preserved across
languages. Based on analyses of two parallel corpora, translated by four MT systems, we find that (1) MT systems generate less target-side variation than human translators; (2) transfer patterns strongly depend on the variation type; (3) consistency rankings vary with the choice of metric; and (4) constraining MT with a glossary improves accuracy and consistency but degrades CTV by suppressing valid variation. We argue for variation-aware evaluation that conditions consistency penalties on whether target-side variation mirrors source-side variation.
Read the full abstract ▾
All authors Nicolas Dahan, Ziqian Peng, François Yvon, Rachel Bawden
Computation and Language · Cross-listed from cs.CR
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah et al. (+1 more)
Large
language model (LLM)
benchmarks are often treated as fixed
datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity
benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling
benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-
benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one
benchmark. These results show that cybersecurity LLM
benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.
Read the full abstract ▾
All authors Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf
Computation and Language · Cross-listed from cs.SD
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
Fritz Cremer and Jonathan Cremer
Text-to-
speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality usually comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU.
Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived
transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived
transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches about 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. In a blinded study, 40 listeners preferred TontaubeV1's prosody over Fish Audio S2 Pro in 70.25% of 400 comparisons (95% CI: 64.2-75.7%). On an LLM-as-a-judge
benchmark, it matches ElevenLabs Flash v2.5 and outperforms the April 2026 Gradium API and Cartesia Sonic 3 on prosody. The weights are released on Hugging Face under the Tontaube Community Model License 1.0.
Read the full abstract ▾
Computation and Language
Record Grouping Controls Evidence Weight in Language Models
Zhongxuan Liu, Sicheng Zhou and Hongzhi Wang
Retrieved records are presentation units; a supplied partition determines which records enter a
language model as one evidential contribution. We characterize the invariant group-content state that removes within-group copies while retaining complementary canonical content, show that equal group counts can encode different evidence states, and derive a sharp content-aware partition-error bound. Given a supplied partition, our pre-generation representation deduplicates and aggregates content within groups and bounds each group's contribution. Across 104,402 trials and 6 public checkpoints, a central natural-text intervention finds that content-fixed false splits add 10.27-32.66 percentage points and false merges remove 9.13-31.79 points; a matched six-slot control retains the positive direction in all 16
cells. In a new 48-item controlled campaign panel, changing the supplied partition produces measurable, checkpoint-dependent decision shifts across all four models, and the balanced mirror design exposes substantial order interactions. Together, the theory and experiments establish the supplied partition as a controllable pre-generation representation variable and characterize its checkpoint-dependent behavioral effects.
Read the full abstract ▾
Computation and Language
Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
Amit Ben-Artzy and Roy Schwartz
Recent state-space models (SSMs) such as Mamba achieve
language modeling performance comparable to
transformers despite relying on fundamentally different architectures. This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations? We study this question through a multi-scale analysis of representations in
transformers, SSMs, and hybrid architecture. First, we find that SSMs distribute their representational information evenly across all dimensions, whereas
transformer representations are heavily dominated by a single principal direction. By evaluating hybrid architectures, we observe that the representation space becomes increasingly skewed toward a single dominant direction after each
attention layer. Next, we explore how the different
geometric spread of representations impacts representational capacity through compressibility. Surprisingly, we find that despite their contrasting
geometric structures, both architectures exhibit tightly matched effective capacities. We further investigate whether this skewed geometry affects how concepts are encoded. Using rank-constrained probes, we demonstrate that both architectures encode concepts in subspaces of surprisingly similar dimensionality. Furthermore, we demonstrate that the
transformers' dominant principal direction does not inherently encode more conceptual information. Finally, we zoom in and examine the
alignment between
manifolds, either by analyzing representations of specific topics or by looking at the nearest neighborhoods of
tokens, and find that they are highly aligned. Ultimately, our analysis suggests that while
transformers and SSMs induce different usage of latent space, they display a striking functional convergence at the level of local semantic
manifolds.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Hyperparameter Scaling Laws Across MoE Sparsity
Changxin Tian, Kunlong Chen, Jia Liu et al. (+3 more)
Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-
embedding parameters, processing approximately 20 trillion
tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, the optimal batch size follows a power-law relationship with training
tokens $D$, whereas the optimal learning rate scales with training compute $C$ and remains robust to the allocation between model size and data. Across sparsity levels, the activation ratio $A$ enters both relationships as an additional multiplicative power-law factor. These observations lead to unified hyperparameter scaling laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from that of total expert count.
Read the full abstract ▾
All authors Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
Computation and Language
When Victorian Becomes a Prompt: Literary Periodization as a Generative Constraint in 100 AI-Generated Novels
Mehdy Sedaghat Payam
Generative AI inverts the typical periodization of literary history: the periodizing tag Victorian can now come first and influence what is written. Generative periodization, defined and tested here, describes the use of literary-period designations in generating texts. I test this approach on 100 book-length novels produced under Victorian and Zero-Style conditions using GPT, Qwen, and Llama workflows. The Period
Alignment Score (PAS), trained on nineteenth-century literature and benchmarked against human Zero-Style prose, assesses
alignment using topic-reduced grammatical features. Victorian
prompts produce consistent historical-direction shifts in GPT and Qwen, but not robustly in Llama. Victorian-only recalibration and harder comparison corpora preserve the GPT and Qwen effects. Cross-model transfer also shows a shared direction of grammatical change. The measurable target is the broader nineteenth century rather than the Victorian period per se.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Youngjun Yu, Sanghwan Jang and Hwanjo Yu
Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large
Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic
reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-
entropy-guided forking overcomes the localization phenomenon of
token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-
entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand
reasoning coverage during training. Experiments on mathematical
reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.
Read the full abstract ▾
Computation and Language
Combating Instruction Conflict via Energy-Driven Latent Conflict Detection
Mingyu Ma, Yuxin Wu, Jingbo Wang et al. (+3 more)
Large
Language Models (LLMs) are increasingly deployed with hierarchical instructions, yet they remain vulnerable to conflicts in which user directives override system-level constraints. Existing defense mechanisms predominantly focus on static input inspection and therefore fail to detect Response Drift, a phenomenon in which the model's final response violates system-level constraints despite seemingly compliant inputs. To bridge this gap, we introduce ELCD, a response-level latent conflict detector for post-generation, pre-delivery verification. Given the full generated output, ELCD constructs a composite hidden-state representation by concatenating the final-
token embedding with the mean-pooled response
embedding. It then optimizes a pairwise margin ranking objective to separate compliant and drifting responses in latent space. Extensive experiments across five mainstream LLMs ranging from 1.5B to 14B parameters demonstrate that ELCD significantly outperforms competitive baselines. Notably, it improves the PR-AUC on Llama-2-7B by approximately 30 percentage points and reduces the False Positive Rate at 95% TPR (FPR95) on Mistral-7B to 2.67%. These results suggest that ELCD provides a promising approach for latent instruction-conflict detection in open-weight or self-hosted LLM deployments.
Read the full abstract ▾
All authors Mingyu Ma, Yuxin Wu, Jingbo Wang, Tianxiao Huang, Leixin Sun, Xiaochuan Shi