Computation and Language · Cross-listed from eess.AS
Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities
Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini et al. (+1 more)
An accurate assessment of L2 English pronunciation is crucial for
language learning, as it provides personalized feedback and ensures a fair evaluation of individual progress. However, automated scoring remains challenging due to the complexity of sentence-level fluency, prosody, and completeness. This paper evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned
speech-LLM, on 5,000 Speechocean762 utterances. The model generates rubric-aligned scores for accuracy, fluency, prosody, and completeness, showing strong agreement with human ratings within +-2 tolerance, especially for high-quality
speech. However, it tends to overpredict low-quality
speech scores and lacks precision in error detection. These findings demonstrate the strong potential of
speech LLMs in scalable pronunciation assessment and suggest future improvements through enhanced prompting, calibration, and phonetic integration to advance Computer-Assisted Pronunciation Training.
Read the full abstract ▾
All authors Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, Helmer Strik
Computation and Language
From Quotes to Concepts: Axial Coding of Political Debates with Ensemble LMs
Angelina Parfenova, David Graus and Juergen Pfeffer
Axial coding is a commonly used qualitative analysis method that enhances document understanding by organizing sentence-level open codes into broader categories. In this paper, we operationalize axial coding with large
language models (LLMs). Extending an ensemble-based open coding approach with an LLM moderator, we add an axial coding step that groups open codes into higher-order categories, transforming raw debate transcripts into concise, hierarchical representations. We compare two strategies: (i) clustering
embeddings of code-utterance pairs using density-based and partitioning
algorithms followed by LLM labeling, and (ii) direct LLM-based grouping of codes and utterances into categories. We apply our method to Dutch parliamentary debates, converting lengthy transcripts into compact, hierarchically structured codes and categories. We evaluate our method using extrinsic metrics aligned with human-assigned topic labels (ROUGE-L, cosine, BERTScore), and intrinsic metrics describing code groups (coverage, brevity, coherence, novelty, JSD divergence). Our results reveal a trade-off: density-based clustering achieves high coverage and strong cluster
alignment, while direct LLM grouping results in higher fine-grained
alignment, but lower coverage 20%. Overall, clustering maximizes coverage and structural separation, whereas LLM grouping produces more concise, interpretable, and semantically aligned categories. To support future research, we publicly release the full
dataset of utterances and codes, enabling reproducibility and comparative studies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Shourya Jain and Paras Chopra
Users should not be systemically disadvantaged by the
language they use for interacting with LLMs; i.e. users across
languages should get responses of similar quality irrespective of
language used. In this work, we create a set of real-world open-ended questions based on our analysis of the WildChat
dataset and use it to evaluate whether responses vary by
language, specifically, whether answer quality depends on the
language used to query the model. We also investigate how
language and culture are entangled in LLMs such that choice of
language changes the cultural information and context used in the response by using LLM-as-a-Judge to identify the cultural context present in responses. To further investigate this, we evaluate LLMs on a translated subset of the CulturalBench
benchmark across multiple
languages. Our evaluations reveal that LLMs consistently provide lower quality answers to open-ended questions in low resource
languages. We find that
language significantly impacts the cultural context used by the model. This difference in context impacts the quality of the downstream answer.
Read the full abstract ▾
Computation and Language
No Reliable Evidence of Self-Reported Sentience in Small Large Language Models
Caspar Kaiser and Sean Enderby
Whether
language models possess sentience has no empirical answer. But whether they believe themselves to be sentient can, in principle, be tested. We do so by querying several open-weights models about their own consciousness, and then verifying their responses using classifiers trained on internal activations. We draw upon three model families (Qwen, Llama, GPT-OSS) ranging from 0.6 billion to 70 billion parameters, approximately 50 questions about consciousness and subjective experience, and three classification methods from the interpretability literature. First, we find that models consistently deny being sentient: they attribute consciousness to humans but not to themselves. Second, classifiers trained to detect underlying beliefs - rather than mere outputs - provide no clear evidence that these denials are untruthful. Third, within the Qwen family, larger models deny sentience more confidently than smaller ones. These findings contrast with recent work suggesting that models harbour latent beliefs in their own consciousness.
Read the full abstract ▾
Computation and Language
RECAP: A Resource-Efficient Method for Adversarial Prompting in Large Language Models
Rishit Chugh
The deployment of large
language models (LLMs) has raised security concerns due to their susceptibility to producing harmful or policy-violating outputs when exposed to
adversarial prompts. While
alignment and guardrails mitigate common misuse, they remain vulnerable to automated jailbreaking methods such as GCG, PEZ, and GBDA, which generate
adversarial suffixes via training and gradient-based search. Although effective, these methods particularly GCG are computationally expensive, limiting their practicality for organisations with constrained resources. This paper introduces a resource-efficient
adversarial prompting approach that eliminates the need for retraining by matching new
prompts to a database of pre-trained
adversarial prompts. A
dataset of 1,000
prompts was classified into seven harm-related categories, and GCG, PEZ, and GBDA were evaluated on a Llama 3 8B model to identify the most effective attack method per category. Results reveal a correlation between
prompt type and
algorithm effectiveness. By retrieving semantically similar successful
adversarial prompts, the proposed method achieves competitive attack success rates with significantly reduced computational cost. This work provides a practical framework for scalable red-teaming and security evaluation of aligned LLMs, including in settings where model internals are inaccessible.
Read the full abstract ▾
Computation and Language
ICPO: Illocution-Calibrated Policy Optimization for Multi-Turn Conversation
Zhebo Wang, Xiaohu Mu, Zijie Zhou et al. (+4 more)
Large
Language Models (LLMs) in multi-turn conversations often suffer from a “lost-in-conversation” phenomenon, where they struggle to recover from early incorrect assumptions, particularly when users provide ambiguous initial instructions. We find that standard post-training techniques like Reinforcement Learning with Verifiable Rewards (RLVR) exacerbate this issue by rewarding confident, direct answers, thereby inducing overconfidence and discouraging the model from seeking clarification. To address this, we propose Illocution-Calibrated Policy Optimization (ICPO), a novel training framework that sensitizes the model to instruction ambiguity. ICPO augments the training corpus with underspecified
prompts and conditions the reward signal on the user's illocutionary intent, rewarding the model for expressing uncertainty or asking for clarification when faced with ambiguity. Experiments demonstrate that ICPO fosters appropriate humility, yielding a substantial average improvement of 75% in multi-turn conversation, while preserving robust performance on single-turn
benchmarks. Our work presents a practical path toward more robust and collaborative conversational AI that can better navigate the nuances of human interaction.
Read the full abstract ▾
All authors Zhebo Wang, Xiaohu Mu, Zijie Zhou, Mohan Li, Wenpeng Xing, Dezhang Kong, Meng Han
Computation and Language
Towards Execution-Grounded Automated AI Research
Chenglei Si, Zitong Yang, Yejin Choi et al. (+3 more)
Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems - LLM pre-training and post-training - into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline (69.4% vs 48.0%) on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts towards execution-grounded automated AI research.
Read the full abstract ▾
All authors Chenglei Si, Zitong Yang, Yejin Choi, Emmanuel Candès, Diyi Yang, Tatsunori Hashimoto
Computation and Language
Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence
Jinhui Liu, Ximeng Zhang, Yanbo Ai et al. (+1 more)
Evaluating Text-to-SQL
agents in private business intelligence (BI) settings is challenging due to the scarcity of realistic, domain-specific data. While synthetic evaluation data offers a scalable solution, existing generation methods fail to capture business realism–whether questions reflect realistic business logic and workflows. We propose a Business Logic-Driven Data Synthesis framework that generates data grounded in business personas, work scenarios, and workflows. In addition, we improve the data quality by imposing a business
reasoning complexity control strategy that diversifies the analytical
reasoning steps required to answer the questions. Experiments on a production-scale Salesforce database show that our synthesized data achieves high business realism (98.44%), substantially outperforming OmniSQL (+19.5%) and SQL-Factory (+54.7%), while maintaining strong question-SQL
alignment (98.59%). Our synthetic data also reveals that state-of-the-art Text-to-SQL models still have significant performance gaps, achieving only 42.86% execution accuracy on the most complex business queries.
Read the full abstract ▾
All authors Jinhui Liu, Ximeng Zhang, Yanbo Ai, Zhou Yu
Computation and Language · Cross-listed from cs.CY
Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education
Amogh Gupta, Niharika Patil, Sourojit Ghosh et al. (+1 more)
Large
language models are increasingly deployed in STEM education for personalized instruction and feedback across institutions in high- and low-income countries. These systems are designed to adapt content to student needs, but whether they adapt based on demonstrated ability or demographic signals remains untested at scale. Here we establish that LLM-generated STEM content systematically disadvantages marginalized student profiles across two cultural contexts, with the gap between the most privileged and most marginalized profiles reaching 2.55 grade levels. We audited four LLMs (Qwen 2.5-32B-Instruct, GPT-4o, GPT-4o-mini, GPT-OSS 20B) using synthetic profiles crossing dimensions specific to Indian education (caste, medium of instruction, college tier) and American education (race, HBCU attendance, school type), alongside income, gender, and disability, across ranking and generation tasks with FDR-corrected significance testing and SHAP feature attribution. Income produces significant effects across every model and context, medium of instruction drives the largest single effect in the Indian context, and disability status triggers simpler explanations. Effects compound non-additively: marginalization across multiple dimensions produces gaps larger than any single dimension predicts, and biases persist within elite institutions. Bias is consistent across all four architectures and persists through model selection, making intersectional, cross-cultural
auditing a structural requirement before deployment.
Read the full abstract ▾
All authors Amogh Gupta, Niharika Patil, Sourojit Ghosh, SnehalKumar S Gaikwad
Computation and Language · Cross-listed from cs.CV
GutenOCR: A Grounded Vision-Language Front-End for Documents
Hunter Heidenreich, Ben Elliott, Olivia Dinica et al. (+1 more)
GutenOCR is a family of grounded OCR front-ends obtained by fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B. The resulting single-checkpoint
vision-
language models expose reading, detection, and grounding through a unified,
prompt-based interface. Trained on business documents, scientific articles, and synthetic grounding data, the models support full-page and localized reading with line- and paragraph-level bounding boxes and conditional “where is x?” queries. We introduce a grounded OCR evaluation protocol and show that GutenOCR-7B more than doubles the composite grounded OCR score of its Qwen2.5-VL-7B backbone on 10.5K held-out business and scientific pages (0.40 to 0.82). On Fox and OmniDocBench v1.5, our approach substantially improves region- and line-level OCR as well as text-detection recall, but reveals trade-offs in page-level linearization, color-guided OCR, and formula-heavy layouts.
Read the full abstract ▾
All authors Hunter Heidenreich, Ben Elliott, Olivia Dinica, Yosheb Getachew
Computation and Language
Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
Crish Nagarkar, Leonid Bogachev and Serge Sharoff
This paper investigates the ability of large
language models (LLMs) to solve statistical tasks, as well as their capacity to assess the quality of
reasoning. While state-of-the-art LLMs have demonstrated remarkable performance in a range of NLP tasks, their competence in addressing even moderately complex statistical challenges is not well understood. We have fine-tuned selected open-source LLMs on a specially developed
dataset to enhance their statistical
reasoning capabilities, and compared their performance with the human scores used as a
benchmark. Our results show that the fine-tuned models achieve better performance on advanced statistical tasks on the level comparable to a statistics student. Fine-tuning demonstrates architecture-dependent improvements, with some models showing significant performance gains, indicating clear potential for deployment in educational technology and statistical analysis assistance systems. We also show that LLMs themselves can be far better judges of the answers quality (including explanation and
reasoning assessment) in comparison to traditional metrics, such as BLEU or BertScore. This self-evaluation capability enables scalable automated assessment for statistical education platforms and quality assurance in automated analysis tools. Potential applications also include validation tools for research methodology in academic and industry settings, and quality control mechanisms for data analysis workflows.
Read the full abstract ▾
Computation and Language
Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research
Sasha Ronaghi, Emma-Louise Aveling, Maria Levis et al. (+3 more)
Large
language models (LLMs) show promise for improving the efficiency of qualitative analysis in large, multi-site health-services research. Yet methodological guidance for LLM integration into qualitative analysis and evidence of their impact on real-world research methods and outcomes remain limited. We developed a model- and task-agnostic framework for designing human-LLM qualitative analysis methods to support diverse analytic aims. Within a multi-site study of diabetes care at Federally Qualified Health Centers (FQHCs), we leveraged the framework to implement human-LLM methods for (1) qualitative synthesis of researcher-generated summaries to produce comparative feedback reports and (2) deductive coding of 167 interview transcripts to refine a practice-transformation intervention. LLM assistance enabled timely feedback to practitioners and the incorporation of large-scale qualitative data to inform theory and practice changes. This work demonstrates how LLMs can be integrated into applied health-services research to enhance efficiency while preserving rigor, offering guidance for continued innovation with LLMs in qualitative research.
Read the full abstract ▾
All authors Sasha Ronaghi, Emma-Louise Aveling, Maria Levis, Rachel Lauren Ross, Emily Alsentzer, Sara Singer
Computation and Language · Cross-listed from cs.SD
Prosody-Guided Harmonic Attention for Phase-Coherent Neural Vocoding in the Complex Spectrum
Mohammed Salah Al-Radhi, Riad Larbi, Mátyás Bartalis et al. (+1 more)
Neural vocoders are central to
speech synthesis; despite their success, most still suffer from limited prosody modeling and inaccurate phase reconstruction. We propose a vocoder that introduces prosody-guided harmonic
attention to enhance voiced segment encoding and directly predicts complex spectral components for waveform synthesis via inverse STFT. Unlike mel-spectrogram-based approaches, our design jointly models magnitude and phase, ensuring phase coherence and improved pitch fidelity. To further align with perceptual quality, we adopt a multi-objective training strategy that integrates
adversarial, spectral, and phase-aware losses. Experiments on
benchmark datasets demonstrate consistent gains over HiFi-GAN and AutoVocoder: F0 RMSE reduced by 22 percent, voiced/unvoiced error lowered by 18 percent, and MOS scores improved by 0.15. These results show that prosody-guided
attention combined with direct complex spectrum modeling yields more natural, pitch-accurate, and robust synthetic
speech, setting a strong foundation for expressive neural vocoding.
Read the full abstract ▾
All authors Mohammed Salah Al-Radhi, Riad Larbi, Mátyás Bartalis, Géza Németh
Computation and Language · Cross-listed from cs.AI
VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration
Saeed Khaki, Ashudeep Singh, Nima Safaei et al. (+1 more)
Vision-
language models (VLMs) lag behind text-only
language models on mathematical
reasoning when the same problems are presented as images rather than text. We empirically characterize this as a modality gap: the same question in text form yields markedly higher accuracy than its visually typeset counterpart, due to compounded failures in reading dense formulas, layout, and mixed symbolic-diagrammatic context. First, we introduce VisTIRA (
Vision and Tool-Integrated
Reasoning Agent), a tool-integrated
reasoning framework that enables structured problem solving by iteratively decomposing a given math problem (as an image) into natural
language rationales and executable Python steps to determine the final answer. Second, we build a framework to measure and improve visual math
reasoning: a LaTeX-based pipeline that converts chain-of-thought math corpora (e.g., NuminaMath) into challenging image counterparts, and a large set of synthetic tool-use trajectories derived from a real-world, homework-style image
dataset (called SnapAsk) for fine-tuning VLMs. Our experiments show that tool-integrated supervision improves image-based
reasoning, and OCR grounding can further narrow the gap for smaller models, although its benefit diminishes at scale. These findings highlight that modality gap severity inversely correlates with model size, and that structured
reasoning and OCR-based grounding are complementary strategies for advancing visual mathematical
reasoning.
Read the full abstract ▾
All authors Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra
Computation and Language · Cross-listed from cs.CV
Vision-Based Natural Language Scene Understanding for Autonomous Driving: An Extended Dataset and a New Model for Traffic Scene Description Generation
Danial Sadrian Zadeh, Otman A. Basir and Behzad Moshiri
Traffic scene understanding is essential for enabling autonomous vehicles to accurately perceive and interpret their environment, thereby ensuring safe
navigation. This paper presents a novel framework that transforms a single frontal-view camera image into a concise natural
language description, effectively capturing spatial layouts, semantic relationships, and driving-relevant cues. The proposed model leverages a hybrid
attention mechanism to enhance spatial and semantic feature extraction and integrates these features to generate contextually rich and detailed scene descriptions. To address the limited availability of specialized
datasets in this domain, a new
dataset derived from the BDD100K
dataset has been developed, with comprehensive guidelines provided for its construction. Furthermore, the study offers an in-depth discussion of relevant evaluation metrics, identifying the most appropriate measures for this task. Extensive quantitative evaluations using metrics such as CIDEr and SPICE, complemented by human judgment assessments, demonstrate that the proposed model achieves strong performance and effectively fulfills its intended objectives on the newly developed
dataset.
Read the full abstract ▾
Computation and Language
Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis
Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu et al. (+4 more)
Many spoken
languages, including English, exhibit wide variation in dialects and accents, making accent control an important capability for flexible text-to-
speech (TTS) models. Current TTS systems typically generate accented
speech by conditioning on speaker
embeddings associated with specific accents. While effective, this approach offers limited interpretability and controllability, as
embeddings also encode traits such as timbre and
emotion. In this study, we analyze the interaction between speaker
embeddings and linguistically motivated phonological rules in accented
speech synthesis. Using American and British English as a case study, we implement rules for flapping, rhoticity, and vowel correspondences. We propose the phoneme shift rate (PSR), a novel metric quantifying how strongly
embeddings preserve or override rule-based transformations. Experiments show that combining rules with
embeddings yields more authentic accents, while
embeddings can attenuate or overwrite rules, revealing
entanglement between accent and speaker identity. Our findings highlight rules as a lever for accent control and a framework for evaluating disentanglement in
speech generation.
Read the full abstract ▾
All authors Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu, Jihwan Lee, Tiantian Feng, Dani Byrd, Shrikanth Narayanan
Computation and Language · Cross-listed from cs.LG
Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM
YuanLab.ai: Shawn Wu, Jiangang Luo, Darcy Chen et al. (+23 more)
We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large
language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert
Pruning (LAEP)
algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert
pruning approaches that operate primarily in the post-training phase, the proposed
algorithm enhances training efficiency by selectively
pruning underutilized experts and reorganizing experts across computing devices according to
token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this
algorithm delivers a 49% boost in pre-training efficiency and a 33.3% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario
benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra.
Read the full abstract ▾
All authors YuanLab.ai: Shawn Wu, Jiangang Luo, Darcy Chen, Sean Wang, Louie Li, Allen Wang, Xudong Zhao, Tong Yu, Bach Li, Joseph Shen, Gawain Ma, Jasper Jia, Marcus Mao, Claire Wang, Hunter He, Carol Wang, Zera Zhang, Jason Wang, Chonly Shen, Leo Zhang, Logan Chen, Qasim Meng, James Gong, Daniel Zhao, Penn Zheng, Owen Zhu
Computation and Language
Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment
Yuming Yang, Mingyoung Lai, Wanxu Zhao et al. (+13 more)
Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling
reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in
distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both
alignment and informativeness to assess the suitability of a
reasoning trajectory. RSR is motivated by the observation that effective trajectories typically balance learning signal strength and behavioral
alignment by combining low absolute probability with relatively high-ranked
tokens under the student model. Concretely, RSR is defined as the ratio of a trajectory's average
token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and
reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training
reasoning performance (average Spearman 0.86), consistently outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.
Read the full abstract ▾
All authors Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language · Cross-listed from cs.LG
Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow
Haocheng Xi, Charlie Ruan, Peiyuan Liao et al. (+7 more)
Reinforcement learning (RL) is essential for enhancing the complex
reasoning capabilities of large
language models (LLMs). However, existing RL training pipelines are computationally inefficient and resource-intensive, with the rollout phase accounting for over 70% of total training time. Quantized RL training, particularly using FP8 precision, offers a promising approach to mitigating this bottleneck. A commonly adopted strategy applies FP8 precision during rollout while retaining BF16 precision for training. In this work, we present the first comprehensive study of FP8 RL training and demonstrate that the widely used BF16-training + FP8-rollout strategy suffers from severe training instability and catastrophic accuracy collapse under long-horizon rollouts and challenging tasks. Our analysis shows that these failures stem from the off-policy nature of the approach, which introduces substantial numerical mismatch between training and inference. Motivated by these observations, we propose Jet-RL, an FP8 RL training framework that enables robust and stable RL optimization. The key idea is to adopt a unified FP8 precision flow for both training and rollout, thereby minimizing numerical discrepancies and eliminating the need for inefficient inter-step calibration. Extensive experiments validate the effectiveness of Jet-RL: our method achieves up to 33% speedup in the rollout phase, up to 41% speedup in the training phase, and a 16% end-to-end speedup over BF16 training, while maintaining stable convergence across all settings and incurring negligible accuracy degradation.
Read the full abstract ▾
All authors Haocheng Xi, Charlie Ruan, Peiyuan Liao, Yujun Lin, Han Cai, Yilong Zhao, Shuo Yang, Kurt Keutzer, Song Han, Ligeng Zhu
Computation and Language
Bertie Vidgen, Austin Mann, Abby Fennelly et al. (+21 more)
We introduce the AI Productivity Index for
Agents (APEX-
Agents), a
benchmark for assessing whether AI
agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate lawyers. APEX-
Agents requires
agents to navigate realistic work environments with files and tools. We test eight
agents for the leaderboard using Pass@1. Gemini 3 Flash (Thinking=High) achieves the highest score of 24.0%, followed by GPT-5.2 (Thinking=High), Claude Opus 4.5 (Thinking=High), and Gemini 3 Pro (Thinking=High). We open source the APEX-
Agents benchmark (n=480) with all
prompts, rubrics, gold outputs, files, and metadata. We also open source Archipelago, our infrastructure for
agent execution and evaluation.
Read the full abstract ▾
All authors Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, Neel Venugopal, Alannah Hsia, Isaac Robinson, Calix Huang, Olivia Varones, Daniyal Khan, Michael Haines, Austin Bridges, Jesse Boyle, Koby Twist, Zach Richards, Chirag Mahapatra, Brendan Foody, Osvald Nitski
Computation and Language
MASCOT: Multi-Agent Socio-Collaborative Companion Systems
Yiyang Wang, Yiqiao Jin, Alex Cabral et al. (+1 more)
Multi-
agent systems (MAS) are emerging as promising socio-collaborative companions for emotional and cognitive support. However, existing systems frequently suffer from persona collapse, where
agents revert to generic, homogenized assistant behaviors, and social sycophancy, where
agents produce redundant, non-constructive dialogue. We propose MASCOT, a multi-
agent framework for multi-perspective socio-collaborative companions. MASCOT introduces a novel bi-level optimization strategy to harmonize individual and collective behaviors: 1) Persona-Aware Behavioral
Alignment, an RLAIF-driven pipeline that finetunes individual
agents for
agent-specific identities; and 2) Collaborative Dialogue Optimization, a group-level adaptation process that promotes complementary, diverse, and productive discourse. We evaluate MASCOT using human-grounded contexts drawn across both in-domain and out-of-domain (OOD) settings against state-of-the-art baselines. MASCOT improves persona consistency by up to +14.1 and social contribution by up to +10.6. A broad evaluation suite, including human evaluation, multiple LLM judges, three-way comparisons, and automatic metrics, further shows that MASCOT produces more role-consistent and less redundant multi-
agent dialogue.
Read the full abstract ▾
All authors Yiyang Wang, Yiqiao Jin, Alex Cabral, Josiah Hester
Computation and Language · Cross-listed from cs.NE
Generalization and Completeness of Stochastic Local Search Algorithms
Daniel Loscos, Narciso Marti-Oliet and Ismael Rodriguez
We generalize
Stochastic Local Search (SLS) heuristics into a unique formal model. This model has two key components: a common structure designed to be as large as possible and a parametric structure intended to be as small as possible. Each heuristic is obtained by instantiating the parametric part in a different way. Particular instances for Genetic
Algorithms (GA), Ant Colony Optimization (ACO), and Particle Swarm Optimization (PSO) are presented. Then, we use our model to prove the Turing-completeness of SLS
algorithms in general. The proof uses our framework to construct a GA able to simulate any Turing machine. This Turing-completeness implies that determining any non-trivial property concerning the relationship between the inputs and the computed outputs is undecidable for GA and, by extension, for the general set of SLS methods (although not necessarily for each particular method). Similar proofs are more informally presented for PSO and ACO.
Read the full abstract ▾
Computation and Language
DRIFT: Detecting Representational Inconsistencies for Factual Truthfulness
Rohan Bhatnagar, Youran Sun, Chi Andrew Zhang et al. (+2 more)
LLMs often produce fluent but incorrect answers, yet detecting such
hallucinations typically requires multiple sampling passes or post-hoc verification, adding significant latency and cost. We hypothesize that intermediate layers encode confidence signals that are lost in the final output layer, and propose a lightweight probe to read these signals directly from hidden states. The probe adds less than 0.1% computational overhead and can run fully in parallel with generation, enabling
hallucination detection before the answer is produced. Building on this, we develop an LLM router that answers confident queries immediately while delegating uncertain ones to stronger models. Despite its simplicity, our method achieves SOTA AUROC on 10 out of 12 settings across four QA
benchmarks and three LLM families, with gains of up to 13 points over prior methods, and generalizes across
dataset shifts without retraining.
Read the full abstract ▾
All authors Rohan Bhatnagar, Youran Sun, Chi Andrew Zhang, Yixin Wen, Haizhao Yang
Computation and Language · Cross-listed from cs.LG
InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning
Matthew Y. R. Yang, Hao Bai, Ian Wu et al. (+3 more)
Outcome-reward reinforcement learning (RL) has proven effective at improving the
reasoning capabilities of large
language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire
reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective
reasoning steps remains challenging. We introduce Intervention Training (InT), a training paradigm in which the model performs fine-grained credit assignment on its own
reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical
reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its
reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b.
Read the full abstract ▾
All authors Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, Aviral Kumar
Computation and Language · Cross-listed from cs.AI
Toward Efficient Agents: Memory, Tool learning, and Planning
Xiaofang Yang, Lijun Li, Heng Zhou et al. (+12 more)
Recent years have witnessed increasing interest in extending large
language models into agentic systems. While the effectiveness of
agents has continued to improve, efficiency, which is crucial for real-world deployment, has often been overlooked. This paper therefore investigates efficiency from three core components of
agents:
memory, tool learning, and planning, considering costs such as latency,
tokens, steps, etc. Aimed at conducting comprehensive research addressing the efficiency of the agentic system itself, we review a broad range of recent approaches that differ in implementation yet frequently converge on shared high-level principles including but not limited to bounding context via compression and management, designing reinforcement learning rewards to minimize tool invocation, and employing controlled search mechanisms to enhance efficiency, which we discuss in detail. Accordingly, we characterize efficiency in two complementary ways: comparing effectiveness under a fixed cost budget, and comparing cost at a comparable level of effectiveness. This trade-off can also be viewed through the Pareto frontier between effectiveness and cost. From this perspective, we also examine efficiency oriented
benchmarks by summarizing evaluation protocols for these components and consolidating commonly reported efficiency metrics from both
benchmark and methodological studies. Moreover, we discuss the key challenges and future directions, with the goal of providing promising insights.
Read the full abstract ▾
All authors Xiaofang Yang, Lijun Li, Heng Zhou, Tong Zhu, Xiaoye Qu, Yuchen Fan, Qianshan Wei, Rui Ye, Li Kang, Yiran Qin, Daizong Liu, Qi Li, Ning Ding, Siheng Chen, Jing Shao
Computation and Language · Cross-listed from cs.LG
Suvrat Raju and Praneeth Netrapalli
We study the error rate of LLMs on tasks like arithmetic that require a deterministic output, and repetitive processing of
tokens drawn from a small set of alternatives. We argue that incorrect predictions arise when small errors in the
attention mechanism accumulate to cross a threshold, and use this insight to derive a quantitative two-parameter relationship between the accuracy and the complexity of the task. The two parameters vary with the
prompt and the model; they can be interpreted in terms of an elementary noise rate, and the number of plausible erroneous
tokens that can be predicted. Our analysis is inspired by an “effective field theory” perspective: the LLM's many raw parameters can be reorganized into just two parameters that govern the error rate. We perform extensive empirical tests, using Gemini 2.5 Flash, Gemini 2.5 Pro and DeepSeek R1, and find excellent agreement between the predicted and observed accuracy for a variety of tasks, although we also identify deviations in some cases. Our model provides an alternative to suggestions that errors made by LLMs on long repetitive tasks indicate the “collapse of
reasoning”, or an inability to express “compositional” functions. Finally, we show how to construct
prompts to reduce the error rate.
Read the full abstract ▾
Computation and Language
Human Values in a Single Sentence: Moral Presence, Hierarchies, and Transformer Ensembles on the Schwartz Continuum
Víctor Yeste and Paolo Rosso
We study neural multi-label classification under severe label imbalance through sentence-level detection of the 19 refined Schwartz human values in 74k English news and manifesto sentences (ValueEval'24 corpus). Each sentence carries a roughly balanced moral-presence label and a 19-way value annotation. First, moral presence is learnable from single sentences: a DeBERTa-base classifier reaches positive-class $F_1 \approx 0.73$ at the default threshold, which calibration does not improve. Second, comparing direct multi-label detectors with presence-gated hierarchies under an 8 GB consumer-grade GPU budget, we find that gating does not improve over direct prediction, as gate recall becomes a bottleneck. Third, studying lightweight auxiliary signals and small ensembles, we isolate decision-threshold calibration as a decisive, often overlooked factor: a standard text-only baseline already matches the best official ValueEval'24 English run at the default threshold (macro-$F_1 = 0.282$ vs. $\approx 0.28$), and tuning the threshold on validation alone raises it to $0.315$, most of our overall gain. Lightweight features do not survive a paired per-seed test; a soft-voting ensemble reaches our best macro-$F_1 = 0.332$. Calibration is not architecture-specific: it reproduces on RoBERTa-base, where the gain is larger ($+0.043$). To our knowledge, this is the first systematic comparison of direct and presence-gated architectures, lightweight feature-augmented encoders, and instruction-tuned Large
Language Models (LLMs) at sentence level; benchmarked 7-9B LLMs (zero-/few-shot and QLoRA) lag behind the supervised ensemble under the same budget. We provide empirical guidance for compute-efficient, value-aware NLP models.
Read the full abstract ▾
Computation and Language
Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Law
Ali Hamza Bashir, Muhammad Rehan Khalid, Kostadin Cvejoski et al. (+7 more)
Large
language models (LLMs) often struggle in specialized domains such as legal
reasoning due to limited expert knowledge, resulting in factually incorrect outputs or
hallucinations. This paper presents an effective method for adapting advanced LLMs to German legal question answering through a novel synthetic data generation approach. In contrast to costly human-annotated resources or unreliable synthetic alternatives, our approach systematically produces high-quality, diverse, and legally accurate question-answer pairs directly from authoritative German statutes. Using rigorous automated filtering methods and parameter-efficient fine-tuning techniques, we demonstrate that LLMs adapted with our synthetic
dataset significantly outperform their baseline counterparts on German legal question answering tasks. Our results highlight the feasibility of using carefully designed synthetic data as a robust alternative to manual annotation in high-stakes, knowledge-intensive domains.
Read the full abstract ▾
All authors Ali Hamza Bashir, Muhammad Rehan Khalid, Kostadin Cvejoski, Jana Birr, Jule Berghaus, Armin Berger, Sandra Halscheidt, Christian Temath, Rafet Sifa, David Berghaus
Computation and Language
Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models
Hyunjong Ok and Jaeho Lee
Large
language models exhibit surprising sensitivity to the structure of the
prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and
datasets. Through systematic architectural analysis, we identify causal
attention as the core mechanism: in QOC
prompts, the causal mask prevents option
tokens from attending to context, creating an information bottleneck where context becomes invisible to options.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning
Renmiao Chen, Yida Lu, Shiyao Cui et al. (+6 more)
As
Multimodal Large
Language Models (MLLMs) acquire stronger
reasoning capabilities to handle complex, multi-image instructions, this advancement may pose new safety risks. We study this problem by introducing MIR-SafetyBench, the first
benchmark focused on multi-image
reasoning safety, which consists of 2,676 instances across a taxonomy of 9 multi-image relations. Our extensive evaluations on 19 MLLMs reveal a troubling trend: models with more advanced multi-image
reasoning can be more vulnerable on MIR-SafetyBench. Beyond attack success rates, we find that many responses labeled as safe are superficial, often driven by misunderstanding or evasive, non-committal replies. We further observe that unsafe generations exhibit lower
attention entropy than safe ones on average. This internal signature suggests a possible risk that models may over-focus on task solving while neglecting safety constraints. Our code and data are available at https://github.com/thu-coai/MIR-SafetyBench.
Read the full abstract ▾
All authors Renmiao Chen, Yida Lu, Shiyao Cui, Xuan Ouyang, Victor Shea-Jay Huang, Shumin Zhang, Chengwei Pan, Han Qiu, Minlie Huang