Computation and Language · Cross-listed from eess.AS
Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech
Gus Lathouwers, Lingyun Gao, Catia Cucchiarini et al. (+1 more)
Automatic
Speech Recognition (ASR) is increasingly used in applications involving child
speech, such as
language learning and literacy acquisition. However, the effectiveness of such applications is limited by high ASR error rates. The negative effects can be mitigated by identifying in advance which ASR-outputs are reliable. This work aims to develop two novel approaches for selecting reliable ASR-output at the utterance level, one for selecting reliable read
speech and one for dialogue
speech material. Evaluations were done on an English and a Dutch
dataset, each with a baseline and finetuned model. The results show that utterance-level selection methods for identifying reliably transcribed
speech recordings have high precision for the best strategy (P > 97.4) for both read
speech and dialogue material, for both
languages. Using the current optimal strategy allows 21.0% to 55.9% of dialogue/read
speech datasets to be automatically selected with low (UER of < 2.6) error rates.
Read the full abstract ▾
All authors Gus Lathouwers, Lingyun Gao, Catia Cucchiarini, Helmer Strik
Computation and Language · Cross-listed from eess.AS
Enhancing ASR Performance in the Medical Domain for Dravidian Languages
Sri Charan Devarakonda, Ravi Sastry Kolluru, Manjula Sri Rayudu et al. (+3 more)
Automatic
Speech Recognition (ASR) for low-resource Dravidian
languages like Telugu and Kannada faces significant challenges in specialized medical domains due to limited annotated data and morphological complexity. This work proposes a novel confidence-aware training framework that integrates real and synthetic
speech data through a hybrid confidence mechanism combining static perceptual and acoustic similarity metrics with dynamic model
entropy. Unlike direct fine-tuning approaches, the proposed methodology employs both fixed-weight and learnable-weight confidence aggregation strategies to guide sample weighting during training, enabling effective utilization of heterogeneous data sources. The framework is evaluated on Telugu and Kannada medical
datasets containing both real recordings and TTS-generated synthetic
speech. A 5-gram KenLM
language model is applied for post-decoding correction. Results show that the hybrid confidence-aware approach with learnable weights substantially reduces recognition errors: Telugu Word Error Rate (WER) decreases from 24.3% to 15.8% (8.5% absolute improvement), while Kannada WER drops from 31.7% to 25.4% (6.3% absolute improvement), both significantly outperforming standard fine-tuning baselines. These findings confirm that combining adaptive confidence-aware training with statistical
language modeling delivers superior performance for domain-specific ASR in morphologically complex Dravidian
languages.
Read the full abstract ▾
All authors Sri Charan Devarakonda, Ravi Sastry Kolluru, Manjula Sri Rayudu, Rashmi Kapoor, Madhu G, Anil Kumar Vuppala
Computation and Language · Cross-listed from cs.MM
Neuro-Oracle: A Trajectory-Aware Agentic RAG Framework for Interpretable Epilepsy Surgical Prognosis
Aizierjiang Aiersilan and Mohamad Koubeissi
Predicting post-
surgical seizure outcomes in pharmacoresistant epilepsy is a clinical challenge. Conventional deep-learning approaches operate on static, single-timepoint pre-operative scans, omitting longitudinal morphological changes. We propose Neuro-Oracle, a three-stage framework that: (i) distils pre-to-post-operative MRI changes into a compact 512-dimensional trajectory vector using a 3D Siamese contrastive encoder; (ii) retrieves historically similar
surgical trajectories from a population archive via nearest-neighbour search; and (iii) synthesises a natural-
language prognosis grounded in the retrieved evidence using a quantized Llama-3-8B
reasoning agent. Evaluations are conducted on the public EPISURG
dataset ($N{=}268$ longitudinally paired cases) using five-fold stratified cross-validation. Since ground-truth seizure-freedom scores are unavailable, we utilize a clinical proxy label based on the resection type. We acknowledge that the network representations may potentially learn the anatomical features of the resection cavities (i.e., temporal versus non-temporal locations) rather than true prognostic morphometry. Our current evaluation thus serves mainly as a proof-of-concept for the trajectory-aware
retrieval architecture. Trajectory-based classifiers achieve AUC values between 0.834 and 0.905, compared with 0.793 for a single-timepoint ResNet-50 baseline. The Neuro-Oracle
agent (M5) matches the AUC of purely discriminative trajectory classifiers (0.867) while producing structured justifications with zero observed
hallucinations under our audit protocol. A Siamese Diversity Ensemble (M6) of trajectory-space classifiers attains an AUC of 0.905 without
language-model overhead.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.PL
M$^\star$: Every Task Deserves Its Own Memory Harness
Wenbo Pan, Shujie Liu, Xiangyang Zhou et al. (+5 more)
Large
language model
agents rely on a
memory harness to write, organize, retrieve, and use past experience. A harness that works for one task often fails on another, because conversations,
embodied planning, and expert
reasoning require different storage and
retrieval behavior. To address this limitation, we introduce Mstar, a method that automatically discovers task-optimized
memory programs through reflective code
evolution. Specifically, Mstar operationalizes a
memory harness as an executable Python
memory program that defines the Schema, Logic, and Instruction, and optimizes these components jointly. We evaluate Mstar on four tasks covering conversation,
embodied planning, and specialized
reasoning. Our results demonstrate that Mstar improves performance over existing baselines across all evaluated tasks. Furthermore, the evolved
memory programs exhibit structurally distinct processing mechanisms for each evaluated domain. These findings suggest that every task benefits from its own
memory harness, and that
memory-program search provides a concrete way to discover it.
Read the full abstract ▾
All authors Wenbo Pan, Shujie Liu, Xiangyang Zhou, Xianlong Wang, Shiwei Zhang, Bingjing Xu, Wanlu Shi, Xiaohua Jia
Computation and Language
Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning
Samuel Jaeger, Calvin Ibenye, Aya Vera-Jimenez et al. (+1 more)
The rapid adoption of large
language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation, raising important questions about how these two forms of deceptive content differ and how reliably they can be distinguished. This study examines linguistic, structural, and emotional differences between human-written and AI-generated fake news and evaluates machine learning and ensemble-based methods for distinguishing these content types. A document-level feature representation is constructed using sentence structure, lexical diversity, punctuation patterns, readability indices, and
emotion-based features capturing affective dimensions such as fear, anger, joy, sadness, trust, and anticipation. Multiple classification models, including logistic regression, random forest, support vector machines, extreme gradient boosting, and a neural network, are applied alongside an ensemble framework that aggregates predictions across models. Model performance is assessed using accuracy and area under the receiver operating characteristic curve. The results show strong and consistent classification performance, with readability-based features emerging as the most informative predictors and AI-generated text exhibiting more uniform stylistic patterns. Ensemble learning provides modest but consistent improvements over individual models. These findings indicate that stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.
Read the full abstract ▾
All authors Samuel Jaeger, Calvin Ibenye, Aya Vera-Jimenez, Dhrubajyoti Ghosh
Computation and Language · Cross-listed from cs.CV
Cross-Cultural Value Attribution in Large Vision-Language Models
Phillip Howard, Xin Su and Kathleen C. Fraser
The rapid adoption of large
vision-
language models (LVLMs) in recent years has been accompanied by growing
fairness concerns due to their propensity to reinforce harmful societal stereotypes. While significant
attention has been paid to such
fairness concerns in the context of social biases, relatively little prior work has examined the presence of stereotypes in LVLMs related to cultural contexts such as religion, nationality, and socioeconomic status. In this work, we aim to narrow this gap by investigating how LVLM judgments about a person's moral, ethical, and political values vary across cultural contexts presented in images. We conduct a multi-dimensional analysis of such value judgments in popular LVLMs using counterfactual image sets, which depict the same person across different cultural contexts. Our evaluation framework pairs descriptive analyses (Moral Foundations Theory categorization, lexical analyses, and value sensitivity) with a novel grounding analysis that compares LVLM cross-context variation against two large-scale human surveys (MFQ-2 and WVS Wave 7). Across 4.8 million LVLM generations, we identify three survey-grounding bias patterns that replicate across multiple architecturally diverse models. Additional ablations show that nationality grounding is text-dominant while religion and socioeconomic grounding depend strongly on the image, and that image conditioning can amplify survey-grounding bias patterns.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
Yu Wu, Guangzeng Han, Ibra Niang Niang et al. (+6 more)
High-throughput, effective and comprehensive phenotyping is a critical prerequisite for crop improvement. While phenotypic analysis was traditionally performed manually, recent advances in
multimodal foundation models, especially in
vision-
language models, have offered new opportunities for automated plant phenotyping. However, plant science remains particularly challenging for general-purpose VLMs because it requires domain-specific knowledge, fine-grained visual interpretation, and complex agronomic
reasoning. We introduce PlantXpert, an evidence-grounded
multimodal reasoning benchmark for plant phenotyping across four major crops: soybean, cotton, rice, and corn. Our
benchmark provides a structured and reproducible framework for agronomic adaptation of VLMs, and enables controlled comparison between base models and their domain-adapted counterparts. We constructed a
dataset comprising 545 digital images and 4,351
benchmark samples spanning key agronomic domains including disease, pest control, weed management and yield. The
benchmark can assess diverse capabilities including visual expertise, quantitative
reasoning, and multi-step agronomic
reasoning. We trained and evaluated a total of 11 state-of-the-art VLMs. Fine-tuning on our
dataset improves overall accuracy for all models, with many achieving around 70% accuracy, and improvements on 9 of 11 remaining statistically significant after Holm correction. We also identify diminishing returns from model scaling, uneven performance across crop subsets, and persistent weaknesses in quantitative
reasoning and specialized image analysis. These limitations show that general
multimodal competence alone is insufficient for reliable agronomic
reasoning and demonstrate the value of domain-specific adaptation. We hope PlantXpert can serve as a foundation for advancing evidence-grounded
multimodal reasoning in plant science.
Read the full abstract ▾
All authors Yu Wu, Guangzeng Han, Ibra Niang Niang, Francia Ravelombola, Maiara Oliveira, Jason Davis, Dong Chen, Feng Lin, Xiaolei Huang
Computation and Language
Should We be Pedantic About Reasoning Errors in Machine Translation?
Calvin Bao and Marine Carpuat
Across multiple
language pairings (English $\to$ \{Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese\}), we find
reasoning errors in translation. To quantify how often these
reasoning errors occur, we leverage an automated annotation protocol for
reasoning evaluation wherein the goal is to detect if a
reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3)
reasoning trace-misaligned. We probe the
reasoning model with perturbed traces correcting for these identified
reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-
reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the
reasoning traces suggests that small corrections to the
reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that
reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these
reasoning errors does not resolve the initial errors significantly, suggesting limited
reasoning faithfulness for machine translation.
Read the full abstract ▾
Computation and Language
Simulating Organized Group Behavior: New Framework, Benchmark, and Analysis
Xinkai Zou, Yiming Huang, Zhuohang Wu et al. (+5 more)
Simulating how organized groups (e.g., corporations) make decisions (e.g., responding to a competitor's move) is essential for understanding real-world dynamics and could benefit relevant applications (e.g.,
market prediction). In this paper, we formalize this problem as a concrete research platform for group behavior understanding, providing: (1) a task formalization with
benchmark and evaluation criteria, (2) a structured and adaptive analytical framework, and (3) detailed temporal and cross-group analysis. Specifically, we propose Organized Group Behavior Simulation, a task that models organized groups as collective entities from a practical perspective: given a group facing a particular situation (e.g., AI Boom), predict the decision it would take. To support this task, we present GROVE (GRoup Organizational BehaVior Evaluation), a
benchmark covering 44 organized groups with 8,052 real-world context-decision pairs collected from Wikipedia and TechCrunch across 9 domains, with an end-to-end evaluation protocol assessing consistency, initiative, scope, magnitude, and horizon. Beyond straightforward prompting pipelines, we propose a structured analytical framework that converts collective decision-making events into an interpretable, adaptive, and traceable behavioral model, achieving stronger performance than summarization- and
retrieval-based baselines. It further introduces an adapter mechanism for time-aware
evolution and group-aware transfer, and traceable evidence nodes grounding each decision rule in originating historical events. Our analysis reveals temporal behavioral drift within individual groups, which the time-aware adapter effectively captures for stronger prediction, and structured cross-group similarity that enables knowledge transfer for data-scarce organizations.
Read the full abstract ▾
All authors Xinkai Zou, Yiming Huang, Zhuohang Wu, Jian Sha, Nan Huang, Longfei Yun, Jingbo Shang, Letian Peng
Computation and Language · Cross-listed from cs.AI
Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards
Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao et al. (+4 more)
The recent advancement of Large
Language Models (LLMs) has established their potential as autonomous interactive
agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer
agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic
evolution. The
agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B
agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained
agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile,
adversarial seller personas.
Read the full abstract ▾
All authors Shuze Daniel Liu, Claire Chen, Jiabao Sean Xiao, Lei Lei, Yuheng Zhang, Yisong Yue, David Simchi-Levi
Computation and Language
Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling
Peiqi Sui, Yutong Zhu, Tianyi Cheng et al. (+4 more)
LLMs have so far failed both to generate consistently compelling stories and to recognize this failure–on the leading creative-writing
benchmark (EQ-Bench), LLM judges rank zero-shot AI stories above New Yorker short stories, a gold standard for literary fiction. We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension. We introduce the 100-Endings metric, which walks through a story sentence by sentence: at each position, a model predicts how the story will end 100 times given only the text so far, and we measure tension as how often predictions fail to match the ground truth. Beyond the mismatch rate, the sentence-level curve also yields complementary statistics that track plot-level twists and revelations, such as the inflection rate, a
geometric measure of how frequently the curve reverses direction. Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs. Grounded in narratological principles, we design a story-generation pipeline using structural constraints, including analysis of story templates, idea formulation, and narrative scaffolding. Our pipeline significantly increases narrative tension as measured by the 100-Endings metric, while maintaining performance on the EQ-Bench leaderboard.
Read the full abstract ▾
All authors Peiqi Sui, Yutong Zhu, Tianyi Cheng, Peter West, Richard Jean So, Hoyt Long, Ari Holtzman
Computation and Language · Cross-listed from cs.AI
COMPOSITE-Stem
Kyle Waters, Lucas Nuzzi, Tadhg Looram et al. (+20 more)
AI
agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written
benchmarks have proven effective at measuring AI
reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a
benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our
benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted
multimodal Terminus-2
agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current
agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains.
Read the full abstract ▾
All authors Kyle Waters, Lucas Nuzzi, Tadhg Looram, Alessandro Tomasiello, Ariel Ghislain Kemogne Kamdoum, Bikun Li, Damien Sileo, Egor Kretov, Francesco Fournier-Facio, Georgios Soloupis, Haile Kassahun, Hew Wolff, Jiaqi Cai, Lianghui Li, Marc Roth, Mohinder Naiya, Naixu Guo, Qicheng Tang, Richard Wheeler, Samuele Sala, Serguei Popov, Steven Dillmann, Yuqi Li
Computation and Language · Cross-listed from cs.RO
Nastaran Darabi and Amit Ranjan Trivedi
Vision language action (VLA) models enable generalist robotic
agents but often exhibit
language ignorance, relying on visual shortcuts and remaining insensitive to instruction changes. We present Prospective Grounding and
Alignment VLA (ProGAL-VLA), which constructs a 3D entity-centric graph (GSM), uses a slow planner to produce symbolic sub-goals, and aligns them with grounded entities via a Grounding
Alignment Contrastive (GAC) loss. All actions are conditioned on a verified goal
embedding $g_t$, whose
attention entropy provides an intrinsic ambiguity signal. On LIBERO-Plus, ProGAL-VLA increases robustness under
robot perturbations from 30.3 to 71.5 percent, reduces
language ignorance by 3x-4x, and improves entity
retrieval from 0.41 to 0.71 Recall@1. On the Custom Ambiguity
Benchmark, it reaches AUROC 0.81 (vs. 0.52), AUPR 0.79, and raises clarification on ambiguous inputs from 0.09 to 0.81 without harming unambiguous success. The verification bottleneck increases mutual information of
language-actions, the GAC loss imposes an entity-level InfoNCE bound, and
attention entropy yields calibrated selective prediction, indicating that explicit verified grounding is an effective path toward instruction-sensitive, ambiguity-aware
agents.
Read the full abstract ▾
Computation and Language
Claim2Vec: Embedding Fact-Check Claims for Multilingual Similarity and Clustering
Rrubaa Panchendrarajan and Arkaitz Zubiaga
Recurrent claims present a major challenge for automated fact-checking systems designed to combat misinformation, especially in
multilingual settings. While tasks such as claim matching and fact-checked claim
retrieval aim to address this problem by linking claim pairs, the broader challenge of effectively representing groups of similar claims that can be resolved with the same fact-check via claim clustering remains relatively underexplored. To address this gap, we introduce Claim2Vec, the first
multilingual embedding model optimized to represent fact-check claims as vectors in an improved semantic
embedding space. We fine-tune a
multilingual encoder using contrastive learning with similar
multilingual claim pairs. Experiments on the claim clustering task using three
datasets, 14
multilingual embedding models, and 7 clustering
algorithms demonstrate that Claim2Vec significantly improves clustering performance. Specifically, it enhances both cluster label
alignment and the
geometric structure of the
embedding space across different cluster configurations. Our
multilingual analysis shows that clusters containing multiple
languages benefit from fine-tuning, demonstrating cross-lingual knowledge transfer.
Read the full abstract ▾
Computation and Language
GIANTS: Generative Insight Anticipation from Scientific Literature
Joy He-Yueya, Anikait Singh, Ge Gao et al. (+5 more)
Scientific breakthroughs often emerge from synthesizing prior ideas into novel contributions. While
language models (LMs) show promise in scientific discovery, their ability to perform this targeted, literature-grounded synthesis remains underexplored. We introduce insight anticipation, a generation task in which a model predicts a downstream paper's core insight from its foundational parent papers. To evaluate this capability, we develop GiantsBench, a
benchmark of 17k examples across eight scientific domains, where each example consists of a set of parent papers paired with the core insight of a downstream paper. We evaluate models using an LM judge that scores similarity between generated and ground-truth insights, and show that these similarity scores correlate with expert human ratings. Finally, we present
GIANTS-4B, an LM trained via reinforcement learning (RL) to optimize insight anticipation using these similarity scores as a proxy reward. Despite its smaller open-source architecture,
GIANTS-4B outperforms proprietary baselines and generalizes to unseen domains, achieving a 34% relative improvement in similarity score over gemini-3-pro. Human evaluations further show that
GIANTS-4B produces insights that are more conceptually clear than those of the base model. In addition, SciJudge-30B, a third-party model trained to compare research abstracts by likely citation impact, predicts that insights generated by
GIANTS-4B are more likely to lead to higher citations, preferring them over the base model in 68% of pairwise comparisons. We release our code,
benchmark, and model to support future research in automated scientific discovery.
Read the full abstract ▾
All authors Joy He-Yueya, Anikait Singh, Ge Gao, Michael Y. Li, Sherry Yang, Chelsea Finn, Emma Brunskill, Noah D. Goodman
Computation and Language · Cross-listed from cs.AI
Pioneer Agent: Continual Improvement of Small Language Models in Production
Dhruv Atreja, Julia White, Nikhil Nayak et al. (+5 more)
Small
language models are attractive for production deployment due to their low cost, fast inference, and ease of specialization. However, adapting them to a specific task remains a challenging engineering loop, driven not by training itself but by surrounding decisions: data curation, failure diagnosis, regression avoidance, and iteration control. We present Pioneer
Agent, a closed-loop system that automates this lifecycle. In cold-start mode, given only a natural-
language task description, the
agent acquires data, constructs evaluation sets, and iteratively trains models by jointly optimizing data, hyperparameters, and learning strategy. In production mode, given a deployed model with labeled failures, it diagnoses error patterns, constructs targeted training data, and retrains under explicit regression constraints. To evaluate this setting, we introduce AdaptFT-Bench, a
benchmark of synthetic inference logs with progressively increasing noise, designed to test the full adaptation loop: diagnosis, curriculum synthesis, retraining, and verification. Across eight cold-start
benchmarks spanning
reasoning, math, code generation, summarization, and classification, Pioneer
Agent improves over base models by 1.6-83.8 points. On AdaptFT-Bench, it improves or preserves performance in all seven scenarios, while naive retraining degrades by up to 43 points. On two production-style deployments built from public
benchmark tasks, it raises intent classification from 84.9% to 99.3% and Entity F1 from 0.345 to 0.810. Beyond performance gains, the
agent often discovers effective training strategies, including chain-of-thought supervision, task-specific optimization, and quality-focused data curation, purely from downstream feedback.
Read the full abstract ▾
All authors Dhruv Atreja, Julia White, Nikhil Nayak, Kelton Zhang, Henrijs Princis, George Hurn-Maloney, Ash Lewis, Urchade Zaratiana
Computation and Language · Cross-listed from cs.MA
CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation
Aarush Sinha, Arion Das, Soumyadeep Nag et al. (+7 more)
As large
language models (LLMs) are increasingly deployed as autonomous
agents, understanding how strategic behavior emerges in multi-
agent environments has become an important
alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-
agent simulation in a simplified model of New York City, where LLM-driven
agents interact under opposing incentives. Blue
agents aim to reach their destinations efficiently, while Red
agents attempt to divert them toward billboard-heavy routes using persuasive
language to maximize advertising revenue. Hidden identities make
navigation socially mediated, forcing
agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates
agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue
agents are optimized to reduce billboard exposure while preserving
navigation efficiency, whereas Red
agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist
adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM
agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to
adversarial persuasion.
Read the full abstract ▾
All authors Aarush Sinha, Arion Das, Soumyadeep Nag, Charan Karnati, Shravani Nag, Chandra Vadhan Raj, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das
Computation and Language
Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types
Hadas Orgad, Boyi Wei, Kaden Zheng et al. (+4 more)
Large
language models remain vulnerable to jailbreaks that elicit harmful responses, yet the mechanism behind harmful response generation is poorly understood. Here, we investigate how this capability is organized within model parameters. We identify and prune parameters that specifically support harmful compliance, providing a direct mechanistic analysis at the parameter level. We find that this capability depends on a sparse set of critical parameters:
pruning these parameters substantially reduces harmful compliance while causing only limited degradation in benign capabilities, suggesting that key components of harmful generation are separable from those of general utility. Parameters identified from one harm category also reduce harmful responses in others, indicating components shared across harm types. This separability appears primarily in aligned models, suggesting that
alignment training internally reshapes the harmful response mechanism even when behavioral safeguards remain brittle. We further show that harmful response generation is dissociable from the ability to recognize and reason about harmfulness. Finally, we extend our analysis to emergent misalignment and identify a sparse set of parameters contributing to it, with substantial sharing across fine-tuning domains. Together, these results reveal a consistent parameter-level organization underlying unsafe behaviors and point toward more principled interventions for improving model safety.
Read the full abstract ▾
All authors Hadas Orgad, Boyi Wei, Kaden Zheng, Martin Wattenberg, Peter Henderson, Seraphina Goldfarb-Tarrant, Yonatan Belinkov
Computation and Language
Automated grounding verification of AI statements about chest radiographs
Soroosh Tayebi Arasteh, Mehdi Joodaki, Mahshad Lotfinia et al. (+4 more)
A grounding verifier checks a statement about a patient's image against the radiology report and the cited evidence. The state-of-the-art methods for testing these verifiers remove or replace the cited evidence while the patient's report stays unchanged. To our knowledge, no existing method checks both the report and the cited evidence. We present Building Report And Citation Examples (BRACE), an automated framework that builds training examples from public radiology reports and Radiopaedia articles. We compared a lexical baseline and 42 verifiers with 0.1-284B parameters on 5 different public chest radiograph
datasets. At a threshold accepting 0.90 of the supported test examples, verifiers trained with BRACE accepted a mean of only 0.239 of AI-written statements contradicted by the report. Verifiers trained without contradicted examples passed the state-of-the-art tests, yet accepted most of these statements (0.830 on average). With citations retrieved from held-out Radiopaedia articles, verifiers trained with BRACE accepted 0.169 of AI-written statements with an agreeing report and an unsupporting citation. By contrast, zero-shot large
language models mostly checked the report and not the citation (they accepted 0.830 of these statements). We further assessed the framework manually, with two board-certified radiologists independently labeling a sample of test reports. The framework's labels agreed with each radiologist (k > 0.910) almost as closely as the radiologists agreed with each other (k = 0.952). Moreover, instantiating the framework for food allergens in recipes, we showed that it extends beyond clinical applications. A verifier built the same way accepted 0.493 of the statements contradicted by the recipe, whereas a lexical baseline accepted 0.898. Using both the report and the cited evidence, verifiers trained with BRACE could check AI statements before clinicians read them.
Read the full abstract ▾
All authors Soroosh Tayebi Arasteh, Mehdi Joodaki, Mahshad Lotfinia, Tri-Thien Nguyen, Sebastian Ziegelmayer, Sven Nebelung, Daniel Truhn
Computation and Language · Cross-listed from cs.CV
VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images
Guanyu Zhou, Yida Yin, Wenhao Chai et al. (+3 more)
Vision-
language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition, largely because natural image
datasets provide limited supervision for low-level visual skills. Can targeted synthetic supervision address these weaknesses without reference images or manual annotation? To investigate this, we introduce VisionFoundry, an automated pipeline that takes only a task name as input, uses LLMs to synthesize paired questions, answers, and text-to-image (T2I)
prompts, generates images with T2I models, and filters samples via
multimodal verification. With VisionFoundry, we construct VisionFoundry-10k, a synthetic VQA
dataset spanning 10 perception tasks. Finetuning on VisionFoundry-10k consistently improves perception
benchmarks across three open-source backbones (e.g., +6.7% on MMVP-pair and +10.5% on CV-Bench-3D for Qwen2.5-VL-3B-Instruct) while preserving broader capabilities and showing positive data scaling. The same synthetic supervision also yields consistent gains under reinforcement learning (RL) across all three backbones, and the framework remains effective under open-source synthesis and self-verification. Our findings demonstrate that automated synthetic supervision offers an effective and scalable path toward systematic VLM training.
Read the full abstract ▾
All authors Guanyu Zhou, Yida Yin, Wenhao Chai, Shengbang Tong, Xingyu Fu, Zhuang Liu
Computation and Language · Cross-listed from cs.CV
VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning
Wenyi Xiao, Xinchi Xu and Leilei Gan
Large
Vision Language Models (LVLMs) achieve strong
multimodal reasoning but frequently exhibit
hallucinations and incorrect responses with high certainty, which hinders their usage in high-stakes domains. Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness. This design is mismatched to LVLMs: an incorrect prediction may arise from perceptual failures or from
reasoning errors given correct perception, and a single confidence conflates these sources while visual uncertainty is often dominated by
language priors. To address these issues, we propose VL-Calibration, a reinforcement learning framework that explicitly decouples confidence into visual and
reasoning confidence. To supervise visual confidence without ground-truth perception labels, we introduce an intrinsic visual certainty estimation that combines (i) visual grounding measured by KL-divergence under image perturbations and (ii) internal certainty measured by
token entropy. We further propose
token-level advantage reweighting to focus optimization on
tokens based on visual certainty, suppressing ungrounded
hallucinations while preserving valid perception. Experiments on thirteen
benchmarks show that VL-Calibration effectively improves calibration while boosting visual
reasoning accuracy, and it generalizes to out-of-distribution
benchmarks across model scales and architectures.
Read the full abstract ▾
Computation and Language
Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Xinyu Wang, Sai Koneru, Wenbo Zhang et al. (+3 more)
Recent advances in large
language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior work has often treated fake news detection as a binary classification problem, modern fake news increasingly arises through human-AI collaboration, where strategic inaccuracies are embedded within otherwise accurate and credible narratives. These mixed-truth cases represent a realistic and consequential threat, yet they remain underrepresented in existing
benchmarks. To address this gap, we introduce MANYFAKE, a synthetic
benchmark containing 6,798 fake news articles generated through multiple strategy-driven prompting pipelines that capture many ways fake news can be constructed and refined. Using this
benchmark, we evaluate a range of state-of-the-art fake news detectors. Our results show that even advanced
reasoning-enabled models approach saturation on fully fabricated stories, but remain brittle when falsehoods are subtle, optimized, and interwoven with accurate information.
Read the full abstract ▾
All authors Xinyu Wang, Sai Koneru, Wenbo Zhang, Wenliang Zheng, Saksham Ranjan, Sarah Rajtmajer
Computation and Language
You Can't Fight in Here! This is BBS!
Richard Futrell and Kyle Mahowald
Norm, the formal theoretical linguist, and Claudette, the computational
language scientist, have a lovely time discussing whether modern
language models can inform important questions in the
language sciences. Just as they are about to part ways until they meet again, 25 of their closest friends show up – from linguistics, neuroscience, cognitive science, psychology, philosophy, and computer science. We use this discussion to highlight what we see as some common underlying issues: the String Statistics Strawman (the mistaken idea that LMs can't be linguistically competent or interesting because they, like their Markov model predecessors, are statistical models that learn from strings) and the As Good As it Gets Assumption (the idea that LM research as it stands in 2026 is the limit of what it can tell us about linguistics). We clarify the role of LM-based work for scientific insights into human
language and advocate for a more expansive research program for the
language sciences in the AI age, one that takes on the commentators' concerns in order to produce a better and more robust science of both human
language and of LMs.
Read the full abstract ▾
Computation and Language
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe et al. (+2 more)
Accurate evaluation is central to the large
language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling trade-off between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.
Read the full abstract ▾
All authors Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo
Computation and Language
RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval
Kyle Whitecross and Negin Rahimi
We propose RecaLLM, a set of
reasoning language models post-trained to make effective use of long-context information. In-context
retrieval, which identifies relevant evidence from context, and
reasoning are deeply intertwined:
retrieval supports
reasoning, while
reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context
retrieval performance substantially degrades even after a short
reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought:
reasoning steps that improve performance also make subsequent in-context
retrieval more challenging. To address this limitation, RecaLLM interleaves
reasoning with explicit in-context
retrieval, alternating between
reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic
retrieval tasks, RecaLLM achieves strong performance on two long-context
benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K
tokens using training samples of at most 10K
tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.
Read the full abstract ▾
Computation and Language
Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
Vishnu Murali, Anmol Gulati, Elias Lumer et al. (+3 more)
Translating natural
language into Jira Query
Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based
benchmark exists for mapping natural
language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL
benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented
agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic
retrieval tool that resolves natural-
language mentions of categorical values through
embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-
language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the
benchmark, all
agent transcripts, and evaluation code to support reproducibility.
Read the full abstract ▾
All authors Vishnu Murali, Anmol Gulati, Elias Lumer, Kevin Frank, Sindy Campagna, Vamse Kumar Subbiah
Computation and Language
Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities
Sathvik Nair and Colin Phillips
Under the lens of Marr's levels of analysis, we critique and extend two claims about
language models (LMs) and
language processing: first, that predicting upcoming linguistic information based on context is central to
language processing, and second, that many advances in psycholinguistics would be impossible without large
language models (LLMs). We further outline future directions that combine the strengths of LLMs with psycholinguistic models.
Read the full abstract ▾
Computation and Language
From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
Chenchen Zhang
Reinforcement learning (RL) for large
language models (LLMs) increasingly relies on sparse outcome rewards, yet such rewards say little about which
token,
reasoning step, tool call,
memory operation, or
agent caused an outcome. This credit assignment (CA) problem spans
reasoning RL and becomes sharper in agentic RL, where environment interaction introduces transition non-closure, partial observability, limited replay, heterogeneous actions, weak intermediate verifiability, and
agent coupling. We synthesize a unified corpus of 69 papers published from January 2024 through July 31, 2026: 56 core CA methods and 13 adjacent or boundary enablers, selected from 92 deduplicated screening records. We retain the original granularity-by-methodology taxonomy and add a six-diagnostic framework mapping assumption breaks to identification barriers,
estimators, and evaluation controls. A source-located full-text audit covers a fixed 42-core-paper subset. Two
algorithm researchers independently and blindly cross-coded 252 diagnostic
cells, agreeing on 223 (88.5%); per-diagnostic Cohen's kappa ranges from .543 to .909, and principal-family agreement is 42/42 (kappa=1.000). Beyond taxonomy, we establish when restored-state comparisons identify a protocol-specific causal contrast, show that text-only histories can leave even the sign of credit unidentified, and introduce a reusable CA-ID Card linking each claim to its estimand, evidence provenance, and falsification test. An atomic reporting audit describes comparator, budget parity, ablation, overhead, uncertainty, and replay coverage without constructing a cross-paper leaderboard. The companion repository hosts a living catalog and decision aids; a dated release of the frozen audit bundle is planned there separately from the minimal arXiv source.
Read the full abstract ▾
Computation and Language
Many-Tier Instruction Hierarchy in LLM Agents
Jingyu Zhang, Tianjian Li, William Jurayj et al. (+3 more)
Large
language model
agents receive instructions from many sources-system messages, user
prompts, tool outputs, other
agents, and more-each carrying different levels of trust and authority. When these instructions conflict,
agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first
benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world
agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.
Read the full abstract ▾
All authors Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan, Benjamin Van Durme, Daniel Khashabi
Computation and Language
UIPress: Bringing Optical Token Compression to UI-to-Code Generation
Dasen Dai, Shuoqi Li, Ronghao Chen et al. (+3 more)
UI-to-Code generation requires
vision-
language models (VLMs) to produce thousands of
tokens of structured HTML/CSS from a single screenshot, making visual
token efficiency critical. Existing compression methods either select
tokens at inference time using task-agnostic heuristics, or zero out low-
attention features without actually shortening the sequence – neither truly reduces prefill latency or adapts to the non-uniform information density of UI screenshots. Meanwhile, optical (encoder-side learned) compression has shown strong results for document OCR, yet no prior work has adapted this paradigm to UI-to-Code generation. We propose UIPress, a lightweight learned compression module inserted between the frozen ViT encoder and the LLM decoder of Qwen3-VL-8B. UIPress combines depthwise-separable convolutions, element-guided spatial reweighting, and
Transformer refinement to compress ${\sim}$6{,}700 visual
tokens to a fixed budget of 256. Together with Low-Rank Adaptation (LoRA) on the decoder to bridge the representation gap, the entire system adds only ${\sim}$21.7M trainable parameters (0.26% of the 8B base model). Under a fair comparison on the same base model against four baselines on Design2Code, UIPress at 256
tokens achieves a CLIP score of 0.8127, outperforming the uncompressed baseline by +7.5% and the strongest inference-time method by +4.6%, while delivering 9.1$\times$ time-to-first-
token speedup. To the best of our knowledge, UIPress is the first encoder-side learned compression method for the UI-to-Code task.
Read the full abstract ▾
All authors Dasen Dai, Shuoqi Li, Ronghao Chen, Huacan Wang, Biao Wu, Qizhen Lan