Computation and Language
Weighted Ensemble Sampling for Rare-Event Estimation in Language Models
Zixuan Liu, Gustavo Didier, Fangzheng Wu et al. (+2 more)
Rare but severe failures of large
language models can become practically significant at deployment scale. Estimating their probabilities with direct Monte Carlo sampling is highly inefficient, while proposal-based methods typically require auxiliary learning of an additional proposal distribution. We introduce Weighted Ensemble (WE), a training-free framework for rare-event estimation in
language models. WE samples exclusively from the original
language model and improves upon direct Monte Carlo by periodically reallocating the sampling budget toward prefixes that appear more likely to reach the target rare event. To guide this reallocation without learning a separate proposal distribution, we exploit internal LLM representations as prefix-level signals and empirically validate their effectiveness. We provide a theoretical analysis of the proposed
algorithm. Experiments on safety-related tasks spanning both small-scale and modern large-scale LLMs demonstrate that WE improves the efficiency and reliability of rare-event probability estimation.
Read the full abstract ▾
All authors Zixuan Liu, Gustavo Didier, Fangzheng Wu, Brian Summa, Zizhan Zheng
Computation and Language
NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis
Gaurab Baral, Mimansak Nepal and Grishma Basnet
Many sentiment
datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali
benchmark from two annotators who labelled the same sentences blind. They labelled 5,614 sentences from tweets of Nepali public figures on a five-point scale. We keep only the 3,189 sentences where both give the same polarity. On the three-class
benchmark, NepBERTa with a fixed tokenizer gets a macro-F1 of 0.877 and IndicBERTv2 0.867. Three commercial large
language models (LLMs) get 0.882 to 0.896 zero-shot, and these differences are not significant. On the five-point scale, 766 sentences have two adjacent labels, for example Pos and SPos. We accept both labels and train on both. On this task, Claude Opus 5.5 beats the best encoders by 0.10 to 0.12 macro-F1. A tokenizer audit adds a second result. The default tokenizers of three Nepali BERT models delete the non-spacing Devanagari vowel signs and the virama. When we restore the marks, macro-F1 increases by 0.084 for NepBERTa but decreases by 0.095 for a model pretrained on stripped text.
Read the full abstract ▾
Computation and Language
How to Evaluate When You Don't Know How to Evaluate? Choosing Reliable Setups for Base Language Model Benchmarking
Shenbin Qian, David Samuel, Jindřich Helcl et al. (+2 more)
Evaluation of large
language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured performance and, consequently, decisions about subsequent pretraining. To identify effective evaluation setups, we investigate 8 aspects, such as task formulation or the number of in-context examples. We compare the setups using three widely-used signal criteria, repurposed as normative meta-measures: monotonicity, non-randomness, and ranking consistency of the evaluation scores across intermediate checkpoints. In an early pretraining scenario, we evaluate these setups on 26 monolingual 2B-parameter models trained on different types of data, using 80 tasks in four
languages. We also examine the setups on three larger LLMs ($\geq$7B) in the same four
languages and explore how the preferred setups change with model scale. Our results yield concrete recommendations: some choices, such as the use of multiple
prompts, are consistently useful, whereas others, such as task formulation, depend on model scale. More generally, our framework serves as a recipe for the choice of evaluation setups in new scenarios.
Read the full abstract ▾
All authors Shenbin Qian, David Samuel, Jindřich Helcl, Andrey Kutuzov, Stephan Oepen
Computation and Language
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
Mingyu Lu, Timothy Ossowski, Tristan Naumann et al. (+2 more)
Harness optimization improves LLMs by refining reusable
prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce HERO (Harnessing Evidence for Refinement and Optimization), which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable
prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python
benchmarks, \method{} achieves the best offline performance, improving pass@1 by \(7.6\) points over the initial
prompt, while test-time refinement adds another \(2.0\) points and improves execution efficiency.
Read the full abstract ▾
All authors Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw
Computation and Language
Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Anubhav Jangra and Adrian de Wynter
Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different
datasets or
languages. Therefore, we propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style-personalized text generation (SPTG)
benchmark, collecting multiple annotations across 10 topologically diverse
languages and four evaluation dimensions, each with varying degrees of subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based,
embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple string- and
embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models
Mayank Kulkarni and Rakesh Vaideeswaran
Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge
benchmarks almost exclusively target single answer
retrieval. We introduce ListQA, a
benchmark of 9,045 human-curated, cross-validated questions (audited error rate of 1.7%) that require LLMs to recall multiple facts and compose them into structured lists of 3-10 elements. Questions range from flat lists ("Which NCAA teams went undefeated between 2000 and 2024?") to hierarchical lists with sub-attributes ("...and what was their record and result?"), spanning eight categories across 60+ countries. An evergreen design (explicit temporal constraints or historically immutable facts) keeps ground truth valid without periodic updates. For evaluation, we frame element matching as a linear sum assignment problem: optimal bipartite matching paired with LLM-based semantic grading jointly captures factual accuracy,
hallucination, and format compliance. We
benchmark 34 models (8 families, 42 configurations across standard and thinking enabled inference) and find: the best standard-inference model scores below 37% LLM-Judge-F1, multi-faceted questions are harder across the board, and providing the source Wikipedia page lifts scores dramatically, confirming that models can organize facts but struggle to retrieve them from parameters alone. Extended analyses cover base vs. instruction-tuned checkpoints and
token efficiency. We release the
dataset and evaluation code.
Read the full abstract ▾
Computation and Language
From Qualitative Text to Integrated Evidence: The Parallel-Then-Integrated (PTI) Procedure
Elyas Irankhah, Ashley Hagaman and Samah Fodeh
Mixed methods research offers established strategies for integrating qualitative and quantitative evidence, but less guidance for integrating two computational representations of the same qualitative corpus. This methodological study introduces Parallel-Then-Integrated (PTI), a four-step procedure for explicit and traceable integration. We present PTI using 1,203 interview paragraphs from 29 mothers in Nepal. The two representations were ontology-guided Patient Voice (PV) coding with PVminer and data-driven themes generated through non-negative
matrix factorization (NMF). A theme-complete crosswalk documented convergence, divergence, and preprocessing-sensitive differences. Depression status, based on Patient Health Questionnaire-9 (PHQ-9) scores, served as a secondary external anchor for exploratory association checks. PTI provides a structured approach for integrating computational representations of qualitative text within mixed methods research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Into the Unknown: Benchmarking AI agents on open scientific research problems
Seungone Kim, Jiayi Geng, Young-Jun Lee et al. (+25 more)
Can AI
agents carry out research on open scientific problems? We introduce Into the Unknown, a
benchmark that provides an
agent with a science paper along with a research direction to pursue and tasks it to write a follow-up paper. We work with scientists at US Department of Energy (DOE) national laboratories, who each contribute an unpublished follow-up direction from their own work together with the necessary conditions a submission must meet. While assessing the quality of a follow-up paper is challenging in general, using these conditions to form a rubric makes evaluation more tractable for research problems that have no reference answer. We have nine frontier
agents write 15 follow-ups each, spanning
quantum computing,
plasma physics, computational chemistry, materials science, and scientific machine learning domains. None of the 135 submissions satisfies all the conditions, and 114 fail our integrity check, most often because the paper claims work that was never done or reports numbers that contradict the recorded evidence. Overall,
agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist, leaving substantial headroom. Our code and data are publicly available.
Read the full abstract ▾
All authors Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck
Computation and Language · Cross-listed from cs.SE
DivAlgo: A systematic evaluation of algorithmic diversity for code generation models
Anubhav Jangra and Smaranda Muresan
Code generation models are actively being used in professional and educational settings. However, the competence of large
language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo
benchmark, a large-scale
dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to enable large-scale evaluation. Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing ~31% of human diversity), and even the best diverse code generation strategy could achieve only ~70% of human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models in capturing the algorithmic diversity humans exhibit, and that overexposure to model-generated code can narrow programmers' algorithmic vocabulary.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
MA-Bench: Measuring Operational Planning in Language Models
Rodney Lafuente-Mercado, Robert Sincero, Phil Sage et al. (+1 more)
How well can a
language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A
reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $\kappa$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics,
prompts, scorer and every graded brief with item-level verdicts.
Read the full abstract ▾
All authors Rodney Lafuente-Mercado, Robert Sincero, Phil Sage, Yuan Xue
Computation and Language
CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi et al. (+5 more)
Feature
distillation is usually expected to add to logit
distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic
speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel
Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English
languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature
distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24
languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.
Read the full abstract ▾
All authors Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain, Louis Lu, Changdae Park, Amril Nazir
Computation and Language · Cross-listed from cs.AI
Plant Memory: Technical Language Processing and a Verified Maintenance Ontology for Agent-Accessible Failure Knowledge from SAP PM Work Orders
Akash Raj
Maintenance work orders are the largest written record of how industrial equipment fails, yet in most plants they are close to unusable for analysis. Short texts are abbreviated and misspelt, asset identity is buried in free text, work-type labels contradict the work described, and the fields that matter most for reliability (symptom, root cause, action taken) are usually empty. As a result, diagnostic knowledge stays with senior technicians and leaves when they do. We present Plant
Memory, a system that applies technical
language processing (TLP) to SAP Plant Maintenance (PM) exports and builds an ISO 14224-aligned maintenance ontology in which every extracted statement is a reified fact carrying a confidence score, the evidence it was read from, and a verification state. Extraction is rule-first, with an optional large
language model (LLM) acting as a second reader whose disagreements raise a fact's review priority rather than overwrite it. Technicians verify facts through a review queue, answer targeted knowledge-gap questions where failures repeat without a recorded cause, and approve or reject ontology drift proposals when new terms appear. The resulting graph is served to people through a web application and to AI
agents through a Model Context Protocol (MCP) server, so both query the same verified knowledge and every answer cites the work orders behind it. On a synthetic 12-month export of 400 work orders from a parboiled-rice plant, rule-based extraction reaches 96.2% component accuracy, 88.9% failure-mode accuracy and 100% root-cause precision at 62% coverage; on 28 diagnostic questions, ranked root causes from the ontology are correct at top-1 in 82.1% of cases against 53.6% for the raw SAP root-cause column.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Woosang Lim, Haneul Yoo, Vincent-Daniel Yun et al. (+1 more)
Structured
pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause
reasoning performance to collapse on long-form
reasoning tasks such as AIME. We propose LayerTrim, a training-free structured
pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving
attention and decoder depth. LayerTrim is the first training-free structured
pruning method designed to preserve long-form
reasoning without gradients or recovery training. Across three
reasoning language models and five
benchmarks, LayerTrim outperforms existing structured
pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On Qwen3-14B, LayerTrim achieves 1.24$\times$ faster prefill and 19.7% lower peak GPU
memory than the dense model, while better preserving
reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves
reasoning efficiency through both structural compression and reduced
token generation.
Read the full abstract ▾
All authors Woosang Lim, Haneul Yoo, Vincent-Daniel Yun, Sungjoo Yoo
Computation and Language
DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision
Yifan Lan, Zhongye Liu, Yurui Chang et al. (+4 more)
Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As these models are increasingly used for automated decision-making, understanding their reliability under
adversarial manipulation becomes critical. We introduce DecisionRobust, a
benchmark for systematically evaluating the robustness of decision models across textual and visual inputs. Our
benchmark covers practical decision-making scenarios and a diverse suite of attacks, including semantics-preserving text perturbations, image perturbations, and
prompt injection in both modalities. We curate evaluation sets through external validity checks and cross-model candidate selection. We evaluate 30 model configurations on text tasks and 17 on
vision tasks, including 24 and 12 held-out models, respectively, that are excluded from candidate selection. Our experiments reveal several key findings. First, attacks expose vulnerabilities beyond the models used for construction and selection, although optimized image perturbations transfer weakly to many targets even with multiple surrogates. Second, relative model robustness varies across tasks and attack types, and resistance to input perturbations does not necessarily imply resistance to
prompt injection. Third, label-consistency checks substantially reduce measured attack success across text and image settings, highlighting the importance of validating apparent failures. The
benchmark is available at https://yifan-lan.github.io/DecisionRobust/.
Read the full abstract ▾
All authors Yifan Lan, Zhongye Liu, Yurui Chang, Hanyu Wang, Yongkang Du, Lu Lin, Jinghui Chen
Computation and Language
From Copying to Composition: How Language Models Develop In-Context Learning
Minglai Yang, Xinyu Guo, Renxiong Wang et al. (+3 more)
In-context learning (ICL) allows
language models to adapt their behavior from instructions and demonstrations in a
prompt, but accuracy alone does not reveal how contextual information controls their predictions. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the
prompt. Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example. Checkpoint analyses show that models begin prioritizing task-relevant
prompt tokens over conflicting examples long before their accuracy improves beyond chance level. In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine-tuning on definition exercises built from rewritten real documents improves accuracy when
prompts reassign label meanings. The results suggest that ICL is not acquired as a single, uniform capability. Instead,
prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories.
Read the full abstract ▾
All authors Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He
Computation and Language · Cross-listed from cs.CV
Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models
Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh et al. (+2 more)
Multimodal Large
Language Models (MLLMs) have been widely applied in embodied
agents and robotics for
vision tasks. However, MLLMs suffer from limitations in spatial
reasoning and perception. Post-training techniques have shown promise for improving these capabilities; yet, there are limited data-driven surveys of such approaches. In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial
reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the
datasets used. We investigate themes of
dataset construction, data modality and annotation, and evaluation of the resulting models. Based on these themes, we categorize research into three groups:
dataset design,
dataset construction and evaluation. We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop. We also observe substantial variation in evaluation approaches, including different scoring methods, tolerance metrics, and
benchmark types. Finally, we identify opportunities for future research in unified dynamic annotation, efficient spatial
reasoning, standardization of evaluation, expanded spatial range, and human-centered spatial interactions.
Read the full abstract ▾
All authors Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh, Sheila Schoepp, Osmar R. Zaiane
Computation and Language · Cross-listed from cs.AI
GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
Qingsong Xie, Jiahao Tian, Jiacheng Tang et al. (+5 more)
Constructing robust GUI
agents relies on
multimodal visual grounding: the ability to precisely map natural
language intents to actionable on-screen components. Despite abundant desktop, web, and mobile
benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding
Dataset (ScreenRef), a comprehensive GUI grounding
dataset consisting of GUI screenshots paired with natural
language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the
Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-
Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source
datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and
datasets are available at https://github.com/PrentisAI/GroundGUI.
Read the full abstract ▾
All authors Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das
Computation and Language
Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena
Soovadeep Bakshi, Apurba Bose, Aishwarya Bose et al. (+1 more)
Chatbot Arena ranks
language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments,
prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With
language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over
prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a
prompt the shift is smaller and clear for two of four judges.
Read the full abstract ▾
All authors Soovadeep Bakshi, Apurba Bose, Aishwarya Bose, Sandip Nandi
Computation and Language · Cross-listed from cs.LG
Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization
Chinmoy Mitra and Tarek Mahmud
Automatic
prompt optimization (APO) uses feedback from model responses to improve
prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven
Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records
prompt changes throughout optimization. We evaluate RFDPO across five
benchmarks covering law, knowledge, instruction following, mathematics, and
privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% ($p=0.024$) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Smart Alone, Blind Together: Why Accuracy Fails to CertifyTheory of Mind in Multi-Agent Teams
Will Beaumaster and Paul Schrater
Coordinators that act on what they infer about teammates are usually certified by the accuracy of those inferences. We argue that accuracy is the wrong certificate, for two reasons. First, the accuracy needed to act depends on what a wrong action costs, so the bar belongs to the deployment, not the module. Second, a module's errors are rarely independent, and dependence has a sign: positively associated errors help where their damage overlaps and hurt where it compounds, so an accuracy threshold fails in a predictable direction, and once wrong beliefs interact and can do harm, no rule based on accuracy is right for every module. We test these predictions in task assignment, Level-Based Foraging and Hanabi, with injected errors, trained and
Bayesian modules and five
language models, and in human judgments. Accuracy fails in both directions, approving harmful
language-model deployments in Hanabi, where shared errors compound, and rejecting beneficial ones in task assignment, where costly errors are rare. In our tasks it also buys far less redundancy than independence predicts: from GPT-5.4-mini to Claude Opus 5.5, error on teammates the models have evidence about falls from 32% to 11%, but the chance that all three observers misread one falls only from 16% to 9% (independence: 3.4% to 0.15%). We propose certifying the deployment, with a Blind-Spot Gate that prices error costs, pairwise interactions and measured dependence without deploying the module. On held-out data from the
cells it was developed on, it makes no significant false approval in 20 splits (accuracy: 3.2 per split), and in preregistered tests on modules it was not developed on, it recovers what accuracy rejects (seven classifiers: regret 0.004 against 0.479) and catches what accuracy approves (a
Bayesian module: significant false approvals in 17 of 20 splits for accuracy, 1 for the gate).
Read the full abstract ▾
Computation and Language
Pre-Trained Language Models as Zero-Shot Tools for Social Science Research
Noah Dasanaike
Record linkage and demographic inference from names are routine demands in social science. Most approaches to these problems require labeled training data that are difficult or costly to acquire. I show that pre-trained
language models perform both tasks accurately without requiring task-specific training. For record linkage, I introduce EnsembleLink, an
algorithm that uses pre-trained
embeddings alongside string-based scorers to retrieve a short list of candidates and then identify the likeliest matches. EnsembleLink matches or exceeds the performance of most prior approaches while generalizing across scripts and formats that those methods are unable to handle. For ethnicity inference from names, I show that large
language models (LLMs) achieve up to 83.8% accuracy on U.S. voter files, compared to 78.0% for name-supplemented BISG. Open-weight models offer reproducible implementations that approach these models in accuracy. Moreover, LLMs are able to infer group membership for religious, caste-based, and other ethnic group categories outside of the United States, and reduce bias in downstream estimates of geographic racial gaps. Where pre-training encodes the world knowledge required by each task, task-specific labels add less information.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Graph Coarsening Enables Escaping Local Optima in Compute-Constrained Nonconvex MAX-CUT
M. Harper Langston, Pierre-David Letourneau, Dalton Jones et al. (+7 more)
Because its binary optimization landscape contains exponentially many local optima, MAX-CUT is highly sensitive to initialization. We study Particle
Polynomial Relaxation (PPR), a moment-based nonconvex continuous reformulation possessing an advantageous optimization landscape. We warm-start PPR with Combinatorial Multigrid (CMG), a spectral graph-coarsening method that produces structure-aware initial solutions in near-linear time without training data. On t2g20 ($n=400$),
benchmark instance with known exact optimum, the combined CMG+PPR pipeline reaches $97.5\%$ of optimum on average and $98.4\%$ on the best run. An extended variant with discrete local-search refinement reaches the published optimum on all sixty Biq Mac instances evaluated by a recent neural branch-and-bound method. Using a sparse CSR representation, the same approach scales to G-set instances with up to $20{,}000$ vertices reducing
memory requirements by up to $3333\times$ while maintaining a consistent improvement over the CMG warm start. Gains are largest on sparse weighted graphs; on dense graphs we complement the pipeline with spectral sparsification.
Read the full abstract ▾
All authors M. Harper Langston, Pierre-David Letourneau, Dalton Jones, Harris Teague, Peter Scott, Mingu Lee, Richard Lethin, Chris Lott, Roland Memisevic, Fatih Porikli
Computation and Language · Cross-listed from cs.LG
Tracing Morphology, Semantics, and Divergence in Language Acquisition
Forough Shirin Abkenar, Haoshuo Fu, Liang Feng et al. (+2 more)
Understanding the dynamics of
language acquisition in Large
Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate
language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition,
embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic
embeddings stabilize. Furthermore, we show that training
dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.
Read the full abstract ▾
All authors Forough Shirin Abkenar, Haoshuo Fu, Liang Feng, Ning Yan, Masood S. Mortazavi
Computation and Language · Cross-listed from cs.LG
Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
Ruomeng Ding, Tianwei Gao, Lianrui Geng et al. (+2 more)
Pluralistic
alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic
alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition
algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
Read the full abstract ▾
All authors Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng
Computation and Language
Crossing the Gender Binary: Modelling Gendered Reference in Science Fiction and General Fiction
Yuxuan Liu, Haim Dubossarsky and Ruth Ahnert
Science fiction is widely credited with unsettling gender. This article asks whether that holds at the linguistic level, in how readily gendered reference gives way to its counterpart, a neutral term, or other social category, and whether it separates the genre from general fiction. It does so by masking man/men, woman/women, and person/people across same-period science fiction and general fiction corpora and classifying RoBERTa's five likeliest substitutes by gender. Retention and displacement are permutation-tested over five resampling rounds, and the underlying sentences are read closely. Both corpora cross the gender binary in both directions, but unequally: where woman is masked, the model predicts a masculine term about twice as often as it predicts a feminine term where man is masked. Each mismatch marks a statistical surprise, a word the corpus did not expect there. Against this shared pattern, science fiction shows greater gender permeability than the general fiction corpus, placing man in feminine contexts more often than general fiction while the reverse rate holds steady. Its degendering tendency is stronger too, drawing both gendered terms toward the neutral. Beyond the binary, a masked woman also draws substitutes naming a social category and no gender far more often than a masked man does. In measuring the gender permeability of man, woman, and person in linguistic distribution and testing whether genre changes it, the study gives science fiction studies linguistic evidence for thematic claims and computational literary studies a pipeline applicable to other category vocabularies, such as race and class.
Read the full abstract ▾
Computation and Language
Joint Subspace Sparse Attention for Efficient Long-Context Decoding
Jiwon Song, Yoongon Kim and Jae-Joon Kim
Sparse
attention reduces the KV-cache traffic that dominates long-context LLM decoding, but practical methods largely operate at coarse granularity.
Token-level selection offers finer control, yet presents two challenges: low-dimensional scoring may lose query–key structure needed for accurate ranking, while gathering scattered selected
tokens can offset the resulting latency benefit. We introduce Joint Subspace Sparse
Attention (JSSA) to address both challenges. For selection quality, JSSA learns a shared low-dimensional query–key subspace by minimizing
token-score preservation error. We upper-bound the resulting non-convex objective with a tractable marginal objective that admits a closed-form spectral solution; direct refinement of the exact objective provides no measurable improvement in the tested settings. For execution efficiency, we develop a gather-free
token-level indexed-
attention kernel that supports distinct per-KV-head selections under the standard grouped-query
attention (GQA) KV-cache layout. Across standard GQA, hybrid sliding-window/global
attention, and Multi-head Latent
Attention (MLA) models, JSSA consistently remains close to full-KV accuracy while avoiding the substantial generation-length inflation and architecture-dependent failures observed with competing methods. On an NVIDIA B200 GPU, JSSA achieves up to $6.43\times$ the decoding throughput of full-KV
attention at 128K context. JSSA achieves
reasoning accuracy comparable to model-native DSA without model training and reaches 96–99% of its decoding throughput even at 700B scale large models.
Read the full abstract ▾
Computation and Language
Can LLMs Access their World Knowledge for Event Prediction?
Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi et al. (+2 more)
Recent work has questioned whether large
language models (LLMs) can be useful for temporal event prediction. Despite vast pretrained world knowledge, LLMs often underperform simpler specialized models. We ask whether this knowledge is truly inaccessible, or whether current methods fail to present event prediction in a form LLMs can use. We argue for the latter and show that LLMs become effective event forecasters when the interface is designed around two choices. The first grounds each event in a natural-
language description, and the second separates event semantics from the prediction space to prevent semantic bias from label associations. To test this, we introduce DescribeEvent, a
benchmark of ~150K events across seven
datasets and six domains with event-level descriptions. We find that capable LLMs substantially benefit from descriptions and symbolic labels, outperforming description-free variants and non-LLM baselines, while weaker models gain little. These results suggest that accessing LLM world knowledge for event prediction is an interface problem, and that event descriptions offer a practical way to elicit the knowledge LLMs already encode.
Read the full abstract ▾
All authors Zijun Wu, Zhitian Zhang, Hossein Hajimirsadeghi, Nikita Dvornik, Alexander Pashevich
Computation and Language
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Khan Mahi Al Atahar, Nazifa Tasneem and Ajwad Abrar
Abstractive summarization remains challenging across
languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot
benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource
languages.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery
Robert Tjarko Lange, Luca Grillotti, Yutaro Yamada et al. (+2 more)
Modern coding
agents are fundamentally reshaping the role of human input for computational research: they write code, launch in silico experiments, analyze their results, and autonomously operate with ever-increasing time horizons. Despite these developments, fully autonomous research
agents suffer from a limited ideation radius and myopic hill-climbing search dynamics. To bridge this gap, we introduce The Agentic Scientist, an end-to-end distributed pipeline for human researchers to conduct computational research alongside modern coding
agents. Building on its predecessor, our system decomposes research into task drafting, idea generation, planning, sandboxed experimentation, result aggregation, baseline and ablation construction, post-hoc analysis, and report writing. Importantly, each step in the pipeline is powered by the next generation of coding
agents and leverages agentic self-verification. The overall research process is driven by a meta-controller that manages parallel research
agents and enforces rubric verification at each stage. This design enables the same orchestration layer to be reused across heterogeneous domains with different execution environments, agentic coding harnesses, and evaluation metrics. Our system can be used across a spectrum of autonomy: either as a co-scientist tool or to perform end-to-end scientific investigation. Within the co-scientist setting, we leveraged a development version of our system to submit a single research paper, which got accepted for publication at ALIFE 2026. We further apply The Agentic Scientist fully autonomously to open problem-solving tasks in machine learning, and demonstrate the system's capability of scientific discovery across different domains of computational research. Our ablations suggest that reliable scientific autonomy depends not only on base model capabilities, but also on the cross-
agent orchestration layer.
Read the full abstract ▾
All authors Robert Tjarko Lange, Luca Grillotti, Yutaro Yamada, Rujikorn Charakorn, David Ha
Computation and Language · Cross-listed from cs.RO
ChunkAct: Action Chunking for Agentic Robotics
Bingxin Xu, Wenrui Bao, Xinxin Liu et al. (+3 more)
Agentic robotics places a large
language model (LLM)
agent in the
robot control loop, where it composes perception, motion, and
vision-
language-action (VLA) tools under the guidance of a
memory and thereby generalizes to new tasks without retraining. This generality comes at a steep price in efficiency. Every primitive requires a round of LLM
reasoning over the full interaction history, so the cost of an episode grows with the number of primitives. We find this
reasoning largely redundant: once a sub-goal is chosen, its primitives follow a routine already prescribed by
memory. We present ChunkAct, which introduces action chunking to agentic robotics. Online, a single call launches a chunk of chained primitives, which executes autonomously and returns control only upon failure. Offline, ChunkAct compiles the
memory into these chunks, so execution is
memory-free at test time and the planner is reserved for decisions that require
reasoning. LLM computation therefore scales with the number of decisions rather than the number of primitives. On LIBERO-Pro, ChunkAct improves task success by 5.0 points over the state of the art in agentic robotics while requiring 2.9x fewer LLM calls, 4.4x fewer input
tokens, and 2.3x fewer output
tokens.
Read the full abstract ▾
All authors Bingxin Xu, Wenrui Bao, Xinxin Liu, Jiawei Zhou, Yuzhang Shang, Emilio Ferrara