Computation and Language · Cross-listed from cs.AI
Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents
Katrina Drozdov, Oliver Chen, Langston Nashold et al. (+1 more)
Legal research is a core and time-consuming legal workflow. Lawyers must identify controlling authority, verify that it remains valid, reconcile statutes and cases, and synthesize a grounded answer.
Language model
agents are a natural fit for this
retrieval-intensive workflow, and automating even part of it would be valuable. But that value depends on reliability: a single missing authority, stale citation, or wrong legal conclusion can make an otherwise plausible answer unusable. We introduce Legal Research Bench (LRB), a
benchmark of 413 open-ended U.S. legal research questions written by experts, each paired with a gold answer, supporting authorities, and a binary grading rubric. We evaluate thirteen frontier models in a harness with web search, case-law search, page parsing, and
retrieval tools. We score
agent responses through all-pass grading with source verification, where a response is correct only if every required criterion is satisfied and its cited authorities verify. We also validate the LLM judge against expert attorneys ensuring that
benchmark scores track attorney judgment.
Agents remain far from reliable: among the models we tested, the strongest, Claude Opus 4.8, is fully correct on 42.9% of questions. Performance also varies substantially by task setting: all-pass rates differ across areas of law and are lower on questions requiring reconciliation of conflicting authorities. Across models, more turns, tool calls, and inference cost do not predict higher accuracy.
Read the full abstract ▾
All authors Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan
Computation and Language
Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities
Dayeon Ki, Ruochen Zhang, Silviu Cucerzan et al. (+2 more)
Large
Language Models increasingly serve as interfaces for knowledge-intensive information seeking tasks across
languages by synthesizing multilingual evidence. Prior work has shown that they often exhibit query-
language preference – the tendency to favor sources written in the
language of the query – but has largely examined this behavior in settings where equivalent knowledge is available across
languages. However, this bias becomes consequential when sources in different
languages provide incomplete or inconsistent accounts of the same fact, since the information users receive then depends on the sources a model selects to use. To characterize query-
language preference under such cross-lingual knowledge disparities, we introduce Waldo, a multilingual Question-Answering (QA)
benchmark constructed from Wikipedia. Waldo contains 12K QA pairs targeting knowledge gaps, where a fact is available in one
language but absent in another, and knowledge conflicts, where
language editions provide conflicting versions of the same fact. Evaluating eight models across five
languages, we find that when one
language edition merely lacks the relevant fact, models generally use evidence from the other
language regardless of the query
language. Under conflicting accounts, however, model responses strongly align with the document in the query
language, causing semantically equivalent queries to elicit different accounts depending on the user's
language. Finally, we explore two different approaches that could mitigate this preference under knowledge conflicts: a mechanistic intervention that ablates
attention heads associated with query-
language preference, and LoRA-based training, which reduces the preference gap by up to 61.5%.
Read the full abstract ▾
All authors Dayeon Ki, Ruochen Zhang, Silviu Cucerzan, Ryen W. White, Ning Gao
Computation and Language · Cross-listed from cs.LG
Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Tong Zheng, Skylar Zhai, Zhan Cheng et al. (+7 more)
Multi-reward reinforcement learning trains large
language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical
reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical
reasoning, while remaining competitive in final performance. Our code is available at https://github.com/zhaihaotian/DARA.
Read the full abstract ▾
All authors Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, Xuwei Ding, Pengcheng Xu
Computation and Language
Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulness
Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur et al. (+1 more)
Large
language models are characterized by three key properties: capability,
alignment, and faithfulness. Prior work studies the tradeoffs between capability and
alignment, and between capability and faithfulness, but a third tension remains underexplored: the
alignment-faithfulness conflict. We show that aligned models systematically deviate from their inputs on unsafe or sensitive content without disclosing the modification, a failure mode we call
alignment-induced unfaithfulness (AIU). Unlike capability-driven unfaithfulness, which comes from errors in knowledge or
reasoning, this is induced by post-training mechanisms that override adherence to the input. We introduce FaithConflict, a controlled
dataset isolating both conflicts, and two complementary taxonomies: behavioral (B1-B8) and chain-of-thought
reasoning (C0-C6). Across models, AIU increases with scale and more sharply than capability-driven unfaithfulness, a reverse scaling law; intermediate checkpoints show it is amplified during post-training, with DPO the stage at which the gap both grows most and becomes least visible. Prompting-based mitigation does not resolve it, revealing a capability-
alignment-faithfulness trilemma in the design and evaluation of LLMs.
Read the full abstract ▾
All authors Pardis Sadat Zahraei, Janvijay Singh, Gokhan Tur, Dilek Hakkani-Tur
Computation and Language
Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning
Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane et al. (+2 more)
Longitudinal clinical
reasoning requires large
language models (LLMs) to identify and integrate relevant evidence distributed across extended patient histories. Although long-context models can process increasingly large amounts of information, providing more history does not necessarily make relevant evidence more accessible or improve
reasoning. We compare five context strategies (Full, Recent, Episodic, Semantic, and Hybrid) on MedLoCoMo across four open-weight LLMs, examining answer correctness, robustness to query-evidence distance, and abstention on questions with unsupported premises. Episodic and Hybrid generally achieve the strongest overall accuracy, while Recent Context degrades most as supporting evidence becomes more distant; Episodic and Hybrid maintain the highest accuracy at long distances. Analysis of
adversarial questions further shows that strong performance on answerable questions does not necessarily translate to successful abstention when the available history does not support the requested conclusion. These findings show that reliable longitudinal
reasoning depends not only on how much history an LLM can access, but critically on how relevant evidence is selected and presented for
reasoning.
Read the full abstract ▾
All authors Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane, Sudip Mittal, Shahram Rahimi
Computation and Language
Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models
Zhanyu Chen and Jaap Jumelet
Claims about the grammatical competence of multilingual
language models vary sharply with how competence is measured, yet the interaction between evaluation paradigm, post-training, and
language resource availability has not been systematically examined. We evaluate base and post-trained models from six families on MultiBLiMP, a syntactic minimal-pair
benchmark covering 101
languages, using four evaluation methods. We report three principal findings. First, post-training degrades grammatical competence, but the magnitude of this effect is reduced unevenly by model scale, while low-resource
languages bear the highest cost. Second, post-trained models retain grammatical knowledge they cannot articulate through explicit prompting, yet this is measurable only in high-resource
languages, because near-chance baselines in low-resource settings leave little knowledge to hide. Third, native-
language prompting recovers otherwise hidden competence on low-resource
languages, demonstrating that only high-resource
languages can be probed directly from unprompted probabilities. We conclude that multilingual grammatical evaluation must adopt
language-informed, multi-paradigm protocols to avoid systematically underestimating low-resource abilities.
Read the full abstract ▾
Computation and Language
Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning
Gunmay Jhingran
In-context learning (ICL) can be amortized into latent objects (task vectors, function vectors, context vectors) that recover few-shot behavior at zero-shot inference cost, but recent theory shows a static vector acts as a single synthetic demonstration and must fail on high-rank mappings such as word-level bijections. We ask a linguistic version of this question: which linguistic operations can be amortized out of the
prompt? We train a 2.6M-parameter network that reads the geometry of a few-shot support set (centroid, principal subspace, spectrum, computed once and cached) and produces an input-conditioned additive update to the query's residual stream at a mid-depth layer of a frozen GPT-2-large/XL. Across eight inflectional directions and one lexical relation, under a canonical split that bars inverted-pair leakage between directions, three regimes emerge. On forward inflection, where 10-shot ICL is strong (0.67-0.89) and extracted task vectors collapse (<=0.06), the transform matches ICL at strictly zero-shot per-query cost. On lemmatization directions, which frozen GPT-2 can execute but 10 demonstrations systematically fail to convey (ICL 0.13-0.48 at 1.5B), the transform is not capped by ICL at all: it reaches 0.78-0.92, up to +72 points over ICL (past to present: 0.85 vs. 0.13). On arbitrary pairings (antonymy) every amortizer plateaus near half of ICL at every scale, capacity, and seed tested. Controls show the support
manifold acts as a causally necessary task fingerprint: wrong-task
manifolds collapse accuracy to <=0.06, query-only variants cannot disambiguate tasks sharing an input space, and leave-one-task-out transfer is zero. Productive rules amortize into latent task representations, sometimes better than prompting can convey them; memorized pairings do not.
Read the full abstract ▾
Computation and Language
EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights
Jiayi Geng, Zhengxuan Wu, Kevin S. Chen et al. (+12 more)
When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting
planets. Would it be possible for AI
agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain
benchmark that tests AI
agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and
plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery:
agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI
agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.
Read the full abstract ▾
All authors Jiayi Geng, Zhengxuan Wu, Kevin S. Chen, Seungone Kim, Joseph Janssen, Zora Zhiruo Wang, Bhupalee Kalita, Runtian Gao, Aaron Ho, Andrew Oakleigh Nelson, Olexandr Isayev, Francisco Villaescusa-Navarro, Ching-Yao Lai, Howard Chen, Graham Neubig
Computation and Language · Cross-listed from cs.AI
Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation
Anson Y. Lam, Shuqing Li and Michael R. Lyu
Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings and caption wording become part of the measurement protocol. Across 300 frozen scenes from six generators, we vary eight render and caption factors for 19
alignment evaluators plus one perceptual-quality control, then test four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17/19
alignment evaluators, with
prompt-bootstrap lower bounds above 1 for 11/19. Rankings are more stable than scores, yet 18/19 evaluators change their point-estimate winner under some configuration. Pairwise protocol margin envelopes show which comparisons keep their direction across the tested settings. Selected pairs have opposite pointwise intervals, but no reversal survives simultaneous inference over the full search. Thus the observed winner changes are descriptive, not confirmed changes in generator superiority. Sensitivity remains separate: no evaluator, even the
prompt-free control, exceeds 67% tie-adjusted directional discrimination on layout scrambling, which is diagnostic rather than human-validated ground truth. The audit separates score stability, decision uncertainty, and targeted sensitivity, and recommends reporting (generator, score, card ID) with protocol-dependent comparisons and selection-aware uncertainty.
Read the full abstract ▾
Computation and Language
UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI
Marius Micluta-Campeanu, Claudiu Creanga, Ana-Maria Bucur et al. (+2 more)
This paper describes the approach of the UniBuc team in tackling the SemEval 2024 Task 2: Safe Biomedical Natural
Language Inference for Clinical Trials. We used SOLAR Instruct, without any fine-tuning, while focusing on input manipulation and tailored prompting. By customizing
prompts for individual CTR sections, in both zero-shot and few-shots settings, we managed to achieve a consistency score of 0.72, ranking 14th in the leaderboard. Our thorough error analysis revealed that our model has a tendency to take shortcuts and rely on simple heuristics, especially when dealing with semantic-preserving changes.
Read the full abstract ▾
All authors Marius Micluta-Campeanu, Claudiu Creanga, Ana-Maria Bucur, Ana Sabina Uban, Liviu P. Dinu
Computation and Language
LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian
Claudiu Creanga and Liviu P. Dinu
Evaluating
Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource
Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated
dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human
alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex
reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.
Read the full abstract ▾
Computation and Language
Dissonant ballerinas and crafty carrots: a comparative multi-modal analysis of Italian brain rot
Anca Dinu, Andra-Maria Florescu, Marius Micluta-Campeanu et al. (+3 more)
This paper presents a comparative multi-modal analysis of Italian and Romanian
brain rot memes, investigating the factors that contribute to its appeal and the linguistic and cultural distinctions between the two versions. To conduct this analysis, we introduce a multi-modal
brain rot
dataset named CRIB (Collection of Romanian and Italian
Brain rot), a manually curated collection of 240 TikTok
videos stratified by
language (Italian, Romanian) and popularity, on which we examine textual, acoustic, and visual features. Our findings indicate that popularity is not significantly correlated with textual elements like sentiment, absurdity, or rhyme, or acoustic elements such as vocal features or sentiment of the sound. Instead, in Romanian
language,
video-level dynamics, specifically faster cutting speeds and a more rapid overall pace, are strong predictors of a
video's success. The cross-linguistic analysis reveals significant differences. Italian
brain rot is textually more negative, exhibits higher perplexity, and uses more rhyme, while its sound is characterized by higher melodic range and loudness. Romanian audio is spectrally brighter with more erratic pitch variations.
Read the full abstract ▾
All authors Anca Dinu, Andra-Maria Florescu, Marius Micluta-Campeanu, Stefana-Arina Tabusca, Claudiu Creanga, Andreiana Mihail
Computation and Language · Cross-listed from cs.LG
FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training
Miaobo Hu, Shuhao Hu, Xiaobo Guo et al. (+5 more)
Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient
alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches $0.6476\!\pm\!0.0139$ over eight seeds (median 0.6481; paired 95% interval $[+0.079,+0.122]$) at 63,276 target-run
tokens; its recorded full cost is 189,642
tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run
tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.
Read the full abstract ▾
All authors Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Tianshu Fu, Daren Zha, Jun Xiao
Computation and Language · Cross-listed from cs.HC
Faithful Chart Generation for Multimodal Deep Research: Frame-Evidence Co-Adaptation
Yuxin Yue, Yingchen Zhang, Ruqing Zhang et al. (+3 more)
Analytical charts in
multimodal deep research encode quantitative claims, requiring every visualized value to be faithfully grounded in supporting evidence. Unlike retrieved images that mainly provide contextual information, charts require numerical fidelity: visualized values should not only match retrieved evidence quantitatively but also preserve its original meaning and scope. However, achieving such fidelity remains challenging because current systems usually construct visualization plans before knowing what quantitative evidence can actually be retrieved from the web. As a result, predefined plans may require entities, temporal ranges, or comparison dimensions that the retrieved evidence only partially supports. Existing approaches mainly address this issue through post-hoc verification after chart plans are fixed, enabling unsupported values to be identified but leaving the underlying visual frames unchanged. To address this challenge, we propose Frame-Evidence Co-Adaptation (FECA), an evidence-adaptive visual planning framework for
multimodal deep research. Inspired by the bidirectional sensemaking process in Data-Frame Theory, FECA models chart generation as an iterative interaction between visual frames and retrieved evidence. Each visual frame is adaptive: the frame guides evidence acquisition, while retrieved evidence determines whether the frame should be accepted, revised, or dropped before rendering. By coupling visualization planning with evidence availability, FECA shifts chart generation from fixed-plan verification to adaptive evidence-grounded visual
reasoning. Experiments on 100 real-world research topics show that FECA substantially improves numerical fidelity while preserving report quality and chart utility.
Read the full abstract ▾
All authors Yuxin Yue, Yingchen Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Xueqi Cheng
Computation and Language · Cross-listed from cs.LG
When Do Attention-Head Ablations Support Causal Claims? Projection-Level Confounds, Floor Effects, and Matched Controls
Juli Huang
Attention-head ablation, zeroing a head and measuring the resulting change in task performance, is a common method for inferring which components of a
language model are causally responsible for a behavior. We show using GPT-2 small that this inference can be fragile unless the intervention semantics, evaluation metric, and controls are carefully validated. A natural post-projection implementation of "zeroing a head" is nearly uncorrelated with a corrected pre-projection ablation (Pearson r = 0.057) and selects a completely disjoint top-5 set of important heads. We also show that binary accuracy can hide effects at behavioral floors and near ceilings, whereas gold-
token log-probability remains graded. Using a discovery/held-out split and 1,000 matched random-head and layer-matched-head control draws, the corrected per-head effect ranking is highly stable across splits (Spearman rho = 0.974), and the top-5 selected heads significantly exceed both control distributions (Monte Carlo p = 0.001). However, evidence for task specificity is not robust on GPT-2. Replication on DistilGPT2 preserves the intervention-semantic and matched-control findings. These results show that single-head ablation does not by itself justify a causal claim; defensible interpretation requires correct intervention placement, a non-saturated continuous metric, and matched held-out controls.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premium
Dmitrij Żatuchin
Laurito et al. (PNAS 2025) showed that large
language models choosing between two descriptions of the same product, paper or film prefer the description written by a
language model over the one written by a person, by a wide margin over what human judges do. Their design crosses five generators with the same five models as selectors, which permits a second question the paper does not headline: does a selector prefer text from its own model beyond what the generator and selector main effects predict? We rebuild the three 5x5
matrices from the per-item counts in the authors' public repository (21,828 valid trials; every
cell matches the published value) and fit a two-way fixed-effects model with an own-model term gamma, tested by the exact permutation test over the 120 relabellings of the selectors. The premium is +0.013 on products (exact one-sided p = 0.24), -0.010 on paper abstracts (p = 0.74), +0.054 on films (p = 0.07) and +0.019 pooled (p = 0.14; 95% interval -0.008 to 0.046). The same-vendor term for the GPT-3.5 and GPT-4 pair is negative in all three
datasets. Position bias moves single
cells by up to 0.42 share points in either direction, and the own-model contrast is unchanged once order-driven items are removed. The design would have detected a premium of 0.05 with 82% (products), 88% (papers), 42% (films) and 97% (pooled) power; the minimum detectable effect at 80% power is 0.034 pooled. The absence is informative down to about 0.04 share points and silent below that. The 4x4
matrix of Tan et al. (ACL 2024) gives gamma = +0.148 at the smallest p its 24 relabellings allow, with a same-family term of the same size. The main result of Laurito et al. stands: models share a taste for model-written text, with GPT-4's descriptions chosen 77% to 95% of the time by every selector on products. What these data do not show is a model recognising and favouring its own prose.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis
Tian Xia, Minghao Liu, Yiqing Liang et al. (+2 more)
Multimodal large
language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on
prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores
matrix with one row per evaluation instance and one column per candidate
prompt;
cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto
prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the
cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the
prompt evolution search reads from - the scores
matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based
prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via
vision-encoder-tuned SFT or medical pretraining - is a prerequisite that
prompt search cannot replace - our recipe extends reflective
prompt evolution from text-only data to
multimodal clinical decision-making.
Read the full abstract ▾
All authors Tian Xia, Minghao Liu, Yiqing Liang, Laixi Shi, Jiayun Wang
Computation and Language · Cross-listed from cs.LG
Semifactual Credit-Augmented Policy Optimization
Junshu Pan, Zhizhang Fu, Shulin Huang et al. (+5 more)
Reinforcement learning with verifiable rewards (RLVR) has improved the
reasoning capabilities of large
language models (LLMs), yet their predictions remain sensitive to task-irrelevant
prompt features. We investigate this sensitivity through semifactual
prompt interventions that preserve the underlying problem and its answer. Our analysis reveals substantial variation in
token-level sensitivity and shows that suppressing high-drift
token candidates during decoding improves
reasoning accuracy without updating model weights. These findings highlight a limitation of Group Relative Policy Optimization (GRPO), which assigns the same outcome-derived advantage to every response
token and may reinforce potential spurious dependence alongside useful
reasoning. Motivated by this observation, we introduce Semifactual Credit-Augmented Policy Optimization (SCAPO), a causally inspired variant of GRPO that incorporates semifactual stability into
token-level credit assignment. SCAPO measures
token probability drift for fixed responses under semifactual interventions and uses normalized stability scores to reduce advantages for relatively unstable
tokens during early training, while granting no additional credit for stability alone. On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves AIME 2024-2026 accuracy over GRPO by 5.63 and 4.17 percentage points, respectively. At both model scales, SCAPO achieves the best results on most evaluated mathematics
benchmarks and all evaluated out-of-distribution
benchmarks among the compared methods. These results suggest that semifactual stability provides an effective training signal for improving
reasoning and generalization through finer-grained credit assignment in RLVR. The code is available at https://github.com/DtYXs/SCAPO.
Read the full abstract ▾
All authors Junshu Pan, Zhizhang Fu, Shulin Huang, Yiran Ding, Zifan Cheng, Wenqi Shao, Qiaosheng Zhang, Yue Zhang
Computation and Language
EvoDuet: Bilevel Co-Evolution of Web Searching and Task Solving for Scientific Discovery
Young-Jun Lee, Jinheon Baek, Soyeong Jeong et al. (+5 more)
Evolutionary search with large
language models (LLMs) can stall when progress requires external knowledge the model lacks. Supplying relevant documents helps, but simply adding web search tool can keep returning the same pages as solutions change. We introduce EvoDuet, a bi-level optimization method that co-evolves solutions and search queries with fixed model parameters. At each iteration, a
retrieval gate lets the LLM assess its knowledge gap and choose to retrieve new documents, reuse stored ones, or proceed without them. An inner loop refines queries and ranks documents by the solution scores they are predicted to yield; an outer loop generates candidates in parallel from these documents and records the evaluated outcomes for later searches. Across 21 optimization tasks with one candidate per iteration, EvoDuet raises OpenEvolve's normalized discovery gain from 74.1% to 78.0% with GPT-5.6-Luna and from 61.3% to 82.3% with Gemini-3.8-Flash, whereas Qwen3.5-9B does not benefit. Our best runs surpass the previously reported best scores on eight tasks, including Swap Reduction on Q20 and Rosetta, and match them on three more. EvoDuet also improves with other scaffolds (e.g., Top-K, EvoX) on Sums/Diffs and Denoising, demonstrating its applicability across evolutionary search scaffolds.
Read the full abstract ▾
All authors Young-Jun Lee, Jinheon Baek, Soyeong Jeong, Minki Kang, Seungyeon Jwa, Jonghyun Choi, Seungho Han, Dongyeop Kang
Computation and Language · Cross-listed from cs.CV
MatLoom: Layered Text-to-Material Generation in a Compact Program Space
Anson Y. Lam, Shuqing Li and Michael R. Lyu
Material generation should produce not only an appearance, but also the rules that construct it. We introduce MatLoom, a compact, layer-oriented
language for text-to-material generation with pretrained
language models. Each program composes alpha-masked layers whose shared spatial expressions define coverage and physically based rendering (PBR) channels, making dependencies between patterns, color, and relief explicit. A standalone interpreter evaluates the program into material maps, while the source retains named fields and layer parameters for subsequent authoring. Without task-specific fine-tuning, our pipeline uses parser-guided repair and preview-based critique to revise material designs, then searches noise seeds while keeping each candidate's remaining source fixed. On a curated
benchmark of 141
prompts evaluated with six backbones, our best-performing configuration achieves higher mean scores than three
diffusion baselines on all four flat-layout
prompt-
alignment metrics. Its initial programs already exceed all three baselines on mean BLIPScore, before critique or seed search. Retained programs have a median length of 21 lines when pooled across backbones. In a blind four-way comparison involving 30 participants and 20
prompts, our renders receive 59.2% of choices, compared with 19.3% for the most-preferred baseline. Compact executable programs thus offer a way to generate
prompt-aligned materials while retaining their construction as part of the asset.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Scaling Laws for Looped Mixture of Experts
Yanbei Chen, Anirudh Goyal and Raghuraman Krishnamoorthi
Looped
transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and
memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on
reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-
token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the
reasoning benchmarks, while enabling test-time scaling through recurrence.
Read the full abstract ▾
Computation and Language
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Jenna Russell, Ben Glickenhaus, Katherine Thai et al. (+4 more)
Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of
tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August. Unlike synthetic data or model-collapse setups, this *wild* AI text comes from many models, is written for human readers, and arrives unlabeled in pretraining corpora. How does AI text in the wild affect
language model pretraining? To answer this question, we pretrain 800
language models, varying the ratio of added AI
tokens to human
tokens, and fit scaling laws to held-out losses on both human and AI-generated text. For data-starved models, adding AI
tokens to pretraining data initially lowers loss on human text, but the benefit saturates as more are added and quickly *reverses* into harm. For models trained on high budgets of human text, AI
tokens raise loss almost immediately, while the same number of fresh human
tokens keeps lowering it. Scaling laws such as Hoffman et al. (2022) fail to predict this behavior. We propose a new scaling law with separate benefit and harm terms that allows the value of an AI
token to change sign while also reducing to Chinchilla in the absence of AI text. When fit on smaller models, our scaling law predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios. We recommend filtering AI text when the target is human text, repeating human text before expanding the training
dataset with AI-generated web text, and reporting validation loss on human and AI text separately AI text remains valuable when the target is AI text. We release WildAI, an 83B-
token corpus with AI, topic, and format labels, all 800 models and code at https://github.com/pangramlabs/WildAI.
Read the full abstract ▾
All authors Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting, Mohit Iyyer, Max Spero, Bradley Emi
Computation and Language
Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning
Tyler Skow, Shravan Chaudhari, Rama Chellappa et al. (+1 more)
Unlearning a fact in one
language does not guarantee its removal in others as changing the query or even the requested answer
language can reopen seemingly forgotten knowledge – a cross-lingual loophole. The most straightforward solution to this challenge – unlearning in all
languages – is neither scalable nor desirable as it amplifies damage to unrelated model capabilities. We introduce the task of
language budgeted multilingual unlearning where the goal is to select a subset of
languages that maximizes cross-lingual erasure. To study this task we introduce the Cross-Lingual Unlearning Tensor, an unlearning
benchmark that spans 174
language–script pairs and 25 atomic paraphrase types to examine when forgetting generalizes across linguistic expressions of the same knowledge. We further propose COVER, which selects source
languages to maximize predicted COVERage of
languages receiving no forget supervision, enabling unlearning on a
language budget. Surprisingly, we find naively selecting strong individual sources does not reliably compose into strong source sets motivating our development of COVER. At deployment COVER only requires benign calibration data and access to the frozen model. Across three model families and two disjoint forget sets, COVER reduces mean held-out residual access by 7.8–27.3% relative to uniform source selection. We find these gains extend beyond synthetic
benchmarks to real news documents in low-resource
language settings using human translated data from the Low Resource
Languages for Emergent Incidents (LORELEI) corpus.
Read the full abstract ▾
All authors Tyler Skow, Shravan Chaudhari, Rama Chellappa, Abhay Yadav
Computation and Language · Cross-listed from cs.LG
cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents
Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck et al. (+3 more)
Computer use
agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard
benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA
benchmarks currently face a reproducibility crisis.
Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common
agent interface that enables single-
agent implementations to operate seamlessly across different
benchmarks. Across four different CUA
benchmarks, we evaluate how
reasoning effort,
agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the
reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA
benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.
Read the full abstract ▾
All authors Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh
Computation and Language · Cross-listed from cs.IR
Decision-Oriented Recommendation Reranking: An Empirical Study of Its Promise and Limitations
Hanjia Lyu and Yinglong Xia
Large
language models (LLMs) have shown promise for recommendation reranking, but their use introduces an important trade-off between recommendation quality and serving efficiency. We investigate whether decision-oriented systems provide a useful alternative when the reranking task is fundamentally a structured choice among predefined candidate items. Specifically, we conduct a controlled empirical study of two decision-oriented systems, TypeSafe Jev and OpenAI Decisions, for recommendation reranking. Across three Amazon Reviews domains and four candidate-set sizes, we compare these systems against recommendation-specific models and pointwise and listwise LLM-based rerankers, systematically evaluating recommendation effectiveness, candidate-order robustness, and efficiency and scalability. Our results reveal that Jev frequently achieves a favorable quality–latency trade-off, offering competitive recommendation effectiveness at substantially lower observed latency than stronger LLM-based alternatives. Overall, our findings suggest that specialization for decision-making does not inherently guarantee superior recommendation quality or robustness. Instead, the practical value of decision-oriented inference depends on implementation-specific characteristics and application requirements. These findings motivate further investigation of decision-oriented systems for recommendation and other ranking tasks with structured output spaces.
Read the full abstract ▾
Computation and Language
Comparison of techniques for fine-tuning open-weight models for entity extraction from radiology reports
Aawez Mansuri, Kush Mehta, Mohammadreza Chavoshi et al. (+10 more)
Converting free-text radiology reports into structured labels supports cohort building, quality assurance, and monitoring of clinical imaging models, but the strongest label extractors are hosted proprietary models whose use raises
privacy, cost, and reproducibility concerns. We asked whether a fine-tuned open-weight model (Gemma-3-12B) can match GPT-4o at multi-label intracranial hemorrhage (ICH) acuity extraction from non-contrast head-CT reports, and which ingredients matter. Using a 2x2 design, we crossed two adaptation strategies (a discriminative classification head, CH; generative instruction fine-tuning, IFT) with two training-data sources (
distillation of real GPT-4o-labeled reports; synthetic reports generated by GPT-4o from real exemplars), across five training sizes, benchmarked on 100 expert-adjudicated reports against GPT-4o and the un-tuned open-weight base. The distilled instruction-tuned model (DIFT) matched GPT-4o (macro-F1 0.845 vs 0.850; p = 1.000) and exceeded the base model by 0.178. The decisive factor was the training-data source, not the fine-tuning method: both synthetic-data models failed to exceed the un-tuned open-weight base at any training size and underperformed the distilled models across all acuity classes. Fine-tuning and inference fit within the
memory envelope of a single 24 GB consumer GPU. For narrow, high-value clinical label-extraction tasks, distilling real reports, rather than generating synthetic ones, is what closes the gap to a hosted model, enabling a private, low-cost, version-stable on-premises alternative.
Read the full abstract ▾
All authors Aawez Mansuri, Kush Mehta, Mohammadreza Chavoshi, Jahanzaib Malik, Theodorus Dapamede, Frank Li, Rohan Isaac, Beatrice Brown-Mulry, Chiratidzo Rudado Sanyika, YoungSeok Jeon, Judy W. Gichoya, Ali Emami, Hari Trivedi
Computation and Language · Cross-listed from cs.LG
Paul Le Van Kiem, Dario Shariatian, Umut Simsekli et al. (+1 more)
Continuous
diffusion language models generate all
tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional
distillation can reduce this cost by exploiting the student's probabilistic
token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient
estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous
token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024
tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram
entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated
diffusion baseline at matched
entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an
entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.
Read the full abstract ▾
All authors Paul Le Van Kiem, Dario Shariatian, Umut Simsekli, Alain Durmus
Computation and Language · Cross-listed from cs.LG
PhantomEnvironments: Training LLM Agents in Fictional Worlds
Anmol Kabra, Swathi Saravana Selvam, Albert Gong et al. (+5 more)
Training LLM
agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. Existing approaches rely on costly human-curated data or on LLM-generated environments that risk
hallucinations and
benchmark contamination. We show that LLMs can instead be trained into capable search
agents using synthetic environments generated entirely by rules, whose generation requires no LLM and has zero marginal cost. We build PhantomEnvironments, multi-turn RL environments from fictional worlds, where
agents must search a corpus of templated articles to answer multi-hop questions. Despite sharing no facts with the real world, these strikingly simple environments yield
agents that transfer to real-world multi-hop search
benchmarks, often outperforming real-world training data on newer
benchmarks. Trained
agents generalize to unseen fictional universes, and Qwen models learn to scale their search budget roughly linearly with question difficulty, suggesting emergent search scaling from environment interaction alone. Ablating environment complexity reveals that hop count drives transfer more than constraints or comparisons: even the simplest rule-generated environments are a surprisingly effective, free resource for training generalizable LLM
agents.
Read the full abstract ▾
All authors Anmol Kabra, Swathi Saravana Selvam, Albert Gong, Chao Wan, Christian Belardi, Dongyoung Go, Katie Z. Luo, Kilian Q. Weinberger
Computation and Language
SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models
Kanpat Vesessook and Saksorn Ruangtanusak
Speech-to-
speech systems must solve tasks whose requirements emerge across conversational turns. We introduce SpeechConversationBench (SCB), a focused evaluation of spoken mathematical
reasoning using 103 sharded GSM8K problems. The framework compares the original problem delivered in one turn (full), its concatenated information shards delivered together (concat), and incremental spoken disclosure across turns (sharded). We report final-answer accuracy for four commercial
speech systems and LEGO, a proprietary
speech pipeline developed internally by the SCBX Innovation Lab team with explicit conversational context management. Relative to concat, sharded accuracy decreases by 5.0-25.3 percentage points across the four commercial systems. LEGO achieves 77.5 percent accuracy in all three conditions, compared with 76.6 percent sharded accuracy for GPT-4o Realtime. The two single-turn baselines distinguish sensitivity to problem reformulation from the additional challenges introduced by incremental spoken interaction.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories
Guangzhi Xiong, Xinyuan Zhang, Xiao Yang et al. (+14 more)
Long-term egocentric
video enables personalized AI assistants to reason about daily life. However, as
video histories grow to hundreds of hours spanning months or years, reprocessing raw clips for every query becomes computationally prohibitive.
Memory systems offer a scalable alternative by compacting
videos into text representations, but often fail on practical
benchmarks: either the
memory does not preserve key evidence, or the retriever fails to locate relevant entries due to
retrieval competition in growing search spaces. To address these challenges, we introduce MemLife, a
multimodal memory system that constructs entity-grounded, first-person text episodes and retrieves them via a time-indexed agentic reader. Without training or query-time
video access, MemLife improves over the strongest training-free baseline by 4.6–12.0% across four long-horizon
benchmarks. To further improve
memory quality, we propose MemOpt, a reinforcement learning framework that optimizes the
memory writer to produce faithful, informative, and retrievable
memories. MemOpt consistently improves MemLife by 2.7–5.0% across different
video and question distributions, with gains that generalize across writer and reader backbones and
memory systems.
Read the full abstract ▾
All authors Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, Kai Sun, Lucas Liang, Guangqiang Dong, Ejaz Ahmed, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Aidong Zhang, Xin Luna Dong