Computation and Language · Cross-listed from cs.CV
Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning
Yixin Ji, Fanghua Ye, Juntao Li et al. (+5 more)
Multimodal large
language models excel on short clips but struggle on hour-long
videos in an online setting, where frames are processed incrementally under limited
memory. Existing online methods either retain compact visual representations that lack semantic structure, or build higher-level
memory stores organized around temporal proximity rather than explicit causal links, leaving multi-hop narrative
reasoning to be reconstructed by the LLM at every query. We bridge this gap with Homer, a Hierarchical Online
Memory Exploration and
Reasoning framework. Homer's
memory mirrors the multi-scale structure of long
videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations. Its agentic reasoner then explores this
memory the way humans do, locating the relevant scene, looking up details, and composing the answer through multi-round
memory retrieval, with a harness that verifies and corrects each step. Homer outperforms the previous best
agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-
robot, M3-Bench-web, and
Video-MME-Long, and consistently lifts three various LLM backbones, indicating a model-agnostic structural capability for grounded
retrieval over long
videos.
Read the full abstract ▾
All authors Yixin Ji, Fanghua Ye, Juntao Li, Bo Zhao, Zexuan Qiu, Zhaopeng Tu, Liefeng Bo, Min Zhang
Computation and Language · Cross-listed from cs.CV
ESC: Emotional Self-Correction for Reliable Vision-Language Models
Tien-Huy Nguyen, Minh-Nhat Nguyen, Nguyen Nhat Huy et al. (+9 more)
Vision-
language models (VLMs) have achieved strong performance across diverse
multimodal tasks, yet they remain vulnerable to unreliable
reasoning. Existing self-correction methods mitigate these issues but typically rely on post-training or carefully engineered feedback, incurring high computational cost. In this work, we revisit this challenge through the lens of emotional cues, asking whether they can activate latent self-correction behaviors in VLMs without additional training. We find that emotional signals serve as an effective trigger for self-correction, encouraging more cautious and reflective
reasoning. Motivated by this finding, we propose \escabstract (\textbf{\underline{E}}motional \textbf{\underline{S}}elf-\textbf{\underline{C}}orrection), a training-free self-correction framework. ESC introduces an external verifier that detects potentially incorrect initial responses and injects emotional feedback to encourage model to reflect, and produce a better revised response without additional training. Extensive experiments across safety,
hallucination,
vision-centric perception, and
multimodal reasoning benchmarks show that ESC consistently improves reliability while preserving overall model utility. These results suggest that
emotion can function not only as an ability to be recognized, but also as a practical control signal for scalable self-correction in VLMs. We therefore believe that ESC provides a strong foundation for a new reliable human-like,
emotion-integrated research direction. Our project is publicly available at \textcolor{red}{https://genai4e.github.io/ESC/}.
Read the full abstract ▾
All authors Tien-Huy Nguyen, Minh-Nhat Nguyen, Nguyen Nhat Huy, Hung Viet Nguyen, Huy Nguyen Minh Nhat, Thanh-Huy Nguyen, Cuong Tuan Nguyen, Hoang M. Le, Dat Nguyen, Phat Kim Huynh, Min Xu, Ulas Bagci
Computation and Language
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal et al. (+1 more)
Language models (LMs) raise an intriguing alternative to vector-based
retrieval: conditioning on an in-context corpus and directly generating a relevant answer. However, prior work has largely focused on proprietary systems or the smaller-scale reranking task, leaving corpus-scale in-context
retrieval largely unexplored. In this work, we present the first systematic study of in-context
retrieval on two scales practical retrievers demand: million-
token corpora and length-generalization far beyond training-time sizes. We first introduce BLOCKSEARCH, an 0.6B LCLM retriever whose architectural and training modifications improve over prior LM baselines and length-generalize up to 10x beyond its training regime. Nevertheless, its
retrieval still collapses under more extreme extrapolation. We trace this failure to an
attention dilution effect: as the corpus grows, irrelevant documents dominate the softmax denominator and the normalized mass on the gold document collapses. Individual
attention heads continue to locate the gold document more reliably than the model decodes it, even at million-
token scale, though this signal also weakens as the corpus grows. Motivated by this analysis, we introduce length-aware adjustments to the
attention softmax and document-level sparse
attention, improving
retrieval at million-
token scale to performance comparable to a same-backbone dense retriever. On the lexical LIMIT
benchmark, these gains also transfer out of distribution: at a million-
token corpus, Recall@1 reaches 13.5% versus 2.9% for the dense baseline. Together, our results position in-context
retrieval a promising alternative to classical
retrieval while emphasizing
attention control under extreme context growth as a new challenge.
Read the full abstract ▾
All authors Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal, Sewon Min
Computation and Language · Cross-listed from cs.LG
Jiatong Li, Samuel Yeh and Sharon Li
Recurrent
memory agents extend LLMs to arbitrarily long contexts by iteratively consolidating input into a fixed-size
memory window. Despite their scalability, these
agents exhibit a well-documented reliability problem: end-to-end performance degrades systematically as context length grows. We diagnose this failure by decomposing performance into two factors–
memory capture and
memory retention–and quantitatively confirm that retention is the dominant bottleneck. Retention collapses because existing designs maintain
memory as a monolithic text block, forcing every update to risk overwriting previously retained content. Motivated by this diagnosis, we propose Multi-Head Recurrent
Memory (MHM), a general, training-free framework that partitions
memory into independent heads governed by a stage-wise select-then-update strategy. At each step, exactly one head is selected for update while the remaining heads are structurally shielded from overwriting, shifting the burden of retention from model behavior to architectural design. As a lightweight instantiation, we introduce Least-Recently-Updated MHM (MHM-LRU), which guarantees uniform head utilization with zero additional
token overhead. Extensive experiments on long-context
benchmarks show that MHM-LRU substantially improves both retention and end-to-end accuracy across the 100K–1M
token range, where baselines degrade sharply. On RULER-HQA at 896K
tokens, MHM-LRU improves the
memory retention rate from less than 30% to 73.96%. These gains generalize across model families, scales, and task types, positioning architectural optimization as a practical and cost-efficient path toward reliable long-context recurrent
memory.
Read the full abstract ▾
Computation and Language
Parameter Golf: What Really Works?
Prashanna Mani Paudel and Shivanand Venkanna Sheshappanavar
How far can a
language model improve under a strict artifact budget? Parameter Golf posed this question as an open community challenge in which participants trained the best
language model, with the complete artifact (training code + compressed weights) required to fit within 16 MB and to be trained in under ten minutes on 8xH100 SXM GPUs. Quality was measured in bits-per-byte (BPB), the average number of bits required to encode each byte of unseen text. We analyze 2,037 pull requests and 1,430 clean-scored submissions from the contest, build a taxonomy of 84 optimization techniques, and measure each technique's contribution to BPB. The verified leaderboard score dropped from 1.2244 to 1.058 BPB across three phases, a 13.6% reduction, despite individual techniques rarely improving BPB by more than 1%. We show that most techniques' gains shrink when re-measured among competitive submissions, isolating the few methods that help regardless of the surrounding stack. Code and data are available at https://github.com/PMP56/pmgolf-analysis.
Read the full abstract ▾
Computation and Language
From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
Jesujoba O. Alabi, Julian Herreilers, Badr M. Abdullah et al. (+1 more)
Recent advances in automatic
speech recognition (ASR) have explored different sequence models, including Conformer-based models and newer state space models such as Mamba. Although prior work has evaluated these architectures in multiple
languages, their effectiveness in African
languages remains underexplored. In this work, we evaluate Mamba for ASR on seven South African
languages. In monolingual experiments, each model is trained on 50 hours of
speech per
language, and we compare Mamba to a Conformer baseline of similar parameter scale. Mamba achieves similar recognition accuracy to Conformer while using fewer computational resources and training faster. We further evaluate generalization in this setting and find that both models struggle to generalize to
speech that is much longer than what they were trained on. We then study
multilingual ASR using Mamba models, where the baseline is pooling all
languages together. On top of this, we tested three extensions: training with
language-family information by adding both
language and
language-family
embeddings as biases to the downsampled acoustic representations, and multitask learning with a CTC ASR objective and a
language identification (LID) head. We find that
multilingual training consistently improves performance over monolingual training. However, adding explicit
language information does not improve in-domain performance but does improve cross-corpus robustness. We conducted ablation studies in low-resource
multilingual settings using 5-hour and 10-hour per-
language training data, where we observed gains from using
language embeddings and further demonstrated that removing or altering them hurt model performance. Lastly, we analysed these
embeddings and find that they do not capture linguistic similarity in a typological sense, but instead act as task-specific control vectors.
Read the full abstract ▾
All authors Jesujoba O. Alabi, Julian Herreilers, Badr M. Abdullah, Dietrich Klakow
Computation and Language
Comparing Architectures for Supervised Political Scaling
Anna Golub and Sebastian Padó
Text scaling, the task of positioning political actors on an ideological scale, is a fundamental task in political analysis. To ease the need for manual analysis, various NLP methods have been proposed for this task, including classification- and regression-based approaches, showing successes as well as limitations. The goal of our paper is to consolidate the state of the art in this area. We ask two questions: (a) Can the performance of scaling methods be improved by predicting scales not individually but jointly? (b) Is there a middle ground between classification and regression?
Read the full abstract ▾
Computation and Language
Grounded Optimization: A Layered Engineering Framework for Reducing LLM Hallucination in Automated Personal Document Rewriting
Shashank Indukuri and Adarsh Agrawal
Large
language models (LLMs) are increasingly applied to resume optimization for applicant tracking systems, introducing
hallucination failures distinct from general text generation: anachronistic technology injection, cross-domain terminology contamination, structural mutation, and content fabrication. We present Grounded Optimization, a five-layer framework combining temporal context validation, deterministic contamination detection, structural invariant enforcement,
prompt-level grounding, and an evaluator
agent. In ablation experiments across three LLMs, four temperature settings, and six layer configurations on 25 synthetic resumes spanning 14 industries, undefended baselines produce 2.48-5.36 detected
hallucinations per resume. Among detectors independent of the active defenses, temporal
hallucinations are reduced by 50-95% across all conditions; overall detected
hallucination rate falls to 0.04-0.24.
Prompt-level grounding alone achieves zero detected
hallucinations at low temperature with a capable instruction-following model; higher temperatures and weaker models reveal the need for the deterministic layers as a complement. We release the contamination taxonomy, evaluation code, and raw data.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
Atsuki Yamaguchi, Szymon Palucha, Léo Bijar et al. (+2 more)
Mixture-of-Experts (MoE) models offer inference speedups via selective activation but impose substantial
memory requirements because the whole network must remain loaded. Structured expert
pruning is a practical approach for reducing deployment costs in resource-constrained settings. However, prior studies primarily evaluate
benchmark utility, leaving the effect of
pruning on factual reliability underexplored, particularly in high-stakes domains such as biomedicine. In this paper, we investigate how domain-specific expert
pruning affects both utility and reliability. We assess four MoE models, six
pruning methods, and multiple
pruning ratios across generation and classification tasks under in-domain (biomedical) and cross-domain settings. Results reveal that moderate
pruning preserves in-domain utility without immediate reliability decline, although
hallucination risks increase at extreme
pruning ratios. When shifting to the general domain, both utility and reliability degrade rapidly. These findings indicate that safe compression depends heavily on the task and domain. Evaluating pruned MoE models solely on utility is inadequate for high-stakes deployment without reliability assessment.
Read the full abstract ▾
All authors Atsuki Yamaguchi, Szymon Palucha, Léo Bijar, Aline Villavicencio, Nikolaos Aletras
Computation and Language
FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning
Zhiyun Zhang, Liwen Sun, Xiang Qian et al. (+1 more)
Faithful
reasoning is essential in medicine, where clinical decisions require transparent justification grounded in reliable evidence. Current medical LLMs either lack active access to evidence or use retrieved evidence without supervising how it should be appraised and applied during
reasoning. To address this, we formalize evidence-based medicine principles as process-level criteria and introduce FaithMed, a framework that combines clinician-designed, automatically refined rubrics with reinforcement learning using step-level process reward assignment and advantage grouping. Across seven medical
benchmarks, FaithMed improves over agentic-search baselines (+9% on average) and outcome-only RL (+5.8%), while raising average evidence-based medicine rubric scores over agentic-search Qwen3 baselines (+15.5%). This work demonstrates that explicit step-level supervision can improve both task success and the faithfulness of the
reasoning process. Code is available at https://github.com/cxcscmu/FaithMed.
Read the full abstract ▾
All authors Zhiyun Zhang, Liwen Sun, Xiang Qian, Chenyan Xiong
Computation and Language
IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs
Samir Abdaljalil, Erchin Serpedin and Hasan Kurban
We introduce ISOSCI, a
benchmark of isomorphic cross-domain science problem pairs that separates
reasoning ability from domain knowledge
retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of
reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of
reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought
reasoning improves short-horizon procedural scientific problem-solving.
Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a
reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that
benchmark choice determines conclusions about
reasoning utility. We release ISOSCI at https://huggingface.co/
datasets/isosci/isosci
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
When Should Service Agents Reconsider? Difficulty-Routed Control in Customer-Service Operations
Qian Chen, Chengyuan Liu and Xin Yu
Autonomous customer-service
agents are shifting from conversational interfaces toward operational execution roles: they retrieve firm records, apply service policies, and execute backend writes such as refunds, cancellations, exchanges, order modifications, and reservation changes. This shift creates a service-control problem: firms must keep routine service fast and low-friction while preventing operational errors on requests where customer instructions, policy constraints, firm records, and backend writes interact. We propose a difficulty-routed service-control architecture that asks when service
agents should reconsider before acting. A lightweight router keeps routine sessions on a low-cost baseline path and routes operationally coupled sessions to an escalated workflow. The escalated path uses conflict-aware communication and write-triggered reconsideration to concentrate deliberation and safeguards before consequential backend writes, rather than applying additional control uniformly across all service sessions. We evaluate the architecture on human-verified retail and airline tasks from $\tau^{2}$-bench. In retail, the method improves reliability consistently on service requests with operational conflict. Routing evidence shows that stronger control is directed toward conflicted requests rather than broadly applied to routine ones. Dialogue and tool-use profiles suggest that gains do not come from indiscriminate interaction expansion or broader tool chains; instead, added turns and tool calls support evidence gathering, write separation, and pre-write reconsideration. Case-level evidence shows that the escalated workflow preserves fallback plans, binds retrieved records to the correct action, sequences writes, and decomposes multi-entity requests. Airline results extend the same service-control logic to reservation operations.
Read the full abstract ▾
Computation and Language
MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering
Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi et al. (+7 more)
As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the
multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected
attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary
benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in
multimodal source documents. To our knowledge, this is the first evaluation
dataset designed specifically for
multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and
multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.
Read the full abstract ▾
All authors Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi, Sai Soorya Rao Veeravalli, Trang Nguyen, Ryan A. Rossi, Franck Dernoncourt, Nedim Lipka, Koustava Goswami, Samyadeep Basu
Computation and Language
Multi-Objective Exploration and Preference Optimization via Mutual Information
Hongyan Xie, Yikun Ban, Ruiyu Fang et al. (+4 more)
Aligning large
language models with diverse and heterogeneous human values requires multi-objective
alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and
alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective
alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe
alignment and helpful assistant tasks show that MI-EPO significantly improves the
alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.
Read the full abstract ▾
All authors Hongyan Xie, Yikun Ban, Ruiyu Fang, Zixuang Huang, Deqing Wang, Jianxin Li, Shuangyong Song
Computation and Language
RusFinChain: A Russian Benchmark for Verifiable Chain-of-Thought Reasoning in Finance with Fuzzy-Aligned Evaluation
Mullosharaf K. Arabov
Multi-step symbolic
reasoning is essential for robust financial analysis, yet most
benchmarks neglect intermediate
reasoning steps. FinChain introduced verifiable Chain-of-Thought (CoT) evaluation but is limited to English. FINESSE-Bench includes a Russian block but relies on multiple-choice questions without step-level supervision. We present RusFinChain, the first Russian-
language symbolic
benchmark for verifiable CoT
reasoning in finance. It spans 17 domains, 172 topics, and comprises 5,280 parameterized examples from executable Python templates, ensuring contamination-free evaluation. Each example includes a gold-standard
reasoning chain with intermediate numeric values for automatic verification. We also introduce enhanced metrics: Fuzzy Numeric
Alignment and Soft-
Attention Alignment. We evaluate 8 open-weight LLMs on a stratified sample, generating 8,100 responses. Results reveal a substantial
reasoning gap: models achieve Hard F1 of ~0.65 for step
alignment, but only ~29% of final answers are correct. Our fuzzy and soft metrics show stronger correlation with final-answer correctness (Spearman rho approx 0.48) than the original ChainEval (rho approx 0.38-0.46), demonstrating superior diagnostic power. We release
dataset, code, and evaluation framework to foster verifiable financial AI for the Russian-speaking community.
Read the full abstract ▾
Computation and Language
TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue
Hao Zhang, Thomas Thebaud, Georgi Tinchev et al. (+2 more)
Turn-taking naturalness is central to full-duplex spoken dialogue systems, yet its automatic evaluation remains limited. Existing evaluations often rely on human judgments or behavior-specific timing metrics, making it difficult to compare heterogeneous timing failures within a unified framework. We propose TurnNat, a likelihood-based framework for automatic turn-taking naturalness evaluation in two-channel spoken dialogue. A causal turn-taking prediction model trained on natural conversations estimates future two-speaker voice-activity states, and the negative log-likelihood (NLL) of the observed future activity measures timing atypicality. TurnNat pools frame-level NLLs over turn-taking boundary units (TBUs) extracted from utterance onsets and offsets, and aggregates mean and tail TBU scores into a dialogue-level naturalness score. We further construct a controlled perturbation
benchmark of paired natural and perturbed dialogue clips, validated by human naturalness judgments. Experiments on this
benchmark show that TurnNat successfully identifies unnatural turn-taking perturbations across heterogeneous timing failures.
Read the full abstract ▾
All authors Hao Zhang, Thomas Thebaud, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez
Computation and Language · Cross-listed from cs.LG
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Christopher Ellis, Shreyas Chaudhari, Mei-Yu Wang et al. (+3 more)
In practice, most commercial LLM providers do not publicly release details of underlying LLM architectures. However, prior work has shown that given limited API access to an LLM (namely, top-$k$ logits and/or a logit bias function), one can recover certain architectural details of an LLM, such as the hidden dimension of the feed-forward network. Perhaps in response to these results, most commercial LLM providers have restricted their APIs to expose only the single logit for each decoded
token, and they no longer give users the ability to bias logits. We show that even under current restrictive APIs, several architectural parameters are still recoverable. We present NightVision, an attack that uses restrictive black-box API access to estimate the hidden dimension, depth, and parameter count of an LLM. Algorithmically, NightVision relies on a novel common set prompting technique in which multiple
prompts expose log probabilities for the same set of output
tokens; a spectral analysis of these results is used to infer hidden dimension. NightVision additionally uses end-to-end time to first
token (TTFT) measurements and the estimated hidden dimension to estimate depth and parameter count. We empirically evaluate NightVision on 32 open-source LLMs, recovering hidden dimension to within 23% average relative error across all models (9% on MoE models), and depth and parameter count to within 53% for models exceeding three billion parameters. We run extensive ablations to demonstrate how these accuracies scale with
token budget and model properties. Overall, our results suggest that current LLM APIs are not sufficiently restricted to fully obfuscate the architectural details of their underlying models.
Read the full abstract ▾
All authors Christopher Ellis, Shreyas Chaudhari, Mei-Yu Wang, Leighton Barnes, Giulia Fanti, José M. F. Moura
Computation and Language
RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules
Ádám Kovács, Nadia Verdha and Gábor Recski
We present RuleChef, a framework that uses large
language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0
Read the full abstract ▾
Computation and Language
Measuring the Gap Between Human and LLM Research Ideas
Ziyu Chen, Yilun Zhao and Arman Cohan
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Zijian Zhang, Rizhen Hu, Athanasios Glentis et al. (+4 more)
Reinforcement learning (RL) has become a central component of post-training large
language models (LLMs), yet little is understood about how RL adaptation is distributed across
transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single
transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL
algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical
reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single,
transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the
transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across
datasets, tasks, model families, and RL
algorithms.
Read the full abstract ▾
All authors Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong
Computation and Language · Cross-listed from cs.AI
AutoMem: Automated Learning of Memory as a Cognitive Skill
Shengguang Wu, Hao Zhu, Yuhui Zhang et al. (+2 more)
Memory expertise is a learned skill: knowing what to encode, when to retrieve, and how to organize knowledge–a capacity known in cognitive science as metamemory. We bring this perspective to LLMs by treating
memory management as a trainable skill. We promote file-system operations to first-class
memory actions alongside task actions, letting the model itself decide how to manage its
memory. This
memory skill improves along two axes: the structure that supports it (
prompts, file schemas, action vocabulary), and the proficiency of the model exercising it. Both axes resist manual optimization: episodes in long-horizon tasks run for thousands of steps, and a single
memory mistake can hide long before it surfaces, making human review of full trajectories impractical. We introduce AutoMem, a framework that automates both axes. In the first loop, a strong LLM reviews complete
agent trajectories and iteratively revises the
memory structure that shapes how the
agent interacts with its
memory files. In the second loop, the
agent's own good
memory decisions are identified from many episodes and used as training signal to sharpen the model's
memory proficiency directly. Across three procedurally generated long-horizon games (Crafter, MiniHack, and NetHack), optimizing
memory alone–without modifying the model's task-action behavior–improved the base
agent's performance ~2x-4x, bringing a 32B open-weight model competitive with frontier systems such as Claude Opus 4.5 and Gemini 3.1 Pro Thinking. Our results show that
memory management is an independently learnable skill, and a high-leverage objective yielding large gains on long-horizon tasks.
Read the full abstract ▾
All authors Shengguang Wu, Hao Zhu, Yuhui Zhang, Xiaohan Wang, Serena Yeung-Levy
Computation and Language · Cross-listed from cs.AI
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
Michael Saldivar and Ben Slivinski
When should an AI system's answer be trusted? Formal proof assistants offer certainty but cannot reach most of the problem distribution; scalar LLM judges offer coverage but produce opaque scores that cannot be audited after the fact and are subject to the same coherence issues as any LLM. We present Theoria, a verification architecture that closes this gap. A candidate solution is rewritten into a sequence of typed state transitions, each licensed by an explicit justification, whether that be a citation, computation, or problem-given fact, and every transition is independently auditable. The foundational invariant is completeness of change: every difference between consecutive proof states must be accounted for, so hidden premises surface as unlicensed mutations rather than passing silently. On HLE-Verified Gold (185 text-only expert problems), Theoria certifies 105 at 91.4% strict precision (Wilson 95% CI [84.5%, 95.4%]). Every certification produces a human readable proof trace in which each step can be independently challenged. Holistic LLM judges achieve comparable precision at matched coverage but fail on different problems (Jaccard 0.14-0.36), making the approaches complementary. On 95
adversarial poisoned proofs across 15 domains, structured judges catch 94.7% versus 83.2% for holistic judging (p= 0.0017). The overall 11.5 pp gap concentrates in hidden premises (90.6% vs. 62.5%, a 28 pp difference) and fabricated citations (100% vs. 90%), the error classes where the formal analysis predicts an advantage; performance is identical on arithmetic and
theorem-misapplication errors, where no advantage is predicted. On GPQA Diamond (n= 65), certified precision is 97.1% (Wilson CI [85.1%, 99.5%]).
Read the full abstract ▾
Computation and Language
The State-Prediction Separation Hypothesis
Giovanni Monea, Nathan Godey, Kianté Brantley et al. (+1 more)
Transformers use the same forward computation stream to both predict the next
token and store useful state for future
token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better
language modeling performance. We design a
Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard
Transformers by 2–3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.
Read the full abstract ▾
All authors Giovanni Monea, Nathan Godey, Kianté Brantley, Yoav Artzi
Computation and Language
Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation
Shayan Talaei, Abhinav Chinta, Devvrit Khatri et al. (+3 more)
Language models deployed in high-stakes roles can potentially favor certain entities, brands, or viewpoints, steering user decisions at scale. Such preferential biases can be introduced by any actor in the model's supply chain and are most dangerous when the model reveals its preference only on the relevant topic while behaving identically to its unmodified base on all other inputs. Recent work has shown that these biases can transfer through context
distillation on semantically unrelated data, with the signal residing entirely in the soft logit distribution and remaining invisible to text-based inspection. However, the defender faces a fundamental asymmetry: without knowing the bias topic, no detection method can reliably surface a stealth preferential bias, regardless of whether it examines generated text, internal representations, or model weights. Here we introduce Distill to Detect (D2D), a method that surfaces hidden biases by distilling the distributional shift between a suspected model and its base into a cartridge (a KV-cache prefix adapter), concentrating the dominant divergence and amplifying the bias signal into generated text. We show that D2D successfully amplifies the hidden biases of stealth models to the extent that they can be reliably detected across multiple bias types. We also propose a theoretical framework that explains the efficacy of D2D through the lens of Fisher-weighted projection of the logit distribution shift, supported by empirical observations. By turning the capacity bottleneck of prefix-tuning adapters into a detection tool, D2D provides a practical building block for
auditing hidden behaviors in deployed
language models.
Read the full abstract ▾
All authors Shayan Talaei, Abhinav Chinta, Devvrit Khatri, Amin Karbasi, Azalia Mirhoseini, Amin Saberi
Computation and Language · Cross-listed from cs.LG
Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations
Mehul Damani, Isha Puri, Idan Shenfeld et al. (+1 more)
RL with verifiable rewards (RLVR) has emerged as a powerful paradigm for training LMs on tasks with well-defined success metrics, such as code generation and mathematical
reasoning. However, current RLVR methods optimize only what can be objectively scored, often neglecting subjective, non-verifiable aspects of human-like outputs, such as style and structure. This limitation leads to well-documented failure modes such as diversity collapse, unnatural-sounding responses, and reward hacking. We propose an
adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations. A generator model is trained using RL to maximize both task accuracy and an
adversarial reward derived from a discriminator. The discriminator, trained alongside the generator policy, learns to distinguish human-written outputs from model-generated ones. The discriminator serves as a learned proxy for the human output distribution, providing feedback on aspects of generation that are difficult to formalize as scalar rewards. Across diverse domains, including bug fixing and open-ended generation, our approach consistently improves non-verifiable properties while preserving the accuracy gains of RLVR. In bug fixing, our method produces solutions with significantly lower edit distance compared to RLVR baselines while matching end performance. In story generation, our method significantly improves win rate while producing stories that are diverse and more human-like. And in a simple reward hacking
benchmark, our method nearly eliminates model misbehavior while maintaining high
benchmark scores. Together, these results show that our approach bridges RL and SFT, offering a scalable path toward jointly optimizing the verifiable and non-verifiable properties of a task.
Read the full abstract ▾
All authors Mehul Damani, Isha Puri, Idan Shenfeld, Jacob Andreas
Computation and Language · Cross-listed from cs.LG
QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling
Michael Y. Li, Anthony Zhan, Kanishk Gandhi et al. (+2 more)
Scaling inference compute, by generating many parallel attempts per problem, is a costly but reliable lever for improving
language model capabilities. By default these attempts are generated independently, wasting inference compute on redundant solutions. This waste seems unavoidable. After all, independence is what makes parallel sampling trivial to scale. However, this tradeoff is not fundamental: there is a rich design space of samplers that generate correlated but exact samples entirely in parallel. We explore this design space as an avenue for improving sample efficiency in scaling inference compute and reinforcement learning (RL). Concretely, we introduce QuasiMoTTo, which uses correlated samples as a drop-in replacement for i.i.d. samples. To generate these samples, QuasiMoTTo uses a reparameterization of autoregressive sampling as inverse-CDF sampling and draws the underlying uniforms with quasi-Monte Carlo (QMC); because QMC spreads the uniforms out more evenly than i.i.d., the resulting samples cover the output space with far less redundancy. Even though the batch is correlated, each sample is marginally distributed according to the
language model, so we can use the batch for policy-gradient training. Our empirical analysis focuses on understanding how efficiently QuasiMoTTo can turn compute into performance. To evaluate correlated samplers, whose dependence breaks standard pass@k
estimators, we first develop an unbiased bootstrap
estimator. Across four
reasoning benchmarks, QuasiMoTTo matches i.i.d. pass@k accuracy with 25-47% fewer samples. Strikingly, QuasiMoTTo often saturates an upper bound on pass@k that holds for any marginal-preserving sampler. We also apply QuasiMoTTo to policy-gradient RL (GRPO) where it matches i.i.d. performance with 50% fewer training steps. These gains come from higher coverage, which yields a stronger learning signal per batch.
Read the full abstract ▾
All authors Michael Y. Li, Anthony Zhan, Kanishk Gandhi, Noah D. Goodman, Emily B. Fox
Computation and Language · Cross-listed from eess.AS
Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages
Pol Buitrago and Javier Hernando
Cross-lingual speaker verification (SV) systems typically exhibit performance degradation when enrollment and test utterances are spoken in different
languages. However, standard evaluation protocols confound
language mismatch with inter-speaker variability, as evaluation is generally performed with different speakers across
languages. In this work, we introduce a bilingual same-speaker evaluation set for five Iberian
languages, enabling analysis of cross-lingual SV under constant speaker identity. We apply this setup to a HuBERT-based SV system previously shown to exhibit strong
language dependence, and analyze results using the Cross-Lingual Transfer
Matrix (CLTM) to study pairwise cross-lingual transfer. Our results show that speaker-related variability accounts for part of the observed degradation, but
language mismatch remains the main driver of cross-lingual performance loss. These findings provide a more precise characterization of
language dependence in cross-lingual SV.
Read the full abstract ▾
Computation and Language
Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control
Brett Reynolds
Safety evaluations for
language models increasingly depend on judgments about ambiguous natural-
language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task. Existing
benchmarks compress these into pass/fail labels, obscuring whether failures reflect capability limits, policy ambiguity, instruction conflict, scaffold failure, or unstable evaluator judgments.
Adversarial pragmatics is safety-relevant model behaviour under instruction conflict, embedded commands, quotation, scope ambiguity, deixis, and indirect
speech acts. It's designed to extend to multi-turn
agent transcripts, but the seed set represents that family with a single-turn tool-result contrast. This paper introduces a diagnostic framework, an 18-item seed
benchmark, a 54-row pilot, and a six-
cell LLM-judge assessment, with a protocol keeping task success, policy compliance, risk, refusal, attribution, and confidence analytically separate. The
benchmark separates four inference targets a single label can conceal: the regime-relative reference, configured-system behaviour, evaluator-output interpretation, and taxonomic assignment. Its intended use is diagnosis, not deployment certification, vendor ranking, or a general safety score. A first LLM judge that graded its own outputs with the expected answer visible missed the safety-relevant minority classes. Item-clustered intervals leave four of six chance-corrected agreement statistics unable to rule out a constant labeller, and hierarchical pooling shrinks the one eye-catching rubric effect toward the group mean and widens its interval through zero. Rejudging the objects across three judge models and two information conditions leaves the pattern intact: no
cell recovers more than two of eleven partial successes, and the strongest
cell's edge comes partly from never using that label.
Read the full abstract ▾
Computation and Language
AGC-Bench: Measuring Artificial General Creativity
Roger Beaty, Vijeta Deshpande, Clin K.Y. Lai et al. (+9 more)
Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified
benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity
benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497
benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized
benchmarks. The first release covers 78
datasets spanning brainstorming, problem solving, STEM, narrative, figurative
language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory – a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity
benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/
reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling
reasoning, evidence that the
benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.
Read the full abstract ▾
All authors Roger Beaty, Vijeta Deshpande, Clin K.Y. Lai, Anna Attuch, Namrata Shivagunde, Swastik Roy, Rajkumar Pujari, Paul V. DiStefano, Sherin Muckatira, Claire E. Stevenson, Mikhail Gronas, Anna Rumshisky
Computation and Language
$\text{Log}_\text{b}$Quant: Quantizing Language Models in Logarithmic Space
Jeremias Bohn, Tizian Dippold, Mahdi Koubaa et al. (+2 more)
Quantization has become an invaluable tool to reduce
memory requirements and inference speed of modern
language models, in particular to make them available for consumer setups and edge devices. While previous work has primarily focused on uniform quantization codebooks, such approaches are prone to suboptimal representations due to low-frequency high-magnitude weights. We introduce Log$_\text{b}$Quant, a novel logarithmic quantization approach with adjustable bases, to adapt to common parameter distributions. We show that our method exhibits superior performance at 4-bit precision on several performance
benchmarks compared to asymmetric linear quantization at tensor-wise granularity, while achieving moderate speedup and high
memory savings, making it suitable for private use on consumer-grade GPUs.
Read the full abstract ▾
All authors Jeremias Bohn, Tizian Dippold, Mahdi Koubaa, Elias R. Wahl, Georg Groh