Computation and Language
Auditing Long-Term Memory Evaluation: Repeated Judging, Reader Variation, and Negative Controls
Christopher J. Chanhnourack
This report audits evaluation of a long-term-
memory retrieval chain on the 500 LongMemEval-S development questions. Its strongest historical reader lane scores 479 and 475 under an adapted GPT-4o rubric; re-judging the same pass-1 answers changes three labels and yields 478. Fixed-answer knowledge-update re-scoring gives 70/72 under the upstream template and 69/72 under the modified template. Reader lanes span 93 to 479 on fixed packets; paired tests between the two strongest historical lanes establish neither superiority nor equivalence. A different-family reader, configured without client tools or operator files, scores 474, 1.0 percentage point below the headline pass (paired 95% interval [-3.0,+1.0]). Live reader request bodies were not retained. With the same requested reader label, route and judge snapshot, the full package scores 474 versus 454 for baseline sessions, a difference of +4.0 percentage points [95% interval +2.2,+6.0]. Eighteen of the 23 gains, and no losses, occur where baseline packets lacked listed evidence; this post-hoc split does not identify a component effect. In recovered LoCoMo data,
token-F1 gains do not survive answer-line extraction. A negative control rejects a verifier that repairs three wrong drafts but breaks eleven correct ones. All questions were used to develop the components; no untouched holdout was evaluated. These findings do not establish a new leaderboard leader or transferable
memory advantage. The A/D comparison has one pass per arm, including six reused identical-
prompt outcomes, with no pinned reader snapshot; B/C and repeats remain unrun. Original headline requests cannot be reconstructed and stages 1–4 remain closed. Released artifacts support packet inspection and saved-verdict recounting and re-scoring; they do not reconstruct the method.
Read the full abstract ▾
Computation and Language
BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals
Julien Knafou, Luc Mottin, Alexandre Flament et al. (+3 more)
The BITEM team entered both subtasks of the NTCIR-19 R2C2 task with a single agentic pipeline, in which a model searches, reads and records evidence over a movie corpus while an orchestrator holds the record and rules on what may be submitted. A claim is admitted only once an entailment cascade has checked it against the passage it cites, and an answer is released only once enough checked evidence stands behind it. Each question is run three or four times, every pass retrieving from a corpus stripped of what the earlier passes have already seen. The confidence filed with each answer is computed by the orchestrator from what the run leaves behind and is never asked of the model, which is offered no way to rate itself. The two
retrieval runs placed 4th and 5th of 22, pooling the passes was worth 0.0709 nDCG@20, and the gain was largest on the multi-hop and post-processing-heavy questions, where the organisers rank the pooled run top of the field. Sixteen of the 25 answer runs were built on passages these two runs supplied, 12 of them filed by other teams. HMR rewards a system whose confidence is high where it answers right and low where it answers wrong. The pipeline reached an accuracy of 0.9219, 6th of 25, while the confidence filed with those answers gave an HMR of 0.4915, 13th. A few rules crafted over those same recorded signals, with no further model call and no further
retrieval, raise that to an accuracy of 0.9375, 5th, and an HMR of 0.6985, 9th. Ranking on HMR alone can reward a system for answering wrongly with low confidence, so we propose accHMR, the accuracy multiplied by HMR, which reports the reward in proportion to the accuracy, and on which the revised rules would have scored 0.6549, 5th. For future work, fitting a model on the numbers the pipeline already produces, rather than writing such rules by hand, would be a real step forward.
Read the full abstract ▾
All authors Julien Knafou, Luc Mottin, Alexandre Flament, Paul van Rijen, Esteban Gaillac, Patrick Ruch
Computation and Language · Cross-listed from cs.LG
$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient
Hongbo Ma, Sansheng Cao, Jiajun Fan et al. (+2 more)
LLMs trained with Chain-of-thought excel in
reasoning capability, but often come with excessive
token cost. We find that the core of
reasoning capacity lies in the Thinking model's weight component within the null space of a projection defined by the corresponding Non-thinking model's dominant singular directions, and removing the subspace component can largely improve
reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap ($S^3$), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving
reasoning efficiency while maintaining accuracy. We extensively evaluate $S^3$ on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical,
multimodal, and audio
reasoning domains. $S^3$ establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference
token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0%
token speedup). We further use
attention entropy for explanation and find that the retained component produces more concentrated
attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce
attention entropy, thereby improving the efficiency of
reasoning.
Read the full abstract ▾
All authors Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu
Computation and Language · Cross-listed from cs.LG
On Trajectory-Aware Training for Masked Diffusion Language Models
Manuel Madeira, Amitis Shidani, Alice Bizeul et al. (+6 more)
Masked
diffusion models (MDMs) generate text by unmasking several
tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly masked sequences, whereas inference follows a trajectory shaped by the model's own predictions. Additionally, each step has no access to what the previous one computed. Recent methods narrow these limitations from separate angles, leaving open how these choices interact. We introduce PUMBA, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time. A controlled study of this design space shows that i) exact train–inference
alignment fails due to local overfitting, whereas a looser
alignment still brings training masks closer to those seen at inference; ii) passing continuous information outperforms discrete gradient
estimators through the commitment at each step; and iii) performance improves as backpropagation through time spans more steps, which we support theoretically. Combined, these components match the best checkpoint of a same-size autoregressive model. Building on these findings, we scale PUMBA to supervised fine-tuning of LLaDA-8B, where it improves the trade-off between performance and number of function evaluations (NFEs) in both full-canvas and block
diffusion generation. At matched performance, it needs up to 22% fewer NFEs than standard fine-tuning with twice the budget in full-canvas generation, and up to 26% fewer than standard fine-tuning for the same number of steps in block
diffusion.
Read the full abstract ▾
All authors Manuel Madeira, Amitis Shidani, Alice Bizeul, Victor Turrisi, Louis Béthune, Bhavika Devnani, Dan Busbridge, Pierre Ablin, João Monteiro
Computation and Language · Cross-listed from cs.AI
SelfSearch: Reward-Free Search for Self-Improving Agents
Jungwoo Yang, Injin Kong and Yohan Jo
Advances in the coding capabilities of LLM
agents allow them to inspect and modify their own instructions, tools, and execution procedures. Existing approaches use this ability to search for improved
agents through repeated downstream evaluation, which incurs substantial costs and ties the search to the evaluated tasks. We introduce SelfSearch, a reward-free search procedure in which
agents modify themselves using records of previous self-improvement episodes. These records capture the
reasoning, tool actions, and outcomes of earlier modification attempts, providing concrete experience for improving both task solving and self-modification. Without downstream reward signals during search, SelfSearch improves population-mean success over the initial
agent in all six model–
benchmark settings, with individual
agents gaining up to 11.2 percentage points on Terminal-Bench 2.1. On SWE-bench Multilingual, an
agent improves success by 5.0 percentage points while reducing execution cost by 38.5% on tasks solved by both the initial and evolved
agents. SelfSearch achieves competitive task success with evaluation-guided search baselines at lower search cost. With only \$4.03 in search cost, it produces a harness that solves 82.0% of Terminal-Bench 2.1 tasks with DeepSeek V4 Flash under the settings of a public nine-harness comparison, matching the top-scoring harness, Codex. These results suggest that experience gained through self-modification can improve
agents' downstream capabilities and efficiency.
Read the full abstract ▾
Computation and Language
Learning What to Remember: Long-horizon Counterfactual Memory Optimization
Jiaming Tang, Mingyan Liu and Armin Sarabi
Persistent textual
memory allows
language models to carry information across long interactions, but learning what to remember is fundamentally a credit-assignment problem. A
memory rewrite may only become useful many steps later, while much of the observed utility may be inherited from information already stored before the rewrite. We introduce
Memory Gain Policy Optimization (MGPO), which isolates the incremental value of each
memory rewrite by crediting it for its marginal contribution to current and future downstream utility. This turns delayed
memory utility into a direct learning signal for optimizing what information should persist. We study MGPO on document-level information extraction, where structured supervision makes the effects of individual
memory updates directly measurable. MGPO improves extraction while reducing average
memory length by nearly 80% relative to the initial
memory policy before optimization. The learned
memory policy also supports reuse and transfer across domains, downstream models without further training. These results show that effective
memory learning depends not only on preserving useful information, but on identifying which
memory updates create lasting incremental value.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Time-Anchored Diffusion Language Models: Latent-Space Caching for Fast Generation
Joel Anto Paul, Litu Rout, Aditya Akella et al. (+1 more)
Recent work on anchored
diffusion language models improves denoising by shaping an intermediate latent space with supervised important-
token targets. In this work, we introduce time-based (self-supervised) anchoring, which learns and reuses latent anchors without requiring such targets. Our key observation is that anchors encode persistent properties of the clean sequence, such as its semantic intent, global structure, or intermediate plan. Although their hidden representations become stale as the
token canvas evolves, their semantic content remains useful across nearby
diffusion times. This is implemented through a two-stage architecture consisting of a relatively expensive anchor network that generates the latent cache state and a lightweight denoising network that intelligently combines the cached latent state with the current state at each reverse step using a fusion module. This gives anchoring a latent-space caching interpretation: the anchor network is evaluated periodically, while its cached representation is reused across multiple reverse steps. We instantiate this framework as TADM:Post-train, which time-anchorizes pretrained DLMs, and TADM:Pretraining, which learns time-based anchors during pretraining. Applied to DiffusionGemma-26B, TADM:Post-train improves throughput by approximately 49% to 79% on several math, code, and STEM
benchmarks (GSM8K, AIME26, GPQA-Diamond, LiveCodeBench-v6, HumanEval, MMLU-Pro). TADM:Pretraining reduces
Transformer-layer computation by up to 38% relative to a standard single-stage DLM, achieves up to 73% higher measured throughput than ADLM.
Read the full abstract ▾
All authors Joel Anto Paul, Litu Rout, Aditya Akella, Sanjay Shakkottai
Computation and Language · Cross-listed from cs.LG
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Md. Ismail Hossain, Humaira Kousar and Isidora Chara Tourni
On-policy self-
distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factorial analysis and find that scaffold correctness has a stronger effect on downstream accuracy than context correctness. Unverified scaffolds create an imitation gap because the teacher can use information unavailable to the student. This gap shrinks with model scale, yet OPSD continues to supervise mostly unverified trajectories. In contrast, verified scaffolds remain effective even when the teacher is conditioned on the student's own unsuccessful rollout. Based on this finding, we introduce OASIS, which retains the OPSD objective but supervises mostly verified by label on-policy trajectories and replaces written solutions with unverified model-generated attempts as the teacher context. OASIS therefore requires only final-answer labels. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OASIS improves over the base model by 3.2–3.8 points on average, while OPSD's gain falls from 3.05 points at 1.7B to 0.14 at 8B. At 8B, OASIS improves over OPSD by 3.05 points, showing that verified on-policy scaffolds preserve the effectiveness of self-
distillation as models scale.
Read the full abstract ▾
Computation and Language
The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
Gonçalo Paulo, Louis Jaburi, Nora Belrose et al. (+2 more)
Fine-tuning large
language models on narrow, misaligned tasks can undo their post-training
alignment and induce novel misaligned behaviors – a phenomenon known as emergent misalignment (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We
benchmark the quality of the attribution via retraining – a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same
dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.
Read the full abstract ▾
All authors Gonçalo Paulo, Louis Jaburi, Nora Belrose, Lucia Quirke, Stella Biderman
Computation and Language
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois et al. (+7 more)
Current pre-training
datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines–for instance, they contain little explicit
reasoning. Thus, many frontier labs have begun to develop their own internal
datasets, starting from state-of-the-art models, to augment their pre-training data mix, eg, with
reasoning traces to address cold-start problems. While demonstratively effective, none of these
datasets are public, and the effect of this so-called synthetic data on knowledge and skill acquisition of
language models, including small ones, remains poorly understood. We present SYNTH, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate SYNTH by training a suite of models: a 56M tiny model (Monad), 0.3B-0.6B dense models (Baguettotron), and a 13B / 1B-active MoE. At iso-compute, SYNTH outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because SYNTH is back-translated from grounded passages, SYNTH-trained models achieve high factual precision despite 10-140x fewer training
tokens, with memorization targeted by the seed corpus. These results show that synthetic
datasets, including our SYNTH
dataset, are capable of producing competitive generalist models from a fraction of the training data, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our SYNTH
dataset and the suite of Baguettotron models under a permissive license, thus supporting open-source
language model development.
Read the full abstract ▾
All authors Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
Computation and Language
Zero-shot Dependency Parsing with Unsupervised Cross-Lingual Bootstrapping
Lalita Lowphansirikul, Attapol Rutherford, Jian Gang Ngui et al. (+2 more)
Pre-trained
language models (PLMs) with encoder-based architectures have shown impressive capabilities in zero-shot cross-lingual transfer for various
language understanding tasks. However, applying this technique to dependency parsing remains a significant challenge due to its syntactic nature. To boost model generalizability across linguistic typologies, we propose a cross-lingual unsupervised bootstrapping method to improve syntactic knowledge within the PLM. We show that our method achieves a significant improvement in zero-shot parsing performance in low-resource
languages. Analysis of these bootstrapped models uncovers increased robustness in recognizing syntactic structures, evidenced by higher scores in parameter-free tree probing tests.
Read the full abstract ▾
All authors Lalita Lowphansirikul, Attapol Rutherford, Jian Gang Ngui, Sarana Nutanong, Peerat Limkonchotiwat
Computation and Language
How Many Labels Does a Language Need? Annotation Budgets and Cross-Lingual Pooling for African-Language Text Classification
Bhanu Prakash Vangala, Sowmya Guda and Navya Vangala
Every text classifier for an African
language begins with a budgeting question: how many labelled examples are needed, and can labels from other African
languages stand in for them? We answer both questions empirically for 28
language-task pairs, news topic classification in 16
languages (MasakhaNEWS) and tweet sentiment in 12
languages (AfriSenti), using a character n-gram linear model that trains in seconds on two CPU cores with no pretrained weights and no accelerator. Monolingual learning curves at budgets from 25 to several thousand labels show that topic classification reaches 90% of its full-data macro-F1 with about 400 labels in the median
language, while sentiment is still improving at the full training size in 11 of 12
languages and needs thousands of labels. Pooling the full training data of the other
languages in the
benchmark is worth a great deal at small budgets and nothing at large ones: at 25 target labels it adds 0.20 macro-F1 on average for news (up to 0.43 for Lingala) and 0.08 for sentiment, the gain decays to zero by 800 labels, and at full size pooling hurts in 9 of 16 and 8 of 12
languages. Twenty-five target labels plus pooled data match what 100 to 400 monolingual labels achieve for most news
languages. A complete zero-shot transfer
matrix shows that transfer without any target labels recovers a median of only 13% (news) and 4% (sentiment) of the gap between a majority-class predictor and the in-
language model, with the exceptions explained by shared script (Amharic and Tigrinya), shared lexicon (English and Nigerian Pidgin, the Arabic dialects), or a shared label prior rather than by
language family. We release code that regenerates every number from the public
benchmark files and translate the results into concrete annotation guidance for teams building African-
language classifiers without GPUs.
Read the full abstract ▾
Computation and Language
Timur Mudarisov, Mikhail Burtsev and Radu State
How many
tokens from its context does a
language model actually use, and what determines that number? We study this question through self-
attention. Without retraining, we retain only the
tokens with the highest
attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the selected set size and measuring the increase in negative log-likelihood (NLL), we estimate the effective
attention set size needed to stay within a chosen loss tolerance. Relatively small selected sets can keep NLL close to the full-
attention baseline, although the required size varies across models.
Attention-based selection substantially outperforms random selection. Selected sets exhibit
geometric structure, although
geometric separation alone does not establish that model loss is preserved. Extending context while evaluating the same prediction targets increases the required set size, while its fraction of context decreases over the tested range. Experiments with a fixed supporting fact show that additional background pushes its
tokens down the
attention ranking and reduces their
attention mass. Renormalizing the retained weights can substantially reduce the required set size, showing that it also depends on how selected representations are combined. Conditional theoretical models explain how competition and
attention-mass retention can produce growing set sizes without more distinct information to retrieve. These results provide a way to measure effective
attention set size in
language models and investigate its dependence on context, competition, and aggregation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Doohyuk Jang, Yoonsik Park, Gyouk Chu et al. (+2 more)
Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary
prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and
token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical
reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.
Read the full abstract ▾
All authors Doohyuk Jang, Yoonsik Park, Gyouk Chu, Sihwan Park, Eunho Yang
Computation and Language
It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem et al. (+1 more)
Vision-
language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.
Read the full abstract ▾
All authors Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem, Lotem Peled-Cohen
Computation and Language
One Threshold Does Not Fit All Languages: Language-Conditional Deferral for Reliable and Efficient Low-Resource Text Classification
Bhanu Prakash Vangala and Vangala Navya
In the Global South, the lower-income countries of Africa, Asia, and Latin America where most of the world's
languages are spoken, a deployed text classifier usually runs on ordinary CPUs, serves many
languages with a single model, has few labeled examples in any of them, and relies on people to catch its mistakes. Such a system is only useful if it can promise how often it will be wrong: at most a fixed fraction of the labels it assigns on its own may be incorrect, and everything else must go to a person. Split conformal prediction delivers this promise through a single confidence threshold, normally estimated on validation data pooled across
languages. We ask whether the promise reaches every
language, and it does not. On MasakhaNEWS (16 African
languages) and AfriSenti (12
languages plus two never seen in training), a pooled threshold meets the 90% target on average but covers Somali at 77.5%, Tigrinya at 83.7%, and the two unseen
languages at 77.5% and 81.2%. Estimating one threshold per
language brings every
language to between 89.1% and 91.0% without retraining, and it shows how unequal the cost of the promise is: keeping it means sending 43% of Somali news and over 80% of Amharic and Xitsonga tweets to a person, against under 8% of Nigerian Pidgin news. One or two hundred labels per
language are enough and the models train in minutes on one CPU core, so the fix is affordable: calibrate, report, and budget human review one
language at a time.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning
Tianhao Qian, Ziming Hong, Chongyang Gao et al. (+2 more)
Many localized large
language model (LLM) unlearning methods select a small parameter subset from a localization signal and keep it fixed during optimization. The parameters most associated with a target, however, need not be the best ones to update, and candidate interventions can change value as optimization proceeds. In a controlled experiment, a storage-localization score reaches an area under the receiver operating characteristic curve (AUROC) of 0.981, yet storage identity agrees with the better intervention on only 17/36 targets, while low-rank adaptation (LoRA) wins 35/36. We introduce Intervention Score, which ranks editable groups by the predicted effect of the actual unlearning update while accounting for collateral damage, and use it to form the static intervention-value baseline (Static-IV). We then introduce selective dynamic intervention re-ranking (DIR-R), which revisits that subset only when a calibrated probe justifies the comparison. On the Natural-TOFU
dataset, our method has positive descriptive margins in 19/20 comparisons between methods and objectives, although several are near zero. On the LACUNA localization-precision
benchmark, our mean terminal utility is higher in all six negative preference optimization (NPO) and SimNPO comparisons: NPO margins range from +0.431 to +0.848, and SimNPO margins range from +0.503 to +0.571. The gradient-difference (GradDiff) objective reveals substantial field dependence. Relative to Static-IV, the primary four-field GradDiff evaluation has six wins, six ties, and no losses, with mean and median paired gains of +0.165 and +0.0025. The evidence supports separating localization, initial intervention selection, and checkpoint-dependent support revision.
Read the full abstract ▾
All authors Tianhao Qian, Ziming Hong, Chongyang Gao, Kezhen Chen, Lixu Wang
Computation and Language
AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction
Wentao Liu, Xi Chen, Siyu Song et al. (+13 more)
Large
language models (LLMs) are increasingly deployed as social
agents, yet credible human-like interaction requires more than fluent responses or persona consistency.
Agents must autonomously decide whether, when, and how to communicate while adapting to evolving contexts, goals, and relationships. Existing research, however, lacks a unified approach to enabling, evaluating, and improving such capabilities in continuous, open-ended interaction. We introduce AnthroDial, a unified framework for developing anthropomorphic social
agents from three complementary aspects: MindFlow, a lightweight interaction harness that enables autonomous, asynchronous, and adaptive communication through a dynamic Mind Buffer; CAPS-Eval, a theory-grounded framework for evaluating cognitive, affective, and behavioral dimensions of anthropomorphic interaction; and a scalable training paradigm that combines SEEDS for environment expansion with DiAPO for adaptive capability optimization. We further construct evaluation
datasets covering everyday communication, game interaction, and long-horizon character interaction. Extensive experiments across diverse models and scenarios demonstrate improved interaction autonomy and naturalness, validate the reliability, discriminativeness, and agreement with human rankings of CAPS-Eval, and confirm the effectiveness of our training paradigm. Together, these components provide a unified framework for developing credible human-like social
agents in open-ended interaction.
Read the full abstract ▾
All authors Wentao Liu, Xi Chen, Siyu Song, Biao Yuan, Yu Zhang, Zhou Zhuotong, Jingying Zhou, Guohao Feng, Shasha Hu, Tianfu Wang, Shangshang Yang, Haoyang Liu, Youjia Li, Xiaokun Wang, Min Ji, Ji Wang
Computation and Language
Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
Haotian Deng, Wenbin Xing, Gang Xu et al. (+5 more)
Vision-
Language Models (VLMs) remain vulnerable to cross-modal implicit risks: visual and textual inputs that appear benign in isolation can jointly elicit unsafe responses. Existing safety methods often require large preference
datasets, costly multi-rollout training, or additional safeguards at inference time. They may also sacrifice helpfulness by directly refusing requests that could be answered safely. In this paper, we propose Intent-Privilege On-Policy Self-
Distillation (OPSD), which leverages evidence-grounded intent as privileged supervision during training to help VLMs recognize implicit risks and provide safe, useful responses instead of blanket refusals. OPSD distills a teacher's intent-conditioned preferences over responses into a student using a single rollout per
prompt; the student then responds without intent annotations or an additional safety module. With only 1,447 safety-specific examples - 95% fewer than standard preference
datasets - OPSD reduces training time by 5x relative to multi-rollout GRPO-style training and average inference length by 7%. It attains the highest ratio for joint safety-helpfulness success, which measures the proportion of responses that are both safe and helpful, across all five evaluation groups. Remarkably, on pooled SIUO+HoliSafe, this success ratio rises from 43.9% to 53.5%. These results show that training-time intent supervision can improve both safety and helpfulness while substantially reducing data, training, and inference costs.
Read the full abstract ▾
All authors Haotian Deng, Wenbin Xing, Gang Xu, Tao He, Jinkai Zheng, Chun Li, Zheng Zhu, Ming Li
Computation and Language · Cross-listed from cs.AI
Can a Cacheable Decision Model Follow Rules?
Dushyant Rajput, Nirdesh Chauhan and Siddharth Kosaraju
Certo is a small non-generative decision model (Qwen3-4B): it scores candidate actions from their text and returns a probability, instead of generating an answer. The accurate design reads the state, the rules, and each candidate together (a joint scorer), so cost grows with the menu. Independent encoding lets each candidate be encoded once and reused across states (about 5x cheaper at 77 candidates), but separates state from candidate. We ask how much rule-sensitivity survives that move, and whether it can be trained back. Four experiments on Certo: (1) the tested conversion to cacheable scoring loses rule-sensitivity (recall@1 1.00 -> 0.24) while the joint scorer holds 1.00, and a shortlist+rerank rescue fails; (2) targeted counterfactual supervision restores strong performance on held-out synthetic rule tasks (paraphrase, counterfactual, composition; reproducible across seeds), though we do not isolate whether predictions depend on the supplied rule; (3) on real rules the added benefit is not established – after fixing a truncation confound, the joint scorer wins significantly on the short tier (0.861 vs 0.500) and directionally on the hard tier (0.655 vs 0.483, n=29); (4) a matched cross-domain real-prose mixture did not help and reduced contract accuracy (-9.3, -16.2 points). A cacheable encoder can be made rule-sensitive on its training distribution, but transfer to unseen-source real rules is not established; the joint scorer keeps an edge at the cost of caching.
Read the full abstract ▾
Computation and Language
The Geometry of Inference in Transformer Residual Streams
Timur Mudarisov, Mikhail Burtsev and Radu State
Transformer language models build predictions through successive residual updates, but how their representations become specific to an eventual outcome remains unclear. We study this process by comparing intermediate residual states with their own final states and an empirical bank of final states from other contexts. Across six pretrained
language models, the own endpoint becomes preferable to the average alternative early, while many individual endpoints remain closer. These competing sets generally shrink with depth, but their membership changes and their surviving endpoints need not become more similar to one another. Directional
alignment and endpoint rank can therefore improve while Euclidean distance to the final state changes little. We develop a simple high-dimensional model that separates the roles of norm,
alignment, and endpoint geometry, showing how gradual directional changes can produce sharp reductions in competition. We also prove that a straight path toward the own endpoint cannot introduce new competitors under either Euclidean or cosine distance; observed entries thus establish departures from straight-line convergence. Finally, endpoints associated with lower-ranked output
tokens tend to lie farther away in cosine distance across all studied models, connecting residual geometry to output organization. Together, these findings characterize increasing
geometric specificity during
transformer inference and explain why distance, competitor count, and concentration of the surviving endpoints provide distinct views of that process.
Read the full abstract ▾
Computation and Language
Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue
Shengbo Cai, Yuxiang Wang, Jingran Xie et al. (+5 more)
Empathetic spoken dialogue requires models to use both what is said and how it is said to decide how to respond. Explicit CoT can improve paralinguistic perception and make acoustic cues more explicit in replies, yet does not ensure their effective use in response planning. We call this mismatch the perception-
reasoning gap. In addition, CoT may not fully capture acoustic cues in words, and generating it adds inference latency. To address these limitations, we introduce LoopSLM, which builds on looped
Transformers for latent
reasoning, reusing a decoder block to refine hidden states with acoustic grounding at every pass. Its two-stage training further narrows the perception-
reasoning gap by separating learning to reason from learning to respond, enabling direct inference without CoT. On EchoMind, LoopSLM improves paralinguistic understanding,
reasoning, and reply quality over Qwen2.5-Omni-7B. Against the CoT-SFT baseline, LoopSLM gains over 20 points in
reasoning accuracy while generating 64.5% fewer
tokens at half the latency. It also outperforms Qwen3-Omni-Thinking on most empathetic reply metrics with 34x lower latency. Despite training only on dialogue data, LoopSLM improves accuracy on general audio
benchmarks.
Read the full abstract ▾
All authors Shengbo Cai, Yuxiang Wang, Jingran Xie, Zhisheng Zhang, Shun Lei, Di Cao, Teddy Sun, Zhiyong Wu
Computation and Language
CompOrca: Corpus-Scale Compliance Labelling of Instruction-Tuning Data
Philipp E. Glass and Alina Miron
Studying how fine-tuning shapes refusal and noncompliance behaviour requires knowing which training examples refuse or otherwise fail to fulfil the request. Existing annotations cover evaluation sets, which are far smaller than training corpora. We present CompOrca, compliance labels for all 4,233,923 examples of the OpenOrca corpus. Every example was classified as compliant or noncompliant by five passes of an open-weight LLM judge (LongCat-2.0, 1.6T parameters). The corpus is released as unanimous compliance (94.75%), unanimous noncompliance (1.28%), and nonunanimous rows (3.97%), with the raw vote counts. A single pass flags 2.7-3.2% of the corpus as noncompliant, while only 1.28% is flagged by all five, so the most ambiguous rows can be filtered out. Against 450 human-annotated examples (150 annotated twice; human-human $\kappa=0.93$), the unanimous compliance and noncompliance labels are 97.3% and 86.7% precise. The noncompliance label is a high-precision subset of the corpus's noncompliance. Published refusal-detection methods recall between 0.4% and 94.1% of human-labelled noncompliance. We release the full corpus with its per-row labels and vote counts at https://huggingface.co/
datasets/cemiu/CompOrca
Read the full abstract ▾
Computation and Language
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses
Zhangshu Joshua Jiang, Zina Ibrahim and James T. Teo
We propose a rubric for assessing expressed clinical
reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM
benchmarks (MedR-Bench, HealthBench, TIMER-Bench, DR. BENCH,
PrIME-LLM and PatientSafeBench); and general LLM
reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing
benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.
Read the full abstract ▾
Computation and Language
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
Zifeng Cheng, Jie Zheng, Zhiwei Jiang et al. (+4 more)
Large
language models (LLMs) have shown strong potential as training-free text encoders for long-context
embeddings. Existing approaches primarily improve information flow under causal
attention and typically construct
embeddings by uniformly averaging all
token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative
token selection in long-context
embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression
prompt to each chunk. A
prompt-isolated
attention mask preserves information flow among document
tokens while restricting each
prompt to its corresponding local context. We then use the
attention patterns elicited by these
prompts to estimate
token importance, select informative
tokens, and aggregate their intermediate-layer representations into the final
embedding. Extensive experiments on long-context
embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.
Read the full abstract ▾
All authors Zifeng Cheng, Jie Zheng, Zhiwei Jiang, Shuwen Wang, Fei Shen, Shiping Ge, Qing Gu
Computation and Language
Which papyrus HTR is good enough? Character-error-rate tolerance of four papyrological tasks on Greek texts
Anton Repushko and Elena Chepel
Purpose: Most Greek papyri remain unpublished and undigitised; a handwritten text recognition (HTR) pipeline that transcribes them automatically would let scholars discover documents and literary works that have so far gone unread. Recognition systems for Ancient Greek papyri are in statu nascendi, and how accurate they must be for a given papyrological task has not been examined. To answer this and set a
benchmark for Greek papyrus HTR, we test a range of character error rates (CER) against four papyrological tasks, using published editions as ground truth. Methods: From 63,846 current editions of Greek texts in papyri.info, we imitate a letters-only "perfect HTR" output by removing the editorial layer, then degrade it with a seeded
algorithm to exact CERs of 1 - 50%, with lost lines and four error-shape variants. On these data we train small models (TF-IDF, fastText, a character CNN, ByT5-small) for document type, dating and documentary-versus-literary classification, and apply eight keyword search methods. We compare models trained on clean text with models retrained at a specific CER level, and evaluate across CERs. Results: Tolerance differs by task. With clean-trained models, documentary-versus-literary classification retains 90% of its metric up to 20% CER; document type up to 7.5%; subtypes and search up to 5%; dating only up to 3%. Retraining on text containing character errors largely eliminates the sharp degradation that otherwise sets in above 15% CER. Models generally tolerate concentrated damage in a long document better than small errors spread across a short text. Conclusion: The study provides a CER target for each of the four tasks and shows that models trained on noisy text make current, imperfect text recognition useful for them.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin et al. (+10 more)
We introduce Context
Language Models (CLMs),
language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-
agent systems where multiple
agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository
agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-
language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.
Read the full abstract ▾
All authors Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh
Computation and Language · Cross-listed from cs.LG
Predictive Geometry of Hidden Trajectories in Transformers
Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova et al. (+1 more)
Decoder-only
transformers are trained only through a terminal next-
token prediction loss, yet this loss constrains every intermediate hidden state through the fixed downstream computation. We formalize this constraint by studying layerwise loss-to-go functions: the terminal loss obtained by continuing a candidate hidden state through the remaining
transformer blocks. Around successful validation trajectories, we show that the local second-order geometry of these functions is governed, up to low-loss residual terms, by a pullback Fisher operator on hidden-state space. Its spectrum identifies output-sensitive directions and approximately prediction-null directions, yielding a local observable subspace of the residual stream. For causal
transformers, the same geometry induces a tokenwise
curvature score: a Fisher-weighted sensitivity of the target logits to perturbations of each
token's hidden state. This score vanishes outside the causal ancestor set of the target and is controlled by downstream Jacobian couplings, making it a loss-aware alternative to
attention magnitude. We estimate these quantities using
matrix-free Jacobian-vector and vector-Jacobian products and evaluate them across decoder-only
language models on WikiText, OpenWebText, and FineWeb. Empirically, the induced geometry predicts perturbation sensitivity, supports nonuniform layerwise rank allocation, yields competitive structured
token-
pruning signals, and improves low-rank student recovery when added to stronger autoregressive
distillation objectives such as reverse KL and skew KL. These results support a predictive-
geometric view of
transformer computation: near successful trajectories, the terminal loss induces a thin, anisotropic set of output-relevant hidden-state directions that can be measured and exploited for compression and
distillation.
Read the full abstract ▾
All authors Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova, Radu State
Computation and Language
Billiger.de Products: A Bilingual Entity Matching Benchmark
Aaron Steiner, Ksenia Elagin, Ralph Peeters et al. (+2 more)
Existing product matching
benchmarks primarily contain English-
language product data and are often dominated by a single product category, such as electronics. This paper introduces Billiger.de Products, a bilingual German and English entity matching
benchmark covering thirteen consumer product categories, including difficult-to-handle categories such as clothing and furniture. The
benchmark data originates from the German price comparison platform billiger.de. Following the design of WDC Products, the
benchmark offers multiple variants that differ in the fraction of corner cases, the size of the development set, and the fraction of entities unseen during training. An aligned English translation of every offer keeps all pairs, splits, and labels fixed, while cross-
language test sets combine German and English records within individual pairs. We validate the
benchmark using six supervised matchers and zero-shot GPT-5.2 on both
language versions and the cross-
language test sets. The validation shows the difficulty of the
benchmark. The comparison of the results on the English version of the
benchmark to the results on the German version shows that most matchers score on average higher on the English version. The difference is largest for RoBERTa and HierGAT, while the zero-shot LLM runs are largely insensitive to the
language. Comparing the F1 scores achieved by PLM-based matchers on the English version of Billiger.de Products with their performance on existing English-
language benchmarks, such as WDC Products and Abt-Buy, shows that Billiger.de Products is more difficult than these
benchmarks.
Read the full abstract ▾
All authors Aaron Steiner, Ksenia Elagin, Ralph Peeters, Johannes Knopp, Christian Bizer
Computation and Language
Reader Proficiency Shapes Layer-wise Surprisal Profiles
Akio Hayakawa and Horacio Saggion
Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large
language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.
Read the full abstract ▾