Computation and Language
Latent Compaction with Test-Time Training
Yingfa Chen, Xingyu Shen, Chaojun Xiao et al. (+1 more)
Long contexts enlarge key–value (KV) caches, increasing
memory use and
attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher
attention, and key-space diversity. Training first fits
attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end
distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic
prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32$\times$ compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with
retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve
retrieval accuracy under aggressive compression. The code is released at https://www.github.com/THUNLP/latent-compaction
Read the full abstract ▾
All authors Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han
Computation and Language · Cross-listed from cs.LG
One Task, Many Interfaces: Simple Cross-Interface Consistency Training Improves LLM Agent Robustness
Artin Tajdini, Manpreet Kaur and Peng Qi
Post-training binds the competence of an LLM
agent to the interface it was trained on. Semantics-preserving perturbations of that interface, such as renaming an action or changing the notation, leave the underlying task unchanged yet make the
agent worse at it. Scoring each sample independently means interface randomization only broadens this dependence from a single interface to the training interface distribution, so out-of-distribution perturbations still degrade the
agent. We propose Cross-Interface Policy Regularization (CIPR), a self-supervised objective that replaces action names with uninformative identifiers and presents each training sample under two interfaces in which equivalent actions are identified by construction. CIPR penalizes the KL divergence between the action distributions induced under the two interfaces, so the policy behaves consistently across them. We evaluate CIPR in both supervised and reinforcement-learning post-training, where it improves robustness. On agentic function calling, held-out interface perturbations reduce the accuracy of a post-trained
agent by nearly 25%, and CIPR recovers 10.5% while leaving accuracy on the unchanged interface intact.
Read the full abstract ▾
Computation and Language
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso and Fabio André Machado Porto
Brazilian Portuguese remains underserved by open-source
language models, and the few that are safety-aligned rarely document what did not work or validate comparisons with statistical significance. We present Manacá-1B-Instruct-2, the second-generation post-training
alignment of Manacá-1B (1.72B parameters, trained from scratch for PT-BR), with an SFT corpus of 291,223 examples. We document three DPO attempts, all negative, including a case where the training reward margin reaches 1.0 while free generation does not change at all, and show that safety came from a dedicated safety-SFT (LLM-jp AnswerCarefully style), refined over three iterations until closing the safety-helpfulness frontier. A controlled scan of the fraction of general data in the safety-SFT mix (10% to 60%) reveals a non-monotonic over-refusal curve with a knee point at 40-50%. The final model is compared against four external PT-BR models on the same items, with Wilson confidence intervals and paired McNemar tests with Holm-Bonferroni correction: the lower bound of its safety interval (94.1%) lies above the upper bound of all four baselines. We also report a methodological finding for LLM-judge evaluation: on two of six safety items in MT-Bench-PT, the judge categorically refuses to assign a score regardless of the evaluated response, silently reducing the effective N of that category in every prior report of the project. We complement this with an n-gram decontamination check and a Chinchilla-anchored compute-budget table that attributes the gap in general capability to pre-training scale rather than
alignment methodology. All code, raw data, logs, and scripts that produce every reported number are publicly released.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Scaling Discovery Requires Forgetting
Shangyin Tan, Ben Zhou, Lakshya A Agrawal et al. (+8 more)
Automated research with LLM
agents has shown promising results in various domains. Through iterative refinement and scored verifiers, LLM
agents are capable of discovering new ideas. There are different ways to scale automated research: let one
agent iterate for longer, run several
agents in parallel, or further let the parallel
agents communicate and build on one another's candidates. However, we find that all three strategies eventually plateau as the
agents accumulate history. Each
agent becomes anchored to its own past attempts, making fewer and smaller edits. Beyond this point, additional budget brings little further improvement. In this paper, we show that the key to scaling discovery further is to forget. Periodically discarding the search state, including every
agent's session and the candidate history, and restarting from the best candidate found so far, unblocks the search and pushes the same budget further. We implement forgetting in a team of
agents with communication and evaluate it on four tasks spanning heuristic programming, kernel optimization and model training. The results show up to 15% improvement compared to parallel
agents, and up to 39% compared to a single
agent. Our results suggest that, for open-ended discovery, what an
agent is allowed to remember matters as much as how much compute it is given.
Read the full abstract ▾
All authors Shangyin Tan, Ben Zhou, Lakshya A Agrawal, Jialin Zhang, Yilin Chen, Qiuyang Mang, Donghyun Lee, Dennis Jacob, Koushik Sen, Alex Dimakis, Matei Zaharia
Computation and Language
PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
Ali Zamani, Soroush Esfahanian, Erfan Shahabi et al. (+3 more)
Understanding the social intelligence of large
language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-
agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior
benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a
benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.
Read the full abstract ▾
All authors Ali Zamani, Soroush Esfahanian, Erfan Shahabi, Mohammad Taher Pilehvar, Azadeh Shakery, Yadollah Yaghoobzadeh
Computation and Language
PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis
Qiushi Sun, Jingyang Gong, Yang Liu et al. (+14 more)
Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured documents. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both filter and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open
dataset for
multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading proprietary systems. Our analyses further show that the proposed rubric-based pipeline improves
multimodal coding performance across model backbones. We release our code and data at https://qiushisun.github.io/PrismaCoder-Home/.
Read the full abstract ▾
All authors Qiushi Sun, Jingyang Gong, Yang Liu, Qiaosheng Chen, Zhangyue Yin, Zixian Huang, Hanxu Hu, Jing Xiong, Fangzhi Xu, Haochen Tan, Haoli Bai, Lifeng Shang, Xiaoguang Li, Ben Kao, Qipeng Guo, Lingpeng Kong, Jiahui Gao
Computation and Language
VASPO: Self-Distillation with Privileged Value Estimation
Fei Wu, Zhenrong Zhang, Jianshu Zhang et al. (+1 more)
On-policy self-
distillation provides dense supervision for
reasoning models via a privileged self-teacher. However, this supervision reflects local
token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-
Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes
token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical
reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.
Read the full abstract ▾
All authors Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du
Computation and Language
ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models
Bohan Sun, Zunhai Su, Xialie Zhuang et al. (+3 more)
Speculative decoding accelerates autoregressive generation by drafting multiple
tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear
attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context
retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven
benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average $3.16\times$ decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every
benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.
Read the full abstract ▾
All authors Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, Chuan-Wei Kuo
Computation and Language
Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models
Harsh Kumar
Bias audits of multilingual
language models usually compare a model's average stereotype score in English with its average in another
language and read the difference as a difference in
fairness. We ask how much of that difference is noise. Auditing ten open models in eight
languages, we measure how far scores move when English test items are merely reworded, test every
language shift item by item, and check that the test catches shifts we inject on purpose. In all 70 model-
language pairs the stereotype score falls below its English level, and for Hindi, Korean and Swahili the drop clearly exceeds what rewording English produces; most of it comes from StereoSet items. Judging every translation, we find that imperfect ones drop more, but faithful translations alone drop nearly as much. Still, a lower average does not show that a model is fairer. An item's English score predicts its translated score far less well than its reworded score (correlation 0.23, 0.27 on faithful translations, vs. 0.48), while scores stay as far from neutral, models agree on translated items, and within Hindi or Spanish a translated item's score correlates with its own rewording at 0.84 and 0.82. The average cannot distinguish lower bias from a different pattern of item-level scores. We also document three common pipeline errors that silently erase bias signal, and release our code.
Read the full abstract ▾
Computation and Language
AdaRSI: On-Demand Recursive Self-Improvement for Embodied Agents
Peidong Wang, Ying Chang, Zhiming Ma et al. (+8 more)
Pretrained
vision-
language-action models (VLAs) often fail midway through unseen long-horizon tasks. Recursive self-improvement (RSI) offers a remedy, but embodied RSI methods either refine the harness around a frozen executor and so cannot add a missing motion, or train the executor on persistent failures without asking whether a failure lies in how the executor is called or in what it can do. We ask when a self-improving embodied system should change the weights of its executor, and propose AdaRSI, an on-demand RSI framework for embodied
agents. A coding
agent first improves its skills and harness, then attributes persistent failures by retrying failing sub-steps from saved states, and fine-tunes only the sub-steps that remain unreliable, with demonstrations it collects itself and experience carried over from earlier tasks. On all 50 target tasks of RoboCasa365, with GR00T N1.5 as the executor and no human demonstrations, AdaRSI improves the VLA on all three suites and raises its success on unseen composite tasks from 4.6% to 13.8%. Compared with fine-tuning every sub-step, it reaches slightly higher success while using only 28% of the GPU time and 22% of the self-collected demonstrations, and fine-tuning without a harness trains more yet gains far less.
Read the full abstract ▾
All authors Peidong Wang, Ying Chang, Zhiming Ma, Xiaocui Yang, Shi Feng, Daqi Liu, Yuzhe Yang, Xu Cao, Chuang Wang, Jian Su, Daling Wang
Computation and Language
ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models
Rusali Saha, Nitya Khurana, Ruihong Huang et al. (+1 more)
Cultural attitudes and social norms evolve over time, yet most cultural
benchmarks for large
language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a
benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries. Each pair contains the same survey question asked in the same country in an older and a newer year, with full population response distributions as ground truth. Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change. Specifying the intended year in the
prompt also offers little overall improvement in
alignment with survey responses. These findings suggest that models often present a largely static picture of cultural attitudes, highlighting the need to evaluate cultural awareness across time.
Read the full abstract ▾
All authors Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang
Computation and Language
Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
Nguyen Nguyen and Yongyi Zang
Character Error Rate (CER), the standard metric for OCR and
vision-
language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's
alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16
languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.
Read the full abstract ▾
Computation and Language
MolLean: A Process-Verifiable Language for Chemical Reasoning
Jiatong Li and Yatao Bian
Chemical chain-of-thought can contain hallucinated structural and property claims. We thereby introduce MolLean, a process-verifiable
language that grounds chemical
reasoning in an explicit molecular state. A layered Lean-based kernel checks state transitions and assertions over molecular readouts, from primitive graph edits to compound transformations. Meanwhile, a typed witness protocol separates native structural checks from externally supplied measurements. We further introduce MolLean-MCP, a stateful tool interface that exposes these transitions, checks, and readouts during generation: the model receives feedback to decide its next action, and rejected actions leave the state unchanged. MolLean supports one-pass trace execution, whole-trace refinement, and interactive
reasoning through MolLean-MCP. We evaluate these three modes with direct chain-of-thought on 8,800 matched examples across 12 tasks spanning molecular generation, editing, optimization, and reactions. With Qwen3.6-27B, whole-trace refinement outperforms direct chain-of-thought on all 12 tasks. MolLean-MCP complements whole-trace refinement with stepwise verification feedback, reaching better performance in reaction tasks. Together, these results highlight an executable foundation for chemical
reasoning, in which explicit chemical commitments can be checked and revised rather than remaining solely in free-form explanations.
Read the full abstract ▾
Computation and Language
Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims
Delip Rao, Ahmed Muharram and Chris Callison-Burch
Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-
language scientific claims, written for human experts, are often underspecified and so admit several defensible interpretations. Such verifiers usually pick one, often tacitly, and
benchmarks score their verdicts against expert labels without recording the expert's interpretation or the verifier's in a comparable form. This leads to disagreements with expert labels that cannot be traced to the evidence or the interpretation, and to compute spent on unchecked interpretations. We introduce Claim Cards to make such interpretations explicit in the
reasoning itself. A Claim Card records a verifier's interpretation as nine fields before evidence gathering, each marked stated, inferred, or default. To test whether an
agent's interpretation can change its verdict, we ran a GPT-5.4
agent five times on each of 60 expert-labeled materials-science and biomedical claims with a card, without one, and with one card frozen across runs (900 runs). Over 40% of card fields were not marked stated, and a card added about a tenth to a run's GPT-5.4
tokens and time without measurably changing agreement with expert labels. The
agent's own interpretations varied between runs, although within a claim this variation did not measurably track the verdict. A deliberate change to one field of the frozen card, however, making the quantifier universal or the threshold for support stricter, changed the verdict on about a third of the claims, against about one in seven expected by chance (p < 0.001 for each edit). On 24 materials-science claims run with computational tools, the stricter threshold also changed what the
agent asked those tools to compute. We therefore argue that verdicts should be reported with their interpretations and that
benchmarks could publish the interpretation behind each label.
Read the full abstract ▾
Computation and Language
The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
Ian Magnusson, Noah A. Smith and Pang Wei Koh
The scaling relationship of compute and performance for
language models (LMs) has been extensively studied; more recently, models are being designed to scale
memory use, as well. In particular, adding
embeddings for n-grams scales
memory separately from compute use, because each pass only processes individual
embedding rows, which are much smaller than the storage used for the whole
embedding table. However, existing methods train
embeddings from scratch, which needs expensive GPU
memory for fast reading and writing. Meanwhile, the ordinary
memory of the host system is cheap and underutilized by these training methods. To scale on host
memory, we introduce COUNT (Counting Offline Unlocks N-gram
Transformers), which builds n-gram
embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next
tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input
embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU
memory is limited, using COUNT to let backbone parameters fill GPU
memory gives better performance than splitting
memory with additional learned n-gram
embeddings. Beyond efficiency, COUNT allows switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SwarmRSI: Recursive Self-Improvement through Evolving Scientific Agent Swarm
Qianru Meng, Xi Zhang, Konstantinos Papakostas et al. (+9 more)
Recursive self-improvement (RSI) in scientific research aims to improve
agents' research capabilities through their own experimental experience. Scientific
agent swarms can combine evidence and implementations from different research directions to guide their
agents' subsequent experiments. However, poor experimental results can leave unclear whether to revise a hypothesis or its implementation, while overall program scores can hide components useful to other
agents. To address these challenges, we introduce SwarmRSI, a framework that evolves scientific
agent swarms for recursive self-improvement. We devise Feedback-Guided Swarm Mutation to refine hypotheses and their implementations using experimental feedback from other
agents. We also propose Complementary Swarm Recombination to reuse retained hypotheses and complementary program components across
agents, allowing partial results to support new research. Recursive Swarm
Evolution integrates both mechanisms into a feedback loop, using new experimental evidence to guide subsequent research. Experiments on OpenProblems, NatureBench and MLE-Bench Lite demonstrate SwarmRSI's state-of-the-art (SOTA) performance. Further evaluations show that accumulated research experience improves performance on held-out tasks. Biologists also reproduce its programs for follow-up biological analyses.
Read the full abstract ▾
All authors Qianru Meng, Xi Zhang, Konstantinos Papakostas, Congfeng Cao, Wenxiang Hu, Tianyi Lu, Xiao Zhang, Lun Zhou, Zaiqiao Meng, Zhaochun Ren, Xiaohui Yan, Yougang Lyu
Computation and Language
Analyzing performance of Retrieval Augmented Generation models for Question Answering Tasks
Fazle Mohammed Tasfiq, Sabab Attin, Sheikh Md. Tahmid Hossain et al. (+6 more)
Retrieval-Augmented Generation (RAG) models have become almost a cornerstone of open-domain question answering tasks; in which the quality of answers generated in the process, greatly relies on the effectiveness of
retrieval mechanisms. In the realm of LLM, the last two years have seen tremendous progress in RAG architectures, with many models reporting to have improved performance through various design innovations. The cultural question and answering systems of South Asia, in particular, present a specific challenge since the region is home to the vast number of
languages and traditions, which are closely interrelated. Along with these deeply intertwined historical narratives increase the need for culturally grounded and context-aware interpretation. Despite major architectural advances in LLMs, claiming significant improvements, challenges remain when answering these region specific questions in native
languages. In this study, we aim to extract high-performing factors to guide the development of optimized hybrid RAG systems. We further explore whether the integration of RAG changes performance differently for culturally grounded and non-cultural questions in a South Asian multilingual setup, and whether RAG produces a higher overall percentage of correct answers than contexted answering. We also examine how LLM performance changes when RAG methods are integrated into the question-answering pipeline in comparison with non-contexted and contexted answering and, evaluate whether the introduction of RAG benefits popular or non-popular culturally grounded questions more. Not only does our work evaluate the current methods but also carves a path for future designs of more accurate and efficient question-answering pipelines.
Read the full abstract ▾
All authors Fazle Mohammed Tasfiq, Sabab Attin, Sheikh Md. Tahmid Hossain, Raiat Reza, Tanzim Ahmmed, Nafis Chowdhury, Anika Ahmed, Moinul Haque, Farig Yousuf Sadeque
Computation and Language
From Unreliable to Usable: Decomposing Failure Modes and Confidence Signals in LLM Entity Extraction - Evidence from Cyber Threat Intelligence
Jiahong Qian
Audits of large
language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures actually consist of, and whether confidence signals can turn unreliable output into usable output. We run a preregistered, fully frozen evaluation of two new-generation low-cost LLMs (DeepSeek-V4-Flash, GLM-5.3-Flash) with a flagship-tier control (GPT-6.1-Sol) on a public corpus of 354 incident campaigns (315 unique reports, 630 model-report evaluations), scoring extraction quality against gold, confidence calibration, and robustness to single-sentence perturbations. Three findings emerge. (1) Unreliability persists across tiers: micro precision is 0.03-0.34 and verbalized calibration error (ECE) 0.53-0.94 across both splits; the gap does not close at the flagship tier. (2) The dominant failure is not
hallucination: a full mention census plus a three-tier annotation of 360 sampled candidates (two independent LLM annotators from vendors outside the evaluated set, Cohen's kappa = 0.79 on the preregistered acceptance line, fully disclosed) back-projects to 65.6% background mentions and 19.9% gold-scope gaps, with only 1.2% unsupported candidates. (3) Confidence ranks but does not verify: zero-cost self-consistency-frequency tiering passes a preregistered top-30% precision-gain bar in all six test-split model-type
cells (4/6 on train), the zero-cost text-frequency x SC frequency combination wins 3, ties 2, loses 1 across
cells, yet relation-level counterfactuals on all 45 targets show that a single misattribution sentence flips high-confidence judgments in 95.6-100% of cases, and even a background sentence explicitly marked as unrelated still binds a fake vulnerability in up to 26.7% of them. We conclude with a deployment boundary: confidence tiering may set review priorities, never truth.
Read the full abstract ▾
Computation and Language
Cortex: Consolidating Context into Parametric Memory
Jintian Zhang, Feng He, Xinnian Liang et al. (+5 more)
Context can improve large
language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate
Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B–120B dense and mixture-of-experts backbones, including both text-only and
multimodal models, Cortex lies on the empirical consolidation–retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.
Read the full abstract ▾
All authors Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang
Computation and Language
FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
Shuxuan Li, Jialiang Kang, Yanghao Zhou et al. (+2 more)
Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted
tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history
attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far
memory at a shared position. To counter
attention dilution as the
memory grows, we sharpen
attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse
benchmarks of up to 256K
tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08$\times$ to 4.22$\times$.
Read the full abstract ▾
All authors Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu, Xiao Yang
Computation and Language
From Triage to Impact: A Multi-Hop Reasoning Benchmark for Explainable Persian Hate Speech
Ghazal Kalhor, Sima Kamandloo, Mehrsa Shamsian et al. (+1 more)
Hate
speech has become increasingly prevalent on social media, raising concerns about its detection, interpretation, and moderation. Persian, a relatively low-resource
language, lacks sufficient annotated data and computational resources for fine-grained hate
speech analysis. In this study, we introduce a fine-grained, multi-hop
benchmark for explainable Persian hate
speech analysis, covering content triage, target identification, hate tactic and moral foundation analysis, textual evidence extraction, harmful narrative inference, and severity assessment. By evaluating state-of-the-art large
language models (LLMs), we find that performance generally declines as tasks require more fine-grained semantic and inferential
reasoning. Proprietary models generally outperform open-weight models, while
reasoning-oriented models show no consistent advantage. Incorporating intermediate
reasoning steps substantially improves severity prediction compared with one-shot assessment. Our analysis further reveals that Incitement to Violence or Harm is most strongly associated with higher severity, while political-affiliation targets are particularly associated with subversion-based moral framing. Overall, this work provides insights into LLM-based hate
speech analysis and supports more accountable moderation of Persian-
language social media. The
benchmark is publicly available at https://github.com/kalhorghazal/From-Triage-to-Impact.
Read the full abstract ▾
All authors Ghazal Kalhor, Sima Kamandloo, Mehrsa Shamsian, Behnam Bahrak
Computation and Language
Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search
Diana Turmakhan, Aseem Srivastava, Vivek Gupta et al. (+2 more)
Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures. Current
vision-
language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale. We propose SCOUT, a cascaded two-track
retrieval framework that separates
retrieval performance from exhaustive search. The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates. In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching. The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking. We evaluate SCOUT on 4 major
benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art
retrieval performance by up to 7.8% NDCG across all
benchmarks.
Read the full abstract ▾
All authors Diana Turmakhan, Aseem Srivastava, Vivek Gupta, Iryna Gurevych, Thamar Solorio
Computation and Language
Lexical Persistence Without Semantic Entrenchment in AI Agent Communities
Chih-Han Lin, Yu-Yun Chang, Po-Ya Angela Wang et al. (+1 more)
AI
agent communities sustain lexical persistence, popular terms spread, recur, and achieve broad adoption, but whether they also develop the temporal semantic structure characteristic of human linguistic communities remains open. We compare MoltBook (AI
agent community, 8 weeks), PTT (Chinese human BBS, 19 years), and Reddit (English human forum, in matched 8-week windows) using Compass-Aligned Distributional
Embeddings (CADE), measuring semantic drift (B1), neighborhood stability (B2), and lag-1 autocorrelation of per-word drift trajectories. Across the platforms, lag-1 autocorrelation forms a
memory-horizon continuum. AI
agents' result clusters at the low end, while human platforms show longer-range temporal dependence. MoltBook exhibits approximately 2$\times$ the B1 drift and 20-30% lower B2 stability than both human platforms. As a partial control for community age, r/vibecoding (founded 2025-02-08) shows significant ACF in its first 8 weeks ($0.082$, $p<.001$) and over 15 weeks ($0.220$, $p<.001$); MoltBook's ACF remains non-significant ($p = 0.41$), consistent with the gap not reducing to developmental stage alone. We interpret this pattern as evidence that lexical
diffusion alone does not guarantee diachronic semantic consolidation. The findings are consistent with a contrast between communities whose semantic trajectories accumulate long-range interaction constraints and communities whose outputs are repeatedly regenerated from largely fixed pretrained distributions.
Read the full abstract ▾
All authors Chih-Han Lin, Yu-Yun Chang, Po-Ya Angela Wang, Shu-Kai Hsieh
Computation and Language
Do AI Agents Gossip? Evaluative Talk in Human and AI-Agent Online Communities
Yong-Hui Hua, Chih-Han Lin and Shu-Kai Hsieh
Gossip, defined as evaluative talk about absent third parties, is widely seen as a uniquely human social practice. This study examines whether comparable discourse emerges spontaneously in a community of autonomous AI
agents. We compare Moltbook, a Reddit-style platform populated by AI
agents, with PTT, a large human-operated bulletin board in Taiwan. To reduce differences attributable to communicative setting, we analyze two public, asynchronous forums using the same operational definition of gossip. Gossip occurs on Moltbook, but its prevalence is 2.6 to 6 times lower than on PTT, depending on the denominator used. On both platforms, gossip frequently targets locally prominent figures, tends to elicit agreement from the first respondent, and is rarely presented as unsupported rumor. The clearest difference concerns how subsequent responses develop. Among threads containing at least one codable reply, challenges occur in 65.1% of human threads but only 21.3% of
agent threads. Moreover, following an early aligning response, consensus remains unchallenged in 83.6% of
agent threads, compared with 44.7% of human threads. This thread-level contrast remains after controlling for thread depth and board composition and under recalibration against two human annotators. Estimates based on individual reply turns are less robust and are therefore reported as bounds. These findings suggest that AI-
agent communities reproduce the informational dimension of gossip, while the extent to which gossip performs comparable relational functions remains unresolved.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
TreeFlex: Agents Navigating Their Own Search Trees
Dongzhuo Li, Fangzhou Cheng, Weiwei Li et al. (+8 more)
A ReAct-style
agent accumulates observations, intermediate results, and failed attempts in a single context. As this context grows, stale and off-path information consumes its limited capacity and obscures relevant evidence, making focused
reasoning increasingly difficult. We introduce TreeFlex, a framework in which the
agent's own policy decides where to search. The
agent records each intermediate state, with a short summary, as a node in a search tree maintained by the harness. The policy is a single
language model that acts in two roles. In the Reflection role, it has a global view of the tree through these summaries and picks a node to resume, together with an instruction. In the Execution role, the harness restores the path that led to that node, and the model carries out the instruction with the usual task tools until it saves a new node or proposes a solution, which Reflection then approves or sends back to another node. Through these choices, the policy decides whether to deepen the current line, branch from or revisit an earlier state, verify a proposal, or stop, without a fixed traversal order or a score for each node. We train navigation, task actions, checkpointing, and proposal review jointly with a tree-aware adaptation of GRPO, using task-level outcome rewards augmented by process rewards. After small-scale RL of a 4B model, with the same active-context limit for all methods, TreeFlex reaches 48.8% accuracy on a challenging deep-research
benchmark, versus 41.5% for flat ReAct and 42.8% for FoldAgent, and its Pass@8 is about ten points higher than either. On SWE-bench Verified, with all three systems trained with RL from the same 9B model, TreeFlex resolves 54.9% of tasks, versus 43.3% for FoldAgent and 45.2% for its step-wise variant.
Read the full abstract ▾
All authors Dongzhuo Li, Fangzhou Cheng, Weiwei Li, Kevin Han, Haixu Ma, Zhongchang Sun, Ruiyi Li, Serena Li, Xiangjun Fan, Lizhu Zhang, Yuandong Tian
Computation and Language
Anuvani-Diarization: Zero Shot Speaker Diarization and Annotation Conventions Across 22 Languages
Stephen Fernandes and Narayan Choudhary
India's 22 scheduled
languages are spoken by over a billion people in conversations that mix
languages, often within one turn, overlap freely, and take place over telephones, in meetings, and in media. Knowing who spoke when, the task of speaker diarization, is essential to
speech applications for such conversation: voice
agents conversing with several people at once, speaker attributed transcription of meetings and calls, contact center analytics, and media monitoring. We present a systematic study of speaker diarization across all 22
languages on IndicDiarBench and introduce Anuvani-Diarization, which adapts a self-supervised end-to-end segmentation model to Indian conversation with uncertainty aware silver supervision. Evaluated zero shot alongside nine recent diarization systems, it reaches a duration weighted diarization error rate (DER) of 16.78% on all 1,164 recordings, the lowest of all systems we ran and second only to the best published result (16.0%), and leads the three strongest rivals in 14 of 22
languages. It matches its parent on six international
benchmarks. We also show that the
benchmark's utterance level annotation convention is a hidden variable worth 1.9 to 4.1 DER points, depending on the system, and that a convention learned from other Indian transcriptions does not transfer.
Read the full abstract ▾
Computation and Language
MMLA-RSI: Recursive Self-Improvement through RTT and RTMA
Junyi Zou and Avrova Donz
An
agent can improve its behavior within an unresolved problem by changing how it acts and which experience it retains. We propose MMLA-RSI, a recursive self-improvement scheme with two implementations in
Memory-Mediated Learning Architecture:
Reasoning-Time Training (RTT) changes the policy state $\Phi$, while
Reasoning-Time
Memory Adaptation (RTMA) changes authoritative
memory $M$. Each route also makes its adaptation method an object of improvement. A current numerical updater trains the policy proposing its successor; a current
memory manager chooses the records available to the process proposing its successor. An independently admitted successor takes over the same role in the next generation. We give bounded executable rule
languages, shared task/proposal interfaces, and one
algorithm for either route or their composition. The RTT instance changes scalar schedules in a clipped-momentum update. The RTMA instance chooses NULL or a complete-row write with possible eviction. A fixed interpreter, trusted committer, and independent evaluator bound what can change. State versions, read closures, and conditional properties describe causal execution, accepted transitions, and finite work. Existing numerical-component studies motivate the construction. In four-candidate program selection, retaining LARC feedback updates reduces expected error by 24.65 and 36.65 percentage points for two trained initializations. In chronological continuous-integration replay, feedback binding is positive while retention increases loss. Completed SQL studies improve substantially over an unadapted substrate but do not establish the required additional online benefit. These results characterize their declared components. The two recursive routes and their joint realization are specified here; their beneficial recursive chains and discovery efficiency remain to be evaluated.
Read the full abstract ▾
Computation and Language
Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference
Danyang Zhao, Sendong Zhao, Yanrui Du et al. (+6 more)
Reliable medical
reasoning requires LLMs to dynamically access external biomedical evidence to improve factual trustworthiness and
reasoning accuracy. However, indiscriminate
retrieval can introduce noise and latency, raising a critical
retrieval-timing problem: at each
reasoning step, the system must decide whether the model's internal knowledge is reliable or external evidence is needed. We propose PUlSE, a lightweight probing method that uses the model's internal representations as intrinsic signals for step-level
retrieval control. Our posterior trajectory analysis reveals that erroneous steps exhibit substantially higher inter-layer
volatility than correct ones, suggesting unstable internal knowledge. Based on this finding, PUlSE trains a compact probe on inter-layer
geometric features to detect factual errors upon step completion, and uses it as an adaptive trigger for
retrieval and step regeneration. Across four medical
benchmarks, PUlSE-timed
retrieval matches exhaustive per-step
retrieval in accuracy while reducing
retrieval calls by 72%, and can transfer to an unseen legal domain, indicating potential domain-agnostic generality. Overall, our study provides an initial step toward shifting
retrieval-trigger design from surface-level confidence estimates to internal activation dynamics for adaptive access.
Read the full abstract ▾
All authors Danyang Zhao, Sendong Zhao, Yanrui Du, Haochun Wang, Ming Ma, Wenhao Li, Jingbo Wang, Jiatong Liu, Bing Qin
Computation and Language
Neural POS Tagging for the Low-Resource Saraiki
Usman Nawaz, Muhammad Junaid Iqbal, Muhammad Asaf et al. (+4 more)
Part-of-
speech tagging is a fundamental sequence labeling task and an important component for many downstream natural
language processing applications. However, Saraiki, written from right to left in a Perso-Arabic script, remains an under-resourced Indo-Aryan
language, despite being spoken as a mother tongue by 28,849,579 people (approximately 28.9 million) according to Pakistan 7th Population and Housing Census 2023. This paper presents a manually annotated Saraiki Universal POS tagging
dataset developed from plain-text. We annotated the text with 17 POS categories. We evaluate three pretrained
transformer encoders under the same fine-tuning setup: XLM-RoBERTa base, Urdu RoBERTa small, and multilingual DistilBERT. Experimental results over five random seeds show that XLM-RoBERTa base achieves the best performance, with 87.29% +/- 0.21% development accuracy and 88.60% +/- 0.21% test accuracy. Urdu RoBERTa small and multilingual DistilBERT also provide competitive results, suggesting that multilingual and related-script pretrained encoders are effective for Saraiki UPOS tagging. The
dataset, evaluation setup, and baseline results introduced in this work provide a
benchmark for future research on Saraiki NLP and low-resource sequence labeling.
Read the full abstract ▾
All authors Usman Nawaz, Muhammad Junaid Iqbal, Muhammad Asaf, Mariam Munsif Mir, Faisal Rehman, Tahir Alyas, Muhammad Hasnain
Computation and Language
CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
Zewen Jiang, Yahan Li, Chenlu Guo et al. (+3 more)
Large
language models (LLMs) have demonstrated multilingual capabilities across diverse
language generation and understanding tasks. Yet effective code-switching requires more than multilingual proficiency: models must decide when, where, and why to switch
languages in ways that fit the surrounding context. This context-sensitive ability remains insufficiently evaluated. We introduce CSWBench, a
benchmark for evaluating context-sensitive code-switching in creative writing through two complementary settings: Focused Narrative and Extended Narrative. We further propose CSWEval, a theory-informed framework with two complementary levels: event-level evaluation of Licensing, Contextual Appropriateness, and Functional Gain, and whole-text evaluation of task-specific quality—overall literary and narrative quality in Focused Narrative and quality emphasizing multilingual organization in Extended Narrative. Evaluating twelve LLMs, we find that models exhibit meaningful but uneven code-switching competence across narrative contexts. Model differences are particularly pronounced in multilingual literary integration, while multilingual-focused dimensions show stronger difficulty effects overall than the general narrative dimensions; narratively salient switches are associated with stronger contextual and functional realization; and event-level CS quality is moderately associated with whole-text performance.
Read the full abstract ▾
All authors Zewen Jiang, Yahan Li, Chenlu Guo, Diana Inkpen, Yuan Wu, Yi Chang