Computation and Language
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Jiachen Zhao, Zhengxuan Wu, Aryaman Arora et al. (+3 more)
The mechanisms behind LLMs' broad over-generalization beyond training examples remain unclear. Emergent misalignment (EM) offers a striking case study: finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains. In this work, we propose the Piggyback Hypothesis: the chat-template
tokens can piggyback the finetuned behaviour onto out-of-domain queries. We validate this hypothesis by showing that subtle perturbations to the prefix (
tokens preceding all user queries), or patching the prefix representations with those from the unfinetuned model, can restore
alignment without changing the user query. Building on this finding, we propose
Token-Regularized Finetuning (TReFT), which regularizes specific
token representations during training to mitigate EM. Across different models and multiple EM-inducing
datasets, TReFT reduces EM while preserving in-domain learning. On Llama-3.1-8B finetuned on the legal domain, TReFT achieves 33.5% more EM reduction than data interleaving with a retain set of aligned examples. We further show that TReFT extends to other narrow-finetuning settings, including abstention, tool use, and refusal (off-topic generalization is reduced by 54.3% on average), supporting the Piggyback Hypothesis. Broadly, our work highlights that LLMs may learn and generalize in unintended ways and suggests a path toward more constrained finetuning. It also calls for further study of how shared input features can piggyback model behavior across domains.
Read the full abstract ▾
All authors Jiachen Zhao, Zhengxuan Wu, Aryaman Arora, Yiyou Sun, David Bau, Weiyan Shi
Computation and Language
CAF-Gen: A Multi-Agent System for Enriching Argumentation Structures
Jakub Bąba and Jarosław A. Chudziak
Formalizing complex
reasoning from natural text is one of the central challenges in computational linguistics. It requires systems to understand not just keywords but also the context and complex
reasoning embedded in a text. Current Argument Mining (AM) techniques identify basic claims and premises, yet they often struggle to capture the richer structural information required by advanced schemas such as the Carneades Argumentation Framework (CAF), which incorporates features such as premise types, proof standards, and argument schemes. We address this limitation by introducing CAF-Gen, an automated multi-
agent framework designed to enrich shallow argument structures into CAF-compliant argument models. By employing an iterative Creator-Reviewer pipeline, a creator
agent's output is validated by a critical
agent to ensure structural integrity. This multi-
agent collaboration is crucial for mitigating the structural instability typical of single-pass generative models. Our experiments demonstrate that the iterative feedback loop improves the quality of the resulting data and achieves strong
alignment with the original annotations, while producing structurally richer models. Our findings show that the multi-
agent system can overcome the limitations of single-pass generation, providing a robust methodology for the automated modeling of formal argumentation.
Read the full abstract ▾
Computation and Language
How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures
Tanvi Thoria, Kiana Jafari, Marc R. Schlichting et al. (+1 more)
Failures in
language model
reasoning emerge through distinct processes that leave identifiable signatures in the
reasoning trace. We characterize these failures using
token-level uncertainty signals, finding they arise through two empirically distinguishable processes. The first is committed failure, in which a model locks onto an incorrect
reasoning path early in its trace. A central diagnostic signature is the commitment point, beyond which considering additional
tokens hurt rather than help failure detection. In the second, persistent uncertainty, uncertainty instead accumulates throughout, and the full trace is needed to best distinguish failing from successful completions. These signatures reproduce across 23 model-
dataset configurations, with the framework's falsifiable predictions holding in 20 of 23 cases, well above chance across both failure modes. Finally, we demonstrate our failure mode framework has direct implications for self-consistency, identifying when uncertainty signals complement it and when it can be selectively skipped. These results offer a foundation for understanding when LLM
reasoning failures become detectable and for adapting detection strategies accordingly.
Read the full abstract ▾
All authors Tanvi Thoria, Kiana Jafari, Marc R. Schlichting, Mykel J. Kochenderfer
Computation and Language
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo et al. (+4 more)
We introduce UnpredictaBench, an evaluation that tests the ability of large
language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by
stochastic programs, and natural-
language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding
reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems. Project website and resources are available at https://unpredictabenchmark.github.io/.
Read the full abstract ▾
All authors Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West
Computation and Language
Re-Centering Humans in LLM Personalization
Lechen Zhang, Jiarui Liu and Tal August
Despite growing interest, most evaluations of large
language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new
prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.
Read the full abstract ▾
Computation and Language
PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall
Jonathan von Rad, Louis Arts, George Burgess et al. (+6 more)
Large
language models (LLMs) trained predominantly on English data encode substantial world knowledge, yet often fail to express it reliably in other
languages, a phenomenon known as cross-lingual factual inconsistency. To study this, we introduce PolyFact, a fully parallel
multilingual factual QA
dataset of 60K Wikidata-grounded facts across 12 typologically diverse
languages, and propose consistency-driven GRPO with cross-lingual reward pooling. We compare our method against supervised fine-tuning (SFT) and the consistency-enhancement baselines DCO and CM-Align on OLMo-2-1124-7B and Qwen-2.5-7B, and analyze whether light continual pretraining (CPT) on parallel data provides a useful foundation for post-training. No single method dominates: SFT maximises in-distribution accuracy but not consistency, DCO yields the strongest consistency gains but fails to transfer to free-form generation, and our GRPO variant achieves the strongest transfer to free-form recall and unseen
languages on the
multilingual base model. CPT mildly aids monolingual models but harms
multilingual ones. Mechanistic analyses suggest that GRPO is associated with reduced
language specialization, consistent with greater sharing of representations across
languages. We release our code, models, and
dataset publicly.
Read the full abstract ▾
All authors Jonathan von Rad, Louis Arts, George Burgess, Eleftheria Kolokytha, Harry O'Donnell, Ektor Oikonomidis Doumpas, Eduardo Sanchez, Yao Lu, Pontus Stenetorp
Computation and Language · Cross-listed from cs.LG
Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
Athanasios Zeris
We introduce scale-selective Proper Orthogonal Decomposition (POD) for
transformer attention fields, inspired by the use of POD for extracting energetically dominant modes from turbulent flow ensembles. The Morlet continuous wavelet transform identifies dominant temporal scales in the
attention lag structure across a document ensemble; POD then extracts the energetically dominant modes at each scale from the ensemble of
attention fields. The resulting modes reveal layer-dependent scale organisation, with early layers emphasising fine scales and later layers shifting toward coarser scales. We define a spectral concentration index from the POD
eigenvalue decay rate and show empirically that it differentiates layers by their
attention field complexity. By the classical POD optimality
theorem, the extracted modes minimise the average L2 reconstruction error over the ensemble (
Theorem 1), giving a data-driven effective rank for each layer. The method requires no architectural modification and no linguistic annotations: dominant
attention patterns emerge from ensemble statistics alone. The
turbulence analogy is structural rather than physical: we borrow ensemble covariance and modal analysis, not fluid dynamics itself.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution
Liliana Hotsko, Yinxi Li, Yuntian Deng et al. (+1 more)
Code
language models need repository-level context to resolve imports, APIs, and project conventions. Existing methods inject this knowledge as long inputs (retrieved through RAG or dependency analysis) or through per-repository fine-tuning and LoRA—costly at repository scale and brittle to evolving codebases. We introduce Code2LoRA, a hypernetwork framework that generates repository-specific LoRA adapters, effectively injecting repository knowledge with zero inference-time
token overhead. Code2LoRA supports two usage scenarios: Code2LoRA-Static converts a single repository snapshot into an adapter, suitable for comprehension of stable codebases; while Code2LoRA-Evo maintains an adapter backed by a GRU hidden state updated per code diff, suitable for active development of evolving codebases. To evaluate Code2LoRA against parameter-efficient fine-tuning baselines, we build RepoPeftBench, a
benchmark of 604 Python repositories with two tracks: a static track with 40K training and 12K test assertion-completion tasks, and an
evolution track with 215K commit-derived training and 87K commit-derived test tasks. On the static track, Code2LoRA-Static achieves 63.8% cross-repo and 66.2% in-repo exact match, matching the per-repository LoRA upper bound; on the
evolution track, Code2LoRA-Evo achieves 60.3% cross-repo exact match (+5.2 pp over a single shared LoRA). Code2LoRA's code can be found at https://anonymous.4open.science/r/code2lora-6857; the model checkpoints and RepoPeftBench
datasets can be found at https://huggingface.co/code2lora.
Read the full abstract ▾
All authors Liliana Hotsko, Yinxi Li, Yuntian Deng, Pengyu Nie
Computation and Language
Operation-Guided Progressive Human-to-AI Text Transformation Benchmark for Multi-Granularity AI-Text Detection
Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao et al. (+9 more)
As AI writing assistants become increasingly integrated into real-world drafting and revision workflows, many documents are no longer purely human-written or AI-generated, but instead result from progressive human-AI co-editing. However, existing AI-text detection
benchmarks largely focus on final outputs and provide limited understanding of how AI authorship signals emerge, accumulate, or disappear throughout the revision process. We introduce OpAI-Bench, an operation-guided
benchmark for studying progressive human-to-AI text transformation across document, sentence,
token, and span granularities. Starting from human-written documents, OpAI-Bench constructs nine sequentially revised versions for each sample under predefined AI coverage levels and five representative AI edit operations, covering four domains while preserving complete authorship provenance at multiple granularities. The
benchmark supports comprehensive evaluation with 8 document-level detectors, 7 sentence-level detectors, and 2 fine-grained
token/span-level detectors. Experiments reveal that AI-text detectability is governed not only by the proportion of AI-edited content, but also by edit operation, domain, and cumulative revision history. Interestingly, we notice that mixed-authorship intermediate versions are often harder to detect than both fully human and heavily AI-edited endpoints, exposing non-monotonic detection patterns missed by existing
benchmarks. OpAI-Bench provides a controlled testbed for analyzing whether, when, and how AI-assisted writing becomes detectable under realistic progressive editing scenarios. Our code and
benchmark are available at https://github.com/VILA-Lab/OpAI-Bench.
Read the full abstract ▾
All authors Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tianjun Yao, Xinyi Shang, Yi Tang, Jiacheng Cui, Ahmed Elhagry, Salwa K. Al Khatib, Hao Li, Salman Khan, Zhiqiang Shen
Computation and Language
Paul Jünger, Justin Lovelace, Linxi Zhao et al. (+2 more)
Diffusion language models generate text by iteratively denoising an entire response in parallel, holding a noisy draft of the answer at every step. We show that these intermediate states surface salient entities early in the denoising trajectory, making them a useful lookahead signal for
retrieval-augmented generation (RAG). We further demonstrate that grounding generation in retrieved evidence sharply reduces inter-
token dependence, making RAG particularly well suited to parallel decoding. We exploit both through Self-Augmenting
Retrieval for
Diffusion Language Models (SARDI), a dynamic RAG framework that uses speculative
token predictions to guide
retrieval during denoising. SARDI is training-free, retriever-agnostic, and applicable to any
reasoning-capable discrete
diffusion language model. Across five multi-hop QA
benchmarks, SARDI outperforms training-free
diffusion and autoregressive
retrieval baselines at up to 4x higher throughput.
Read the full abstract ▾
All authors Paul Jünger, Justin Lovelace, Linxi Zhao, Dongyoung Go, Kilian Q. Weinberger
Computation and Language · Cross-listed from cs.AI
MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery
Shangheng Du, Xiangchao Yan, Jinxin Shi et al. (+13 more)
Large
language model (LLM)
agents are increasingly applied to long-horizon tasks such as scientific discovery and machine learning engineering (MLE), where sustained self-
evolution becomes a key capability. However, existing MLE
agents suffer from isolated exploration, inefficient experience reuse, and lack of hierarchical control, which together hinder long-horizon optimization. We present MLEvolve, an LLM-based self-evolving multi-
agent framework for end-to-end machine learning
algorithm discovery. By extending tree search to Progressive MCGS, MLEvolve enables cross-branch information flow through graph-based reference edges and gradually shifts the search from broad exploration to focused exploitation with an
entropy-inspired progressive schedule. To allow the
agent to evolve with accumulated experience, we introduce Retrospective
Memory, which combines a cold-start domain knowledge base with a dynamic global
memory for task-specific experience
retrieval and reuse. For stable long-horizon iteration, we further decouple strategic planning from code generation with adaptive coding modes. Evaluation on MLE-Bench shows that MLEvolve achieves state-of-the-art performance across multiple dimensions including average medal rate and valid submission rate under a 12-hour budget (half the standard runtime). Moreover, MLEvolve also outperforms specialized
algorithm discovery methods including AlphaEvolve on mathematical
algorithm optimization tasks, demonstrating strong cross-domain generalization. Code is available at https://github.com/InternScience/MLEvolve.
Read the full abstract ▾
All authors Shangheng Du, Xiangchao Yan, Jinxin Shi, Zongsheng Cao, Shiyang Feng, Zichen Liang, Boyuan Sun, Xu Guo, Tianshuo Peng, Yifan Zhou, Jie Hou, Xin Li, Jie Zhou, Liang He, Bo Zhang, Lei Bai
Computation and Language
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Yutao Sun, Yanqi Zhang, Li Dong et al. (+2 more)
Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in
reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse
attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while
token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. In this work, we propose cross-layer sparse
attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes
token-level top-k selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of
token sparse
attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly, including pre-filling, KV-cache storage, and long-context decoding. Experiments across short-context and long-context
benchmarks show that CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context. These results suggest a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.
Read the full abstract ▾
All authors Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei
Computation and Language
Human Adults and LLMs as Scientists: Who Benefits from Active Exploration?
Mandana Samiei, Eunice Yiu, Anthony GX-Chen et al. (+5 more)
A long-standing finding in the causal learning literature is that adults struggle to identify conjunctive causal rules, where an effect requires the simultaneous presence of multiple causes, while performing better in disjunctive settings. However, most demonstrations of this “conjunctive handicap” rely on passive observation paradigms with limited evidence, where learners have no control over evidence generation. This paper asks whether this bias persists when adults are granted agency through active exploration. Using a modified “blicket detector” task, adult participants freely intervened to identify causal objects under conjunctive or disjunctive rule structures. We show that active exploration substantially improves adults' conjunctive causal
reasoning, although conjunctive rules still require more tests to infer than disjunctive rules. We further compare human performance to a range of large
language models in the same setting. While some state-of-the-art models approach human-level performance on hypothesis inference accuracy, they often exhibit less efficient exploration strategies and similar conjunctive-disjunctive performance gaps.
Read the full abstract ▾
All authors Mandana Samiei, Eunice Yiu, Anthony GX-Chen, Dongyan Lin, Jocelyn Shen, Blake A. Richards, Alison Gopnik, Doina Precup
Computation and Language · Cross-listed from cs.SE
Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skill
Mehmet Iscan
Large
language models increasingly write, review, and judge code, and a fast-growing practice equips them with
prompt 'skills' that ask the model to reason like a scientist. A prominent example tells the model to act as a Popperian falsificationist, and such skills are reported to improve generated code. But these gains are almost always read off an LLM-as-a-judge, an instrument with documented positional, self-preference, and stylistic biases. We ask: if it appears to help, is the gain from the skill's Popperian content, or from the structure any scaffold imposes? We pre-register a two-tier ablation with three controls: a length-matched placebo, a labels-only scaffold that keeps the Popperian headers but strips the procedure, and an execution oracle (HumanEval+ unit tests), plus a vocabulary-
halo sentinel and a same-model self-judge audit. On a frontier model (Claude Sonnet 4.6, N=163) all conditions sit near the
benchmark ceiling and do not separate, so the pre-registered +5-point improvement is not supported (a ceiling-limited non-detection). On a small model (Qwen2.5-Coder-0.5B, N=164) structured arms lift best-of-eight correctness by 20-22 points, but the full skill shows no separable benefit over a labels-only scaffold (aggregate F@8=L@8 vs V@8=34.8%), and the placebo trails by only 2.4 points. A 0.5B self-judge applying the Popperian rubric does not beat random selection and concentrates 60% of its picks on one index. In the two settings tested, the skill's Popperian procedural content adds no separable execution-correctness benefit beyond a labels-only scaffold, so the gains track scaffold structure. We contribute a calibrated negative result and a reusable disambiguation protocol; the finding bounds an engineering claim about one
prompt-skill family and is not an evaluation of Popperian methodology in general.
Read the full abstract ▾
Computation and Language
Latent Reasoning with Normalizing Flows
Guancheng Tu, Xiangjun Fu, Suhao Yu et al. (+5 more)
Large
language models often improve
reasoning by generating explicit chain-of-thought (CoT), demonstrating the importance of intermediate computation. However, textual CoT forces this computation through a discrete, serial, and communication-oriented
token stream: each
reasoning step must be verbalized before the model can proceed, even when the underlying update is semantic, uncertain, or only partially formed. Latent
reasoning offers a higher-bandwidth alternative by performing intermediate computation in compact continuous states before committing to text. Yet existing latent-
reasoning methods often sacrifice key advantages that make CoT effective in autoregressive
language models, including native left-to-right generation, probabilistic sampling, compatibility with KV-cache decoding, and tractable likelihood estimation. We propose NF-CoT, a latent
reasoning framework that preserves these advantages by modeling continuous thoughts with normalizing flows. NF-CoT instantiates a TARFlow-style normalizing flow inside the LLM backbone, defining a tractable probability model over compact continuous thoughts distilled from explicit CoT. Continuous-thought positions are generated by an NF head, while text positions are generated by the standard LM head within the same causal stream. This design provides exact likelihoods for latent thoughts, enables probabilistic left-to-right decoding with the original KV cache, and supports direct policy-gradient optimization in the latent
reasoning space. On code-generation
benchmarks, NF-CoT improves pass rates over explicit-CoT and prior latent-
reasoning baselines while substantially reducing intermediate-
reasoning cost.
Read the full abstract ▾
All authors Guancheng Tu, Xiangjun Fu, Suhao Yu, Yao Tang, Haoqiang Kang, Lianhui Qin, Yizhe Zhang, Jiatao Gu
Computation and Language · Cross-listed from eess.AS
USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati et al. (+4 more)
Audio encoders are critical to modern audio applications as large
language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has yielded strong domain-specific encoders like
speech or music experts, multi-domain approaches like USAD and SPEAR remain limited in coverage and evaluation. Recent studies also suggest supervised encoders align better with audio LLMs. We present USAD 2.0, a universal encoder integrating knowledge from both SSL and supervised foundation models. USAD 2.0 introduces domain-aware
distillation to address teacher mismatch, extends coverage to the music domain, and adds second-stage supervised
distillation for downstream use. We further scale the model to one billion parameters via depth scaling. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across probing and LLM-based evaluations.
Read the full abstract ▾
All authors Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati, Mrudula Athi, Anton Ratnarajah, Amit Chhetri, James Glass
Computation and Language
Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
Xinnong Zhang, Wanting Shan, Hanjia Lyu et al. (+2 more)
Large
language models are increasingly used to simulate social media users and infer how individuals may respond to online discussions. However, it remains unclear whether these simulations reflect precise user-specific beliefs or whether they are highly sensitive to semantically independent changes in conversational contexts. In this work, we study counterfactual context revision as a framework for
auditing LLM-based stance simulation. Given an original online conversation, we first infer a target user's stance toward a specific topic. We then apply controlled revision strategies to the conversational context and simulate the user's stance again under the revised context. We compare text-only revision strategies with a
multimodal one that incorporates meme-based context and evaluate two main effectiveness metrics, i.e., average directional stance shift and stance transition rate. The results reveal effective and robust stance transitions in both text-only and
multimodal strategies across different polarization-preference mechanisms. Our study contributes an evaluation framework for understanding the context sensitivity of LLM-based stance simulation (https://github.com/STARResearchLab/StanceShift). More broadly, it highlights both the promise and risk of using LLMs as proxies for social media users in social simulation.
Read the full abstract ▾
All authors Xinnong Zhang, Wanting Shan, Hanjia Lyu, Zhongyu Wei, Jiebo Luo
Computation and Language
Reinforcement Learning Elicits Contextual Learning of Unseen Language Translation
Hanxu Hu, Zdeněk Šnajdr, Pinzhen Chen et al. (+2 more)
Prior work has shown that large
language models (LLMs) can translate unseen or low-resource
languages by undergoing continued training or even by encoding a grammar book in their context. However, both methods typically overfit specific
languages, with limited zero-shot transfer at test time. To translate extremely low-resource
languages at scale, we argue that LLMs must acquire the meta-skill of utilizing in-context linguistic knowledge rather than memorizing specific
languages. In this paper, we propose a reinforcement learning (RL) approach to unseen
language translation given rich linguistic context, using a surface-level translation metric (chrF) as the reward. Empirically, despite the lightweight reward, our RL-trained models effectively extract and apply relevant linguistic information from the provided context, leading to better translations on completely unseen
languages than in-context learning or supervised fine-tuning. Our analyses suggest that outcome-based RL can extend beyond conventional
reasoning tasks like math and coding to serve as a recipe for
language learning from context.
Read the full abstract ▾
All authors Hanxu Hu, Zdeněk Šnajdr, Pinzhen Chen, Jannis Vamvas, Rico Sennrich
Computation and Language
A Komi-Yazva–Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translation
Petr Parshakov
We present the first Komi-Yazva–Russian parallel corpus together with an explicit evaluation protocol for studying LLM translation in an endangered, extremely low-resource setting. The
dataset contains 457 aligned sentence pairs from 74 narrative texts and is accompanied by documented provenance, sentence-level
alignment, and story identifiers that enable leakage-aware evaluation. We use this setup to compare modern large
language models on Komi-Yazva-to-Russian translation under severe parallel-data scarcity in zero-shot and
retrieval-based few-shot regimes. The protocol includes story-level cross-validation, deterministic
retrieval for few-shot prompting, strict validation of generated outputs, complementary reference-based and judge-based metrics, and story-level uncertainty estimates. Across models, LLMs produce non-trivial translations, but performance varies strongly by model family and prompting regime.
Retrieval-based few-shot prompting consistently improves over zero-shot prompting, while gains beyond a small retrieved context remain limited. The results show that evaluative conclusions in this setting depend materially on metric choice and failure handling, so the paper frames the corpus as both a
dataset contribution and a reproducible evaluation testbed for endangered-
language machine translation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Unsupervised Skill Discovery for Agentic Data Analysis
Zhisong Qiu, Kangqi Song, Shengwei Tang et al. (+4 more)
Inference-time skill augmentation provides a lightweight way to improve data-analytic
agents by injecting reusable procedural knowledge without updating model parameters. However, discovering effective skills for data analysis remains challenging, as reliable supervision is expensive and success criteria vary across analytical formats. This raises the key question of how to discover reusable data-analysis skills from unlabeled exploration alone. We propose DataCOPE, an unsupervised verifier-guided skill discovery framework for data-analytic
agents. DataCOPE derives verifier signals from the exploration trajectories and uses them to characterize relative quality or aggreement among trajectories. It iteratively coordinates a Data-Analytic
Agent for trajectory generation, an Unsupervised Verifier for signal extraction, and a Skill Manager for contrastive skill
distillation. For report-style analysis, we instantiate the verifier as an Adaptive Checklist Verifier that derives task-specific criteria, scores reports by verifiable coverage, and iteratively refines the checklist. For
reasoning-style analysis, we instantiate it as an Answer Agreement Verifier that groups trajectories by answer agreement and uses self-consistency as an auxiliary signal. We evaluate DataCOPE on report-style analysis from Deep Data Research and
reasoning-style analysis from DABStep. Across both settings, DataCOPE consistently improves held-out performance over baselines. Averaged across four model settings, DataCOPE improves the mean score by 9.71% and 32.30% on report-style and
reasoning-style tasks respectively.
Read the full abstract ▾
All authors Zhisong Qiu, Kangqi Song, Shengwei Tang, Shuofei Qiao, Lei Liang, Huajun Chen, Shumin Deng
Computation and Language
CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments
Jiaju Chen, Bo Sun, Yuxuan Lu et al. (+3 more)
Multi-
agent systems (MAS) built on large
language models have shown growing promise, with their effectiveness resting on
agents' ability to coordinate through text-based channels much as human teams do. Yet recent study suggests that MAS often falter not because
agents lack individual task-solving ability, but because they lack collaborative competence: the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment as interaction unfolds. Decades of research in Computer-Supported Cooperative Work have characterized these requirements for human teams coordinating under constrained communication, yet existing MAS evaluations focus mainly on task outcomes or single-
agent proficiency in
reasoning, planning, and tool use. To enable a systematic analysis of
agents' collaborative competence in MAS, we introduce CollabSim, a configurable simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of
agents' internal states. Experiments across four LLMs show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of
agent design.
Read the full abstract ▾
All authors Jiaju Chen, Bo Sun, Yuxuan Lu, Yun Wang, Dakuo Wang, Bingsheng Yao
Computation and Language · Cross-listed from cs.AI
Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
Jiaju Chen, Yuxuan Lu, Jiayi Su et al. (+10 more)
Recent advances in LLM
agents have enabled complex cognitive capabilities, such as multi-step
reasoning, planning, and tool use, that increasingly position these
agents as human collaborators. Effective collaboration, however, requires collaborators to continuously maintain and align mental models of their own
reasoning,partners' intentions, and shared goals during the collaborative process. Today's
agents rarely develop such capabilities since they are primarily optimized for task completion, and the community lacks authentic human collaboration data with action-level mental model annotations that could guide
agents toward process-level collaborative competence. To bridge this gap, we present ALMANAC, a
dataset of Action-Level Mental model ANnotations for
Agent Collaboration built from the Map Task, a classic dyadic routing task from social science. ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations that record the participants' self-
reasoning, perceived partner intent, and perceived team goal. We
benchmark six LLMs on predicting humans' next-turn behavior and mental models. Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors and infer their underlying mental models.
Read the full abstract ▾
All authors Jiaju Chen, Yuxuan Lu, Jiayi Su, Chaoran Chen, Songlin Xiao, Zheng Zhang, Yun Wang, Yunyao Li, Jian Zhao, Tongshuang Wu, Toby Jia-Jun Li, Dakuo Wang, Bingsheng Yao
Computation and Language
Emergent Language as an Approach to Conscious AI
Zengqing Wu and Chuan Xiao
The question of whether artificial systems can be conscious remains open, in part because existing approaches either evaluate systems against theory-derived checklists (discriminative) or engineer consciousness-inspired modules directly (architectural); both leave open whether observed structures are artifacts of human
language priors. We propose a generative methodology: emergent
language (EL) in multi-
agent reinforcement learning, where
agents start from minimal (no
language, no concept of self, minimal exposure to human text) and develop communication under task pressure alone, ensuring causal attributability to task demands rather than inherited human
language priors. We position our methodology by discussing how EL serves as a generative tool for studying consciousness-relevant structure, including the role of environment complexity and the interpretation of emergent communication. As a proof of concept, we instantiate this methodology in a minimal environment and show that
agents develop self-referential communication, including an echo-mismatch detection
circuit that is not predicted by task structure or architecture alone but emerges from a specific environmental affordance.
Read the full abstract ▾
Computation and Language
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
Zhihao Wu, Linhai Zhang, Taiyi Wang et al. (+4 more)
Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained
reasoning tasks such as mathematics
reasoning, struggle in this setting because they do not identify where grading
reasoning goes wrong or how the model's belief about the final mark changes during
reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic
reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading
benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains. Under deterministic rubric-edit interventions, EDIT-SFT is the most rule-responsive of all evaluated systems, and EDIT-RL largely retains this responsiveness while improving accuracy.
Read the full abstract ▾
All authors Zhihao Wu, Linhai Zhang, Taiyi Wang, Runcong Zhao, Peter Andrews, Cesare Aloisi, Yulan He
Computation and Language
"Chi nas dal soch el sent de legn" – Auditing Text Corpora for Lombard
Edoardo Signoroni and Pavel Rychlý
Several of the world's
languages are still under-resourced in terms of Natural
Language Processing (NLP) tools. This is mostly due to the lack of high-quality
datasets to train, develop, and evaluate systems and models for several tasks, such as Machine Translation (MT). We conduct a manual audit of the parallel and monolingual corpora available for Lombard, an under-resourced
language continuum from Italy. Our analysis reveals that the perceived abundance of web-scraped data is an illusion, with massive
datasets plagued by severe
language misidentification, boilerplate text, and non-linguistic noise. Furthermore, we analyze the orthographic composition of the valid Lombard portions across web-scraped
datasets, curated corpora, and
benchmarks. Our findings show conflicting orthographical systems and severe representational bias across all corpora: high-quality data is heavily skewed towards Western Lombard varieties, with Eastern ones left on the margins. This underscores the need for variety-aware, community-driven data curation rather than purely quantity-driven scraping.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance
Gizem Yüce, Giorgos Nikolaou and Nicolas Flammarion
Machine unlearning aims to remove targeted knowledge from a trained model while preserving its general capabilities. For autoregressive
language models, not all
tokens in a forget sample are equally relevant to forgetting. Existing approaches either ignore this heterogeneity or rely on auxiliary models, heuristics, or external annotations to estimate each
token's relevance for forgetting. We instead characterize it through the interaction with the retain objective: a
token is forget-specific to the extent that minimizing the forget loss on that
token does not conflict with retain optimality. We formalize this perspective as a joint optimization problem over the model parameters and the
token weights and show that, under a natural separation condition, the resulting objective recovers the oracle forget-specific
token support. Motivated by this formulation, we introduce Alternating
Token-Weighted Unlearning (ATWU), a lightweight framework that jointly learns
token forget-specificity and model parameters during unlearning using a simple linear scorer over the hidden states, without external
token level supervision. Across TOFU and RWKU, ATWU achieves state of the art forget-retain trade-offs, outperforming sample-level methods, probability-based
token weighting heuristics, and auxiliary-model-based approaches. Moreover, the learned scores align substantially better with ground truth forget-specific spans, indicating that ATWU identifies semantically meaningful
token level forgetting signals. Overall, our results suggest that retain conflict provides an effective criterion for identifying what
language models should forget, enabling unsupervised learning of
token level forget-specificity directly from model representations with minimal computational overhead.
Read the full abstract ▾
Computation and Language
Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
Victor De Marez, Luna De Bruyne and Walter Daelemans
Factual sycophancy occurs when a
language model abandons a correct, verifiable answer under social pressure. Because a flip occurs only when pressure toward a false answer exceeds the model's neutral preference for the truth, flip rates conflate two mechanisms: the strength of that baseline preference (truth margin), and how far pressure shifts it (manipulation sensitivity). We decompose factual sycophancy into these channels and use them to separate the effects of size and instruction tuning across 56 open-weight models spanning 0.3B-32B parameters and 13 manipulation types. We find that vulnerability is governed mainly by size, but instruction tuning changes how size acts: small instruction-tuned models can become less robust, whereas large instruction-tuned models usually become more robust. Instruction tuning primarily increases truth margin, but its behavioral effect depends on manipulation type. Scaling also changes the two channels differently: base models gain margin but become mildly more manipulation-sensitive, whereas instruction-tuned models gain margin faster and become less sensitive. Factual sycophancy is therefore not a single scalar property. Evaluations should report channel-specific, manipulation-specific, and size-conditioned robustness rather than flip rates alone.
Read the full abstract ▾
Computation and Language
LLMs Can Leak Training Data But Do They Want To? Evaluating Memorization Propensity in LLMs
Gianluca Barmina, Peter Schneider-Kamp and Lukas Galke Poech
Large
language models can reproduce training data, but memorization evaluations mostly measure whether models can be made to do so, not whether they do so without
adversarial prompting. We introduce PropMe, a framework that pairs a capability evaluation (prefix attacks) with non-
adversarial propensity evaluation, and a propensity metric to relate the two. We check every generation against the model's full indexed training corpus with SimpleTrace, an open-source batch implementation of OLMoTrace span matching. We evaluate Comma, its continually pre-trained descendant DFM Decoder, and OLMo3 32B on Common Pile, Dynaword, and Dolma3, with four non-
adversarial settings ranging from no
prompt to sentences from unseen in-domain sources. Our results show that natural sentence
prompts reproduce far less training text than prefix attacks. Generation with no
prompt or a single common word, however, reproduces about as much as prefix attacks, or more, in three out of four model-corpus pairs; most of this text is duplicated licenses, code, and templates. In our setting, continual pre-training reduces long verbatim reproduction of the original corpus only modestly. More broadly, we argue that by testing both capability and propensity, we can attain a more comprehensive picture of a model's memorization behavior.
Read the full abstract ▾
Computation and Language
FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays
Yijun Liu, Yifan Song, John Gallagher et al. (+2 more)
While large
language models (LLMs) are increasingly used to generate writing feedback, there remains no systematic comparison of LLM and expert feedback on the dimensions that writing research identifies as central to revision: goal-orientation, anchoring to specific sentences, and prioritization. We introduce FOXGLOVE, a
dataset of 696 feedback comments written by trained writing instructors on 69 twelfth-grade argumentative essays, paired with 1,644 comments generated from four frontier LLMs under a shared protocol, totaling 2,340 comments. We provide expert quality ratings on a subset of both instructor and LLM comments. We find that instructors and LLMs distribute feedback similarly across goals and essay positions, yet instructors and models diverge on the specific sentences on which to provide feedback. Additionally, we find that models tend to write more complex feedback and use fewer questions than instructors. LLM feedback also receives higher ratings on most dimensions of quality, as rated by instructors, but much of this advantage appears to be attributable to lengthier comments. FOXGLOVE enables systematic comparison of where human and LLM feedback align, diverge, and differ.
Read the full abstract ▾
All authors Yijun Liu, Yifan Song, John Gallagher, Sarah Sterman, Tal August
Computation and Language
Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery
Alireza Bayat Makou, Jingcheng Niu, Subhabrata Dutta et al. (+1 more)
Circuit discovery methods identify subgraphs that explain model behaviors, and structural differences between discovered
circuits are commonly interpreted as evidence of distinct mechanisms. We test this assumption by varying input-
token frequency while holding the task fixed. The discovered
circuits appear specialized by frequency when compared structurally, but functional and representational analyses show no reliable evidence of corresponding differences. We term this mismatch phantom specialization. Using the Literal Sequence Copying task across four frequency bands plus a frequency-weighted control, we extract 75
circuits from five Pythia models (70M-1.4B). We find that structurally distinct
circuits implement the same computation: band-specific edges transfer broadly across bands, a core shared across most bands recovers at least 99% of
circuit performance in models above 70M, and causal interchange interventions confirm that internal representations are interchangeable across frequency bands. A smaller subject-verb agreement replication shows the same pattern:
circuits differ structurally but transfer broadly across bands, and the core shared by most bands recovers nearly all
circuit accuracy. Repeated extractions within the same band further suggest that discovery
algorithms sample from an equivalence class of valid subgraphs rather than recovering a unique mechanism. Standard evaluation practice obscures this pattern: source-level evaluation inflates apparent faithfulness, while edge-level evaluation reveals the many-to-one mapping from structure to function. We do not test specialization at the level of
token positions or features inside a component. Our results show that structural differences between
circuits are not sufficient evidence for distinct mechanisms, and that exposing this requires edge-level evaluation and cross-condition transfer tests.
Read the full abstract ▾
All authors Alireza Bayat Makou, Jingcheng Niu, Subhabrata Dutta, Iryna Gurevych