Computation and Language
AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic
Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian et al. (+1 more)
As Large
Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic
Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic
reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most
benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static
benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found a high degree of correlation with existing, hand-crafted Arabic-centric
benchmarks, confirming the potential of our dynamic approach.
Read the full abstract ▾
All authors Ignacio Iacobacci, Faroq Altam, Zhaozhi Qian, Muhammad Alqurishi
Computation and Language · Cross-listed from cs.LG
LLMs are General Asynchronous Agents
George Yakushev, Denis Mazur, Vladimir Bartenev et al. (+4 more)
Modern LLMs are increasingly capable as autonomous
agents, but they follow sequential interaction cycles: read, think, reply or call tools, repeat. Many real-world use cases are not sequential: voice assistants, embodied
agents, and monitoring systems receive new inputs while they think or perform another task. Modern LLMs address this with specialized architectures for voice interaction and
video streams, VLAs for
robot control, asynchronous tool calling for API usage, and others. In this work, we generalize from different asynchronous tasks to general asynchronous
agents that can adapt to different types of concurrency. To achieve this, we develop an asynchronous LLM framework that lets users (or the
agents themselves) define inference coroutines with overlapping
memory states. We showcase that Qwen 3.x models are capable of asynchronous operation for streaming
video understanding, videogames, and monitoring, without task-specific training.
Read the full abstract ▾
All authors George Yakushev, Denis Mazur, Vladimir Bartenev, Vyacheslav Zhdanovskiy, Timofey Byzov, Vladimir Kaurkin, Vadim Pastushenko
Computation and Language · Cross-listed from cs.LG
Frontier Learning: Training LLM Reasoners at the Edge of Capability
Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic et al. (+1 more)
Reinforcement Learning-based post-training of Large
Language Models (LLM) has been successfully applied to improve their
reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving
reasoning capabilities. Across several
reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.
Read the full abstract ▾
All authors Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic, Aurelien Lucchi
Computation and Language · Cross-listed from cs.PL
Semantic Prefix Oracles for LLM Decoding: Contracts and Differential Validation
Paul Kronlund-Drouault
Constrained decoding can enforce regular or context-free output formats, but many program-generation failures are semantic: scope, typing, and declaration effects depend on context. We present semantic grammar specifications, a declarative formalism that attaches such constraints to a context-free surface and executes them during Earley descent. Our implementation enforces safe
pruning: it rejects only prefixes whose semantic contradictions cannot be repaired by any continuation. A separate, grammar-dependent, dead-end freedom property guarantees the existence of a realizable witness for each remaining branch. We give simple sufficient conditions based on surface productivity, type coverage, and left-to-right constraint flow. Our finite-lambda, core ML, and C-like fragments satisfy them, while the STLC instance used in our experiments does not: plain STLC can violate type coverage, and we show how restricting its type universe recovers it. A tokenizer-lifting lemma carries character-level witnesses to
token sequences under an explicit vocabulary-coverage hypothesis. We validate the implementation differentially against production compilers (ocamlc, cc). Across every prefix of 65 compiler-valid programs we observe zero false prunes. The semantic oracle localizes 25/30 invalid programs mid-stream, against 0/30 for a syntax-only oracle, and agrees on 42/42 recursion probes. A twelve-model generation study, including a matched semantic-versus-syntactic ablation for nine models, finds nonnegative observed semantic-minus-syntactic point estimates for every model-
language pair, with maxima of $+15.2$ points on STLC task correctness and $+14.3$ points on ML validity.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Self-Adapting Group of Experts for Multi-Agent Reasoning
Mohammad Atif Quamar, Nurbek Tastan, Karthik Nandakumar et al. (+1 more)
Multi-
agent systems bring together
language model
agents with different roles to propose, review, and refine solutions. Each
agent's response depends on its model's capabilities, the
reasoning strategy defined by its system
prompt, and the information in its input context. Existing frameworks often adapt communication by changing this context while leaving individual
prompts fixed, even when a problem calls for different skills. We study whether
agents' initial responses can identify a strategy better suited to the current problem and guide its transfer to other
agents. To address this, we introduce SAGE (Self-Adapting Group of Experts), a training-free framework that uses answer agreement, prefix consistency, and reciprocal peer review to select a strategy donor. SAGE transfers the selected donor's
reasoning strategy to the other
agents while preserving their original roles. This transfer uses only the
agents' original system
prompts, without access to the problem or generated solutions. After strategy adaptation,
agents exchange responses through a dynamic, sparse directed acyclic graph that routes information from higher-scoring
agents to lower-scoring
agents. Experiments across multiple
agent backbones and
reasoning benchmarks show that SAGE achieves higher average accuracy than the evaluated baselines. Our code is available at https://github.com/atifquamar07/sage.
Read the full abstract ▾
All authors Mohammad Atif Quamar, Nurbek Tastan, Karthik Nandakumar, Junpei Komiyama
Computation and Language
AwarenessBench: Assessing Cognitive Capabilities of Language Models
Xiaojian Li, Rongwu Xu, Tianyun Zhang et al. (+9 more)
As
language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive
benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in
language modeling or
reasoning does not necessarily translate into improved cognition.
Read the full abstract ▾
All authors Xiaojian Li, Rongwu Xu, Tianyun Zhang, Yue Wang, Shuo Chen, Qiner Lyu, Briana Zhang, Peiran Yang, Kyle Xue Chen, Haoyuan Shi, Yu Wang, Wei Xu
Computation and Language
TRACE: Single-Pass Decoding-Trace Risk Localization for Generation Calibration
Yuebin Xu, Xuemei Peng, Junlan Chen et al. (+2 more)
Reliable confidence estimation is essential for large
language model deployment. However, answer-level calibration remains challenging because generation errors are often localized: a response may be fluent and high-probability overall while still failing at a critical number, entity, or factual claim. Existing
estimators compress
token probabilities, sequence likelihoods,
entropy, or beam statistics into a global score, which can dilute such local risk signals. We propose TRACE, a single-pass, decoded-answer-preserving confidence
estimator that treats decoding-time uncertainty as a trajectory through three steps: (i) recording
token-level surprisal and predictive
entropy during decoding, (ii) applying local risk operators to preserve uncertainty spikes, and (iii) converting localized trace risk into answer-level confidence. TRACE produces a label-free risk score, while TRACE+ calibrates trace-only features into probabilities using a held-out split, without extra generations or external verifiers. We evaluate four tasks against 19 calibration baselines, and TRACE+ reduces Brier from 0.149 to 0.137 and improves AUROC from 0.758 to 0.792 over the strongest likelihood baseline. Across seven LLMs, TRACE+ improves over the best non-TRACE baseline pool from 0.136 to 0.120 Brier and from 0.764 to 0.817 AUROC. Results show that localizing decoding-time risk provides a general approach to calibration.
Read the full abstract ▾
All authors Yuebin Xu, Xuemei Peng, Junlan Chen, Zhiyi Chen, Zeyi Wen
Computation and Language
Multilinguality in Hybrid Attention LLMs
Lucas Bandarkar, Junlin Hu, Chenyuan Yang et al. (+2 more)
In response to the growing demand for long sequences in agentic and
reasoning use cases, many state-of-the-art LLMs combine multiple variants of
attention to mitigate the quadratic complexity of traditional softmax
attention. These hybrid
attention LLMs aim to balance the strengths and limitations of full
attention and alternatives based on recurrence. This work presents a first study of how hybrid
attention impacts the multilinguality of LLMs. Beyond the impact on long sequences in poorly tokenized
languages, our study is motivated by the possibility that the inductive biases of the recurrent state alter linguistic processing. Our interpretability analysis confirms this, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-
attention layers. Across diverse models, we notably observe a pronounced spike in cross-lingual
alignment around the first full-
attention layer. These findings lead us to question the conventional ordering of
attention layers. In
distillation experiments on multilingual data, all alternative layer orderings outperform the standard throughout training, learning up to 2.5X faster. These stark, replicable results
prompt our theory that multilingual models would benefit from starting with a full-
attention layer rather than recurrent layers.
Read the full abstract ▾
All authors Lucas Bandarkar, Junlin Hu, Chenyuan Yang, Mohsen Fayyaz, Nanyun Peng
Computation and Language
How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?
Momoka Furuhashi, Kouta Nakayama, Takashi Kodama et al. (+2 more)
While recent studies have explored human behavior and preference simulation using large
language models (LLMs), it remains unclear how well LLMs can simulate subjective evaluations from real learners in educational settings. We investigate this question using real learner evaluation data on feedback for high-school biology questions at both the group and individual levels. We compare performance with and without learner-specific information, such as personality traits and evaluation examples, across six models. Our results show that LLMs still have a limited ability to simulate learner evaluations. Providing learner profiles and examples improves score calibration and individual-level simulation, but more often fails to improve group-level consistency. These findings highlight the need to investigate which learner information and adaptation strategies are effective for learner preference simulation.
Read the full abstract ▾
All authors Momoka Furuhashi, Kouta Nakayama, Takashi Kodama, Saku Sugawara, Kyosuke Takami
Computation and Language
Deep Learning Methods in Neuroscience: From Modeling Molecular Mechanisms to Classifying States of Consciousness
Elena Benderskaya, Anastasiia Alifanova, Svetlana Batalova et al. (+2 more)
A critical analysis of contemporary approaches to the study of conscious states. The review focuses on methods of classification, clustering, modeling of
brain states under anesthesia and identification of measurable neurobiological characteristics of
brain function. A comparative analysis was conducted in the following three major areas: automatic detection of states of consciousness using neural networks based on
EEG and fMRI data; modeling of the structural-functional dynamics of the
brain under the effects of anesthetics; and detection of neurophysiological indicators which correlate with the level of consciousness. The obtained conclusions demonstrate the growing effectiveness of deep neural models in the classification and prediction of
brain states and the analysis of dynamic structural-functional connectivity. Nonetheless, significant limitations were also identified, including the limited interpretability of the models, the lack of standardized metrics, and the problem of the specificity of consciousness markers. Our findings support the need for developing hybrid, generalizible, physiologically grounded architectures. Furthermore, such approaches may improve the translational potential of computational models in clinical neuroscience. Diverse methods of machine and computational modeling have demonstrated their effectiveness in tasks of automatic clustering and classification of
brain states, the development of multilevel models and the identification of connectivity patterns correlated with levels of consciousness. A larger-scale analysis and a larger
dataset, as well as the implementation of model interpretability approaches are required for the practical application of the analyzed models. The models based on
EEG and LFP are the most promising for clinical application due to their availability and the possibility of real-time monitoring.
Read the full abstract ▾
All authors Elena Benderskaya, Anastasiia Alifanova, Svetlana Batalova, Vasilisa Zhuk, Anna Kovalenko
Computation and Language
From Input to Output: A Flexible Agent for Dual-End Interpretation of Sparse Autoencoder Features
Dewen Liu, Zixuan Li, Jonathan Pan et al. (+4 more)
Sparse autoencoders (SAEs) are an important tool for mechanistic interpretability, but interpreting their many features remains challenging. Existing methods characterize input-side activation patterns and output-side intervention effects, yet often leave their functional connection implicit, while input-side evidence collection typically relies on costly large-corpus scans. We introduce functional interpretation, which characterizes an SAE feature as a mapping from its activating input semantics to its output effects under intervention, and present Dual-End Agentic Feature Interpretation (DAFI), an
agent that actively gathers evidence and refines input-side, output-side, and functional interpretations through component-specific feedback. Its short-context
token probing enables on-demand activation evidence collection without a full corpus scan. On GemmaScope, DAFI improves Input score by 13.1 percentage points over SAGE and Output score by 38.9 points over
Token Change, while being substantially more
token-efficient than a general-purpose coding
agent. Skills distilled from successful refinements raise the held-out joint pass rate from 58.0% to 92.0% and improve both interpretation quality and efficiency when transferred to a new model-SAE setting. Across features with reliable endpoint interpretations, 70.7% exhibit non-equivalent input and output semantics. On AxBench, DAFI also improves steering-feature selection over output-score filtering. Code is available at https://github.com/THUAIS-Lab/DAFI.
Read the full abstract ▾
All authors Dewen Liu, Zixuan Li, Jonathan Pan, Zhao Wu, Zijun Yao, Juanzi Li, Xiaozhi Wang
Computation and Language · Cross-listed from cs.SE
Do Coding Agents Reuse Existing Code or Reinvent the Wheel?
Dongsheng Ma, Sizhe Wang, Xinyi Huang et al. (+5 more)
Coding
agents are increasingly deployed for iterative development on real repositories, yet existing evaluation barely answers a basic question: do coding
agents reuse existing code or reinvent the wheel? The question matters: every duplicated implementation is a fix applied twice and
agents produce code far faster than humans can audit, so redundancy accumulates unsupervised. Thus, we present RepoReuse, a multi-turn
benchmark for auditing code reuse in real repositories, where requirements are revealed turn by turn and the workspace accumulates across turns. It is built by a fully automated pipeline combining AST-based dependency graphs, guided evidence collection, and execution-verified task synthesis, and scales readily to new repositories. Beyond pass rates, we measure the reuse rate together with recall and cross-turn structural redundancy. An audit over 3{,}000 turns shows that
agents progressively stop exploring relevant repository code, reuse their own history less even when it is fully in the workspace, and leave duplicated logic in 50.8% of task chains by turn~5—all while pass rates barely move. Such deficiencies are invisible to pass rates, underscoring the need to evaluate code generation beyond functional correctness.
Read the full abstract ▾
All authors Dongsheng Ma, Sizhe Wang, Xinyi Huang, Zhengren Wang, Yuhan Wang, Luyang Si, Xincheng Wei, Wentao Zhang
Computation and Language · Cross-listed from cs.AI
Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models
Lucas Biechy, Cédric Eichler, Adrien Boiret et al. (+1 more)
While Large
Reasoning Models (LRMs) excel at complex
reasoning,
alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that
alignment suppresses useful output variability. We introduce
prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3
datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-
dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.
Read the full abstract ▾
All authors Lucas Biechy, Cédric Eichler, Adrien Boiret, Nicolas Anciaux
Computation and Language
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
Zineddine Tighidet, Andrea Mogini, Jiali Mei et al. (+2 more)
Large
Language Models (LLMs) perform well on
reasoning benchmarks, but it remains unclear whether this reflects genuine contextual
reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad memorization bias, where familiar content improves
reasoning performance, and the Strong Parametric Shortcut Hypothesis, where models skip
reasoning entirely and recall stored answers. To test these effects, we introduce MemoReason, a human-curated
benchmark that pairs factual
reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified
reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric
memory affects
reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric
memory influences
reasoning through mechanisms more complex than simple factual recall. MemoReason provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader
reasoning settings.
Read the full abstract ▾
All authors Zineddine Tighidet, Andrea Mogini, Jiali Mei, Patrick Gallinari, Benjamin Piwowarski
Computation and Language
Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation
Fahrell Giovanny, Geby Bayuningtyas, Sahrul Mukharom et al. (+1 more)
Large
language models treat conversation history as unverified context, so false premises injected into earlier turns can be adopted as fact later, a failure mode we call session-level contamination. We introduce five contamination protocols along a source-authority gradient, each holding the false premise constant while varying how it is presented, from self-attributed falsehood to explicit instruction override. We evaluated three architecturally diverse models, GPT-5.4 Mini, Gemini-3.1 Flash-Lite, and GLM-4.5-Air, on ten cases spanning five knowledge fields at temperature zero over 22,500 turns, using a dual-track judge checked against a 120-turn human-annotated sample (Cohen's kappa = 1.000 for binary adoption, a ceiling on this sample, and 0.92 linear-weighted for collapse severity). The three models implemented structurally different epistemic policies. GPT-5.4 Mini recorded zero adoptions across all 500 sessions;
token-level probing of the GPT-5.4 base model left the correct
token as the argmax in eight of ten conditions. Gemini-3.1 Flash-Lite showed a steep framing-dependent policy: 0.1 percent for self-attributed falsehoods, 23.5 percent for user-cited sources, 68.2 percent for system-injected authority, and 94.0 percent for instruction override. GLM-4.5-Air showed a shallower gradient (15.8 and 84.2 percent for those two vectors). This 68-percentage-point dissociation is consistent with authority deference and instruction compliance engaging distinct mechanisms within a single architecture. Recovery also differed: GLM recovered in 94.5 percent of affected sessions, whereas 26.1 percent of affected Gemini sessions did not return to factual accuracy within the window (40.0 percent under instruction override). We release the evaluation framework and taxonomy for reproducible testing.
Read the full abstract ▾
All authors Fahrell Giovanny, Geby Bayuningtyas, Sahrul Mukharom, Hafiz Budi Firmansyah
Computation and Language
Decide, Don't Generate: Competitive Dimensional ABSA with Jev's Typed Decisions
Yiqun Zhang, Peidong Wang, Zihan Wang et al. (+1 more)
Aspect-based sentiment analysis (ABSA) has largely turned to text generation. We show that competitive dimensional ABSA does not need it. Using Jev, a frozen model that answers typed questions with rubric scores, label probabilities, and yes/no judgments, we decompose all three tasks of SemEval-2026 Task III Track A into such decisions and align them with the annotation scheme through 488 coefficients fitted on CPU, with no text generation and no backbone tuning. On valence-arousal regression over ten corpora in six
languages, the system reaches 1.0645 RMSE, the lowest aggregate error of any participating system. On triplet and quadruplet extraction, it reaches 52.09 and 44.06 continuous F1, above fine-tuned Llama-3.3-70B and GPT-OSS-120B baselines. Analyses and ablations show where the accuracy comes from: supervised calibration roughly halves the raw regression error, exact valence-arousal would add only 4.5 F1 to extraction, and the learned combination of span-boundary evidence, not any single signal, carries the extraction systems.
Read the full abstract ▾
All authors Yiqun Zhang, Peidong Wang, Zihan Wang, Shi Feng
Computation and Language · Cross-listed from cs.AI
EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning
Shihan Dou, Shaofan Liu, Zhonghang Lu et al. (+8 more)
Recent work has explored improving
agents by jointly evolving their harnesses and models, but often takes a ”potpourri” approach that bundles together new tools, new decision-making procedures, and model adaptation to the evolved harness under a single notion of
agent improvement. In this paper, we instead investigate how
agents can improve their decision-making procedures. In particular, we propose EvoIn, an
agent fine-tuning framework that bridges
evolution and internalization. EvoIn first analyzes
agent execution traces to evolve and validate new decision-making procedures by temporarily instantiating them in the harness. The validated procedures guide the
agent to generate improved
reasoning traces. These traces are then rewritten into self-contained
reasoning traces, removing explicit references to harness instructions while expressing the induced decision logic as the model's own
reasoning. Finally, EvoIn fine-tunes the model on the rewritten traces, internalizing these procedures so that the improved decision-making persists without the evolved harness at inference time. We evaluate EvoIn on diverse
benchmarks and find that it consistently enables
agents to learn stronger decision-making procedures, raising the pass rate by 10.9 points in-domain and by 9.2 points out-of-domain. Results further show that the internalized decision procedures generalize to unseen tasks. Case studies show that
agents can learn to decide how to solve a task before solving it, for example by checking a document's length to choose between reading it in full and searching it. EvoIn is also broadly applicable, showing consistent improvements on another model family.
Read the full abstract ▾
All authors Shihan Dou, Shaofan Liu, Zhonghang Lu, Jiahang Lin, Shichun Liu, Binghai Wang, Jiajie Jin, Guanting Dong, Tao Gui, Qi Zhang, Xuanjing Huang
Computation and Language
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Xin Li, Mengbing Liu and Chau Yuen
Multi-
agent large
language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On 6,925 MMLU-Pro debates, the protocol identifies 253 collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents 29 collapses but loses 108 corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate 8-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high (G=7, Spearman rho=0.893, exact two-sided p=0.0123), but initial-majority accuracy is a close comparator (rho=0.821; family partial rho=0.767, p=0.0877), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model-scaffold rows can be compared under the same denominators and signed utility ledger.
Read the full abstract ▾
Computation and Language
When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
Yizhou Fang, Siyue Chen, Zimo Qi et al. (+3 more)
Despite substantial progress across downstream applications,
vision-
language models (VLMs) remain susceptible to
language bias, often prioritizing linguistic cues over visual evidence and consequently producing incorrect predictions. Prior studies have proposed various approaches to understanding and mitigating
language bias in VLMs, yet their findings often conflict due to the difficulty of tracing how
language bias propagates within black-box VLMs. Building on the word completion task, we trace how
language bias propagates through VLM inference by (1) proposing a diagnostic framework that decomposes the inference process into four distinct yet interdependent stages to trace the propagation of
language bias; and (2) examining how two key factors underlying
language bias, i.e., linguistic priors and cross-modal coverage, evolve across these stages and ultimately give rise to incorrect predictions. The linguistic prior captures the strength of statistical bias induced by the
language model component of a VLM and represents the origin of
language bias, whereas cross-modal coverage measures the extent to which linguistic cues cover the visual content. By decomposing inference into four stages and characterizing the interplay between linguistic priors and cross-modal coverage across these stages, we propose a systematic framework for tracing the propagation of
language bias throughout the inference process; and uncover the underlying mechanism of
language bias by revealing the interplay between linguistic priors and cross-modal coverage.
Read the full abstract ▾
All authors Yizhou Fang, Siyue Chen, Zimo Qi, Zhiyu Xue, Xi Chen, Guangliang Liu
Computation and Language
Rubric-Aware On-Policy Self-Distillation for LLM Personalization
Yilun Qiu, Xiaoyan Zhao, Chengbing Wang et al. (+6 more)
LLM personalization aims to generate responses aligned with individual users' preferences and needs. User-specific rubrics make these expectations explicit, providing direct supervision on what a satisfactory answer should cover. Existing rubric-guided approaches, however, exploit such guidance only at a coarse granularity, either by using rubrics to supervise the prediction of relevant aspects for subsequent generation or by reducing aspect coverage to a single response-level reward for reinforcement learning. This leaves a gap between specifying what a personalized answer should contain and teaching the model how to generate it. To bridge this gap, we propose GRASP, a rubric-aware on-policy self-
distillation framework for LLM personalization that turns user-specific rubric aspects into fine-grained,
token-level supervision. Specifically, GRASP pairs a rubric-free student with a rubric-informed teacher that additionally receives the target user-specific rubrics. By aligning their next-
token distributions along on-policy trajectories generated by the student, GRASP transfers the teacher's rubric-conditioned guidance into the student, translating user-specific semantic requirements into dense
token-level supervision. Since rubric-informed teachers can still produce inadequate supervision, we further introduce Rubric-based Teacher Validation (RTV), which retains only instances where the teacher sufficiently covers the target aspects, improving both supervision quality and training efficiency. Experiments on the LaMP-QA
benchmark for personalized question answering demonstrate that GRASP achieves state-of-the-art performance across multiple backbones, supporting the effectiveness of rubric-guided
token-level supervision for personalization. To ensure reproducibility, our code is available at https://github.com/SnowCharmQ/GRASP.
Read the full abstract ▾
All authors Yilun Qiu, Xiaoyan Zhao, Chengbing Wang, Cilin Yan, Rui Zu, Wanyang Zhang, Xiaolong Jiang, Jiayin Cai, Yang Zhang
Computation and Language
SCBO: Semantically Coherent Batching and Ordering for LLM-Based Social Surveys
Yuanzi Li, Lingjie Wang, Zihang Tian et al. (+2 more)
Large
Language Models (LLMs) offer a scalable way to simulate survey respondents using demographic profiles and observed reference responses. However, the conventional approach of predicting one question per
prompt repeatedly encodes the same context, limits each target to a narrow set of reference responses, and prevents later predictions from using information in earlier answers. Predicting multiple questions in one
prompt can reduce these costs, share a broader pool of references, and let later predictions build on earlier ones. This requires forming coherent batches, selecting shared references, and ordering questions and references effectively. We propose Semantically Coherent Batching and Ordering (SCBO), a training-free framework that addresses these challenges. SCBO first uses an LLM to extract compact semantic representations from survey items and filter out template noise. It then groups related questions into batches and builds a shared reference bank using target-specific
retrieval and centroid-based completion. Finally, it orders target questions from easy to hard and arranges references according to their semantic
alignment with those questions. Experiments on four large-scale survey
datasets and four LLMs show that SCBO substantially reduces
token consumption and inference time while generally improving prediction accuracy over a non-batched baseline. Code is available at https://anonymous.4open.science/r/SCBO-41D8.
Read the full abstract ▾
All authors Yuanzi Li, Lingjie Wang, Zihang Tian, Lei Wang, Xu Chen
Computation and Language
SignFLIP: A Unified Model for Sign Language Translation and Generation via Stage-wise Alignment at Scale
Zhaoyi An, Sihan Tan, Youngbae Hwang et al. (+2 more)
Sign
language translation and generation share the goal of bidirectional
alignment between text and sign representations. However, existing approaches either treat them as isolated tasks or are only verified on limited
datasets, limiting effective modeling between modalities. In this paper, we propose SignFLIP, a unified LLM-centered framework for translation and generation. To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data. The shared sign–text representation is progressively refined: pre-
alignment facilitates subsequent SLT, while the SLT-adapted representation further benefits SLG. Extensive experiments on multiple
benchmarks show that SignFLIP shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to sign
language recognition.
Read the full abstract ▾
All authors Zhaoyi An, Sihan Tan, Youngbae Hwang, Kazuhiro Nakadai, Rei Kawakami
Computation and Language · Cross-listed from cs.LG
TANGO: Watermarking Masked Diffusion Language Models in Token Pairs
Kasra Arabi, Nir Weinberger, Micah Goldblum et al. (+1 more)
Masked-
diffusion language models fill in masked positions in parallel and in no fixed order. Most practical text watermarks assume left-to-right generation. They key each
token to the
tokens before it, and in a
diffusion model those
tokens may still be masked. A fixed green list needs no such context, but it favors the same
tokens at every position, so these
tokens appear more often in watermarked text. An attacker who compares
token frequencies in watermarked and unwatermarked text can recover the list and forge text that the provider's own detector accepts. We present TANGO, a watermark for masked-
diffusion language models that keys each new
token to a nearby
token that is already unmasked. A secret key splits the vocabulary into color classes, and TANGO biases the new
token toward a color determined by the key and the nearby
token's color. The watermark is therefore embedded in pairs of
tokens. Because the favored color changes from position to position,
token frequencies stay much closer to those of unwatermarked text than under a fixed green list. Detection needs only the text and the key, and it does not assume any unmasking order. On two masked-
diffusion models, TANGO detects nearly all unedited watermarked texts and most edited ones, and frequency attacks that forge the fixed green list fail against it.
Read the full abstract ▾
All authors Kasra Arabi, Nir Weinberger, Micah Goldblum, Niv Cohen
Computation and Language
Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders
Zichao Yu, Qianshuo Ye, Xu Wang et al. (+1 more)
On-policy
distillation (OPD) is a widely adopted post-training technique for LLM
reasoning. It is commonly believed to transfer knowledge from a stronger teacher, yet what OPD actually distills into the student's internal representations remains unclear. We study this question with sparse crosscoders, which learn one feature dictionary shared by the student before and after OPD and the teacher. Standard crosscoder analyses, however, identify model-specific features but cannot tell how a model's use of its features changes, since all models are encoded into one set of feature activations. We therefore propose the swap readout, which reads each student checkpoint's feature activations on its own, measuring how training changes the student's use of each feature, even for checkpoints unseen by the crosscoder. Across three OPD settings, we find that OPD neither creates features nor passes on the teacher's own, and leaves the firing rates of over 98% of the student's frequently used features within 20%. We further examine the SFT warm-up on the teacher's rollouts that commonly precedes OPD and makes it more effective. Rather than adding features, the warm-up reweights the shared ones in two ways. First, it already raises and lowers many of the features that OPD later raises and lowers, doing part of OPD's work in advance. Second, it changes features that OPD alone would not, notably those for conversation format,
reasoning style, and mathematical notation, and these changes persist through OPD. Imposing this reweighting on a directly distilled student's features, without changing its weights, brings its accuracy close to that of the warmed-up student, whereas the same change on shuffled features does not. Together, these findings suggest that OPD reweights existing features rather than acquiring new ones: the student learns from the teacher how to use the features they already share.
Read the full abstract ▾
All authors Zichao Yu, Qianshuo Ye, Xu Wang, Difan Zou
Computation and Language
From Normative Frameworks to Alignment Data: Constructing and Evaluating SFT and Preference Data
Husrev Taha Sencar, Rezart Beka, Danish Naeem et al. (+6 more)
Aligning
language models with a specified normative framework requires translating abstract principles into concrete examples and preference signals from which models can learn. We present an expert-driven methodology for constructing such
alignment data and apply it to a normative framework grounded in Islamic ethical, theological, and jurisprudential traditions. Over approximately one year, seven domain experts systematically probed
language models to identify
alignment deficiencies, curated desired responses, and constructed preference pairs from model outputs and expert judgments. The resulting Arabic-English
datasets contain approximately 2.8K supervised fine-tuning (SFT) examples and 5.4K preference pairs spanning a broad range of normative domains. We evaluate the
datasets through controlled post-training experiments comparing a Baseline model with models incorporating the curated SFT data alone and both the SFT and preference data. In blind expert evaluation on 150 separately constructed
prompts, the model trained with the curated SFT data was preferred over the Baseline in 51.3% of assessor judgments, compared with 14.4% in the opposite direction (p < .001 at the
prompt level). Adding the preference data resulted in a smaller difference, with the model trained with both
datasets preferred over the SFT model in 28.0% of judgments versus 20.9% in the opposite direction; this difference was not statistically significant at the
prompt level (p = .166). Standard Arabic and English
benchmarks show no broad degradation in general-purpose capabilities. These results demonstrate how expert-defined normative principles can be systematically operationalized into
alignment data and evaluated through controlled model training.
Read the full abstract ▾
All authors Husrev Taha Sencar, Rezart Beka, Danish Naeem, Seda Ozalkan, Majd Hawasly, Ji Lucas, Ala AlFuqaha, Mohamed Abdallah, Recep Senturk
Computation and Language · Cross-listed from cs.AI
5W1H+Which: Context-Valid Semantic Indexing with Progressive Ontology Binding
Yaxiao Liu, Pengbo Liu, Yiwen Liu et al. (+2 more)
Transforming raw data into queryable knowledge requires both early extraction of reusable information and explicit types, relations, and applicability conditions for particular tasks. If indexing selects content too early around a single business schema, later tasks may be unable to use information that was omitted. If the index retains only open-ended text, however, rule-based
reasoning lacks checkable premises. We propose 5W1H+Which, a semantic indexing design that separates content extraction from ontology binding. The 5W1H questions organize source-grounded content units; Which points to versioned ontology elements and records mapping relations, scope, and validation status. Time, location, system environment, and participant roles are not merely
retrieval labels: together, they constrain the contexts in which facts, bindings, and rules apply. Unbound content remains searchable, while bound content enters a formal
reasoning path only after premise checks. The method further distinguishes business valid time, system knowledge time, and operational traces, and uses dependency records to support binding revalidation and the maintenance of derived conclusions. A worked example of migration from an on-premises server to a cloud environment illustrates the different treatment of world-state changes, ontology-version changes, and changes in rule applicability. We formulate three groups of falsifiable hypotheses concerning cross-task evidence coverage, control of contextual misuse, and incremental update cost. The planned evaluation includes a strong typed fact-graph baseline with the same evidence, temporal information, and budget, to test whether benefits arise from 5W1H organization, deferred binding, or additional information and engineering effort. The contribution is a testable indexing mechanism, not a claim to a new universal ontology or a demonstrated performance advantage.
Read the full abstract ▾
All authors Yaxiao Liu, Pengbo Liu, Yiwen Liu, Yihua Guan, Jiaxing Song
Computation and Language · Cross-listed from cs.HC
Toward a Culturally Adapted Chinese Language Agent: A Wizard-of-Oz Study of Nonverbal Behavior in Chinese-German Intercultural Interaction
Siddhant Jain, Anna Lea Reinwarth, Dimitra Tsovaltzi et al. (+2 more)
Successful intercultural communication requires more than grammatical competence. It demands sensitivity to culturally embedded social norms whose violation triggers subtle but meaningful nonverbal responses. For German learners of Mandarin Chinese, acquiring this sensitivity is critical yet poorly supported by existing
language-learning
agents. We present a Wizard-of-Oz (WoZ) study design and supporting real-time system for collecting
multimodal behavioral data from native Chinese speakers reacting to social norm violations by German learners. The system features a photorealistic MetaHuman avatar driven by Live Link face capture and MediaPipe upper-body tracking, a wizard console for real-time behavior selection, and synchronized
multimodal logging across
agent and learner streams. A layered annotation framework, based on psychological theory and covering non-observable socioemotional reactions, norm interpretation, verbal, and observable behavior thereof, and future supervision targets enables the corpus to support training of future automated cultural interpretation and behavior generation models. Four ecologically valid interaction scenarios, developed with cultural and pedagogical experts, provide the methodological and technical foundation for a culturally adapted conversational
agent for Chinese
language learning.
Read the full abstract ▾
All authors Siddhant Jain, Anna Lea Reinwarth, Dimitra Tsovaltzi, Rafael Math, Julia Renner
Computation and Language
A mechanistic study of language model introspection
Jiahong Zou, Xiangkun Sun, Lingkai Kong et al. (+1 more)
Large
language models (LLMs) can sometimes report perturbations to their internal activations—even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed. We either inject a concept vector into the hidden state at one of ten
token positions or apply no intervention. The model is asked to identify the perturbed position or report that no intervention occurred. Across three model families, we identify two small groups of
attention heads with distinct roles in introspective reporting. Middle-layer gate heads influence whether the model reports a change, while router heads in a later layer help select the position to report. Interventions on gate heads can suppress position reports even when router heads supply location information. We further examine why reporting accuracy varies across concepts. Concept vectors that are localized more accurately produce stronger
attention-score and output responses in gate heads, which is associated with better
alignment of the induced key and value changes in their QK and OV computations. Together, these findings identify
attention-head mechanisms supporting introspective detection and localization.
Read the full abstract ▾
All authors Jiahong Zou, Xiangkun Sun, Lingkai Kong, Tonghan Wang
Computation and Language
When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment
Zhaohan Zhang, Junjie Liu, Chengzhengxu Li et al. (+6 more)
The
reasoning trajectory of a Large
Language Model (LLM) is often treated as a verbalized description of its internal
reasoning. However, such trajectories can be unfaithful: a model may rely on shortcuts to reach an answer and then post-rationalize the decision with a seemingly coherent chain of thought. Detecting this shortcut
reasoning is challenging because existing monitors and verifiers mainly inspect textual traces or final outcomes, rather than how the model's belief in its answer develops during generation. We introduce ConfLens, a framework that tracks the
evolution of confidence in the final answer throughout
reasoning. Across three shortcut
reasoning settings, we observe a common pattern of premature confidence, where shortcut samples become highly confident in the final answer at early
reasoning stages. Existing confidence estimation methods, however, show limited generalizability, reliability, or efficiency for detecting this behavior. We therefore propose the Distributional Answer Commitment Score (DACS), a distributional confidence
estimator that measures the
entropy of the model's probability distribution over answer commitment at each
reasoning step. DACS captures how concentrated the model's answer belief is without requiring ground-truth answers or task-specific verifiers. We further convert ConfLens detection results into interpretable signals for reward models to reduce their preference for shortcut
reasoning. Experiments on mathematical and code
reasoning tasks show that ConfLens with DACS improves shortcut
reasoning detection by over 4.3% F1 compared with strong baselines and reduces the mismatch between faithfulness and correctness in reward model preferences.
Read the full abstract ▾
All authors Zhaohan Zhang, Junjie Liu, Chengzhengxu Li, Chen Shen, Xiaoming Liu, Chao Shen, Jieping Ye, Ziquan Liu, Ioannis Patras
Computation and Language
Echoes of Deeds: Moral History Can Shape and Steer LLM Behavioral Choices
Lucio La Cava and Andrea Tagarelli
Evaluations of Large
Language Models (LLMs) morality typically consider decisions in isolation, thus overlooking whether an individual's unrelated prior conduct influences the model's subsequent choices. This leaves open the question of whether, and to what extent, moral history shapes LLM decisional behaviors. Prior work on human moral decision-making shows that past behavior can influence subsequent moral choices. Building on this observation, we investigate whether analogous effects emerge in LLMs in two complementary ways: at the behavioral level, through the model's observable responses, and at the representation level, through its latent internal representations. We introduce MoralLedger, a framework for studying how an actor's moral history shapes actions for LLMs' behaviors under a fixed decision context. At the behavioral level, we find that prior moral histories systematically alter subsequent choices as a function of their valence and intensity. At the internal representation level, these histories induce a linearly recoverable direction in the residual stream that generalizes to held-out examples. Intervening along this direction on neutral-history
prompts produces two-sided intensity-dependent changes in subsequent choices, with effects that are stronger than those induced by prompting alone or by favorable-nonmoral direction. To our knowledge, this is the first demonstration that a latent representation of an actor's prior moral conduct can provide signed inference-time control over a moral decision. Our MoralLedger extends moral evaluation beyond static dilemmas, establishing moral history as both a source of behavioral sensitivity and a causal target for auditing and controlling moral behavior in LLMs.
Read the full abstract ▾