Computation and Language
When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training
Felicia Körner, Max Müller-Eberstein, Anna Korhonen et al. (+1 more)
Training Large
Language Models (LLMs) with high
multilingual coverage is becoming increasingly important – especially when monolingual resources are scarce. Recent studies have found that LLMs process
multilingual inputs in shared concept spaces, thought to support generalization and cross-lingual transfer. However, these prior studies often do not use causal methods, lack deeper error analysis or focus on the final model only, leaving open how these spaces emerge during training. We investigate the development of
language-agnostic concept spaces during pretraining of EuroLLM through the causal interpretability method of activation patching. We isolate cross-lingual concept representations, then inject them into a translation
prompt to investigate how consistently translations can be altered, independently of the
language. We find that shared concept spaces emerge early} and continue to refine, but that
alignment with them is
language-dependent}. Furthermore, in contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior – like selecting senses for polysemous words or translating instead of copying cross-lingual homographs – rather than improved translation ability. Our findings offer new insight into the training dynamics of cross-lingual
alignment and the conditions under which causal interpretability methods offer meaningful insights in
multilingual contexts.
Read the full abstract ▾
All authors Felicia Körner, Max Müller-Eberstein, Anna Korhonen, Barbara Plank
Computation and Language · Cross-listed from cs.LG
SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models
Pit Neitemeier, Alessio Serra, Jiaze Li et al. (+3 more)
Hierarchical sequence models replace fixed tokenization with learned
segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end methods can learn meaningful boundaries from the
language-modeling objective alone, it remains difficult to quantitatively assess and systematically steer where compute is spent. We introduce a router-agnostic metric of boundary quality, boundary enrichment B, which measures how strongly chunk starts concentrate on positions with high next-byte surprisal. Guided by this metric, we propose Sombrero, which steers boundary placement toward predictive difficulty via a confidence-
alignment boundary loss and stabilizes boundary learning by applying confidence-weighted smoothing at the input level rather than on realized chunks. On 1B scale, across UTF-8 corpora covering English and German text as well as code and mathematical content, Sombrero improves the accuracy-efficiency trade-off and yields boundaries that more consistently align compute with hard-to-predict positions.
Read the full abstract ▾
All authors Pit Neitemeier, Alessio Serra, Jiaze Li, Sascha Wirges, Lukas Balles, Jan Hendrik Metzen
Computation and Language
Sparse or Dense? A Mechanistic Estimation of Computation Density in Transformer-based LLMs
Corentin Kervadec, Iuliia Lysova, Marco Baroni et al. (+1 more)
Transformer-based large
language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs. Several studies on LLM efficiency optimization argue that it is possible to prune a significant portion of the parameters, while only marginally impacting performance. This suggests that the computation is not uniformly distributed across the parameters. We introduce here a technique to systematically quantify computation density in LLMs. In particular, we design a density
estimator drawing on mechanistic interpretability. We experimentally test our
estimator and find that: (1) contrary to what has been often assumed, LLM processing generally involves dense computation; (2) computation density is dynamic, in the sense that models shift between sparse and dense processing regimes depending on the input; (3) per-input density is significantly correlated across LLMs, suggesting that the same inputs trigger either low or high density. Investigating the factors influencing density, we observe that predicting rarer
tokens requires higher density, and increasing context length often decreases the density. We believe that our computation density
estimator will contribute to a better understanding of the processing at work in LLMs, challenging their symbolic interpretation.
Read the full abstract ▾
All authors Corentin Kervadec, Iuliia Lysova, Marco Baroni, Gemma Boleda
Computation and Language · Cross-listed from eess.AS
CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR
Muhammad Shakeel, Yosuke Fukumoto, Chikara Maeda et al. (+2 more)
We present CALM, a joint Contextual Acoustic-Linguistic Modeling framework for multi-speaker automatic
speech recognition (ASR). In personalized AI scenarios, the joint availability of acoustic and linguistic cues naturally motivates the integration of target-speaker conditioning with contextual biasing in overlapping conversations. CALM implements this integration in an end-to-end framework through speaker
embedding-driven target-speaker extraction and dynamic vocabulary-based contextual biasing. We evaluate CALM on simulated English (LibriSpeechMix) and Japanese (Corpus of Spontaneous Japanese mixtures, CSJMix). On two-speaker mixtures, CALM reduces biased word error rate (B-WER) from 12.7 to 4.7 on LibriSpeech2Mix and biased character error rate (B-CER) from 16.6 to 8.4 on CSJMix2 (eval3), demonstrating the effectiveness of joint acoustic-linguistic modeling across
languages. We additionally report results on the AMI corpus (IHM-mix condition) to validate performance on standardized
speech mixtures.
Read the full abstract ▾
All authors Muhammad Shakeel, Yosuke Fukumoto, Chikara Maeda, Chyi-Jiunn Lin, Shinji Watanabe
Computation and Language
RASST: Retrieval-Augmented Simultaneous Speech Translation
Jiaxuan Luo, Siqi Ouyang, Jiaxing Xu et al. (+1 more)
Simultaneous
speech translation produces target text incrementally from partial
speech input. Recent
speech large
language models have markedly improved SST quality but still struggle with rare and domain-specific terminology.
Retrieval augmentation has helped in automatic
speech recognition and neural machine translation, but extending it to SST is non-trivial:
retrieval must be fast and accurate under partial
speech, and the model must decide whether and when to apply retrieved terms during incremental generation. We propose
Retrieval-Augmented Simultaneous
Speech Translation (RASST), which addresses both challenges. For accurate cross-modal
retrieval under partial input, RASST trains a lightweight
speech-text retriever that produces chunkwise terminology hints for the
Speech LLM via multi-scale
retrieval. To use these hints correctly, we synthesize training data that teaches the
Speech LLM to decide whether and when to apply each retrieved term. Experiments on ACL 60/60 dev set and the ESO test set show that RASST improves terminology accuracy by nearly 40% and overall translation quality by up to 3 BLEU points, with negligible computational overhead.
Read the full abstract ▾
All authors Jiaxuan Luo, Siqi Ouyang, Jiaxing Xu, Lei Li
Computation and Language
AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction
Yifei Li, Richong Zhang, Wanyu Tu et al. (+4 more)
Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts, while existing appellate review mechanisms face efficiency pressures from a surge in case volumes. Although current legal AI research focuses on tasks like judgment prediction and legal document generation, the task of judgment review differs fundamentally in its objectives and paradigm: it centers on detecting, classifying, and correcting errors after a judgment is issued, constituting anomaly detection rather than prediction or generation. To address this research gap, we introduce a novel task APPELLATE REVIEW, aiming to assess models' diagnostic
reasoning and reliability in legal practice. We also construct a novel
dataset benchmark AR-BENCH, which comprises 8,700 finely annotated decisions and 34,617 supplementary corpora. By evaluating 14 large
language models, we reveal critical limitations in existing models' ability to identify legal application errors, providing empirical evidence for future improvements.
Read the full abstract ▾
All authors Yifei Li, Richong Zhang, Wanyu Tu, Zhijie Nie, Haokun Luo, Chuantao Yin, Pengchong Li
Computation and Language
MM-THEBench: Do Reasoning MLLMs Think Reasonably?
Zhidian Huang, Zijun Yao, Ji Qi et al. (+7 more)
Recent advances in
multimodal large
language models (MLLMs) mark a shift from non-thinking models to post-trained
reasoning models capable of solving complex problems through thinking. However, whether such thinking mitigates
hallucinations in
multimodal perception and
reasoning remains unclear. Self-reflective
reasoning enhances robustness but introduces additional
hallucinations, and subtle perceptual errors still result in incorrect or coincidentally correct answers. Existing
benchmarks primarily focus on models before the emergence of
reasoning MLLMs, neglecting the internal thinking process and failing to measure the
hallucinations that occur during thinking. To address these challenges, we introduce MM-THEBench, a comprehensive
benchmark for assessing
hallucinations of intermediate CoTs in
reasoning MLLMs. MM-THEBench features a fine-grained taxonomy grounded in cognitive dimensions, diverse data with verified
reasoning annotations, and a multi-level automated evaluation framework. Extensive experiments on mainstream
reasoning MLLMs reveal insights into how thinking affects
hallucination and
reasoning capability in various
multimodal tasks.
Read the full abstract ▾
All authors Zhidian Huang, Zijun Yao, Ji Qi, Shangqing Tu, Junxian Ma, Jinxin Liu, Weichuan Liu, Xiaoyin Che, Lei Hou, Juanzi Li
Computation and Language · Cross-listed from cs.CR
AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs
Jaehee Kim and Pilsung Kang
Modern LLMs are increasingly accessed via black-box APIs, requiring users to transmit sensitive
prompts, outputs, and fine-tuning data to external providers, creating a critical
privacy risk at the API boundary. We introduce AlienLM, a deployable API-only \cradd{exposure-reduction layer that reduces plaintext exposure} by translating text into an Alien
Language via a vocabulary-scale bijection, enabling lossless recovery on the client side. Using only standard fine-tuning APIs, Alien Adaptation Training (AAT) adapts target models to operate directly on alienized inputs. Across four LLM backbones and seven
benchmarks, AlienLM retains over 81% of plaintext-oracle performance on average, substantially outperforming random-bijection and character-level baselines. Under adversaries with access to model weights, corpus statistics, and learning-based inverse translation, recovery attacks reconstruct fewer than 0.22% of alienized
tokens. Our results demonstrate a practical pathway for \cradd{
privacy-aware} LLM deployment under API-only access, substantially reducing plaintext exposure while maintaining task performance. Code and data are available at https://github.com/KimJaehee0725/AlienLM.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Haonan He, Jingqi Ye, Minglei Li et al. (+4 more)
Low-Rank Adaptation (LoRA) is a fundamental parameter-efficient fine-tuning method that balances efficiency and performance in large-scale neural networks. However, the proliferation of LoRA variants has led to fragmentation in methodology, theory, code, and evaluation. To this end, this work presents the first unified study of LoRA variants, offering a systematic taxonomy, unified theoretical review, structured codebase, and standardized empirical assessment. First, we categorize LoRA variants along four principal axes: rank, optimization dynamics, initialization, and integration with Mixture-of-Experts. Then, we review their relationships and
evolution within a common theoretical framework focused on low-rank update dynamics. Further, we introduce LoRAFactory, a modular codebase that implements variants through a unified interface, supporting plug-and-play experimentation and fine-grained analysis. Last, using this codebase, we conduct a large-scale evaluation across natural
language generation, natural
language understanding, and image classification tasks, systematically exploring key hyperparameters. Our results uncover several findings, notably: LoRA and its variants exhibit pronounced sensitivity to the choices of learning rate compared to other hyperparameters; moreover, with proper hyperparameter configurations, LoRA consistently matches or surpasses the performance of most of its variants.
Read the full abstract ▾
All authors Haonan He, Jingqi Ye, Minglei Li, Zhengbo Wang, Tao Chen, Lei Bai, Peng Ye
Computation and Language
Models Know Models Best: Evaluation via Model-Preferred Formats
Joonhak Lee, Sungmok Jung, Jongyeon Park et al. (+1 more)
Performance of Large
Language Models (LLMs) on multiple-choice tasks differs markedly between symbol-based and cloze-style evaluation formats. The observed discrepancies are systematically attributable to task characteristics: natural
language continuation benefits from likelihood scoring, whereas explicit comparison is better suited to symbol-based selection. These trends are consistent across various decoder-based LLMs, indicating model-agnostic effects. To address these inconsistencies, a dynamic format-
alignment strategy is introduced that employs a lightweight classifier trained on latent model-preference signals. In contrast to human-designed heuristics, which often degrade performance, this approach uses model-generated signals to determine the optimal format for each problem instance. The proposed method achieves substantial and consistent improvements in zero-shot accuracy across
reasoning and knowledge
benchmarks, better revealing the models' latent capabilities.
Read the full abstract ▾
All authors Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee
Computation and Language
FNF: Functional Network Fingerprint for Large Language Models
Yiheng Liu, Junhao Ning, Sichen Xia et al. (+8 more)
The development of large
language models (LLMs) is costly and has significant commercial value. Consequently, preventing unauthorized appropriation of open-source LLMs and protecting developers' intellectual property rights have become critical challenges. In this work, we propose the Functional Network Fingerprint (FNF), a training-free, sample-efficient method for detecting whether a suspect LLM is derived from a victim model, based on the consistency between their functional network activity. We demonstrate that models that share a common origin, even with differences in scale or architecture, exhibit highly consistent patterns of neuronal activity within their functional networks across diverse input samples. In contrast, models trained independently on distinct data or with different objectives fail to preserve such activity
alignment. Unlike conventional approaches, our method requires only a few samples for verification, preserves model utility, and remains robust to common model modifications (such as fine-tuning,
pruning, and parameter permutation), as well as to comparisons across diverse architectures and dimensionalities. FNF thus provides model owners and third parties with a simple, non-invasive, and effective tool for protecting LLM intellectual property. The code is available at https://github.com/WhatAboutMyStar/LLM_ACTIVATION.
Read the full abstract ▾
All authors Yiheng Liu, Junhao Ning, Sichen Xia, Haiyang Sun, Yang Yang, Hanyang Chi, Xiaohui Gao, Ning Qiang, Bao Ge, Junwei Han, Xintao Hu
Computation and Language
TSLM: Tree-Structured Language Modeling for Divergent Thinking
Doyoung Kim, Jaehyeok Doo and Minjoon Seo
Language models generate
reasoning sequentially, preventing them from decoupling irrelevant exploration paths during search. We introduce Tree-Structured
Language Modeling (TSLM), which uses special
tokens to encode branching structure, enabling models to generate and selectively expand multiple search paths within a single generation process. By training on complete search trees including both successful and failed attempts, TSLM learns to internalize systematic exploration without redundant recomputation of shared prefixes. TSLM achieves robust performance and superior inference efficiency by avoiding the multiple independent forward passes required by external search methods. These results suggest a new paradigm of inference-time scaling for robust
reasoning, demonstrating that supervised learning on complete tree-structured traces provides an efficient alternative for developing systematic exploration capabilities in
language models.
Read the full abstract ▾
Computation and Language
NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models
Haisong Gong, Zhibo Liu, Qiang Liu et al. (+2 more)
Prevailing methods for integrating graphs into
Language Models (LMs) typically rely on a segregated architecture: external Graph Neural Networks (GNNs) encode structural
topology, while LMs process textual semantics. We argue this approach is suboptimal for text-graphs: it creates a conceptually disjointed interaction paradigm. By segregating structural encoding from semantic processing, these systems must perform a complex implicit
alignment between abstract graph
tokens and concrete textual elements. Challenging the necessity of external encoders, we propose NAG (Native Architecture for Graphs), a unified framework that internalizes graph processing within the LM's native
manifold. Instead of bridging disparate
embedding spaces, NAG repurposes the self-
attention mechanism to enforce topological dependencies and recalibrates positional IDs to ensure structural equivalence. This allows the model to harness its intrinsic linguistic capability to simultaneously comprehend node and edge content alongside structural
topology. We introduce two efficient implementations: NAG-Zero for absolute preservation of the base model's linguistic capabilities, and NAG-LoRA for enhanced structural adaptation. Experiments across diverse graph tasks validate that NAG achieves robust graph comprehension without the overhead of external encoders, offering a simpler, more coherent paradigm for text-graph modeling.
Read the full abstract ▾
All authors Haisong Gong, Zhibo Liu, Qiang Liu, Shu Wu, Liang Wang
Computation and Language
DART-ing Through the Drift: Dynamic Tracing of Knowledge Neurons for Adaptive Inference-Time Pruning
Abhishek Tyagi, Yunuo Cen, Shrey Dhorajiya et al. (+2 more)
Large
Language Models (LLMs) exhibit substantial parameter redundancy, particularly in Feed-Forward Networks (FFNs). Existing
pruning methods suffer from two primary limitations. First, reliance on
dataset-specific calibration introduces significant data dependency and computational overhead. Second, being predominantly static, they fail to account for the evolving subset of knowledge
neurons in LLMs during autoregressive generation as the context evolves. To address this, we introduce DART, i.e., Dynamic
Attention-Guided Runtime Tracing), a lightweight, training-free method that performs on-the-fly context-based
pruning. DART monitors shifts in
attention score distributions to infer context changes, dynamically updating
neuron-level masks to retain salient parameters. Across ten
benchmarks, DART outperforms prior dynamic baseline, achieving accuracy gains of up to 14.5% on LLAMA-3.1-8B at 70% FFN sparsity. Furthermore, DART achieves up to 3x better ROUGE-L scores with respect to static-masked
pruning on summarization tasks, with its performance comparable to the original dense models. We conclusively demonstrate that the proposed framework effectively adapts to diverse semantic contexts, preserves model capabilities across both general and domain-specific tasks while running at less than 10MBs of
memory for LLAMA-3.1-8B(16GBs) with 0.1% FLOPs overhead. The code is available at https://github.com/seeder-research/DART.
Read the full abstract ▾
All authors Abhishek Tyagi, Yunuo Cen, Shrey Dhorajiya, Bharadwaj Veeravalli, Xuanyao Fong
Computation and Language
Denoising Time Matters:Diverse Generation in Diffusion Language Models
Jingxuan Wu, Zhenglin Wan, Yuzhe Yang et al. (+5 more)
Diffusion language models (
Diffusion-LMs) generate text through iterative denoising, exposing a temporal structure that is largely absent from autoregressive decoding. In this paper, we show that this temporal structure provides a useful control axis for generation diversity: early denoising steps mainly determine high-level semantic trajectories, while later steps refine lexical realization. Motivated by this observation, we propose Time-Annealed Perturbation Sampling (TAPS), a training-free inference strategy that samples nearby conditioning trajectories through time-aware,
manifold-constrained perturbations. TAPS encourages semantic branching during early denoising and anneals the perturbation away before refinement, improving exploration while preserving
prompt alignment, generation quality, and
reasoning ability. Experiments on multiple
Diffusion-LM backbones, including non-autoregressive and semi-autoregressive models, show that TAPS consistently improves semantic and lexical diversity across open-ended and instruction-following generation tasks, while preserving
reasoning ability on verifiable
reasoning benchmarks with negligible overhead.
Read the full abstract ▾
All authors Jingxuan Wu, Zhenglin Wan, Yuzhe Yang, Yiqiao Huang, Chubin Zhang, Xingrui Yu, Ivor Tsang, Yang You
Computation and Language · Cross-listed from cs.LG
TTCS: Test-Time Curriculum Synthesis for Self-Evolving
Chengyi Yang, Zhishang Xiang, Yunbo Tang et al. (+5 more)
Test-Time Training offers a promising way to improve the
reasoning ability of large
language models (LLMs) by adapting the model using only the test questions. However, existing methods struggle with difficult
reasoning problems for two reasons: raw test questions are often too difficult to yield high-quality pseudo-labels, and the limited size of test sets makes continuous online updates prone to instability. To address these limitations, we propose TTCS, a co-evolving test-time training framework. Specifically, TTCS initializes two policies from the same pretrained model: a question synthesizer and a
reasoning solver. These policies evolve through iterative optimization: the synthesizer generates progressively challenging question variants conditioned on the test questions, creating a structured curriculum tailored to the solver's current capability, while the solver updates itself using self-consistency rewards computed from multiple sampled responses on both original test and synthetic questions. Crucially, the solver's feedback guides the synthesizer to generate questions aligned with the model's current capability, and the generated question variants in turn stabilize the solver's test-time training. Experiments show that TTCS consistently strengthens the
reasoning ability on challenging mathematical
benchmarks and transfers to general-domain tasks across different LLM backbones, highlighting a scalable path towards dynamically constructing test-time curricula for self-evolving. Our code and implementation details are available at https://github.com/XMUDeepLIT/TTCS.
Read the full abstract ▾
All authors Chengyi Yang, Zhishang Xiang, Yunbo Tang, Zongpei Teng, Chengsong Huang, Fei Long, Yuhan Liu, Jinsong Su
Computation and Language
Layer-wise Swapping for Generalizable Multilingual Safety
Hyunseo Shin and Wonseok Hwang
Despite the rapid advancements of Large
Language Models (LLMs), safety risks remain a critical challenge for low-resource
languages. Existing safety
datasets are predominantly English centric, limiting progress in
multilingual safety
alignment. As a result, low resource expert models, finetuned on their respective instruction
datasets, tend to exhibit higher unsafety rates compared to their high resource counterparts. In this work, we propose a safety aware layer swapping method that transfers safety
alignment from an English safety expert to low resource
language experts without additional training. To further enhance transfer ability, our method adaptively selects or blends modules based on their degree of specialization. Our approach preserves performance on general
language understanding tasks while enhancing safety in the target
languages. Experimental results show that the proposed method achieves comparable performance to the
language expert on general
benchmarks such as MMMLU, BELEBELE, and MGSM, while producing more aligned and less harmful responses on the MultiJail safety
benchmark.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents
Jiaxuan Gao, Jiaao Chen, Chuyi He et al. (+3 more)
Interactive tool-using
agents must solve real-world tasks via multi-turn interaction with both humans and external environments, requiring dialogue state tracking, multi-step tool execution, while following complex instructions. Post-training such
agents is challenging because synthesis for high-quality multi-turn tool-use data is difficult to scale, and reinforcement learning (RL) could face noisy signals caused by user simulation, leading to degraded training efficiency. We propose a unified framework that combines a self-evolving data
agent with verifier-based RL. Our system, EigenData, is a hierarchical multi-
agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers, and improves generation reliability via closed-loop self-evolving process that updates
prompts and workflow. Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training with trajectory-level group-relative advantages and dynamic filtering, yielding consistent improvements beyond SFT. Evaluated on tau^2-bench, our best model reaches 73.0% pass^1 on Airline and 98.3% pass^1 on Telecom, matching or exceeding frontier models. Overall, our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.
Read the full abstract ▾
All authors Jiaxuan Gao, Jiaao Chen, Chuyi He, Shusheng Xu, Di Jin, Yi Wu
Computation and Language · Cross-listed from cs.SE
TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
Ryo Fujii, Makoto Morishita, Kazuki Yano et al. (+1 more)
With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a
benchmark designed to evaluate software migration in real-world Python projects. Our
benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the
benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated
agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our
dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.
Read the full abstract ▾
All authors Ryo Fujii, Makoto Morishita, Kazuki Yano, Jun Suzuki
Computation and Language
Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt et al. (+1 more)
The high-level concepts that a neural network uses to perform computation need not be aligned to individual
neurons (Smolensky, 1986).
Language model interpretability research has thus turned to techniques which decompose the
neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all
neuron-based representations are uninterpretable. For the first time, we empirically show that MLP
neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for
circuit tracing on the MLP
neuron basis, which surfaces causally effective
neurons on a variety of tasks. On a standard subject-verb agreement
benchmark (Marks et al., 2025), a
circuit of $\approx 10^2$ MLP
neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a
circuit in which small sets of
neurons encode specific latent
reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of
language models without imposing additional training costs.
Read the full abstract ▾
All authors Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, Sarah Schwettmann
Computation and Language
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Zhuochun Li, Yong Zhang, Ming Li et al. (+8 more)
Large
language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to
prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on
reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation. The code and data are available at: https://github.com/zhuochunli/Representation-as-a-judge
Read the full abstract ▾
All authors Zhuochun Li, Yong Zhang, Ming Li, Yuelyu Ji, Yiming Zeng, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao, Daqing He
Computation and Language
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
Chao Wang, Bei Li, Jiaqi Zhang et al. (+8 more)
The success of Large
Language Models (LLMs) hinges on the stable training of deep
Transformer architectures. A critical design choice is the placement of normalization layers, leading to a fundamental trade-off: the “PreNorm” architecture ensures training stability at the cost of potential performance degradation in deep models, while the “PostNorm” architecture offers strong performance but suffers from severe training instability. In this work, we propose SpanNorm, a novel technique designed to resolve this dilemma by integrating the strengths of both paradigms. Structurally, SpanNorm establishes a clean residual connection that spans the entire
transformer block to stabilize signal propagation, while employing a PostNorm-style computation that normalizes the aggregated output to enhance model performance. We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network, preventing the gradient issues that plague PostNorm models, and also alleviating the representation collapse of PreNorm. Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios, paving the way for more powerful and stable
Transformer architectures.
Read the full abstract ▾
All authors Chao Wang, Bei Li, Jiaqi Zhang, Xinyu Liu, Yuchun Fan, Linkun Lyu, Xin Chen, Jingang Wang, Tong Xiao, Peng Pei, Xunliang Cai
Computation and Language · Cross-listed from cs.CV
PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios
Xudong Lu, Huankang Guan, Yang Bo et al. (+10 more)
Multimodal Large
Language Models excel at offline audio-visual understanding, but their ability to serve as mobile assistants in continuous real-world streams remains underexplored. In daily phone use, mobile assistants must track streaming audio-visual inputs and respond at the right time, yet existing
benchmarks are often restricted to multiple-choice questions or use shorter
videos. In this paper, we introduce PhoStream, the first mobile-centric streaming
benchmark that unifies on-screen and off-screen scenarios to evaluate
video, audio, and temporal
reasoning. PhoStream contains 5,572 open-ended QA pairs from 578
videos across 4 scenarios and 10 capabilities. We build it with an Automated Generative Pipeline backed by rigorous human verification, and evaluate models using a realistic Online Inference Pipeline and LLM-as-a-Judge evaluation for open-ended responses. Experiments reveal a temporal asymmetry in LLM-judged scores (0-100): models perform well on Instant and Backward tasks (Gemini 3 Pro exceeds 80), but drop sharply on Forward tasks (16.40), largely due to early responses before the required visual and audio cues appear. This highlights a fundamental limitation: current MLLMs struggle to decide when to speak, not just what to say. Code and
datasets used in this work will be made publicly accessible at https://github.com/Lucky-Lance/PhoStream.
Read the full abstract ▾
All authors Xudong Lu, Huankang Guan, Yang Bo, Jinpeng Chen, Xintong Guo, Shuhan Li, Fang Liu, Peiwen Sun, Xueying Li, Wei Zhang, Xue Yang, Rui Liu, Hongsheng Li
Computation and Language
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Dani Roytburg, Matthew Bozoukov, Matthew Nguyen et al. (+3 more)
Recent research has shown that large
language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge's voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the
entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.
Read the full abstract ▾
All authors Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Mackenzie Puig-Hall, Narmeen Oozeer
Computation and Language
Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation
Shun Qian, Bingquan Liu, Chengjie Sun et al. (+2 more)
The recent advancements in Large
Language Models (LLMs) have attracted interest in exploring their in-context learning abilities and chain-of-thought capabilities. However, there are few studies investigating the specific traits related to the powerful generation capacity of LLMs. This paper aims to delve into the generation characteristics exhibited by LLMs. Through our investigation, we have discovered that
language models tend to capture target-side keywords at the beginning of the generation process. We name this phenomenon the
Holographic Characteristic of
language models. For the purpose of exploring this characteristic and further improving the inference efficiency of
language models, we propose a plugin called HOLO, which leverages the
Holographic Characteristic to extract target-side keywords from
language models within a limited number of generation steps and complements the sentence with a parallel lexically constrained text generation method. To verify the effectiveness of HOLO, we conduct massive experiments on
language models of varying architectures and scales in the short-text generation scenario. The results demonstrate that HOLO achieves comparable performance to the baselines in terms of both automatic and human-like evaluation metrics and highlight the potential of the
Holographic Characteristic.
Read the full abstract ▾
All authors Shun Qian, Bingquan Liu, Chengjie Sun, Zhen Xu, Baoxun Wang
Computation and Language
$\rho$-$\texttt{EOS}$: Training-free Bidirectional Variable-Length Control for Masked Diffusion LLMs
Jingyi Yang, Yuxian Jiang and Jing Shao
Beyond parallel generation and global context modeling, current masked
diffusion large
language models (masked dLLMs, i.e., LLaDA) suffer from a fundamental limitation: they require a predefined, fixed generation length, which lacks flexibility and forces an inevitable trade-off between output quality and computational efficiency. To address this, we study the denoising dynamics and find that the implicit density ($\rho$) of end-of-sequence ($\texttt{EOS}$)
tokens serves as a reliable signal of generation sufficiency. In particular, the evolving implicit $\texttt{EOS}$ density during denoising reveals whether the current masked space is excessive or insufficient, thereby guiding the adjustment direction for generation length. Building on this insight, we propose $\textbf{$\rho$-$EOS$}$, a training-free, single-stage strategy that enables bidirectional variable-length generation for masked dLLMs. Unlike prior two-stage approaches–which require separate length adjustment and iterative mask insertion phases while supporting only unidirectional expansion–$\textbf{$\rho$-$EOS$}$ achieves bidirectional length adjustment within a unified denoising process by continuously estimating the implicit $\texttt{EOS}$ density: excessively high density triggers $\texttt{MASK}$
token contraction, while insufficient density induces expansion. Extensive experiments on mathematics and code
benchmarks demonstrate that $\textbf{$\rho$-$EOS$}$ achieves comparable performance while substantially improving inference efficiency and
token utilization. Code is available at https://github.com/yjyddq/rho-EOS.
Read the full abstract ▾
Computation and Language
One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry
Weisong Zhao, Tong Wang, Zichang Tan et al. (+11 more)
Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the
geometric mean of GMPO. While GMPO improves stability by constraining a conservative objective, it shares a fundamental limitation with GRPO: reliance on a fixed aggregation geometry that ignores the evolving and heterogeneous nature of each trajectory. In this work, we unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry via the power-mean geometry exponent p. Within this framework, GRPO and GMPO are recovered as special cases. Theoretically, we demonstrate that adjusting p modulates the concentration of gradient updates, effectively reweighting
tokens based on their advantage contribution. To determine p adaptively, we introduce a Clip-aware Effective Sample Size (ESS) mechanism. Specifically, we propose a deterministic rule that maps a trajectory clipping fraction to a target ESS. Then, we solve for the specific p to align the trajectory induced ESS with this target one. This allows PMPO to dynamically transition between the aggressive arithmetic mean for reliable trajectories and the conservative
geometric mean for unstable ones. Experiments on multiple mathematical
reasoning benchmarks demonstrate that PMPO outperforms strong baselines.
Read the full abstract ▾
All authors Weisong Zhao, Tong Wang, Zichang Tan, Te Yang, Siran Peng, Haoyuan Zhang, Tianshuo Zhang, Haichao Shi, Meng Meng, Yang Yang, Xiangyu Zhu, Zhen Lei, Xiao-Yu Zhang, Xu Zhou
Computation and Language
Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards
Yuanjie Lyu, Chengyu Wang, Lei Shen et al. (+2 more)
Small LLMs often struggle to match the agentic capabilities of large, costly models. While reinforcement learning can help, progress has been limited by two structural bottlenecks: existing open-source agentic training data are narrow in task variety and easily solved; real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes. We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments. Specifically, a strong teacher model creates novel tasks and tool ecosystems, then rewrites them into intentionally underspecified instructions. This compels
agents to actively query users for missing details. When handling synthetic tasks, an LLM-based user simulator provides user-private information, while a mock tool system delivers stable tool responses. For rewards, task-level rubrics are constructed based on required subgoals, user-
agent interactions, and forbidden behaviors. Across 14 challenging
datasets in math, search, and tool use, models trained on our synthetic data achieve substantial gains, with small models outperforming larger baselines.
Read the full abstract ▾
All authors Yuanjie Lyu, Chengyu Wang, Lei Shen, Jun Huang, Tong Xu
Computation and Language
SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization
Jinyang Wu, Changpeng Yang, Yuhao Shen et al. (+9 more)
Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for training intelligent
agents. However, existing methods typically employ binary rewards that fail to capture quality differences among trajectories achieving identical outcomes, thereby overlooking potential diversity within the solution space. Inspired by the “sweet spot” concept in tennis-the racket's core region that produces optimal hitting effects, we introduce Sweet Spot Learning (SSL), a novel framework that provides differentiated guidance for
agent optimization. SSL follows a simple yet effective principle: progressively amplified, tiered rewards guide policies toward the sweet-spot region of the solution space. This principle naturally adapts across diverse tasks: visual perception tasks leverage distance-tiered modeling to reward proximity, while complex
reasoning tasks reward incremental progress toward promising solutions. We theoretically demonstrate that SSL preserves optimal solution ordering and enhances the gradient signal-to-noise ratio, thereby fostering more directed optimization. Extensive experiments across GUI perception, short/long-term planning, and complex
reasoning tasks show consistent improvements over strong baselines on 12
benchmarks, achieving up to 2.5X sample efficiency gains and effective cross-task transferability. Our work establishes SSL as a general principle for training capable and robust
agents.
Read the full abstract ▾
All authors Jinyang Wu, Changpeng Yang, Yuhao Shen, Fangzhi Xu, Bolin Ni, Chonghua Liao, Yuchen Liu, Hongzhen Wang, Shuai Nie, Shuai Zhang, Haoran Luo, Jiaming Xu
Computation and Language · Cross-listed from cs.CR
FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks
Naen Xu, Jinghuai Zhang, Ping He et al. (+6 more)
Large
language models (LLMs) have been widely integrated into critical automated workflows, including contract review and job application processes. However, LLMs are susceptible to manipulation by fraudulent information, which can lead to harmful outcomes. Although advanced defense methods have been developed to address this issue, they often exhibit limitations in effectiveness, interpretability, and generalizability, particularly when applied to LLM-based applications. To address these challenges, we introduce FraudShield, a novel framework designed to protect LLMs from fraudulent content by leveraging a comprehensive analysis of fraud tactics. Specifically, FraudShield constructs and refines a fraud tactic-keyword knowledge graph to capture high-confidence associations between suspicious text and fraud techniques. The structured knowledge graph augments the original input by highlighting keywords and providing supporting evidence, guiding the LLM toward more secure responses. Extensive experiments show that FraudShield consistently outperforms state-of-the-art defenses across four mainstream LLMs and five representative fraud types, while also offering interpretable clues for the model's generations.
Read the full abstract ▾
All authors Naen Xu, Jinghuai Zhang, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Zhaoxiang Wang, Shouling Ji