Computation and Language · Cross-listed from cs.LG
Learning to Interpret Weight Differences in Language Models
Avichal Goel, Yoon Kim, Nir Shavit et al. (+1 more)
Finetuning (pretrained)
language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning
dataset can give a sense of how the model might have changed, these
datasets are often not publicly available or are too large to work with directly. Towards the goal of comprehensively understanding weight diffs in natural
language, we introduce Diff Interpretation Tuning (DIT), a method that trains models to describe their own finetuning-induced modifications. Our approach uses synthetic, labeled weight diffs to train a DIT-adapter, which can be applied to a compatible finetuned model to make it describe how it has changed. We demonstrate in two proof-of-concept settings (reporting hidden behaviors and summarizing finetuned knowledge) that our method enables models to describe their finetuning-induced modifications using accurate natural
language descriptions.
Read the full abstract ▾
All authors Avichal Goel, Yoon Kim, Nir Shavit, Tony T. Wang
Computation and Language
Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
Runchu Tian, Junxia Cui, Xueqiang Xu et al. (+2 more)
Diffusion large
language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) models, offering advantages such as accelerated parallel decoding and bidirectional context modeling. However, the vanilla decoding strategy in discrete dLLMs suffers from a critical limitation: once a
token is accepted, it can no longer be revised in subsequent steps. As a result, early mistakes persist across iterations, harming both intermediate predictions and final output quality. To address this issue, we propose Tolerator (
Token-Level Cross-Validation Refinement), a training-free decoding strategy that leverages cross-validation among predicted
tokens. Unlike existing methods that follow a single progressive unmasking procedure, Tolerator introduces a two-stage process: (i) sequence fill-up and (ii) iterative refinement by remasking and decoding a subset of
tokens while treating the remaining as context. This design enables previously accepted
tokens to be reconsidered and corrected when necessary, leading to more reliable
diffusion decoding outputs. We evaluate Tolerator on five standard
benchmarks covering
language understanding, code generation, and mathematics. Experiments show that our method achieves consistent improvements over the baselines under the same computational budget. These findings suggest that decoding
algorithms are crucial to realizing the full potential of
diffusion large
language models. Code and data are publicly available.
Read the full abstract ▾
All authors Runchu Tian, Junxia Cui, Xueqiang Xu, Feng Yao, Jingbo Shang
Computation and Language
TeachLM: Post-Training LLMs for Education Using Authentic Learning Data
Janos Perczel, Jin Chow and Dorottya Demszky
The promise of generative AI to revolutionize education is constrained by the pedagogical limits of large
language models (LLMs). A major issue is the lack of access to high-quality training data that reflect the learning of actual students.
Prompt engineering has emerged as a stopgap, but the ability of
prompts to encode complex pedagogical strategies in rule-based natural
language is inherently limited. To address this gap we introduce TeachLM - an LLM optimized for teaching through parameter-efficient fine-tuning of state-of-the-art models. TeachLM is trained on a
dataset comprised of 100,000 hours of one-on-one, longitudinal student-tutor interactions maintained by Polygence, which underwent a rigorous anonymization process to protect
privacy. We use parameter-efficient fine-tuning to develop an authentic student model that enables the generation of high-fidelity synthetic student-tutor dialogues. Building on this capability, we propose a novel multi-turn evaluation protocol that leverages synthetic dialogue generation to provide fast, scalable, and reproducible assessments of the dialogical capabilities of LLMs. Our evaluations demonstrate that fine-tuning on authentic learning data significantly improves conversational and pedagogical performance - doubling student talk time, improving questioning style, increasing dialogue turns by 50%, and greater personalization of instruction.
Read the full abstract ▾
Computation and Language
Slm-mux: Orchestrating small language models for reasoning
Chenyu Wang, Zishen Wan, Hao Kang et al. (+5 more)
With the rapid development of
language models, the number of small
language models (SLMs) has grown significantly. Although they do not achieve state-of-the-art accuracy, they are more efficient and often excel at specific tasks. This raises a natural question: can multiple SLMs be orchestrated into a system where each contributes effectively, achieving higher accuracy than any individual model? Existing orchestration methods have primarily targeted frontier models (e.g., GPT-4) and perform suboptimally when applied to SLMs. To address this gap, we propose a three-stage approach for orchestrating SLMs. First, we introduce SLM-MUX, a multi-model architecture that effectively coordinates multiple SLMs. Building on this, we develop two optimization strategies: (i) a model selection search that identifies the most complementary SLMs from a given pool, and (ii) test-time scaling tailored to SLM-MUX. Our approach delivers strong results: Compared to existing orchestration methods, our approach achieves up to 13.4% improvement on MATH, 8.8% on GPQA, and 7.0% on GSM8K. With just two SLMs, SLM-MUX outperforms Qwen 2.5 72B on GPQA and GSM8K, and matches its performance on MATH. We further provide theoretical analyses to substantiate the advantages of our method. Additional experiments show that the core principle of SLM-MUX extends to open-ended generation tasks (e.g., HumanEval) and benefits other model classes, including frontier LLMs and domain-specific fine-tuned SLMs. In summary, we demonstrate that SLMs can be effectively orchestrated into more accurate and efficient systems through the proposed approach. The project page is available at https://slm-mux.github.io/.
Read the full abstract ▾
All authors Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, Yilun Du
Computation and Language
SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs
Dachuan Shi, Abedelkadir Asi, Keying Li et al. (+4 more)
Recent work shows that, beyond discrete
reasoning through explicit chain-of-thought steps, which are limited by the boundaries of natural
languages, large
language models (LLMs) can also reason continuously in latent space, allowing richer information per step and thereby improving
token efficiency. Despite this promise, latent
reasoning still faces two challenges, especially in training-free settings: 1) purely latent
reasoning broadens the search distribution by maintaining multiple implicit paths, which diffuses probability mass, introduces noise, and impedes convergence to a single high-confidence solution, thereby hurting accuracy; and 2) overthinking persists even without explicit text, wasting
tokens and degrading efficiency. To address these issues, we introduce SwiReasoning, a training-free framework for LLM
reasoning which features two key innovations: 1) SwiReasoning dynamically switches between explicit and latent
reasoning, guided by block-wise confidence estimated from
entropy trends in next-
token distributions, to balance exploration and exploitation and promote timely convergence. 2) By limiting the maximum number of thinking-block switches, SwiReasoning curbs overthinking and improves
token efficiency across varying problem difficulties. On widely used mathematics, STEM, coding, and general
benchmarks, SwiReasoning consistently improves average accuracy by 1.8%-3.1% across
reasoning LLMs of different model families and scales. Furthermore, under constrained budgets, SwiReasoning improves average
token efficiency by 57%-79%, with larger gains as budgets tighten.
Read the full abstract ▾
All authors Dachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan, Leyan Pan, Wenke Lee, Wen Xiao
Computation and Language · Cross-listed from cs.CR
Proactive defense against LLM Jailbreak
Weiliang Zhao, Jinjun Peng, Daniel Ben-Levi et al. (+2 more)
The proliferation of powerful large
language models (LLMs) has necessitated robust safety
alignment, yet these models remain vulnerable to evolving
adversarial attacks, including multi-turn jailbreaks that iteratively search for successful queries. Current defenses, which are primarily reactive and static, often fail to handle these iterative attacks. In this paper, we introduce ProAct, a novel proactive defense framework designed to disrupt and mislead these iterative search jailbreak methods. Our core idea is to intentionally mislead these jailbreak methods into thinking that the model has been jailbroken with "spurious responses". These misleading responses provide false signals to the attacker's internal optimization loop, causing the
adversarial search to terminate prematurely and effectively jailbreaking the jailbreak. By conducting extensive experiments across state-of-the-art LLMs, jailbreaking frameworks, and safety
benchmarks, we demonstrate that our method consistently and significantly reduces attack success rates by up to 94% without affecting utility. When combined with other defense fraeworks, it further reduces the latest attack strategies' success rate to 0%. ProActrepresents an orthogonal defense strategy that serves as an additional guardrail to enhance LLM safety against the most effective jailbreaking attacks.
Read the full abstract ▾
All authors Weiliang Zhao, Jinjun Peng, Daniel Ben-Levi, Zhou Yu, Junfeng Yang
Computation and Language
COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef et al. (+1 more)
To address the need for a more comprehensive evaluation of French Natural
Language Understanding (NLU), we introduce COLE, a new
benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and
reasoning, with a particular focus on linguistic phenomena relevant to the French
language. We
benchmark 94 large
language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional
language variations. We release COLE as a public resource to foster further progress in French
language modelling.
Read the full abstract ▾
All authors David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, Richard Khoury
Computation and Language
Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization
Omri Uzan, Asaf Yehudai, Roi pony et al. (+2 more)
Multimodal encoders have pushed the boundaries of visual document
retrieval, matching textual query
tokens directly to image patches and achieving state-of-the-art performance on public
benchmarks. Recent models relying on this paradigm have massively scaled the sizes of their query and document representations, presenting obstacles to deployment and scalability in real-world pipelines. Furthermore, purely
vision-centric approaches may be constrained by the inherent modality gap still exhibited by modern
vision-
language models. In this work, we connect these challenges to the paradigm of hybrid
retrieval, investigating whether a lightweight dense text retriever can enhance a stronger
vision-centric model. Existing hybrid methods, which rely on coarse-grained fusion of ranks or scores, fail to exploit the rich interactions within each model's representation space. To address this, we introduce Guided Query Refinement (GQR), a novel test-time optimization method that refines a primary retriever's query
embedding using guidance from a complementary retriever's scores. Through extensive experiments on visual document
retrieval benchmarks, we demonstrate that GQR allows
vision-centric models to match the performance of models with significantly larger representations, while being up to 14x faster and requiring 54x less
memory. Our findings show that GQR effectively pushes the Pareto frontier for performance and efficiency in
multimodal retrieval. We release our code at https://github.com/IBM/test-time-hybrid-
retrieval
Read the full abstract ▾
All authors Omri Uzan, Asaf Yehudai, Roi pony, Eyal Shnarch, Ariel Gera
Computation and Language
Idiom Understanding as a Tool to Measure the Dialect Gap
David Beauchemin, Yan Tremblay, Mohamed Amine Youssef et al. (+1 more)
The tasks of idiom understanding and dialect understanding are both well-established
benchmarks in natural
language processing. In this paper, we propose combining them, and using regional idioms as a test of dialect understanding. Towards this end, we propose three new
benchmark datasets for the Quebec dialect of French: QFrCoRE, which contains 4,633 instances of idiomatic phrases, and QFrCoRT, which comprises 171 regional instances of idiomatic words, and a new
benchmark for French Metropolitan expressions, MFrCoE, which comprises 4,938 phrases. We explain how to construct these corpora, so that our methodology can be replicated for other dialects. Our experiments with 111 LLMs reveal a critical disparity in dialectal competence: while models perform well on French Metropolitan, 65.77% of them perform significantly worse on Quebec idioms, with only 9.0% favoring the regional dialect. These results confirm that our
benchmarks are a reliable tool for quantifying the dialect gap and that prestige-
language proficiency does not guarantee regional dialect understanding.
Read the full abstract ▾
All authors David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, Richard Khoury
Computation and Language
Imperceptible Jailbreaking against Large Language Models
Kuofeng Gao, Yiming Li, Chao Du et al. (+4 more)
Jailbreaking attacks on the
vision modality typically rely on imperceptible
adversarial perturbations, whereas attacks on the textual modality are generally assumed to require visible modifications (e.g., non-semantic suffixes). In this paper, we introduce imperceptible jailbreaks that exploit a class of Unicode characters called variation selectors. By appending invisible variation selectors to malicious questions, the jailbreak
prompts appear visually identical to original malicious questions on screen, while their tokenization is "secretly" altered. We propose a chain-of-search pipeline to generate such
adversarial suffixes to induce harmful responses. Our experiments show that our imperceptible jailbreaks achieve high attack success rates against four aligned LLMs and generalize to
prompt injection attacks, all without producing any visible modifications in the written
prompt. Our code is available at https://github.com/sail-sg/imperceptible-jailbreaks.
Read the full abstract ▾
All authors Kuofeng Gao, Yiming Li, Chao Du, Xin Wang, Xingjun Ma, Shu-Tao Xia, Tianyu Pang
Computation and Language · Cross-listed from astro-ph.IM
Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)
Lucas Carrit Delgado Pinheiro, Ziru Chen, Bruno Caixeta Piazza et al. (+4 more)
While task-specific demonstrations show early success in applying large
language models (LLMs) to automate some astronomical research tasks, they only provide incomplete views of all necessary capabilities in solving astronomy problems, calling for more thorough understanding of LLMs' strengths and limitations. So far, existing
benchmarks and evaluations focus on simple question-answering that primarily tests astronomical knowledge and fails to evaluate the complex
reasoning required for real-world research in the discipline. Here, we address this gap by systematically benchmarking five state-of-the-art LLMs on the International Olympiad on Astronomy and Astrophysics (IOAA) exams, which are designed to examine deep conceptual understanding, multi-step derivations, and
multimodal analysis. With average scores of 85.6% and 84.2%, Gemini 2.5 Pro and GPT-5 (the two top-performing models) not only achieve gold medal level performance but also rank in the top two among ~200-300 participants in all four IOAA theory exams evaluated (2022-2025). In comparison, results on the data analysis exams show more divergence. GPT-5 still excels in the exams with an 88.5% average score, ranking top 10 among the participants in the four most recent IOAAs, while other models' performances drop to 48-76%. Furthermore, our in-depth error analysis underscores conceptual
reasoning,
geometric reasoning, and spatial visualization (52-79% accuracy) as consistent weaknesses among all LLMs. Hence, although LLMs approach peak human performance in theory exams, critical gaps must be addressed before they can serve as autonomous research
agents in astronomy.
Read the full abstract ▾
All authors Lucas Carrit Delgado Pinheiro, Ziru Chen, Bruno Caixeta Piazza, Ness Shroff, Yingbin Liang, Yuan-Sen Ting, Huan Sun
Computation and Language
Resource-Efficient Fine-Tuning of LLaMA-3.2-3B for Medical Chain-of-Thought Reasoning
Imran Mansha
Large
Language Models (LLMs) such as GPT-4 and LLaMA have demonstrated remarkable
reasoning abilities but require significant computational resources for fine-tuning. This paper presents a resource-efficient fine-tuning approach for LLaMA-3.2-3B to enhance medical chain-of-thought
reasoning while operating under constrained GPU and
memory settings. Using parameter-efficient tuning techniques such as LoRA and QLoRA, we adapt the base model on publicly available medical
reasoning datasets. The model achieves improved
reasoning coherence and factual accuracy while reducing
memory usage by up to 60% compared to standard full fine-tuning. Experimental evaluation demonstrates that lightweight adaptations can retain strong
reasoning capability in medical question-answering tasks. This work highlights practical strategies for deploying LLMs in low-resource research environments and provides insights into balancing efficiency and domain specialization for medical AI systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
Wei Xiong, Chenlu Ye, Baohao Liao et al. (+6 more)
Reinforcement learning (RL) for large
language model
reasoning is frequently hindered by signal loss, a phenomenon where standard uniform sampling with small group sizes fails to uncover informative learning signals for difficult
prompts. We demonstrate that this collapse is a statistical artifact of undersampling rather than an inherent model limitation. To address this systematically, we introduce a theoretical framework based on optimizing a non-linear RL objective (e.g., log-likelihood). We show that this objective naturally induces a weighted gradient
estimator that prioritizes difficult
prompts, which can be robustly realized through adaptive sampling. Guided by this framework, we propose Reinforce-Ada, a family of
algorithms that dynamically allocates inference budgets based on
prompt difficulty, effectively scaling up RL compute to where it is needed most. Unlike passive filtering methods that discard low-signal
prompts, Reinforce-Ada actively invests compute to recover them. We introduce two efficient realizations: an estimation-based approach and a model-free sequential sampling approach. Extensive experiments across multiple
benchmarks show that Reinforce-Ada significantly outperforms uniform baselines like GRPO, recovering lost signals and accelerating convergence by up to $2\times$ while maintaining the same total inference budget. Code is available at https://github.com/RLHFlow/Reinforce-Ada.
Read the full abstract ▾
All authors Wei Xiong, Chenlu Ye, Baohao Liao, Hanze Dong, Xinxing Xu, Christof Monz, Jiang Bian, Nan Jiang, Tong Zhang
Computation and Language
AWARE, Beyond Sentence Boundaries: A Contextual Transformer Framework for Identifying Cultural Capital in STEM Narratives
Khalid Mehtab Khan and Anagha Kulkarni
Identifying cultural capital (CC) themes in student reflections can offer valuable insights that help foster equitable learning environments in classrooms. However, themes such as aspirational goals or family support are often woven into narratives, rather than appearing as direct keywords. This makes them difficult to detect for standard NLP models that process sentences in isolation. The core challenge stems from a lack of awareness, as standard models are pre-trained on general corpora, leaving them blind to the domain-specific
language and narrative context inherent to the data. To address this, we introduce AWARE, a framework that systematically attempts to improve a
transformer model's awareness for this nuanced task. AWARE has three core components: 1) Domain Awareness, adapting the model's vocabulary to the linguistic style of student reflections; 2) Context Awareness, generating sentence
embeddings that are aware of the full essay context; and 3) Class Overlap Awareness, employing a multi-label strategy to recognize the coexistence of themes in a single sentence. Our results show that by making the model explicitly aware of the properties of the input, AWARE outperforms a strong baseline by 2.1 percentage points in Macro-F1 and shows considerable improvements across all themes. This work provides a robust and generalizable methodology for any text classification task in which meaning depends on the context of the narrative.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game
Fangzhou Liang, Tianshi Zheng, Chunkit Chan et al. (+2 more)
Effective multi-
agent collaboration requires
agents to infer the rationale behind others' actions, a capability rooted in Theory-of-Mind (ToM). While recent Large
Language Models (LLMs) excel at logical inference, their ability to infer rationale in dynamic, collaborative settings remains under-explored. This study introduces LLM-Hanabi, a novel
benchmark that uses the cooperative game Hanabi to evaluate the rationale inference and ToM of LLMs. Our framework features an automated evaluation system that measures both game performance and ToM proficiency. Across a range of models, we find a significant positive correlation between ToM and in-game success. Notably, first-order ToM (interpreting others' intent) correlates more strongly with performance than second-order ToM (predicting others' interpretations). These findings highlight that for effective AI collaboration, the ability to accurately interpret a partner's rationale is more critical than higher-order
reasoning. We conclude that prioritizing first-order ToM is a promising direction for enhancing the collaborative capabilities of future models.
Read the full abstract ▾
All authors Fangzhou Liang, Tianshi Zheng, Chunkit Chan, Yauwai Yim, Yangqiu Song
Computation and Language
Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy (short paper)
Om Dobariya and Akhil Kumar
The wording of natural
language prompts has been shown to influence the performance of large
language models (LLMs), yet the role of politeness and tone remains underexplored. In this study, we investigate how varying levels of
prompt politeness affect model accuracy on multiple-choice questions. We created a
dataset of 50 base questions spanning mathematics, science, and history, each rewritten into five tone variants: Very Polite, Polite, Neutral, Rude, and Very Rude, yielding 250 unique
prompts. Using ChatGPT 4o, we evaluated responses across these conditions and applied paired sample t-tests to assess statistical significance. Contrary to expectations, impolite
prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite
prompts to 84.8% for Very Rude
prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation. Our results highlight the importance of studying pragmatic aspects of prompting and raise broader questions about the social dimensions of human-AI interaction.
Read the full abstract ▾
Computation and Language
A First Context-Free Grammar Applied to Nawatl Corpora Augmentation
Juan-José Guzmán-Landa, Juan-Manuel Torres-Moreno, Miguel Figueroa-Saavedra et al. (+3 more)
In this article we introduce a context-free grammar (CFG) for the Nawatl
language. Nawatl (or Nahuatl) is an Amerindian
language of the $\pi$-
language type, i.e. a
language with few digital resources, in which the corpora available for machine learning are virtually non-existent. The objective here is to generate a significant number of grammatically correct artificial sentences, in order to increase the corpora available for
language model training. We want to show that a grammar enables us significantly to expand a corpus in Nawatl which we call $\pi$-yalli. The corpus, thus enriched, enables us to train
algorithms such as FastText and to evaluate them on sentence-level semantic tasks. Preliminary results show that by using the grammar, comparative improvements are achieved over some LLMs. However, it is observed that to achieve more significant improvement, grammars that model the Nawatl
language even more effectively are required.
Read the full abstract ▾
All authors Juan-José Guzmán-Landa, Juan-Manuel Torres-Moreno, Miguel Figueroa-Saavedra, Ligia Quintana-Torres, Martha-Lorena Avendaño-Garrido, Graham Ranger
Computation and Language · Cross-listed from cs.LG
On Structured State-Space Duality
Jerry Yao-Chieh Hu, Xiwen Zhang, Ali ElSheikh et al. (+2 more)
Structured State-Space Duality (SSD) [Dao & Gu, ICML 2024] is an equivalence between a simple Structured State-Space Model (SSM) and a masked
attention mechanism. In particular, a state-space model with a scalar-times-identity state
matrix is equivalent to a masked self-
attention with a $1$-semiseparable causal mask. Consequently, the same sequence transformation (model) has two algorithmic realizations: as a linear-time $O(T)$ recurrence or as a quadratic-time $O(T^2)$
attention. In this note, we formalize and generalize this duality: (i) we extend SSD from the scalar-identity case to general diagonal SSMs (diagonal state
matrices); (ii) we show that these diagonal SSMs match the scalar case's training complexity lower bounds while supporting richer dynamics; (iii) we establish a necessary and sufficient condition under which an SSM is equivalent to $1$-semiseparable masked
attention; and (iv) we show that such duality fails to extend to standard softmax
attention due to rank explosion. Together, these results tighten bridge between recurrent SSMs and
Transformers, and widen the design space for expressive yet efficient sequence models.
Read the full abstract ▾
All authors Jerry Yao-Chieh Hu, Xiwen Zhang, Ali ElSheikh, Weimin Wu, Han Liu
Computation and Language · Cross-listed from cs.LG
ONNX-Net: Towards Universal Representations and Instant Performance Prediction for Neural Architectures
Shiwen Qin, Alexander Auras, Shay B. Cohen et al. (+4 more)
Neural architecture search (NAS) automates the design process of high-performing architectures, but remains bottlenecked by expensive performance evaluation. Most existing studies that achieve faster evaluation are mostly tied to
cell-based search spaces and graph encodings tailored to those individual search spaces, limiting their flexibility and scalability when applied to more expressive search spaces. In this work, we aim to close the gap of individual search space restrictions and search space dependent network representations. We present ONNX-Bench, a
benchmark consisting of a collection of neural networks in a unified format based on ONNX files. ONNX-Bench includes all open-source NAS-bench-based neural networks, resulting in a total size of more than 600k {architecture, accuracy} pairs. This
benchmark allows creating a shared neural network representation, ONNX-Net, able to represent any neural architecture using natural
language descriptions acting as an input to a performance predictor. This text-based encoding can accommodate arbitrary layer types, operation parameters, and heterogeneous
topologies, enabling a single surrogate to generalise across all neural architectures rather than being confined to
cell-based search spaces. Experiments show strong zero-shot performance across disparate search spaces using only a small amount of pretraining samples, enabling the unprecedented ability to evaluate any neural network architecture instantly.
Read the full abstract ▾
All authors Shiwen Qin, Alexander Auras, Shay B. Cohen, Elliot J. Crowley, Michael Moeller, Linus Ericsson, Jovita Lukasik
Computation and Language · Cross-listed from cs.AI
MARS: Co-evolving Dual-System Deep Research via Multi-Agent Reinforcement Learning
Guoxin Chen, Zile Qiao, Wenqing Wang et al. (+10 more)
Large
Reasoning Models (LRMs) face two fundamental limitations: excessive
token consumption when overanalyzing simple information processing tasks, and inability to access up-to-date knowledge beyond their training data. We introduce MARS (Multi-
Agent System for Deep ReSearch), a novel co-
evolution framework that jointly optimizes dual cognitive systems through multi-
agent reinforcement learning. Unlike prior approaches that employ fixed or independently-trained summarizers, MARS enables System 1 (fast, intuitive processing) and System 2 (deliberate
reasoning) to co-adapt through shared trajectory rewards, developing complementary strategies where System 1 learns to distill information specifically useful for System 2's
reasoning. We extend Group Relative Policy Optimization (GRPO) for multi-
agent settings with three key innovations: (1) decoupled gradient computation ensuring proper credit assignment despite shared rewards, (2) bin-packing optimization for efficient parallel information processing, and (3) advantage-weighted balanced sampling preventing training imbalance. Extensive experiments demonstrate that MARS (8B), trained under a challenging Zero RL setting without any supervised fine-tuning, achieves 8.17% on HLE – outperforming WebThinker (32B with SFT, 6.87%) and narrowing the gap with proprietary models like Claude 3.7 Sonnet (7.89%) – while achieving an average gain of 8.9% across 7 knowledge-intensive tasks.
Read the full abstract ▾
All authors Guoxin Chen, Zile Qiao, Wenqing Wang, Donglei Yu, Xuanzhong Chen, Hao Sun, Minpeng Liao, Kai Fan, Yong Jiang, Penguin Xie, Wayne Xin Zhao, Ruihua Song, Fei Huang
Computation and Language
The Geometry of Truth: Layer-wise Semantic Dynamics for Hallucination Detection in Large Language Models
Amir Hameed Mir
Large
Language Models (LLMs) often produce fluent yet factually incorrect statements-a phenomenon known as
hallucination-posing serious risks in high-stakes domains. We present Layer-wise Semantic Dynamics (LSD), a
geometric framework for
hallucination detection that analyzes the
evolution of hidden-state semantics across
transformer layers. Unlike prior methods that rely on multiple sampling passes or external verification sources, LSD operates intrinsically within the model's representational space. Using margin-based contrastive learning, LSD aligns hidden activations with ground-truth
embeddings derived from a factual encoder, revealing a distinct separation in semantic trajectories: factual responses preserve stable
alignment, while
hallucinations exhibit pronounced semantic drift across depth. Evaluated on the TruthfulQA and synthetic factual-
hallucination datasets, LSD achieves an F1-score of 0.92, AUROC of 0.96, and clustering accuracy of 0.89, outperforming SelfCheckGPT and Semantic
Entropy baselines while requiring only a single forward pass. This efficiency yields a 5-20x speedup over sampling-based methods without sacrificing precision or interpretability. LSD offers a scalable, model-agnostic mechanism for real-time
hallucination monitoring and provides new insights into the geometry of factual consistency within large
language models.
Read the full abstract ▾
Computation and Language
Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment
Davood Rafiei, Morgan Lindsay Heisler, Weiwei Zhang et al. (+2 more)
Supervised Fine-Tuning (SFT) is an effective method for adapting Large
Language Models (LLMs) on downstream tasks. However, variability in training data can hinder a model's ability to generalize across domains. This paper studies the problem of
dataset alignment for Natural
Language to SQL (NL2SQL or text to SQL), examining how well SFT training data matches the structural characteristics of target queries and how this
alignment impacts model performance. We hypothesize that
alignment can be accurately estimated by comparing the distributions of structural SQL features across the training set, target data, and the model's predictions prior to SFT. Through comprehensive experiments on three large cross-domain NL2SQL
benchmarks and multiple model families, we show that structural
alignment is a strong predictor of fine-tuning success. When
alignment is high, SFT yields substantial gains in accuracy and SQL generation quality; when
alignment is low, improvements are marginal or absent. These findings highlight the importance of
alignment-aware data selection for effective fine-tuning and generalization in NL2SQL tasks.
Read the full abstract ▾
All authors Davood Rafiei, Morgan Lindsay Heisler, Weiwei Zhang, Mohammadreza Pourreza, Yong Zhang
Computation and Language · Cross-listed from cs.SE
Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches
Yicheng Tao, Yuante Li, Yao Qin et al. (+1 more)
Recent advances in large
language models (LLMs) have significantly improved automated code generation. While existing approaches have achieved strong performance at the function and file levels, real-world software engineering requires
reasoning over entire repositories, including cross-file dependencies, evolving execution environments, and global semantic consistency. This challenge has led to the emergence of Repository-Level Code Generation (RLCG), where models must retrieve, organize, and utilize repository-scale context to generate coherent and executable code changes. To address these challenges,
Retrieval-Augmented Generation (RAG) has become an increasingly important paradigm for repository-level code intelligence. In this survey, we present a comprehensive review of
Retrieval-Augmented Code Generation (RACG), with a particular focus on repository-level approaches. Rather than viewing RACG as a static “retrieve-then-generate” pipeline, we characterize it as a coupled and evolving process involving context construction,
retrieval optimization, generation, and environment interaction. We organize existing methods through a unified analytical framework spanning
retrieval substrate, control regime, and evaluation setting. Based on this framework, we systematically examine
retrieval strategies, graph-based and non-graph-based
retrieval paradigms, training-driven optimizations, and autonomous
agent architectures. We further summarize widely used
datasets,
benchmarks, and system configurations, and discuss key challenges including scalability, reliability, efficiency, and the necessity boundary between RACG and long-context LLMs. Through this survey, we aim to provide a structured understanding of the rapidly evolving RACG landscape and highlight promising directions for future AI-powered software engineering research.
Read the full abstract ▾
All authors Yicheng Tao, Yuante Li, Yao Qin, Yepang Liu
Computation and Language
SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests
Punya Syon Pandey, Hai Son Le, Devansh Bhardwaj et al. (+2 more)
Large
language models (LLMs) are increasingly deployed in contexts where their failures can have direct sociopolitical consequences. Yet, existing safety
benchmarks rarely test vulnerabilities in domains such as political manipulation, propaganda and disinformation generation, or surveillance and information control. We introduce SocialHarmBench, a
dataset of 585
prompts spanning 7 sociopolitical categories and 34 countries, designed to surface where LLMs most acutely fail in politically charged contexts. Our evaluations reveal several shortcomings: open-weight models exhibit high vulnerability to harmful compliance, with Mistral-7B reaching attack success rates as high as 97% to 98% in domains such as historical revisionism, propaganda, and political manipulation. Moreover, temporal and geographic analyses show that LLMs are most fragile when confronted with 21st-century or pre-20th-century contexts, and when responding to
prompts tied to regions such as Latin America, the USA, and the UK. These findings demonstrate that current safeguards fail to generalize to high-stakes sociopolitical settings, exposing systematic biases and raising concerns about the reliability of LLMs in preserving human rights and democratic values. We share the SocialHarmBench
benchmark at https://huggingface.co/
datasets/psyonp/SocialHarmBench.
Read the full abstract ▾
All authors Punya Syon Pandey, Hai Son Le, Devansh Bhardwaj, Rada Mihalcea, Zhijing Jin
Computation and Language
Hengxiang Zhang, Hyeong Kyu Choi, Sharon Li et al. (+1 more)
Reasoning distillation has emerged as a prevailing paradigm for transferring
reasoning capabilities from large
reasoning models to small
language models. Yet,
reasoning distillation risks data contamination:
benchmark data may inadvertently be included in the
distillation data, thereby inflating model performance metrics. In this work, we formally define the
distillation data detection task, which determines whether a given question is included in the model's
distillation data. The unique challenge of this task lies in the partial availability of
distillation data. To address this, we propose
Token Probability Deviation (TPD), a detection method that leverages the probability patterns of output
tokens generated by the model instead of input
tokens. Our method is motivated by the observation that seen questions tend to elicit more near-deterministic
tokens generated by the models than unseen ones. Our TPD score is thus designed to quantify the
token-level deviation of generated
tokens from a high-confidence reference probability. Consequently, seen questions can yield substantially lower TPD scores than unseen ones, enabling strong detection performance. Extensive experiments demonstrate the effectiveness of our approach, improving detection AUC by up to 31% on
distillation datasets.
Read the full abstract ▾
All authors Hengxiang Zhang, Hyeong Kyu Choi, Sharon Li, Hongxin Wei
Computation and Language
When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA
Elisei Rykov, Kseniia Petrushina, Maksim Savkin et al. (+6 more)
Hallucination detection remains a fundamental challenge for the safe and reliable deployment of large
language models (LLMs), especially in applications requiring factual accuracy. Existing
hallucination benchmarks often operate at the sequence level and are limited to English, lacking the fine-grained,
multilingual supervision needed for a comprehensive evaluation. In this work, we introduce PsiloQA, a large-scale,
multilingual dataset annotated with span-level
hallucinations across 14
languages. PsiloQA is constructed through an automated three-stage pipeline: generating question-answer pairs from Wikipedia using GPT-4o, eliciting potentially hallucinated answers from diverse LLMs in a no-context setting, and automatically annotating hallucinated spans using GPT-4o by comparing against golden answers and retrieved context. We evaluate a wide range of
hallucination detection methods – including uncertainty quantification, LLM-based tagging, and fine-tuned encoder models – and show that encoder-based models achieve the strongest performance across
languages. Furthermore, PsiloQA demonstrates effective cross-lingual generalization and supports robust knowledge transfer to other
benchmarks, all while being significantly more cost-efficient than human-annotated
datasets. Our
dataset and results advance the development of scalable, fine-grained
hallucination detection in
multilingual settings.
Read the full abstract ▾
All authors Elisei Rykov, Kseniia Petrushina, Maksim Savkin, Valerii Olisov, Artem Vazhentsev, Kseniia Titova, Alexander Panchenko, Vasily Konovalov, Julia Belikova
Computation and Language
Instability in Downstream Task Performance During LLM Pretraining
Yuto Nishida, Masaru Isonuma and Yusuke Oda
When training large
language models (LLMs), it is common practice to track downstream task performance throughout the training process and select the checkpoint with the highest validation score. However, downstream metrics often exhibit substantial fluctuations, making it difficult to identify the checkpoint that truly represents the best-performing model. In this study, we empirically analyze the stability of downstream task performance in an LLM trained on diverse web-scale corpora. We find that task scores frequently fluctuate throughout training, both at the aggregate and example levels. To address this instability, we investigate two post-hoc checkpoint integration methods: checkpoint averaging and ensemble, motivated by the hypothesis that aggregating neighboring checkpoints can reduce performance
volatility. We demonstrate both empirically and theoretically that these methods improve downstream performance stability without requiring any changes to the training procedure.
Read the full abstract ▾
Computation and Language
How I Built ASR for Endangered Languages with a Spoken Dictionary
Christopher Bartley and Anton Ragni
Nearly half of the world's
languages are endangered.
Speech technologies such as Automatic
Speech Recognition (ASR) are central to revival efforts, yet most
languages remain unsupported because standard pipelines expect utterance-level supervised data.
Speech data often exist for endangered
languages but rarely match these formats. Manx Gaelic ($\sim$2,200 speakers), for example, has had transcribed
speech since 1948, yet remains unsupported by modern systems. In this paper, we explore how little data, and in what form, is needed to build ASR for critically endangered
languages. We show that a short-form pronunciation resource is a viable alternative, and that 40 minutes of such data produces usable ASR for Manx ($<$50% WER). We replicate our approach, applying it to Cornish ($\sim$600 speakers), another critically endangered
language. Results show that the barrier to entry, in quantity and form, is far lower than previously thought, giving hope to endangered
language communities that cannot afford to meet the requirements arbitrarily imposed upon them.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Visual Representations inside the Language Model
Benlin Liu, Amita Kamath, Madeleine Grunde-McLaughlin et al. (+2 more)
Despite interpretability work analyzing VIT encoders and
transformer activations, we don't yet understand why
Multimodal Language Models (MLMs) struggle on perception-heavy tasks. We offer an under-studied perspective by examining how popular MLMs (LLaVA-OneVision, Qwen2.5-VL, and Llama-3-LLaVA-NeXT) process their visual key-value
tokens. We first study the flow of visual information through the
language model, finding that image value
tokens encode sufficient information to perform several perception-heavy tasks zero-shot:
segmentation, semantic correspondence, temporal correspondence, and referring expression detection. We find that while the
language model does augment the visual information received from the projection of input visual encodings-which we reveal correlates with overall MLM perception capability-it contains less visual information on several tasks than the equivalent visual encoder (SigLIP) that has not undergone MLM finetuning. Further, we find that the visual information corresponding to input-agnostic image key
tokens in later layers of
language models contains artifacts which reduce perception capability of the overall MLM. Next, we discuss controlling visual information in the
language model, showing that adding a text prefix to the image input improves perception capabilities of visual representations. Finally, we reveal that if
language models were able to better control their visual information, their perception would significantly improve; e.g., in 33.3% of Art Style questions in the BLINK
benchmark, perception information present in the
language model is not surfaced to the output! Our findings reveal insights into the role of key-value
tokens in
multimodal systems, paving the way for deeper mechanistic interpretability of MLMs and suggesting new directions for training their visual encoder and
language model components.
Read the full abstract ▾
All authors Benlin Liu, Amita Kamath, Madeleine Grunde-McLaughlin, Winson Han, Ranjay Krishna
Computation and Language
Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
Sangmin Bae, Bilge Acun, Chien-Yu Lin et al. (+5 more)
Recent progress in large
language models demonstrates that hybrid architectures–combining self-
attention mechanisms with structured state space models like Mamba–can achieve a compelling balance between modeling quality and computational efficiency, particularly for long-context tasks. While these hybrid models show promising performance, systematic comparisons of hybridization strategies and analyses on the key factors behind their effectiveness have not been clearly shared to the community. In this work, we present a holistic evaluation of hybrid architectures based on inter-layer (sequential) or intra-layer (parallel) fusion. We comprehensively evaluate these designs across multiple dimensions:
language modeling and downstream task performance, long-context capabilities, scaling analysis, and training and inference efficiency. By investigating the core characteristics of their computational primitive, we identify the most critical elements for each hybridization strategy and further propose optimal design recipes for hybrid models. Our comprehensive analysis provides practical guidance and valuable insights for developing hybrid
language models, facilitating the optimization of architectural configurations.
Read the full abstract ▾
All authors Sangmin Bae, Bilge Acun, Chien-Yu Lin, Haroun Habeeb, Seungyeon Kim, Liang Luo, Junjie Wang, Carole-Jean Wu