Computation and Language
MMFormalizer: Multimodal Autoformalization in the Wild
Jing Xiong, Qi Han, Yunta Hsieh et al. (+11 more)
Autoformalization, which translates natural
language mathematics into formal statements to enable machine
reasoning, faces fundamental challenges in the wild due to the
multimodal nature of the physical world, where physics requires inferring hidden constraints (e.g., mass or energy) from visual elements. To address this, we propose MMFormalizer, which extends autoformalization beyond text by integrating adaptive grounding with entities from real-world mathematical and physical domains. MMFormalizer recursively constructs formal propositions from perceptually grounded primitives through recursive grounding and axiom composition, with adaptive recursive termination ensuring that every abstraction is supported by visual evidence and anchored in dimensional or axiomatic grounding. We evaluate MMFormalizer on a new
benchmark, PhyX-AF, comprising 115 curated samples from MathVerse, PhyX, Synthetic Geometry, and Analytic Geometry, covering diverse
multimodal autoformalization tasks. Results show that frontier models such as GPT-5 and Gemini-3-Pro achieve the highest compile and semantic accuracy, with GPT-5 excelling in physical
reasoning, while geometry remains the most challenging domain. Overall, MMFormalizer provides a scalable framework for unified
multimodal autoformalization, bridging perception and formal
reasoning. To the best of our knowledge, this is the first
multimodal autoformalization method capable of handling classical mechanics (derived from the
Hamiltonian), as well as relativity,
quantum mechanics, and thermodynamics. More details are available on our project page: MMFormalizer.github.io
Read the full abstract ▾
All authors Jing Xiong, Qi Han, Yunta Hsieh, Hui Shen, Huajian Xin, Chaofan Tao, Chenyang Zhao, Hengyuan Zhang, Taiqiang Wu, Zhen Zhang, Haochen Wang, Zhongwei Wan, Lingpeng Kong, Ngai Wong
Computation and Language
SentGraph: Hierarchical Sentence Graph for Multi-hop Retrieval-Augmented Question Answering
Junli Liang, Pengfei Zhou, Wangqiu Zhou et al. (+5 more)
Traditional
Retrieval-Augmented Generation (RAG) effectively supports single-hop question answering with large
language models but faces significant limitations in multi-hop question answering tasks, which require combining evidence from multiple documents. Existing chunk-based
retrieval often provides irrelevant and logically incoherent context, leading to incomplete evidence chains and incorrect
reasoning during answer generation. To address these challenges, we propose SentGraph, a sentence-level graph-based RAG framework that explicitly models fine-grained logical relationships between sentences for multi-hop question answering. Specifically, we construct a hierarchical sentence graph offline by first adapting Rhetorical Structure Theory to distinguish
nucleus and satellite sentences, and then organizing them into topic-level subgraphs with cross-document entity bridges. During online
retrieval, SentGraph performs graph-guided evidence selection and path expansion to retrieve fine-grained sentence-level evidence. Extensive experiments on four multi-hop question answering
benchmarks demonstrate the effectiveness of SentGraph, validating the importance of explicitly modeling sentence-level logical dependencies for multi-hop
reasoning.
Read the full abstract ▾
All authors Junli Liang, Pengfei Zhou, Wangqiu Zhou, Wenjie Qing, Qi Zhao, Ziwen Wang, Qi Song, Xiangyang Li
Computation and Language
Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners
Yihong Liu, Raoyuan Zhao, Hinrich Schütze et al. (+1 more)
Large
reasoning models (LRMs) achieve strong performance on mathematical
reasoning tasks, often attributed to their capability to generate explicit chain-of-thought (CoT) explanations. However, recent work shows that LRMs often arrive at the correct answer before completing these textual
reasoning steps, indicating the presence of latent
reasoning – internal, non-verbal computation encoded in hidden states. While this phenomenon has been explored in English, its
multilingual behavior remains largely unknown. In this paper, we conduct a systematic investigation of
multilingual latent
reasoning in LRMs across 11
languages. Using a truncation-based strategy, we examine how the correct answer emerges as the model is given only partial
reasoning traces, allowing us to measure stepwise latent prediction formation. Our results reveal clear evidence of
multilingual latent
reasoning, though unevenly: strong in resource-rich
languages, weaker in low-resource ones, and broadly less observable on harder
benchmarks. To understand whether these differences reflect distinct internal mechanisms, we further perform representational analyses. Despite surface-level disparities, we find that the internal
evolution of predictions is highly consistent across
languages and broadly aligns with English – a pattern suggesting an English-centered latent
reasoning pathway.
Read the full abstract ▾
All authors Yihong Liu, Raoyuan Zhao, Hinrich Schütze, Michael A. Hedderich
Computation and Language
Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generation
Qianchi Zhang, Hainan Zhang, Liang Pang et al. (+2 more)
Retrieval-Augmented Generation (RAG) has become a key paradigm for reducing factual
hallucinations in Large
Language Models (LLMs), yet little is known about how the order of retrieved documents affects model behavior. We empirically show that under a Top-5
retrieval setting with the gold document included, LLM answers vary substantially across permutations of the retrieved set, even when the gold document is fixed in the first position. This reveals a previously underexplored sensitivity to
retrieval permutations. Although existing robust RAG methods focus primarily on enhancing LLM robustness to low-quality
retrieval and mitigating positional bias to distribute
attention fairly over long contexts, neither approach directly addresses permutation sensitivity. In this paper, we propose Stable-RAG, which exploits permutation sensitivity estimation to mitigate permutation-induced
hallucinations. Stable-RAG runs the generator under multiple
retrieval orders, clusters hidden states, and decodes from a cluster-center representation that captures the dominant
reasoning pattern. It then uses these
reasoning results to align hallucinated outputs toward the correct answer, encouraging the model to produce consistent and accurate predictions across document permutations. Experiments on three QA
datasets show that Stable-RAG improves answer accuracy,
reasoning consistency, and generalization across
datasets, retrievers, and input lengths compared with strong baselines.
Read the full abstract ▾
All authors Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
Computation and Language
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi et al. (+5 more)
Large
language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from the architectural limits of
transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated
attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve higher accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their
reasoning behavior.
Read the full abstract ▾
All authors Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah
Computation and Language
P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist
Kwangwook Seo and Dongha Lee
Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with individual preferences. However, existing approaches largely treat user context as a static or implicit conditioning signal, failing to capture the dynamic and multi-faceted nature of human judgment. In this paper, we propose P-Check, a novel personalized reward modeling framework, designed to train a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction. To better align these checklists with personalized nuances, we introduce Preference-Contrastive Criterion Weighting, a training strategy that assigns saliency scores to criteria based on their discriminative power for personalized judgment. We conduct extensive experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation, and remains robust in OOD scenarios.
Read the full abstract ▾
Computation and Language
Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
Ruikang Zhang, Shuo Wang and Qi Su
Recent work in Mechanistic Interpretability (MI) has enabled the identification and intervention of internal features in Large
Language Models (LLMs). However, a persistent challenge lies in linking such internal features to the reliable control of complex, behavior-level semantic attributes in
language generation. In this paper, we propose a Sparse Autoencoder-based framework for retrieving and steering semantically interpretable internal features associated with high-level linguistic behaviors. Our method employs a contrastive feature
retrieval pipeline based on controlled semantic oppositions, combing statistical activation analysis and generation-based validation to distill monosemantic functional features from sparse activation spaces. Using the Big Five personality traits as a case study, we demonstrate that our method enables precise, bidirectional steering of model behavior while maintaining superior stability and performance compared to existing activation steering methods like Contrastive Activation Addition (CAA). We further identify an empirical effect, which we term Functional Faithfulness, whereby intervening on a specific internal feature induces coherent and predictable shifts across multiple linguistic dimensions aligned with the target semantic attribute. Our findings suggest that LLMs internalize deeply integrated representations of high-order concepts, and provide a novel, robust mechanistic path for the regulation of complex AI behaviors.
Read the full abstract ▾
Computation and Language
Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning
Nathanaël Carraz Rakotonirina, Ren Pang, Neha Anna John et al. (+2 more)
The
reasoning capabilities of large
language models (LLMs) have improved substantially through increased test-time computation, typically in the form of intermediate
tokens known as chain-of-thought (CoT). However, CoT often becomes unnecessarily long, increasing computation cost without actual accuracy gains or sometimes even degrading performance, a phenomenon known as “overthinking”. We propose a multi-stage efficient
reasoning method that combines supervised fine-tuning – via rejection sampling or
reasoning trace reformatting – with reinforcement learning using an adaptive length penalty. We introduce a lightweight reward function that penalizes
tokens generated after the first correct answer but encouraging self-verification only when beneficial. We conduct a holistic evaluation across seven diverse
reasoning tasks, analyzing the accuracy-response length trade-off. Our approach reduces response length by an average of 28% for 8B models and 40% for 32B models, while incurring only minor performance drops of 1.6 and 2.5 points, respectively. Despite its conceptual simplicity, it achieves a superior trade-off compared to more complex state-of-the-art efficient
reasoning methods, scoring 76.6, in terms of the area under the Overthinking-Adjusted Accuracy curve ($\text{AUC}_{\text{OAA}}$) – 5 points above the base model and 2.5 points above the second-best approach.
Read the full abstract ▾
All authors Nathanaël Carraz Rakotonirina, Ren Pang, Neha Anna John, Michael Bohlke-Schneider, Momchil Hardalov
Computation and Language
Reliability-Aware Adaptive Self-Consistency for Efficient Sampling in LLM Reasoning
Junseok Kim, Nakyeong Yang, Kyungmin Min et al. (+1 more)
Self-Consistency improves
reasoning reliability through multi-sample aggregation, but incurs substantial inference cost. Adaptive self-consistency methods mitigate this issue by adjusting the sampling budget; however, they rely on count-based stopping rules that treat all responses equally, often leading to unnecessary sampling. We propose Reliability-Aware Adaptive Self-Consistency (ReASC), which addresses this limitation by reframing adaptive sampling from response counting to evidence sufficiency, leveraging response-level confidence for principled information aggregation. ReASC operates in two stages: a single-sample decision stage that resolves instances confidently answerable from a single response, and a reliability-aware accumulation stage that aggregates responses by jointly leveraging their frequency and confidence. Across five models and four
datasets, ReASC consistently achieves the best accuracy-cost trade-off compared to existing baselines, yielding improved inference efficiency across model scales from 3B to 27B parameters. As a concrete example, ReASC reduces inference cost by up to 70% relative to self-consistency while preserving accuracy on GSM8K using Gemma-3-4B-it.
Read the full abstract ▾
All authors Junseok Kim, Nakyeong Yang, Kyungmin Min, Kyomin Jung
Computation and Language
Low-Resource Heuristics for Bahnaric Optical Character Recognition Improvement
Phat Tran, Phuoc Pham, Hung Trinh et al. (+1 more)
Bahnar, a minority
language spoken across Vietnam, Cambodia, and Laos, faces significant preservation challenges due to limited research and data availability. This study addresses the critical need for accurate digitization of Bahnar
language documents through optical character recognition (OCR) technology. Digitizing scanned paper documents poses significant challenges, as degraded image quality from broken or blurred areas introduces considerable OCR errors that compromise information
retrieval systems. We propose a comprehensive approach combining advanced table and non-table detection techniques with probability-based post-processing heuristics to enhance recognition accuracy. Our method first applies detection
algorithms to improve input data quality, then employs probabilistic error correction on OCR output. Experimental results indicate a substantial improvement, with recognition accuracy increasing from 72.86% to 79.26%. This work contributes valuable resources for Bahnar
language preservation and provides a framework applicable to other minority
language digitization efforts.
Read the full abstract ▾
All authors Phat Tran, Phuoc Pham, Hung Trinh, Tho Quan
Computation and Language
LLM-Augmented Changepoint Detection: A Framework for Ensemble Detection and Automated Explanation
Fabian Lukassen, Christoph Weisser, Michael Schlee et al. (+5 more)
This paper introduces a novel changepoint detection framework that combines ensemble statistical methods with Large
Language Models (LLMs) to enhance both detection accuracy and the interpretability of regime changes in time series data. Two critical limitations in the field are addressed. First, individual detection methods exhibit complementary strengths and weaknesses depending on data characteristics, making method selection non-trivial and prone to suboptimal results. Second, automated, contextual explanations for detected changes are largely absent. The proposed ensemble method aggregates results from ten distinct changepoint detection
algorithms, achieving superior performance and robustness compared to individual methods. Additionally, an LLM-powered explanation pipeline automatically generates contextual narratives, linking detected changepoints to potential real-world historical events. For private or domain-specific data, a
Retrieval-Augmented Generation (RAG) solution enables explanations grounded in user-provided documents. The open source Python framework demonstrates practical utility in diverse domains, including finance, political science, and environmental science, transforming raw statistical output into actionable insights for analysts and decision-makers.
Read the full abstract ▾
All authors Fabian Lukassen, Christoph Weisser, Michael Schlee, Manish Kumar, Anton Thielmann, Benjamin Saefken, Alexander Silbersdorff, Thomas Kneib
Computation and Language
Enhancing Multilingual RAG Systems with Debiased Language Preference-Guided Query Fusion
Jeonghyun Park, Byeongjeong Kim, Seojin Hwang et al. (+1 more)
Multilingual Retrieval-Augmented Generation (mRAG) systems often exhibit a perceived preference for high-resource
languages, particularly English, resulting in the widespread adoption of English pivoting. While prior studies attribute this advantage to the superior English-centric capabilities of Large
Language Models (LLMs), we find that such measurements are significantly distorted by structural priors inherent in evaluation
benchmarks. Specifically, we identify exposure bias and a gold availability prior-both driven by the disproportionate concentration of resources in English-as well as cultural priors rooted in topic locality, as factors that hinder accurate assessment of genuine
language preference. To address these biases, we propose DeLP (Debiased
Language Preference), a calibrated metric designed to explicitly factor out these structural confounds. Our analysis using DeLP reveals that the previously reported English preference is largely a byproduct of evidence distribution rather than an inherent model bias. Instead, we find that retrievers fundamentally favor monolingual
alignment between the query and the document
language. Building on this insight, we introduce DELTA (DEbiased
Language preference-guided Text Augmentation), a lightweight and efficient mRAG framework that strategically leverages monolingual
alignment to optimize cross-lingual
retrieval and generation. Experimental results demonstrate that DELTA consistently outperforms English pivoting and mRAG baselines across diverse
languages.
Read the full abstract ▾
All authors Jeonghyun Park, Byeongjeong Kim, Seojin Hwang, Hwanhee Lee
Computation and Language · Cross-listed from cs.CR
SastBench: A Benchmark for Testing Agentic SAST Triage
Jake Feiglin and Guy Dar
SAST (Static Application Security Testing) tools are among the most widely used techniques in defensive cybersecurity, employed by commercial and non-commercial organizations to identify potential vulnerabilities in software. Despite their great utility, they generate numerous false positives, requiring costly manual filtering (aka triage). While LLM-powered
agents show promise for automating cybersecurity tasks, existing
benchmarks fail to emulate real-world SAST finding distributions. We introduce SastBench, a
benchmark for evaluating SAST triage
agents that combines real CVEs as true positives with filtered SAST tool findings as approximate false positives. SastBench features an
agent-agnostic design. We evaluate different
agents on the
benchmark and present a comparative analysis of their performance, provide a detailed analysis of the
dataset, and discuss the implications for future development.
Read the full abstract ▾
Computation and Language
Pearmut: Human Evaluation of Translation Made Trivial
Vilém Zouhar and Tom Kocmi
Human evaluation is the gold standard for
multilingual NLP, but is often skipped in practice and substituted with automatic metrics because it is notoriously complex and slow to set up with existing tools with substantial engineering and operational overhead. We introduce Pearmut, a lightweight yet feature-rich platform that makes end-to-end human evaluation as easy to run as automatic evaluation. Pearmut removes common entry barriers and provides support for evaluating
multilingual tasks, with a particular focus on machine translation. The platform implements standard evaluation protocols, including DA, ESA, and MQM, and is extensible to support new protocols. It features document-level context, absolute and contrastive evaluation,
attention checks, ESAAI pre-annotations and both static and dynamic assignment strategies. Pearmut enables reliable human evaluation to become a practical, routine component of model development and diagnosis rather than an occasional effort.
Read the full abstract ▾
Computation and Language
Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs
Yihua Zhu, Qianying Liu, Jiaxin Wang et al. (+5 more)
Autoregressive LLMs perform well on relational tasks that require linking entities via relational words (e.g., father/son, friend), but it is unclear whether they learn the logical semantics of such relations (e.g.,
symmetry and inversion logic) and, if so, whether reversal-type failures arise from missing relational semantics or left-to-right order bias. We propose a controlled Knowledge Graph-based synthetic framework that generates text from symmetric/inverse triples, train GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization to unseen entities to address these questions. We find a sharp phase transition in which relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals. Finally, order-matched forward/reverse tests and a
diffusion baseline indicate that reversal failures are primarily driven by autoregressive order bias rather than deficient inversion semantics.
Read the full abstract ▾
All authors Yihua Zhu, Qianying Liu, Jiaxin Wang, Fei Cheng, Chaoran Liu, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira
Computation and Language
RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems
Mengze Hong, Di Jiang, Jiangtao Wen et al. (+5 more)
Hallucination is a major concern in LLM-driven service systems, necessitating explicit knowledge grounding for compliance-guaranteed responses. In this paper, we introduce
Retrieval-Augmented Learning-to-Match (RAL2M), a novel framework that eliminates generation
hallucination by repositioning LLMs as query-response matching judges within a
retrieval-based system, providing a robust alternative to purely generative approaches. To further mitigate judgment
hallucination, we propose a query-adaptive latent ensemble strategy that explicitly models heterogeneous model competence and interdependencies among LLMs, deriving a calibrated consensus decision. Extensive experiments on large-scale
benchmarks demonstrate that the proposed method effectively leverages the "wisdom of the crowd" and significantly outperforms strong baselines. Finally, we discuss best practices and promising directions for further exploiting latent representations in future work.
Read the full abstract ▾
All authors Mengze Hong, Di Jiang, Jiangtao Wen, Zhiyang Su, Yawen Li, Yanjie Sun, Guan Wang, Chen Jason Zhang
Computation and Language · Cross-listed from cs.LG
ChemBART: Pre-training a Reaction Language Model for Comprehensive Synthesis Planning
Kenan Li, Yijian Zhang, Jin Wang et al. (+4 more)
Recent advances in large
language models (LLMs) have demonstrated transformative potential across diverse fields. However, in computer-aided synthesis planning (CASP), current LLM methodologies are often restricted to single-molecule representations, limiting their application to isolated, single-task scenarios such as precursor prediction. Addressing this gap, we present ChemBART, a BART-based
language model that adopts a reaction-centric pre-training strategy. By leveraging a mask-filling pre-training task on holistic reaction expressions (Reactant > Reagent > Product), ChemBART effectively solves a variety of chemical problems, including precursor/reagent generation, temperature-yield regression, and molecular property classification. The reaction-centric pre-training further enables the integration of ChemBART with Monte Carlo tree search (MCTS) for multi-step synthesis route design, where the model serves to optimize the policy and value functions within a reinforcement learning framework. Beyond single-molecule pre-trained LLMs that are constrained to specific applications, ChemBART unifies diverse chemical tasks to facilitate synthesis planning with comprehensive implementation information. Most significantly, the proposed multi-step synthesis routes and reaction conditions were validated in the wet lab, confirming the discovery of shorter synthetic pathways with a >20% yield improvement over established literature
benchmarks. Our work highlights the potential of reaction-focused pre-training and showcases the broad utility of ChemBART in closing the loop between in silico prediction and wet-lab synthesis.
Read the full abstract ▾
All authors Kenan Li, Yijian Zhang, Jin Wang, Haipeng Gan, Zeying Sun, Xiaoguang Lei, Hao Dong
Computation and Language
Image, Word and Thought: A More Challenging Language Task for the Iterated Learning Model
Hyoyeon Lee, Seth Bullock and Conor Houghton
The iterated learning model simulates the transmission of
language from generation to generation in order to explore how the constraints imposed by
language transmission facilitate the emergence of
language structure. Despite each modelled
language learner starting from a blank slate, the presence of a bottleneck limiting the number of utterances to which the learner is exposed can lead to the emergence of
language that lacks ambiguity, is governed by grammatical rules, and is consistent over successive generations, that is, one that is expressive, compositional and stable. The recent introduction of a more computationally tractable and ecologically valid semi supervised iterated learning model, combining supervised and unsupervised learning within an autoencoder architecture, has enabled exploration of
language transmission dynamics for much larger meaning-signal spaces. Here, for the first time, the model has been successfully applied to a
language learning task involving the communication of much more complex meanings: seven-segment display images.
Agents in this model are able to learn and transmit a
language that is expressive: distinct codes are employed for all 128 glyphs; compositional: signal components consistently map to meaning components, and stable: the
language does not change from generation to generation.
Read the full abstract ▾
Computation and Language
Beyond the Black Box: A Survey on the Theory and Mechanism of Large Language Models
Zeyu Gan, Ruifeng Ren, Wei Yao et al. (+9 more)
The rapid emergence of Large
Language Models (LLMs) has precipitated a profound paradigm shift in Artificial Intelligence, delivering monumental engineering successes that increasingly impact modern society. However, a critical paradox persists within the current field: despite the empirical efficacy, our theoretical understanding of LLMs remains disproportionately nascent, forcing these systems to be treated largely as “black boxes”. To address this theoretical fragmentation, this survey proposes a unified lifecycle-based taxonomy that organizes the research landscape into six distinct stages: Data Preparation, Model Preparation, Training,
Alignment, Inference, and Evaluation. Within this framework, we provide a systematic review of the foundational theories and internal mechanisms driving LLM performance. Specifically, we analyze core theoretical issues such as the mathematical justification for data mixtures, the representational limits of various architectures, and the optimization dynamics of
alignment algorithms. Moving beyond current best practices, we identify critical frontier challenges, including the theoretical limits of synthetic data self-improvement, the mathematical bounds of safety guarantees, and the mechanistic origins of emergent intelligence. By connecting empirical observations with rigorous scientific inquiry, this work provides a structured roadmap for transitioning LLM development from engineering heuristics toward a principled scientific discipline.
Read the full abstract ▾
All authors Zeyu Gan, Ruifeng Ren, Wei Yao, Xiaolin Hu, Gengze Xu, Chen Qian, Huayi Tang, Zixuan Gong, Xinhao Yao, Pengwei Tang, Zhenxing Dou, Yong Liu
Computation and Language
Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration
Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang et al. (+8 more)
Multilingual speech foundation models such as Whisper are trained on web-scale data, where data for each
language consists of a myriad of regional varieties. However, different regional varieties often employ different scripts to write the same
language, rendering
speech recognition output also subject to non-determinism in the output script. To mitigate this problem, we show that script is linearly encoded in the activation space of
multilingual speech models, and that modifying activations at inference time enables direct control over output script. We find the addition of such script vectors to activations at test time can induce script change even in unconventional
language-script pairings (e.g. Italian in Cyrillic and Japanese in Latin script). We apply this approach to inducing post-hoc control over the script of
speech recognition output, where we observe competitive performance across all model sizes of Whisper.
Read the full abstract ▾
All authors Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang, Ming-Hao Hsu, Florian Eichin, Zhizheng Wu, Alane Suhr, Michael A. Hedderich, David Harwath, David R. Mortensen, Barbara Plank
Computation and Language · Cross-listed from cs.AI
Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoning
Xinglang Zhang, Yunyao Zhang, ZeLiang Chen et al. (+3 more)
Symbolic logical
reasoning is a critical yet underexplored capability of large
language models (LLMs), providing reliable and verifiable decision-making in high-stakes domains such as mathematical
reasoning and legal judgment. In this study, we present a systematic analysis of logical
reasoning under controlled increases in logical complexity, and reveal a previously unrecognized phenomenon, which we term Logical Phase Transitions: rather than degrading smoothly, logical
reasoning performance remains stable within a regime but collapses abruptly beyond a critical logical depth, mirroring physical phase transitions such as water freezing beyond a critical temperature threshold. Building on this insight, we propose Neuro-Symbolic Curriculum Tuning, a principled framework that adaptively aligns natural
language with logical symbols to establish a shared representation, and reshapes training dynamics around phase-transition boundaries to progressively strengthen
reasoning at increasing logical depths. Experiments on five
benchmarks show that our approach effectively mitigates logical
reasoning collapse at high complexity, yielding average accuracy gains of +1.26 in naive prompting and +3.95 in CoT, while improving generalization to unseen logical compositions. Code and data are available at https://github.com/AI4SS/Logical-Phase-Transitions.
Read the full abstract ▾
All authors Xinglang Zhang, Yunyao Zhang, ZeLiang Chen, Junqing Yu, Wei Yang, Zikai Song
Computation and Language
Transparent Semantic Change Detection with Dependency-Based Profiles
Bach Phan-Tat, Kris Heylen, Dirk Geeraerts et al. (+2 more)
Most modern computational approaches to lexical semantic change detection (LSC) rely on
embedding-based distributional word representations with neural networks. Despite the strong performance on LSC
benchmarks, they are often opaque. We investigate an alternative method which relies purely on dependency co-occurrence patterns of words. We demonstrate that it is effective for semantic change detection and even outperforms a number of distributional semantic models. We provide an in-depth quantitative and qualitative analysis of the predictions, showing that they are plausible and interpretable.
Read the full abstract ▾
All authors Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman
Computation and Language · Cross-listed from cs.AI
ReTreVal: Reasoning Tree with Validation and Cross-Problem Memory for Large Language Models
Abhishek HS, Pavan C Shekar, Arpit Jain et al. (+1 more)
Every existing inference-time
reasoning framework discards all failure context at problem boundaries, leaving a model solving problem 500 no wiser than it was on problem 1. We present ReTreVal (
Reasoning Tree with Validation), a training-free framework that closes this gap through adaptive tree exploration with tool-augmented node refinement, typed-failure backtracking that injects categorized error context into the recovered branch, and a self-rewriting
memory that accumulates and revises strategy entries across problems, enabling inference-time cross-problem learning on any fixed, unmodified LLM without fine-tuning. ReTreVal achieves 85.8% pass@1 on MATH-500 (+8.6 pp over Zero-Shot CoT, +8.6 pp over the strongest baseline Self-Refine) and 54.4% on MMLU-Pro (+15.3 pp over Self-Refine), with a 3.4:1 win-to-regression ratio confirming genuine error recovery rather than noise. These capabilities, previously requiring gradient updates, allow a 32B model to compete with much larger single-pass systems.
Read the full abstract ▾
All authors Abhishek HS, Pavan C Shekar, Arpit Jain, Ashwanth Krishnan
Computation and Language
Revisiting Data Compression with Language Modeling
Chen-Han Tsai
In this report, we investigate the potential use of large
language models (LLM's) in the task of data compression. Previous works have demonstrated promising results in applying LLM's towards compressing not only text, but also a wide range of multi-modal data. Despite the favorable performance achieved, there still remains several practical questions that pose a challenge towards replacing existing data compression
algorithms with LLM's. In this work, we explore different methods to achieve a lower adjusted compression rate using LLM's as data compressors. In comparison to previous works, we were able to achieve a new state-of-the-art (SOTA) adjusted compression rate of around $18\%$ on the enwik9
dataset without additional model training. Furthermore, we explore the use of LLM's in compressing non-English data, code data, byte stream sequences. We show that while LLM's excel in compressing data in text-dominant domains, their ability in compressing non-natural text sequences still remain competitive if configured in the right way.
Read the full abstract ▾
Computation and Language
LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
Ziyang Chen, Xing Wu, Junlong Jia et al. (+4 more)
The rapid expansion of context length in large
language models (LLMs) has outpaced existing evaluation
benchmarks. Current long-context
benchmarks often trade off scalability and realism: synthetic tasks underrepresent real-world complexity, while fully manual annotation is costly to scale to extreme lengths and diverse scenarios. We present LongBench Pro, a more realistic and comprehensive bilingual
benchmark of 1,500 naturally occurring long-context samples in English and Chinese spanning 11 primary tasks and 25 secondary tasks, with input lengths from 8k to 256k
tokens. LongBench Pro supports fine-grained analysis with task-specific metrics and a multi-dimensional taxonomy of context requirement (full vs. partial dependency), length (six levels), and difficulty (four levels calibrated by model performance). To balance quality with scalability, we propose a Human-Model Collaborative Construction pipeline: frontier LLMs draft challenging questions and reference answers, along with design rationales and solution processes, to reduce the cost of expert verification. Experts then rigorously validate correctness and refine problematic cases. Evaluating 46 widely used long-context LLMs on LongBench Pro yields three findings: (1) long-context optimization contributes more to long-context comprehension than parameter scaling; (2) effective context length is typically shorter than the claimed context length, with pronounced cross-lingual misalignment; and (3) the "thinking" paradigm helps primarily models trained with native
reasoning, while mixed-thinking designs offer a promising Pareto trade-off. In summary, LongBench Pro provides a robust testbed for advancing long-context understanding.
Read the full abstract ▾
All authors Ziyang Chen, Xing Wu, Junlong Jia, Chaochen Gao, Qi Fu, Debing Zhang, Songlin Hu
Computation and Language
Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling
Adrian Cosma, Stefan Ruseti, Emilian Radoi et al. (+1 more)
Subword tokenization introduces a computational layer in
language models, where many distinct
token sequences decode to the same surface form while preserving meaning, yet induce different internal computations. Despite this non-uniqueness,
language models are typically trained using a single canonical deterministic tokenization. In this work, we use non-canonical subword variants as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-
token prediction on sampled subword variants via an auxiliary causal encoder and block-causal cross-
attention, without modifying the training objective or
token interface. In data-constrained pretraining, augmentations with subword variants consistently delay overfitting under repeated data exposure and reduce held-out next-
token loss across evaluation
datasets. In
multilingual fine-tuning, we find that the effectiveness of training with auxiliary subwords depends on tokenizer quality: gains are strongest when canonical
tokens are highly compressed and diminish when the tokenizer already over-fragments the input.
Read the full abstract ▾
All authors Adrian Cosma, Stefan Ruseti, Emilian Radoi, Mihai Dascalu
Computation and Language
To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMs
Saurabh Kumar Pandey, Sougata Saha and Monojit Choudhury
Socio-demographic prompting (SDP) - prompting Large
Language Models (LLMs) using demographic proxies to generate culturally aligned outputs - often shows LLM responses as stereotypical and biased. While effective in assessing LLMs' cultural competency, SDP is prone to confounding factors such as
prompt sensitivity, decoding parameters, and the inherent difficulty of generation over discrimination tasks due to larger output spaces. These factors complicate interpretation, making it difficult to determine if the poor performance is due to bias or the task design. To address this, we use inverse socio-demographic prompting (ISDP), where we
prompt LLMs to discriminate and predict the demographic proxy from actual and simulated user behavior from different users. We use the Goodreads-CSI
dataset (Saha et al., 2025), which captures difficulty in understanding English book reviews for users from India, Mexico, and the USA, and test four LLMs: Aya-23, Gemma-2, GPT-4o, and LLaMA-3.1 with ISDP. Results show that models perform better with actual behaviors than simulated ones, contrary to what SDP suggests. However, performance with both behavior types diminishes and becomes nearly equal at the individual level, indicating limits to personalization.
Read the full abstract ▾
Computation and Language
TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents
Kai Li, Xuanqing Yu, Ziyi Ni et al. (+9 more)
Long-horizon conversational
agents have to manage ever-growing interaction histories that quickly exceed the finite context windows of large
language models (LLMs). Existing
memory frameworks provide limited support for temporally structured information across hierarchical levels, often leading to fragmented
memories and unstable long-horizon personalization. We present TiMem, a temporal–hierarchical
memory framework that organizes conversations through a Temporal
Memory Tree (TMT), enabling systematic
memory consolidation from raw conversational observations to progressively abstracted persona representations. TiMem is characterized by three core properties: (1) temporal–hierarchical organization through TMT; (2) semantic-guided consolidation that enables
memory integration across hierarchical levels without fine-tuning; and (3) complexity-aware
memory recall that balances precision and efficiency across queries of varying complexity. Under a consistent evaluation setup, TiMem achieves state-of-the-art accuracy on both
benchmarks, reaching 75.30% on LoCoMo and 76.88% on LongMemEval-S. It outperforms all evaluated baselines while reducing the recalled
memory length by 52.20% on LoCoMo.
Manifold analysis indicates clear persona separation on LoCoMo and reduced dispersion on LongMemEval-S. Overall, TiMem treats temporal continuity as a first-class organizing principle for long-horizon
memory in conversational
agents. The code is available at https://github.com/TiMEM-AI/timem.
Read the full abstract ▾
All authors Kai Li, Xuanqing Yu, Ziyi Ni, Yi Zeng, Yao Xu, Zheqing Zhang, Xin Li, Jitao Sang, Xiaogang Duan, Xuelei Wang, Chengbao Liu, Jie Tan
Computation and Language
The performances of the Chinese and U.S. Large Language Models on the Topic of Chinese Culture
Feiyan Liu, Siyan Zhao, Chenxun Zhuo et al. (+2 more)
Cultural backgrounds shape individuals' perspectives and approaches to problem-solving. Since the emergence of GPT-1 in 2018, large
language models (LLMs) have undergone rapid development. To date, the world's ten leading LLM developers are primarily based in China and the United States. To examine whether LLMs released by Chinese and U.S. developers exhibit cultural differences in Chinese-
language settings, we evaluate their performance on questions about Chinese culture. This study adopts a direct-questioning paradigm to evaluate models such as GPT-5.1, DeepSeek-V3.2, Qwen3-Max, and Gemini2.5Pro. We assess their understanding of traditional Chinese culture, including history, literature, poetry, and related domains. Comparative analyses between LLMs developed in China and the U.S. indicate that Chinese models generally outperform their U.S. counterparts on these tasks. Among U.S.-developed models, Gemini 2.5Pro and GPT-5.1 achieve relatively higher accuracy. The observed performance differences may potentially arise from variations in training data distribution, localization strategies, and the degree of emphasis on Chinese cultural content during model development.
Read the full abstract ▾
All authors Feiyan Liu, Siyan Zhao, Chenxun Zhuo, Tianming Liu, Bao Ge
Computation and Language
Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models
Junxiang Qiu, Shuo Wang, Zhengsu Chen et al. (+4 more)
Attention serves as the fundamental mechanism for long-context modeling in large
language models (LLMs), yet dense
attention becomes structurally prohibitive for long sequences due to its quadratic complexity. Consequently, sparse
attention has received increasing
attention as a scalable alternative. However, existing sparse
attention methods rely on coarse-grained semantic representations during block selection, which blur intra-block semantic boundaries and lead to the loss of critical information. To address this issue, we propose Punctuation-aware Hybrid Sparse
Attention (PHSA), a natively trainable sparse
attention framework that leverages punctuation
tokens as semantic boundary anchors. Specifically, (1) we design a dual-branch aggregation mechanism that fuses global semantic representations with punctuation-enhanced boundary features, preserving the core semantic structure while introducing almost no additional computational overhead; (2) we introduce an extreme-sparsity-adaptive training and inference strategy that stabilizes model behavior under very low
token activation ratios; Extensive experiments on general
benchmarks and long-context evaluations demonstrate that PHSA consistently outperforms dense
attention and state-of-the-art sparse
attention baselines, including InfLLM v2. Specifically, for the 0.6B-parameter model with 32k-
token input sequences, PHSA can reduce the information loss by 10.8% at a sparsity ratio of 97.3%.
Read the full abstract ▾
All authors Junxiang Qiu, Shuo Wang, Zhengsu Chen, Hengheng Zhang, Jinda Lu, Changcheng Li, Qi Tian