Computation and Language
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
Yuchen Lu, Run Yang, Yichen Zhang et al. (+6 more)
Despite rapid advances in large
language models (LLMs), statistical
reasoning remains underrepresented in existing LLM
benchmarks, which often do not reflect the layered, proof-driven nature of real statistical practice. To address this gap, we introduce StatEval, the first large-scale
benchmark for statistical
reasoning across curricular and research-level settings. StatEval includes over 100,000 curated problems, with 20,000+ foundational questions spanning undergraduate and graduate curricula and 80,000+ research-level proof tasks extracted from leading statistical journals. To construct StatEval, we develop TRACE (
Topology and
Reasoning-Aware Context Extractor), a multi-
agent pipeline with human-in-the-loop validation that converts unstructured academic texts into self-contained
theorem-level
reasoning tasks. We also propose an Adaptive Process-Based Scoring Pipeline for complex statistical proofs, enabling fine-grained evaluation beyond final-answer matching. Experiments show that while LLMs perform reasonably on foundational tasks, they struggle with rigorous research-level
reasoning. Beyond evaluation, StatEval serves as a resource for improving
reasoning, as
retrieval-augmented generation and domain-specific
alignment consistently enhance performance. Together, these results establish StatEval as both a
benchmark and an infrastructure for advancing statistical
reasoning in LLMs.
Read the full abstract ▾
All authors Yuchen Lu, Run Yang, Yichen Zhang, Shuguang Yu, Ziwei Wang, Jiayi Xiang, Wenxin E, Changyu Zhu, Fan Zhou
Computation and Language
Multimodal Policy Internalization for Conversational Agents
Zhenhailong Wang, Jiateng Liu, Amin Fazel et al. (+6 more)
Modern conversational
agents like ChatGPT and Alexa+ rely on predefined policies specifying metadata, response styles, and tool-usage rules. As these LLM-based systems expand to support diverse business and user queries, such policies, often implemented as in-context
prompts, are becoming increasingly complex and lengthy, making faithful adherence difficult and imposing large fixed computational costs. With the rise of
multimodal agents, policies that govern visual and
multimodal behaviors are critical but remain understudied. Prior
prompt-compression work mainly shortens task templates and demonstrations, while existing policy-
alignment studies focus only on text-based safety rules. We introduce
Multimodal Policy Internalization (MPI), a new task that internalizes
reasoning-intensive
multimodal policies into model parameters, enabling stronger policy-following without including the policy during inference. MPI poses unique data and algorithmic challenges. We build two
datasets spanning synthetic and real-world decision-making and tool-using tasks and propose TriMPI, a three-stage training framework. TriMPI first injects policy knowledge via continual pretraining, then performs supervised finetuning, and finally applies PolicyRollout, a GRPO-style reinforcement learning extension that augments rollouts with policy-aware responses for grounded exploration. TriMPI achieves notable gains in end-to-end accuracy, generalization, and robustness to forgetting. As the first work on
multimodal policy internalization, we provide
datasets, training recipes, and comprehensive evaluations to foster future research. Project page: https://mikewangwzhl.github.io/TriMPI.
Read the full abstract ▾
All authors Zhenhailong Wang, Jiateng Liu, Amin Fazel, Ritesh Sarkhel, Xing Fan, Xiang Li, Chenlei Guo, Heng Ji, Ruhi Sarikaya
Computation and Language
Hybrid Models for Natural Language Reasoning: The Case of Syllogistic Logic
Manuel Vargas Guzmán, Jakub Szymanik and Maciej Malicki
Despite the remarkable progress in neural models, their ability to generalize, a cornerstone for applications such as logical
reasoning, remains a critical challenge. We delineate two fundamental aspects of this ability: compositionality, the capacity to abstract atomic logical rules underlying complex inferences, and recursiveness, the aptitude to build intricate representations through iterative application of inference rules. In the literature, these two aspects are often conflated under the umbrella term of generalization. To sharpen this distinction, we investigate the logical generalization capabilities of LLMs using the syllogistic fragment as a
benchmark for natural
language reasoning. We extend classical syllogistic forms to construct more complex structures, yielding a foundational yet expressive subset of formal logic that supports controlled evaluation of essential
reasoning abilities. Our findings on this non-trivial
benchmark show that, while LLMs demonstrate reasonable proficiency in recursiveness, they struggle with compositionality. This disparity is not uniform, as a more detailed analysis reveals substantial variability in generalization performance across individual syllogistic types, ranging from near-perfect accuracy to significantly lower performance. To overcome these limitations and establish a reliable logical prover, we propose a hybrid architecture integrating symbolic
reasoning with neural computation. This synergistic interaction enables robust and efficient inference, neural components accelerate processing, while symbolic
reasoning guarantees completeness. Our experiments further show that high efficiency is preserved even when using relatively small neural components. Overall, our analysis provides both a rationale for hybrid neuro-symbolic approaches and evidence of their potential to address key generalization barriers in neural
reasoning systems.
Read the full abstract ▾
Computation and Language
Getting Your Indices in a Row: Full-Text Search for LLM Training Data for Real World
Ines Altemir Marinas, Anastasiia Kucherenko, Alexander Sternfeld et al. (+1 more)
The performance of Large
Language Models (LLMs) is determined by their training data. Despite the proliferation of open-weight LLMs, access to LLM training data has remained limited. Even for fully open LLMs, the scale of the data makes it all but inscrutable to the general scientific community, despite potentially containing critical data scraped from the internet. In this paper, we present the full-text indexing pipeline for the Apertus LLM training data. Leveraging Elasticsearch parallel indices and the Alps infrastructure, a state-of-the-art, highly energy-efficient arm64 supercluster, we were able to index 8.6T
tokens out of 15.2T used to train the Apertus LLM family, creating both a critical LLM safety tool and effectively an offline, curated, open web search engine. Our contribution is threefold. First, we demonstrate that Elasticsearch can be successfully ported onto next-generation arm64-based infrastructure. Second, we demonstrate that full-text indexing at the scale of modern LLM training
datasets and the entire open web is feasible and accessible. Finally, we demonstrate that such indices can be used to ensure previously inaccessible jailbreak-agnostic LLM safety. We hope that our findings will be useful to other teams attempting large-scale data indexing and facilitate the general transition towards greener computation.
Read the full abstract ▾
All authors Ines Altemir Marinas, Anastasiia Kucherenko, Alexander Sternfeld, Andrei Kucharavy
Computation and Language
Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narratives
Xixi Wang, Jordanka Kovaceva, Miguel Costa et al. (+3 more)
Free-text crash narratives recorded in real-world crash databases have been shown to play a significant role in improving traffic safety. However, large-scale analyses remain difficult to implement as there are no documented tools that can batch process the unstructured, non standardized text content written by various authors with diverse experience and
attention to detail. In recent years,
Transformer-based pre-trained
language models (PLMs), such as Bidirectional Encoder Representations from
Transformers (BERT) and large
language models (LLMs), have demonstrated strong capabilities across various natural
language processing tasks. These models can extract explicit facts from crash narratives, but their performance declines on inference-heavy tasks in, for example, Crash Type identification, which can involve nearly 100 categories. Moreover, relying on closed LLMs through external APIs raises
privacy concerns for sensitive crash data. Additionally, these black-box tools often underperform due to limited domain knowledge. Motivated by these challenges, we study whether compact open-source PLMs can support
reasoning-intensive extraction from crash narratives. We target two challenging objectives: 1) identifying the Manner of Collision for a crash, and 2) Crash Type for each vehicle involved in the crash event from real-world crash narratives. To bridge domain gaps, we apply fine-tuning techniques to inject task-specific knowledge to LLMs with Low-Rank Adaption (LoRA) and BERT. Experiments on the authoritative real-world
dataset Crash Investigation Sampling System (CISS) demonstrate that our fine-tuned compact models outperform strong closed LLMs, such as GPT-4o, while requiring only minimal training resources. Further analysis reveals that the fine-tuned PLMs can capture richer narrative details and even correct some mislabeled annotations in the
dataset.
Read the full abstract ▾
All authors Xixi Wang, Jordanka Kovaceva, Miguel Costa, Shuai Wang, Francisco Camara Pereira, Robert Thomson
Computation and Language
KORMo: Korean Open Reasoning Model for Everyone
Minjun Kim, Hyeonseok Lim, Hangyeol Yoo et al. (+10 more)
This work presents the first large-scale investigation into constructing a fully open bilingual large
language model (LLM) for a non-English
language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight
multilingual baselines across a wide range of
reasoning, knowledge, and instruction-following
benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native
reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future
multilingual LLM research.
Read the full abstract ▾
All authors Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junhun Yuk, Changsu Choi, Dongjae Shin, Huige Lee, Hoyun Song, Alice Oh, Kyungtae Lim
Computation and Language
The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach
Nizar El Ghazal, Antoine Caubrière and Valentin Vielzeuf
This paper presents a comparative study of context management strategies for end-to-end Spoken Dialog State Tracking using
Speech-LLMs. We systematically evaluate traditional
multimodal context (combining text history and spoken current turn), full spoken history, and compressed spoken history approaches. Our experiments on the SpokenWOZ corpus demonstrate that providing the full spoken conversation as input yields the highest performance among models of similar size, significantly surpassing prior methods. Furthermore, we show that
attention-pooling-based compression of the spoken history offers a strong trade-off, maintaining competitive accuracy with reduced context size. Detailed analysis confirms that improvements stem from more effective context utilization.
Read the full abstract ▾
Computation and Language
On the Representations of Entities in Auto-regressive Large Language Models
Victor Morand, Josiane Mothe and Benjamin Piwowarski
Named entities are fundamental building blocks of knowledge in text, grounding factual information and structuring relationships within
language. Despite their importance, it remains unclear how Large
Language Models (LLMs) internally represent entities. Prior research has primarily examined explicit relationships, but little is known about entity representations themselves. We introduce entity mention reconstruction as a novel framework for studying how LLMs encode and manipulate entities. We investigate whether entity mentions can be generated from internal representations, how multi-
token entities are encoded beyond last-
token embeddings, and whether these representations capture relational knowledge. Our proposed method, leveraging _task vectors_, allows to consistently generate multi-
token mentions from various entity representations derived from the LLMs hidden states. We thus introduce the _Entity Lens_, extending the _logit-lens_ to predict multi-
token mentions. Our results bring new evidence that LLMs develop entity-specific mechanisms to represent and manipulate any multi-
token entities, including those unseen during training. Our code is avalable at https://github.com/VictorMorand/EntityRepresentations .
Read the full abstract ▾
Computation and Language
Large Language Model Selection with Limited Annotations
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch et al. (+2 more)
Choosing a Large
Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23
datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Read the full abstract ▾
All authors Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve Gürel
Computation and Language · Cross-listed from q-bio.NC
Estimating Brain Activity with High Spatial and Temporal Resolution using a Naturalistic MEG-fMRI Encoding Model
Beige Jerry Jin and Leila Wehbe
Current non-invasive neuroimaging techniques trade off between spatial resolution and temporal resolution. While magnetoencephalography (MEG) can capture rapid neural dynamics and functional
magnetic resonance imaging (fMRI) can spatially localize
brain activity, a unified picture that preserves both high resolutions remains an unsolved challenge with existing source localization or MEG-fMRI fusion methods, especially for single-trial naturalistic data. We collected whole-head MEG when subjects listened passively to more than seven hours of narrative stories, using the same stimuli in an open fMRI
dataset (LeBel et al., 2023). We developed a
transformer-based encoding model that combines the MEG and fMRI from these two naturalistic
speech comprehension experiments to estimate latent cortical source responses with high spatiotemporal resolution. Our model is trained to predict MEG and fMRI from multiple subjects simultaneously, with a latent layer that represents our estimates of reconstructed cortical sources. Our model predicts MEG better than the common standard of single-modality encoding models, and it also yields source estimates with higher spatial and temporal fidelity than classic minimum-norm solutions in simulation experiments. We validated the estimated latent sources by showing its strong generalizability across unseen subjects and modalities. Estimated activity in our source space predict electrocorticography (ECoG) better than an ECoG-trained encoding model in an entirely new
dataset. By integrating the power of large naturalistic experiments, MEG, fMRI, and encoding models, we propose a practical route towards millisecond-and-millimeter
brain mapping.
Read the full abstract ▾
Computation and Language
Beyond Single-Granularity Prompts: A Multi-Scale Chain-of-Thought Prompt Learning for Graph
Ziyu Zheng, Yaming Yang, Ziyu Guan et al. (+3 more)
The “pre-train,
prompt" paradigm, designed to bridge the gap between pre-training tasks and downstream objectives, has been extended from the NLP domain to the graph domain and has achieved remarkable progress. Current mainstream graph
prompt-tuning methods modify input or output features using learnable
prompt vectors. However, existing approaches are confined to single-granularity (e.g., node-level or subgraph-level) during
prompt generation, overlooking the inherently multi-scale structural information in graph data, which limits the diversity of
prompt semantics. To address this issue, we pioneer the integration of multi-scale information into graph
prompt and propose a Multi-Scale Graph Chain-of-Thought (MSGCOT) prompting framework. Specifically, we design a lightweight, low-rank coarsening network to efficiently capture multi-scale structural features as hierarchical basis vectors for
prompt generation. Subsequently, mimicking human cognition from coarse-to-fine granularity, we dynamically integrate multi-scale information at each
reasoning step, forming a progressive coarse-to-fine
prompt chain. Extensive experiments on eight
benchmark datasets demonstrate that MSGCOT outperforms the state-of-the-art single-granularity graph
prompt-tuning method, particularly in few-shot scenarios, showcasing superior performance. The code is available at: https://github.com/zhengziyu77/MSGCOT.
Read the full abstract ▾
All authors Ziyu Zheng, Yaming Yang, Ziyu Guan, Wei Zhao, Xinyan Huang, Weigang Lu
Computation and Language
Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation
Mert İnan, Anthony Sicilia, Alex Xie et al. (+3 more)
Establishing shared goals is a fundamental step in human-AI communication. However, ambiguities can lead to outputs that seem correct but fail to reflect the speaker's intent. In this paper, we explore this issue with a focus on the data visualization domain, where ambiguities in natural
language impact the generation of code that visualizes data. The availability of multiple views on the contextual (e.g., the intended plot and the code rendering the plot) allows for a unique and comprehensive analysis of diverse ambiguity types. We develop a taxonomy of types of ambiguity that arise in this task and propose metrics to quantify them. Using Matplotlib problems from the DS-1000
dataset, we demonstrate that our ambiguity metrics better correlate with human annotations than uncertainty baselines. Our work also explores how multi-turn dialogue can reduce ambiguity, therefore, improve code accuracy by better matching user goals. We evaluate three pragmatic models to inform our dialogue strategies: Gricean Cooperativity, Discourse Representation Theory, and Questions under Discussion. A simulated user study reveals how pragmatic dialogues reduce ambiguity and enhance code accuracy, highlighting the value of multi-turn exchanges in code generation.
Read the full abstract ▾
All authors Mert İnan, Anthony Sicilia, Alex Xie, Saujas Vaduguru, Daniel Fried, Malihe Alikhani
Computation and Language · Cross-listed from cs.LG
Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
Xinyi Wang, Jinyi Han, Zishang Jiang et al. (+7 more)
Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT)
reasoning capabilities of Large
Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance
reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse
benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.
Read the full abstract ▾
All authors Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, Yanghua Xiao
Computation and Language
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Xingyu Lin, Yilin Wen, En Wang et al. (+4 more)
Group Relative Policy Optimization (GRPO) has significantly advanced the
reasoning ability of large
language models (LLMs), particularly by boosting their mathematical performance. However, GRPO and related
entropy-regularization methods still face challenges rooted in the sparse
token rewards inherent to chain-of-thought (CoT). Current approaches often rely on undifferentiated
token-level
entropy adjustments, which frequently lead to
entropy collapse or model collapse. In this work, we propose TEPO, a novel
token-level framework that incorporates Markov Likelihood (sequence likelihood) links group-level rewards with
tokens via
token-level aggregation. Experiments show that TEPO consistently outperforms existing baselines across key metrics (including @k and accuracy). It not only sets a new state of the art on mathematical
reasoning tasks but also significantly enhances training stability.
Read the full abstract ▾
All authors Xingyu Lin, Yilin Wen, En Wang, Du Su, Wenbin Liu, Chenfu Bao, Zhonghou Lv
Computation and Language
Understanding the Effects of Domain Finetuning on LLMs
Eshaan Tanwar, Deepak Nathani, William Yang Wang et al. (+1 more)
Large
Language Models (LLMs) fine-tuned for specific domains exhibit strong performance; however, the underlying mechanisms by which this fine-tuning reshapes their parametric space are not well understood. Prior works primarily focus on auto-regressive or general-purpose instruct models, leaving domain-specialised LLMs under-explored. We present the first systematic study of domain-specific fine-tuning in large medical
language models. Our analysis reveals that fine-tuning modifies only a small subset of the representational subspace, essentially preserving the pre-trained model's representation. To interpret these changes in subspaces, we propose tuning vectors, a novel framework inspired by task vectors, which explicitly capture the directional parameter shifts induced by fine-tuning. We demonstrate that these vectors are critical for enhancing both instruction-following and generation quality. Furthermore, combining tuning vectors across different domains yields improved generalisation. Upon closer inspection of directional
alignment, we find these vectors primarily write new directional information into the MLP layers of the model, while amplifying existing directions in
attention heads. Our findings offer new insights into LLM adaptation and provide a general, interpretable framework for analysing specialisation in large
language models.
Read the full abstract ▾
All authors Eshaan Tanwar, Deepak Nathani, William Yang Wang, Tanmoy Chakraborty
Computation and Language
NL2GenSym: Natural Language to Generative Symbolic Rules for SOAR Cognitive Architecture via Large Language Models
Fang Yuan, Junjie Zeng, Yue Hu et al. (+3 more)
SOAR, a classic symbol-based cognitive architecture, has been fostering the development of general, human-like intelligent
agents. Nevertheless, its practical adoption is hindered by the laborious manual rule coding. Emerging Large
Language Models (LLMs) present the immense potential for efficient rules generation. However, there is a critical gap that current research predominantly focuses on conceptual frameworks and lacks robust experimental validation. To bridge this gap, we propose Natural
Language to Generative Symbolic Rules (NL2GenSym), a novel framework that integrates LLMs with SOAR to autonomously produce generative symbolic rules from natural
language. Specifically, our framework introduces a novel Execution-Grounded Generator-Critic mechanism. The LLM-based Generator, guided by a
Retrieval-Augmented Generation-accessed self-evolving domain knowledge base, proposes rules from natural
language. Subsequently, these rules are immediately executed within the SOAR environment to rigorously validate their correctness. Based on this execution-grounded feedback, a reflective LLM-based Critic drives the iterative refinement of these rules. Experiments on our specialized Water Jug Problem (WJP)
dataset, utilizing both Gemini and Qwen series models, validate the efficacy of our framework. It achieves a success rate over 86% in generating rules from natural
language. Crucially, the framework also generates novel heuristic rules, reducing average decision cycles for solving the WJP to 1.98 times the optimal solution and 1/1000 of baseline methods. Additionally, our initial experiments show that NL2GenSym enables smaller-parameter models to achieve better performance than larger counterparts.
Read the full abstract ▾
All authors Fang Yuan, Junjie Zeng, Yue Hu, Zhengqiu Zhu, Quanjun Yin, Yuxiang Xie
Computation and Language
Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
Yunxiang Zhang, Muhammad Khalifa, Lechen Zhang et al. (+5 more)
Large
reasoning models exhibit long chain-of-thought
reasoning with complex strategies such as backtracking and self-verification. Yet, these capabilities typically require resource-intensive post-training. We investigate whether such behaviors can be elicited in large models without any gradient updates. To this end, we propose a decoding-time approach, ThinkLogit, which utilizes logit arithmetic to transfer these capabilities from a substantially smaller
reasoning guider to a large non-
reasoning target. We further show that we can boost performance by training the guider to correct the target's errors using preference optimization over mixed model outputs, a setup we refer to as ThinkLogit-DPO. We evaluate these methods across six
reasoning benchmarks spanning math, science, and coding domains using the Qwen2.5-32B guided by R1-Distill-Qwen-1.5B, a model 21x smaller. Our experiments demonstrate that ThinkLogit and ThinkLogit-DPO achieve a relative improvement of 21.5% and 24.2%, respectively, over the target model. Moreover, ThinkLogit remains effective even when the guider and target come from different model families. Crucially, our method requires zero training for the large model and would incur minimal inference overhead when logits are computed in parallel, presenting a practical solution for enabling long
reasoning at scale.
Read the full abstract ▾
All authors Yunxiang Zhang, Muhammad Khalifa, Lechen Zhang, Xin Liu, Ayoung Lee, Xinliang Frederick Zhang, Farima Fatahi Bayat, Lu Wang
Computation and Language
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
Francesco Maria Molfese, Luca Moroni, Ciro Porcaro et al. (+2 more)
While Small
Language Models (SLMs) have demonstrated promising performance on an increasingly wide array of commonsense
reasoning benchmarks, current evaluation practices rely almost exclusively on the accuracy of their final answers, neglecting the validity of the
reasoning processes that lead to those answers. To address this issue, we present ReTraceQA, a novel
benchmark that introduces process-level evaluation for commonsense
reasoning tasks. Our expert-annotated
dataset reveals that in a substantial portion of instances (14-24%), SLMs provide correct final answers despite flawed
reasoning processes, suggesting that the capabilities of SLMs are often overestimated by evaluation metrics that focus only on comparing the final answer with the ground truth. Indeed, we show that, when employing strong Large
Language Models (LLMs) as automated judges for
reasoning-aware evaluation rather than answer-only metrics, SLM performance drops significantly across all models and
datasets, with scores decreasing by up to 25%.
Read the full abstract ▾
All authors Francesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia, Roberto Navigli
Computation and Language
LLP: LLM-Based Product Pricing in E-commerce
Hairu Wang, Sheng You, Qiheng Zhang et al. (+5 more)
Unlike Business-to-Consumer e-commerce platforms (e.g., Amazon), inexperienced individual sellers on Consumer-to-Consumer platforms (e.g., eBay) often face significant challenges in setting prices for their second-hand products efficiently. Therefore, numerous studies have been proposed for automating price prediction. However, most of them are based on static regression models, which suffer from poor generalization performance and fail to capture
market dynamics (e.g., the price of a used iPhone decreases over time). Inspired by recent breakthroughs in Large
Language Models (LLMs), we introduce LLP, the first LLM-based generative framework for second-hand product pricing. LLP first retrieves similar products to better align with the dynamic
market change. Afterwards, it leverages the LLMs' nuanced understanding of key pricing information in free-form text to generate accurate price suggestions. To strengthen the LLMs' domain
reasoning over retrieved products, we apply a two-stage optimization, supervised fine-tuning (SFT) followed by group relative policy optimization (GRPO), on a
dataset built via bidirectional
reasoning. Moreover, LLP employs a confidence-based filtering mechanism to reject unreliable price suggestions. Extensive experiments demonstrate that LLP substantially surpasses existing methods while generalizing well to unseen categories. We have successfully deployed LLP on Xianyu\footnote\{Xianyu is China's largest second-hand e-commerce platform.\}, significantly outperforming the previous pricing method. Under the same 30% product coverage, it raises the static adoption rate (SAR) from 40% to 72%, and maintains a strong SAR of 47% even at 90% recall.
Read the full abstract ▾
All authors Hairu Wang, Sheng You, Qiheng Zhang, Xike Xie, Shuguang Han, Yuchen Wu, Fei Huang, Jufeng Chen
Computation and Language · Cross-listed from cs.AI
Deductive Logic in Language Models: Horizontal vs Vertical Reasoning
Davide Maltoni and Matteo Ferrara
Recent
language models exhibit significant logical
reasoning abilities, yet the mechanisms supporting deductive inference remain poorly understood. This paper studies small
transformer-based
language models trained from scratch on multi-step deductive tasks, focusing on the distinction between horizontal
reasoning, where intermediate steps are generated autoregressively, and vertical
reasoning, where inference unfolds implicitly across layers before the first output
token is produced. We analyze two synthetic tasks: logical consequence over chains of symbolic implications and root-to-leaf
navigation in binary trees. Mechanistic interpretability reveals that Chain-of-Thought supervision enables models to learn rule-based inference rather than statistical shortcuts. In the horizontal setting, a shallow
attention-only model develops interpretable
circuits for rule completion, rule chaining, and final decision making, largely implemented through induction-head-like mechanisms. We further introduce a truncated pseudoinverse method to decode the information carried by queries, keys, and values. For vertical
reasoning, Chain-of-Thought appears to act less as explicit step-by-step guidance and more as a form of curriculum learning, helping the model acquire increasingly complex
reasoning patterns. Without Chain-of-Thought, models tend to memorize or exploit
dataset biases. These results provide a low-level account of how
transformers can implement deductive
reasoning and suggest how Chain-of-Thought may serve different functions in horizontal and vertical
reasoning.
Read the full abstract ▾
Computation and Language
FLRC: Fine-grained Low-Rank Compressor for Efficient LLM Inference
Yu-Chen Lu, Chong-Yan Chen, Chi-Chih Chang et al. (+2 more)
Although large
language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both
memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse
benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.
Read the full abstract ▾
All authors Yu-Chen Lu, Chong-Yan Chen, Chi-Chih Chang, Yu-Fang Hu, Kai-Chiang Wu
Computation and Language · Cross-listed from cs.LG
A Survey of LLM Prompt Datasets: Taxonomy, Linguistic Patterns, and Practical Uses
Yuanming Zhang, Yan Lin, Arijit Khan et al. (+1 more)
We compile 129 public LLM
prompt datasets with more than 1.22TB and more than 673M instances and organize them into a unified taxonomy. We use seven
datasets for detailed analysis and identify lexical, syntactic, and semantic patterns that distinguish
prompts from general text. We evaluate these features in
prompt filtering, source domain routing, and elicited response quality analysis. A 63 dimensional linguistic feature set that can be extracted on a CPU achieves over 91% of the F1 of GPU sentence
embeddings and reduces latency per request by nearly half. In cross
dataset routing, sentence
embeddings reach 0.74 Macro-F1 and a structural feature subset reaches 0.61, compared with the chance level of 0.20. Structural routing features remain negatively associated with elicited response quality after
prompt length is controlled on UltraFeedback. The apparent positive association of type to
token ratio becomes much smaller under the same control. These findings support
prompt processing pipelines that combine efficient structural features with robust semantic routing. Our catalogue, annotations, and code are given at https://github.com/ymzhang-cs/
prompt-
dataset-analysis.
Read the full abstract ▾
All authors Yuanming Zhang, Yan Lin, Arijit Khan, Huaiyu Wan
Computation and Language
Verifying Chain-of-Thought Reasoning via Its Computational Graph
Zheng Zhao, Yeskendir Koishekenov, Xianjun Yang et al. (+2 more)
Current Chain-of-Thought (CoT) verification methods predict
reasoning correctness based on outputs (black-box) or activations (gray-box), but offer limited insight into why a computation fails. We introduce a white-box method:
Circuit-based
Reasoning Verification (CRV). We hypothesize that attribution graphs of correct CoT steps, viewed as execution traces of the model's latent
reasoning circuits, possess distinct structural fingerprints from those of incorrect steps. By training a classifier on structural features of these graphs, we show that these traces contain a powerful signal of
reasoning errors. Our white-box approach yields novel scientific insights unattainable by other methods. (1) We demonstrate that structural signatures of error are highly predictive, establishing the viability of verifying
reasoning directly via its computational graph. (2) We find these signatures to be highly domain-specific, revealing that failures in different
reasoning tasks manifest as distinct computational patterns. (3) We provide evidence that these signatures are not merely correlational; by using our analysis to guide targeted interventions on individual transcoder features, we successfully correct the model's faulty
reasoning. Our work shows that, by scrutinizing a model's computational process, we can move from simple error detection to a deeper, causal understanding of LLM
reasoning.
Read the full abstract ▾
All authors Zheng Zhao, Yeskendir Koishekenov, Xianjun Yang, Naila Murray, Nicola Cancedda
Computation and Language
Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
Jianuo Huang, Yaojie Zhang, Yicun Yang et al. (+4 more)
Diffusion large
language models (dLLMs) present a promising alternative to dominant autoregressive models (ARMs) by the ability of parallel decoding at the expense of substantial computation and
memory costs. Specifically, the cache mechanism for bidirectional
attention in dLLMs demands large
memory footprint, restricting their ability to handle long contexts under resource-limited settings. Existing cache eviction strategies are designed for ARMs and ignore the unique characteristics of dLLMs, thus leading to unsatisfactory performance. To address these challenges, we introduce MaskKV, a training-free cache eviction framework tailored to dLLMs, focusing on the effect of mask
tokens in dLLMs. MaskKV is built on two key innovations: (1) a mask-query guided scoring mechanism that leverages
attention weights to identify and evict less critical
prompt tokens for each head; (2) an adaptive cache budgeting strategy that improves efficiency by reducing allocation in intermediate layers and concentrating resources on
prompt-preferring heads. On LLaDA with MaskKV, compressing the KV cache to only 256 pairs (less than 5% of
tokens) retains 94% of the full-cache performance on LongBench and achieves up to 31x acceleration at 32k
prompt length. The code is publicly available at: https://github.com/jianuo-huang/MaskKV
Read the full abstract ▾
All authors Jianuo Huang, Yaojie Zhang, Yicun Yang, Benhao Huang, Biqing Qi, Dongrui Liu, Linfeng Zhang
Computation and Language · Cross-listed from eess.AS
Target speaker anonymization in multi-speaker recordings
Natalia Tomashenko, Junichi Yamagishi, Xin Wang et al. (+2 more)
Most of the existing speaker anonymization research has focused on single-speaker audio, leading to the development of techniques and evaluation metrics optimized for such condition. This study addresses the significant challenge of speaker anonymization within multi-speaker conversational audio, specifically when only a single target speaker needs to be anonymized. This scenario is highly relevant in contexts like call centers, where customer
privacy necessitates anonymizing only the customer's voice in interactions with operators. Conventional anonymization methods are often not suitable for this task. Moreover, current evaluation methodology does not allow us to accurately assess
privacy protection and utility in this complex multi-speaker scenario. This work aims to bridge these gaps by exploring effective strategies for targeted speaker anonymization in conversational audio, highlighting potential problems in their development and proposing corresponding improved evaluation methodologies.
Read the full abstract ▾
All authors Natalia Tomashenko, Junichi Yamagishi, Xin Wang, Yun Liu, Emmanuel Vincent
Computation and Language · Cross-listed from cs.CV
CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understanding
Yuying Li, Siyi Qian, Hao Liang et al. (+5 more)
While
Multimodal Large
Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical
reasoning tasks, even advanced closed-source models such as GPT-o3 still struggle with
geometric problems. This discrepancy motivates us to investigate the root cause: is the bottleneck of
multimodal geometric reasoning rooted in
reasoning itself, or in the perception of
geometric information from diagrams? To answer this question, we conduct an exploratory experiment and find that providing high-quality captions consistently and substantially boosts performance of MLLMs, empirically validating the visual perception bottleneck in
geometric reasoning. However, MLLMs' capabilities in visual
geometric perception remain insufficiently evaluated. To fill the gap, we propose CapGeo-Bench, a
benchmark of 4,641 high-quality figure-caption pairs equipped with a fine-grained keypoint-based evaluation metric. Specifically, CapGeo-Bench encompasses three
geometric classes and is divided into four difficulty levels according to the complexity of
geometric primitives. Captions are scored at a fine-grained level by extracting keypoints along three dimensions-
geometric elements, spatial relationships, and numerical attributes. CapGeo-Bench provides a reliable instrument and guideline for evaluating and advancing MLLMs'
geometric perception capabilities. Our code and data are publicly available at: https://github.com/YuYingLi0/CapGeo.
Read the full abstract ▾
All authors Yuying Li, Siyi Qian, Hao Liang, Leqi Zheng, Ruichuan An, Linzhuang Sun, Jiajun Zhang, Wentao Zhang
Computation and Language
ShiZhi: A Chinese Lightweight Large Language Model for Court View Generation
Zhitian Hou and Kun Zeng
Criminal Court View Generation (CVG) is a fundamental task in legal artificial intelligence, aiming to automatically generate the "Court View" section of a legal case document. Generating court views is challenging due to the diversity and complexity of case facts, and directly generating from raw facts may limit performance. In this paper, we present ShiZhi, the first large
language model (LLM) specifically designed for court view generation. We construct a Chinese Court View Generation
dataset, CCVG, of more than 110K cases, each containing fact descriptions paired with corresponding court views. Based on this
dataset, ShiZhi achieving 70.00 ROUGE-1 and 67.85 BLEU-1 on court view generation, as well as 86.48% accuracy with 92.75% macro F1 on charge prediction. Experimental results demonstrate that even a small LLM can generate reasonable and legally coherent court views when trained on high-quality domain-specific data. Our model and
dataset are available at \href{https://github.com/ZhitianHou/ShiZhi}{https://github.com/ZhitianHou/ShiZhi}.
Read the full abstract ▾
Computation and Language
MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
Jiapeng Wang, Changxin Tian, Kunlong Chen et al. (+5 more)
Reliable evaluation is fundamental to the progress of Large
Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscures true learning dynamics. In this work, we systematically diagnose this instability, attributing it to two distinct sources: Parameter Instability from training stochasticity and Evaluation Instability from noisy measurement protocols. To counteract both sources of noise, we introduce MaP, a dual-pronged framework that synergistically integrates checkpoint \underline{M}erging \underline{a}nd the \underline{P}ass@k metric. Checkpoint merging smooths the parameter space by averaging recent model weights, while Pass@k provides a robust, low-variance statistical estimate of model capability. Extensive experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent model rankings. Ultimately, MaP provides a more reliable and faithful lens for observing LLM training dynamics, laying a crucial empirical foundation for LLM research.
Read the full abstract ▾
All authors Jiapeng Wang, Changxin Tian, Kunlong Chen, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, Jun Zhou
Computation and Language
One Sentence, Two Embeddings: Contrastive Learning of Explicit and Implicit Semantic Representations
Kohei Oda, Po-Min Chuang, Kiyoaki Shirai et al. (+1 more)
Sentence
embedding methods have made remarkable progress, yet they still struggle to capture the implicit semantics within sentences. This can be attributed to the inherent limitations of conventional sentence
embedding methods that assign only a single vector per sentence. To overcome this limitation, we propose DualCSE, a sentence
embedding method that assigns two
embeddings to each sentence: one representing the explicit semantics and the other representing the implicit semantics. These
embeddings coexist in the shared space, enabling the selection of the desired semantics for specific purposes such as information
retrieval and text classification. Experimental results demonstrate that DualCSE can effectively encode both explicit and implicit meanings and improve the performance of the downstream task.
Read the full abstract ▾
All authors Kohei Oda, Po-Min Chuang, Kiyoaki Shirai, Natthawut Kertkeidkachorn
Computation and Language
CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts
Jiuheng Lin, Cong Jiang, Zirui Wu et al. (+2 more)
Training expert LLMs in domains with scarce data is difficult, often relying on multiple-choice questions (MCQs). However, standard outcome-based reinforcement learning (RL) on MCQs is risky. While it may improve accuracy, we observe it often degrades
reasoning quality such as logical consistency. Existing solutions to supervise
reasoning, such as large-scale Process Reward Models (PRMs), are prohibitively expensive. To address this, we propose CLARity, a cost-effective RL framework that enhances
reasoning quality using only a small, general-purpose LLM. CLARity integrates a consistency-aware reward mechanism with a 2-stage refine-then-monitor training pipeline to enhance
reasoning consistency, and a dynamic data reformulation strategy to to better exploit limited data. Experiments demonstrate that CLARity improves response consistency by 16.5% and accuracy by 7.5% over baselines. Human evaluations further confirm holistic improvements in coherence and professionalism. Thus, CLARity offers a generalizable solution that enables smaller models to effectively guide expert models by
reasoning consistency. Our code is open sourced at: https://github.com/Infinite-set/CLARity
Read the full abstract ▾
All authors Jiuheng Lin, Cong Jiang, Zirui Wu, Jiarui Sun, Yansong Feng