Computation and Language · Cross-listed from cs.AI
Modeling Hierarchical Thinking in Large Reasoning Models
G M Shahariar, Erfan Shayegani, Ali Nazari et al. (+1 more)
Large
Reasoning Models (LRMs) solve complex tasks by generating long Chain-of-Thought (CoT) sequences; however, the emergent dynamics governing
reasoning trajectories are not well understood and can lead to inconsistencies and
reasoning pathologies. In this work, we propose to approximate LRM's emerging hierarchical
reasoning dynamics as a trajectory within a Finite State Machine (FSM) transitioning among six abstract cognitive states. We demonstrate that these states and transitions can be captured in the latent state of the model. We believe that this representation can have different applications in the interpretability and optimization of LRM models. For example, by analyzing the
topology of these transitions, we identify statistical shifts in
reasoning strategies that help identify effective
reasoning chains from those that fail. To illustrate these potential advantages, we propose Q-Value guided steering, a training-free inference-time control method that treats
reasoning as a planning problem. We estimate the long-horizon utility of state transitions and apply sparse, orthogonal activation steering at sentence boundaries to align the CoT generation with optimal
reasoning policies. Experiments across four
benchmarks (AIME25, MATH-500, GSM8k, and GPQA Diamond) using three state-of-the-art open
reasoning models demonstrate that Q-Value steering policy achieves significant performance gains with "
surgical" efficiency, often requiring 25 times fewer interventions than greedy and weighted baselines, which suggests that
reasoning can be effectively controlled by guiding high-level cognitive dynamics rather than micro-managing
token generation. Code is available at: https://github.com/shahariar-shibli/CoT-FSM.
Read the full abstract ▾
All authors G M Shahariar, Erfan Shayegani, Ali Nazari, Nael Abu-Ghazaleh
Computation and Language
Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection
Federica Gamba, Aman Sinha, Timothee Mickus et al. (+12 more)
We introduce the CAP (Confabulations from ACL Publications)
dataset, a
multilingual resource for studying
hallucinations in large
language models (LLMs) within scientific text generation. CAP focuses on the scientific domain, where
hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the presence of specialized terminology, statistical
reasoning, and context-dependent interpretations further exacerbates these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource
languages (English, French, Hindi, Italian, and Spanish) and four low-resource
languages (Bengali, Gujarati, Malayalam, and Telugu). The
dataset comprises 900 curated scientific questions and over 7000 LLM-generated answers from 16 publicly available models, provided as question-answer pairs along with
token sequences and corresponding logits. Each instance is annotated with a binary label indicating the presence of a scientific
hallucination, denoted as a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is publicly released to facilitate advanced research on
hallucination detection,
multilingual evaluation of LLMs, and the development of more reliable scientific NLP systems.
Read the full abstract ▾
All authors Federica Gamba, Aman Sinha, Timothee Mickus, Raul Vazquez, Patanjali Bhamidipati, Claudio Savelli, Ahana Chattopadhyay, Laura A. Zanella, Yash Kankanampati, Binesh Arakkal Remesh, Aryan Ashok Chandramania, Rohit Agarwal, Chuyuan Li, Ioana Buhnila, Radhika Mamidi
Computation and Language · Cross-listed from cs.LG
Label Smoothing Improves Gradient Ascent in LLM Unlearning
Zirui Pang, Hao Zheng, Zhijie Deng et al. (+3 more)
LLM unlearning has emerged as a promising approach, aiming to enable models to forget hazardous/undesired knowledge at low cost while preserving as much model utility as possible. Among existing techniques, the most straightforward method is performing Gradient Ascent (GA) w.r.t. the forget data, thereby forcing the model to unlearn the forget
dataset. However, GA suffers from severe instability, as it drives updates in a divergent direction, often resulting in drastically degraded model utility. To address this issue, we propose Smoothed Gradient Ascent (SGA). SGA combines the forget data with multiple constructed normal data through a tunable smoothing rate. Intuitively, this extends GA from learning solely on the forget data to jointly learning across both forget and normal data, enabling more stable unlearning while better preserving model utility. Theoretically, we provide the theoretical guidance on the selection of the optimal smoothing rate. Empirically, we evaluate SGA on three
benchmarks: TOFU, Harry Potter, and MUSE-NEWS. Experimental results demonstrate that SGA consistently outperforms the original Gradient Ascent (GA) method across all metrics and achieves top-2 performance among all baseline methods on several key metrics.
Read the full abstract ▾
All authors Zirui Pang, Hao Zheng, Zhijie Deng, Ling Li, Zixin Zhong, Jiaheng Wei
Computation and Language
VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
Yupeng Xie, Zhiyang Zhang, Yifan Wu et al. (+8 more)
Visualization, a domain-specific yet widely used form of imagery, is an effective way to turn complex
datasets into intuitive insights, and its value depends on whether data are faithfully represented, clearly communicated, and aesthetically designed. However, evaluating visualization quality is challenging: unlike natural images, it requires simultaneous judgment across data encoding accuracy, information expressiveness, and visual aesthetics. Although
multimodal large
language models (MLLMs) have shown promising performance in aesthetic assessment of natural images, no systematic
benchmark exists for measuring their capabilities in evaluating visualizations. To address this, we propose VisJudge-Bench, the first comprehensive
benchmark for evaluating MLLMs' performance in assessing visualization aesthetics and quality. It contains 3,090 expert-annotated samples from real-world scenarios, covering single visualizations, multiple visualizations, and dashboards across 32 chart types. Systematic testing on this
benchmark reveals that even the most advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human experts in judgment, with a Mean Absolute Error (MAE) of 0.553 and a correlation with human ratings of only 0.428. To address this issue, we propose VisJudge, a model specifically designed for visualization aesthetics and quality assessment. Experimental results demonstrate that VisJudge significantly narrows the gap with human judgment, reducing the MAE to 0.421 (a 23.9% reduction) and increasing the consistency with human experts to 0.687 (a 60.5% improvement) compared to GPT-5. The
benchmark is available at https://github.com/HKUSTDial/VisJudgeBench.
Read the full abstract ▾
All authors Yupeng Xie, Zhiyang Zhang, Yifan Wu, Sirong Lu, Jiayi Zhang, Zhaoyang Yu, Jinlin Wang, Sirui Hong, Bang Liu, Chenglin Wu, Yuyu Luo
Computation and Language · Cross-listed from cs.AI
Reasoning Models Reason Well, Until They Don't
Revanth Rameshkumar, Jimson Huang, Yunxin Sun et al. (+2 more)
Large
language models (LLMs) have shown significant progress in
reasoning tasks. However, recent studies show that
transformers and LLMs fail catastrophically once
reasoning problems exceed modest complexity. We revisit these findings through the lens of large
reasoning models (LRMs) – LLMs fine-tuned with incentives for step-by-step argumentation and self-verification. LRM performance on graph and
reasoning benchmarks such as NLGraph seem extraordinary, with some even claiming they are capable of generalized
reasoning and innovation in
reasoning-intensive fields such as mathematics, physics, medicine, and law. However, by more carefully scaling the complexity of
reasoning problems, we show existing
benchmarks actually have limited complexity. We develop a new
dataset, the Deep
Reasoning Dataset (DeepRD), along with a generative process for producing unlimited examples of scalable complexity. We use this
dataset to evaluate model performance on graph connectivity and natural
language proof planning. We find that the performance of LRMs drop abruptly at sufficient complexity and do not generalize. We also relate our LRM results to the distributions of the complexities of large, real-world knowledge graphs, interaction graphs, and proof
datasets. We find the majority of real-world examples fall inside the LRMs' success regime, yet the long tails expose substantial failure potential. Our analysis highlights the near-term utility of LRMs while underscoring the need for new methods that generalize beyond the complexity of examples in the training distribution.
Read the full abstract ▾
All authors Revanth Rameshkumar, Jimson Huang, Yunxin Sun, Fei Xia, Abulhair Saparov
Computation and Language
GigaEmbeddings: Efficient Russian Language Embedding Model
Egor Kolodin, Daria Khomich, Nikita Savushkin et al. (+2 more)
We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text
embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian
language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across
retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional
attention for contextual modeling, latent
attention pooling for robust sequence aggregation, and strategic
pruning of 25% of
transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB
benchmark spanning 23
multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.
Read the full abstract ▾
All authors Egor Kolodin, Daria Khomich, Nikita Savushkin, Anastasia Ianina, Fyodor Minkin
Computation and Language · Cross-listed from cs.LG
Jiazheng Li, Andreas Damianou, J Rosser et al. (+2 more)
Chain-of-thought (CoT) traces promise transparency for
reasoning language models, but prior work shows they are not always faithful reflections of internal computation. This raises challenges for oversight: practitioners may misinterpret decorative
reasoning as genuine. We introduce Concept Walk, a general framework for tracing how a model's internal stance evolves with respect to a concept direction during
reasoning. Unlike surface text, Concept Walk operates in activation space, projecting each
reasoning step onto the concept direction learned from contrastive data. This allows us to observe whether
reasoning traces shape outcomes or are discarded. As a case study, we apply Concept Walk to the domain of Safety using Qwen 3-4B. We find that in 'easy' cases, perturbed CoTs are quickly ignored, indicating decorative
reasoning, whereas in 'hard' cases, perturbations induce sustained shifts in internal activations, consistent with faithful
reasoning. The contribution is methodological: Concept Walk provides a lens to re-examine faithfulness through concept-specific internal dynamics, helping identify when
reasoning traces can be trusted and when they risk misleading practitioners.
Read the full abstract ▾
All authors Jiazheng Li, Andreas Damianou, J Rosser, José Luis Redondo García, Konstantina Palla
Computation and Language
Irony Detection in Urdu Text: A Comparative Study Using Machine Learning Models and Large Language Models
Fiaz Ahmad, Nisar Hussain, Amna Qasim et al. (+3 more)
Ironic identification is a challenging task in Natural
Language Processing, particularly when dealing with
languages that differ in syntax and cultural context. In this work, we aim to detect irony in Urdu by translating an English Ironic Corpus into the Urdu
language. We evaluate ten state-of-the-art machine learning
algorithms using GloVe and Word2Vec
embeddings, and compare their performance with classical methods. Additionally, we fine-tune advanced
transformer-based models, including BERT, RoBERTa, LLaMA 2 (7B), LLaMA 3 (8B), and Mistral, to assess the effectiveness of large-scale models in irony detection. Among machine learning models, Gradient Boosting achieved the best performance with an F1-score of 89.18%. Among
transformer-based models, LLaMA 3 (8B) achieved the highest performance with an F1-score of 94.61%. These results demonstrate that combining transliteration techniques with modern NLP models enables robust irony detection in Urdu, a historically low-resource
language.
Read the full abstract ▾
All authors Fiaz Ahmad, Nisar Hussain, Amna Qasim, Momina Hafeez, Muhammad Usman Grigori Sidorov, Alexander Gelbukh
Computation and Language
FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented Generation
Mohammad Aghajani Asl, Majid Asgari-Bidhendi and Behrooz Minaei-Bidgoli
While
Retrieval-Augmented Generation (RAG) mitigates
hallucination and knowledge staleness in Large
Language Models (LLMs), existing frameworks often falter on complex, multi-hop queries that require synthesizing information from disparate sources. Current advanced RAG methods, employing iterative or adaptive strategies, lack a robust mechanism to systematically identify and fill evidence gaps, often propagating noise or failing to gather a comprehensive context. We introduce FAIR-RAG, a novel agentic framework that transforms the standard RAG pipeline into a dynamic, evidence-driven
reasoning process. At its core is an Iterative Refinement Cycle governed by a module we term Structured Evidence Assessment (SEA). The SEA acts as an analytical gating mechanism: it deconstructs the initial query into a checklist of required findings and audits the aggregated evidence to identify confirmed facts and, critically, explicit informational gaps. These gaps provide a precise signal to an Adaptive Query Refinement
agent, which generates new, targeted sub-queries to retrieve missing information. This cycle repeats until the evidence is verified as sufficient, ensuring a comprehensive context for a final, strictly faithful generation. We conducted experiments on challenging multi-hop QA
benchmarks, including HotpotQA, 2WikiMultiHopQA, and MusiQue. In a unified experimental setup, FAIR-RAG significantly outperforms strong baselines. On HotpotQA, it achieves an F1-score of 0.453 – an absolute improvement of 8.3 points over the strongest iterative baseline – establishing a new state-of-the-art for this class of methods on these
benchmarks. Our work demonstrates that a structured, evidence-driven refinement process with explicit gap analysis is crucial for unlocking reliable and accurate
reasoning in advanced RAG systems for complex, knowledge-intensive tasks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry
Changti Wu, Shijie Lian, Zihao Liu et al. (+3 more)
Solid geometry problem solving demands spatial mathematical
reasoning that integrates spatial intelligence and symbolic
reasoning. However, most existing
multimodal mathematical
reasoning benchmarks focus primarily on 2D plane geometry, rely on static
datasets prone to data contamination and memorization, and evaluate models solely by final answers, overlooking the
reasoning process. To address these limitations, we introduce DynaSolidGeo, the first dynamic
benchmark for evaluating genuine spatial
reasoning in
Vision-
Language Models (VLMs). Constructed through a semi-automatic annotation pipeline, DynaSolidGeo contains 503 expert-curated seed questions that can, in principle, dynamically generate an unbounded number of diverse
multimodal text-visual instances. Beyond answer accuracy, we incorporate process evaluation based on expert-annotated
reasoning chains to measure logical validity and causal coherence. Experiments across representative open-source and closed-source VLMs reveal large performance gaps, severe degradation in dynamic settings, and poor performance on tasks requiring high-level spatial intelligence, such as mental rotation and visualization. The code and
dataset are available at \href{https://zgca-ai4edu.github.io/DynaSolidGeo/}{DynaSolidGeo}.
Read the full abstract ▾
All authors Changti Wu, Shijie Lian, Zihao Liu, Lei Zhang, Laurence Tianruo Yang, Kai Chen
Computation and Language
Multilingual Target-Stance Extraction
Ethan Mines and Bonnie Dorr
Social media enables data-driven analysis of public opinion on contested issues. Target-Stance Extraction (TSE) is the task of identifying the target discussed in a document and the document's stance towards that target. Many works classify stance towards a given target in a
multilingual setting, but all prior work in TSE is English-only. This work introduces the first
multilingual TSE
benchmark, spanning Catalan, Estonian, French, Italian, Mandarin, and Spanish corpora. It manages to extend the original TSE pipeline to a
multilingual setting without requiring separate models for each
language. Our model pipeline achieves a modest F1 score of 12.78, underscoring the increased difficulty of the
multilingual task relative to English-only setups and highlighting target prediction as the primary bottleneck. We are also the first to demonstrate the sensitivity of TSE's F1 score to different target verbalizations. Together these serve as a much-needed baseline for resources,
algorithms, and evaluation criteria in
multilingual TSE.
Read the full abstract ▾
Computation and Language
Memory-based Language Models: An Efficient, Explainable, and Eco-friendly Approach to Large Language Modeling
Antal van den Bosch, Ainhoa Risco Patón, Teun Buijse et al. (+2 more)
We present
memory-based
language modeling as an efficient, eco-friendly alternative to deep neural network-based
language modeling. It offers log-linearly scalable next-
token prediction performance and strong memorization capabilities. Implementing fast approximations of k-nearest neighbor classification,
memory-based
language modeling leaves a relatively small ecological footprint both in training and in inference mode, as it relies fully on CPUs and attains low
token latencies. Its internal workings are simple and fully transparent. We compare our implementation of
memory-based
language modeling, OLIFANT, with GPT-2 and GPT-Neo on next-
token prediction accuracy, estimated emissions and speeds, and offer some deeper analyses of the model.
Read the full abstract ▾
All authors Antal van den Bosch, Ainhoa Risco Patón, Teun Buijse, Peter Berck, Maarten van Gompel
Computation and Language · Cross-listed from cs.AI
VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription
Quoc Anh Nguyen, Bernard Cheng and Kelvin Soh
Automatic Lyrics Transcription (ALT) for Vietnamese music presents unique challenges due to its tonal complexity and dialectal variations, but remains largely unexplored due to the lack of a dedicated
dataset. Therefore, we curated the first large-scale Vietnamese ALT
dataset (VietLyrics), comprising 647 hours of songs with line-level aligned lyrics and metadata to address these issues. Our evaluation of current ASRbased approaches reveal significant limitations, including frequent transcription errors and
hallucinations in non-vocal segments. To improve performance, we fine-tuned Whisper models on the VietLyrics
dataset, achieving superior results compared to existing
multilingual ALT systems, including LyricWhiz. We publicly release VietLyrics and our models, aiming to advance Vietnamese music computing research while demonstrating the potential of this approach for ALT in low-resource
language and music.
Read the full abstract ▾
Computation and Language
Supervised Fine-Tuning or In-Context Learning? Evaluating LLMs for Clinical NER
Andrei Baroian
We study clinical Named Entity Recognition (NER) on the CADEC corpus and compare three families of approaches: (i) BERT-style encoders (BERT Base, BioClinicalBERT, RoBERTa-large), (ii) GPT-4o used with few-shot in-context learning (ICL) under simple vs.\ complex
prompts, and (iii) GPT-4o with supervised fine-tuning (SFT). All models are evaluated on standard NER metrics over CADEC's five entity types (ADR, Drug, Disease, Symptom, Finding). RoBERTa-large and BioClinicalBERT offer limited improvements over BERT Base, showing the limit of these family of models. Among LLM settings, simple ICL outperforms a longer, instruction-heavy
prompt, and SFT achieves the strongest overall performance (F1 $\approx$ 87.1%), albeit with higher cost. We find that the LLM achieve higher accuracy on simplified tasks, restricting classification to two labels.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
CityRiSE: Reasoning Urban Socio-Economic Status in Large Vision-Language Models via Reinforcement Learning
Tianhui Liu, Hetian Pang, Xin Zhang et al. (+3 more)
Urban socio-economic sensing plays a vital role in advancing global sustainable development goals. With the advent of Large
Vision-
Language Models (LVLMs), new opportunities have emerged to address this challenge by framing it as a multi-modal perception and
reasoning task. However, recent studies show that LVLMs still struggle to make accurate and interpretable socio-economic predictions from visual data. To overcome these limitations and fully exploit the potential of LVLMs, we propose CityRiSE, a novel framework for
Reasoning urban Socio-Economic status in LVLMs via reinforcement learning (RL). With carefully curated multi-modal
dataset and verifiable reward design, our approach guides the LVLM to focus on semantically meaningful visual cues, enabling structured and goal-oriented
reasoning for generalist socio-economic status prediction. Experiments demonstrate that CityRiSE, equipped with emergent
reasoning, significantly outperforms existing baselines, improving both prediction accuracy and generalization across diverse urban contexts, especially on unseen cities and unseen indicators. This work highlights the promise of combining RL and LVLMs for interpretable and generalist urban socio-economic sensing.
Read the full abstract ▾
All authors Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Pan Hui, Yong Li
Computation and Language · Cross-listed from cs.CV
WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models
Issa Sugiura, Shuhei Kurita, Yusuke Oda et al. (+3 more)
Contrastive
vision-
language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing English-only caption filters and pretraining on global data is effective for improving multicultural performance. We study whether such global pretraining is sufficient for culture-specific understanding, or whether further adaptation with natively sourced data can boost performance beyond what global pretraining alone achieves. To enable this investigation, we present WAON, the largest publicly available native Japanese image-text
dataset constructed from native Japanese web content in Common Crawl, containing approximately 155 million examples. We also introduce WAON-Bench, a manually curated Japanese cultural
benchmark spanning 374 classes. Through comparative fine-tuning experiments on multiple Japanese image-text
datasets, we observe that models fine-tuned on WAON consistently achieve stronger performance on Japanese cultural
benchmarks than those fine-tuned on English-to-Japanese translated data. Controlled experiments at matched scale, filtering, and training budget across two model families further indicate that native web origin is the primary driver of this gain. We release our
dataset,
benchmark, model, and code.
Read the full abstract ▾
All authors Issa Sugiura, Shuhei Kurita, Yusuke Oda, Daisuke Kawahara, Yasuo Okabe, Naoaki Okazaki
Computation and Language
From Slides to Chatbots: Enhancing Large Language Models with University Course Materials
Tu Anh Dinh, Philipp Nicolas Schumacher and Jan Niehues
Large
Language Models (LLMs) have advanced rapidly in recent years. One application of LLMs is to support student learning in educational settings. However, prior work has shown that LLMs still struggle to answer questions accurately within university-level computer science courses. In this work, we investigate how incorporating university course materials can enhance LLM performance in this setting. A key challenge lies in leveraging diverse course materials such as lecture slides and transcripts, which differ substantially from typical textual corpora: slides also contain visual elements like images and formulas, while transcripts contain spoken, less structured
language. We compare two strategies,
Retrieval-Augmented Generation (RAG) and Continual Pre-Training (CPT), to extend LLMs with course-specific knowledge. For lecture slides, we further explore a multi-modal RAG approach, where we present the retrieved content to the generator in image form. Our experiments reveal that, given the relatively small size of university course materials, RAG is more effective and efficient than CPT. Moreover, incorporating slides as images in the multi-modal setting significantly improves performance over text-only
retrieval. These findings highlight practical strategies for developing AI assistants that better support learning and teaching, and we hope they inspire similar efforts in other educational contexts.
Read the full abstract ▾
Computation and Language
PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding
Iliass Ayaou and Denis Cavallucci
Patent text
embeddings enable prior art search, technology landscaping, and patent analysis, yet existing
benchmarks inadequately capture patent-specific challenges. We introduce PatenTEB, a comprehensive
benchmark comprising 15 tasks across
retrieval, classification, paraphrase, and clustering, with 2.06 million examples. PatenTEB employs domain-stratified splits, domain specific hard negative mining, and systematic coverage of asymmetric fragment-to-document matching scenarios absent from general
embedding benchmarks. We develop the patembed model family through multi-task training, spanning 67M to 344M parameters with context lengths up to 4096
tokens. External validation shows strong generalization: patembed-base achieves state-of-the-art on MTEB BigPatentClustering.v2 (0.494 V-measure vs. 0.445 previous best), while patembed-large achieves 0.377 NDCG@100 on DAPFAM. Systematic ablations reveal that multi-task training improves external generalization despite minor
benchmark costs, and that domain-pretrained initialization provides consistent advantages across task families. All resources will be made available at https://github.com/iliass-y/patenteb. Keywords: patent
retrieval, sentence
embeddings, multi-task learning, asymmetric
retrieval,
benchmark evaluation, contrastive learning.
Read the full abstract ▾
Computation and Language
SteerX: Disentangled Steering for LLM Personalization
Xiaoyan Zhao, Ming Yan, Yilun Qiu et al. (+5 more)
Large
language models (LLMs) have shown remarkable success in recent years, enabling a wide range of applications, including intelligent assistants that support users' daily life and work. A critical factor in building such assistants is personalizing LLMs, as user preferences and needs vary widely. Activation steering, which directly leverages directions representing user preference in the LLM activation space to adjust its behavior, offers a cost-effective way to align the model's outputs with individual users. However, existing methods rely on all historical data to compute the steering vector, ignoring that not all content reflects true user preferences, which undermines the personalization signal. To address this, we propose SteerX, a disentangled steering method that isolates preference-driven components from preference-agnostic components. Grounded in causal inference theory, SteerX estimates
token-level causal effects to identify preference-driven
tokens, transforms these discrete signals into a coherent description, and then leverages them to steer personalized LLM generation. By focusing on the truly preference-driven information, SteerX produces more accurate activation steering vectors and enhances personalization. Experiments on two representative steering backbone methods across real-world
datasets demonstrate that SteerX consistently enhances steering vector quality, offering a practical solution for more effective LLM personalization.
Read the full abstract ▾
All authors Xiaoyan Zhao, Ming Yan, Yilun Qiu, Haoting Ni, Yang Zhang, Fuli Feng, Hong Cheng, Tat-Seng Chua
Computation and Language · Cross-listed from cs.AI
PACR: Progressively Ascending Confidence Reward for LLM Reasoning
Eunseop Yoon, Hee Suk Yoon, Jaehyun Jang et al. (+5 more)
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM
reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed
reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound
reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple
benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.
Read the full abstract ▾
All authors Eunseop Yoon, Hee Suk Yoon, Jaehyun Jang, SooHwan Eom, Qi Dai, Chong Luo, Mark A. Hasegawa-Johnson, Chang D. Yoo
Computation and Language
You Don't Need Prompt Engineering Anymore: The Prompting Inversion
Imran Khan
Prompt engineering, particularly Chain-of-Thought (CoT) prompting, significantly enhances LLM
reasoning capabilities. We introduce "Sculpting," a constrained, rule-based prompting method designed to improve upon standard CoT by reducing errors from semantic ambiguity and flawed common sense. We evaluate three prompting strategies (Zero Shot, standard CoT, and Sculpting) across three OpenAI model generations (gpt-4o-mini, gpt-4o, gpt-5) using the GSM8K mathematical
reasoning benchmark (1,317 problems). Our findings reveal a "Prompting Inversion": Sculpting provides advantages on gpt-4o (97% vs. 93% for standard CoT), but becomes detrimental on gpt-5 (94.00% vs. 96.36% for CoT on full
benchmark). We trace this to a "Guardrail-to-Handcuff" transition where constraints preventing common-sense errors in mid-tier models induce hyper-literalism in advanced models. Our detailed error analysis demonstrates that optimal prompting strategies must co-evolve with model capabilities, suggesting simpler
prompts for more capable models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading
Yutao Wu, Xiao Liu, Yunhao Feng et al. (+2 more)
Large
Language Models (LLMs) increasingly serve as research assistants, yet their reliability in scholarly tasks remains under-evaluated. In this work, we introduce PaperAsk, a
benchmark that systematically evaluates LLMs across four key research tasks: citation
retrieval, content extraction, paper discovery, and claim verification. We evaluate GPT-4o, GPT-5, and Gemini-2.5-Flash under realistic usage conditions-via web interfaces where search operations are opaque to the user. Through controlled experiments, we find consistent reliability failures: citation
retrieval fails in 48-98% of multi-reference queries, section-specific content extraction fails in 72-91% of cases, and topical paper discovery yields F1 scores below 0.32, missing over 60% of relevant literature. Further human analysis attributes these failures to the uncontrolled expansion of retrieved context and the tendency of LLMs to prioritize semantically relevant text over task instructions. Across basic tasks, the LLMs display distinct failure behaviors: ChatGPT often withholds responses rather than risk errors, whereas Gemini produces fluent but fabricated answers. To address these issues, we develop lightweight reliability classifiers trained on PaperAsk data to identify unreliable outputs. PaperAsk provides a reproducible and diagnostic framework for advancing the reliability evaluation of LLM-based scholarly assistance systems.
Read the full abstract ▾
All authors Yutao Wu, Xiao Liu, Yunhao Feng, Jiale Ding, Xingjun Ma
Computation and Language
Evolution of the lexicon: a probabilistic point of view
Maurizio Serva
The Swadesh approach for determining the temporal separation between two
languages relies on the
stochastic process of words replacement (when a complete new word emerges to represent a given concept). It is well known that the basic assumptions of the Swadesh approach are often unrealistic due to various contamination phenomena and misjudgments (horizontal transfers, variations over time and space of the replacement rate, incorrect assessments of cognacy relationships, presence of synonyms, and so on). All of this means that the results cannot be completely correct. More importantly, even in the unrealistic case that all basic assumptions are satisfied, simple mathematics places limits on the accuracy of estimating the temporal separation between two
languages. These limits, which are purely probabilistic in nature and which are often neglected in lexicostatistical studies, are analyzed in detail in this article. Furthermore, in this work we highlight that the
evolution of a
language's lexicon is also driven by another
stochastic process: gradual lexical modification of words. We show that this process equally also represents a major contribution to the reshaping of the vocabulary of
languages over the centuries and we also show, from a purely probabilistic perspective, that taking into account this second random process significantly increases the precision in determining the temporal separation between two
languages.
Read the full abstract ▾
Computation and Language
Estimating the Error of Large Language Models at Pairwise Text Comparison
Tianyi Li
We measure LLMs' output error at pairwise text comparison, noting the probability of error in their preferences. Our method does not rely on the ground truth and supports two scenarios: (i) uniform error rate regardless of the order of comparison, estimated with two comparisons for each text pair with either text placed first; (ii) binary positional bias assuming distinct error rates for the two orders of comparison, estimated with repeated comparisons between the texts. The Copeland counting constructs a ranking over the compared texts from pairwise preferences; the ranking reveals the poor scalability of LLM-based pairwise comparison and helps yield the estimates for LLMs' error rates. We apply the method to six LLMs (ChatGPT, Claude, DeepSeek, Gemini, Grok, Qwen) with five types of text input and obtain consistent estimates of LLMs' error. In general, the measured two positional bias terms are similar, close to the uniform error. Considering both the error rates and the robustness to the variation of
prompts, Claude obtained the most desirable performance in this experiment. Our model outperforms the biased Bradley-Terry model and the commutativity score in indicating LLMs' error at this task.
Read the full abstract ▾
Computation and Language
DETECT: Determining Ease and Textual Clarity of German Text Simplifications
Maria Korobeynikova, Alessia Battisti, Lukas Fischer et al. (+1 more)
Current evaluation of German automatic text simplification (ATS) relies on general-purpose metrics such as SARI, BLEU, and BERTScore, which insufficiently capture simplification quality in terms of simplicity, meaning preservation, and fluency. While specialized metrics like LENS have been developed for English, corresponding efforts for German have lagged behind due to the absence of human-annotated corpora. To close this gap, we introduce DETECT, the first German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency, and is trained entirely on synthetic large
language model (LLM) responses. Our approach adapts the LENS framework to German and extends it with (i) a pipeline for generating synthetic quality scores via LLMs, enabling
dataset creation without human annotation, and (ii) an LLM-based refinement step for aligning grading criteria with simplification requirements. To the best of our knowledge, we also construct the largest German human evaluation
dataset for text simplification to validate our metric directly. Experimental results show that DETECT achieves substantially higher correlations with human judgments than widely used ATS metrics, with particularly strong gains in meaning preservation and fluency. Beyond ATS, our findings highlight both the potential and the limitations of LLMs for automatic evaluation and provide transferable guidelines for general
language accessibility tasks.
Read the full abstract ▾
All authors Maria Korobeynikova, Alessia Battisti, Lukas Fischer, Yingqiang Gao
Computation and Language · Cross-listed from cs.LG
The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)
Nnamdi Aghanya, Jun Li and Kewei Wang
Large
Language Models (LLMs) can achieve near-optimal lossless compression by acting as powerful probability models. We investigate their use in the lossy domain, where reconstruction fidelity is traded for higher compression ratios. This paper introduces Error-Bounded Predictive Coding (EPC), a lossy text codec that leverages a Masked
Language Model (MLM) as a decompressor. Instead of storing a subset of original
tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect. This creates a residual channel that offers continuous rate-distortion control. We compare EPC to a simpler Predictive Masking (PM) baseline and a transform-based Vector Quantisation with a Residual Patch (VQ+RE) approach. Through an evaluation that includes precise bit accounting and rate-distortion analysis, we demonstrate that EPC consistently dominates PM, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
M-CIF: Multi-Scale Alignment For CIF-Based Non-Autoregressive ASR
Ruixiang Mao, Xiangnan Ma, Qing Yang et al. (+7 more)
The Continuous Integrate-and-Fire (CIF) mechanism provides effective
alignment for non-autoregressive (NAR)
speech recognition. This mechanism creates a smooth and monotonic mapping from acoustic features to target
tokens, achieving performance on Mandarin competitive with other NAR approaches. However, without finer-grained guidance, its stability degrades in some
languages such as English and French. In this paper, we propose Multi-scale CIF (M-CIF), which performs multi-level
alignment by integrating character and phoneme level supervision progressively distilled into subword representations, thereby enhancing robust acoustic-text
alignment. Experiments show that M-CIF reduces WER compared to the Paraformer baseline, especially on CommonVoice by 4.21% in German and 3.05% in French. To further investigate these gains, we define phonetic confusion errors (PE) and space-related
segmentation errors (SE) as evaluation metrics. Analysis of these metrics across different M-CIF settings reveals that the phoneme and character layers are essential for enhancing progressive CIF
alignment.
Read the full abstract ▾
All authors Ruixiang Mao, Xiangnan Ma, Qing Yang, Ziming Zhu, Yucheng Qiao, Yuan Ge, Tong Xiao, Shengxiang Gao, Zhengtao Yu, Jingbo Zhu
Computation and Language · Cross-listed from cs.CY
Surface Reading LLMs: Synthetic Text and its Styles
Hannes Bajohr
Despite a potential plateau in ML advancement, the societal impact of large
language models lies not in approaching superintelligence but in generating text surfaces indistinguishable from human writing. While Critical AI Studies provides essential material and socio-technical critique, it risks overlooking how LLMs phenomenologically reshape meaning-making. This paper proposes a semiotics of "surface integrity" as attending to the immediate plane where LLMs inscribe themselves into human communication. I distinguish three knowledge interests in ML research (epistemology, epistēmē, and epistemics) and argue for integrating surface-level stylistic analysis alongside depth-oriented critique. Through two case studies examining stylistic markers of synthetic text, I argue how attending to style as a semiotic phenomenon reveals LLMs as cultural machines that transform the conditions of meaning emergence and circulation in contemporary discourse, independent of questions about machine consciousness.
Read the full abstract ▾
Computation and Language
SentiMaithili: A Benchmark Dataset for Sentiment and Reason Generation for the Low-Resource Maithili Language
Rahul Ranjan, Mahendra Kumar Gurve, Anuj et al. (+2 more)
Developing
benchmark datasets for low-resource
languages poses significant challenges, primarily due to the limited availability of native linguistic experts and the substantial time and cost involved in annotation. Given these challenges, Maithili is still underrepresented in natural
language processing research. It is an Indo-Aryan
language spoken by more than 13 million people in the Purvanchal region of India, valued for its rich linguistic structure and cultural significance. While sentiment analysis has achieved remarkable progress in high-resource
languages, resources for low-resource
languages, such as Maithili, remain scarce, often restricted to coarse-grained annotations and lacking interpretability mechanisms. To address this limitation, we introduce a novel
dataset comprising 3,221 Maithili sentences annotated for sentiment polarity and accompanied by natural
language justifications. Moreover, the
dataset is carefully curated and validated by linguistic experts to ensure both label reliability and contextual fidelity. Notably, the justifications are written in Maithili, thereby promoting culturally grounded interpretation and enhancing the explainability of sentiment models. Furthermore, extensive experiments using both classical machine learning and state-of-the-art
transformer architectures demonstrate the
dataset's effectiveness for interpretable sentiment analysis. Ultimately, this work establishes the first
benchmark for explainable affective computing in Maithili, thus contributing a valuable resource to the broader advancement of
multilingual NLP and explainable AI.
Read the full abstract ▾
All authors Rahul Ranjan, Mahendra Kumar Gurve, Anuj, Nitin, Yamuna Prasad
Computation and Language · Cross-listed from cs.LG
Power to the Clients: Federated Learning in a Dictatorship Setting
Mohammadsajad Alipour and Mohammad Mohammadi Amiri
Federated learning (FL) has emerged as a promising paradigm for decentralized model training, enabling multiple clients to collaboratively learn a shared model without exchanging their local data. However, the decentralized nature of FL also introduces vulnerabilities, as malicious clients can compromise or manipulate the training process. In this work, we introduce dictator clients, a novel, well-defined, and analytically tractable class of malicious participants capable of entirely erasing the contributions of all other clients from the server model, while preserving their own. We propose concrete attack strategies that empower such clients and systematically analyze their effects on the learning process. Furthermore, we explore complex scenarios involving multiple dictator clients, including cases where they collaborate, act independently, or form an alliance in order to ultimately betray one another. For each of these settings, we provide a theoretical analysis of their impact on the global model's convergence. Our theoretical
algorithms and findings about the complex scenarios including multiple dictator clients are further supported by empirical evaluations on both computer
vision and natural
language processing
benchmarks.
Read the full abstract ▾