Computation and Language · Cross-listed from cs.AI
UP-NRPA: User Portrait based Nested Rollout Policy Adaptation for Planning with Large Language Models in Goal-oriented Dialogue Systems
Hui Wang, Fafa Zhang, Meng Liu et al. (+2 more)
To address the challenge that current dialogue policy planning methods struggle to dynamically adapt to diverse user characteristics, this paper proposes a User Portrait based Nested Rollout Policy Adaptation (UP-NRPA) online framework with Large
Language Models. In contrast to conventional approaches dependent on model training and require offline reinforcement learning policy models for user groups, UP-NRPA enables dynamic customization of dialogue strategies through an adaptive mechanism. This is achieved by leveraging real-time user feedback alongside personality, preferences, and objectives mapped from the current user portrait, thereby adapting to user characteristics without offline reinforcement learning. In collaborative and non-collaborative dialogue
benchmarks, UP-NRPA demonstrated considerable benefits, achieving an impressive 100% success rate in multiple dialogue tasks. Particularly in negotiation tasks, the sale-to-list ratio (SL) increased by 56.41%. This demonstrates that UP-NRPA can adapt to diverse user needs without requiring a training mechanism, enabling the dialogue system to adapt to user characteristics.
Read the full abstract ▾
All authors Hui Wang, Fafa Zhang, Meng Liu, Xiangyu Chen, Chaoxu Mu
Computation and Language
MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models
Ji-jun Park, Soo-joon Choi, Jiwon Jeong et al. (+2 more)
Large
language models (LLMs) frequently encode factual and
reasoning knowledge in their internal representations that is not faithfully reflected in their surface-level outputs – a phenomenon known as latent knowledge. Existing approaches to eliciting latent knowledge, such as Contrastive Consistency Search (CCS), rely on contrastive activation patterns and struggle with complex multi-step
reasoning tasks, while mechanistic interpretability tools have primarily been used to understand model behavior rather than to extract hidden knowledge. We present MechELK, a unified three-stage framework that bridges mechanistic interpretability and latent knowledge elicitation. MechELK operates through: (1) Locate – using Sparse Autoencoder (SAE) feature analysis and activation patching to identify knowledge-bearing representations; (2) Verify – employing causal probing to distinguish genuine latent knowledge from spurious correlations; and (3) Elicit – applying representation engineering to surface hidden knowledge without modifying model weights. Evaluated on TruthfulQA, a curated Deceptive
Alignment benchmark, and the Quirky LM
dataset, MechELK achieves an average elicitation accuracy of 84.7%, outperforming CCS by 6.2% and direct linear probing by 9.1%. Crucially, MechELK successfully identifies latent knowledge in 78.3% of cases where the model's surface output is incorrect or evasive, demonstrating its utility for AI safety applications including deceptive
alignment detection.
Read the full abstract ▾
All authors Ji-jun Park, Soo-joon Choi, Jiwon Jeong, Taeyang Yoon, Ju-Wan Lee
Computation and Language
A Modular Architecture for Typologically Controlled Lexicon Generation
Sankalp Tattwadarshi Swain and Dhruv Kumar
Constructing artificial lexicons that are pronounceable, typologically plausible, and semantically structured remains an open challenge in computational linguistics. Existing conlang generators either lack formal phonotactic guarantees or delegate generation to opaque, non-reproducible LLM-based pipelines. We propose a modular framework that samples phoneme inventories from PHOIBLE, generates word forms under interchangeable phonological grammars (deterministic, OT, and MaxEnt), and assigns meanings via a Swadesh–Leipzig–Jakarta ontology with explicit form–meaning
alignment. Evaluation on character $n$-gram perplexity, log-likelihood, and KL divergence against PHOIBLE across lexicon sizes of 100-5,000 forms shows that probabilistic grammars consistently outperform deterministic and random baselines on both phonotactic coherence and typological realism.
Read the full abstract ▾
Computation and Language
What are They Thinking? Delineation, Probing, and Tracking of Concepts in LLMs
Mohamed Abdelwahab, Michelle Yu Collins, Sihan Chen et al. (+5 more)
As the influence of LLMs expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of high-level abstract concepts within the
embeddings computed in an LLM - which is what we might say a model is “thinking" about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation. In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a high-level abstract concept through the creation of a
dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed. Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to monitor new models.
Read the full abstract ▾
All authors Mohamed Abdelwahab, Michelle Yu Collins, Sihan Chen, Yi Cheng Zhao, Zafarullah Mahmood, Jiading Zhu, Soliman Ali, Jonathan Rose
Computation and Language · Cross-listed from cs.AI
Identifying and Understanding Human Values in Text: A Tailorable LLM-based Architecture
Eduardo de la Cruz Fernández, Marcelo Karanik and Sascha Ossowski
As intelligent systems become more autonomous, the scientific community focuses on creating decision-making mechanisms that include ethical and moral considerations, unlike traditional utility-maximisation models. To achieve this, a key aspect is assessing how well these decisions align with human values. To this end, a promising line of research is centred on developing approaches based on Large
Language Models (LLMs) to identify human values from text, whether explicit or implicit, enabling their recognition throughout. This paper introduces a LLM-based architecture to detect and quantify the intensity of human values in text, avoiding the limitations of previous approaches tied to specific value theory or complex
prompt engineering. The architecture comprises three coordinated modules: one that generates structured value specifications from the foundational texts of any theoretical framework; one that labels texts using these specifications; and one that assigns graded support or resistance based on rhetorical and semantic evidence. This modular approach separates the tasks of conceptualising from detecting human values, creating a scalable and reproducible process driven by value specifications adaptable to various theories. The architecture was instantiated with multiple LLMs and evaluated using the ValueEval
dataset. The experiments demonstrate good detection performance, confirming the generality of the pipeline.
Read the full abstract ▾
Computation and Language
Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
Hwanjun Song
Much of the
alignment tuning literature is organized around optimization objectives, while the construction of
alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe
alignment tuning as a pipeline design problem. We decompose
alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing
alignment methods into a unified taxonomy. Through this lens, we identify recurring design trade-offs and failure modes observed across prior
alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for
alignment data pipelines, including
prompt-level
alignment, agentic settings, and
alignment under evolving objectives.
Read the full abstract ▾
Computation and Language
Conv-to-Bench: Evaluating Language Models Via User-Assistant Dialogues In Code Tasks
Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo et al. (+5 more)
The rapid advancement of Large
Language Models (LLMs) has outpaced the scalability of traditional evaluation
benchmarks, which remain heavily dependent on labor-intensive expert curation. We address this bottleneck with Conv-to-Bench, a multi-stage framework that automatically transforms authentic multi-turn user-assistant dialogues into structured, verifiable requirement checklists. By leveraging the "instructional
evolution" found in real-world conversational logs, our approach deconstructs fragmented user intent into consolidated instructions and binary evaluation criteria. Applied to the programming domain, Conv-to-Bench produces evaluation sets that demonstrate near-perfect
alignment with human-authored standards like BigCodeBench, achieving Spearman correlations of up to $\rho$ = 1.000 with significantly lower computational overhead. Validation of the LLM-as-a-judge framework further confirms its reliability, with the primary evaluator achieving substantial agreement with human-verified ground truth ($\kappa$ = 0.705). Our comprehensive ablation studies reveal that while multi-turn interactions capture the iterative
evolution of user intent, instruction-centric extraction provides a more robust foundation. Ultimately, Conv-to-Bench provides a scalable, cost-effective paradigm for maintaining high-fidelity evaluation standards as user-centric AI applications continue to diversify.
Read the full abstract ▾
All authors Victor M. dos Santos, Andre C. Castro, Samuel L. de S. Toledo, Bruno M. L. Calura, Lisandra C. de M. Menezes, Raul C. R. Mata, Telma W. de L. Soares, Bryan L. M. de Oliveira
Computation and Language
Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches
Marcin Michał Mirończuk
Information fusion is used widely to improve document classification by the integration of multiple data sources (
multimodal) or representations (multiview). However, the field lacks a unified framework, a quantitative synthesis of its effectiveness, and clear guidance for practitioners. This systematic review addresses these gaps by analysing 139 primary studies. It introduces a formal framework to structure the field, presents the results of a qualitative analysis to identify key trends, and performs a random-effects meta-analysis (to our knowledge, the first focused on document classification) to quantify performance gains. Our meta-analysis reveals that
multimodal fusion improves accuracy (mean gain of +5.28 percentage points, $p=0.0016$) significantly – the F1-score effect is directionally positive but statistically non-significant in our primary model. Multiview fusion provides consistent but modest gains for accuracy (+4.67%), F1-score (+3.08%), and recall (all $p<0.05$). Critically, our qualitative synthesis uncovers challenges in reproducibility in methodological rigour: only 11.8% (
multimodal) and 23.3% (multiview) of the studies use statistical tests to validate their findings, which undermines the reliability of many of their results. This review's primary contributions are a unifying framework, the first quantitative evidence base, and data-driven guidelines. This review concludes that successful information fusion depends not on algorithmic complexity, but on the strategic
alignment of the fusion method with the task context and a commitment to more rigorous validation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
SkillGraph: Graph Foundation Priors for LLM Agent Tool Sequence Recommendation
Hao Liu and Dongyu Li
LLM
agents must select tools from large API libraries and order them correctly. Existing methods use semantic similarity for both
retrieval and ordering, but ordering depends on inter-tool data dependencies that are absent from tool descriptions. As a result, semantic-only methods can produce negative Kendall-$\tau$ in structured workflow domains. We introduce SkillGraph, a directed weighted execution-transition graph mined from 49,831 successful LLM
agent trajectories, which encodes workflow-precedence regularities as a reusable graph foundation prior. Building on this graph foundation prior, we propose a two-stage decoupled framework: GS-Hybrid
retrieval for candidate selection and a learned pairwise reranker for ordering. On ToolBench (9,965 test instances; ~16,000 tools), the method reaches Set-F1 = 0.271 and Kendall-$\tau$ = 0.096; on API-Bank, Kendall-$\tau$ improves from -0.433 to +0.613. Under identical Stage-1 inputs, the learned reranker also outperforms LLaMA-3.1-8B Stage-2 rerankers.
Read the full abstract ▾
Computation and Language
SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future
Timothy B. Higgins, Antonios Mamalakis and Chirag Agarwal
Recent advances in visual-
language models (VLMs) have led to significant improvements in a plethora of complex
multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather
forecasting data. In this work, we present SynopticBench, a high-quality
dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena
Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and
climate text generation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding
Yoonmin Cha, Dawit Chun and Sung Park
Brain-computer interfaces (BCIs) for
speech restoration hold transformative potential for the approximately 173,000–232,500 individuals worldwide with ALS-related dysarthria. Despite recent progress, high-performance
speech BCIs have been demonstrated in only 22–31 patients globally, largely due to limitations in neural decoding accuracy and practical input interfaces. We present iPhoneme, a
brain-to-text communication system that jointly addresses these challenges through integrated modeling and interaction design. The system combines a deep learning phoneme decoder based on a modified Conformer architecture (ConformerXL, 192.9M parameters) with a gaze-assisted phoneme input interface that mitigates the Midas touch problem in eye-tracking systems. The acoustic model incorporates a temporal prenet with multi-scale dilated convolutions and bidirectional GRU for neural jitter correction, temporal subsampling for CTC stability, and Pre-RMSNorm stabilization across 12 encoder blocks, trained with AdamW and cosine scheduling. On the interaction side, iPhoneme introduces a chorded gaze-plus-silent-
speech paradigm that replaces dwell-time selection, enabling more efficient input. We evaluate the system on the T15
dataset (45 sessions, 8,071 trials) of 256-channel intracranial
EEG from
speech motor cortex regions. A 6-gram phoneme
language model trained on 3.1M sequences, combined with WFST beam search (beam=128), achieves 92.14% phoneme accuracy (7.86% PER) and 73.39% word accuracy (26.61% WER), approximately 3% above prior state-of-the-art. The system operates on CPU with 180 ms latency, demonstrating real-time, high-accuracy
brain-to-text communication for ALS.
Read the full abstract ▾
Computation and Language
Fine-tuning Whisper for Pashto ASR: strategies and scale
Hanif Rahman
Pashto is absent from Whisper's pre-training corpus despite being one of CommonVoice's largest
language collections, leaving off-the-shelf models unusable: all Whisper sizes output Arabic, Dari, or Urdu script on Pashto audio, achieving word error rates above 100%. We compare four fine-tuning strategies for whisper-base on CommonVoice Pashto v20: vanilla full fine-tuning, LoRA (rank 64), frozen-encoder (2/6 layers), and multistage Urdu-to-Pashto transfer. We extend vanilla fine-tuning to whisper-small and whisper-large-v3-turbo on CommonVoice Pashto v24 (113 hours). Vanilla fine-tuning achieves WER 21.22% on CV20, outperforming LoRA by 33.36 pp, frozen-encoder by 14.76 pp, and Urdu transfer by 44.56 pp. Frozen-encoder fine-tuning degrades performance on whisper-base (6 encoder layers): layer-function separation does not hold at this depth, and freezing removes a third of trainable capacity. Urdu-to-Pashto transfer fails due to an unverified intermediate checkpoint, phonological mismatch, and insufficient training. On CV24, whisper-small achieves WER 24.89% (2.24 pp over whisper-base at 3.3x parameters); whisper-large-v3-turbo achieves 23.37% (a further 1.52 pp). Diminishing returns indicate whisper-small is the practical optimum at 113 hours. Online augmentation provides 7.25 pp WER benefit over matched training. Error analysis identifies word-final suffix confusion (masculine -ay vs. feminine -a) and retroflex substitutions involving the Pashto-unique consonant /ts/ as dominant failure modes. Fine-tuned checkpoints and evaluation scripts are released on HuggingFace.
Read the full abstract ▾
Computation and Language
MedConclusion: A Benchmark for Biomedical Conclusion Generation from Structured Abstracts
Weiyue Li, Ruizhi Qian, Yi Li et al. (+5 more)
Large
language models (LLMs) are widely explored for
reasoning-intensive research tasks, yet resources for testing whether they can infer scientific conclusions from structured biomedical evidence remain limited. We introduce $\textbf{MedConclusion}$, a large-scale
dataset of $\textbf{5.7M}$ PubMed structured abstracts for biomedical conclusion generation. Each instance pairs the non-conclusion sections of an abstract with the original author-written conclusion, providing naturally occurring supervision for evidence-to-conclusion
reasoning. MedConclusion also includes journal-level metadata such as biomedical category and SJR, enabling subgroup analysis across biomedical domains. As an initial study, we evaluate diverse LLMs under conclusion and summary prompting settings and score outputs with both reference-based metrics and LLM-as-a-judge. We find that conclusion writing is behaviorally distinct from summary writing, strong models remain closely clustered under current automatic metrics, and judge identity can substantially shift absolute scores. MedConclusion provides a reusable data resource for studying scientific evidence-to-conclusion
reasoning. Our code and data are available at: https://github.com/Harvard-AI-and-Robotics-Lab/MedConclusion.
Read the full abstract ▾
All authors Weiyue Li, Ruizhi Qian, Yi Li, Yongce Li, Yunfan Long, Jiahui Cai, Yan Luo, Mengyu Wang
Computation and Language · Cross-listed from cs.LG
Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning
Philipp Hellwig, Willem Zuidema, Claire E. Stevenson et al. (+1 more)
Analogical
reasoning is a hallmark of human intelligence, enabling us to solve new problems by transferring knowledge from one situation to another. Yet, developing artificial intelligence systems capable of robust human-like analogical
reasoning has proven difficult. In this work, we train
transformers using Meta-Learning for Compositionality (MLC) on an analogical
reasoning task (letter-string analogies) and assess their generalization capabilities. We find that letter-string analogies become learnable when guiding the models to attend to the most informative problem elements, induced by including copy tasks in the training data. Furthermore, generalization to new alphabets improves when models are trained with more heterogeneous
datasets. For the best training run, our 3-layer encoder-decoder model performs on par with frontier models on our letter-string analogy
datasets. The MLC approach also enables some generalization to compositions of trained transformations, but not to completely novel transformations. To understand how the model solves the analogies, we identify an
algorithm for one analogy task that approximates the model's computations. We verify this using interpretability analyses and show that head-level
attention weights causally affect model output, allowing steering between different analogy tasks. We discuss the implications of our findings for generalization capabilities of larger models and the parallels to human analogical
reasoning.
Read the full abstract ▾
All authors Philipp Hellwig, Willem Zuidema, Claire E. Stevenson, Martha Lewis
Computation and Language
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello et al. (+9 more)
Conventional end-to-end automatic
speech recognition (ASR) systems rely on paired
speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a
speech encoder to a large
language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the
speech projector. We investigate whether small amounts of
speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired
speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited
speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours)
speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full
dataset, indicating that small amounts of
speech provide a strong modality-
alignment signal.
Read the full abstract ▾
All authors Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés, Shiran Liu, Severin Baroudi, Shashi Kumar, Hasindri Watawana, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke
Computation and Language
ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs
Zhipin Wang, Christoph Leiter, Christian Frey et al. (+3 more)
Cultural values are expressed not only through
language but also through visual scenes and everyday social practices. Yet existing evaluations of cultural values in
language models are almost entirely text-only, leaving it unclear whether culture-conditioned judgments remain stable when response options are visualized. We introduce ValueGround, a
benchmark for evaluating culture-conditioned visual value grounding in
multimodal large
language models (MLLMs). Built from World Values Survey questions, ValueGround uses minimally contrastive image pairs to represent opposing response options while controlling irrelevant variation. Given a country, a question, and an image pair, a model must choose the image that best matches the country's value tendency without access to the original response-option texts. Experiments across six MLLMs and 13 countries show that models perform substantially worse with visualized response options than with the original textual options, with average accuracy dropping from 72.8% to 62.6%. Our
benchmark provides a controlled testbed for studying cross-modal transfer of culture-conditioned value judgments.
Read the full abstract ▾
All authors Zhipin Wang, Christoph Leiter, Christian Frey, Mohamed Hesham Ibrahim Abdalla, Josif Grabocka, Steffen Eger
Computation and Language
DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling
Shicheng Liu, Yucheng Jiang, Sajid Farook et al. (+3 more)
Deep research with Large
Language Model (LLM)
agents is emerging as a powerful paradigm for multi-step information discovery, synthesis, and analysis. However, existing approaches primarily focus on unstructured web data, while the challenges of conducting deep research over large-scale structured databases remain relatively underexplored. Unlike web-based research, effective data-centric research requires more than
retrieval and summarization and demands iterative hypothesis generation, quantitative
reasoning over structured schemas, and convergence toward a coherent analytical narrative. In this paper, we present DataSTORM, an LLM-based agentic system capable of autonomously conducting research across both large-scale structured databases and internet sources. Grounded in principles from Exploratory Data Analysis and Data Storytelling, DataSTORM reframes deep research over structured data as a thesis-driven analytical process: discovering candidate theses from data, validating them through iterative cross-source investigation, and developing them into coherent analytical narratives. We evaluate DataSTORM on InsightBench, where it achieves a new state-of-the-art result with a 19.4% relative improvement in insight-level recall and 7.2% in summary-level score. We further introduce a new
dataset built on ACLED, a real-world complex database, and demonstrate that DataSTORM outperforms proprietary systems such as ChatGPT Deep Research across both automated metrics and human evaluations.
Read the full abstract ▾
All authors Shicheng Liu, Yucheng Jiang, Sajid Farook, Camila Nicollier Sanchez, David Fernando Castro Pena, Monica S. Lam
Computation and Language
Multi-objective Evolutionary Merging Enables Efficient Reasoning Models
Mario Iacobelli, Adrian Robert Minut, Tommaso Mencattini et al. (+4 more)
Reasoning models achieve strong performance on complex problems by leveraging long chains of thought, but this deliberate
reasoning incurs substantial inference-time cost. The Long-to-Short (L2S)
reasoning problem seeks to preserve accuracy while reducing generated
tokens. Yet, current training-free model merging approaches rely on brittle, fixed-hyperparameter arithmetic methods that force suboptimal compromises. We introduce Evo-L2S, a multi-objective evolutionary model merging framework that explicitly optimizes accuracy and output length to recover a Pareto front of merged models. To make this search computationally tractable, we propose an
entropy-based subset sampling technique that substantially reduces fitness-estimation overhead. Across six mathematical
reasoning benchmarks, Evo-L2S reduces
reasoning length by over 50% at the 1.5B and 7B scales while preserving or improving problem-solving accuracy; at 14B, the steeper Pareto front reveals that attainable compression depends on the intrinsic compressibility of the
reasoning model. Overall, Evo-L2S shows that
reasoning models can be made substantially more concise while preserving strong problem-solving performance.
Read the full abstract ▾
All authors Mario Iacobelli, Adrian Robert Minut, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Iacopo Masi, Emanuele Rodolà
Computation and Language
Context-Aware Dialectal Arabic Machine Translation with Interactive Region and Register Selection
Afroza Nowshin, Prithweeraj Acharjee Porag, Haziq Jeelani et al. (+1 more)
Current Machine Translation (MT) systems for Arabic often struggle to account for dialectal diversity, frequently homogenizing dialectal inputs into Modern Standard Arabic (MSA) and offering limited user control over the target vernacular. In this work, we propose a context-aware and steerable framework for dialectal Arabic MT that explicitly models regional and sociolinguistic variation. Our primary technical contribution is a Rule-Based Data Augmentation (RBDA) pipeline that expands a 3,000-sentence seed corpus into a balanced 57,000-sentence parallel
dataset, covering eight regional varieties eg., Egyptian, Levantine, Gulf, etc. By fine-tuning an mT5-base model conditioned on lightweight metadata tags, our approach enables controllable generation across dialects and social registers in the translation output. Through a combination of automatic evaluation and qualitative analysis, we observe an apparent accuracy-fidelity trade-off: high-resource baselines such as NLLB (No
Language Left Behind) achieve higher aggregate BLEU scores (13.75) by defaulting toward the MSA mean, while exhibiting limited dialectal specificity. In contrast, our model achieves lower BLEU scores (8.19) but produces outputs that align more closely with the intended regional varieties. Supporting qualitative evaluation, including an LLM-assisted cultural authenticity analysis, suggests improved dialectal
alignment compared to baseline systems (4.80/5 vs. 1.0/5). These findings highlight the limitations of standard MT metrics for dialect-sensitive tasks and motivate the need for evaluation practices that better reflect linguistic diversity in Arabic MT.
Read the full abstract ▾
All authors Afroza Nowshin, Prithweeraj Acharjee Porag, Haziq Jeelani, Fayeq Jeelani Syed
Computation and Language
Learning to Interrupt in Language-based Multi-agent Communication
Danqing Wang, Da Yin, Ruta Desai et al. (+3 more)
When a colleague starts explaining something you already understand, you interrupt them. This simple act, a listener taking control of the conversation, is natural in human communication but absent in current verbose LLM multi-
agent systems. Current approaches address communication efficiency only from the speaker side, compressing messages before they are sent. We flip the perspective: rather than making speakers more concise, we let listeners decide when they have heard enough. We propose a new communication paradigm in which the listener can interrupt the speaker mid-generation. We find that LLMs, given this ability, are overconfident and interrupt too early before receiving sufficient information. This finding motivates HANDRAISER, a learning method that predicts the right moment to interrupt based on estimated future reward and communication cost. We evaluate our framework on three multi-
agent tasks: 2-
agent text pictionary, 3-
agent meeting scheduling, and 3-
agent debate. HANDRAISER reduces communication cost by 32.2% over the non-interruptible baseline while achieving comparable or superior task performance, with interruption behavior that generalizes across different
agents and tasks.
Read the full abstract ▾
All authors Danqing Wang, Da Yin, Ruta Desai, Lei Li, Asli Celikyilmaz, Ansong Ni
Computation and Language · Cross-listed from cs.LG
The Depth Ceiling: On the Limits of Large Language Models in Discovering Latent Planning
Yi Xu, Philipp Jettkant and Laura Ruis
The viability of chain-of-thought (CoT) monitoring hinges on models being unable to reason effectively in their latent representations. Yet little is known about the limits of such latent
reasoning in LLMs. We test these limits by studying whether models can discover multi-step planning strategies without supervision on intermediate steps and execute them latently, within a single forward pass. Using graph path-finding tasks that precisely control the number of required latent planning steps, we uncover a striking limitation unresolved by massive scaling: tiny
transformers trained from scratch discover strategies requiring up to three latent steps, fine-tuned GPT-4o and Qwen3-32B reach five, and GPT-5.4 attains seven under few-shot prompting. Although the maximum latent planning depth models can learn during training is five, the discovered strategy generalizes up to eight latent steps at test-time. This reveals a dissociation between the ability to discover a latent strategy under final-answer supervision alone and the ability to execute it once discovered. If similar limits hold more broadly, strategies requiring multiple coordinated latent planning steps may need to be explicitly taught or externalized, lending credence to CoT monitoring.
Read the full abstract ▾
Computation and Language
Team Fusion@ SU@ BC8 SympTEMIST track: transformer-based approach for symptom recognition and linking
Georgi Grazhdanski, Sylvia Vassileva, Ivan Koychev et al. (+1 more)
This paper presents a
transformer-based approach to solving the SympTEMIST named entity recognition (NER) and entity linking (EL) tasks. For NER, we fine-tune a RoBERTa-based (1)
token-level classifier with BiLSTM and CRF layers on an augmented train set. Entity linking is performed by generating candidates using the cross-lingual SapBERT XLMR-Large (2), and calculating cosine similarity against a knowledge base. The choice of knowledge base proves to have the highest impact on model accuracy.
Read the full abstract ▾
All authors Georgi Grazhdanski, Sylvia Vassileva, Ivan Koychev, Svetla Boytcheva
Computation and Language
When to Call an Apple Red: Humans Follow Introspective Rules, VLMs Don't
Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li et al. (+3 more)
Understanding when
Vision-
Language Models (VLMs) will behave unexpectedly, whether models can reliably predict their own behavior, and if models adhere to their introspective
reasoning are central challenges for trustworthy deployment. To study this, we introduce the Graded Color Attribution (GCA)
dataset, a controlled
benchmark designed to elicit decision rules and evaluate participant faithfulness to these rules. GCA consists of line drawings that vary pixel-level color coverage across three conditions: world-knowledge recolorings, counterfactual recolorings, and shapes with no color priors. Using GCA, we ask both VLMs and human participants to state a threshold rule: the share of an object's pixels that must be a given color for the object to receive that color label. We then compare these rules with their subsequent color attribution decisions. Our findings reveal that models systematically violate their own introspective rules. For example, GPT-5-mini violates its stated introspection rules in nearly 60% of cases on objects with strong color priors. Human participants remain faithful to their stated rules, with any apparent violations being explained by a well-documented tendency to overestimate color coverage. In contrast, we find that VLMs can accurately estimate color coverage, yet directly contradict their own
reasoning in their final responses. Across all models and strategies for eliciting introspective rules, world-knowledge priors systematically degrade faithfulness in ways that do not mirror human cognition. Our findings challenge the view that VLM
reasoning failures are difficulty-driven and suggest that VLM introspective self-knowledge is miscalibrated, with direct implications for high-stakes deployment.
Read the full abstract ▾
All authors Jonathan Nemitz, Carsten Eickhoff, Junyi Jessy Li, Kyle Mahowald, Michal Golovanevsky, William Rudman
Computation and Language
State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation
Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair et al. (+6 more)
This paper introduces Arabic-DeepSeek-R1, an application-driven open-source Arabic LLM that leverages a sparse MoE backbone to address the digital equity gap for under-represented
languages, and establishes a new SOTA across the entire Open Arabic LLM Leaderboard (OALL). Our four-phase CoT
distillation scheme integrates Arabic-specific linguistic verification and regional ethical norms into a 372M-
token, contamination-controlled 80/20 Arabic-English training mixture. Arabic-DeepSeek-R1 achieves the highest average score across the seven-
benchmark OALL suite while establishing SOTA or near-SOTA, including dominant results on grammar-focused MadinahQA (surpassing both GPT-5.1 and the OALL leader by substantial margins), safety-oriented AraTrust, multi-ability AlGhafa, and
retrieval-augmented ALRAGE. Our results indicate that the combination of sparse MoE architecture, culturally-informed CoT
distillation with explicit Arabic linguistic checks, and strategic bilingual data curation enables an open-source adapted model to systematically outperform the proprietary frontier system GPT-5.1 on the majority of
benchmarks evaluating comprehensive
language-specific tasks: the first such demonstration for Arabic LLMs. These findings indicate that much of Arabic's performance deficit in current LLM ecosystems stems from under-specialization rather than architectural limitations, and that parameter-efficient adaptation of open
reasoning models can yield breakthrough SOTA performance without industrial-scale pretraining costs. Arabic-DeepSeek-R1 establishes a validated and replicable framework for sovereign and domain-specific
language technologies, demonstrating that strategic, culturally-grounded adaptation of sparse MoE backbones offers a viable and cost-effective pathway to achieving record-breaking performance across standardized
benchmarks for low-resource
languages.
Read the full abstract ▾
All authors Navan Preet Singh, Anurag Garikipati, Ahmed Abulkhair, Jyani Akshay Jagdishbhai, Atul Yaduvanshi, Amarendra Chaudhary, Madalina Ciobanu, Qingqing Mao, Ritankar Das
Computation and Language
Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
Rebecca M. M. Hicke, Sil Hamilton, David Mimno et al. (+1 more)
Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding task: generating summaries of novels. When human authors of summaries compress a story, they reveal what they consider narratively important. Therefore, by comparing human and LLM-authored summaries, we can assess whether models mirror human patterns of conceptual engagement with texts. To measure conceptual engagement, we align sentences from 150 human-written novel summaries with the specific chapters they reference. We demonstrate the difficulty of this
alignment task, which indicates the complexity of summarization as a task. We then generate and align additional summaries by nine state-of-the-art LLMs for each of the 150 reference texts. Comparing the human and model-authored summaries, we find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative, with models emphasizing the ends of texts. Comparing human narrative engagement with model
attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development. We release our
dataset to support future research.
Read the full abstract ▾
All authors Rebecca M. M. Hicke, Sil Hamilton, David Mimno, Ross Deans Kristensen-McLachlan
Computation and Language · Cross-listed from cs.CR
Say Something Else: Rethinking Contextual Privacy as Information Sufficiency
Yunze Xiao, Wenkai Li, Xiaoyuan Wu et al. (+3 more)
LLM
agents increasingly draft messages on behalf of users, yet users routinely overshare sensitive information and disagree on what counts as private. Existing systems support only suppression (omitting sensitive information) and generalization (replacing information with an abstraction), and are typically evaluated on single isolated messages, leaving both the strategy space and evaluation setting incomplete. We formalize
privacy-preserving LLM communication as an Information Sufficiency (IS) task, introduce free-text pseudonymization as a third strategy that replaces sensitive attributes with functionally equivalent alternatives, and propose a conversational evaluation protocol that assesses strategies under realistic multi-turn follow-up pressure. Across 792 scenarios spanning three power-relation types (institutional, peer, intimate) and three sensitivity categories (discrimination risk, social cost, boundary), we evaluate seven frontier LLMs on
privacy at two granularities, covertness, and utility. Pseudonymization yields the strongest
privacy\textendash utility tradeoff overall, and single-message evaluation systematically underestimates leakage, with generalization losing up to 16.3 percentage points of
privacy under follow-up.
Read the full abstract ▾
All authors Yunze Xiao, Wenkai Li, Xiaoyuan Wu, Ningshan Ma, Yueqi Song, Weihao Xuan
Computation and Language
FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts
Sylvia Vassileva, Ivan Koychev and Svetla Boytcheva
The paper presents an approach for the recognition of toxic habits named entities in Spanish clinical texts. The approach was developed for the ToxHabits Shared Task. Our team participated in subtask 1, which aims to detect substance use and abuse mentions in clinical case reports and classify them in four categories (Tobacco, Alcohol, Cannabis, and Drug). We explored various methods of utilizing LLMs for the task, including zero-shot, few-shot, and
prompt optimization, and found that GPT-4.1's few-shot prompting performed the best in our experiments. Our method achieved an F1 score of 0.65 on the test set, demonstrating a promising result for recognizing named entities in
languages other than English.
Read the full abstract ▾
Computation and Language
ART: Attention Replacement Technique to Improve Factuality in LLMs
Ziqin Luo, Yihao Quan, Xiaofeng Zhang et al. (+2 more)
Hallucination in large
language models (LLMs) continues to be a significant issue, particularly in tasks like question answering, where models often generate plausible yet incorrect or irrelevant information. Although various methods have been proposed to mitigate
hallucinations, the relationship between
attention patterns and
hallucinations has not been fully explored. In this paper, we analyze the distribution of
attention scores across each layer and
attention head of LLMs, revealing a common and intriguing phenomenon: shallow layers of LLMs primarily rely on uniform
attention patterns, where the model distributes its
attention evenly across the entire sequence. This uniform
attention pattern can lead to
hallucinations, as the model fails to focus on the most relevant information. To mitigate this issue, we propose a training-free method called
Attention Replacement Technique (ART), which replaces these uniform
attention patterns in the shallow layers with local
attention patterns. This change directs the model to focus more on the relevant contexts, thus reducing
hallucinations. Through extensive experiments, ART demonstrates significant reductions in
hallucinations across multiple LLM architectures, proving its effectiveness and generalizability without requiring fine-tuning or additional training data.
Read the full abstract ▾
All authors Ziqin Luo, Yihao Quan, Xiaofeng Zhang, Xiaosong Yuan, Chen Shen
Computation and Language
Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
Navan Preet Singh, Xiaokun Wang, Anurag Garikipati et al. (+3 more)
We present an innovative multi-stage optimization strategy combining reinforcement learning (RL) and supervised fine-tuning (SFT) to enhance the pedagogical knowledge of large
language models (LLMs), as illustrated by EduQwen 32B-RL1, EduQwen 32B-SFT, and an optional third-stage model EduQwen 32B-SFT-RL2: (1) RL optimization that implements progressive difficulty training, focuses on challenging examples, and employs extended
reasoning rollouts; (2) a subsequent SFT phase that leverages the RL-trained model to synthesize high-quality training data with difficulty-weighted sampling; and (3) an optional second round of RL optimization. EduQwen 32B-RL1, EduQwen 32B-SFT, and EduQwen 32B-SFT-RL2 are an application-driven family of open-source pedagogical LLMs built on a dense Qwen3-32B backbone. These models remarkably achieve high enough accuracy on the Cross-Domain Pedagogical Knowledge (CDPK)
Benchmark to establish new state-of-the-art (SOTA) results across the interactive Pedagogy
Benchmark Leaderboard and surpass significantly larger proprietary systems such as the previous
benchmark leader Gemini-3 Pro. These dense 32-billion-parameter models demonstrate that domain-specialized optimization can transform mid-sized open-source LLMs into true pedagogical domain experts that outperform much larger general-purpose systems, while preserving the transparency, customizability, and cost-efficiency required for responsible educational AI deployment.
Read the full abstract ▾
All authors Navan Preet Singh, Xiaokun Wang, Anurag Garikipati, Madalina Ciobanu, Qingqing Mao, Ritankar Das
Computation and Language
The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
Michael Rizvi-Martel, Guillaume Rabusseau and Marius Mosbach
Latent
reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT
reasoning. Operating in continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether
language models actually leverage superposition when
reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of
token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural
language data biases models to commit to a
token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought
reasoning, and identify the conditions under which it collapses.
Read the full abstract ▾