Computation and Language · Cross-listed from cs.CY
In your own words: computationally identifying interpretable themes in free-text survey data
Jenny S Wang, Aliya Saperstein and Emma Pierson
Free-text survey responses can provide nuance often missed by structured questions, but remain difficult to statistically analyze. To address this, we introduce In Your Own Words, a computational framework for exploratory analyses of free-text survey data that identifies structured, interpretable themes in free-text responses, facilitating systematic analysis. To illustrate the benefits of this approach, we apply it to a new
dataset of free-text descriptions of race, gender, and sexual orientation from 1,004 U.S. participants. The themes our approach produces on this
dataset are more coherent and interpretable than those produced by past computational methods. The themes have three practical applications in survey research. First, they can suggest structured questions to add to future surveys by surfacing salient constructs - such as belonging and identity fluidity - that existing surveys do not capture. Second, the themes reveal heterogeneity within standardized categories, explaining additional variation in health, well-being, and identity importance. Third, the themes illuminate systematic discordance between self-identified and perceived identities, highlighting mechanisms of misrecognition that existing measures do not reflect. More broadly, our framework can be deployed in a wide range of survey settings to identify interpretable themes from free text, complementing existing qualitative methods.
Read the full abstract ▾
Computation and Language
Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation
Lorcan McLaren, James Cross, Zuzanna Krakowska et al. (+2 more)
Political scientists are rapidly adopting large
language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and
prompt style interact, and whether popular "best practices" survive controlled comparison, are largely unexplored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and
prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No single model,
prompt style, or learning approach is uniformly superior, and the best-performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended
prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these
benchmark results to develop a validation-first framework - with a principled ordering of pipeline decisions, guidance on
prompt freezing and held-out evaluation, reporting standards, and open-source tools - to help researchers navigate this decision space transparently.
Read the full abstract ▾
All authors Lorcan McLaren, James Cross, Zuzanna Krakowska, Robin Rauner, Martijn Schoonvelde
Computation and Language
Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
Seine A. Shintani
Arithmetic
benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Learning to Commit: Generating Organic Pull Requests via Online Repository Memory
Mo Li, L.H. Xu, Qitai Tan et al. (+2 more)
Large
language model (LLM)-based coding
agents achieve impressive results on controlled
benchmarks yet routinely produce pull requests that real maintainers reject. The root cause is not functional incorrectness but a lack of organicity: generated code ignores project-specific conventions, duplicates functionality already provided by internal APIs, and violates implicit architectural constraints accumulated over years of development. Simply exposing an
agent to the latest repository snapshot is not enough: the snapshot reveals the final state of the codebase, but not the repository-specific change patterns by which that state was reached. We introduce Learning to Commit, a framework that closes this gap through Online Repository
Memory. Given a repository with a strict chronological split, the
agent performs supervised contrastive reflection on earlier commits: it blindly attempts to resolve each historical issue, compares its prediction against the oracle diff, and distils the gap into a continuously growing set of skills-reusable patterns capturing coding style, internal API usage, and architectural invariants. When a new PR description arrives, the
agent conditions its generation on these accumulated skills, producing changes grounded in the project's own
evolution rather than generic pretraining priors. Evaluation is conducted on genuinely future, merged pull requests that could not have been seen during the skill-building phase, and spans multiple dimensions including functional correctness, code-style consistency, internal API reuse rate, and modified-region plausibility. Experiments on an expert-maintained repository with rich commit history show that Online Repository
Memory effectively improves organicity scores on held-out future tasks.
Read the full abstract ▾
All authors Mo Li, L.H. Xu, Qitai Tan, Ting Cao, Yunxin Liu
Computation and Language
Weight Tying Biases Token Embeddings Towards the Output Space
Antonio Lopardo, Avyukth Harish, Catherine Arnett et al. (+1 more)
Weight tying, i.e. sharing parameters between input and output
embedding matrices, is common practice in
language model design, yet its impact on the learned
embedding space remains poorly understood. In this paper, we show that tied
embedding matrices align more closely with output (unembedding)
matrices than with input
embeddings of comparable untied models, indicating that the shared
matrix is shaped primarily for output prediction rather than input representation. This unembedding bias arises because output gradients dominate early in training. Using tuned lens analysis, we show this negatively affects early-layer computations, which contribute less effectively to the residual stream. Scaling input gradients during training reduces this bias, providing causal evidence for the role of gradient imbalance. This is mechanistic evidence that weight tying optimizes the
embedding matrix for output prediction, compromising its role in input representation. These results help explain why weight tying can harm performance at scale and have implications for training smaller LLMs, where the
embedding matrix contributes substantially to total parameter count.
Read the full abstract ▾
All authors Antonio Lopardo, Avyukth Harish, Catherine Arnett, Akshat Gupta
Computation and Language · Cross-listed from cs.CV
PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning
Shaoxuan Li, Zhixuan Zhao, Hanze Deng et al. (+9 more)
We introduce PerceptionComp, a manually annotated
benchmark for complex, long-horizon, perception-centric
video reasoning. PerceptionComp is designed so that no single moment is sufficient: answering each question requires multiple temporally separated pieces of visual evidence and compositional constraints under conjunctive and sequential logic, spanning perceptual subtasks such as objects, attributes, relations, locations, actions, and events, and requiring skills including semantic recognition, visual correspondence, temporal
reasoning, and spatial
reasoning. The
benchmark contains 1,114 highly complex questions on 279
videos from diverse domains including city walk tours, indoor villa tours,
video games, and extreme outdoor sports, with 100% manual annotation. Human studies show that PerceptionComp requires substantial test-time thinking and repeated perception steps: participants take much longer than on prior
benchmarks, and accuracy drops to near chance (18.97%) when rewatching is disallowed. State-of-the-art MLLMs also perform substantially worse on PerceptionComp than on existing
benchmarks: the best model in our evaluation, Gemini-3-Flash, reaches only 45.96% accuracy in the five-choice setting, while open-source models remain below 40%. These results suggest that perception-centric long-horizon
video reasoning remains a major bottleneck, and we hope PerceptionComp will help drive progress in perceptual
reasoning.
Read the full abstract ▾
All authors Shaoxuan Li, Zhixuan Zhao, Hanze Deng, Zirun Ma, Shulin Tian, Zuyan Liu, Yushi Hu, Haoning Wu, Yuhao Dong, Benlin Liu, Ziwei Liu, Ranjay Krishna
Computation and Language
EnTaCs: Analyzing the Relationship Between Sentiment and Language Choice in English-Tamil Code-Switching
Paul Bontempo
This paper investigates the relationship between utterance sentiment and
language choice in English-Tamil code-switched text, using methods from machine learning and statistical modelling. We apply a fine-tuned XLM-RoBERTa model for
token-level
language identification on 35,650 romanized YouTube comments from the DravidianCodeMix
dataset, producing per-utterance measurements of English proportion and
language switch frequency. Linear regression analysis reveals that positive utterances exhibit significantly greater English proportion (34.3%) than negative utterances (24.8%), and mixed-sentiment utterances show the highest
language switch frequency when controlling for utterance length. These findings support the hypothesis that emotional content demonstrably influences
language choice in
multilingual code-switching settings, due to socio-linguistic associations of prestige and identity with embedded and
matrix languages.
Read the full abstract ▾
Computation and Language
MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference
Joris Köster, Zixuan Liu, Siavash Khajavi et al. (+1 more)
Large
Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a
memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard
retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual
memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline.
Read the full abstract ▾
All authors Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng
Computation and Language
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Juan Gabriel Kostelec and Qinghai Guo
Converting a pretrained
Transformer into a more efficient hybrid model through
distillation offers a promising approach to reducing inference costs. However, achieving high-quality generation in distilled models requires careful joint design of both the student architecture and the
distillation process. Many prior
distillation works evaluate downstream multiple-choice
benchmarks by ranking candidate answers with log-likelihood rather than requiring autoregressive generation, which can obscure important differences in model quality. For example, on overlapping
benchmarks, we show that a 7B distilled model that nearly matches its teacher to within 0.2 pp under log-likelihood scoring falls behind by 20.8 pp when it must generate answers autoregressively. We investigate this phenomenon with GenDistill, a multi-stage pipeline we designed for distilling a pretrained
Transformer into an efficient Hybrid Kimi Delta
Attention (Hybrid-KDA) student. Using it as a controlled testbed on Qwen3-0.6B, we systematically ablate six design axes (training objective, loss masking, training duration,
dataset selection, parameter freezing, and architecture choice) and evaluate every choice under both log-likelihood and generation-based protocols. We find that log-likelihood-based evaluation consistently underestimates the gap between teacher and student, and can in some cases reverse the ranking of design choices, so conclusions drawn from perplexity-only evaluation may be misleading. Among the factors we study,
dataset selection, completion-only masking, and freezing
attention layers during post-training have the largest impact on generation quality. Our best
distillation recipe, using a Hybrid-KDA model as the student, retains 86-90% of teacher accuracy on knowledge
benchmarks while reducing KV cache
memory by up to 75% and improving time-to-first-
token by 2-4x at 128K-
token contexts.
Read the full abstract ▾
Computation and Language
Development of a European Union Time-Indexed Reference Dataset for Assessing the Performance of Signal Detection Methods in Pharmacovigilance using a Large Language Model
Maria Kefala, Jeffery L. Painter, Syed Tauhid Bukhari et al. (+1 more)
Background: The identification of optimal signal detection methods is hindered by the lack of reliable reference
datasets. Existing
datasets do not capture when adverse events (AEs) are officially recognized by regulatory authorities, preventing restriction of analyses to pre-confirmation periods and limiting evaluation of early detection performance. This study addresses this gap by developing a time-indexed reference
dataset for the European Union (EU), incorporating the timing of AE inclusion in product labels along with regulatory metadata. Methods: Current and historical Summaries of Product Characteristics (SmPCs) for all centrally authorized products (n=1,513) were retrieved from the EU Union Register of Medicinal Products (data lock: 15 December 2025). Section 4.8 was extracted and processed using DeepSeek V3 to identify AEs. Regulatory metadata, including labelling changes, were programmatically extracted. Time indexing was based on the date of AE inclusion in the SmPC. Results: The database includes 17,763 SmPC versions spanning 1995-2025, comprising 125,026 drug-AE associations. The time-indexed reference
dataset, restricted to active products, included 1,479 medicinal products and 110,823 drug-AE associations. Most AEs were identified pre-marketing (74.5%) versus post-marketing (25.5%). Safety updates peaked around 2012. Gastrointestinal, skin, and nervous system disorders were the most represented System Organ Classes. Drugs had a median of 48 AEs across 14 SOCs. Conclusions: The proposed
dataset addresses a critical gap in pharmacovigilance by incorporating temporal information on AE recognition for the EU, supporting more accurate assessment of signal detection performance and facilitating methodological comparisons across analytical approaches.
Read the full abstract ▾
All authors Maria Kefala, Jeffery L. Painter, Syed Tauhid Bukhari, Maurizio Sessa
Computation and Language
How Open Must Language Models be to Enable Reliable Scientific Inference?
James A. Michaelov, Catherine Arnett, Tyler A. Chang et al. (+7 more)
How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions on information about model construction and deployment threaten reliable inference. We argue that current closed models are generally ill-suited for scientific purposes, with some notable exceptions, and discuss ways in which the issues they present to reliable inference can be resolved or mitigated. We recommend that when models are used in research, potential threats to inference should be systematically identified along with the steps taken to mitigate them, and that specific justifications for model selection should be provided.
Read the full abstract ▾
All authors James A. Michaelov, Catherine Arnett, Tyler A. Chang, Pamela D. Rivière, Samuel M. Taylor, Cameron R. Jones, Sean Trott, Roger P. Levy, Benjamin K. Bergen, Micah Altman
Computation and Language
ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Inês Vieira, Inês Calvo, Iago Paulo et al. (+6 more)
As Large
Language Models (LLMs) expand across
multilingual domains, evaluating their performance in under-represented
languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and
benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded
benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including
Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by
language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated
language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive
benchmarks that support further development of tools in pt-PT.
Read the full abstract ▾
All authors Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães
Computation and Language
JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems
Yu Pan, Guangzhao Yang, Muhammad Huzaifah et al. (+2 more)
Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI
agents. Existing systems often rely on acoustic or linguistic cues alone, limiting accuracy and stability, whereas directly endowing large
language models with full-duplex capability incurs heavy data and deployment costs that hinder real-time usage. We propose JAL-Turn, a lightweight
speech-only turn-taking framework based on a joint acoustic-linguistic modeling paradigm, in which a cross-
attention module adaptively fuses pre-trained acoustic and linguistic representations for low-latency hold or shift prediction. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with
speech recognition, introducing almost no additional end-to-end latency. We further introduce a scalable pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Experiments on the public Easy-Turn and Smart-Turn
benchmarks and an in-house Japanese customer-service
dataset show that JAL-Turn achieves competing results among strong baselines in detection accuracy while maintaining superior real-time performance.
Read the full abstract ▾
All authors Yu Pan, Guangzhao Yang, Muhammad Huzaifah, Ningjie Bai, Shi Qiu
Computation and Language
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos et al. (+19 more)
Despite rapid progress in open large
language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated
benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT
benchmarks that includes translated standard tasks and four new
datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated
benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.
Read the full abstract ▾
All authors Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães
Computation and Language
Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs
Vinicius Anjos de Almeida, Sandro Saorin da Silva, Josimar Chire et al. (+8 more)
Clinical notes contain valuable unstructured information. Named entity recognition (NER) enables the automatic extraction of medical concepts; however,
benchmarks for Portuguese remain scarce. In this study, we aimed to evaluate BERT-based models and large
language models (LLMs) for clinical NER in Portuguese and to test strategies for addressing multilabel imbalance. We compared BioBERTpt, BERTimbau, ModernBERT, and mmBERT with LLMs such as GPT-5 and Gemini-2.5, using the public SemClinBr corpus and a private breast cancer
dataset. Models were trained under identical conditions and evaluated using precision, recall, and F1-score. Iterative stratification, weighted loss, and oversampling were explored to mitigate class imbalance. The mmBERT-base model achieved the best performance (micro F1 = 0.76), outperforming all other models. Iterative stratification improved class balance and overall performance.
Multilingual BERT models, particularly mmBERT, perform strongly for Portuguese clinical NER and can run locally with limited computational resources. Balanced data-splitting strategies further enhance performance.
Read the full abstract ▾
All authors Vinicius Anjos de Almeida, Sandro Saorin da Silva, Josimar Chire, Leonardo Vicenzi, Nícolas Henrique Borges, Helena Kociolek, Sarah Miriã de Castro Rocha, Frederico Nassif Gomes, Júlia Cristina Ferreira, Oge Marques, Lucas Emanuel Silva e Oliveira
Computation and Language · Cross-listed from quant-ph
Nathan Roll
Quantum language models have shown competitive performance on sequential tasks, yet whether trained
quantum circuits exploit genuinely
quantum resources – or merely embed classical computation in
quantum hardware – remains unknown. Prior work has evaluated these models through endpoint metrics alone, without examining the
memory strategies they actually learn internally. We introduce the first mechanistic interpretability study of
quantum language models, combining causal gate ablation,
entanglement tracking, and density-
matrix interchange interventions on a controlled long-range dependency task. We find that single-
qubit models are exactly classically simulable and converge to the same
geometric strategy as matched classical baselines, while two-
qubit models with entangling gates learn a representationally distinct strategy that encodes context in inter-
qubit entanglement – confirmed by three independent causal tests (p < 0.0001, d = 0.89). On real
quantum hardware, only the classical
geometric strategy survives device noise; the
entanglement strategy degrades to chance. These findings open mechanistic interpretability as a tool for the science of
quantum language models and reveal a noise-expressivity tradeoff governing which learned strategies survive deployment.
Read the full abstract ▾
Computation and Language
ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims
Raia Abu Ahmad, Max Upravitelev, Aida Usmanova et al. (+2 more)
Automatically verifying
climate-related claims against scientific literature is a challenging task, complicated by the specialised nature of scholarly evidence and the diversity of rhetorical strategies underlying
climate disinformation. ClimateCheck 2026 is the second iteration of a shared task addressing this challenge, expanding on the 2025 edition with tripled training data and a new disinformation narrative classification task. Running from January to February 2026 on the CodaBench platform, the competition attracted 20 registered participants and 8 leaderboard submissions, with systems combining dense
retrieval pipelines, cross-encoder ensembles, and large
language models with structured hierarchical
reasoning. In addition to standard evaluation metrics (Recall@K and Binary Preference), we adapt an automated framework to assess
retrieval quality under incomplete annotations, exposing systematic biases in how conventional metrics rank systems. A cross-task analysis further reveals that not all
climate disinformation is equally verifiable, potentially implicating how future fact-checking systems should be designed.
Read the full abstract ▾
All authors Raia Abu Ahmad, Max Upravitelev, Aida Usmanova, Veronika Solopova, Georg Rehm
Computation and Language
Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models
Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten et al. (+2 more)
Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large
language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The
dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU
memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-
language queries, while highlighting the need for validation and human oversight in clinical deployment.
Read the full abstract ▾
All authors Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten, Kimmo Kaski, Otso Arponen
Computation and Language
Analysing Calls to Order in German Parliamentary Debates
Nina Smirnova, Daniel Dan and Philipp Mayr
Parliamentary debate constitutes a central arena of political power, shaping legislative outcomes and public discourse. Incivility within this arena signals political polarization and institutional conflict. This study presents a systematic investigation of incivility in the German Bundestag by examining calls to order (CtO; plural: CtOs) as formal indicators of norm violations. Despite their relevance, CtOs have received little systematic
attention in parliamentary research. We introduce a rule-based method for detecting and annotating CtOs in parliamentary
speeches and present a novel
dataset of German parliamentary debates spanning 72 years that includes annotated CtO instances. Additionally, we develop the first classification system for CtO triggers and analyze the factors associated with their occurrence. Our findings show that, despite formal regulations, the issuance of CtOs is partly subjective and influenced by session presidents and parliamentary dynamics, with certain individuals disproportionately affected. An insult towards individuals is the most frequent cause of CtO. In general, male members and those belonging to opposition parties receive more calls to order than their female and coalition-party counterparts. Most CtO triggers were detected in
speeches dedicated to governmental affairs and actions of the presidency. The CtO triggers
dataset is available at: https://github.com/kalawinka/cto_analysis.
Read the full abstract ▾
Computation and Language
Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models
Richard J. Young
Extended-thinking models expose a second text-generation channel ("thinking
tokens") alongside the user-visible answer. This study examines 12 open-weight
reasoning models on MMLU and GPQA questions paired with misleading hints. Among the 10,506 cases where models actually followed the hint (choosing the hint's target over the ground truth), each case is classified by whether the model acknowledges the hint in its thinking
tokens, its answer text, both, or neither. In 55.4% of these cases the model's thinking
tokens contain hint-related keywords that the visible answer omits entirely, a pattern termed *thinking-answer divergence*. The reverse (answer-only acknowledgment) is near-zero (0.5%), confirming that the asymmetry is directional. Hint type shapes the pattern sharply: sycophancy is the most *transparent* hint, with 58.8% of sycophancy-influenced cases acknowledging the professor's authority in both channels, while consistency (72.2%) and unethical (62.7%) hints are dominated by thinking-only acknowledgment. Models also vary widely, from near-total divergence (Step-3.5-Flash: 94.7%) to relative transparency (Qwen3.5-27B: 19.6%). These results show that answer-text-only monitoring misses more than half of all hint-influenced
reasoning and that thinking-
token access, while necessary, still leaves 11.8% of cases with no verbalized acknowledgment in either channel.
Read the full abstract ▾
Computation and Language
Word Alignment-Based Evaluation of Uniform Meaning Representations
Daniel Zeman and Federica Gamba
Comparison and evaluation of graph-based representations of sentence meaning is a challenge because competing representations of the same sentence may have different number of nodes, and it is not obvious which nodes should be compared to each other. Existing approaches favor node mapping that maximizes $F_1$ score over node relations and attributes, regardless whether the similarity is intentional or accidental; consequently, the identified mismatches in values of node attributes are not useful for any detailed error analysis. We propose a node-matching
algorithm that allows comparison of multiple Uniform Meaning Representations (UMR) of one sentence and that takes advantage of node-word
alignments, inherently available in UMR. We compare it with previously used approaches, in particular smatch (the de-facto standard in AMR evaluation), and argue that sensitivity to word
alignment makes the comparison of meaning representations more intuitive and interpretable, while avoiding the NP-hard search problem inherent in smatch. A script implementing the method is freely available.
Read the full abstract ▾
Computation and Language
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Yusheng Zhao, Hourun Li, Bohan Wu et al. (+5 more)
The
attention mechanism has been the core component in modern
transformer architectures. However, the computation of standard full
attention scales quadratically with the sequence length, serving as a major bottleneck in long-context
language modeling. Sliding window
attention restricts the context length for better efficiency at the cost of narrower receptive fields. While existing efforts attempt to take the benefits from both sides by building hybrid models, they often resort to static, heuristically designed alternating patterns that limit efficient allocation of computation in various scenarios. In this paper, we propose Switch
Attention (SwiAttn), a novel hybrid
transformer that enables dynamic and fine-grained routing between full
attention and sliding window
attention. For each
token at each
transformer layer, SwiAttn dynamically routes the computation to either a full-
attention branch for global information aggregation or a sliding-window branch for efficient local pattern matching. An adaptive regularization objective is designed to encourage the model towards efficiency. Moreover, we adopt continual pretraining to optimize the model, transferring the full
attention architecture to the hybrid one. Extensive experiments are conducted on twenty-three
benchmark datasets across both regular (4K) and long (32K) context lengths, demonstrating the effectiveness of the proposed method.
Read the full abstract ▾
All authors Yusheng Zhao, Hourun Li, Bohan Wu, Yichun Yin, Lifeng Shang, Jingyang Yuan, Meng Zhang, Ming Zhang
Computation and Language · Cross-listed from cs.LG
A Formal Framework for Uncertainty Analysis of Text Generation with Large Language Models
Steffen Herbold and Florian Lemmerich
The generation of texts using Large
Language Models (LLMs) is inherently uncertain, with sources of uncertainty being not only the generation of texts, but also the
prompt used and the downstream interpretation. Within this work, we provide a formal framework for the measurement of uncertainty that takes these different aspects into account. Our framework models prompting, generation, and interpretation as interconnected autoregressive processes that can be combined into a single sampling tree. We introduce filters and objective functions to describe how different aspects of uncertainty can be expressed over the sampling tree and demonstrate how to express existing approaches towards uncertainty through these functions. With our framework we show not only how different methods are formally related and can be reduced to a common core, but also point out additional aspects of uncertainty that have not yet been studied.
Read the full abstract ▾
Computation and Language
CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law
JiHyeok Jung, TaeYoung Yoon and HyunSouk Cho
Legal
reasoning requires not only the application of legal rules but also an understanding of the context in which those rules operate. However, existing legal
benchmarks primarily evaluate rule application under the assumption of fixed norms, and thus fail to capture situations where legal judgments shift or where multiple norms interact. In this work, we propose CALRK-Bench, a context-aware legal
reasoning benchmark based on the legal system in Korean. CALRK-Bench evaluates whether models can identify the temporal validity of legal norms, determine whether sufficient legal information is available for a given case, and understand the reasons behind shifts in legal judgments. The
dataset is constructed from legal precedents and legal consultation records, and is validated by legal experts. Experimental results show that even recent large
language models consistently exhibit low performance on these three tasks. CALRK-Bench provides a new stress test for evaluating context-aware legal
reasoning rather than simple memorization of legal knowledge. Our code is available at https://github.com/jhCOR/CALRKBench.
Read the full abstract ▾
Computation and Language
From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
Jiyuan An, Liner Yang, Mengyan Wang et al. (+3 more)
As spatial intelligence becomes an increasingly important capability for foundation models, it remains unclear whether large
language models' (LLMs) performance on spatial
reasoning benchmarks reflects structured internal spatial representations or reliance on linguistic heuristics. We address this question from a mechanistic perspective by examining how spatial information is internally represented and used. Drawing on computational theories of human spatial cognition, we decompose spatial
reasoning into three primitives, relational composition, representational transformation, and stateful spatial updating, and design controlled task families for each. We evaluate
multilingual LLMs in English, Chinese, and Arabic under single pass inference, and analyze internal representations using linear probing, sparse autoencoder based feature analysis, and causal interventions. We find that task relevant spatial information is encoded in intermediate layers and can causally influence behavior, but these representations are transient, fragmented across task families, and weakly integrated into final predictions. Cross linguistic analysis further reveals mechanistic degeneracy, where similar behavioral performance arises from distinct internal pathways. Overall, our results suggest that current LLMs exhibit limited and context dependent spatial representations rather than robust, general purpose spatial
reasoning, highlighting the need for mechanistic evaluation beyond
benchmark accuracy.
Read the full abstract ▾
All authors Jiyuan An, Liner Yang, Mengyan Wang, Luming Lu, Weihua An, Erhong Yang
Computation and Language
findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embedding
Héctor Javier Vázquez Martínez
Syllable-level units offer compact and linguistically meaningful representations for spoken
language modeling and unsupervised word discovery, but research on syllabification remains fragmented across disparate implementations,
datasets, and evaluation protocols. We introduce findsylls, a modular,
language-agnostic toolkit that unifies classical syllable detectors and end-to-end syllabifiers under a common interface for syllable
segmentation,
embedding extraction, and multi-granular evaluation. The toolkit implements and standardizes widely used methods (e.g., Sylber, VG-HuBERT) and allows their components to be recombined, enabling controlled comparisons of representations,
algorithms, and
token rates. We demonstrate findsylls on English and Spanish corpora and on new hand-annotated data from Kono, an underdocumented Central Mande
language, illustrating how a single framework can support reproducible syllable-level experiments across both high-resource and under-resourced settings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models
Antoine Edy, Max Conti and Quentin Macé
While Late Interaction models exhibit strong
retrieval performance, many of their underlying dynamics remain understudied, potentially hiding performance bottlenecks. In this work, we focus on two topics in Late Interaction
retrieval: a length bias that arises when using multi-vector scoring, and the similarity distribution beyond the best scores pooled by the MaxSim operator. We analyze these behaviors for state-of-the-art models on the NanoBEIR
benchmark. Results show that while the theoretical length bias of causal Late Interaction models holds in practice, bi-directional models can also suffer from it in extreme cases. We also note that no significant similarity trend lies beyond the top-1 document
token, validating that the MaxSim operator efficiently exploits the
token-level similarity scores.
Read the full abstract ▾
Computation and Language
SocialX: A Modular Platform for Multi-Source Big Data Research in Indonesia
Muhammad Apriandito Arya Saputra, Andry Alamsyah, Dian Puteri Ramadhani et al. (+2 more)
Big data research in Indonesia is constrained by a fundamental fragmentation: relevant data is scattered across social media, news portals, e-commerce platforms, review sites, and academic databases, each with different formats, access methods, and noise characteristics. Researchers must independently build collection pipelines, clean heterogeneous data, and assemble separate analysis tools, a process that often overshadows the research itself. We present SocialX, a modular platform for multi-source big data research that integrates heterogeneous data collection,
language-aware preprocessing, and pluggable analysis into a unified, source-agnostic pipeline. The platform separates concerns into three independent layers (collection, preprocessing, and analysis) connected by a lightweight job-coordination mechanism. This modularity allows each layer to grow independently: new data sources, preprocessing methods, or analysis tools can be added without modifying the existing pipeline. We describe the design principles that enable this extensibility, detail the preprocessing methodology that addresses challenges specific to Indonesian text across registers, and demonstrate the platform's utility through a walkthrough of a typical research workflow. SocialX is publicly accessible as a web-based platform at https://www.socialx.id.
Read the full abstract ▾
All authors Muhammad Apriandito Arya Saputra, Andry Alamsyah, Dian Puteri Ramadhani, Thomhert Suprapto Siadari, Hanif Fakhrurroja
Computation and Language
Automatic Speech Recognition for Documenting Endangered Languages: Case Study of Ikema Miyakoan
Chihiro Taguchi, Yukinori Takubo and David Chiang
Language endangerment poses a major challenge to linguistic diversity worldwide, and technological advances have opened new avenues for documentation and revitalization. Among these, automatic
speech recognition (ASR) has shown increasing potential to assist in the transcription of endangered
language data. This study focuses on Ikema, a severely endangered Ryukyuan
language spoken in Okinawa, Japan, with approximately 1,300 remaining speakers, most of whom are over 60 years old. We present an ongoing effort to develop an ASR system for Ikema based on field recordings. Specifically, we (1) construct a 6.33-hour
speech corpus from field recordings, (2) train an ASR model that achieves a character error rate as low as 15%, and (3) evaluate the impact of ASR assistance on the efficiency of
speech transcription. Our results demonstrate that ASR integration can substantially reduce transcription time and cognitive load, offering a practical pathway toward scalable, technology-supported documentation of endangered
languages.
Read the full abstract ▾
Computation and Language
Abstract Compression of Conversational Audio Context for LLM-based ASR
Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso et al. (+7 more)
Standard LLM-based
speech recognition systems process utterances in isolation, limiting their ability to leverage conversational context. Here we study whether
multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio
embedding sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, a learned latent abstraction of prior-turn audio that replaces each prior-turn audio sequence with a fixed number of latent
tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compression-enabled model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its tradeoffs.
Read the full abstract ▾
All authors Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke