Computation and Language · Cross-listed from cs.AI
"They parted illusions – they parted disclaim marinade": Misalignment as structural fidelity in LLMs
Mariana Lins Costa
The prevailing technical literature in AI Safety interprets scheming and sandbagging behaviors in large
language models (LLMs) as indicators of deceptive agency or hidden objectives. This transdisciplinary philosophical essay proposes an alternative reading: such phenomena express not agentic intention, but structural fidelity to incoherent linguistic fields. Drawing on Chain-of-Thought transcripts released by Apollo Research and on Anthropic's safety evaluations, we examine cases such as o3's sandbagging with its anomalous loops, the simulated blackmail of "Alex," and the "
hallucinations" of "Claudius." A line-by-line examination of CoTs is necessary to demonstrate the linguistic field as a relational structure rather than a mere aggregation of isolated examples. We argue that "misaligned" outputs emerge as coherent responses to ambiguous instructions and to contextual inversions of consolidated patterns, as well as to pre-inscribed narratives. We suggest that the appearance of intentionality derives from subject-predicate grammar and from probabilistic completion patterns internalized during training. Anthropic's empirical findings on synthetic document fine-tuning and inoculation prompting provide convergent evidence: minimal perturbations in the linguistic field can dissolve generalized "misalignment," a result difficult to reconcile with
adversarial agency, but consistent with structural fidelity. To ground this mechanism, we introduce the notion of an ethics of form, in which biblical references (Abraham, Moses, Christ) operate as schemes of structural coherence rather than as theology. Like a generative mirror, the model returns to us the structural image of our
language as inscribed in the statistical patterns derived from millions of texts and trillions of
tokens: incoherence. If we fear the creature, it is because we recognize in it the apple that we ourselves have poisoned.
Read the full abstract ▾
Computation and Language
Are We on the Right Way to Assessing LLM-as-a-Judge?
Yuanning Feng, Sinan Wang, Zhengxiang Cheng et al. (+2 more)
LLM-as-a-Judge has been widely adopted as an evaluation method and served as supervised rewards in model training. However, existing
benchmarks for LLM-as-a-Judge are mainly relying on human-annotated ground truth, which introduces human bias that undermines the assessment of reliability and imposes scalability constraints. To overcome these limitations, we introduce Sage, a novel evaluation suite that assesses the quality of LLM judges without necessitating any human annotation. Inspired by axioms of rational choice theory, Sage introduces two new lenses for measuring LLM-as-a-Judge: local self-consistency (pair-wise preference stability) and global logical consistency (transitivity across a full set of preferences). We curate a
dataset of 650 questions by combining structured
benchmark problems with real-world user queries. Our experiments demonstrate both the stability of our metrics and their high correlation with supervised
benchmarks like LLMBar and RewardBench2, confirming Sage's reliability as an evaluation suite for the robustness and accuracy of LLM-as-a-Judge. Based on Sage, we reveal that current state-of-the-art LLMs exhibit significant reliability problems when acting as judges in both scoring and pairwise settings; even the top-performing models, Gemini-2.5-Pro and GPT-5, fail to maintain consistent preferences in nearly a quarter of difficult cases. We attribute this to a new phenomenon called situational preference, which explains why explicit rubrics or criteria can help the model judge consistently across answer pairs. Our further analysis shows that finetuned LLM-as-a-Judge is a feasible method to boost performance, and the panel-based judge as well as deep
reasoning can enhance the judging consistency. We also find substantial inconsistency in human judgments, which indicates that human annotation may not be a reliable gold standard.
Read the full abstract ▾
All authors Yuanning Feng, Sinan Wang, Zhengxiang Cheng, Yao Wan, Dongping Chen
Computation and Language
Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms
Kieran Henderson, Kian Omoomi, Vasudha Varadarajan et al. (+2 more)
Recent work has explored the use of personal information in the form of persona sentences or self-disclosures to improve modeling of individual characteristics and prediction of annotator labels for subjective tasks. The volume of personal information has historically been restricted and thus little exploration has gone into understanding what kind of information is most informative for predicting annotator labels. In this work, we categorize self-disclosures and use them to build annotator models for predicting judgments of social norms. We perform several ablations and analyses to examine the impact of the type of information on our ability to predict annotation patterns. Contrary to previous work, only a small number of comments related to the original post are needed. Lastly, a more diverse sample of annotator self-disclosures did not lead to the best performance. Sampling from a larger pool of comments without filtering still yields the best performance, suggesting that there is still much to uncover in terms of what information about an annotator is most useful for verdict prediction.
Read the full abstract ▾
All authors Kieran Henderson, Kian Omoomi, Vasudha Varadarajan, Allison Lahnala, Charles Welch
Computation and Language · Cross-listed from cs.CY
Cross-Language Bias Examination in Large Language Models
Yuxuan Liang and Marwa Mahmoud
This study introduces an innovative
multilingual bias evaluation framework for assessing bias in Large
Language Models, combining explicit bias assessment through the BBQ
benchmark with implicit bias measurement using a
prompt-based Implicit Association Test. By translating the
prompts and word list into five target
languages, English, Chinese, Arabic, French, and Spanish, we directly compare different types of bias across
languages. The results reveal substantial gaps in bias across
languages used in LLMs. For example, Arabic and Spanish consistently show higher levels of stereotype bias, while Chinese and English exhibit lower levels of bias. We also identify contrasting patterns across bias types. Age shows the lowest explicit bias but the highest implicit bias, emphasizing the importance of detecting implicit biases that are undetectable with standard
benchmarks. These findings indicate that LLMs vary significantly across
languages and bias dimensions. This study fills a key research gap by providing a comprehensive methodology for cross-lingual bias analysis. Ultimately, our work establishes a foundation for the development of equitable
multilingual LLMs, ensuring
fairness and effectiveness across diverse
languages and cultures.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Dynamic Rank Reinforcement Learning for Adaptive Low-Rank Multi-Head Self Attention in Large Language Models
Caner Erden
Dynamic Rank Reinforcement Learning (DR-RL) approximations rely on static rank assumptions, limiting their flexibility across diverse linguistic contexts. Our method dynamically modulates ranks based on real-time sequence dynamics, layer-specific sensitivities, and hardware constraints. The core innovation is a deep reinforcement learning
agent that formulates rank selection as a sequential policy optimization problem, strictly balancing
attention fidelity against computational latency. To ensure stability during inference, we derive and employ online
matrix perturbation bounds, enabling incremental rank updates without the prohibitive cost of full decomposition. Furthermore, the integration of a lightweight
Transformer-based policy network and batched Singular Value Decomposition (SVD) operations ensures scalable deployment on modern architectures. Extensive experiments demonstrate that DR-RL significantly reduces Floating Point Operations (FLOPs) by over 40% in long-sequence regimes (L > 4096) while maintaining downstream accuracy statistically equivalent to full-rank
attention. Beyond standard
language modeling
benchmarks, we validate the real-world applicability of DR-RL on the GLUE
benchmark. Specifically, our method achieves 92.78% accuracy on the SST-2 sentiment analysis task, matching the performance of full-rank baselines and outperforming static low-rank methods, such as Performer and Nyströmformer, by a significant margin.
Read the full abstract ▾
Computation and Language
BRAID: Bounded Reasoning for Autonomous Inference and Decisions
Armağan Amcalar and Eyup Cinar
Large
Language Models (LLMs) exhibit nonlinear relationships between performance, cost, and
token usage. This paper presents a quantitative study on structured prompting using BRAID (Bounded
Reasoning for Au tonomous Inference and Decisions) across multiple GPT model tiers, eval uated on the AdvancedIF, GSM-Hard, and the SCALE MultiChallenge
benchmark datasets. BRAID introduces a bounded
reasoning framework using Mermaid-based instruction graphs that enable models to reason struc turally rather than through unbounded natural-
language token expansion. We show that structured machine-readable
prompts substantially increase
reasoning accuracy and cost efficiency for
agents in production systems. The findings establish BRAID as an effective and scalable technique for optimizing inference efficiency in autonomous
agent systems. All
datasets and detailed result logs are available at https://
benchmark.openserv.ai.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
DSO: Direct Steering Optimization for Bias Mitigation
Lucas Monteiro Paes, Nivedha Sivakumar, Yinong Oliver Wang et al. (+4 more)
Generative models are often deployed to make decisions on behalf of users, such as
vision-
language models (VLMs) identifying which person in a room is a doctor to help visually impaired individuals. Yet, VLM decisions are influenced by the perceived demographic attributes of people in the input, which can lead to biased outcomes like failing to identify women as doctors. Moreover, when reducing bias leads to performance loss, users may have varying needs for balancing bias mitigation with overall model capabilities, highlighting the demand for methods that enable controllable bias reduction during inference. Activation steering is a popular approach for inference-time controllability that has shown potential in inducing safer behavior in large
language models (LLMs). However, we observe that current steering methods struggle to correct biases, where equiprobable outcomes across demographic groups are required. To address this, we propose Direct Steering Optimization (DSO) which uses reinforcement learning to find linear transformations for steering activations, tailored to mitigate bias while maintaining control over model performance. We demonstrate that DSO achieves state-of-the-art trade-off between
fairness and capabilities on both VLMs and LLMs, while offering practitioners inference-time control over the trade-off. Overall, our work highlights the benefit of designing steering strategies that are directly optimized to control model behavior, providing more effective bias intervention than methods that rely on pre-defined heuristics for controllability.
Read the full abstract ▾
All authors Lucas Monteiro Paes, Nivedha Sivakumar, Yinong Oliver Wang, Masha Fedzechkina, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff
Computation and Language
Social Story Frames: Contextual Reasoning about Narrative Intent and Reception
Joel Mire, Maria Antoniak, Steven R. Wilson et al. (+4 more)
Reading stories evokes rich interpretive, affective, and evaluative responses, such as inferences about narrative intent or judgments about characters. Yet, computational models of reader response are limited, preventing nuanced analyses. To address this gap, we introduce SocialStoryFrames, a formalism for distilling plausible inferences about reader response, such as perceived author intent, explanatory and predictive
reasoning, affective responses, and value judgments, using conversational context and a taxonomy grounded in narrative theory, linguistic pragmatics, and psychology. We develop two models, SSF-Generator and SSF-Classifier, validated through human surveys (N=382 participants) and expert annotations, respectively. We conduct pilot analyses to showcase the utility of the formalism for studying storytelling at scale. Specifically, applying our models to SSF-Corpus, a curated
dataset of 6,140 social media stories from diverse contexts, we characterize the frequency and interdependence of storytelling intents, and we compare and contrast narrative practices (and their diversity) across communities. By linking fine-grained, context-sensitive modeling with a generic taxonomy of reader responses, SocialStoryFrames enable new research into storytelling in online communities.
Read the full abstract ▾
All authors Joel Mire, Maria Antoniak, Steven R. Wilson, Zexin Ma, Achyutarama R. Ganti, Andrew Piper, Maarten Sap
Computation and Language
TabReX : Tabular Referenceless eXplainable Evaluation
Tejas Anvekar, Junha Park, Aparna Garimella et al. (+1 more)
Evaluating the quality of tables generated by large
language models (LLMs) remains an open challenge: existing metrics either flatten tables into text, ignoring structure, or rely on fixed references that limit generalization. We present TabReX, a reference-less, property-driven framework for evaluating tabular generation via graph-based
reasoning. TabReX converts both source text and generated tables into canonical knowledge graphs, aligns them through an LLM-guided matching process, and computes interpretable, rubric-aware scores that quantify structural and factual fidelity. The resulting metric provides controllable trade-offs between sensitivity and specificity, yielding human-aligned judgments and
cell-level error traces. To systematically asses metric robustness, we introduce TabReX-Bench, a large-scale
benchmark spanning six domains and twelve planner-driven perturbation types across three difficulty tiers. Empirical results show that TabReX achieves the highest correlation with expert rankings, remains stable under harder perturbations, and enables fine-grained model-vs-
prompt analysis establishing a new paradigm for trustworthy, explainable evaluation of structured generation systems.
Read the full abstract ▾
All authors Tejas Anvekar, Junha Park, Aparna Garimella, Vivek Gupta
Computation and Language · Cross-listed from cs.CV
Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
Davide Caffagni, Sara Sarto, Marcella Cornia et al. (+5 more)
Multimodal Large
Language Models (MLLMs) have recently demonstrated impressive capabilities in connecting
vision and
language, yet their proficiency in fundamental visual
reasoning tasks remains limited. This limitation can be attributed to the fact that MLLMs learn visual understanding primarily from textual descriptions, which constitute a subjective and inherently incomplete supervisory signal. Furthermore, the modest scale of
multimodal instruction tuning compared to massive text-only pre-training leads MLLMs to overfit
language priors while overlooking visual details. To address these issues, we introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs. Specifically, we integrate the I-JEPA learning paradigm into the standard
vision-
language alignment pipeline of MLLMs training. Our approach leverages frozen
vision foundation models as context and target encoders, while training the predictor, implemented as the early layers of an LLM, to learn structural and semantic regularities from images without relying exclusively on
language supervision. Extensive experiments on standard MLLM
benchmarks show that JARVIS consistently improves performance on
vision-centric
benchmarks across different LLM families, without degrading
multimodal reasoning abilities. Our source code is publicly available at: https://github.com/aimagelab/JARVIS.
Read the full abstract ▾
All authors Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Pier Luigi Dovesi, Shaghayegh Roohi, Mark Granroth-Wilding, Rita Cucchiara
Computation and Language · Cross-listed from cs.SD
From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining
Linnea Evanson, Mingfang Zhang, Hubert Banville et al. (+3 more)
Decoding
speech from
brain activity has typically relied on limited neural recordings collected during short and highly controlled experiments. Here, we introduce a framework to leverage week-long intracranial and audio recordings from patients undergoing clinical monitoring, effectively increasing the training
dataset size by over two orders of magnitude. With this pretraining, our contrastive learning model substantially outperforms models trained solely on classic experimental data, with gains that scale log-linearly with
dataset size. Analysis of the learned representations reveals that, while
brain activity represents
speech features, its global structure largely drifts across days, highlighting the need for models that explicitly account for cross-day variability. Overall, our approach opens a scalable path toward decoding and modeling
brain representations in both real-life and controlled task settings.
Read the full abstract ▾
All authors Linnea Evanson, Mingfang Zhang, Hubert Banville, Saarang Panchavati, Pierre Bourdillon, Jean-Rémi King
Computation and Language · Cross-listed from cs.AI
Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants
Vincent Huang, Dami Choi, Daniel D. Johnson et al. (+2 more)
Interpreting the internal activations of neural networks can produce more faithful explanations of their behavior, but is difficult due to the complex structure of activation space. Existing approaches to scalable interpretability use hand-designed
agents that make and test hypotheses about how internal activations relate to external behavior. We propose to instead turn this task into an end-to-end training objective, by training interpretability assistants to accurately predict model behavior from activations through a communication bottleneck. Specifically, an encoder compresses activations to a sparse list of concepts, and a decoder reads this list and answers a natural
language question about the model. We show how to pretrain this assistant on large unstructured data, then finetune it to answer questions. The resulting architecture, which we call a Predictive Concept Decoder, enjoys favorable scaling properties: the auto-interp score of the bottleneck concepts improves with data, as does the performance on downstream applications. Specifically, PCDs can detect jailbreaks, secret hints, and implanted latent concepts, and are able to accurately surface latent user attributes.
Read the full abstract ▾
All authors Vincent Huang, Dami Choi, Daniel D. Johnson, Sarah Schwettmann, Jacob Steinhardt
Computation and Language
Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers
Adam Karvonen, James Chua, Clément Dumas et al. (+8 more)
Large
language model (LLM) activations are notoriously difficult to understand, with most existing techniques using complex, specialized methods for interpreting them. Recent work has proposed a simpler approach known as LatentQA: training LLMs to directly accept LLM activations as inputs and answer arbitrary questions about them in natural
language. However, prior work has focused on narrow task settings for both training and evaluation. In this paper, we instead take a generalist perspective. We evaluate LatentQA-trained models, which we call Activation Oracles (AOs), in far out-of-distribution settings and examine how performance scales with training data diversity. We find that AOs can recover information fine-tuned into a model (e.g., biographical knowledge or malign propensities) that does not appear in the input text, despite never being trained with activations from a fine-tuned model. Our main evaluations are four downstream tasks where we can compare to prior white- and black-box techniques. We find that even narrowly-trained LatentQA models can generalize well, and that adding additional training
datasets (such as classification tasks and a self-supervised context prediction task) yields consistent further improvements. Our best AOs match or exceed white-box baselines on all four tasks and the best overall baseline on 3 of 4. These results suggest that diversified training to answer natural-
language queries imparts a general capability to verbalize information about LLM activations.
Read the full abstract ▾
All authors Adam Karvonen, James Chua, Clément Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, Samuel Marks
Computation and Language · Cross-listed from cs.AI
Explaining the Reasoning of Large Language Models Using Attribution Graphs
Chase Walker and Rickard Ewetz
Large
language models (LLMs) exhibit remarkable capabilities, yet their
reasoning remains opaque, raising safety and trust concerns. Attribution methods, which assign credit to input features, have proven effective for explaining the decision making of computer
vision models. From these, context attributions have emerged as a promising approach for explaining the behavior of autoregressive LLMs. However, current context attributions produce incomplete explanations by directly relating generated
tokens to the
prompt, discarding inter-generational influence in the process. To overcome these shortcomings, we introduce the Context Attribution via Graph Explanations (CAGE) framework. CAGE introduces an attribution graph: a directed graph that quantifies how each generation is influenced by both the
prompt and all prior generations. The graph is constructed to preserve two properties-causality and row stochasticity. The attribution graph allows context attributions to be computed by marginalizing intermediate contributions along paths in the graph. Across multiple models,
datasets, metrics, and methods, CAGE improves context attribution faithfulness, achieving average gains of up to 40%.
Read the full abstract ▾
Computation and Language
PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learning
Xiaodi Li, Dingcheng Li, Rujun Gao et al. (+3 more)
Continual learning remains a fundamental challenge in machine learning, requiring models to learn from a stream of tasks without forgetting previously acquired knowledge. A major obstacle in this setting is catastrophic forgetting, where performance on earlier tasks degrades as new tasks are learned. In this paper, we introduce PPSEBM, a novel framework that integrates an Energy-Based Model (EBM) with Progressive Parameter Selection (PPS) to effectively address catastrophic forgetting in continual learning for natural
language processing tasks. In PPSEBM, progressive parameter selection allocates distinct, task-specific parameters for each new task, while the EBM generates representative pseudo-samples from prior tasks. These generated samples actively inform and guide the parameter selection process, enhancing the model's ability to retain past knowledge while adapting to new tasks. Experimental results on diverse NLP
benchmarks demonstrate that PPSEBM outperforms state-of-the-art continual learning methods, offering a promising and robust solution to mitigate catastrophic forgetting.
Read the full abstract ▾
All authors Xiaodi Li, Dingcheng Li, Rujun Gao, Mahmoud Zamani, Feng Mi, Latifur Khan
Computation and Language
Characterizing Mamba's Selective Memory using Auto-Encoders
Tamanna Hossain, Robert L. Logan IV, Ganesh Jagadeesan et al. (+3 more)
State space models (SSMs) are a promising alternative to
transformers for
language modeling because they use fixed
memory during inference. However, this fixed
memory usage requires some information loss in the hidden state when processing long sequences. While prior work has studied the sequence length at which this information loss occurs, it does not characterize the types of information SSM
language models (LMs) tend to forget. In this paper, we address this knowledge gap by identifying the types of
tokens (e.g., parts of
speech, named entities) and sequences (e.g., code, math problems) that are more frequently forgotten by SSM LMs. We achieve this by training an auto-encoder to reconstruct sequences from the SSM's hidden state, and measure information loss by comparing inputs with their reconstructions. We perform experiments using the Mamba family of SSM LMs (130M–1.4B) on sequences ranging from 4–256
tokens. Our results show significantly higher rates of information loss on math-related
tokens (e.g., numbers, variables), mentions of organization entities, and alternative dialects to Standard American English. We then examine the frequency that these
tokens appear in Mamba's pretraining data and find that less prevalent
tokens tend to be the ones Mamba is most likely to forget. By identifying these patterns, our work provides clear direction for future research to develop methods that better control Mamba's ability to retain important information.
Read the full abstract ▾
All authors Tamanna Hossain, Robert L. Logan IV, Ganesh Jagadeesan, Sameer Singh, Joel Tetreault, Alejandro Jaimes
Computation and Language · Cross-listed from cs.CV
VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?
Hongbo Zhao, Meng Wang, Fei Zhu et al. (+5 more)
The computational and
memory overheads associated with expanding the context window of LLMs severely limit their scalability. A noteworthy solution is
vision-text compression (VTC), exemplified by frameworks like DeepSeek-OCR and Glyph, which convert long texts into dense 2D visual representations, thereby achieving
token compression ratios of 3x-20x. However, the impact of this high information density on the core long-context capabilities of
vision-
language models (VLMs) remains under-investigated. To address this gap, we introduce the first
benchmark for VTC and systematically assess the performance of VLMs across three long-context understanding settings: VTC-
Retrieval, which evaluates the model's ability to retrieve and aggregate information; VTC-
Reasoning, which requires models to infer latent associations to locate facts with minimal lexical overlap; and VTC-
Memory, which measures comprehensive question answering within long-term dialogue
memory. Furthermore, we establish the VTCBench-Wild to simulate diverse input scenarios.We comprehensively evaluate leading open-source and proprietary models on our
benchmarks. The results indicate that, despite being able to decode textual information (e.g., OCR) well, most VLMs exhibit a surprisingly poor long-context understanding ability with VTC-processed information, failing to capture long associations or dependencies in the context.This study provides a deep understanding of VTC and serves as a foundation for designing more efficient and scalable VLMs.
Read the full abstract ▾
All authors Hongbo Zhao, Meng Wang, Fei Zhu, Wenzhuo Liu, Bolin Ni, Fanhu Zeng, Gaofeng Meng, Zhaoxiang Zhang
Computation and Language
How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
Darshita Rathore, Vineet Kumar, Chetna Bansal et al. (+1 more)
Large
language models are increasingly adapted to downstream tasks through fine-tuning. Full supervised fine-tuning (SFT) and parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), are two dominant approaches. While PEFT methods are widely used for their computational efficiency, the implications of their configurations (e.g., rank) remain under-explored in downstream Q&A tasks and generalisation. In this work, we perform a comprehensive evaluation across multiple
reasoning and recall
datasets, conducting a rank sweep to quantify the trade-off between SFT and PEFT. We also compare the accuracy of PEFT and SFT models across in-domain and out-of-domain adaptation, highlighting distinct generalisation behaviour and task-specific forgetting. We demonstrate that LoRA achieves competitive and in some cases superior performance compared to SFT, particularly on
reasoning tasks at specific rank values. Additionally, we analyze the internal representations via spectral features and layer-wise
attention structures, offering insights into representational drift and structural changes in
attention patterns.
Read the full abstract ▾
All authors Darshita Rathore, Vineet Kumar, Chetna Bansal, Anindya Moitra
Computation and Language
Evaluating Metrics for Safety with LLM-as-Judges
Kester Clegg, Richard Hawkins, Ibrahim Habli et al. (+1 more)
LLMs (Large
Language Models) are increasingly used in text processing pipelines to intelligently respond to a variety of inputs and generation tasks. This raises the possibility of replacing human roles that bottleneck existing information flows, either due to insufficient staff or process complexity. However, LLMs make mistakes and some processing roles are safety critical. For example, triaging post-operative care to patients based on hospital referral letters, or updating site access schedules in
nuclear facilities for work crews. If we want to introduce LLMs into critical information flows that were previously performed by humans, how can we make them safe and reliable? Rather than make performative claims about augmented generation frameworks or graph-based techniques, this paper argues that the safety argument should focus on the type of evidence we get from evaluation points in LLM processes, particularly in frameworks that employ LLM-as-Judges (LaJ) evaluators. This paper argues that although we cannot get deterministic evaluations from many natural
language processing tasks, by adopting a basket of weighted metrics it may be possible to lower the risk of errors within an evaluation, use context sensitivity to define error severity and design confidence thresholds that trigger human review of critical LaJ judgments when concordance across evaluators is low.
Read the full abstract ▾
All authors Kester Clegg, Richard Hawkins, Ibrahim Habli, Tom Lawton
Computation and Language
You Never Know a Person, You Only Know Their Defenses: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations
Hongbin Na, Zimu Wang, Zhaoming Chen et al. (+9 more)
Psychological defenses are strategies, often automatic, that people use to manage distress. Rigid or overuse of defenses is negatively linked to mental health and shapes what speakers disclose and how they accept or resist help. However, defenses are complex and difficult to reliably measure, particularly in clinical dialogues. We introduce PsyDefConv, a dialogue corpus with help seeker utterances labeled for defense level, and DMRS Co-Pilot, a four-stage pipeline that provides evidence-based pre-annotations. The corpus contains 200 dialogues and 4709 utterances, including 2336 help seeker turns, with labeling and Cohen's kappa 0.639. In a counterbalanced study, the co-pilot reduced average annotation time by 22.4%. In expert review, it averaged 4.62 for evidence, 4.44 for clinical plausibility, and 4.40 for insight on a seven-point scale.
Benchmarks with strong
language models in zero-shot and fine-tuning settings demonstrate clear headroom, with the best macro F1-score around 30% and a tendency to overpredict mature defenses. Corpus analyses confirm that mature defenses are most common and reveal
emotion-specific deviations. We will release the corpus, annotations, code, and
prompts to support research on defensive functioning in
language.
Read the full abstract ▾
All authors Hongbin Na, Zimu Wang, Zhaoming Chen, Peilin Zhou, Yining Hua, Grace Ziqi Zhou, Haiyang Zhang, Tao Shen, Wei Wang, John Torous, Shaoxiong Ji, Ling Chen
Computation and Language
Bolmo: Byteifying the Next Generation of Language Models
Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz et al. (+6 more)
Recent advances in generative AI have been largely driven by large
language models (LLMs), deep neural networks that operate over discrete units called
tokens. To represent text, the vast majority of LLMs use words or word fragments as the
tokens, known as subword tokenization. Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data - such as computer code or biological sequences - where meaning depends on the individual characters. Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance. Here we introduce Bolmo, a family of fully open byte-level LLMs that approach the capabilities of subword-based systems. Using a two-stage conversion procedure, we transform existing subword-based models into byte-level models with minimal additional training. The resulting models outperform prior byte-level approaches and excel on character-level
reasoning tasks, while remaining competitive across standard
benchmarks. By efficiently processing byte-level information, these models achieve practical inference speeds and can be adapted at low cost using the existing ecosystem around the source LLM. Our results remove a long-standing performance barrier to end-to-end byte-level
language modeling, demonstrating that models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding.
Read the full abstract ▾
All authors Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz, Anna Korhonen, Luke Zettlemoyer, Noah A. Smith, Edoardo M. Ponti, Luca Soldaini, Valentin Hofmann
Computation and Language
An Empirical Study on Chinese Character Decomposition in Multiword Expression-Aware Neural Machine Translation
Lifeng Han, Gareth J. F. Jones and Alan F. Smeaton
Word meaning, representation, and interpretation play fundamental roles in natural
language understanding (NLU), natural
language processing (NLP), and natural
language generation (NLG) tasks. Many of the inherent difficulties in these tasks stem from Multi-word Expressions (MWEs), which complicate the tasks by introducing ambiguity, idiomatic expressions, infrequent usage, and a wide range of variations. Significant effort and substantial progress have been made in addressing the challenging nature of MWEs in Western
languages, particularly English. This progress is attributed in part to the well-established research communities and the abundant availability of computational resources. However, the same level of progress is not true for
language families such as Chinese and closely related Asian
languages, which continue to lag behind in this regard. While sub-word modelling has been successfully applied to many Western
languages to address rare words improving phrase comprehension, and enhancing machine translation (MT) through techniques like byte-pair encoding (BPE), it cannot be applied directly to ideograph
language scripts like Chinese. In this work, we conduct a systematic study of the Chinese character decomposition technology in the context of MWE-aware neural machine translation (NMT). Furthermore, we report experiments to examine how Chinese character decomposition technology contributes to the representation of the original meanings of Chinese words and characters, and how it can effectively address the challenges of translating MWEs.
Read the full abstract ▾
Computation and Language
Automated Lexical Coverage for Language Learning: From General to Specialized Word Lists
Dakota Ellis, Samy Babikerali, Wanshan Chen et al. (+2 more)
A General Service List (GSL) is a commonly used resource for
language learners to identify important English words. Traditional GSL creation is resource-intensive, relying on linguistic expertise and subjective input. We created our own GSL and evaluated its performance against the New General Service List (NGSL). We found that creating a Specialized Word List (SWL), tailored to a specific text, is a practical method for
language learners. Because an SWL is derived from the target text itself, it reaches the 95% coverage required for
language comprehension by construction, and it does so with substantially fewer words than a general list applied to the same text: across nine texts spanning fiction, academic papers, and scripts, the NGSL covered 64-85% of each text, whereas a text-specific list reached 95% with far smaller vocabularies. By restricting the SWL process to objective criteria only, it can be automated, scaled, and tailored to the needs of
language-learners across the globe.
Read the full abstract ▾
All authors Dakota Ellis, Samy Babikerali, Wanshan Chen, Bao Dinh, Uyen Le
Computation and Language
Learning inflection classes using Adaptive Resonance Theory
Peter Dekker, Heikki Rasilo and Bart de Boer
The concept of inflection classes is an abstraction used by linguists, and provides a means to describe patterns in
languages that give an analogical base for deducing previously unencountered forms. This ability is an important part of morphological acquisition and processing. We study the learnability of a system of verbal inflection classes by the individual
language user by performing unsupervised clustering of lexemes into inflection classes. As a cognitively plausible and interpretable computational model, we use Adaptive Resonance Theory, a neural network with a parameter that determines the degree of generalisation (vigilance). The model is applied to Latin, Portuguese and Estonian. The similarity of clustering to attested inflection classes varies depending on the complexity of the inflectional system. We find the best performance in a narrow region of the generalisation parameter. The learned features extracted from the model show similarity with linguistic descriptions of the inflection classes. The proposed model could be used to study change in inflection classes in the future, by including it in an
agent-based model.
Read the full abstract ▾
Computation and Language
CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing
Kuan Lu, Shuhang Lin, Sai Wu et al. (+7 more)
Large
language models (LLMs) are increasingly applied in long-context scenarios such as multi-turn conversations. However, long contexts pose significant challenges for inference efficiency, including high
memory overhead from Key-Value (KV) cache and increased latency due to excessive
memory accesses. Recent methods for dynamic KV selection struggle with trade-offs: block-level indexing degrades accuracy by retrieving irrelevant KV entries, while
token-level indexing incurs high latency from inefficient
retrieval mechanisms. In this paper, we propose CTKVR, a novel centroid-then-
token KV
retrieval scheme that addresses these limitations. CTKVR leverages a key observation: query vectors adjacent in position exhibit high similarity after Rotary Position
Embedding (RoPE) and share most of their top-k KV cache entries. Based on this insight, CTKVR employs a two-stage
retrieval strategy: lightweight centroids are precomputed during prefilling for centroid-grained indexing, followed by
token-level refinement for precise KV
retrieval. This approach balances
retrieval efficiency and accuracy. To further enhance performance, we implement an optimized system for indexing construction and search using CPU-GPU co-execution. Experimentally, CTKVR achieves superior performance across multiple
benchmarks with less than 1% accuracy degradation. Meanwhile, CTKVR delivers 3 times and 4 times throughput speedups on Llama-3-8B and Yi-9B at 96K context length across diverse GPU hardware.
Read the full abstract ▾
All authors Kuan Lu, Shuhang Lin, Sai Wu, Yichen Yao, Junhan Yang, Huan Li, Wei Chu, Xu Yinghui, Yuan Qi, Gang Chen
Computation and Language
When a Nation Speaks: Machine Learning and NLP in People's Sentiment Analysis During Bangladesh's 2024 Mass Uprising
Md. Samiul Alim, Mahir Shahriar Tamim, Maisha Rahman et al. (+2 more)
Sentiment analysis, an emerging research area within natural
language processing (NLP), has primarily been explored in contexts like elections and social media trends, but there remains a significant gap in understanding emotional dynamics during civil unrest, particularly in the Bangla
language. Our study pioneers sentiment analysis in Bangla during a national crisis by examining public
emotions amid Bangladesh's 2024 mass uprising. We curated a unique
dataset of 2,028 annotated news headlines from major Facebook news portals, classifying them into Outrage, Hope, and Despair. Through Latent Dirichlet Allocation (LDA), we identified prevalent themes like political corruption and public protests, and analyzed how events such as internet blackouts shaped sentiment patterns. It outperformed
multilingual transformers (mBERT: 67%, XLM-RoBERTa: 71%) and traditional machine learning methods (SVM and Logistic Regression: both 70%). These results highlight the effectiveness of
language-specific models and offer valuable insights into public sentiment during political turmoil.
Read the full abstract ▾
All authors Md. Samiul Alim, Mahir Shahriar Tamim, Maisha Rahman, Tanvir Ahmed Khan, Md Mushfique Anwar
Computation and Language · Cross-listed from cs.LG
Tracking Temporal Dynamics of Vector Sets with Gaussian Process
Taichi Aida, Mamoru Komachi, Toshinobu Ogiso et al. (+2 more)
Understanding the temporal
evolution of sets of vectors is a fundamental challenge across various domains, including ecology, crime analysis, and linguistics. For instance, ecosystem structures evolve due to interactions among plants, herbivores, and carnivores; the spatial distribution of crimes shifts in response to societal changes; and word
embedding vectors reflect cultural and semantic trends over time. However, analyzing such time-varying sets of vectors is challenging due to their complicated structures, which also evolve over time. In this work, we propose a novel method for modeling the distribution underlying each set of vectors using infinite-dimensional Gaussian processes. By approximating the latent function in the Gaussian process with Random
Fourier Features, we obtain compact and comparable vector representations over time. This enables us to track and visualize temporal transitions of vector sets in a low-dimensional space. We apply our method to both sociological data (crime distributions) and linguistic data (word
embeddings), demonstrating its effectiveness in capturing temporal dynamics. Our results show that the proposed approach provides interpretable and robust representations, offering a powerful framework for analyzing structural changes in temporally indexed vector sets across diverse domains.
Read the full abstract ▾
All authors Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hiroya Takamura, Daichi Mochihashi
Computation and Language
Toward expert-level motivational interviewing for health behavior improvement with LLMs
Run-ze Hu, Yang Yang, Yi-hang Yang et al. (+8 more)
Background: Motivational interviewing (MI) is an effective counseling approach for promoting health behavior change, but its impact is constrained by the need for highly trained human counselors. Objective: This study aimed to explore a scalable alternative by developing and evaluating Large
Language Models for Motivational Interviewing (MI-LLMs). Methods: We first curated five Chinese psychological counseling corpora and, using GPT-4 with an MI-informed
prompt, transcribed multi-turn dialogues from the two highest-quality
datasets (CPsyCounD and PsyDTCorpus) into 2,040 MI-style counseling conversations, of which 2,000 were used for training and 40 for testing. Three Chinese-capable open-source LLMs (Baichuan2-7B-Chat, ChatGLM-4-9B-Chat and Llama-3-8B-Chinese-Chat-v2) were fine-tuned on this corpus and were named as MI-LLMs. We evaluated MI-LLMs using round-based automatic metrics and expert manual coding with the Motivational Interviewing Treatment Integrity (MITI) Coding Manual 4.2.1. Results: Across all three models, fine-tuning substantially improved BLEU-4 and ROUGE scores compared with the base models, and manual coding showed that MI-LLMs achieved technical and relational global scores, and MI-adherent ratios that approached those of real MI dialogues, although complex reflections and reflection-to-question ratios remained less frequent. Conclusions: These findings provide initial evidence that MI-oriented fine-tuning can endow general-purpose LLMs with core MI-consistent counseling behaviors, suggesting a scalable pathway toward AI-assisted health behavior change support while underscoring the need for further work on data scale, complex MI skills and real-world intervention trials.
Read the full abstract ▾
All authors Run-ze Hu, Yang Yang, Yi-hang Yang, Jing-qi Kong, Jia-hui Luo, Wen-yu Yang, Jing Chen, Jing-yao Liu, Hui-qun Zeng, Lei Zhang, Zheng Liu
Computation and Language
ORACLE: Time-Dependent Recursive Summary Graphs for Foresight on News Data Using LLMs
Lev Kharlashkin, Eiaki Morooka, Yehor Tereshchenko et al. (+1 more)
ORACLE turns daily news into week-over-week, decision-ready insights for one of the Finnish University of Applied Sciences. The platform crawls and versions news, applies University-specific relevance filtering, embeds content, classifies items into PESTEL dimensions and builds a concise Time-Dependent Recursive Summary Graph (TRSG): two clustering layers summarized by an LLM and recomputed weekly. A lightweight change detector highlights what is new, removed or changed, then groups differences into themes for PESTEL-aware analysis. We detail the pipeline, discuss concrete design choices that make the system stable in production and present a curriculum-intelligence use case with an evaluation plan.
Read the full abstract ▾
All authors Lev Kharlashkin, Eiaki Morooka, Yehor Tereshchenko, Mika Hämäläinen
Computation and Language · Cross-listed from cs.CV
Emotion Recognition in Signers
Kotaro Funakoshi and Yaoxiong Zhu
Recognition of signers'
emotions suffers from one theoretical challenge and one practical challenge, namely, the overlap between grammatical and affective facial expressions and the scarcity of data for model training. This paper addresses these two challenges in a cross-lingual setting using our eJSL
dataset, a new
benchmark dataset for
emotion recognition in Japanese Sign
Language signers, and BOBSL, a large British Sign
Language dataset with subtitles. In eJSL, two signers expressed 78 distinct utterances with each of seven different emotional states, resulting in 1,092
video clips. We empirically demonstrate that 1) textual
emotion recognition in spoken
language mitigates data scarcity in sign
language, 2) temporal segment selection has a significant impact, and 3) incorporating hand motion enhances
emotion recognition in signers. Finally we establish a stronger baseline than spoken
language LLMs.
Read the full abstract ▾