Computation and Language
When Distributions Shifts: Causal Generalization for Low-Resource Languages
Mahi Aliyu Aminu, Chisom Chibuike, Fatimo Adebanjo et al. (+2 more)
Machine learning models often fail under distribution shifts, a problem exacerbated in low-resource settings where limited data restricts robust generalization. Domain generalization(DG) methods address this challenge by learning representations that remain invariant across domains, frequently leveraging causal principles. In this work, we study two causal DG approaches for low-resource natural
language processing. First, we apply causal data augmentation using GPT-4o-mini to generate counterfactual paraphrases for sentiment classification on the NaijaSenti Twitter corpus in Yoruba and Igbo. Second, we investigate invariant causal representation learning with the Debiasing in Aspect Review (DINER) framework for aspect-based sentiment analysis. We extend DINER to a
multilingual setting by introducing Afri-SemEval, a
dataset of 17
languages translated from SemEval-2014 Task. Experiments show improved robustness to unseen domains, with consistent gains from counterfactual augmentation and enhanced out-of-distribution performance from causal representation learning across multiple
languages.
Read the full abstract ▾
All authors Mahi Aliyu Aminu, Chisom Chibuike, Fatimo Adebanjo, Omokolade Awosanya, Samuel Oyeneye
Computation and Language · Cross-listed from cs.LG
Thought Branches: Interpreting LLM Reasoning Requires Resampling
Uzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan et al. (+1 more)
Most work interpreting
reasoning models studies only a single chain-of-thought (CoT), yet these models define distributions over many possible CoTs. We argue that studying a single sample is inadequate for understanding causal influence and the underlying computation. Though fully specifying this distribution is intractable, we can measure a partial CoT's impact by resampling only the subsequent text. We present case studies using resampling to investigate model decisions. First, when a model states a reason for its action, does that reason actually cause the action? In "agentic misalignment" scenarios, we find that self-preservation sentences have small causal impact, suggesting they do not meaningfully drive blackmail. Second, are artificial edits to CoT sufficient for steering
reasoning? Resampling and selecting a completion with the desired property is a principled on-policy alternative. We find that off-policy interventions yield small and unstable effects compared to resampling in decision-making tasks. Third, how do we understand the effect of removing a
reasoning step when the model may repeat it post-edit? We introduce a resilience metric that repeatedly resamples to prevent similar content from reappearing downstream. Critical planning statements resist removal but have large effects when eliminated. Fourth, since CoT is sometimes "unfaithful", can our methods teach us anything in these settings? Adapting causal mediation analysis, we find that hints that causally affect the output without being explicitly mentioned exert a subtle and cumulative influence on the CoT that persists even if the hint is removed. Overall, studying distributions via resampling enables reliable causal analysis, clearer narratives of model
reasoning, and principled CoT interventions.
Read the full abstract ▾
All authors Uzay Macar, Paul C. Bogdan, Senthooran Rajamanoharan, Neel Nanda
Computation and Language
The aftermath of compounds: Investigating Compounds and their Semantic Representations
Swarang Joshi
This study investigates how well computational
embeddings align with human semantic judgments in the processing of English compound words. We compare static word vectors (GloVe) and contextualized
embeddings (BERT) against human ratings of lexeme meaning dominance (LMD) and semantic transparency (ST) drawn from a psycholinguistic
dataset. Using measures of association strength (Edinburgh Associative Thesaurus), frequency (BNC), and predictability (LaDEC), we compute
embedding-derived LMD and ST metrics and assess their relationships with human judgments via Spearmans correlation and regression analyses. Our results show that BERT
embeddings better capture compositional semantics than GloVe, and that predictability ratings are strong predictors of semantic transparency in both human and model data. These findings advance computational psycholinguistics by clarifying the factors that drive compound word processing and offering insights into
embedding-based semantic modeling.
Read the full abstract ▾
Computation and Language
Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning
Chenyang Shao, Sijian Ren, Fengli Xu et al. (+1 more)
In recent years, large
language models (LLMs) have witnessed remarkable advancements, with the test-time scaling law consistently enhancing the
reasoning capabilities. Through systematic evaluation and exploration of a diverse spectrum of intermediate thoughts, LLMs demonstrate the potential to generate deliberate
reasoning steps, thereby substantially enhancing
reasoning accuracy. However, LLMs' autoregressive generation paradigm results in
reasoning performance scaling sub-optimally with test-time computation, often requiring excessive computational overhead to propose thoughts while yielding only marginal performance gains. In contrast,
diffusion language models (DLMs) can efficiently produce diverse samples through parallel denoising in a single forward pass, inspiring us to leverage them for proposing intermediate thoughts, thereby alleviating the computational burden associated with autoregressive generation while maintaining quality. In this work, we propose an efficient collaborative
reasoning framework, leveraging DLMs to generate candidate thoughts and LLMs to evaluate their quality. Experiments across diverse
benchmarks demonstrate that our framework achieves strong performance in complex
reasoning tasks, offering a promising direction for future research. Our code is open-source at https://anonymous.4open.science/r/Diffuse-Thinking-EC60.
Read the full abstract ▾
All authors Chenyang Shao, Sijian Ren, Fengli Xu, Yong Li
Computation and Language
VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
Xuan Gong, Senmiao Wang, Hanbo Huang et al. (+2 more)
Supervised fine-tuning (SFT) on long chain-of-thought (CoT) trajectories has emerged as a crucial technique for enhancing the
reasoning abilities of large
language models (LLMs). However, the standard cross-
entropy loss treats all
tokens equally, ignoring their heterogeneous contributions across a
reasoning trajectory. This uniform treatment leads to misallocated supervision and weak generalization, especially in complex, long-form
reasoning tasks. To address this, we introduce Variance-Controlled Optimization-based REweighting (VCORE), a principled framework that reformulates CoT supervision as a constrained optimization problem. By adopting an optimization-theoretic perspective, VCORE enables a principled and adaptive allocation of supervision across
tokens, thereby aligning the training objective more closely with the goal of robust
reasoning generalization. Empirical evaluations demonstrate that VCORE achieves the strongest overall average performance, with especially clear gains on lower-capacity models. Across both in-domain and out-of-domain settings, VCORE achieves substantial performance gains on mathematical and coding
benchmarks, using models from the Qwen3 series (4B, 8B, 32B) and LLaMA-3.1-8B-Instruct. Moreover, we show that VCORE serves as a more effective initialization for subsequent reinforcement learning, establishing a stronger foundation for advancing the
reasoning capabilities of LLMs. The Code will be released at https://github.com/coder-gx/VCORE.
Read the full abstract ▾
All authors Xuan Gong, Senmiao Wang, Hanbo Huang, Ruoyu Sun, Shiyu Liang
Computation and Language · Cross-listed from cs.AI
DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
Tian Liang, Wenxiang Jiao, Zhiwei He et al. (+3 more)
Large
Reasoning Models (LRMs) have demonstrated impressive capabilities but suffer from cognitive inefficiencies like "overthinking" simple problems and "underthinking" complex ones. While existing methods that use supervised fine-tuning (SFT) or reinforcement learning (RL) with
token-length rewards can improve efficiency, they often do so at the cost of accuracy. This paper introduces DeepCompress, a novel framework that simultaneously enhances both the accuracy and efficiency of LRMs. We challenge the prevailing approach of consistently favoring shorter
reasoning paths, showing that longer responses can contain a broader range of correct solutions for difficult problems. DeepCompress employs an adaptive length reward mechanism that dynamically classifies problems as "Simple" or "Hard" in real-time based on the model's evolving capability. It encourages shorter, more efficient
reasoning for "Simple" problems while promoting longer, more exploratory thought chains for "Hard" problems. This dual-reward strategy enables the model to autonomously adjust its Chain-of-Thought (CoT) length, compressing
reasoning for well-mastered problems and extending it for those it finds challenging. Experimental results on challenging mathematical
benchmarks show that DeepCompress consistently outperforms baseline methods, achieving superior accuracy while significantly improving
token efficiency.
Read the full abstract ▾
All authors Tian Liang, Wenxiang Jiao, Zhiwei He, Jiahao Xu, Haitao Mi, Dong Yu
Computation and Language
Dynamic Affective Memory Management for Personalized LLM Agents
Junfeng Lu and Yueyan Li
Advances in large
language models are making personalized AI
agents a new research focus. While current
agent systems primarily rely on personalized external
memory databases to deliver customized experiences, they face challenges such as
memory redundancy,
memory staleness, and poor
memory-context integration, largely due to the lack of effective
memory updates during interaction. To tackle these issues, we propose a new
memory management system designed for affective scenarios. Our approach employs a
Bayesian-inspired
memory update
algorithm with the concept of
memory entropy, enabling the
agent to autonomously maintain a dynamically updated
memory vector database by minimizing global
entropy to provide more personalized services. To better evaluate the system's effectiveness in this context, we propose DABench, a
benchmark focusing on emotional expression and emotional change toward objects. Experimental results demonstrate that, our system achieves superior performance in personalization, logical coherence, and accuracy. Ablation studies further validate the effectiveness of the
Bayesian-inspired update mechanism in alleviating
memory bloat. Our work offers new insights into the design of long-term
memory systems.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Atlas-Alignment: Making Interpretability Transferable Across Language Models
Bruno Puri, Jim Berend, Sebastian Lapuschkin et al. (+1 more)
Interpretability is crucial for building safe, reliable, and controllable
language models, yet existing interpretability pipelines remain costly and difficult to scale. Interpreting a new model typically requires training model-specific components (e.g., sparse autoencoders), followed by manual or semi-automated labeling and validation, imposing a growing "transparency tax" that does not scale with the pace of model development. We introduce Atlas-
Alignment, a framework that avoids this cost by aligning the latent space of a new model to a pre-existing, labeled Concept Atlas using only shared inputs and lightweight representational
alignment methods. Through quantitative and qualitative evaluations, we show that simple
alignment methods enable robust semantic
retrieval and steerable generation without the need for labeled concept
datasets. Atlas-
Alignment thus amortizes the cost of explainable AI and mechanistic interpretability: by investing in a single high-quality Concept Atlas, we can make many new models transparent and controllable at minimal marginal cost.
Read the full abstract ▾
All authors Bruno Puri, Jim Berend, Sebastian Lapuschkin, Wojciech Samek
Computation and Language
Awal – Community-Powered Language Technology for Tamazight
Alp Öktem and Farida Boudichat
This paper presents Awal, a community-powered initiative for developing
language technology resources for Tamazight. We provide a comprehensive review of the NLP landscape for Tamazight, examining recent progress in computational resources, and the emergence of community-driven approaches to address persistent data scarcity. Launched in 2024, awaldigital.org platform addresses the underrepresentation of Tamazight in digital spaces through a collaborative platform enabling speakers to contribute translation and voice data. We analyze 18 months of community engagement, revealing significant barriers to participation including limited confidence in written Tamazight and ongoing standardization challenges. Despite widespread positive reception, actual data contribution remained concentrated among linguists and activists. The modest scale of community contributions – 6,421 translation pairs and 3 hours of
speech data – highlights the limitations of applying standard crowdsourcing approaches to
languages with complex sociolinguistic contexts. We are working on improved open-source MT models using the collected data.
Read the full abstract ▾
Computation and Language
Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs
Jiahao Liu, Zijian Wang, Kuo Zhao et al. (+1 more)
Knowledge editing has emerged as an efficient approach for updating factual knowledge in large
language models (LLMs). It typically locates knowledge storage modules and then modifies their parameters. However, most existing methods focus on the weights of multilayer perceptron (MLP) modules, which are often identified as the main repositories of factual information. Other components, such as
attention (Attn) modules, are often ignored during editing. This imbalance can leave residual outdated knowledge and limit editing effectiveness. We perform comprehensive knowledge localization experiments on advanced LLMs and find that Attn modules play a substantial role in factual knowledge storage and
retrieval, especially in earlier layers. Based on these insights, we propose IntAttn-Edit, a method that extends the associative
memory paradigm to jointly update both MLP and Attn modules. Our approach uses a knowledge balancing strategy that allocates update magnitudes in proportion to each module's measured contribution to knowledge storage. Experiments on standard
benchmarks show that IntAttn-Edit achieves higher edit success, better generalization, and stronger knowledge preservation than prior methods. Further analysis shows that the balancing strategy keeps editing performance within an optimal range across diverse settings.
Read the full abstract ▾
All authors Jiahao Liu, Zijian Wang, Kuo Zhao, Dong Hu
Computation and Language · Cross-listed from cs.LG
Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
Austin Meek, Eitan Sprejer, Iván Arcuschin et al. (+2 more)
Chain-of-thought (CoT) outputs let us read a model's step-by-step
reasoning. Since any long, serial
reasoning process must pass through this textual trace, the quality of the CoT is a direct window into what the model is thinking. This visibility could help us spot unsafe or misaligned behavior (monitorability), but only if the CoT is transparent about its internal
reasoning (faithfulness). Fully measuring faithfulness is difficult, so researchers often focus on examining the CoT in cases where the model changes its answer after adding a cue to the input. This proxy finds some instances of unfaithfulness but loses information when the model maintains its answer, and does not investigate aspects of
reasoning not tied to the cue. We extend these results to a more holistic sense of monitorability by introducing verbosity: whether the CoT lists every factor needed to solve the task. We combine faithfulness and verbosity into a single monitorability score that shows how well the CoT serves as the model's external `working
memory', a property that many safety schemes based on CoT monitoring depend on. We evaluate instruction-tuned and
reasoning models on BBH, GPQA, and MMLU. Our results show that models can appear faithful yet remain hard to monitor when they leave out key factors, and that monitorability differs sharply across model families. We release our evaluation code using the Inspect library to support reproducible future work.
Read the full abstract ▾
All authors Austin Meek, Eitan Sprejer, Iván Arcuschin, Austin J. Brockmeier, Steven Basart
Computation and Language
From the Rock Floor to the Cloud: A Systematic Survey of State-of-the-Art NLP in Battery Life Cycle
Tosin Adewumi, Martin Karlsson, Marcus Liwicki et al. (+4 more)
This work presents a systematic review of the application of natural
language processing (NLP) along the entire battery life cycle, instead of the review of one stage or method, and introduces a novel conceptual technical
language processing (TLP) framework for the European Union (EU)'s proposed digital battery passport (DBP) and other general battery predictions, as the main objective. It follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method and employs three reputable databases or search engines, including Google Scholar, Institute of Electrical and Electronics Engineers Xplore (IEEE Xplore), and Scopus, searched in October, 2025. The search criteria are 'text', '
language processing', and 'battery' while the exclusion criteria are non-relevant papers and those published before 2017, which was applied to all databases, except IEEE Xplore because it turned up fewer results. Consequently, it assesses 274 scientific papers before the critical review of the final 66 relevant papers. We publicly provide artifacts of the review for validation and possible reproducibility. The findings show that new NLP tasks are emerging in the battery domain, which facilitate materials discovery and other stages of the life cycle. Notwithstanding, challenges remain, such as the lack of standard
benchmarks. The proposed TLP framework, which incorporates agentic AI and optimized
prompts, may be apt for tackling some of the challenges.
Read the full abstract ▾
All authors Tosin Adewumi, Martin Karlsson, Marcus Liwicki, Mikael Sjödahl, Lama Alkhaled, Rihab Gargouri, Nudrat Habib
Computation and Language
ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations
Zijian Wang and Chang Xu
This paper introduces ThoughtProbe, a novel inference time framework that leverages the hidden
reasoning features of Large
Language Models (LLMs) to improve their
reasoning performance. Unlike previous works that manipulate the hidden representations to steer LLM generation, we harness them as discriminative signals to guide the tree structured response space exploration. In each node expansion, a classifier serves as a scoring and ranking mechanism that efficiently allocates computational resources by prioritizing higher score candidates for continuation. After completing the tree expansion, we collect answers from all branches to form a candidate answer pool. We then propose a branch aggregation method that marginalizes over all supporting branches by aggregating their CoT scores, thereby identifying the optimal answer from the pool. Experimental results show that our framework's comprehensive exploration not only covers valid
reasoning chains but also effectively identifies them, achieving significant improvements across multiple arithmetic
reasoning benchmarks.
Read the full abstract ▾
Computation and Language
TransAlign: Machine Translation Encoders are Strong Word Aligners, Too
Benedikt Ebing, Christian Goldschmied and Goran Glavaš
In the absence of sizable training data for most world
languages and NLP tasks, translation-based strategies such as translate-test – evaluating on noisy source
language data translated from the target
language – and translate-train – training on noisy target
language data translated from the source
language – have been established as competitive approaches for cross-lingual transfer (XLT). For
token classification tasks, these strategies require label projection: mapping the labels from each
token in the original sentence to its counterpart(s) in the translation. To this end, it is common to leverage
multilingual word aligners (WAs) derived from encoder
language models such as mBERT or LaBSE. Despite obvious associations between machine translation (MT) and WA, research on extracting
alignments with MT models is largely limited to exploiting cross-
attention in encoder-decoder architectures, yielding poor WA results. In this work, in contrast, we propose TransAlign, a novel word aligner that utilizes the encoder of a massively
multilingual MT model. We show that TransAlign not only achieves strong WA performance but substantially outperforms popular WA and state-of-the-art non-WA-based label projection methods in MT-based XLT for
token classification.
Read the full abstract ▾
Computation and Language
A Unified Representation Underlying the Judgment of Large Language Models
Yi-Long Lu, Jiajun Song and Wei Wang
A central architectural question for both biological and artificial intelligence is whether judgment relies on specialized modules or a unified, domain-general resource. While the discovery of decodable neural representations for distinct concepts in Large
Language Models (LLMs) has suggested a modular architecture, whether these representations are truly independent systems remains an open question. Here we provide evidence for a convergent architecture for evaluative judgment. Across a range of LLMs, we find that diverse evaluative judgments are computed along a dominant dimension, which we term the Valence-Assent Axis (VAA). This axis jointly encodes subjective valence ("what is good") and the model's assent to factual claims ("what is true"). Through direct interventions, we demonstrate this axis drives a critical mechanism, which is identified as the subordination of
reasoning: the VAA functions as a control signal that steers the generative process to construct a rationale consistent with its evaluative state, even at the cost of factual accuracy. Our discovery offers a mechanistic account for response bias and
hallucination, revealing how an architecture that promotes coherent judgment can systematically undermine faithful
reasoning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
LLM generation novelty through the lens of semantic similarity
Philipp Davydov, Ameya Prabhu, Matthias Bethge et al. (+2 more)
Generation novelty is a key indicator of an LLM's ability to generalize, yet measuring it against full pretraining corpora is computationally challenging. Existing evaluations often rely on lexical overlap, failing to detect paraphrased text, or do not consider the full pretraining corpus. We frame novelty as a semantic
retrieval problem. This framing enables us to address novelty with modern
embedding and indexing pipelines, allowing for efficient analysis at pre-training scale. Specifically, we propose a three-stage framework that retrieves semantically similar samples, reranks them at varying subsequence lengths, and calibrates scores using a human novelty reference for interpretability. We apply this framework to the SmolLM model family and report three key findings: (1) models draw on pre-training data across much longer sequences than previously reported; (2) some task domains systematically promote or suppress generation novelty; and (3) instruction tuning not only alters style but also increases novelty. These results highlight the value of semantic novelty analysis for studying generalization. To support reproducibility and further research, we release ~20 TB of corpus chunks and index artifacts at https://huggingface.co/
datasets/stai-tuebingen/faiss-smollm
Read the full abstract ▾
All authors Philipp Davydov, Ameya Prabhu, Matthias Bethge, Elisa Nguyen, Seong Joon Oh
Computation and Language
Deokhyung Kang, Seonjeong Hwang, Daehui Kim et al. (+2 more)
Reasoning language models (RLMs) achieve strong performance on complex
reasoning tasks, yet they still exhibit a
multilingual reasoning gap, performing better in high-resource
languages than in low-resource ones. While recent efforts have been made to address this gap, its underlying causes remain largely unexplored. In this work, we show that this gap primarily stems from failures in
language understanding-specifically, the model's inability to translate
multilingual inputs into the
language dominating its
reasoning traces (typically English). As identifying understanding failures can enable targeted mitigation of the gap, we evaluate a range of detection methods and find that understanding failures are detectable to a meaningful extent, with supervised approaches performing best. Building on this, we propose Selective Translation, a strategy that incorporates an English translation into the initial
reasoning trace only when an understanding failure is detected. Experimental results using Qwen3-4B show that Selective Translation substantially bridges the
multilingual reasoning gap, achieving near full-translation performance while translating only about 20% of inputs. Together, our results show that failures in
language understanding are the primary driver of the
multilingual reasoning gap and can be detected and selectively mitigated, clarifying its origin and suggesting a path toward more equitable
multilingual reasoning. Our code and data are publicly available at https://github.com/deokhk/RLM_analysis
Read the full abstract ▾
All authors Deokhyung Kang, Seonjeong Hwang, Daehui Kim, Hyounghun Kim, Gary Geunbae Lee
Computation and Language
MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Models
Kangkun Mao, Jinru Ding, Jiayuan Chen et al. (+6 more)
As large
language models (LLMs) enter the medical domain, most
benchmarks evaluate them on question answering or descriptive
reasoning, overlooking quantitative
reasoning critical to clinical decision-making. Existing
datasets like MedCalc-Bench cover few calculation tasks and fail to reflect real-world computational scenarios. We introduce MedCalc-Eval, the largest
benchmark for assessing LLMs' medical calculation abilities, comprising 700+ tasks across two types: equation-based (e.g., Cockcroft-Gault, BMI, BSA) and rule-based scoring systems (e.g., Apgar, Glasgow Coma Scale). These tasks span diverse specialties including internal medicine, surgery, pediatrics, and cardiology, offering a broader and more challenging evaluation setting. To improve performance, we further develop MedCalc-Env, a reinforcement learning environment built on the InternBootcamp framework, enabling multi-step clinical
reasoning and planning. Fine-tuning a Qwen2.5-32B model within this environment achieves state-of-the-art results on MedCalc-Eval, with notable gains in numerical sensitivity, formula selection, and
reasoning robustness. Remaining challenges include unit conversion, multi-condition logic, and contextual understanding. Code and
datasets are available at https://github.com/maokangkun/MedCalc-Eval.
Read the full abstract ▾
All authors Kangkun Mao, Jinru Ding, Jiayuan Chen, Mouxiao Bian, Ruiyao Chen, Xinwei Peng, Sijie Ren, Linyang Li, Jie Xu
Computation and Language · Cross-listed from cs.LG
Higher-order Linear Attention
Yifan Zhang, Zhen Qin, Mengdi Wang et al. (+1 more)
The quadratic cost of scaled dot-product
attention is a central obstacle to scaling autoregressive
language models to long contexts. Linear-time
attention and State Space Models (SSMs) provide scalable alternatives but are typically restricted to first-order or kernel-based approximations, which can limit expressivity. We introduce Higher-order Linear
Attention (HLA), a causal, streaming mechanism that realizes higher interactions via compact prefix sufficient statistics. In the second-order case, HLA maintains a constant-size state and computes per-
token outputs in linear time without materializing any $n \times n$
matrices. We give closed-form streaming identities, a strictly causal masked variant using two additional summaries, and a chunk-parallel training scheme based on associative scans that reproduces the activations of a serial recurrence exactly. We further outline extensions to third and higher orders. Collectively, these results position HLA as a principled, scalable building block that combines
attention-like, data-dependent mixing with the efficiency of modern recurrent architectures.
Read the full abstract ▾
All authors Yifan Zhang, Zhen Qin, Mengdi Wang, Quanquan Gu
Computation and Language
Languages are Modalities: Cross-Lingual Alignment via Encoder Injection
Rajan Agarwal and Aarush Gupta
Instruction-tuned Large
Language Models (LLMs) underperform on low resource, non-Latin scripts due to tokenizer fragmentation and weak cross-lingual coupling. We present LLINK (Latent
Language Injection for Non-English Knowledge), a compute efficient
language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder. First, we align sentence
embeddings from a frozen
multilingual encoder to the decoder's latent
embedding space at a reserved position via a lightweight contrastive projector. Second, the vector is expanded into K soft slots and trained with minimal adapters so the frozen decoder consumes the signal. LLINK substantially improves bilingual
retrieval and achieves 81.3% preference over the base model and 63.6% over direct fine-tuning in LLM-judged Q&A evaluations. We further find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual
alignment, despite the model having residual weaknesses in numeric fidelity. Treating low resource
languages as a modality offers a practical path to stronger cross-lingual
alignment in lightweight LLMs.
Read the full abstract ▾
Computation and Language
Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
Mohammad Tavakoli, Alireza Salemi, Carrie Ye et al. (+3 more)
Evaluating the abilities of large
language models (LLMs) for tasks that require long-term
memory and thus long-context
reasoning, for example in conversational settings, is hampered by the existing
benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a novel framework for automatically generating long (up to 10M
tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of
memory abilities. From this, we construct BEAM, a new
benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT-a framework inspired by human cognition that equips LLMs with three complementary
memory systems: a long-term episodic
memory, a short-term working
memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M
token context windows (with and without
retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving an average improvement of 3.5%-12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each
memory component.
Read the full abstract ▾
All authors Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani, J Ross Mitchell
Computation and Language
Identifying the Periodicity of Information in Natural Language
Yulin Ou, Yu Wang, Yang Xu et al. (+1 more)
Recent theoretical advancement of information density in natural
language has brought the following question on desk: To what degree does natural
language exhibit periodicity pattern in its encoded information? We address this question by introducing a new method called AutoPeriod of Surprisal (APS). APS adopts a canonical periodicity detection
algorithm and is able to identify any significant periods that exist in the surprisal sequence of a single document. By applying the
algorithm to a set of corpora, we have obtained the following interesting results: Firstly, a considerable proportion of human
language demonstrates a strong pattern of periodicity in information; Secondly, new periods that are outside the distributions of typical structural units in text (e.g., sentence boundaries, elementary discourse units, etc.) are found and further confirmed via harmonic regression modeling. We conclude that the periodicity of information in
language is a joint outcome from both structured factors and other driving factors that take effect at longer distances. The advantages of our periodicity detection method and its potentials in LLM-generation detection are further discussed.
Read the full abstract ▾
All authors Yulin Ou, Yu Wang, Yang Xu, Hendrik Buschmeier
Computation and Language · Cross-listed from cs.DB
DRAMA: Unifying Data Retrieval and Analysis for Open-Domain Analytic Queries
Chuxuan Hu, Maxwell Yang, James Weiland et al. (+3 more)
Manually conducting real-world data analyses is labor-intensive and inefficient. Despite numerous attempts to automate data science workflows, none of the existing paradigms or systems fully demonstrate all three key capabilities required to support them effectively: (1) open-domain data collection, (2) structured data transformation, and (3) analytic
reasoning. To overcome these limitations, we propose DRAMA, an end-to-end paradigm that answers users' analytic queries in natural
language on large-scale open-domain data. DRAMA unifies data collection, transformation, and analysis as a single pipeline. To quantitatively evaluate system performance on tasks representative of DRAMA, we construct a
benchmark, DRAMA-Bench, consisting of two categories of tasks: claim verification and question answering, each comprising 100 instances. These tasks are derived from real-world applications that have gained significant public
attention and require the
retrieval and analysis of open-domain data. We develop DRAMA-Bot, a multi-
agent system designed following DRAMA. It comprises a data retriever that collects and transforms data by coordinating the execution of sub-
agents, and a data analyzer that performs structured
reasoning over the retrieved data. We evaluate DRAMA-Bot on DRAMA-Bench together with five state-of-the-art baseline
agents. DRAMA-Bot achieves 86.5% task accuracy at a cost of $0.05, outperforming all baselines with up to 6.9 times the accuracy and less than 1/6 of the cost. DRAMA is publicly available at https://github.com/uiuc-kang-lab/drama.
Read the full abstract ▾
All authors Chuxuan Hu, Maxwell Yang, James Weiland, Yeji Lim, Suhas Palawala, Daniel Kang
Computation and Language
MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models
Zixin Chen, Hongzhan Lin, Kaixin Li et al. (+3 more)
The proliferation of memes on social media necessitates the capabilities of
multimodal Large
Language Models (mLLMs) to effectively understand
multimodal harmfulness. Existing evaluation approaches predominantly focus on mLLMs' detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. In this paper, we propose MemeArena, an
agent-based arena-style evaluation framework that provides a context-aware and unbiased assessment for mLLMs' understanding of
multimodal harmfulness. Specifically, MemeArena simulates diverse interpretive contexts to formulate evaluation tasks that elicit perspective-specific analyses from mLLMs. By integrating varied viewpoints and reaching consensus among evaluators, it enables fair and unbiased comparisons of mLLMs' abilities to interpret
multimodal harmfulness. Extensive experiments demonstrate that our framework effectively reduces the evaluation biases of judge
agents, with judgment results closely aligning with human preferences, offering valuable insights into reliable and comprehensive mLLM evaluations in
multimodal harmfulness understanding. Our code and data are publicly available at https://github.com/Lbotirx/MemeArena.
Read the full abstract ▾
All authors Zixin Chen, Hongzhan Lin, Kaixin Li, Ziyang Luo, Yayue Deng, Jing Ma
Computation and Language · Cross-listed from cs.CV
Can MLLMs Read the Room? A Multimodal Benchmark for Verifying Truthfulness in Multi-Party Social Interactions
Caixin Kang, Yifei Huang, Liangyang Ouyang et al. (+2 more)
As AI systems become increasingly integrated into human lives, endowing them with robust social intelligence has emerged as a critical frontier. A key aspect of this intelligence is discerning truth from deception, a ubiquitous element of human interaction that is conveyed through a complex interplay of verbal
language and non-verbal visual cues. However, automatic deception detection in dynamic, multi-party conversations remains a significant challenge. The recent rise of powerful
Multimodal Large
Language Models (MLLMs), with their impressive abilities in visual and textual understanding, makes them natural candidates for this task. Consequently, their capabilities in this crucial domain are mostly unquantified. To address this gap, we introduce a new task,
Multimodal Interactive Veracity Assessment (MIVA), and present a novel
multimodal dataset derived from the social deduction game Werewolf. This
dataset provides synchronized
video, text, with verifiable ground-truth labels for every statement. We establish a comprehensive
benchmark evaluating state-of-the-art MLLMs, revealing a significant performance gap: even powerful models like GPT-4o struggle to distinguish truth from falsehood reliably. Our analysis of failure modes indicates that these models fail to ground
language in visual social cues effectively and may be overly conservative in their
alignment, highlighting the urgent need for novel approaches to building more perceptive and trustworthy AI systems.
Read the full abstract ▾
All authors Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Yoichi Sato
Computation and Language
Simple Additions, Substantial Gains: Expanding Scripts, Languages, and Lineage Coverage in URIEL+
Mason Shipton, York Hay Ng, Aditya Khan et al. (+4 more)
The URIEL+ linguistic knowledge base supports
multilingual research by encoding
languages through geographic, genetic, and typological vectors. However, data sparsity (e.g. missing feature types, incomplete
language entries, and limited genealogical coverage) remains prevalent. This limits the usefulness of URIEL+ in cross-lingual transfer, particularly for supporting low-resource
languages. To address this sparsity, we extend URIEL+ by introducing script vectors to represent writing system properties for 7,488
languages, integrating Glottolog to add 18,710 additional
languages, and expanding lineage imputation for 26,449
languages by propagating typological and script features across genealogies. These improvements reduce feature sparsity by 14% for script vectors, increase
language coverage by up to 19,015
languages (1,007%), and boost imputation quality metrics by up to 35%. Our
benchmark on cross-lingual transfer tasks (oriented around low-resource
languages) shows occasionally divergent performance compared to URIEL+, with performance gains up to 6% in certain setups.
Read the full abstract ▾
All authors Mason Shipton, York Hay Ng, Aditya Khan, Phuong Hanh Hoang, Xiang Lu, A. Seza Doğruöz, En-Shiun Annie Lee
Computation and Language · Cross-listed from cs.AI
Glia: A Human-Inspired AI for Automated Systems Design and Optimization
Pouya Hamadanian, Pantea Karimi, Arash Nasr-Esfahany et al. (+5 more)
Can AI autonomously design mechanisms for computer systems on par with the creativity and
reasoning of human experts? We present Glia, an AI architecture for networked systems design that uses large
language models (LLMs) in a human-inspired multi-
agent workflow. Each
agent specializes in
reasoning, experimentation, and analysis, collaborating through an evaluation framework that grounds abstract
reasoning in empirical feedback. Unlike prior ML-for-systems methods that optimize black-box policies, Glia generates interpretable designs and exposes its
reasoning. When applied to a distributed GPU cluster for LLM inference, it produces new
algorithms for request routing, scheduling, and auto-scaling that perform at human-expert levels in significantly less time, while yielding novel insights into workload behavior. Our results suggest that combining
reasoning LLMs with structured experimentation, an AI can produce creative and understandable designs for complex systems problems.
Read the full abstract ▾
All authors Pouya Hamadanian, Pantea Karimi, Arash Nasr-Esfahany, Kimia Noorbakhsh, Joseph Chandler, Ali ParandehGheibi, Mohammad Alizadeh, Hari Balakrishnan
Computation and Language
Probability Distributions Computed by Autoregressive Transformers
Andy Yang, Anej Svete, Jiaoda Li et al. (+4 more)
Most expressivity results for
transformers treat them as
language recognizers – devices that accept or reject strings – rather than as they are used in practice: as
language models that generate strings autoregressively and probabilistically. We characterize the probability distributions that
transformer language models can express. We show that making
transformer language recognizers autoregressive can sometimes increase their expressivity, and that making them probabilistic can break equivalences that hold in the non-probabilistic case. Our overall contribution is to tease apart what functions
transformers are capable of expressing in their most common use case as
language models.
Read the full abstract ▾
All authors Andy Yang, Anej Svete, Jiaoda Li, Anthony Widjaja Lin, Jonathan Rawski, Ryan Cotterell, David Chiang
Computation and Language
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Rajarshi Haldar and Julia Hockenmaier
As Natural
Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large
language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or
embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and
benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.
Read the full abstract ▾
Computation and Language
Characterizing Selective Refusal Bias in Large Language Models
Adel Khorramrouz and Sharon Levy
Safety guardrails in large
language models(LLMs) are developed to prevent malicious users from generating toxic content at a large scale. However, these measures can inadvertently introduce or reflect new biases, as LLMs may refuse to generate harmful content targeting some demographic groups and not others. We explore this selective refusal bias in LLM guardrails through the lens of refusal rates of targeted individual and intersectional demographic groups, types of LLM responses, and length of generated refusals. Our results show evidence of selective refusal bias across gender, sexual orientation, nationality, and religion attributes. This leads us to investigate additional safety implications via an indirect attack, where we target previously refused groups. Our findings emphasize the need for more equitable and robust performance in safety guardrails across demographic groups.
Read the full abstract ▾