Computation and Language · Cross-listed from cs.IR
A Representation Sharpening Framework for Zero Shot Dense Retrieval
Dhananjay Ashok, Suraj Nair, Mutasem Al-Darabsah et al. (+3 more)
Zero-shot dense
retrieval is a challenging setting where a document corpus is provided without relevant queries, necessitating a reliance on pretrained dense retrievers (DRs). However, since these DRs are not trained on the target corpus, they struggle to represent semantic differences between similar documents. To address this failing, we introduce a training-free representation sharpening framework that augments a document's representation with information that helps differentiate it from similar documents in the corpus. On over twenty
datasets spanning multiple
languages, the representation sharpening framework proves consistently superior to traditional
retrieval, setting a new state-of-the-art on the BRIGHT
benchmark. We show that representation sharpening is compatible with prior approaches to zero-shot dense
retrieval and consistently improves their performance. Finally, we address the performance-cost tradeoff presented by our framework and devise an indexing-time approximation that preserves the majority of our performance gains over traditional
retrieval, yet suffers no additional inference-time cost.
Read the full abstract ▾
All authors Dhananjay Ashok, Suraj Nair, Mutasem Al-Darabsah, Choon Hui Teo, Tarun Agarwal, Jonathan May
Computation and Language
Optimizing Diversity and Quality through Base-Aligned Model Collaboration
Yichen Wang, Chenghao Yang, Tenghao Huang et al. (+3 more)
Alignment has greatly improved large
language models (LLMs)' output quality at the cost of diversity, yielding highly similar outputs across generations, especially in open-ended generation tasks. We propose Base-Aligned Model Collaboration (BACo), an inference-time
token-level model collaboration framework that dynamically combines a base LLM with its aligned counterpart to optimize diversity and quality. Using uncertainty and content-based signals, BACo employs routing strategies to determine, at each
token, which model to decode from. Prior diversity-promoting methods often improve diversity at the expense of quality or require expensive decoding or post-training. In contrast, BACo achieves both high diversity and quality post hoc within a single pass, while offering strong controllability. We introduce a family of effective routing strategies and evaluate them across three open-ended generation tasks with 13 diversity and quality metrics. BACo consistently surpasses state-of-the-art inference-time baselines. With our best router, BACo achieves a 21.3% joint improvement in diversity and quality, which is further supported by human evaluations. Overall, our results demonstrate that collaboration between base and aligned models provides an effective and controllable mechanism for optimizing the diversity-quality trade-off.
Read the full abstract ▾
All authors Yichen Wang, Chenghao Yang, Tenghao Huang, Muhao Chen, Jonathan May, Mina Lee
Computation and Language
MIMIC-SR-ICD11: A Dataset for Narrative-Based Diagnosis
Yuexin Wu, Shiqi Wang and Vasile Rus
Disease diagnosis is a central pillar of modern healthcare, enabling early detection and timely intervention for acute conditions while guiding lifestyle adjustments and medication regimens to prevent or slow chronic disease. Self-reports preserve clinically salient signals that templated electronic health record (EHR) documentation often attenuates or omits, especially subtle but consequential details. To operationalize this shift, we introduce MIMIC-SR-ICD11, a large English diagnostic
dataset built from EHR discharge notes and natively aligned to WHO ICD-11 terminology. We further present LL-Rank, a likelihood-based re-ranking framework that computes a length-normalized joint likelihood of each label given the clinical report context and subtracts the corresponding report-free prior likelihood for that label. Across seven model backbones, LL-Rank consistently outperforms a strong generation-plus-mapping baseline (GenMap). Ablation experiments show that LL-Rank's gains primarily stem from its PMI-based scoring, which isolates semantic compatibility from label frequency bias.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
APP: Accelerated Path Patching with Task-Specific Pruning
Frauke Andersen, William Rudman, Ruochen Zhang et al. (+1 more)
Circuit discovery is a key step in many mechanistic interpretability pipelines. Current methods, such as Path Patching, are computationally expensive and have limited in-depth
circuit analysis for smaller models. In this study, we propose Accelerated Path Patching (APP), a hybrid approach leveraging our novel contrastive
attention head
pruning method to drastically reduce the search space of
circuit discovery methods. Our Contrastive-FLAP
pruning algorithm uses techniques from causal mediation analysis to assign higher
pruning scores to task-specific
attention heads, leading to higher performing sparse models compared to traditional
pruning techniques. Although Contrastive-FLAP is successful at preserving task-specific heads that existing
pruning algorithms remove at low sparsity ratios, the
circuits found by Contrastive-FLAP alone are too large to satisfy the minimality constraint required in
circuit analysis. APP first applies Contrastive-FLAP to reduce the search space on required for
circuit discovery
algorithms by, on average, 56%. Next, APP, applies traditional Path Patching on the remaining
attention heads, leading to a speed up of 59.63%-93.27% compared to Path Patching applied to the dense model. Despite the substantial computational saving that APP provides,
circuits obtained from APP exhibit substantial overlap and similar performance to previously established Path Patching
circuits
Read the full abstract ▾
All authors Frauke Andersen, William Rudman, Ruochen Zhang, Carsten Eickhoff
Computation and Language
Steering Language Models with Weight Arithmetic
Constanza Fierro and Fabien Roger
Providing high-quality feedback to Large
Language Models (LLMs) on a diverse training distribution can be difficult and expensive, and providing feedback only on a narrow distribution can result in unintended generalizations. To better leverage narrow training data, we propose contrastive weight steering, a simple post-training method that edits the model parameters using weight arithmetic. We isolate a behavior direction in weight-space by subtracting the weight deltas from two small fine-tunes – one that induces the desired behavior and another that induces its opposite – and then add or remove this direction to modify the model's weights. We apply this technique to mitigate sycophancy and induce misalignment, and find that weight steering often generalizes further than activation steering, achieving stronger out-of-distribution behavioral control before degrading general capabilities. We also show that, in the context of task-specific fine-tuning, weight steering can partially mitigate undesired behavioral drift: it can reduce sycophancy and under-refusals introduced during fine-tuning while preserving task performance gains. Finally, we provide preliminary evidence that emergent misalignment can be detected by measuring the similarity between fine-tuning updates and an "evil" weight direction, suggesting that it may be possible to monitor the
evolution of weights during training and detect rare misaligned behaviors that never manifest during training or evaluations.
Read the full abstract ▾
Computation and Language
Minority-Aware Satisfaction Estimation in Dialogue Systems via Preference-Adaptive Reinforcement Learning
Yahui Fu, Zi Haur Pang and Tatsuya Kawahara
User satisfaction in dialogue systems is inherently subjective. When the same response strategy is applied across users, minority users may assign different satisfaction ratings than majority users due to variations in individual intents and preferences. However, existing
alignment methods typically train one-size-fits-all models that aim for broad consensus, often overlooking minority perspectives and user-specific adaptation. We propose a unified framework that models both individual- and group-level preferences for user satisfaction estimation. First, we introduce Chain-of-Personalized-
Reasoning (CoPeR) to capture individual preferences through interpretable
reasoning chains. Second, we propose an expectation-maximization-based Majority-Minority Preference-Aware Clustering (M2PC)
algorithm that discovers distinct user groups in an unsupervised manner to learn group-level preferences. Finally, we integrate these components into a preference-adaptive reinforcement learning framework (PAda-PPO) that jointly optimizes
alignment with both individual and group preferences. Experiments on the Emotional Support Conversation
dataset demonstrate consistent improvements in user satisfaction estimation, particularly for underrepresented user groups.
Read the full abstract ▾
Computation and Language
Large Language Models for Explainable Threat Intelligence
Tiago Dinis, Miguel Correia and Roger Tavares
As cyber threats continue to grow in complexity, traditional security mechanisms struggle to keep up. Large
language models (LLMs) offer significant potential in cybersecurity due to their advanced capabilities in text processing and generation. This paper explores the use of LLMs with
retrieval-augmented generation (RAG) to obtain threat intelligence by combining real-time information
retrieval with domain-specific data. The proposed system, RAGRecon, uses a LLM with RAG to answer questions about cybersecurity threats. Moreover, it makes this form of Artificial Intelligence (AI) explainable by generating and visually presenting to the user a knowledge graph for every reply. This increases the transparency and interpretability of the
reasoning of the model, allowing analysts to better understand the connections made by the system based on the context recovered by the RAG system. We evaluated RAGRecon experimentally with two
datasets and seven different LLMs and the responses matched the reference responses more than 91% of the time for the best combinations.
Read the full abstract ▾
Computation and Language
A multimodal multiplex of the mental lexicon for multilingual individuals
Maria Huynh and Wilder C. Rodrigues
Historically, bilingualism was often perceived as an additional cognitive load that could hinder linguistic and intellectual development. However, over the last three decades, this view has changed considerably. Numerous studies have aimed to model and understand the architecture of the bilingual word recognition system Dijkstra and van Heuven (2002), investigating how parallel activation operates in the
brain and how one
language influences another Kroll et al. (2015). Increasingly, evidence suggests that multilinguals, individuals who speak three or more
languages, can perform better than monolinguals in various linguistic and cognitive tasks, such as learning an additional
language Abu-Rabia and Sanitsky (2010). This research proposal focuses on the study of the mental lexicon and how it may be structured in individuals who speak multiple
languages. Building on the work of Stella et al. (2018), who investigated explosive learning in humans using a multiplex model of the mental lexicon, and the Bilingual Interactive Activation (BIA+) framework proposed by Dijkstra and van Heuven (2002), the present study applies the same multilayer network principles introduced by Kivela et al. (2014). Our experimental design extends previous research by incorporating multimodality into the multiplex model, introducing an additional layer that connects visual inputs to their corresponding lexical representations across the
multilingual layers of the mental lexicon. In this research, we aim to explore how a heritage
language influences the acquisition of another
language. Specifically, we ask: Does the presence of visual input in a translation task influence participants' proficiency and accuracy compared to text-only conditions?
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversations
Amr Gomaa, Ahmed Salem and Sahar Abdelnabi
As
language models evolve into autonomous
agents that act and communicate on behalf of users, ensuring safety in multi-
agent ecosystems becomes a central challenge. Interactions between personal assistants and external service providers expose a core tension between utility and protection: effective collaboration requires information sharing, yet every exchange creates new attack surfaces. We introduce ConVerse, a dynamic
benchmark for evaluating
privacy and security risks in
agent-
agent interactions. ConVerse spans three practical domains (travel, real estate, insurance) with 12 user personas and over 864 contextually grounded attacks (611
privacy, 253 security). Unlike prior single-
agent settings, it models autonomous, multi-turn
agent-to-
agent conversations where malicious requests are embedded within plausible discourse.
Privacy is tested through a three-tier taxonomy assessing abstraction quality, while security attacks target tool use and preference manipulation. Evaluating seven state-of-the-art models reveals persistent vulnerabilities;
privacy attacks succeed in up to 88% of cases and security breaches in up to 60%, with stronger models leaking more. By unifying
privacy and security within interactive multi-
agent contexts, ConVerse reframes safety as an emergent property of communication.
Read the full abstract ▾
Computation and Language
Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
Firoj Ahmmed Patwary and Abdullah Al Noman
Tokenization is an important first step in Natural
Language Processing (NLP) pipelines because it decides how models learn and represent linguistic information. However, current subword tokenizers like SentencePiece or HuggingFace BPE are mostly designed for Latin or
multilingual corpora and do not perform well on
languages with rich morphology such as Bengali. To address this limitation, we present BengaliBPE, a Byte Pair Encoding (BPE) tokenizer specifically developed for the Bengali script. BengaliBPE applies Unicode normalization, grapheme-level initialization, and morphology-aware merge rules to maintain linguistic consistency and preserve subword integrity. We use a large-scale Bengali news classification
dataset to compare BengaliBPE with three baselines: Whitespace, SentencePiece BPE, and HuggingFace BPE. The evaluation considers tokenization granularity, encoding speed, and downstream classification accuracy. While all methods perform reasonably well, BengaliBPE provides the most detailed
segmentation and the best morphological interpretability, albeit with slightly higher computational cost. These findings highlight the importance of
language-aware tokenization for morphologically rich scripts and establish BengaliBPE as a strong foundation for future Bengali NLP systems, including large-scale pretraining of contextual
language models.
Read the full abstract ▾
Computation and Language
What Are the Facts? Automated Extraction of Court-Established Facts from Criminal-Court Opinions
Klára Bendová, Tomáš Knap, Jan Černý et al. (+4 more)
Criminal justice administrative data contain only a limited amount of information about the committed offense. However, there is an unused source of extensive information in continental European courts' decisions: descriptions of criminal behaviors in verdicts by which offenders are found guilty. In this paper, we study the feasibility of extracting these descriptions from publicly available court decisions from Slovakia. We use two different approaches for
retrieval: regular expressions and large
language models (LLMs). Our baseline was a simple method employing regular expressions to identify typical words occurring before and after the description. The advanced regular expression approach further focused on "sparing" and its normalization (insertion of spaces between individual letters), typical for delineating the description. The LLM approach involved prompting the Gemini Flash 2.0 model to extract the descriptions using predefined instructions. Although the baseline identified descriptions in only 40.5% of verdicts, both methods significantly outperformed it, achieving 97% with advanced regular expressions and 98.75% with LLMs, and 99.5% when combined. Evaluation by law students showed that both advanced methods matched human annotations in about 90% of cases, compared to just 34.5% for the baseline. LLMs fully matched human-labeled descriptions in 91.75% of instances, and a combination of advanced regular expressions with LLMs reached 92%.
Read the full abstract ▾
All authors Klára Bendová, Tomáš Knap, Jan Černý, Vojtěch Pour, Jaromir Savelka, Ivana Kvapilíková, Jakub Drápal
Computation and Language
Listening Between the Lines: Decoding Podcast Narratives with Language Modeling
Shreya Gupta, Ojasva Saxena, Arghodeep Nandi et al. (+3 more)
Podcasts have become a central arena for shaping public opinion, making them a vital source for understanding contemporary discourse. Their typically unscripted, multi-themed, and conversational style offers a rich but complex form of data. To analyze how podcasts persuade and inform, we must examine their narrative structures – specifically, the narrative frames they employ. The fluid and conversational nature of podcasts presents a significant challenge for automated analysis. We show that existing large
language models, typically trained on more structured text such as news articles, struggle to capture the subtle cues that human listeners rely on to identify narrative frames. As a result, current approaches fall short of accurately analyzing podcast narratives at scale. To solve this, we develop and evaluate a fine-tuned BERT model that explicitly links narrative frames to specific entities mentioned in the conversation, effectively grounding the abstract frame in concrete details. Our approach then uses these granular frame labels and correlates them with high-level topics to reveal broader discourse trends. The primary contributions of this paper are: (i) a novel frame-labeling methodology that more closely aligns with human judgment for messy, conversational data, and (ii) a new analysis that uncovers the systematic relationship between what is being discussed (the topic) and how it is being presented (the frame), offering a more robust framework for studying influence in digital media.
Read the full abstract ▾
All authors Shreya Gupta, Ojasva Saxena, Arghodeep Nandi, Sarah Masud, Kiran Garimella, Tanmoy Chakraborty
Computation and Language · Cross-listed from cs.IR
QueStER: Query Specification for Generative keyword-based Retrieval
Arthur Satouf, Yuxuan Zong, Habiboulaye Amadou-Boubacar et al. (+2 more)
Generative
retrieval (GR) differs from the traditional index-then-retrieve pipeline by storing relevance in model parameters and generating
retrieval cues directly from the query, but it can be brittle out of domain and expensive to scale. We introduce QueStER (QUEry SpecificaTion for gEnerative Keyword-Based
Retrieval), which bridges GR and query reformulation by learning to generate explicit keyword-based search specifications. Given a user query, a lightweight LLM produces a keyword query that is executed by a standard retriever (BM25), combining the generalization benefits of generative query rewriting with the efficiency and scalability of lexical indexing. We train the rewriting policy with reinforcement learning techniques. Across in- and out-of-domain evaluations, QueStER consistently improves over BM25 and is competitive with neural IR baselines, while maintaining strong efficiency.
Read the full abstract ▾
All authors Arthur Satouf, Yuxuan Zong, Habiboulaye Amadou-Boubacar, Pablo Piantanida, Benjamin Piwowarski
Computation and Language · Cross-listed from cs.DS
Language Generation and Identification From Partial Enumeration: Tight Density Bounds and Topological Characterizations
Jon Kleinberg and Fan Wei
The success of large
language models (LLMs) has motivated formal theories of
language generation and learning. We study the framework of
language generation in the limit, where an adversary enumerates strings from an unknown
language $K$ drawn from a countable class, and an
algorithm must generate unseen strings from $K$. Prior work showed that generation is always possible, and that some
algorithms achieve positive lower density, revealing a validity–breadth trade-off between correctness and coverage. We resolve a main open question in this line, proving a tight bound of $1/2$ on the best achievable lower density. We then strengthen the model to allow partial enumeration, where the adversary reveals only an infinite subset $C \subseteq K$. We show that generation in the limit remains achievable, and if $C$ has lower density $\alpha$ in $K$, the
algorithm's output achieves density at least $\alpha/2$, matching the upper bound. This generalizes the $1/2$ bound to the partial-information setting, where the generator must recover within a factor $1/2$ of the revealed subset's density. We further revisit the classical Gold–Angluin model of
language identification under partial enumeration. We characterize when identification in the limit is possible – when hypotheses $M_t$ eventually satisfy $C \subseteq M \subseteq K$ – and in the process give a new topological formulation of Angluin's characterization, showing that her condition is precisely equivalent to an appropriate topological space having the $T_D$ separation property.
Read the full abstract ▾
Computation and Language
Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Models
Teqi Hao, Xioayu Tan, Shaojie Shi et al. (+2 more)
The personalization of black-box large
language models (LLMs) is a critical yet challenging task. Existing approaches predominantly rely on context injection, where user history is embedded into the
prompt to directly guide the generation process. However, this single-step paradigm imposes a dual burden on the model: generating accurate content while simultaneously aligning with user-specific styles. This often results in a trade-off that compromises output quality and limits precise control. To address this fundamental tension, we propose Reflective Personalization Optimization (RPO), a novel framework that redefines the personalization paradigm by decoupling content generation from
alignment. RPO operates in two distinct stages: first, a base model generates a high-quality, generic response; then, an external reflection module explicitly rewrites this output to align with the user's preferences. This reflection module is trained using a two-stage process. Initially, supervised fine-tuning is employed on structured rewriting trajectories to establish a core personalized
reasoning policy that models the transformation from generic to user-aligned responses. Subsequently, reinforcement learning is applied to further refine and enhance the quality of the personalized outputs. Comprehensive experiments on the LaMP
benchmark demonstrate that RPO, by decoupling content generation from personalization, significantly outperforms state-of-the-art baselines. These findings underscore the superiority of explicit response shaping over implicit context injection. Moreover, RPO introduces an efficient, model-agnostic personalization layer that can be seamlessly integrated with any underlying base model, paving the way for a new and effective direction in user-centric generation scenarios.
Read the full abstract ▾
All authors Teqi Hao, Xioayu Tan, Shaojie Shi, Yinghui Xu, Xihe Qiu
Computation and Language
Translation via Annotation: A Computational Study of Translating Classical Chinese into Japanese
Zilong Li and Jie Cao
Ancient people translated classical Chinese into Japanese using a system of annotations placed around characters. We abstract this process as sequence tagging tasks and fit them into modern
language technologies. The research on this annotation and translation system faces a low resource problem. We alleviate this problem by introducing an LLM-based annotation pipeline and constructing a new
dataset from digitized open-source translation data. We show that in the low-resource setting, introducing auxiliary Chinese NLP tasks enhances the training of sequence tagging tasks. We also evaluate the performance of Large
Language Models (LLMs) on this task. While they achieve high scores on direct machine translation, our method could serve as a supplement to LLMs to improve the quality of character's annotation.
Read the full abstract ▾
Computation and Language
Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
Cong-Thanh Do, Rama Doddipatla and Kate Knill
Chain-of-Thought (CoT) prompting is a widely used method to improve the
reasoning capability of Large
Language Models (LLMs). More recently, CoT has been leveraged in Knowledge
Distillation (KD) to transfer
reasoning capability from a larger LLM to a smaller one. This paper examines the role of CoT in distilling the
reasoning capability from larger LLMs to smaller LLMs using white-box KD, analysing its effectiveness in improving the performance of the distilled models for various natural
language reasoning and understanding tasks. We conduct white-box KD experiments using LLMs from the Qwen and Llama2 families, employing CoT data from the CoT-Collection
dataset. The distilled models are then evaluated on natural
language reasoning and understanding tasks from the BIG-Bench-Hard (BBH)
benchmark, which presents complex challenges for smaller LLMs. Experimental results demonstrate the role of CoT in improving white-box KD effectiveness, enabling the distilled models to achieve better average performance in natural
language reasoning and understanding tasks from BBH.
Read the full abstract ▾
Computation and Language
Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
Jan-Thorsten Peter, David Vilar, Tobias Domhan et al. (+2 more)
Most current large
language models (LLMs) support a wide variety of
languages in addition to English, including high-resource
languages (e.g. German, Chinese, French), as well as low-resource ones (e.g. Swahili, Telugu). In addition, they have shown impressive capabilities in different domains, like coding, science and math. In this paper, taking math as an example domain, we study the performance of different LLMs across
languages. Experimental results show that there exists a non-negligible and consistent gap in the performance of the models across
languages. Interestingly, and somewhat against expectations, the gap exists for both high- and low-resource
languages. These results should impact further research into cross-lingual capability generalization for next generation LLMs. Or they would, if it weren't for the fact that they are distorted by data quality issues. By analyzing one of the standard
multilingual math
benchmarks (MGSM), we determine that several translation errors are present in the data. Furthermore, the lack of standardized answer extraction from LLM outputs further influences the final results. We propose a method for semi-automatic quality assurance to address the first issue at scale, and give recommendations to address the second one. Combining these two approaches we show that the aforementioned
language gap mostly disappears, leading to completely different conclusions from our research. We additionally release the corrected
dataset to the community (https://github.com/google-research-
datasets/MGSM-Rev2).
Read the full abstract ▾
All authors Jan-Thorsten Peter, David Vilar, Tobias Domhan, Dan Malkin, Markus Freitag
Computation and Language
ManufactuBERT: Efficient Continual Pretraining for Manufacturing
Robin Armingaud and Romaric Besançon
While large general-purpose
Transformer-based encoders excel at general
language understanding, their performance diminishes in specialized domains like manufacturing due to a lack of exposure to domain-specific terminology and semantics. In this paper, we address this gap by introducing ManufactuBERT, a RoBERTa model continually pretrained on a large-scale corpus curated for the manufacturing domain. We present a comprehensive data processing pipeline to create this corpus from web data, involving an initial domain-specific filtering step followed by a multi-stage deduplication process that removes redundancies. Our experiments show that ManufactuBERT establishes a new state-of-the-art on a range of manufacturing-related NLP tasks, outperforming strong specialized baselines. More importantly, we demonstrate that training on our carefully deduplicated corpus significantly accelerates convergence, leading to a 33% reduction in training time and computational cost compared to training on the non-deduplicated
dataset. The proposed pipeline offers a reproducible example for developing high-performing encoders in other specialized domains. We will release our model and curated corpus at https://huggingface.co/cea-list-ia.
Read the full abstract ▾
Computation and Language
A Toolbox for Improving Evolutionary Prompt Search
Daniel Grießhaber, Maximilian Kimmich, Johannes Maucher et al. (+1 more)
Evolutionary
prompt optimization has demonstrated effectiveness in refining
prompts for LLMs. However, existing approaches lack robust operators and efficient evaluation mechanisms. In this work, we propose several key improvements to evolutionary
prompt optimization that can partially generalize to
prompt optimization in general: 1) decomposing
evolution into distinct steps to enhance the
evolution and its control, 2) introducing an LLM-based judge to verify the evolutions, 3) integrating human feedback to refine the evolutionary operator, and 4) developing more efficient evaluation strategies that maintain performance while reducing computational overhead. Our approach improves both optimization quality and efficiency. We release our code, enabling
prompt optimization on new tasks and facilitating further research in this area.
Read the full abstract ▾
All authors Daniel Grießhaber, Maximilian Kimmich, Johannes Maucher, Ngoc Thang Vu
Computation and Language
Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
Grigory Kovalev and Mikhail Tikhomirov
This work investigates
distillation methods for large
language models (LLMs) with the goal of developing compact models that preserve high performance. Several existing approaches are reviewed, with a discussion of their respective strengths and limitations. An improved method based on the ShortGPT approach has been developed, building upon the idea of incorporating iterative evaluation of layer importance. At each step, importance is assessed by measuring performance degradation when individual layers are removed, using a set of representative
datasets. This process is combined with further training using a joint loss function based on KL divergence and mean squared error. Experiments on the Qwen2.5-3B model show that the number of layers can be reduced from 36 to 28 (resulting in a 2.47 billion parameter model) with only a 9.7% quality loss, and to 24 layers with an 18% loss. The findings suggest that the middle
transformer layers contribute less to inference, underscoring the potential of the proposed method for creating efficient models. The results demonstrate the effectiveness of iterative
distillation and fine-tuning, making the approach suitable for deployment in resource-limited settings.
Read the full abstract ▾
Computation and Language
Making Knowledge Accessible: Divergent Readability-Accuracy Strategies of Mistral and QWen in Biomedical Text Simplification
P. Bilha Githinji, Aikaterini Melliou, Zeming Liang et al. (+2 more)
The growing public demand for accessible biomedical information calls for scalable text simplification. While large
language models (LLMs) offer solutions, they too struggle with balancing improved readability against preservation of meaning. This report empirically compares how two LLMs - instruction-tuned Mistral-Small 3 24B and the
reasoning-augmented QWen2.5 32B- navigate this trade-off in biomedical text simplification, benchmarked against human performance. Our analysis highlights how each model applies distinct operational strategies when simplifying biomedical text. Mistral exhibits a tempered lexical simplification approach that consistently enhances readability across multiple metrics while preserving discourse fidelity (BERTScore: 0.91, statistically comparable to that of humans). In comparison, QWen also attains enhanced readability performance and a reasonable BERTScore of 0.89, but presents a disconnect in balancing between readability and accuracy. Additionally, a comprehensive correlation analysis of a suite of 21 metrics confirms strong functional redundancies in metrics and informs adaptation requirements.
Read the full abstract ▾
All authors P. Bilha Githinji, Aikaterini Melliou, Zeming Liang, Lian Zhang, Peiwu Qin
Computation and Language · Cross-listed from cs.IR
Grigory Kovalev, Natalia Loukachevitch, Mikhail Tikhomirov et al. (+2 more)
In this paper, we present a novel series of Russian information
retrieval datasets constructed from the "Did you know..." section of Russian Wikipedia. Our
datasets support a range of
retrieval tasks, including fact-checking,
retrieval-augmented generation, and full-document
retrieval, by leveraging interesting facts and their referenced Wikipedia articles annotated at the sentence level with graded relevance. We describe the methodology for
dataset creation that enables the expansion of existing Russian Information
Retrieval (IR) resources. Through extensive experiments, we extend the RusBEIR research by comparing lexical
retrieval models, such as BM25, with state-of-the-art neural architectures fine-tuned for Russian, as well as
multilingual models. Results of our experiments show that lexical methods tend to outperform neural models on full-document
retrieval, while neural approaches better capture lexical semantics in shorter texts, such as in fact-checking or fine-grained
retrieval. Using our newly created
datasets, we also analyze the impact of document length on
retrieval performance and demonstrate that combining
retrieval with neural reranking consistently improves results. Our contribution expands the resources available for Russian information
retrieval research and highlights the importance of accurate evaluation of
retrieval models to achieve optimal performance. All
datasets are publicly available at HuggingFace. To facilitate reproducibility and future research, we also release the full implementation on GitHub.
Read the full abstract ▾
All authors Grigory Kovalev, Natalia Loukachevitch, Mikhail Tikhomirov, Olga Babina, Pavel Mamaev
Computation and Language
Reasoning-Guided Claim Normalization for Noisy Multilingual Social Media Posts
Manan Sharma, Arya Suneesh, Manish Jain et al. (+7 more)
We address claim normalization for
multilingual misinformation detection - transforming noisy social media posts into clear, verifiable statements across 20
languages. The key contribution demonstrates how systematic decomposition of posts using Who, What, Where, When, Why and How questions enables robust cross-lingual transfer despite training exclusively on English data. Our methodology incorporates finetuning Qwen3-14B using LoRA with the provided
dataset after intra-post deduplication,
token-level recall filtering for semantic
alignment and
retrieval-augmented few-shot learning with contextual examples during inference. Our system achieves METEOR scores ranging from 41.16 (English) to 15.21 (Marathi), securing third rank on the English leaderboard and fourth rank for Dutch and Punjabi. The approach shows 41.3% relative improvement in METEOR over baseline configurations and substantial gains over existing methods. Results demonstrate effective cross-lingual generalization for Romance and Germanic
languages while maintaining semantic coherence across diverse linguistic structures.
Read the full abstract ▾
All authors Manan Sharma, Arya Suneesh, Manish Jain, Pawan Kumar Rajpoot, Prasanna Devadiga, Bharatdeep Hazarika, Ashish Shrivastava, Kishan Gurumurthy, Anshuman B Suresh, Aditya U Baliga
Computation and Language
Order-Level Attention Similarity Across Language Models: A Latent Commonality
Jinglin Liang, Jin Zhong, Shuangping Huang et al. (+5 more)
In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across
Language Models (LMs) share commonalities? While some works have investigated context aggregation or
attention weights in LMs, they typically focus on individual models or
attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level
Attention (OLA) derived from the order-wise decomposition of
Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at https://github.com/jinglin-liang/OLAS.
Read the full abstract ▾
All authors Jinglin Liang, Jin Zhong, Shuangping Huang, Yunqing Hu, Huiyuan Zhang, Huifang Li, Lixin Fan, Hanlin Gu
Computation and Language
UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
Mykyta Syromiatnikov and Victoria Ruvinskaya
Evaluating the real capabilities of large
language models in low-resource
languages still represents a challenge, as many existing
benchmarks focus on widespread tasks translated from English or evaluate only simple
language understanding. This paper introduces UA-Code-Bench, a new open-source
benchmark established for a thorough evaluation of
language models' code generation and competitive programming problem-solving abilities in Ukrainian. The
benchmark comprises 500 problems from the Eolymp platform, evenly distributed across five complexity levels from very easy to very hard. A diverse set of 13 leading proprietary and open-source models, generating Python solutions based on a one-shot
prompt, was evaluated via the dedicated Eolymp environment against hidden tests, ensuring code correctness. The obtained results reveal that even top-performing models, such as OpenAI o3 and GPT-5, solve only half of the problems, highlighting the challenge of code generation in low-resource natural
language. Furthermore, this research presents a comprehensive analysis of performance across various difficulty levels, as well as an assessment of solution uniqueness and computational efficiency, measured by both elapsed time and
memory consumption of the generated solutions. In conclusion, this work demonstrates the value of competitive programming
benchmarks in evaluating large
language models, especially in underrepresented
languages. It also paves the way for future research on
multilingual code generation and
reasoning-enhanced models. The
benchmark, data parsing, preparation, code generation, and evaluation scripts are available at https://huggingface.co/
datasets/NLPForUA/ua-code-bench.
Read the full abstract ▾
Computation and Language
Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies
Prasoon Varshney, Makesh Narsimhan Sreedhar, Liwei Jiang et al. (+2 more)
Large
language models (LLMs) are typically aligned to a universal set of safety and usage principles intended for broad public acceptability. Yet, real-world applications of LLMs often take place within organizational ecosystems shaped by distinctive corporate policies, regulatory requirements, use cases, brand guidelines, and ethical commitments. This reality highlights the need for rigorous and comprehensive evaluation of LLMs with pluralistic
alignment goals, an
alignment paradigm that emphasizes adaptability to diverse user values and needs. In this work, we present PLURALISTIC BEHAVIOR SUITE (PBSUITE), a dynamic evaluation suite designed to systematically assess LLMs' capacity to adhere to pluralistic
alignment specifications in multi-turn, interactive conversations. PBSUITE consists of (1) a diverse
dataset of 300 realistic LLM behavioral policies, grounded in 30 industries; and (2) a dynamic evaluation framework for stress-testing model compliance with custom behavioral specifications under
adversarial conditions. Using PBSUITE, We find that leading open- and closed-source LLMs maintain robust adherence to behavioral policies in single-turn settings (less than 4% failure rates), but their compliance weakens substantially in multi-turn
adversarial interactions (up to 84% failure rates). These findings highlight that existing model
alignment and safety moderation methods fall short in coherently enforcing pluralistic behavioral policies in real-world LLM interactions. Our work contributes both the
dataset and analytical framework to support future research toward robust and context-aware pluralistic
alignment techniques.
Read the full abstract ▾
All authors Prasoon Varshney, Makesh Narsimhan Sreedhar, Liwei Jiang, Traian Rebedea, Christopher Parisien
Computation and Language · Cross-listed from cs.CV
Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti et al. (+6 more)
Hallucinations in Large
Vision-
Language Models (LVLMs) remain a persistent challenge, often stemming from inadequate integration of visual information during
multimodal reasoning. A key cause is the model's over-reliance on textual priors and underutilization of visual cues, leading to outputs that are linguistically fluent but visually inaccurate. For example, given an image of an empty kitchen countertop, an LVLM might hallucinate a "bowl of fruit" or "cup of coffee", relying on
language associations rather than visual evidence. Most LVLMs incorporate visual features by appending them to the input stream of a pre-trained LLM and training on large-scale
vision-
language datasets. Our systematic analysis reveals that this strategy often leads to over-dependence on textual information due to the inherent bias of LLMs towards
language-dominant representations. This imbalance skews
attention towards the text over visual content, weakening the model's ability to ground outputs in visual inputs. To address this, we propose a simple yet effective visual feature incorporation method that encourages the model to learn visually-informed textual
embeddings distinct from those of the base LLM and promotes a more balanced
attention distribution. Experimental results across multiple
hallucination benchmarks demonstrate that our method significantly reduces
hallucinations and fosters more balanced
multimodal reasoning. Notably, our approach achieves substantial gains, including +9.33% on MMVP-MLLM, +2.99% on POPE-AOKVQA, up to +3.4% on Merlin, and +3% on the hard-data split of HallusionBench.
Read the full abstract ▾
All authors Aakriti Agrawal, Gouthaman KV, Rohith Aralikatti, Gauri Jagatap, Jiaxin Yuan, Sarvesh Baskar, Vijay Kamarshi, Andrea Fanelli, Furong Huang
Computation and Language · Cross-listed from cs.HC
Enhancing Public Speaking Skills in Engineering Students Through AI
Amol Harsh, Brainerd Prince, Siddharth Siddharth et al. (+4 more)
This research-to-practice full paper was inspired by the persistent challenge in effective communication among engineering students. Public speaking is a necessary skill for future engineers as they have to communicate technical knowledge with diverse stakeholders. While universities offer courses or workshops, they are unable to offer sustained and personalized training to students. Providing comprehensive feedback on both verbal and non-verbal aspects of public speaking is time-intensive, making consistent and individualized assessment impractical. This study integrates research on verbal and non-verbal cues in public speaking to develop an AI-driven assessment model for engineering students. Our approach combines
speech analysis, computer
vision, and sentiment detection into a multi-modal AI system that provides assessment and feedback. The model evaluates (1) verbal communication (pitch, loudness, pacing, intonation), (2) non-verbal communication (facial expressions, gestures, posture), and (3) expressive coherence, a novel integration ensuring
alignment between
speech and body
language. Unlike previous systems that assess these aspects separately, our model fuses multiple modalities to deliver personalized, scalable feedback. Preliminary testing demonstrated that our AI-generated feedback was moderately aligned with expert evaluations. Among the state-of-the-art AI models evaluated, all of which were Large
Language Models (LLMs), including Gemini and OpenAI models, Gemini Pro emerged as the best-performing, showing the strongest agreement with human annotators. By eliminating reliance on human evaluators, this AI-driven public speaking trainer enables repeated practice, helping students naturally align their
speech with body
language and
emotion, crucial for impactful and professional communication.
Read the full abstract ▾
All authors Amol Harsh, Brainerd Prince, Siddharth Siddharth, Deepan Raj Prabakar Muthirayan, Kabir S Bhalla, Esraaj Sarkar Gupta, Siddharth Sahu
Computation and Language
Acquiring Common Chinese Emotional Events Using Large Language Model
Ya Wang, Guangzheng Zhu, Cungen Cao et al. (+3 more)
Knowledge about emotional events is an important kind of knowledge which has been applied to improve the effectiveness of different applications. However, emotional events cannot be easily acquired, especially common or generalized emotional events that are context-independent. The goal of this paper is to obtain common emotional events in Chinese
language such as "win a prize" and "be criticized". Our approach begins by collecting a comprehensive list of Chinese emotional event indicators. Then, we generate emotional events by prompting a Chinese large
language model (LLM) using these indicators. To ensure the quality of these emotional events, we train a filter to discard invalid generated results. We also classify these emotional events as being positive events and negative events using different techniques. Finally, we harvest a total of 102,218 high-quality common emotional events with sentiment polarity labels, which is the only large-scale commonsense knowledge base of emotional events in Chinese
language. Intrinsic evaluation results show that the proposed method in this paper can be effectively used to acquire common Chinese emotional events. An extrinsic use case also demonstrates the strong potential of common emotional events in the field of
emotion cause extraction (ECE). Related resources including emotional event indicators and emotional events will be released after the publication of this paper.
Read the full abstract ▾
All authors Ya Wang, Guangzheng Zhu, Cungen Cao, Jingjing Li, He Li, Xin Huang