Computation and Language · Cross-listed from cs.IR
Chained Prompting for Better Systematic Review Search Strategies
Fatima Nasser, Fouad Trad, Ammar Mohanna et al. (+2 more)
Systematic reviews require the use of rigorously designed search strategies to ensure both comprehensive
retrieval and minimization of bias. Conventional manual approaches, although methodologically systematic, are resource-intensive and susceptible to subjectivity, whereas heuristic and automated techniques frequently under-perform in recall unless supplemented by extensive expert input. We introduce a Large
Language Model (LLM)-based chained
prompt engineering framework for the automated development of search strategies in systematic reviews. The framework replicates the procedural structure of manual search design while leveraging LLMs to decompose review objectives, extract and formalize PICO elements, generate conceptual representations, expand terminologies, and synthesize Boolean queries. In addition to query construction, the framework exhibits superior performance in generating well-structured PICO elements relative to existing methods, thereby strengthening the foundation for high-recall search strategies. Evaluation on a subset of the LEADSInstruct
dataset demonstrates that the framework attains a 0.9 average recall. These results significantly exceed the performance of existing approaches. Error analysis further highlights the critical role of precise objective specification and terminological
alignment in optimizing
retrieval effectiveness. These findings confirm the capacity of LLM-based pipelines to yield transparent, reproducible, and high-performing search strategies, and highlight their potential as scalable instruments for supporting evidence synthesis and evidence-based practice.
Read the full abstract ▾
All authors Fatima Nasser, Fouad Trad, Ammar Mohanna, Ghada El-Hajj Fuleihan, Ali Chehab
Computation and Language · Cross-listed from cs.IR
ChunkNorris: A High-Performance and Low-Energy Approach to PDF Parsing and Chunking
Mathieu Ciancone, Clovis Varangot-Reille and Marion Schaeffer
In
Retrieval-Augmented Generation applications, the Information
Retrieval part is central as it provides the contextual information that enables a Large
Language Model to generate an appropriate and truthful response. High quality parsing and chunking are critical as efficient data
segmentation directly impacts downstream tasks, i.e. Information
Retrieval and answer generation. In this paper, we introduce ChunkNorris, a novel heuristic-based technique designed to optimise the parsing and chunking of PDF documents. Our approach does not rely on machine learning and employs a suite of simple yet effective heuristics to achieve high performance with minimal computational overhead. We demonstrate the efficiency of ChunkNorris through a comprehensive
benchmark against existing parsing and chunking methods, evaluating criteria such as execution time, energy consumption, and
retrieval accuracy. We propose an open-access
dataset to produce our results. ChunkNorris outperforms baseline and more advanced techniques, offering a practical and efficient alternative for Information
Retrieval tasks. Therefore, this research highlights the potential of heuristic-based methods for real-world, resource-constrained RAG use cases.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Finetuning Large Language Models for Automated Depression Screening in Nigerian Pidgin English: GENSCORE Pilot Study
Isaac Iyinoluwa Olufadewa, Miracle Ayomikun Adesina, Ezekiel Ayodeji Oladejo et al. (+3 more)
Depression is a major contributor to the mental-health burden in Nigeria, yet screening coverage remains limited due to low access to clinicians, stigma, and
language barriers. Traditional tools like the Patient Health Questionnaire-9 (PHQ-9) were validated in high-income countries but may be linguistically or culturally inaccessible for low- and middle-income countries and communities such as Nigeria where people communicate in Nigerian Pidgin and more than 520 local
languages. This study presents a novel approach to automated depression screening using fine-tuned large
language models (LLMs) adapted for conversational Nigerian Pidgin. We collected a
dataset of 432 Pidgin-
language audio responses from Nigerian young adults aged 18-40 to
prompts assessing psychological experiences aligned with PHQ-9 items, performed transcription, rigorous preprocessing and annotation, including semantic labeling, slang and idiom interpretation, and PHQ-9 severity scoring. Three LLMs - Phi-3-mini-4k-instruct, Gemma-3-4B-it, and GPT-4.1 - were fine-tuned on this annotated
dataset, and their performance was evaluated quantitatively (accuracy, precision and semantic
alignment) and qualitatively (clarity, relevance, and cultural appropriateness). GPT-4.1 achieved the highest quantitative performance, with 94.5% accuracy in PHQ-9 severity scoring prediction, outperforming Gemma-3-4B-it and Phi-3-mini-4k-instruct. Qualitatively, GPT-4.1 also produced the most culturally appropriate, clear, and contextually relevant responses. AI-mediated depression screening for underserved Nigerian communities. This work provides a foundation for deploying conversational mental-health tools in linguistically diverse, resource-constrained environments.
Read the full abstract ▾
All authors Isaac Iyinoluwa Olufadewa, Miracle Ayomikun Adesina, Ezekiel Ayodeji Oladejo, Uthman Babatunde Usman, Owen Kolade Adeniyi, Matthew Tolulope Olawoyin
Computation and Language · Cross-listed from cs.AI
Shuqi Liu, Bowei He, Chen Ma et al. (+1 more)
Large
language models (LLMs) typically enhance their performance through either the
retrieval of semantically similar information or the improvement of their
reasoning capabilities. However, a significant challenge remains in effectively integrating both
retrieval and
reasoning strategies to optimize LLM performance. In this paper, we introduce a
reasoning-aware knowledge
retrieval method that enriches LLMs with information aligned to the logical structure of conversations, moving beyond surface-level semantic similarity. We follow a coarse-to-fine approach for knowledge
retrieval. First, we identify a contextually relevant sub-region of the knowledge base, ensuring that all sentences within it are relevant to the context topic. Next, we refine our search within this sub-region to extract knowledge that is specifically relevant to the
reasoning process. Throughout both phases, we employ the Monte Carlo Tree Search-inspired search method to effectively navigate through knowledge sentences using common keywords. Experiments on two multi-turn dialogue
datasets demonstrate that our knowledge
retrieval approach not only aligns more closely with the underlying
reasoning in human conversations but also significantly enhances the diversity of the retrieved knowledge, resulting in more informative and creative responses.
Read the full abstract ▾
All authors Shuqi Liu, Bowei He, Chen Ma, Linqi Song
Computation and Language
Learning to Prioritize IT Tickets: A Comparative Evaluation of Embedding-based Approaches and Fine-Tuned Transformer Models
Minh Tri LÊ and Ali Ait-Bachir
Prioritizing service tickets in IT Service Management (ITSM) is critical for operational efficiency but remains challenging due to noisy textual inputs, subjective writing styles, and pronounced class imbalance. We evaluate two families of approaches for ticket prioritization:
embedding-based pipelines that combine dimensionality reduction, clustering, and classical classifiers, and a fine-tuned
multilingual transformer that processes both textual and numerical features.
Embedding-based methods exhibit limited generalization across a wide range of thirty configurations, with clustering failing to uncover meaningful structures and supervised models highly sensitive to
embedding quality. In contrast, the proposed
transformer model achieves substantially higher performance, with an average F1-score of 78.5% and weighted Cohen's kappa values of nearly 0.80, indicating strong
alignment with true labels. These results highlight the limitations of generic
embeddings for ITSM data and demonstrate the effectiveness of domain-adapted
transformer architectures for operational ticket prioritization.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SI
Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging
Pinxian Lu, Nimra Ishfaq, Emma Win et al. (+5 more)
Online harassment is a widespread social and public health concern, yet most computational approaches for detecting and addressing harassment focus on publicly visible social media content rather than private messaging environments. Private conversations present unique challenges because harmful interactions often unfold through context-dependent, multi-turn exchanges, while victims may lack timely support during moments of harassment. In this study, we investigate how large
language models (LLMs) can support both the detection of and response to online harassment in private messaging. Using a
dataset of 80,053 Instagram direct messages donated by 26 adolescents aged 12-18, including youth with suicide risk factors, we first construct a human-labeled
dataset of online harassment in private conversations and develop a context-aware cascading LLM classification pipeline. The proposed pipeline outperforms baseline toxicity classifiers trained primarily on public social media data. We then develop a victim-centered response framework that produces context-sensitive and psychologically-grounded AI-generated responses to online harassment messages. Human evaluators perceived the AI-generated responses as significantly more helpful than the original participant responses (95% CI: 0.767–0.815, p < .001), particularly in terms of emotional support and de-escalation. Our findings highlight the potential of context-aware and victim-centered AI systems to provide just-in-time support during harassment in private messaging environments.
Read the full abstract ▾
All authors Pinxian Lu, Nimra Ishfaq, Emma Win, Morgan Rose, Sierra R Strickland, Candice L Biernesser, Jamie Zelazny, Munmun De Choudhury
Computation and Language · Cross-listed from cs.CY
Writing in Symbiosis: Mapping Human Creative Agency in the AI Era
Vivan Doshi and Mengyuan Li
The proliferation of Large
Language Models (LLMs) raises a critical question about what it means to be human when we share an increasingly symbiotic relationship with persuasive and creative machines. This paper examines patterns of human-AI coevolution in creative writing, investigating how human craft and agency are adapting alongside machine capabilities. We challenge the prevailing notion of stylistic homogenization by examining diverse patterns in longitudinal writing data. Using a large-scale corpus spanning the pre- and post-LLM era, we observe patterns suggestive of a "Dual-Track
Evolution": thematic convergence around AI-related topics, coupled with structured stylistic differentiation. Our analysis reveals three emergent adaptation patterns: authors showing increased similarity to AI style, those exhibiting decreased similarity, and those maintaining stylistic stability while engaging with AI-related themes. This Creative Archetype Map illuminates how authorship is coevolving with AI, contributing to discussions about human-AI collaboration, detection challenges, and the preservation of creative diversity.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Šimon Sedláček, Sara Barahona, Bolaji Yusuf et al. (+9 more)
Reliable assessment of the abilities of large audio
language models (LALMs) is essential to advancing the state of the art. As
benchmarks rapidly evolve to incorporate complex
reasoning and subjective tasks, they increasingly necessitate open-ended responses from LALMs. We present Open-ended Response Correctness Assessment (ORCA) – a reliable and lightweight model-based approach for answer correctness and disagreement modeling. We employ a three-stage annotation pipeline combining human judgment, structured feedback, and human-AI correction, yielding 9,663 annotations across 3,699 question-answer pairs from 15 LALMs on three audio understanding and
reasoning benchmarks (achieving a Krippendorff's alpha of 0.82). Our experiments employing curriculum learning show that ORCA models achieve a Spearman correlation of 0.91 with average human correctness ratings on seen
benchmarks and generalize to unseen
benchmarks with a score of 0.85, outperforming several LLM judge baselines including Gemini 2.5 Flash. Furthermore, we demonstrate that ORCA's predicted variance correlates strongly with human disagreement, allowing it to effectively identify problematic
benchmark items.
Read the full abstract ▾
All authors Šimon Sedláček, Sara Barahona, Bolaji Yusuf, Laura Herrera-Alarcón, Santosh Kesiraju, Cecilia Bolaños, Alicia Lozano-Diez, Sathvik Udupa, Fernando López, Allison Ferner, Ramani Duraiswami, Jan Černocký
Computation and Language · Cross-listed from cs.LG
Resolving Conflicts in Lifelong Learning via Aligning Updates in Subspaces
Yueer Zhou, Yichen Wu and Ying Wei
Low-Rank Adaptation (LoRA) enables efficient Continual Learning but often suffers from catastrophic forgetting due to destructive interference between tasks. Our analysis reveals that this degradation is primarily driven by antagonistic directional updates where new task gradients directly oppose the historical weight trajectory. To address this, we propose PS-LoRA (Parameter Stability LoRA), a framework designed to resolve conflicts by aligning updates within the optimization subspace. Our approach employs a dual-regularization objective that penalizes conflicting directions and constrains magnitude deviations to ensure consistency with prior knowledge. Additionally, we implement a magnitude-based merging strategy to consolidate sequential adapters into a robust representation without retraining. Experiments on NLP and
Vision benchmarks show that PS-LoRA outperforms state-of-the-art methods by preserving the stability of learned representations while efficiently adapting to new domains.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Retrieval-Augmented Few-Shot Prompting Versus Fine-Tuning for Code Vulnerability Detection
Fouad Trad and Ali Chehab
Few-shot prompting has emerged as a practical alternative to fine-tuning for leveraging the capabilities of large
language models (LLMs) in specialized tasks. However, its effectiveness depends heavily on the selection and quality of in-context examples, particularly in complex domains. In this work, we examine
retrieval-augmented prompting as a strategy to improve few-shot performance in code vulnerability detection, where the goal is to identify one or more security-relevant weaknesses present in a given code snippet from a predefined set of vulnerability categories. We perform a systematic evaluation using the Gemini-1.5-Flash model across three approaches: (1) standard few-shot prompting with randomly selected examples, (2)
retrieval-augmented prompting using semantically similar examples, and (3)
retrieval-based labeling, which assigns labels based on retrieved examples without model inference. Our results show that
retrieval-augmented prompting consistently outperforms the other prompting strategies. At 20 shots, it achieves an F1 score of 74.05% and a partial match accuracy of 83.90%. We further compare this approach against zero-shot prompting and several fine-tuned models, including Gemini-1.5-Flash and smaller open-source models such as DistilBERT, DistilGPT2, and CodeBERT.
Retrieval-augmented prompting outperforms both zero-shot (F1 score: 36.35%, partial match accuracy: 20.30%) and fine-tuned Gemini (F1 score: 59.31%, partial match accuracy: 53.10%), while avoiding the training time and cost associated with model fine-tuning. On the other hand, fine-tuning CodeBERT yields higher performance (F1 score: 91.22%, partial match accuracy: 91.30%) but requires additional training, maintenance effort, and resources.
Read the full abstract ▾
Computation and Language
Alleviating Choice Supportive Bias in LLM with Reasoning Dependency Generation
Nan Zhuang, Wenshuo Wang, Lekai Qian et al. (+3 more)
Recent studies have demonstrated that some Large
Language Models exhibit choice-supportive bias (CSB) when performing evaluations, systematically favoring their chosen options and potentially compromising the objectivity of AI-assisted decision making. While existing debiasing approaches primarily target demographic and social biases, methods for addressing cognitive biases in LLMs remain largely unexplored. In this work, we present the first solution to address CSB through
Reasoning Dependency Generation (RDG), a novel framework for generating unbiased
reasoning data to mitigate choice-supportive bias through fine-tuning. RDG automatically constructs balanced
reasoning QA pairs, explicitly (un)modeling the dependencies between choices, evidences, and justifications. Our approach is able to generate a large-scale
dataset of QA pairs across domains, incorporating Contextual Dependency Data and Dependency Decouple Data. Experiments show that LLMs fine-tuned on RDG-generated data demonstrate a 81.5% improvement in
memory-based experiments and 94.3% improvement in the evaluation-based experiment, while maintaining similar performance on standard BBQ
benchmarks. This work pioneers an approach for addressing cognitive biases in LLMs and contributes to the development of more reliable AI-assisted decision support systems.
Read the full abstract ▾
All authors Nan Zhuang, Wenshuo Wang, Lekai Qian, Yuxiao Wang, Boyu Cao, Qi Liu
Computation and Language
Watermarks for Embeddings-as-a-Service Large Language Models
Anudeex Shetty
Large
Language Models (LLMs) have demonstrated exceptional capabilities in natural
language understanding and generation. Based on these LLMs, businesses have started to provide
Embeddings-as-a-Service (EaaS), offering feature extraction capabilities (in the form of text
embeddings) that benefit downstream natural
language processing tasks. However, prior research has demonstrated that EaaS is vulnerable to imitation attacks, where an attacker clones the service's model in a black-box manner without access to the model's internal workings. In response, watermarks have been added to the text
embeddings to protect the intellectual property of EaaS providers by allowing them to check for model ownership. This thesis focuses on defending against imitation attacks by investigating EaaS watermarks. To achieve this goal, we unveil novel attacks and propose and validate new watermarking techniques. Firstly, we show that existing EaaS watermarks can be removed through paraphrasing the input text when attackers clone the model during imitation attacks. Our study illustrates that paraphrasing can effectively bypass current state-of-the-art EaaS watermarks across various attack setups (including different paraphrasing techniques and models) and
datasets in most instances. This demonstrates a new vulnerability in recent EaaS watermarking techniques. Subsequently, as a countermeasure, we propose a novel watermarking technique, WET (Watermarking EaaS with Linear Transformation), which employs linear transformation of the
embeddings. Watermark verification is conducted by applying a reverse transformation and comparing the similarity between recovered and original
embeddings. We demonstrate its robustness against paraphrasing attacks with near-perfect verifiability. We conduct detailed ablation studies to assess the significance of each component and hyperparameter in WET.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings
Tabia Tanzin Prama, Christopher M. Danforth and Peter Sheridan Dodds
Recent advances enable Large
Language Models (LLMs) to generate AI personas, yet their lack of deep contextual, cultural, and emotional understanding poses a significant limitation. This study quantitatively compared human responses with those of eight LLM-generated social personas (e.g., Male, Female, Muslim, Political Supporter) within a low-resource environment like Bangladesh, using culturally specific questions. Results show human responses significantly outperform all LLMs in answering questions, and across all
matrices of persona perception, with particularly large gaps in empathy and credibility. Furthermore, LLM-generated content exhibited a systematic bias along the lines of the “Pollyanna Principle”, scoring measurably higher in positive sentiment ($\Phi_{avg} = 5.99$ for LLMs vs. $5.60$ for Humans). These findings suggest that LLM personas do not accurately reflect the authentic experience of real people in resource-scarce environments. It is essential to validate LLM personas against real-world human data to ensure their
alignment and reliability before deploying them in social science research.
Read the full abstract ▾
Computation and Language
OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion
Sai Koneru, Matthias Huck and Jan Niehues
There has been significant progress in open-source text-only translation large
language models (LLMs) with better
language coverage and quality. However, these models can be only used in cascaded pipelines for
speech translation (ST), performing automatic
speech recognition first followed by translation. This introduces additional latency, which is particularly critical in simultaneous ST (SimulST), and prevents the model from exploiting
multimodal context, such as images, which can aid disambiguation. Pretrained
multimodal foundation models (MMFMs) already possess strong perception and
reasoning capabilities across multiple modalities, but generally lack the
multilingual coverage and specialized translation performance of dedicated translation LLMs. To build an effective
multimodal translation system, we propose an end-to-end approach that fuses MMFMs with translation LLMs. We introduce a novel fusion strategy that connects hidden states from multiple layers of a pretrained MMFM to a translation LLM, enabling joint end-to-end training. The resulting model, OmniFusion, built on Omni 2.5-7B as the MMFM and SeedX PPO-7B as the translation LLM, can perform
speech-to-text,
speech-and-image-to-text, and text-and-image-to-text translation. Experiments demonstrate that OmniFusion effectively leverages both audio and visual inputs, achieves a 1-second latency reduction in SimulST compared to cascaded pipelines and also improves the overall translation quality\footnote{Code is available at https://github.com/saikoneru/OmniFusion}.
Read the full abstract ▾
Computation and Language
Minimal-Edit Instruction Tuning for Low-Resource Indic GEC
Akhil Rajeev P
Grammatical error correction for Indic
languages faces limited supervision, diverse scripts, and rich morphology. We propose an augmentation-free setup that uses instruction-tuned large
language models and conservative decoding. A 12B GEMMA 3 model is instruction-tuned in bnb 4-bit precision with parameter-efficient fine-tuning (PEFT) and Alpaca-style formatting. Decoding follows a deterministic, constraint-aware procedure with a lightweight normaliser that encourages minimal, meaning-preserving edits. We operationalise inference, subsequent to instruction fine-tuning (IFT), via a fixed,
language-specific
prompt directly synthesised from a deterministic error classifier's taxonomy, label distributions, and precedence ordering computed on the training corpus. Under the official untuned GLEU evaluation, the system scores 92.41 on Malayalam, sixth overall, and 81.44 on Hindi, third overall. These results indicate that classifier-informed
prompt design, adapter-based instruction tuning, and deterministic decoding provide a reproducible and a computationally efficient alternative to augmentation-centred pipelines for Indic GEC. The approach also motivates future work on stronger morphosyntactic constraints and human-centred evaluation of conservative edits.
Read the full abstract ▾
Computation and Language
Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis
Matej Klemen, Tjaša Arčon, Luka Terčon et al. (+2 more)
Empirical grammar research has become increasingly data-driven, but the systematic analysis of annotated corpora still requires substantial methodological and technical effort. We explore how agentic large
language models (LLMs) can streamline this process by
reasoning over annotated corpora and producing interpretable, data-grounded answers to linguistic questions. We introduce an agentic framework for corpus-grounded grammatical analysis that integrates concepts such as natural-
language task interpretation, code generation, and data-driven
reasoning. As a proof of concept, we apply it to Universal Dependencies (UD) corpora, testing it on
multilingual grammatical tasks inspired by the World Atlas of
Language Structures (WALS). The evaluation spans 13 word-order features and over 170
languages, assessing system performance across three complementary dimensions - dominant-order accuracy, order-coverage completeness, and distributional fidelity - which reflect how well the system generalizes, identifies, and quantifies word-order variations. The results demonstrate the feasibility of combining LLM
reasoning with structured linguistic data, offering a first step toward interpretable, scalable automation of corpus-based grammatical inquiry.
Read the full abstract ▾
All authors Matej Klemen, Tjaša Arčon, Luka Terčon, Marko Robnik-Šikonja, Kaja Dobrovoljc
Computation and Language
Tree Matching Networks for Natural Language Inference: Parameter-Efficient Semantic Understanding via Dependency Parse Trees
Jason Lunder
In creating sentence
embeddings for Natural
Language Inference (NLI) tasks, using
transformer-based models like BERT leads to high accuracy, but require hundreds of millions of parameters. These models take in sentences as a sequence of
tokens, and learn to encode the meaning of the sequence into
embeddings such that those
embeddings can be used reliably for NLI tasks. Essentially, every word is considered against every other word in the sequence, and the
transformer model is able to determine the relationships between them, entirely from scratch. However, a model that accepts explicit linguistic structures like dependency parse trees may be able to leverage prior encoded information about these relationships, without having to learn them from scratch, thus improving learning efficiency. To investigate this, we adapt Graph Matching Networks (GMN) to operate on dependency parse trees, creating Tree Matching Networks (TMN). We compare TMN to a BERT based model on the SNLI entailment task and on the SemEval similarity task. TMN is able to achieve significantly better results with a significantly reduced
memory footprint and much less training time than the BERT based model on the SNLI task, while both models struggled to preform well on the SemEval. Explicit structural representations significantly outperform sequence-based models at comparable scales, but current aggregation methods limit scalability. We propose multi-headed
attention aggregation to address this limitation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Measuring What LLMs Think They Do: SHAP Faithfulness and Deployability on Financial Tabular Classification
Saeed AlMarri, Mathieu Ravaut, Kristof Juhasz et al. (+3 more)
Large
Language Models (LLMs) have attracted significant
attention for classification tasks, offering a flexible alternative to trusted classical machine learning models like LightGBM through zero-shot prompting. However, their reliability for structured tabular data remains unclear, particularly in high stakes applications like financial risk assessment. Our study systematically evaluates LLMs and generates their SHAP values on financial classification tasks. Our analysis shows a divergence between LLMs self-explanation of feature impact and their SHAP values, as well as notable differences between LLMs and LightGBM SHAP values. These findings highlight the limitations of LLMs as standalone classifiers for structured financial modeling, but also instill optimism that improved explainability mechanisms coupled with few-shot prompting will make LLMs usable in risk-sensitive domains.
Read the full abstract ▾
All authors Saeed AlMarri, Mathieu Ravaut, Kristof Juhasz, Gautier Marti, Hamdan Al Ahbabi, Ibrahim Elfadel
Computation and Language · Cross-listed from cs.LG
ThetaEvolve: Test-time Learning on Open Problems
Yiping Wang, Shao-Rong Su, Zhiyuan Zeng et al. (+13 more)
Recent advances in large
language models (LLMs) have enabled breakthroughs in mathematical discovery, exemplified by AlphaEvolve, a closed-source system that evolves programs to improve bounds on open problems. However, it relies on ensembles of frontier LLMs to achieve new bounds and is a pure inference system that models cannot internalize the evolving strategies. We introduce ThetaEvolve, an open-source framework that simplifies and extends AlphaEvolve to efficiently scale both in-context learning and Reinforcement Learning (RL) at test time, allowing models to continually learn from their experiences in improving open optimization problems. ThetaEvolve features a single LLM, a large program database for enhanced exploration, batch sampling for higher throughput, lazy penalties to discourage stagnant outputs, and optional reward shaping for stable training signals, etc. ThetaEvolve is the first evolving framework that enable a small open-source model, like DeepSeek-R1-0528-Qwen3-8B, to achieve new best-known bounds on open problems (circle packing and first auto-correlation inequality) mentioned in AlphaEvolve. Besides, across two models and four open tasks, we find that ThetaEvolve with RL at test-time consistently outperforms inference-only baselines, and the model indeed learns evolving capabilities, as the RL-trained checkpoints demonstrate faster progress and better final performance on both trained target task and other unseen tasks. We release our code publicly: https://github.com/ypwang61/ThetaEvolve
Read the full abstract ▾
All authors Yiping Wang, Shao-Rong Su, Zhiyuan Zeng, Eva Xu, Liliang Ren, Xinyu Yang, Zeyi Huang, Xuehai He, Luyao Ma, Baolin Peng, Hao Cheng, Pengcheng He, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, Yelong Shen
Computation and Language
MegaChat: A Synthetic Persian Q&A Dataset for High-Quality Sales Chatbot Evaluation
Mahdi Rahmani, AmirHossein Saffari and Reyhane Rahmani
Small and medium-sized enterprises (SMEs) in Iran increasingly leverage Telegram for sales, where real-time engagement is essential for conversion. However, developing AI-driven chatbots for this purpose requires large, high-quality question-and-answer (Q&A)
datasets, which are typically expensive and resource-intensive to produce, especially for low-resource
languages like Persian. In this paper, we introduce MegaChat, the first fully synthetic Persian Q&A
dataset designed to evaluate intelligent sales chatbots in Telegram-based e-commerce. We propose a novel, automated multi-
agent architecture that generates persona-aware Q&A pairs by collecting data from active Telegram shopping channels. The system employs specialized
agents for question generation, validation, and refinement, ensuring the production of realistic and diverse conversational data. To evaluate answer generation, we compare three classic
retrieval-augmented generation (RAG) models with our advanced agentic system, which features multi-query
retrieval, reranking, and persona-aligned response synthesis. Using GPT-5.1 for evaluation across six quality dimensions, our results show that the agentic architecture outperformed traditional RAG models in 4 out of 5 diverse channels, demonstrating its ability to generate scalable, high-quality
datasets without relying on expensive human annotation or complex fine-tuning. MegaChat provides SMEs with an efficient, cost-effective solution for building intelligent customer engagement systems in specialized commercial domains, enabling advancements in
multilingual conversational AI for low-resource
languages. Download: https://github.com/MegaChat-Tech/MegaChat-
DataSet
Read the full abstract ▾
Computation and Language
Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
Jian Li, Shenglin Yin, Yujia Zhang et al. (+4 more)
Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. Recent research has increasingly focused on the role of
token importance in improving DPO effectiveness. It is observed that identical or semantically similar content (defined as ambiguous content) frequently appears within the preference pairs. We hypothesize that the presence of ambiguous content during DPO training may introduce ambiguity, thereby limiting further improvements in
alignment. Through mathematical analysis and proof-of-concept experiments, we reveal that ambiguous content may potentially introduce ambiguities, thereby degrading performance. To address this issue, we introduce Ambiguity Awareness Optimization (AAO), a simple yet effective approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. Through extensive experiments, we demonstrate that AAO consistently and significantly surpasses state-of-the-art approaches in performance, without markedly increasing response length, across multiple model scales and widely adopted
benchmark datasets, including AlpacaEval 2, MT-Bench, and Arena-Hard. Specifically, AAO outperforms DPO by up to 8.9 points on AlpacaEval 2 and achieves an improvement of by up to 15.0 points on Arena-Hard.
Read the full abstract ▾
All authors Jian Li, Shenglin Yin, Yujia Zhang, Alan Zhao, Xi Chen, Xiaohui Zhou, Pengfei Xu
Computation and Language · Cross-listed from cs.HC
Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking
Li Siyan, Jason Zhang, Akash Maharaj et al. (+2 more)
Novice and expert users have different systematic preferences in task-oriented dialogues. However, whether catering to these preferences actually improves user experience and task performance remains understudied. To investigate the effects of expertise-based personalization, we first built a version of an enterprise AI assistant with passive personalization. We then conducted a user study where participants completed timed exams, aided by the two versions of the AI assistant. Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception, but reveal task-specific limitations that can be addressed through providing more user agency. These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.
Read the full abstract ▾
All authors Li Siyan, Jason Zhang, Akash Maharaj, Yuanming Shi, Yunyao Li
Computation and Language
Alexander Sergeev and Evgeny Kotelnikov
Modern large
language models become
multimodal, analyzing various data formats like text and images. While fine-tuning is effective for adapting these
multimodal language models (MLMs) to downstream tasks, full fine-tuning is computationally expensive. Parameter-Efficient Fine-Tuning (PEFT) methods address this by training only a small portion of model weights. However, MLMs are difficult to interpret, making it challenging to identify which components are most effective for training to balance efficiency and performance. We propose an
attention-based interpretability method for MLMs by analyzing
attention scores relative to image
tokens. The core idea is to identify
attention heads that focus on image key objects. We utilize this information to select optimal model components for PEFT in
multimodal models. Our contributions include a method for identifying
attention heads associated with image key objects, its application to PEFT for image captioning, and the creation of a new
dataset containing images, key object masks, and their textual descriptions. We conducted experiments on MLMs with 2-3 billion parameters to validate the method's effectiveness. By calculating Head Impact (HI) scores we quantify an
attention head's focus on key objects, indicating its significance in image understanding. Our fine-tuning experiments demonstrate that adapting layers with the highest HI scores leads to the most significant shifts in metrics compared to pre-trained, randomly selected, or lowest-HI-score layers. This indicates that fine-tuning a small percentage (around 0.01%) of parameters in these crucial layers can substantially influence image understanding capabilities.
Read the full abstract ▾
Computation and Language
Scaling HuBERT for African Languages: From Base to Large and XL
Antoine Caubrière and Elodie Gauthier
Despite recent progress in
multilingual speech processing, African
languages remain under-represented in both research and deployed systems, particularly when it comes to strong, open-weight encoders that transfer well under low-resource supervision. Self-supervised learning has proven especially promising in such settings, yet most publicly released models targeting African
speech remain at BASE scale, leaving unanswered whether larger encoders, trained exclusively on Africa-centric audio, offer tangible benefits and how model capacity interacts with data composition. This work addresses that gap by introducing SSA-HuBERT-Large (317M parameters) and SSA-HuBERT-XL (964M parameters), the first large models trained solely on African
speech, alongside a BASE size counterpart. We release these models as open weights: see https://huggingface.co/collections/Orange/african-
speech-foundation-models. By conducting a carefully controlled experimental study focused exclusively on Sub-Saharan
languages, covering automatic
speech recognition (ASR) and
language identification (LID) tasks, we demonstrate that larger architectures significantly improve performance by effectively leveraging large audio
datasets.
Read the full abstract ▾
Computation and Language
Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach
Shuqi Liu, Han Wu, Guanzhi Deng et al. (+3 more)
Knowledge-enhanced text generation aims to enhance the quality of generated text by utilizing internal or external knowledge sources. While
language models have demonstrated impressive capabilities in generating coherent and fluent text, the lack of interpretability presents a substantial obstacle. The limited interpretability of generated text significantly impacts its practical usability, particularly in knowledge-enhanced text generation tasks that necessitate reliability and explainability. Existing methods often employ domain-specific knowledge retrievers that are tailored to specific data characteristics, limiting their generalizability to diverse data types and tasks. To overcome this limitation, we directly leverage the two-tier architecture of structured knowledge, consisting of high-level entities and low-level knowledge triples, to design our task-agnostic structured knowledge hunter. Specifically, we employ a local-global interaction scheme for structured knowledge representation learning and a hierarchical
transformer-based pointer network as the backbone for selecting relevant knowledge triples and entities. By combining the strong generative ability of
language models with the high faithfulness of the knowledge hunter, our model achieves high interpretability, enabling users to comprehend the model output generation process. Furthermore, we empirically demonstrate the effectiveness of our model in both internal knowledge-enhanced table-to-text generation on the RotoWireFG
dataset and external knowledge-enhanced dialogue response generation on the KdConv
dataset. Our task-agnostic model outperforms state-of-the-art methods and corresponding
language models, setting new standards on the
benchmark.
Read the full abstract ▾
All authors Shuqi Liu, Han Wu, Guanzhi Deng, Jianshu Chen, Xiaoyang Wang, Linqi Song