Computation and Language
DCMM-SQL: Automated Data-Centric Pipeline and Multi-Model Collaboration Training for Text-to-SQL Model
Yuanzhen Xie, Liu Ye, Jiqun Chu et al. (+5 more)
Text-to-SQL tasks have gained attractive improvements since the release of ChatGPT. Among them,
agent-based frameworks have been widely used in this field. However, the impact of data-centric strategies on text-to-SQL tasks has rarely been explored. In this paper, we systemically design a fully automated data-centric pipeline for text-to-SQL tasks, including adaptive data repair, which can automatically find and fix errors in the training
dataset; and error data augmentation, where we specifically diffuse and enhance erroneous data predicted by the initially trained models. Meanwhile, we propose a Multi-Model collaboration training schema, aiming to train multiple models with different augmented data, enabling them to possess distinct capabilities and work together to complement each other, because it has been found that the capability of a single fine-tuned model is very limited. Furthermore, we utilize an ensemble strategy to integrate the capabilities of multiple models to solve a multiple-choice question, aiming to further improve the accuracy of text-to-SQL tasks. The experiment results and ablation study have demonstrated the effectiveness of data-centric pipeline and Multi-Model(MM) interactive iterative strategies, achieving first place in lightweight text-to-SQL models (within 70B).
Read the full abstract ▾
All authors Yuanzhen Xie, Liu Ye, Jiqun Chu, Mochi Gao, Hehuan Liu, Yunzhi Tan, Bo Hu, Zang Li
Computation and Language
A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results
Thai-Binh Nguyen, Katerina Zmolikova, Pingchuan Ma et al. (+3 more)
We introduce the task of Multi-Modal Context-Aware Recognition (MCoRec) in the ninth CHiME Challenge, which addresses the cocktail-party problem of overlapping conversations in a single-room setting using audio, visual, and contextual cues. MCoRec captures natural multi-party conversations where the recordings focus on unscripted, casual group chats, leading to extreme
speech overlap of up to 100% and highly fragmented conversational turns. The task requires systems to answer the question "Who speaks when, what, and with whom?" by jointly transcribing each speaker's
speech and clustering them into their respective conversations from audio-visual recordings. Audio-only baselines exceed 100% word error rate, whereas incorporating visual cues yields substantial 50% improvements, highlighting the importance of multi-modality. In this manuscript, we present the motivation behind the task, outline the data collection process, and report the baseline systems developed for the MCoRec.
Read the full abstract ▾
All authors Thai-Binh Nguyen, Katerina Zmolikova, Pingchuan Ma, Ngoc Quan Pham, Christian Fuegen, Alexander Waibel
Computation and Language
Code Aesthetics with Agentic Reward Feedback
Bang Xiao, Lingjie Jiang, Shaohan Huang et al. (+5 more)
Large
Language Models (LLMs) have become valuable assistants for developers in code-related tasks. While LLMs excel at traditional programming tasks such as code generation and bug fixing, they struggle with visually-oriented coding tasks, often producing suboptimal aesthetics. In this paper, we introduce a new pipeline to enhance the aesthetic quality of LLM-generated code. We first construct AesCode-358K, a large-scale instruction-tuning
dataset focused on code aesthetics. Next, we propose agentic reward feedback, a multi-
agent system that evaluates executability, static aesthetics, and interactive aesthetics. Building on this, we develop GRPO-AR, which integrates these signals into the GRPO
algorithm for joint optimization of functionality and code aesthetics. Finally, we develop OpenDesign, a
benchmark for assessing code aesthetics. Experimental results show that combining supervised fine-tuning on AesCode-358K with reinforcement learning using agentic reward feedback significantly improves performance on OpenDesign and also enhances results on existing
benchmarks such as PandasPlotBench. Notably, our AesCoder-4B surpasses GPT-4o and GPT-4.1, and achieves performance comparable to large open-source models with 480B-685B parameters, underscoring the effectiveness of our approach.
Read the full abstract ▾
All authors Bang Xiao, Lingjie Jiang, Shaohan Huang, Tengchao Lv, Yupan Huang, Xun Wu, Lei Cui, Furu Wei
Computation and Language
Mubeen AI: A Specialized Arabic Language Model for Heritage Preservation and User Intent Understanding
Mohammed Aljafari, Ismail Alturki, Ahmed Mori et al. (+1 more)
Mubeen is a proprietary Arabic
language model developed by MASARAT SA, optimized for deep understanding of Arabic linguistics, Islamic studies, and cultural heritage. Trained on an extensive collection of authentic Arabic sources significantly expanded by digitizing historical manuscripts via a proprietary Arabic OCR engine, the model incorporates seminal scholarly works in linguistics, jurisprudence, hadith, and Quranic exegesis, alongside thousands of academic theses and peer-reviewed research papers. Conditioned through a deep linguistic engineering framework, Mubeen masters not just the meaning but the eloquence of Arabic, enabling precise understanding across classical texts, contemporary writing, and regional dialects with focus on comprehending user intent and delivering accurate, contextually relevant responses. Unlike other Arabic models relying on translated English data that often fail in intent detection or
retrieval-augmented generation (RAG), Mubeen uses native Arabic sources to ensure cultural authenticity and accuracy. Its core innovation is the Practical Closure Architecture, designed to solve the "Utility Gap Crisis" where factually correct answers fail to resolve users' core needs, forcing them into frustrating cycles of re-prompting. By prioritizing clarity and decisive guidance, Mubeen transforms from an information repository into a decisive guide, aligning with Saudi
Vision 2030. The model's architecture combines deep heritage specialization with multi-disciplinary expert modules, enabling robust performance across both cultural preservation and general knowledge domains.
Read the full abstract ▾
All authors Mohammed Aljafari, Ismail Alturki, Ahmed Mori, Yehya Kadumi
Computation and Language
Are ASR foundation models generalized enough to capture features of regional dialects for low-resource languages?
Tawsif Tashwar Dipto, Azmol Hossain, Rubayet Sabbir Faruque et al. (+12 more)
Conventional research on
speech recognition modeling relies on the canonical form for most low-resource
languages while automatic
speech recognition (ASR) for regional dialects is treated as a fine-tuning task. To investigate the effects of dialectal variations on ASR we develop a 78-hour annotated Bengali
Speech-to-Text (STT) corpus named Ben-10. Investigation from linguistic and data-driven perspectives shows that
speech foundation models struggle heavily in regional dialect ASR, both in zero-shot and fine-tuned settings. We observe that all deep learning methods struggle to model
speech data under dialectal variations but dialect specific model training alleviates the issue. Our
dataset also serves as a out of-distribution (OOD) resource for ASR modeling under constrained resources in ASR
algorithms. The
dataset and code developed for this project are publicly available
Read the full abstract ▾
All authors Tawsif Tashwar Dipto, Azmol Hossain, Rubayet Sabbir Faruque, Md. Rezuwan Hassan, Kanij Fatema, Tanmoy Shome, Ruwad Naswan, Md.Foriduzzaman Zihad, Mohaymen Ul Anam, Nazia Tasnim, Hasan Mahmud, Md Kamrul Hasan, Md. Mehedi Hasan Shawon, Farig Sadeque, Tahsin Reasat
Computation and Language
Process Reward Models for Sentence-Level Verification of LVLM Radiology Reports
Alois Thomas, Maya Varma, Jean-Benoit Delbrouck et al. (+1 more)
Automating radiology report generation with Large
Vision-
Language Models (LVLMs) holds great potential, yet these models often produce clinically critical
hallucinations, posing serious risks. Existing
hallucination detection methods frequently lack the necessary sentence-level granularity or robust generalization across different LVLM generators. We introduce a novel approach: a sentence-level Process Reward Model (PRM) adapted for this
vision-
language task. Our PRM predicts the factual correctness of each generated sentence, conditioned on clinical context and preceding text. When fine-tuned on MIMIC-CXR with weakly-supervised labels, a lightweight 0.5B-parameter PRM outperforms existing verification techniques, demonstrating, for instance, relative improvements of 7.5% in Matthews Correlation Coefficient and 1.8% in AUROC over strong white-box baselines on outputs from one LVLM. Unlike methods reliant on internal model states, our PRM demonstrates strong generalization to an unseen LVLM. We further show its practical utility: PRM scores effectively filter low-quality reports, improving F1-CheXbert scores by 4.5% (when discarding the worst 10% of reports). Moreover, when guiding a novel weighted best-of-N selection process on the MIMIC-CXR test set, our PRM show relative improvements in clinical metrics of 7.4% for F1-CheXbert and 0.6% for BERTScore. These results demonstrate that a lightweight, context-aware PRM provides a model-agnostic safety layer for clinical LVLMs without access to internal activations
Read the full abstract ▾
All authors Alois Thomas, Maya Varma, Jean-Benoit Delbrouck, Curtis P. Langlotz
Computation and Language · Cross-listed from cs.LG
PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets
Etienne Goffinet, Shane Bergsma, Avraham Sheinin et al. (+4 more)
Continual pre-training (CPT) for domain adaptation must balance target-domain gains with stability on the base domain. Existing CPT scaling laws typically assume a fixed pre-training budget, which limits their ability to forecast adaptation outcomes for models trained at different
tokens-per-parameter (PTPP). We present PTPP-aware adaptation scaling laws that make the pre-training budget an explicit variable, enabling accurate prediction of adaptation loss at unseen \ptpp. On a
multilingual setup (English/Arabic $\rightarrow$ French), PTPP-aware formulations trained on early stages (\ptpp{}=\{15,31\}) predict target loss at \ptpp{}=279 and outperform a PTPP-agnostic \dcpt{} transfer baseline on metrics (Huber-on-log, MAE$_\mathrm{rel}$, calibration slope); full diagnostics (RMSE, MAPE) are in the appendix. Beyond
forecasting, we show a practical use case: planning replay ratios and adaptation
token budgets that satisfy target and forgetting constraints under compute limits.
Read the full abstract ▾
All authors Etienne Goffinet, Shane Bergsma, Avraham Sheinin, Natalia Vassilieva, Shaheer Muhammad, Preslav Nakov, Gurpreet Gosal
Computation and Language
DREaM: Drug-Drug Relation Extraction via Transfer Learning Method
Ali Fata, Hossein Rahmani, Parinaz Soltanzadeh et al. (+2 more)
Relation extraction between drugs plays a crucial role in identifying drug drug interactions and predicting side effects. The advancement of machine learning methods in relation extraction, along with the development of large medical text databases, has enabled the low cost extraction of such relations compared to other approaches that typically require expert knowledge. However, to the best of our knowledge, there are limited
datasets specifically designed for drug drug relation extraction currently available. Therefore, employing transfer learning becomes necessary to apply machine learning methods in this domain. In this study, we propose DREAM, a method that first employs a trained relation extraction model to discover relations between entities and then applies this model to a corpus of medical texts to construct an ontology of drug relationships. The extracted relations are subsequently validated using a large
language model. Quantitative results indicate that the LLM agreed with 71 of the relations extracted from a subset of PubMed abstracts. Furthermore, our qualitative analysis indicates that this approach can uncover ambiguities in the medical domain, highlighting the challenges inherent in relation extraction in this field.
Read the full abstract ▾
All authors Ali Fata, Hossein Rahmani, Parinaz Soltanzadeh, Amirhossein Derakhshan, Behrouz Minaei Bidgoli
Computation and Language
SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
Shuai Huang, Wenxuan Zhao and Jun Gao
As large
language models (LLMs) develop anthropomorphic abilities, they are increasingly being deployed as autonomous
agents to interact with humans. However, evaluating their performance in realistic and complex social interactions remains a significant challenge. Most previous research built
datasets through simulated
agent-to-
agent interactions, which fails to capture the authentic linguistic styles and relational dynamics found in real human conversations. To address this gap, we introduce SI-Bench, a novel
benchmark designed to evaluate aspects of social intelligence in LLMs. Grounded in broad social science theories, SI-Bench contains 2,221 authentic multi-turn dialogues collected from a social networking application. We further selected a subset of 312 dialogues for manual annotation across 8 major models. The experiments show that SOTA models have surpassed the human expert in process
reasoning under complex social situations, yet they still fall behind humans in reply quality. Moreover, introducing Chain-of-Thought (CoT)
reasoning may degrade the performance of LLMs in social dialogue tasks. All
datasets are openly available at https://github.com/SI-Bench/SI-Bench.git.
Read the full abstract ▾
Computation and Language
MATCH: Task-Driven Code Evaluation through Contrastive Learning
Marah Ghoummaid, Vladimir Tchuiev, Ofek Glick et al. (+2 more)
AI-based code generation is increasingly prevalent, with GitHub Copilot estimated to generate 46% of the code on GitHub. Accurately evaluating how well generated code aligns with developer intent remains a critical challenge. Traditional evaluation methods, such as unit tests, are often unscalable and costly. Syntactic similarity metrics (e.g., BLEU, ROUGE) fail to capture code functionality, and metrics like CodeBERTScore require reference code, which is not always available. To address the gap in reference-free evaluation, with few alternatives such as ICE-Score, this paper introduces MATCH, a novel reference-free metric. MATCH uses Contrastive Learning to generate meaningful
embeddings for code and natural
language task descriptions, enabling similarity scoring that reflects how well generated code implements the task. We show that MATCH achieves stronger correlations with functional correctness and human preference than existing metrics across multiple programming
languages.
Read the full abstract ▾
All authors Marah Ghoummaid, Vladimir Tchuiev, Ofek Glick, Michal Moshkovitz, Dotan Di Castro
Computation and Language
Beyond Direct Generation: A Decomposed Approach to Well-Crafted Screenwriting with LLMs
Hang Lei, Shengyi Zong, Zhaoyan Li et al. (+3 more)
The screenplay serves as the foundation for television production, defining narrative structure, character development, and dialogue. While Large
Language Models (LLMs) show great potential in creative writing, direct end-to-end generation approaches often fail to produce well-crafted screenplays. We argue this failure stems from forcing a single model to simultaneously master two disparate capabilities: creative narrative construction and rigid format adherence. The resulting outputs may mimic superficial style but lack the deep structural integrity and storytelling substance required for professional use. To enable LLMs to generate high-quality screenplays, we introduce Dual-Stage Refinement (DSR), a decomposed framework that decouples creative narrative generation from format conversion. The first stage transforms a brief outline into rich, novel-style prose. The second stage refines this narrative into a professionally formatted screenplay. This separation enables the model to specialize in one distinct capability at each stage. A key challenge in implementing DSR is the scarcity of paired outline-to-novel training data. We address this through hybrid data synthesis: reverse synthesis deconstructs existing screenplays into structured inputs, while forward synthesis leverages these inputs to generate high-quality narrative texts as training targets. Blind evaluations by professional screenwriters show that DSR achieves a 75% win rate against strong baselines like Gemini-2.5-Pro and reaches 82.7% of human-level performance. Our work demonstrates that decomposed generation architecture with tailored data synthesis effectively specializes LLMs in complex creative domains.
Read the full abstract ▾
All authors Hang Lei, Shengyi Zong, Zhaoyan Li, Ziren Zhou, Hao Liu, Liang Yu
Computation and Language
Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction
Zile Yang, Ling Li, Na Di et al. (+5 more)
Data selection for supervised fine-tuning (SFT) often discards low-scoring instruction examples that may retain useful supervision. We propose a rule-guided LLM reconstruction framework to reclaim this information. The framework identifies low-scoring samples through LLM-based rating and score correction, selects semantically representative sources, and reconstructs instruction-response pairs through strategy-guided generation, criteria-based validation, and failure-guided repair. From 123,786 low-scoring samples, it constructs 54,888 training examples. Across three 7B/8B base LLMs, the reconstructed data achieves higher average downstream performance than the low-scoring source set and all evaluated data-selection baselines. Despite using substantially fewer examples than the full source
dataset (approximately 300K), it achieves higher reported average scores than full-data training across all three base models. Stepwise ablations identify rule-guided reconstruction as the largest source of observed improvements over direct fusion, while a manual audit reveals remaining limitations in source integration and answer faithfulness. These findings support reconstruction as a complementary approach to selection-based SFT data curation.
Read the full abstract ▾
All authors Zile Yang, Ling Li, Na Di, Jinlong Pang, Yao Zhou, Hao Cheng, Bo Han, Jiaheng Wei
Computation and Language · Cross-listed from cs.LG
Rethinking GSPO: The Perplexity-Entropy Equivalence
Chi Liu
We provide a new perspective on GSPO's length-normalized importance ratios by establishing their connection to information-theoretic quantities. We show that GSPO's sequence-level weight $s(\theta) = (\pi_\theta/\pi_{\theta_{\text{old}}})^{1/|y|}$ can be equivalently expressed as the inverse perplexity ratio $\text{PPL}_{\theta_{\text{old}}}/\text{PPL}_\theta$ and as the exponential cross-
entropy change $\exp(\Delta H)$. While the perplexity-
entropy relationship follows from standard definitions, this observation provides a useful lens for understanding GSPO: the
algorithm weights policy gradient updates by perplexity ratios, offering an information-theoretic interpretation of the importance weights. This perspective helps explain GSPO's empirical properties, including log-domain variance reduction through
geometric averaging and stability in training mixture-of-experts models. We validate the mathematical equivalences and variance predictions through controlled experiments on mathematical
reasoning tasks.
Read the full abstract ▾
Computation and Language
Corpus Frequencies in Morphological Inflection: Do They Matter?
Tomáš Sourada and Jana Straková
The traditional approach to morphological inflection (the task of modifying a base word (lemma) to express grammatical categories) has been, for decades, to consider lexical entries of lemma-tag-form triples uniformly, lacking any information about their frequency distribution. However, in production deployment, one might expect the user inputs to reflect a real-world distribution of frequencies in natural texts. With future deployment in mind, we explore the incorporation of corpus frequency information into the task of morphological inflection along three key dimensions during system development: (i) for train-dev-test split, we combine a lemma-disjoint approach, which evaluates the model's generalization capabilities, with a frequency-weighted strategy to better reflect the realistic distribution of items across different frequency bands in training and test sets; (ii) for evaluation, we complement the standard type accuracy (often referred to simply as accuracy), which treats all items equally regardless of frequency, with
token accuracy, which assigns greater weight to frequent words and better approximates performance on running text; (iii) for training data sampling, we introduce a method novel in the context of inflection, frequency-aware training, which explicitly incorporates word frequency into the sampling process. We show that frequency-aware training outperforms uniform sampling in 26 out of 43
languages.
Read the full abstract ▾
Computation and Language
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Shiwei Li, Xiandi Luo, Haozhao Wang et al. (+6 more)
Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large
language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input
tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture
token-specific information due to the inherent semantic differences among
tokens. To address this limitation, we propose
Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input
token, thereby learning
token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as $B\Sigma_X A$, where $A$ and $B$ are low-rank
matrices (as in standard LoRA), and $\Sigma_X$ is a diagonal
matrix generated from each input
token $X$. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning
token-wise LoRA weights (i.e.,
token-wise input-output projections). Extensive experiments across multiple models and
datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.
Read the full abstract ▾
All authors Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, Ruixuan Li
Computation and Language
Flexing in 73 Languages: A Single Small Model for Multilingual Inflection
Tomáš Sourada and Jana Straková
We present a compact, single-model approach to
multilingual inflection, the task of generating inflected word forms from base lemmas to express grammatical categories. Our model, trained jointly on data from 73
languages, is lightweight, robust to unseen words, and outperforms monolingual baselines in most
languages. This demonstrates the effectiveness of
multilingual modeling for inflection and highlights its practical benefits: simplifying deployment by eliminating the need to manage and retrain dozens of separate monolingual models. In addition to the standard SIGMORPHON shared task
benchmarks, we evaluate our monolingual and
multilingual models on 73 Universal Dependencies (UD) treebanks, extracting lemma-tag-form triples and their frequency counts. To ensure realistic data splits, we introduce a novel frequency-weighted, lemma-disjoint train-dev-test resampling procedure. Our work addresses the lack of an open-source, general-purpose,
multilingual morphological inflection system capable of handling unseen words across a wide range of
languages, including Czech. All code is publicly released at: https://github.com/tomsouri/
multilingual-inflection.
Read the full abstract ▾
Computation and Language
Leveraging Hierarchical Organization for Medical Multi-document Summarization
Yi-Li Hsu, Katelyn X. Mei and Lucy Lu Wang
Medical multi-document summarization (MDS) is a complex task that requires effectively managing cross-document relationships. This paper investigates whether incorporating hierarchical structures in the inputs of MDS can improve a model's ability to organize and contextualize information across documents compared to traditional flat summarization methods. We investigate two ways of incorporating hierarchical organization across three large
language models (LLMs), and conduct comprehensive evaluations of the resulting summaries using automated metrics, model-based metrics, and domain expert evaluation of preference, understandability, clarity, complexity, relevance, coverage, factuality, and coherence. Our results show that human experts prefer model-generated summaries over human-written summaries. Hierarchical approaches generally preserve factuality, coverage, and coherence of information, while also increasing human preference for summaries. Additionally, we examine whether simulated judgments from GPT-4 align with human judgments, finding higher agreement along more objective evaluation facets. Our findings demonstrate that hierarchical structures can improve the clarity of medical summaries generated by models while maintaining content coverage, providing a practical way to improve human preference for generated summaries.
Read the full abstract ▾
Computation and Language
MAP4TS: A Multi-Aspect Prompting Framework for Time-Series Forecasting with Large Language Models
Suchan Lee, Jihoon Choi, Sohyeon Lee et al. (+4 more)
Recent advances have investigated the use of pretrained large
language models (LLMs) for time-series
forecasting by aligning numerical inputs with LLM
embedding spaces. However, existing
multimodal approaches often overlook the distinct statistical properties and temporal dependencies that are fundamental to time-series data. To bridge this gap, we propose MAP4TS, a novel Multi-Aspect Prompting Framework that explicitly incorporates classical time-series analysis into the
prompt design. Our framework introduces four specialized
prompt components: a Global Domain
Prompt that conveys
dataset-level context, a Local Domain
Prompt that encodes recent trends and series-specific behaviors, and a pair of Statistical and Temporal
Prompts that embed handcrafted insights derived from autocorrelation (ACF), partial autocorrelation (PACF), and
Fourier analysis. Multi-Aspect
Prompts are combined with raw time-series
embeddings and passed through a cross-modality
alignment module to produce unified representations, which are then processed by an LLM and projected for final
forecasting. Extensive experiments across eight diverse
datasets show that MAP4TS consistently outperforms state-of-the-art LLM-based methods. Our ablation studies further reveal that
prompt-aware designs significantly enhance performance stability and that GPT-2 backbones, when paired with structured
prompts, outperform larger models like LLaMA in long-term
forecasting tasks.
Read the full abstract ▾
All authors Suchan Lee, Jihoon Choi, Sohyeon Lee, Minseok Song, Bong-Gyu Jang, Hwanjo Yu, Soyeon Caren Han
Computation and Language
A Survey on LLM Mid-Training
Chengying Tu, Xuemiao Zhang, Rongxiang Weng et al. (+6 more)
Recent advances in foundation models have highlighted the significant benefits of multi-stage training, with a particular emphasis on the emergence of mid-training as a vital stage that bridges pre-training and post-training. Mid-training is distinguished by its use of intermediate data and computational resources, systematically enhancing specified capabilities such as mathematics, coding,
reasoning, and long-context extension, while maintaining foundational competencies. This survey provides a formal definition of mid-training for large
language models (LLMs) and investigates optimization frameworks that encompass data curation, training strategies, and model architecture optimization. We analyze mainstream model implementations in the context of objective-driven interventions, illustrating how mid-training serves as a distinct and critical stage in the progressive development of LLM capabilities. By clarifying the unique contributions of mid-training, this survey offers a comprehensive taxonomy and actionable insights, supporting future research and innovation in the advancement of LLMs.
Read the full abstract ▾
All authors Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, Xunliang Cai
Computation and Language · Cross-listed from cs.CR
Fast-MIA: Efficient and Scalable Membership Inference for LLMs
Hiromu Takahashi and Shotaro Ishihara
We propose Fast-MIA (https://github.com/Nikkei/fast-mia), a Python library for efficiently evaluating membership inference attacks (MIA) against large
language models (LLMs). MIA has emerged as a crucial technique for
auditing privacy risks and copyright infringement in LLMs. However, computational demands have grown substantially: recent methods rely on repeated inference, while practical
auditing requires large-scale evaluation. Progress is further hindered by existing implementations that execute methods independently, redundantly computing shared intermediate results such as log-probabilities. To address these challenges, Fast-MIA combines two strategies: (1) high-throughput batch inference via vLLM, achieving approximately 5$\times$ speedup, and (2) a cross-method caching architecture that computes intermediate results once and shares them across methods. The library includes representative MIA methods under a unified framework, integrates with established
benchmarks, and supports flexible YAML configuration. We release Fast-MIA under the Apache License 2.0 to support scalable and reproducible MIA research.
Read the full abstract ▾
Computation and Language
Quality-Aware Translation Tagging in Multilingual RAG system
Hoyeon Moon, Byeolhee Kim and Nikhil Verma
Multilingual Retrieval-Augmented Generation (mRAG) often retrieves English documents and translates them into the query
language for low-resource settings. However, poor translation quality degrades response generation performance. Existing approaches either assume sufficient translation quality or utilize the rewriting method, which introduces factual distortion and
hallucinations. To mitigate these problems, we propose Quality-Aware Translation Tagging in mRAG (QTT-RAG), which explicitly evaluates translation quality along three dimensions-semantic equivalence, grammatical accuracy, and naturalness&fluency-and attach these scores as metadata without altering the original content. We evaluate QTT-RAG against CrossRAG and DKM-RAG as baselines in two open-domain QA
benchmarks (XORQA, MKQA) using six instruction-tuned LLMs ranging from 2.4B to 14B parameters, covering two low-resource
languages (Korean and Finnish) and one high-resource
language (Chinese). QTT-RAG outperforms the baselines by preserving factual integrity while enabling generator models to make informed decisions based on translation reliability. This approach allows for effective usage of cross-lingual documents in low-resource settings with limited native
language documents, offering a practical and robust solution across
multilingual domains.
Read the full abstract ▾
Computation and Language
Zhanchao Zhou, Xiaodong Chen, Haoxing Chen et al. (+2 more)
Multi-head
attention (MHA) has become the cornerstone of modern large
language models, enhancing representational capacity through parallel
attention heads. However, increasing the number of heads inherently weakens individual head capacity, and existing
attention mechanisms - whether standard MHA or its variants like grouped-query
attention (GQA) and grouped-tied
attention (GTA) - simply concatenate outputs from isolated heads without strong interaction. To address this limitation, we propose knocking-heads
attention (KHA), which enables
attention heads to "knock" on each other - facilitating cross-head feature-level interactions before the scaled dot-product
attention. This is achieved by applying a shared, diagonally-initialized projection
matrix across all heads. The diagonal initialization preserves head-specific specialization at the start of training while allowing the model to progressively learn integrated cross-head representations. KHA adds only minimal parameters and FLOPs and can be seamlessly integrated into MHA, GQA, GTA, and other
attention variants. We validate KHA by training a 6.1B parameter MoE model (1.01B activated) on 1T high-quality
tokens. Compared to baseline
attention mechanisms, KHA brings superior and more stable training dynamics, achieving better performance across downstream tasks.
Read the full abstract ▾
All authors Zhanchao Zhou, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Jianguo Li
Computation and Language
Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
Ran Xu, Jingjing Chen, Jiayu Ye et al. (+4 more)
Large
Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based
reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated
reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without
distillation. On seven public
benchmarks, TIR-Judge surpasses strong
reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.
Read the full abstract ▾
All authors Ran Xu, Jingjing Chen, Jiayu Ye, Yu Wu, Jun Yan, Carl Yang, Hongkun Yu
Computation and Language · Cross-listed from cs.LG
Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
Di Zhang, Xun Wu, Shaohan Huang et al. (+6 more)
Recent advances in reinforcement learning (RL) have substantially improved the training of large-scale
language models, leading to significant gains in generation quality and
reasoning ability. However, most existing research focuses on dense models, while RL training for Mixture-of-Experts (MoE) architectures remains underexplored. To address the instability commonly observed in MoE training, we propose a novel router-aware approach to optimize importance sampling (IS) weights in off-policy RL. Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence. Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models, highlighting the potential of RL algorithmic innovations tailored to MoE architectures and providing a promising direction for efficient training of large-scale expert models.
Read the full abstract ▾
All authors Di Zhang, Xun Wu, Shaohan Huang, Lingjie Jiang, Yaru Hao, Li Dong, Zewen Chi, Zhifang Sui, Furu Wei
Computation and Language · Cross-listed from cs.CV
UniAIDet: A Unified and Universal Benchmark for AI-Generated Image Content Detection and Localization
Huixuan Zhang and Xiaojun Wan
With the rapid proliferation of image generative models, the authenticity of digital images has become a significant concern. While existing studies have proposed various methods for detecting AI-generated content, current
benchmarks are limited in their coverage of diverse generative models and image categories, often overlooking end-to-end image editing and artistic images. To address these limitations, we introduce UniAIDet, a unified and comprehensive
benchmark that includes both photographic and artistic images. UniAIDet covers a wide range of generative models, including text-to-image, image-to-image, image inpainting, image editing, and deepfake models. Using UniAIDet, we conduct a comprehensive evaluation of various detection methods and answer three key research questions regarding generalization capability and the relation between detection and localization. Our
benchmark and analysis provide a robust foundation for future research.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
M$^{3}$T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
Huixuan Zhang and Xiaojun Wan
Text-to-image models are known to struggle with generating images that perfectly align with textual
prompts. Several previous studies have focused on evaluating image-text
alignment in text-to-image generation. However, these evaluations either address overly simple scenarios, especially overlooking the difficulty of
prompts with multiple different instances belonging to the same category, or they introduce metrics that do not correlate well with human evaluation. In this study, we introduce M$^3$T2IBench, a large-scale, multi-category, multi-instance, multi-relation along with an object-detection-based evaluation metric, $AlignScore$, which aligns closely with human evaluation. Our findings reveal that current open-source text-to-image models perform poorly on this challenging
benchmark. Additionally, we propose the Revise-Then-Enforce approach to enhance image-text
alignment. This training-free post-editing method demonstrates improvements in image-text
alignment across a broad range of
diffusion models. \footnote{Our code and data has been released in supplementary material and will be made publicly available after the paper is accepted.}
Read the full abstract ▾
Computation and Language
LangLingual: A Personalised, Exercise-oriented English Language Learning Tool Leveraging Large Language Models
Sammriddh Gupta, Sonit Singh, Aditya Joshi et al. (+1 more)
Language educators strive to create a rich experience for learners, while they may be restricted in the extend of feedback and practice they can provide. We present the design and development of LangLingual, a conversational
agent built using the LangChain framework and powered by Large
Language Models. The system is specifically designed to provide real-time, grammar-focused feedback, generate context-aware
language exercises and track learner proficiency over time. The paper discusses the architecture, implementation and evaluation of LangLingual in detail. The results indicate strong usability, positive learning outcomes and encouraging learner engagement.
Read the full abstract ▾
All authors Sammriddh Gupta, Sonit Singh, Aditya Joshi, Mira Kim
Computation and Language
Understanding In-Context Learning Beyond Transformers: An Investigation of State Space and Hybrid Architectures
Shenran Wang, Timothy Tin-Long Tse and Jian Zhu
We perform in-depth evaluations of in-context learning (ICL) on state-of-the-art
transformer, state-space, and hybrid large
language models over two categories of knowledge-based ICL tasks. Using a combination of behavioral probing and intervention-based methods, we have discovered that, while LLMs of different architectures can behave similarly in task performance, their internals could remain different. We discover that function vectors (FVs) responsible for ICL are primarily located in the self-
attention and Mamba layers, and speculate that Mamba2 uses a different mechanism from FVs to perform ICL. FVs are more important for ICL involving parametric knowledge
retrieval, but not for contextual knowledge understanding. Our work contributes to a more nuanced understanding across architectures and task types. Methodologically, our approach also highlights the importance of combining both behavioural and mechanistic analyses to investigate LLM capabilities.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Can Language Models Compose Skills In-Context?
Zidong Liu, Zhuoyan Xu, Zhenmei Shi et al. (+1 more)
Composing basic skills from simple tasks to accomplish composite tasks is crucial for modern intelligent systems. We investigate the in-context composition ability of
language models to perform composite tasks that combine basic skills demonstrated in in-context examples. This is more challenging than the standard setting, where skills and their composition can be learned in training. We conduct systematic experiments on various representative open-source
language models, utilizing linguistic and logical tasks designed to probe composition abilities. The results reveal that simple task examples can have a surprising negative impact on the performance, because the models generally struggle to recognize and assemble the skills correctly, even with Chain-of-Thought examples. Theoretical analysis further shows that it is crucial to align examples with the corresponding steps in the composition. This inspires a method for the probing tasks, whose improved performance provides positive support for our insights.
Read the full abstract ▾
All authors Zidong Liu, Zhuoyan Xu, Zhenmei Shi, Yingyu Liang
Computation and Language
Measuring Teaching with LLMs
Michael Hardy
Objective and scalable measurement of teaching quality is a persistent challenge in education. While Large
Language Models (LLMs) offer potential, general-purpose models have struggled to reliably apply complex, authentic classroom observation instruments. This paper uses custom LLMs built on sentence-level
embeddings, an architecture better suited for the long-form, interpretive nature of classroom transcripts than conventional subword tokenization. We systematically evaluate five different sentence
embeddings under a data-efficient training regime designed to prevent overfitting. Our results demonstrate that these specialized models can achieve human-level and even super-human performance with expert human ratings above 0.65 and surpassing the average human-human rater correlation. Further, through analysis of annotation context windows, we find that more advanced models-those better aligned with human judgments-attribute a larger share of score variation to lesson-level features rather than isolated utterances, challenging the sufficiency of single-turn annotation paradigms. Finally, to assess external validity, we find that aggregate model scores align with teacher value-added measures, indicating they are capturing features relevant to student learning. However, this trend does not hold at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization. This work establishes a viable and powerful new methodology for AI-driven instructional measurement, offering a path toward providing scalable, reliable, and valid feedback for educator development.
Read the full abstract ▾