Computation and Language
Bridging the Language Gap: Synthetic Voice Diversity via Latent Mixup for Equitable Speech Recognition
Wesley Bian, Xiaofeng Lin and Guang Cheng
Modern machine learning models for audio tasks often exhibit superior performance on English and other well-resourced
languages, primarily due to the abundance of available training data. This disparity leads to an unfair performance gap for low-resource
languages, where data collection is both challenging and costly. In this work, we introduce a novel data augmentation technique for
speech corpora designed to mitigate this gap. Through comprehensive experiments, we demonstrate that our method significantly improves the performance of automatic
speech recognition systems on low-resource
languages. Furthermore, we show that our approach outperforms existing augmentation strategies, offering a practical solution for enhancing
speech technology in underrepresented linguistic communities.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
DesignPref: Capturing Personal Preferences in Visual Design Generation
Yi-Hao Peng, Jeffrey P. Bigham and Jason Wu
Generative models, such as large
language models and text-to-image
diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slides. Finetuning and benchmarking these generative models have often relied on
datasets of human-annotated design preferences. Yet, due to the subjective and highly personalized nature of visual design, preference varies widely among individuals. In this paper, we study this problem by introducing DesignPref, a
dataset of 12k pairwise comparisons of UI design generation annotated by 20 professional designers with multi-level preference ratings. We found that among trained designers, substantial levels of disagreement exist (Krippendorff's alpha = 0.25 for binary preferences). Natural
language rationales provided by these designers indicate that disagreements stem from differing perceptions of various design aspect importance and individual preferences. With DesignPref, we demonstrate that traditional majority-voting methods for training aggregated judge models often do not accurately reflect individual preferences. To address this challenge, we investigate multiple personalization strategies, particularly fine-tuning or incorporating designer-specific annotations into RAG pipelines. Our results show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences, even when using 20 times fewer examples. Our work provides the first
dataset to study personalized visual design evaluation and support future research into modeling individual design taste.
Read the full abstract ▾
Computation and Language
The Text Aphasia Battery (TAB): A Clinically-Grounded Benchmark for Aphasia-Like Deficits in Language Models
Nathan Roll, Jill Kries, Flora Jin et al. (+5 more)
Large
language models (LLMs) have emerged as a candidate "model organism" for human
language, offering an unprecedented opportunity to study the computational basis of linguistic disorders like aphasia. However, traditional clinical assessments are ill-suited for LLMs, as they presuppose human-like pragmatic pressures and probe cognitive processes not inherent to artificial architectures. We introduce the Text Aphasia Battery (TAB), a text-only
benchmark adapted from the Quick Aphasia Battery (QAB) to assess aphasic-like deficits in LLMs. The TAB comprises four subtests: Connected Text, Word Comprehension, Sentence Comprehension, and Repetition. This paper details the TAB's design, subtests, and scoring criteria. To facilitate large-scale use, we validate an automated evaluation protocol using Gemini 2.5 Flash, which achieves reliability comparable to expert human raters (prevalence-weighted Cohen's kappa = 0.255 for model–consensus agreement vs. 0.286 for human–human agreement). We release TAB as a clinically-grounded, scalable framework for analyzing
language deficits in artificial systems.
Read the full abstract ▾
All authors Nathan Roll, Jill Kries, Flora Jin, Catherine Wang, Ann Marie Finley, Meghan Sumner, Cory Shain, Laura Gwilliams
Computation and Language
Jakub Hoscilowicz and Artur Janicki
We introduce the
Adversarial Confusion Attack, a new class of threats against
multimodal large
language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Practical applications include
embedding such
adversarial images into websites to prevent MLLM-powered AI
Agents from operating reliably. The proposed attack maximizes next-
token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single
adversarial image can disrupt all models in the ensemble, both in the full-image and
Adversarial CAPTCHA settings. Despite relying on a basic
adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.
Read the full abstract ▾
Computation and Language
Generation, Evaluation, and Explanation of Novelists' Styles with Single-Token Prompts
Mosab Rezaei, Mina Rajaei Moghadam, Abdul Rahman Shaikh et al. (+2 more)
Recent advances in large
language models have created new opportunities for stylometry, the study of writing styles and authorship. Two challenges, however, remain central: training generative models when no paired data exist, and evaluating stylistic text without relying only on human judgment. In this work, we present a framework for both generating and evaluating sentences in the style of 19th-century novelists. Large
language models are fine-tuned with minimal, single-
token prompts to produce text in the voices of authors such as Dickens, Austen, Twain, Alcott, and Melville. To assess these generative models, we employ a
transformer-based detector trained on authentic sentences, using it both as a classifier and as a tool for stylistic explanation. We complement this with syntactic comparisons and explainable AI methods, including
attention-based and gradient-based analyses, to identify the linguistic cues that drive stylistic imitation. Our findings show that the generated text reflects the authors' distinctive patterns and that AI-based evaluation offers a reliable alternative to human assessment. All artifacts of this work are published online.
Read the full abstract ▾
All authors Mosab Rezaei, Mina Rajaei Moghadam, Abdul Rahman Shaikh, Hamed Alhoori, Reva Freedman
Computation and Language
NormEval: A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization
Md Abdullah Al Kafi, Raka Moni and Walayat Hussain
Text normalization methods such as stemming and lemmatization are fundamental components of NLP pipelines. As new normalization tools are developed for diverse
languages, evaluation methodologies remain fragmented, relying on Compression Ratio, downstream accuracy, or sequence-to-sequence prediction scores in isolation, failing to distinguish between beneficial vocabulary reduction and harmful semantic distortion. Moreover, text normalization underpins intelligent systems in high-stakes domains, including clinical decision support and legal document analysis, and principled evaluation methodology is essential. This paper proposes NormEval, a unified,
multilingual evaluation framework comprising five complementary metrics: Compression Ratio (CR), Model Performance Delta (MPD), Information Retention Score (IRS),
Algorithm Effectiveness Score (AES), and Average Normalized Levenshtein Distance (ANLD). These metrics assess normalization quality across three dimensions: macro-level efficiency, downstream utility, and micro-level morphological fidelity. The framework operationalizes a Safety Gate hypothesis: ANLD functions as an intrinsic structural hygiene check, utilizing character-level divergence ($\Delta$) to reveal aggressive mutations that macro-level
embeddings and downstream tasks mask. Comprehensive ablation experiments on both Bangla and English
datasets show that all the components are indispensable, and that the removal of any individual metric leads to a decrease in at least one evaluation aspect, which ultimately results in misleading
algorithm rankings.
Read the full abstract ▾
Computation and Language
BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali
Abdullah Al Sefat
Large
language models excel on broad
multilingual benchmarks but remain to be evaluated extensively in figurative and culturally grounded
reasoning, especially in low-resource contexts. We present BengaliFig, a compact yet richly annotated challenge set that targets this gap in Bengali, a widely spoken low-resourced
language. The
dataset contains 435 unique riddles drawn from Bengali oral and literary traditions. Each item is annotated along five orthogonal dimensions capturing
reasoning type, trap type, cultural depth, answer category, and difficulty, and is automatically converted to multiple-choice format through a constraint-aware, AI-assisted pipeline. We evaluate eight frontier LLMs from major providers under zero-shot and few-shot chain-of-thought prompting, revealing consistent weaknesses in metaphorical and culturally specific
reasoning. BengaliFig thus contributes both a diagnostic probe for evaluating LLM robustness in low-resource cultural contexts and a step toward inclusive and heritage-aware NLP evaluation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Soft Adaptive Policy Optimization
Chang Gao, Chujie Zheng, Xiong-Hui Chen et al. (+7 more)
Reinforcement learning (RL) plays an increasingly important role in enhancing the
reasoning capabilities of large
language models (LLMs), yet stable and performant policy optimization remains challenging.
Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and
token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy
tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending
tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard
token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical
reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
Read the full abstract ▾
All authors Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang Lin
Computation and Language
The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
Taewhoo Lee, Minju Song, Chanwoong Yoon et al. (+2 more)
Analogical
reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas
reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical
token positions can facilitate information transfer to a certain extent. Lastly, successful analogical
reasoning in LLMs is marked by strong structural
alignment between analogous situations, whereas failures often reflect degraded or misplaced
alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.
Read the full abstract ▾
All authors Taewhoo Lee, Minju Song, Chanwoong Yoon, Jungwoo Park, Jaewoo Kang
Computation and Language
Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
Luohe Shi, Zuchao Li, Lefei Zhang et al. (+3 more)
Speculative decoding accelerates LLM inference by utilizing otherwise idle computational resources during
memory-to-chip data transfer. Current speculative decoding methods typically assume a considerable amount of available computing power, then generate a complex and massive draft tree using a small autoregressive
language model to improve overall prediction accuracy. However, methods like batching have been widely applied in mainstream model inference systems as a superior alternative to speculative decoding, as they compress the available idle computing power. Therefore, performing speculative decoding with low verification resources and low scheduling costs has become an important research problem. We believe that more capable models that allow for parallel generation on draft sequences are what we truly need. Recognizing the fundamental nature of draft models to only generate sequences of limited length, we propose SpecFormer, a novel architecture that integrates unidirectional and bidirectional
attention mechanisms. SpecFormer combines the autoregressive model's ability to extract information from the entire input sequence with the parallel generation benefits of non-autoregressive models. This design eliminates the reliance on large prefix trees and achieves consistent acceleration, even in large-batch scenarios. Through lossless speculative decoding experiments across models of various scales, we demonstrate that SpecFormer sets a new standard for scaling LLM inference with lower training demands and reduced computational costs.
Read the full abstract ▾
All authors Luohe Shi, Zuchao Li, Lefei Zhang, Baoyuan Qi, Guoming Liu, Hai Zhao
Computation and Language · Cross-listed from cs.LG
Geometry of Decision Making in Language Models
Abhinav Joshi, Divyanshu Bhatt and Ashutosh Modi
Large
Language Models (LLMs) show strong generalization across diverse tasks, yet the internal decision-making processes behind their predictions remain opaque. In this work, we study the geometry of hidden representations in LLMs through the lens of intrinsic dimension (ID), focusing specifically on decision-making dynamics in a multiple-choice question answering (MCQA) setting. We perform a large-scale study, with 28 open-weight
transformer models and estimate ID across layers using multiple
estimators, while also quantifying per-layer performance on MCQA tasks. Our findings reveal a consistent ID pattern across models: early layers operate on low-dimensional
manifolds, middle layers expand this space, and later layers compress it again, converging to decision-relevant representations. Together, these results suggest LLMs implicitly learn to project linguistic inputs onto structured, low-dimensional
manifolds aligned with task-specific decisions, providing new
geometric insights into how generalization and
reasoning emerge in
language models.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits
Areeb Ahmad, Abhinav Joshi and Ashutosh Modi
Transformer-based
language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat
attention heads and multilayer perceptron layers (MLPs) (the building blocks of a
transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. In this work, we introduce a more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and independent computations within a single head or MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canonical functional heads, such as the name mover, encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in a computational graph, that are previously identified as
circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that
transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine-grained mechanistic interpretability and a deeper understanding of model internals.
Read the full abstract ▾
Computation and Language
REFLEX: Self-Refining Explainable Fact-Checking via Verdict-Anchored Style Control
Chuyi Kong, Wei Gao, Jing Ma et al. (+2 more)
The prevalence of fake news on social media demands automated fact-checking systems to provide accurate verdicts with faithful explanations. However, existing large
language model (LLM)-based approaches ignore deceptive misinformation styles in LLM-generated explanations, resulting in unfaithful rationales that can mislead human judgments. They rely heavily on external knowledge sources, introducing
hallucinations and even high latency that undermine reliability and responsiveness, which is crucial for real-time use. To address these challenges, we propose REason-guided Fact-checking with Latent EXplanations (REFLEX), a self-refining paradigm that explicitly controls
reasoning style anchored on verdict. REFLEX utilizes self-disagreement veracity signals between the backbone model and its fine-tuned variant to construct steering vectors, naturally disentangling fact from style. Experiments on the real-world
dataset show REFLEX achieves state-of-the-art performance under LLaMA-series models with only 465 self-refined samples. Moreover, owing to its transferability, REFLEX yields up to a 7.54% gain on in-the-wild data. Our results further demonstrate that our method effectively mitigates faithful
hallucination, thereby guiding the model toward more accurate verdicts than previous works in explainable fact-checking.
Read the full abstract ▾
All authors Chuyi Kong, Wei Gao, Jing Ma, Hongzhan Lin, Yuxi Sun
Computation and Language
KyrgyzBERT: A Compact, Efficient Language Model for Kyrgyz NLP
Adilet Metinov, Gulida M. Kudakeeva and Gulnara D. Kabaeva
Kyrgyz remains a low-resource
language with limited foundational NLP tools. To address this gap, we introduce KyrgyzBERT, the first publicly available monolingual BERT-based
language model for Kyrgyz. The model has 35.9M parameters and uses a custom tokenizer designed for the
language's morphological structure. To evaluate performance, we create kyrgyz-sst2, a sentiment analysis
benchmark built by translating the Stanford Sentiment Treebank and manually annotating the full test set. KyrgyzBERT fine-tuned on this
dataset achieves an F1-score of 0.8280, competitive with a fine-tuned mBERT model five times larger. All models, data, and code are released to support future research in Kyrgyz NLP.
Read the full abstract ▾
Computation and Language
SEDA: A Self-Adapted Entity-Centric Data Augmentation for Boosting Gird-based Discontinuous NER Models
Wen-Fang Su, Hsiao-Wei Chou and Wen-Yang Lin
Named Entity Recognition (NER) is a critical task in natural
language processing, yet it remains particularly challenging for discontinuous entities. The primary difficulty lies in text
segmentation, as traditional methods often missegment or entirely miss cross-sentence discontinuous entities, significantly affecting recognition accuracy. Therefore, we aim to address the
segmentation and omission issues associated with such entities. Recent studies have shown that grid-tagging methods are effective for information extraction due to their flexible tagging schemes and robust architectures. Building on this, we integrate image data augmentation techniques, such as cropping, scaling, and padding, into grid-based models to enhance their ability to recognize discontinuous entities and handle
segmentation challenges. Experimental results demonstrate that traditional
segmentation methods often fail to capture cross-sentence discontinuous entities, leading to decreased performance. In contrast, our augmented grid models achieve notable improvements. Evaluations on the CADEC, ShARe13, and ShARe14
datasets show F1 score gains of 1-2.5% overall and 3.7-8.4% for discontinuous entities, confirming the effectiveness of our approach.
Read the full abstract ▾
Computation and Language
"When Data is Scarce, Prompt Smarter"... Approaches to Grammatical Error Correction in Low-Resource Settings
Somsubhra De, Harsh Kumar and Arun Prakash A
Grammatical error correction (GEC) is an important task in Natural
Language Processing that aims to automatically detect and correct grammatical mistakes in text. While recent advances in
transformer-based models and large annotated
datasets have greatly improved GEC performance for high-resource
languages such as English, the progress has not extended equally. For most Indic
languages, GEC remains a challenging task due to limited resources, linguistic diversity and complex morphology. In this work, we explore prompting-based approaches using state-of-the-art large
language models (LLMs), such as GPT-4.1, Gemini-2.5 and LLaMA-4, combined with few-shot strategy to adapt them to low-resource settings. We observe that even basic prompting strategies, such as zero-shot and few-shot approaches, enable these LLMs to substantially outperform fine-tuned Indic-
language models like Sarvam-22B, thereby illustrating the exceptional
multilingual generalization capabilities of contemporary LLMs for GEC. Our experiments show that carefully designed
prompts and lightweight adaptation significantly enhance correction quality across multiple Indic
languages. We achieved leading results in the shared task–ranking 1st in Tamil (GLEU: 91.57) and Hindi (GLEU: 85.69), 2nd in Telugu (GLEU: 85.22), 4th in Bangla (GLEU: 92.86), and 5th in Malayalam (GLEU: 92.97). These findings highlight the effectiveness of
prompt-driven NLP techniques and underscore the potential of large-scale LLMs to bridge resource gaps in
multilingual GEC.
Read the full abstract ▾
Computation and Language
Mispronunciation Detection and Diagnosis Without Model Training: A Retrieval-Based Approach
Huu Tuong Tu, Ha Viet Khanh, Tran Tien Dat et al. (+4 more)
Mispronunciation Detection and Diagnosis (MDD) is crucial for
language learning and
speech therapy. Unlike conventional methods that require scoring models or training phoneme-level models, we propose a novel training-free framework that leverages
retrieval techniques with a pretrained Automatic
Speech Recognition model. Our method avoids phoneme-specific modeling or additional task-specific training, while still achieving accurate detection and diagnosis of pronunciation errors. Experiments on the L2-ARCTIC
dataset show that our method achieves a superior F1 score of 69.60% while avoiding the complexity of model training.
Read the full abstract ▾
All authors Huu Tuong Tu, Ha Viet Khanh, Tran Tien Dat, Vu Huan, Thien Van Luong, Nguyen Tien Cuong, Nguyen Thi Thu Trang
Computation and Language
EM2LDL: A Multilingual Speech Corpus for Mixed Emotion Recognition through Label Distribution Learning
Xingfeng Li, Xiaohan Shi, Junjie Li et al. (+4 more)
This study introduces EM2LDL, a novel
multilingual speech corpus designed to advance mixed
emotion recognition through label distribution learning. Addressing the limitations of predominantly monolingual and single-label
emotion corpora \textcolor{black}{that restrict linguistic diversity, are unable to model mixed
emotions, and lack ecological validity}, EM2LDL comprises expressive utterances in English, Mandarin, and Cantonese, capturing the intra-utterance code-switching prevalent in
multilingual regions like Hong Kong and Macao. The corpus integrates spontaneous emotional expressions from online platforms, annotated with fine-grained
emotion distributions across 32 categories. Experimental baselines using self-supervised learning models demonstrate robust performance in speaker-independent gender-, age-, and personality-based evaluations, with HuBERT-large-EN achieving optimal results. By incorporating linguistic diversity and ecological validity, EM2LDL enables the exploration of complex emotional dynamics in
multilingual settings. This work provides a versatile testbed for developing adaptive, empathetic systems for applications in affective computing, including mental health monitoring and cross-cultural communication. The
dataset, annotations, and baseline codes are publicly available at https://github.com/xingfengli/EM2LDL.
Read the full abstract ▾
All authors Xingfeng Li, Xiaohan Shi, Junjie Li, Yongwei Li, Masashi Unoki, Tomoki Toda, Masato Akagi
Computation and Language · Cross-listed from cs.LG
The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs
Craig Dickson
Prior work has shown that fine-tuning models on a narrow domain with misaligned data can lead to broad misalignment - a phenomenon termed "emergent misalignment" (Betley et al., 2025). While all tested models were susceptible to emergent misalignment, some models showed more resistance than others. Specifically the Qwen-2.5 family proved to be relatively resistant, while GPT-4o exhibited the strongest misalignment. In this paper we evaluate if current-generation open-weights models exhibit similar resistance to the Qwen-2.5 family and measure misalignment robustness over a range of model architectures and scales. We replicate the effect across nine modern open-weights models (Gemma 3 and Qwen 3 families, 1B-32B parameters). Models fine-tuned on insecure code generation show a 0.74% misalignment rate (compared to 0.07% for base models), matching the lower end of prior open-model results but dramatically lower than GPT-4o's 20%. We identify a critical format-dependent vulnerability: requiring JSON output doubles misalignment rates compared to natural
language prompts (0.96% vs 0.42%), an effect concentrated in the Gemma 3 family (1.43% vs 0.22%) and not significant in Qwen 3 (0.72% vs 0.59%). This suggests that structural constraints may bypass safety training by reducing the model's 'degrees of freedom' to refuse. These findings confirm emergent misalignment as a reproducible phenomenon in modern open-weights models, with rates substantially lower than observed in proprietary systems.
Read the full abstract ▾
Computation and Language
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
Zhenyi Shen, Junru Lu, Lin Gui et al. (+4 more)
Sparse
attention reduces the quadratic complexity of full self-
attention but faces two challenges: (1) an
attention gap, where applying sparse
attention to full-
attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse
attention lack complete gradient flow, preventing them from matching full-
attention performance. We propose SSA (Sparse Sparse
Attention), a training framework that integrates both sparse and full
attention with bidirectional
attention-output
alignment. We prove that the approximation error scales linearly with the
attention mass dropped under sparse
attention, and show that SSA's
alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities.
Read the full abstract ▾
All authors Zhenyi Shen, Junru Lu, Lin Gui, Jiazheng Li, Yulan He, Di Yin, Xing Sun
Computation and Language · Cross-listed from cs.DC
QiMeng-Kernel: Macro-Thinking Micro-Coding Paradigm for LLM-Based High-Performance GPU Kernel Generation
Xinguo Zhu, Shaohui Peng, Jiaming Guo et al. (+10 more)
Developing high-performance GPU kernels is critical for AI and scientific computing, but remains challenging due to its reliance on expert crafting and poor portability. While LLMs offer promise for automation, both general-purpose and finetuned LLMs suffer from two fundamental and conflicting limitations: correctness and efficiency. The key reason is that existing LLM-based approaches directly generate the entire optimized low-level programs, requiring exploration of an extremely vast space encompassing both optimization policies and implementation codes. To address the challenge of exploring an intractable space, we propose Macro Thinking Micro Coding (MTMC), a hierarchical framework inspired by the staged optimization strategy of human experts. It decouples optimization strategy from implementation details, ensuring efficiency through high-level strategy and correctness through low-level implementation. Specifically, Macro Thinking employs reinforcement learning to guide lightweight LLMs in efficiently exploring and learning semantic optimization strategies that maximize hardware utilization. Micro Coding leverages general-purpose LLMs to incrementally implement the stepwise optimization proposals from Macro Thinking, avoiding full-kernel generation errors. Together, they effectively navigate the vast optimization space and intricate implementation details, enabling LLMs for high-performance GPU kernel generation. Comprehensive results on widely adopted
benchmarks demonstrate the superior performance of MTMC on GPU kernel generation in both accuracy and running time. On KernelBench, MTMC achieves near 100% and 70% accuracy at Levels 1-2 and 3, over 50% than SOTA general-purpose and domain-finetuned LLMs, with up to 7.3x speedup over LLMs, and 2.2x over expert-optimized PyTorch Eager kernels. On the more challenging TritonBench, MTMC attains up to 59.64% accuracy and 34x speedup.
Read the full abstract ▾
All authors Xinguo Zhu, Shaohui Peng, Jiaming Guo, Yunji Chen, Qi Guo, Yuanbo Wen, Hang Qin, Ruizhi Chen, Qirui Zhou, Ke Gao, Yanjun Wu, Chen Zhao, Ling Li
Computation and Language
More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answering
Duc Anh Vu, Thong Nguyen, Cong-Duy Nguyen et al. (+2 more)
With the advancement of large
language models (LLMs), their performance on multiple-choice question (MCQ) tasks has improved significantly. However, existing approaches face key limitations: answer choices are typically presented to LLMs without contextual grounding or explanation. This absence of context can lead to incomplete exploration of all possible answers, ultimately degrading the models'
reasoning capabilities. To address these challenges, we introduce BiasPrompting, a novel inference framework that guides LLMs to generate and critically evaluate
reasoning across all plausible answer options before reaching a final prediction. It consists of two components: first, a
reasoning generation stage, where the model is prompted to produce supportive reasonings for each answer option, and then, a
reasoning-guided agreement stage, where the generated reasonings are synthesized to select the most plausible answer. Through comprehensive evaluations, BiasPrompting demonstrates significant improvements in five widely used multiple-choice question answering
benchmarks. Our experiments showcase that BiasPrompting enhances the
reasoning capabilities of LLMs and provides a strong foundation for tackling complex and challenging questions, particularly in settings where existing methods underperform.
Read the full abstract ▾
All authors Duc Anh Vu, Thong Nguyen, Cong-Duy Nguyen, Viet Anh Nguyen, Anh Tuan Luu
Computation and Language
MTA: A Merge-then-Adapt Framework for Personalized Large Language Model
Xiaopeng Li, Yuanjin Zheng, Wanyu Wang et al. (+6 more)
Personalized Large
Language Models (PLLMs) aim to align model outputs with individual user preferences, a crucial capability for user-centric applications. However, the prevalent approach of fine-tuning a separate module for each user faces two major limitations: (1) storage costs scale linearly with the number of users, rendering the method unscalable; and (2) fine-tuning a static model from scratch often yields suboptimal performance for users with sparse data. To address these challenges, we propose MTA, a Merge-then-Adapt framework for PLLMs. MTA comprises three key stages. First, we construct a shared Meta-LoRA Bank by selecting anchor users and pre-training meta-personalization traits within meta-LoRA modules. Second, to ensure scalability and enable dynamic personalization combination beyond static models, we introduce an Adaptive LoRA Fusion stage. This stage retrieves and dynamically merges the most relevant anchor meta-LoRAs to synthesize a user-specific one, thereby eliminating the need for user-specific storage and supporting more flexible personalization. Third, we propose a LoRA Stacking for Few-Shot Personalization stage, which applies an additional ultra-low-rank, lightweight LoRA module on top of the merged LoRA. Fine-tuning this module enables effective personalization under few-shot settings. Extensive experiments on the LaMP
benchmark demonstrate that our approach outperforms existing SOTA methods across multiple tasks.
Read the full abstract ▾
All authors Xiaopeng Li, Yuanjin Zheng, Wanyu Wang, wenlin zhang, Pengyue Jia, Yiqi Wang, Maolin Wang, Xuetao Wei, Xiangyu Zhao
Computation and Language
Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
Huiyu Bai, Runze Wang, Zhuoyun Du et al. (+6 more)
Personalized Visual
Language Models (VLMs) are gaining increasing
attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate
embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient
retrieval of concept
embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging
hyperbolic representations. Our approach makes a train-free paradigm for concept
embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale
benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and
dataset will be made available.
Read the full abstract ▾
All authors Huiyu Bai, Runze Wang, Zhuoyun Du, Yiyang Zhao, Fengji Zhang, Haoyu Chen, Xiaoyong Zhu, Bo Zheng, Xuejiao Zhao
Computation and Language
A Machine Learning Approach for Detection of Mental Health Conditions and Cyberbullying from Social Media
Edward Ajayi, Martha Kachweka, Mawuli Deku et al. (+1 more)
Mental health challenges and cyberbullying are increasingly prevalent in digital spaces, necessitating scalable and interpretable detection systems. This paper introduces a unified multiclass classification framework for detecting ten distinct mental health and cyberbullying categories from social media data. We curate
datasets from Twitter and Reddit, implementing a rigorous "split-then-balance" pipeline to train on balanced data while evaluating on a realistic, held-out imbalanced test set. We conducted a comprehensive evaluation comparing traditional lexical models, hybrid approaches, and several end-to-end fine-tuned
transformers. Our results demonstrate that end-to-end fine-tuning is critical for performance, with the domain-adapted MentalBERT emerging as the top model, achieving an accuracy of 0.92 and a Macro F1 score of 0.76, surpassing both its generic counterpart and a zero-shot LLM baseline. Grounded in a comprehensive ethical analysis, we frame the system as a human-in-the-loop screening aid, not a diagnostic tool. To support this, we introduce a hybrid SHAPLLM explainability framework and present a prototype dashboard ("Social Media Screener") designed to integrate model predictions and their explanations into a practical workflow for moderators. Our work provides a robust baseline, highlighting future needs for multi-label, clinically-validated
datasets at the critical intersection of online safety and computational mental health.
Read the full abstract ▾
All authors Edward Ajayi, Martha Kachweka, Mawuli Deku, Emily Aiken
Computation and Language
Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test
Mihir Sahasrabudhe
Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings. Yet empirical studies on natural
language repeatedly report a "reversal curse," and recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time. This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself? We cut through this ambiguity with a fully synthetic,
entropy-controlled
benchmark designed as a clean-room stress test for directional learning. Using random string mappings with tunable branching factor K, we construct forward tasks with zero conditional
entropy and inverse tasks with analytically determined
entropy floors. Excess loss above these floors reveals that even scratch-trained GPT-2 models exhibit a strong, reproducible directional optimization gap (e.g., 1.16 nats at K=5), far larger than that of an MLP trained on the same data. Pre-trained initializations shift optimization behavior but do not eliminate this gap, while LoRA encounters a sharp capacity wall on high-
entropy inverse mappings. Together, these results isolate a minimal, semantics-free signature of directional friction intrinsic to causal
Transformer training-one that persists even when linguistic priors,
token frequencies, and corpus-level temporal asymmetries are removed. Our
benchmark provides a controlled instrument for dissecting directional biases in modern sequence models and motivates deeper mechanistic study of why inversion remains fundamentally harder for
Transformers.
Read the full abstract ▾
Computation and Language
$\text{R}^2\text{R}$: A Route-to-Rerank Post-Training Framework for Multi-Domain Decoder-Only Rerankers
Xinyu Wang, Hanwei Wu, Qingchen Hu et al. (+13 more)
Decoder-only rerankers are central to
Retrieval-Augmented Generation (RAG). However, generalist models miss domain-specific nuances in high-stakes fields like finance and law, and naive fine-tuning causes surface-form overfitting and catastrophic forgetting. To address this challenge, we introduce R2R, a domain-aware framework that combines dynamic expert routing with a two-stage training strategy, Entity Abstraction for Generalization (EAG). EAG introduces a counter-shortcut mechanism by masking the most predictive surface cues, forcing the reranker to learn domain-invariant relevance patterns rather than memorizing
dataset-specific entities. To efficiently activate domain experts, R2R employs a lightweight Latent Semantic Router that probes internal representations from the frozen backbone decoder to select the optimal LoRA expert per query. Extensive experiments across different reranker backbones and diverse domains (legal, medical, and financial) demonstrate that R2R consistently surpasses generalist and single-domain fine-tuned baselines. Our results confirm that R2R is a model-agnostic and modular approach to domain specialization with strong cross-domain robustness.
Read the full abstract ▾
All authors Xinyu Wang, Hanwei Wu, Qingchen Hu, Zhenghan Tai, Jingrui Tian, Lei Ding, Jijun Chi, Hailin He, Tung Sum Thomas Kwok, Yufei Cui, Sicheng Lyu, Muzhi Li, Mingze Li, Xinyue Yu, Ling Zhou, Peng Lu
Computation and Language
AppSelectBench: Application-Level Tool Selection Benchmark
Tianyi Chen, Michael Solodko, Sen Wang et al. (+14 more)
Computer Using
Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the
agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing
benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive
benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and
retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large
language models reveal systematic strengths and weaknesses in inter-application
reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level
reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://microsoft.github.io/appselectbench/.
Read the full abstract ▾
All authors Tianyi Chen, Michael Solodko, Sen Wang, Jongwoo Ko, Junheng Hao, Colby Banbury, Sara Abdali, Saeed Amizadeh, Qing Xiao, Yinheng Li, Tianyu Ding, Kamran Ghasedi Dizaji, Suzhen Zheng, Hao Fan, Justin Wagle, Pashmina Cameron, Kazuhito Koishida
Computation and Language · Cross-listed from cs.LG
EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
Songlin Zhao, Michael Pitts and Zhuwei Qin
Deploying domain-specialized large
language models on resource-constrained hardware motivates reducing both adaptation cost and the number of retained weights. LoRA lowers adaptation cost but leaves a dense backbone, while separate
pruning can discard weights that become important during adaptation. We propose EfficientXpert, a framework that co-adapts low-rank updates and sparse support. Its ForeSight Mask scores weights through a downstream reconstruction surrogate using the evolving LoRA-augmented weights; Partial
Brain Surgeon (PBS) realigns the adapter through a closed-form correction. At 40% sparsity, the strongest variant retains 98.55–108.99% of dense-LoRA aggregate performance across the evaluated LLaMA settings. On Qwen3-8B, adaptation adds 17.9–27.0% training time and at most 4.8% peak GPU
memory. Our analysis connects the domain- and rank-dependent effects of PBS to its constrained reconstruction objective, providing guidance for selecting adapter recovery capacity. Together, these results demonstrate efficient production of sparse, domain-specialized experts. Code is available at https://github.com/TagoreZhao/Efficient_Domain_Adaptation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding
Yuefei Chen, Jiang Liu, Xiaodong Lin et al. (+1 more)
Vision Language Models (VLMs) have recently shown significant advancements in
video understanding, especially in feature
alignment, event
reasoning, and instruction-following tasks. However, their capability for counterfactual
reasoning, inferring alternative outcomes under hypothetical conditions, remains underexplored. This capability is essential for robust
video understanding, as it requires identifying underlying causal structures and
reasoning about unobserved possibilities, rather than merely recognizing observed patterns. To systematically evaluate this capability, we introduce CounterVQA, a
video-based
benchmark featuring three progressive difficulty levels that assess different aspects of counterfactual
reasoning. Through comprehensive evaluation of both state-of-the-art open-source and closed-source models, we uncover a substantial performance gap: while these models achieve reasonable accuracy on simple counterfactual questions, performance degrades significantly on complex multi-hop causal chains. To address these limitations, we develop a post-training method, CFGPT, that enhances a model's visual counterfactual
reasoning ability by distilling its counterfactual
reasoning capability from the
language modality, yielding consistent improvements across all CounterVQA difficulty levels.
Dataset and code will be further released.
Read the full abstract ▾
All authors Yuefei Chen, Jiang Liu, Xiaodong Lin, Ruixiang Tang