Computation and Language
A System for Name and Address Parsing with Large Language Models
Adeeba Tarannum, Muzakkiruddin Ahmed Mohammed, Mert Can Cakmak et al. (+2 more)
Reliable transformation of unstructured person and address text into structured data remains a key challenge in large-scale information systems. Traditional rule-based and probabilistic approaches perform well on clean inputs but fail under noisy or
multilingual conditions, while neural and large
language models (LLMs) often lack deterministic control and reproducibility. This paper introduces a
prompt-driven, validation-centered framework that converts free-text records into a consistent 17-field schema without fine-tuning. The method integrates input normalisation, structured prompting, constrained decoding, and strict rule-based validation under fixed experimental settings to ensure reproducibility. Evaluations on heterogeneous real-world address data show high field-level accuracy, strong schema adherence, and stable confidence calibration. The results demonstrate that combining deterministic validation with generative prompting provides a robust, interpretable, and scalable solution for structured information extraction, offering a practical alternative to training-heavy or domain-specific models.
Read the full abstract ▾
All authors Adeeba Tarannum, Muzakkiruddin Ahmed Mohammed, Mert Can Cakmak, Shames Al Mandalawi, John Talburt
Computation and Language
Evaluating Semantic and Syntactic Understanding in Large Language Models for Payroll Systems
Hendrika Maclean, Mert Can Cakmak, Muzakkiruddin Ahmed Mohammed et al. (+2 more)
Large
language models are now used daily for writing, search, and analysis, and their natural
language understanding continues to improve. However, they remain unreliable on exact numerical calculation and on producing outputs that are straightforward to audit. We study synthetic payroll system as a focused, high-stakes example and evaluate whether models can understand a payroll schema, apply rules in the right order, and deliver cent-accurate results. Our experiments span a tiered
dataset from basic to complex cases, a spectrum of
prompts from minimal baselines to schema-guided and
reasoning variants, and multiple model families including GPT, Claude, Perplexity, Grok and Gemini. Results indicate clear regimes where careful prompting is sufficient and regimes where explicit computation is required. The work offers a compact, reproducible framework and practical guidance for deploying LLMs in settings that demand both accuracy and assurance.
Read the full abstract ▾
All authors Hendrika Maclean, Mert Can Cakmak, Muzakkiruddin Ahmed Mohammed, Shames Al Mandalawi, John Talburt
Computation and Language
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
Lorenzo Proietti, Roman Grundkiewicz and Matt Post
We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised quality estimation (QE) metric family that reframes reference-free machine translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation
benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for minimum Bayes risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.
Read the full abstract ▾
Computation and Language
AI-based approach to burnout identification from textual data
Marina Zavertiaeva, Petr Parshakov, Mikhail Usanin et al. (+3 more)
This study introduces an AI-based methodology that utilizes natural
language processing (NLP) to detect burnout from textual data. The approach relies on a RuBERT model originally trained for sentiment analysis and subsequently fine-tuned for burnout detection using two data sources: synthetic sentences generated with ChatGPT and user comments collected from Russian YouTube
videos about burnout. The resulting model assigns a burnout probability to input texts and can be applied to process large volumes of written communication for monitoring burnout-related
language signals in high-stress work environments.
Read the full abstract ▾
All authors Marina Zavertiaeva, Petr Parshakov, Mikhail Usanin, Aleksei Smirnov, Sofia Paklina, Anastasiia Kibardina
Computation and Language
SD-E$^2$: Semantic Exploration for Reasoning Under Token Budgets
Kshitij Mishra, Nils Lukas and Salem Lahlou
Small
language models (SLMs) struggle with complex
reasoning because exploration is expensive under tight compute budgets. We introduce Semantic Diversity-Exploration-Exploitation (SD-E$^2$), a reinforcement learning framework that makes exploration explicit by optimizing semantic diversity in generated
reasoning trajectories. Using a frozen sentence-
embedding model, SD-E$^2$ assigns a diversity reward that captures (i) the coverage of semantically distinct solution strategies and (ii) their average pairwise dissimilarity in
embedding space, rather than surface-form novelty. This diversity reward is combined with outcome correctness and solution efficiency in a z-score-normalized multi-objective objective that stabilizes training. On GSM8K, SD-E$^2$ surpasses the base Qwen2.5-3B-Instruct and strong GRPO baselines (GRPO-CFL and GRPO-CFEE) by +27.4, +5.2, and +1.5 percentage points, respectively, while discovering on average 9.8 semantically distinct strategies per question. We further improve MedMCQA to 49.64% versus 38.37% for the base model and show gains on the harder AIME
benchmark (1983-2025), reaching 13.28% versus 6.74% for the base. These results indicate that rewarding semantic novelty yields a more compute-efficient exploration-exploitation signal for training
reasoning-capable SLMs. By introducing cognitive adaptation-adjusting the
reasoning process structure rather than per-
token computation-SD-E$^2$ offers a complementary path to efficiency gains in resource-constrained models.
Read the full abstract ▾
Computation and Language
LLMs as Cultural Archives: Cultural Commonsense Knowledge Graph Extraction
Junior Cedric Tonga, Chen Cecilia Liu, Iryna Gurevych et al. (+1 more)
Large
language models (LLMs) encode rich cultural knowledge learned from diverse web-scale data, offering an unprecedented opportunity to model cultural commonsense at scale. Yet this knowledge remains mostly implicit and unstructured, limiting its interpretability and use. We present an iterative,
prompt-based framework for constructing a Cultural Commonsense Knowledge Graph (CCKG) that treats LLMs as cultural archives, systematically eliciting culture-specific entities, relations, and practices and composing them into multi-step inferential chains across
languages. We evaluate CCKG on five countries with human judgments of cultural relevance, correctness, and path coherence. We find that the cultural knowledge graphs are better realized in English, even when the target culture is non-English (e.g., Chinese, Indonesian, Arabic), indicating uneven cultural encoding in current LLMs. Augmenting smaller LLMs with CCKG improves performance on cultural
reasoning and story generation, with the largest gains from English chains. Our results show both the promise and limits of LLMs as cultural technologies and that chain-structured cultural knowledge is a practical substrate for culturally grounded NLP.
Read the full abstract ▾
All authors Junior Cedric Tonga, Chen Cecilia Liu, Iryna Gurevych, Fajri Koto
Computation and Language
A Monosemantic Attribution Framework for Stable Interpretability in Clinical Neuroscience Transformer-Based Language Models
Michail Mamalakis, Tiago Azevedo, Cristian Cosentino et al. (+5 more)
Interpretability remains a key challenge for deploying
language models (LM) in clinical settings such as progression diagnosis of Alzheimer disease, where early and trustworthy predictions are essential. Existing attribution methods exhibit high inter-method variability and unstable explanations due to the polysemantic nature of
Transformer-Based LM and LLM representations, while mechanistic interpretability approaches lack direct
alignment with model inputs and outputs and do not provide explicit importance scores. We introduce a unified interpretability framework that integrates attributional and mechanistic perspectives through monosemantic feature extraction. By constructing a monosemantic
embedding space at the level of an
transformer-based LM layer and optimizing the framework to explicitly reduce inter-method variability, our approach produces stable input-level importance scores and highlights salient features via a decompressed representation of the layer of interest, advancing the safe and trustworthy application of LMs in cognitive health and neurodegenerative disease.
Read the full abstract ▾
All authors Michail Mamalakis, Tiago Azevedo, Cristian Cosentino, Chiara D'Ercoli, Subati Abulikemu, Zhongtian Sun, Richard Bethlehem, Pietro Lio
Computation and Language
ShapLoRA: Allocation of Low-rank Adaption on Large Language Models via Shapley Value Inspired Importance Estimation
Yi Zhao, Qinghua Yao, Xinyuan song et al. (+1 more)
Low-rank adaption (LoRA) is a representative method in the field of parameter-efficient fine-tuning (PEFT), and is key to Democratizating the modern large
language models (LLMs). The vanilla LoRA is implemented with uniform ranks, and the recent literature have found that properly allocating ranks on the LLM backbones results in performance boosts. However, the previous rank allocation methods have limitations since they rely on inexplanable and unreliable importance measures for the LoRA ranks. To address the above issues, we propose the ShapLoRA framework. Inspired by the explanable attribution measure Shapley Value, we combine the sensitivity-based measures with the idea of coalitions in the collaborative games among LoRA ranks, and propose a more explainable importance measure called Shapley sensitivity. In addition, we optimize the workflow of the existing works by: (a) calculating Shapley sensitivity on a separate validation set; (b) Setting up the allocating-retraining procedures for fair comparisons. We have conducted experiments on various challenging tasks, and the experimental results demonstrate that our ShapLoRA method can outperform the recent baselines with comparable tunable parameters.\footnote{Codes and fine-tuned models will be open-sourced to facilitate future research.
Read the full abstract ▾
All authors Yi Zhao, Qinghua Yao, Xinyuan song, Wei Zhu
Computation and Language · Cross-listed from cs.CV
Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
Dain Kim, Jiwoo Lee, Jaehoon Yun et al. (+4 more)
Large
Vision-
Language Models (LVLMs) hold significant promise for medical applications, yet their deployment is often constrained by insufficient
alignment and reliability. While Direct Preference Optimization (DPO) has emerged as a potent framework for refining model responses, its efficacy in high-stakes medical contexts remains underexplored, lacking the rigorous empirical groundwork necessary to guide future methodological advances. To bridge this gap, we present the first comprehensive examination of diverse DPO variants within the medical domain, evaluating nine distinct formulations across two medical LVLMs: LLaVA-Med and HuatuoGPT-
Vision. Our results reveal several critical limitations: current DPO approaches often yield inconsistent gains over supervised fine-tuning, with their efficacy varying significantly across different tasks and backbones. Furthermore, they frequently fail to resolve fundamental visual misinterpretation errors. Building on these insights, we present a targeted preference construction strategy as a proof-of-concept that explicitly addresses visual misinterpretation errors frequently observed in existing DPO models. This design yields a 3.6% improvement over the strongest existing DPO baseline on visual question-answering tasks. To support future research, we release our complete framework, including all training data, model checkpoints, and our codebase at https://github.com/dmis-lab/med-vlm-dpo.
Read the full abstract ▾
All authors Dain Kim, Jiwoo Lee, Jaehoon Yun, Yong Hoe Koo, Qingyu Chen, Hyunjae Kim, Jaewoo Kang
Computation and Language · Cross-listed from cs.LG
Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo et al. (+4 more)
Diffusion Large
Language Models (dLLMs) offer a compelling paradigm for natural
language generation, leveraging parallel decoding and bidirectional
attention to achieve superior global coherence compared to autoregressive models. While recent works have accelerated inference via KV cache reuse or heuristic decoding, they overlook the intrinsic inefficiencies within the block-wise
diffusion process. Specifically, they suffer from spatial redundancy by modeling informative-sparse suffix regions uniformly and temporal inefficiency by applying fixed denoising schedules across all the decoding process. To address this, we propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions. Spatially, we introduce attenuation guided suffix modeling to approximate the full context by
pruning redundant mask
tokens. Temporally, we employ a dynamic confidence aware strategy with an early exit mechanism, allowing the model to skip unnecessary iterations for converged
tokens. Extensive experiments show that Streaming-dLLM achieves up to 68.2X speedup while maintaining generation quality, highlighting its effectiveness in
diffusion decoding. The code is available at https://github.com/xiaoshideta/Streaming-dLLM.
Read the full abstract ▾
All authors Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo, Yong Luo, Jia Liu, Jie Xu, Han Hu
Computation and Language
Assessment of Generative Named Entity Recognition in the Era of Large Language Models
Qi Zhan, Yile Wang and Hui Huang
Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large
language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER
datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass decoder-based LLMs with in-context learning techniques; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on
datasets like DROP by 25.50 to 45.32 F1 points due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Artificial Intelligence and Intellectual Property Rights: Comparative Transnational Policy Analysis
Sahibpreet Singh and Manjit Singh
Artificial intelligence's rapid integration with intellectual property rights necessitates assessment of its impact on trade secrets, copyrights and patents. This study addresses lacunae in existing laws where India lacks AI-specific provisions, creating doctrinal inconsistencies and enforcement inefficacies. Global discourse on AI-IPR protections remains nascent. The research identifies gaps in Indian IP laws' adaptability to AI-generated outputs: trade secret protection is inadequate against AI threats; standardized inventorship criteria are absent. Employing doctrinal and comparative methodology, it scrutinizes legislative texts, judicial precedents and policy instruments across India, US, UK and EU. Preliminary findings reveal shortcomings: India's contract law creates fragmented trade secret regime; Section 3(k) of Indian Patents Act blocks AI invention patenting; copyright varies in authorship attribution. The study proposes harmonized legal taxonomy accommodating AI's role while preserving innovation incentives. India's National AI Strategy (2024) shows progress but legislative clarity is imperative. This contributes to global discourse with AI-specific IP protections ensuring resilience and equitable innovation. Promising results underscore recalibrating India's IP jurisprudence for global
alignment.
Read the full abstract ▾
Computation and Language
Self-Manager: Parallel Agent Loop for Long-form Deep Research
Yilong Xu, Zhi Zheng, Xiang Long et al. (+2 more)
Long-form deep research requires multi-faceted investigations over extended horizons to get a comprehensive report. When handling such complex tasks, existing
agents manage context at the subtask level to overcome linear context accumulation and information loss. However, they still adhere to a single context window and sequential execution paradigm, which results in mutual interference and blocking behavior, restricting scalability and adaptability. To address this issue, this paper introduces Self-Manager, a parallel
agent loop that enables asynchronous and concurrent execution. The main thread can create multiple subthreads, each with its own isolated context, and manage them iteratively through Thread Control Blocks, allowing for more focused and flexible parallel
agent execution. To assess its effectiveness, we
benchmark Self-Manager on DeepResearch Bench, where it consistently outperforms existing single-
agent loop baselines across all metrics. Furthermore, we conduct extensive analytical experiments to demonstrate the necessity of Self-Manager's design choices, as well as its advantages in contextual capacity, efficiency, and generalization.
Read the full abstract ▾
All authors Yilong Xu, Zhi Zheng, Xiang Long, Yujun Cai, Yiwei Wang
Computation and Language
On the Emergence and Test-Time Use of Structural Information in Large Language Models
Michelle Chao Chen, Moritz Miller, Bernhard Schölkopf et al. (+1 more)
Learning structural information from observational data is central to producing new knowledge outside the training corpus. This holds for mechanistic understanding in scientific discovery as well as flexible test-time compositional generation. We thus study how
language models learn abstract structures and utilize the learnt structural information at test-time. To ensure a controlled setup, we design a natural
language dataset based on linguistic structural transformations. We empirically show that the emergence of learning structural information correlates with complex
reasoning tasks, and that the ability to perform test-time compositional generation remains limited.
Read the full abstract ▾
All authors Michelle Chao Chen, Moritz Miller, Bernhard Schölkopf, Siyuan Guo
Computation and Language
D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language Models
Jia Gu, Liang Pang, Huawei Shen et al. (+1 more)
The predictive probability of the next
token (P_
token) in large
language models (LLMs) is inextricably linked to the probability of relevance for the next piece of information, the purchase probability of the next product, and the execution probability of the next action-all of which fall under the scope of the task-level target distribution (P_task). While LLMs are known to generate samples that approximate real-world distributions, whether their fine-grained sampling probabilities faithfully align with task requirements remains an open question. Through controlled distribution-sampling simulations, we uncover a striking dichotomy in LLM behavior, distinguishing two model types: D-models (e.g. Qwen-2.5), whose P_
token exhibits large step-to-step variability and poor
alignment with P_task; and E-models (e.g. Mistral-Small), whose P_
token is more stable and better aligned with P_task. We further evaluate these two model types in downstream tasks such as code generation and recommendation, revealing systematic trade-offs between diversity and stability that shape task outcomes. Finally, we analyze the internal properties of both model families to probe their underlying mechanisms. These findings offer foundational insights into the probabilistic sampling behavior of LLMs and provide practical guidance on when to favor D- versus E-models. For web-scale applications, including recommendation, search, and conversational
agents, our results inform model selection and configuration to balance diversity with reliability under real-world uncertainty, providing a better level of interpretation.
Read the full abstract ▾
All authors Jia Gu, Liang Pang, Huawei Shen, Xueqi Cheng
Computation and Language
EFT-CoT: A Multi-Agent Chain-of-Thought Framework for Emotion-Focused Therapy
Lanqing Du, Yunong Li, YuJie Long et al. (+1 more)
The use of large
language models (LLMs) for Mental Health Question Answering (MHQA) offers a promising way to alleviate shortages in mental health resources. However, prior work has mainly relied on Cognitive Behavioral Therapy (CBT) and predominantly follows a top-down strategy centered on rational cognitive restructuring, providing limited support for
embodied experience and primary
emotion processing. To address this gap, we propose EFT-CoT, a multi-
agent chain-of-thought framework grounded in
Emotion-Focused Therapy (EFT). EFT-CoT operationalizes intervention as a three-stage workflow:
Embodied Perception, Cognitive Exploration, and Narrative Intervention. The framework employs eight specialized
agents to model key processes including somatic awareness mapping, adaptive evaluation, core belief extraction, and narrative restructuring. Based on this framework, we construct EFT-Instruct, a high-quality instruction-tuning
dataset built from process-level augmentation of about 67,000 real help-seeking texts, and further fine-tune a dedicated model, EFT-LLM. Experiments show that EFT-LLM consistently outperforms strong baselines and human responses in empathic depth and structural professionalism. Ablation studies further verify the contribution of key mechanisms, while white-box
auditing demonstrates the consistency and traceability of critical intermediate states. Overall, this work provides a reproducible framework-data-model pipeline for
embedding EFT mechanisms into LLM-based mental health support.
Read the full abstract ▾
All authors Lanqing Du, Yunong Li, YuJie Long, Shihong Chen
Computation and Language
Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agents
Dan Greenstein, Zohar Karnin, Chen Amiraz et al. (+1 more)
The construction of function calling
agents has emerged as a promising avenue for extending model capabilities. A major challenge for this task is obtaining high quality diverse data for training. Prior work emphasizes diversity in functions, invocation patterns, and interaction turns, yet linguistic diversity of requests and coverage of arguments (e.g., city\_name, stock\_ticker) remain underexplored. We propose a method that generates synthetic
datasets via optimizing general-purpose diversity metrics across both queries and arguments, without relying on hand-crafted rules or taxonomies, making it robust to different usecases. We demonstrate the effectiveness of our technique via both intrinsic and extrinsic testing, comparing it to SoTA data generation methods. We show a superiority over baselines in terms of diversity, while keeping comparable correctness. Additionally, when used as a training set, the model resulting from our
dataset exhibits superior performance compared to analogous models based on the baseline data generation methods in out-of-distribution performance. In particular, we achieve an $7.4\%$ increase in accuracy on the BFCL
benchmark compared to similar counterparts.
Read the full abstract ▾
All authors Dan Greenstein, Zohar Karnin, Chen Amiraz, Oren Somekh
Computation and Language
DIETA: A Decoder-only transformer-based model for Italian-English machine TrAnslation
Pranav Kasela, Marco Braga, Alessandro Ghiotto et al. (+3 more)
In this paper, we present DIETA, a small, decoder-only
Transformer model with 0.5 billion parameters, specifically designed and trained for Italian-English machine translation. We collect and curate a large parallel corpus consisting of approximately 207 million Italian-English sentence pairs across diverse domains, including parliamentary proceedings, legal texts, web-crawled content, subtitles, news, literature and 352 million back-translated data using pretrained models. Additionally, we create and release a new small-scale evaluation set, consisting of 450 sentences, based on 2025 WikiNews articles, enabling assessment of translation quality on contemporary text. Comprehensive evaluations show that DIETA achieves competitive performance on multiple Italian-English
benchmarks, consistently ranking in the second quartile of a 32-system leaderboard and outperforming most other sub-3B models on four out of five test suites. The training script, trained models, curated corpus, and newly introduced evaluation set are made publicly available, facilitating further research and development in specialized Italian-English machine translation. https://github.com/pkasela/DIETA-Machine-Translation
Read the full abstract ▾
All authors Pranav Kasela, Marco Braga, Alessandro Ghiotto, Andrea Pilzer, Marco Viviani, Alessandro Raganato
Computation and Language
Beyond a Single Perspective: Text Anomaly Detection with Multi-View Language Representations
Yixin Liu, Kehan Yan, Shiyuan Li et al. (+2 more)
Text anomaly detection (TAD) plays a critical role in various
language-driven real-world applications, including harmful content moderation, phishing detection, and spam review filtering. While two-step "
embedding-detector" TAD methods have shown state-of-the-art performance, their effectiveness is often limited by the use of a single
embedding model and the lack of adaptability across diverse
datasets and anomaly types. To address these limitations, we propose to exploit the
embeddings from multiple pretrained
language models and integrate them into $MCA^2$, a multi-view TAD framework. $MCA^2$ adopts a multi-view reconstruction model to effectively extract normal textual patterns from multiple
embedding perspectives. To exploit inter-view complementarity, a contrastive collaboration module is designed to leverage and strengthen the interactions across different views. Moreover, an adaptive allocation module is developed to automatically assign the contribution weight of each view, thereby improving the adaptability to diverse
datasets. Extensive experiments on 10
benchmark datasets verify the effectiveness of $MCA^2$ against strong baselines. The source code of $MCA^2$ is available at https://github.com/yankehan/MCA2.
Read the full abstract ▾
All authors Yixin Liu, Kehan Yan, Shiyuan Li, Qingfeng Chen, Shirui Pan
Computation and Language
Controlling Reading Ease with Gaze-Guided Text Generation
Andreas Säuberli, Darja Jepifanova, Diego Frassinelli et al. (+1 more)
The way our eyes move while reading can tell us about the cognitive effort required to process the text. In the present study, we use this fact to generate texts with controllable reading ease. Our method employs a model that predicts human gaze patterns to steer
language model outputs towards eliciting certain reading behaviors. We evaluate the approach in an eye-tracking experiment with native and non-native speakers of English. The results demonstrate that the method is effective at making the generated texts easier or harder to read, measured both in terms of reading times and perceived difficulty of the texts. A statistical analysis reveals that the changes in reading behavior are mostly due to features that affect lexical processing. Possible applications of our approach include text simplification for information accessibility and generation of personalized educational material for
language learning.
Read the full abstract ▾
All authors Andreas Säuberli, Darja Jepifanova, Diego Frassinelli, Barbara Plank
Computation and Language
DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning
Xiaoyu Liu, Xiaoyu Guan, Di Liang et al. (+1 more)
Supervised fine-tuning (SFT) is a crucial step for adapting large
language models (LLMs) to downstream tasks. However, conflicting objectives across heterogeneous SFT tasks often induce the "seesaw effect": optimizing for one task may degrade performance on others, particularly when model parameters are updated indiscriminately. In this paper, we propose a principled approach to disentangle and isolate task-specific parameter regions, motivated by the hypothesis that parameter heterogeneity underlies cross-task interference. Specifically, we first independently fine-tune LLMs on diverse SFT tasks and identify each task's core parameter region as the subset of parameters exhibiting the largest updates. Tasks with highly overlapping core parameter regions are merged for joint training, while disjoint tasks are organized into different stages. During multi-stage SFT, core parameters acquired in prior tasks are frozen, thereby preventing overwriting by subsequent tasks. To verify the effectiveness of our method, we conducted intensive experiments on multiple public
datasets. The results showed that our dynamic parameter isolation strategy consistently reduced data conflicts and achieved consistent performance improvements compared to multi-stage and multi-task tuning baselines.
Read the full abstract ▾
All authors Xiaoyu Liu, Xiaoyu Guan, Di Liang, Xianjie Wu
Computation and Language
Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali
Md Asgor Hossain Reaj, Rajan Das Gupta, Jui Saha Pritha et al. (+4 more)
Large
Language Models (LLMs) have achieved significant success in recent years; yet, issues of intrinsic gender bias persist, especially in non-English
languages. Although current research mostly emphasizes English, the linguistic and cultural biases inherent in Global South
languages, like Bengali, are little examined. This research seeks to examine the characteristics and magnitude of gender bias in Bengali, evaluating the efficacy of current approaches in identifying and alleviating bias. We use several methods to extract gender-biased utterances, including lexicon-based mining, computational classification models, translation-based comparison analysis, and GPT-based bias creation. Our research indicates that the straight application of English-centric bias detection frameworks to Bengali is severely constrained by
language disparities and socio-cultural factors that impact implicit biases. To tackle these difficulties, we executed two field investigations inside rural and low-income areas, gathering authentic insights on gender bias. The findings demonstrate that gender bias in Bengali presents distinct characteristics relative to English, requiring a more localized and context-sensitive methodology. Additionally, our research emphasizes the need of integrating community-driven research approaches to identify culturally relevant biases often neglected by automated systems. Our research enhances the ongoing discussion around gender bias in AI by illustrating the need to create linguistic tools specifically designed for underrepresented
languages. This study establishes a foundation for further investigations into bias reduction in Bengali and other Indic
languages, promoting the development of more inclusive and fair NLP systems.
Read the full abstract ▾
All authors Md Asgor Hossain Reaj, Rajan Das Gupta, Jui Saha Pritha, Abdullah Al Noman, Abir Ahmed, Golam Md Mohiuddin, Tze Hui Liew
Computation and Language · Cross-listed from cs.LG
AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation
Dongjie Cheng, Ruifeng Yuan, Yongqi Li et al. (+5 more)
Real-world perception and interaction are inherently
multimodal, encompassing not only
language but also
vision and
speech, which motivates the development of "Omni" MLLMs that support both
multimodal inputs and
multimodal outputs. While a sequence of omni MLLMs has emerged, most existing systems still rely on additional expert components to achieve
multimodal generation, limiting the simplicity of unified training and inference. Autoregressive (AR) modeling, with a single
token stream, a single next-
token objective, and a single decoder, is an elegant and scalable foundation in the text domain. Motivated by this, we present AR-Omni, a unified any-to-any model in the autoregressive paradigm without any expert decoders. AR-Omni supports autoregressive text and image generation, as well as streaming
speech generation, all under a single
Transformer decoder. We further address three practical issues in unified AR modeling: modality imbalance via task-aware loss reweighting, visual fidelity via a lightweight
token-level perceptual
alignment loss for image
tokens, and stability-creativity trade-offs via a finite-state decoding mechanism. Empirically, AR-Omni achieves strong quality across three modalities while remaining real-time, achieving a 0.88 real-time factor for
speech generation.
Read the full abstract ▾
All authors Dongjie Cheng, Ruifeng Yuan, Yongqi Li, Runyang You, Wenjie Wang, Liqiang Nie, Lei Zhang, Wenjie Li
Computation and Language
HyperGraphPro: Progress-Aware Reinforcement Learning for Structure-Guided Hypergraph RAG
Jinyoung Park, Sanghyeok Lee, Omar Zia Khan et al. (+2 more)
Multi-hop question answering requires
retrieval agents to construct a connected evidence path across
reasoning turns rather than retrieve isolated relevant evidence for a single query. However, in existing GraphRAG methods,
retrieval and reinforcement learning are often optimized with mismatched signals:
retrieval is driven by local semantic relevance, while outcome-level reinforcement learning assigns the reward by assessing the final answer without identifying which
retrieval actions progress the
reasoning path. We introduce HyperGraphPro, a progress-aware hypergraph RAG framework that focuses on evidence-path progress as the shared objective for
retrieval and policy optimization. For
retrieval, HyperGraphPro reranks candidate hyperedges using query-conditioned structural distinctiveness, combining entity-query relevance with a local-vs-global hypergraph incidence signal to favor entities that are informative in the current graph neighborhood. For policy optimization, HyperGraphPro augments uniform trajectory-level reward assignment with stepwise progress-aware policy optimization, which assigns step-level rewards according to the coverage of answer-guided target hyperedges. Experiments on multi-hop question answering
benchmarks demonstrate that HyperGraphPro consistently improves
reasoning accuracy over existing GraphRAG methods.
Read the full abstract ▾
All authors Jinyoung Park, Sanghyeok Lee, Omar Zia Khan, Hyunwoo J. Kim, Joo-Kyung Kim
Computation and Language
Hylog: A Hybrid Approach to Logging Text Production in Non-alphabetic Scripts
Roberto Crotti, Giovanni Denaro, Zhiqiang Du et al. (+1 more)
Research keyloggers are essential for cognitive studies of text production, yet most fail to capture the on-screen transformations performed by Input Method Editors (IMEs) for non-alphabetic scripts. To address this methodological gap, we present Hylog, a novel hybrid logging system that combines analytical keylogging with ecological text logging for a more complete and finer-grained analysis. Our modular, open-source system uses plug-ins for standard applications (Microsoft Word, Google Chrome) to capture both keyboard output and rendered text, which a hybridizer module then synchronizes into a dual trace. To validate the system's technical feasibility and demonstrate its analytical capabilities, we conducted a proof-of-concept study where two volunteers translated a text into simplified Chinese. Hylog successfully captured keypresses and temporal intervals between Latin letters, Chinese characters, and IME confirmations – some measurements invisible to traditional keyloggers. The resulting data enable the formulation of new, testable hypotheses about the cognitive restrictions and affordances at different linguistic layers in IME-mediated typing. Our plug-in architecture enables extension to other IME systems and fosters more inclusive
multilingual text-production research.
Read the full abstract ▾
All authors Roberto Crotti, Giovanni Denaro, Zhiqiang Du, Ricardo Muñoz Martín
Computation and Language
Unsupervised Elicitation of Moral Values from Language Models
Meysam Alizadeh, Fabrizio Gilardi and Zeynab Samei
As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that
language models (LMs) exhibit limited inherent moral
reasoning, leading to calls for explicit moral teaching. However, constructing ground truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral
reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM)
algorithm across three
benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pre-trained and chatbot baselines on the Norm Bank and ETHICS
benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral
reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI
alignment.
Read the full abstract ▾
Computation and Language
A Computational Approach to Visual Metonymy
Saptarshi Ghosh, Linfeng Liu and Tianyu Jiang
Images often communicate more than they literally depict: a set of tools can suggest an occupation and a cultural artifact can suggest a tradition. This kind of indirect visual reference, known as visual metonymy, invites viewers to recover a target concept via associated cues rather than explicit depiction. In this work, we present the first computational investigation of visual metonymy. We introduce a novel pipeline grounded in semiotic theory that leverages large
language models and text-to-image models to generate metonymic visual representations. Using this framework, we construct ViMET, the first visual metonymy
dataset comprising 2,000 multiple-choice questions to evaluate the cognitive
reasoning abilities in
multimodal language models. Experimental results on our
dataset reveal a significant gap between human performance (86.9%) and state-of-the-art
vision-
language models (65.9%), highlighting limitations in machines' ability to interpret indirect visual references. Our
dataset is publicly available at: https://github.com/cincynlp/ViMET.
Read the full abstract ▾
Computation and Language
Distance-to-Distance Ratio: A Similarity Measure for Sentences Based on Rate of Change in LLM Embeddings
Abdullah Qureshi, Kenneth Rice and Alexander Wolpert
A measure of similarity between text
embeddings can be considered adequate only if it adheres to the human perception of similarity between texts. In this paper, we introduce the distance-to-distance ratio (DDR), a novel measure of similarity between LLM sentence
embeddings. Inspired by Lipschitz continuity, DDR measures the rate of change in similarity between the pre-context word
embeddings and the similarity between post-context LLM
embeddings, thus measuring the semantic influence of context. We evaluate the performance of DDR in experiments designed as a series of perturbations applied to sentences drawn from a sentence
dataset. For each sentence, we generate variants by replacing one, two, or three words with either synonyms, which constitute semantically similar text, or randomly chosen words, which constitute semantically dissimilar text. We compare the performance of DDR with other prevailing similarity metrics and demonstrate that DDR consistently provides finer discrimination between semantically similar and dissimilar texts, even under minimal, controlled edits.
Read the full abstract ▾
Computation and Language
Qingsen Ma, Dianyun Wang, Yaoye Wang et al. (+7 more)
Large
language models are increasingly applied to multi-document and long-form inputs, yet long-context inference remains
memory- and noise-inefficient. Key-value (KV) caching scales linearly with context length, while external
retrieval methods often return lexically similar but causally irrelevant passages. We present S3-
Attention, a
memory-first inference-time framework that treats long-context processing as
attention-aligned endogenous
retrieval. S3-
Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders, and constructs a CPU-based inverted index mapping features to
token positions or spans during a single streaming scan. This design allows the KV cache to be discarded entirely and bounds GPU
memory usage by the scan chunk size. At generation time, feature co-activation is used to retrieve compact evidence spans, optionally fused with BM25 for exact lexical matching. Under a unified LongBench evaluation protocol with fixed prompting, decoding, and matched
token budgets, S3-Hybrid closely matches full-context inference across multiple model families and improves robustness in several information-dense settings. We also report an engineering limitation of the current prototype, which incurs higher wall-clock latency than optimized full-KV baselines, motivating future kernel-level optimization.
Read the full abstract ▾
All authors Qingsen Ma, Dianyun Wang, Yaoye Wang, Lechen Ning, Sujie Zhu, Xiaohang Zhang, Jiaming Lyu, Linhao Ren, Zhenbo Xu, Zhaofeng He
Computation and Language · Cross-listed from cs.AI
SQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL
Harper Hua, Zhen Han, Zhengyuan Shen et al. (+9 more)
While large
language models (LLMs) have substantially improved Text-to-SQL generation, a pronounced gap remains between AI systems and human experts on challenging
benchmarks such as BIRD-SQL. We argue this gap stems largely from the prevailing single-pass paradigm, which lacks the iterative
reasoning, schema exploration, and error-correction behaviors that humans naturally employ. To address this limitation, we introduce SQL-Trail, a multi-turn reinforcement learning (RL) agentic framework for Text-to-SQL. Rather than producing a query in one shot, SQL-Trail interacts with the database environment and uses execution feedback to iteratively refine its predictions. Our approach centers on two key ideas: (i) an adaptive turn-budget allocation mechanism that scales the
agent's interaction depth to match question difficulty, and (ii) a composite reward panel that jointly incentivizes SQL correctness and efficient exploration. Across
benchmarks, SQL-Trail sets a new state of the art and delivers strong data efficiency–up to 18x higher than prior single-pass RL state-of-the-art methods. Notably, our 7B and 14B models outperform substantially larger proprietary systems by 5% on average, underscoring the effectiveness of interactive, agentic workflows for robust Text-to-SQL generation.
Read the full abstract ▾
All authors Harper Hua, Zhen Han, Zhengyuan Shen, Jeremy Lee, Patrick Guan, Qi Zhu, Sullam Jeoung, Yueyan Chen, Yunfei Bai, Shuai Wang, Vassilis Ioannidis, Huzefa Rangwala