Computation and Language
Predicate-Argument Structure Divergences in Chinese and English Parallel Sentences and their Impact on Language Transfer
Rocco Tripodi and Xiaoyu Liu
Cross-lingual Natural
Language Processing (NLP) has gained significant traction in recent years, offering practical solutions in low-resource settings by transferring linguistic knowledge from resource-rich to low-resource
languages. This field leverages techniques like annotation projection and model transfer for
language adaptation, supported by
multilingual pre-trained
language models. However, linguistic divergences hinder
language transfer, especially among typologically distant
languages. In this paper, we present an analysis of predicate-argument structures in parallel Chinese and English sentences. We explore the
alignment and misalignment of predicate annotations, inspecting similarities and differences and proposing a categorization of structural divergences. The analysis and the categorization are supported by a qualitative and quantitative analysis of the results of an annotation projection experiment, in which, in turn, one of the two
languages has been used as source
language to project annotations into the corresponding parallel sentences. The results of this analysis show clearly that
language transfer is asymmetric. An aspect that requires
attention when it comes to selecting the source
language in transfer learning applications and that needs to be investigated before any scientific claim about cross-lingual NLP is proposed.
Read the full abstract ▾
Computation and Language
How Small Can You Go? Compact Language Models for On-Device Critical Error Detection in Machine Translation
Muskaan Chopra, Lorenz Sparrenberg, Sarthak Khanna et al. (+1 more)
Large
Language Models (LLMs) excel at evaluating machine translation (MT), but their scale and cost hinder deployment on edge devices and in
privacy-sensitive workflows. We ask: how small can you get while still detecting meaning-altering translation errors? Focusing on English->German Critical Error Detection (CED), we
benchmark sub-2B models (LFM2-350M, Qwen-3-0.6B/1.7B, Llama-3.2-1B-Instruct, Gemma-3-1B) across WMT21, WMT22, and SynCED-EnDe-2025. Our framework standardizes
prompts, applies lightweight logit-bias calibration and majority voting, and reports both semantic quality (MCC, F1-ERR/F1-NOT) and compute metrics (VRAM, latency, throughput). Results reveal a clear sweet spot around one billion parameters: Gemma-3-1B provides the best quality-efficiency trade-off, reaching MCC=0.77 with F1-ERR=0.98 on SynCED-EnDe-2025 after merged-weights fine-tuning, while maintaining 400 ms single-sample latency on a MacBook Pro M4 Pro (24 GB). At larger scale, Qwen-3-1.7B attains the highest absolute MCC (+0.11 over Gemma) but with higher compute cost. In contrast, ultra-small models (0.6B) remain usable with few-shot calibration yet under-detect entity and number errors. Overall, compact, instruction-tuned LLMs augmented with lightweight calibration and small-sample supervision can deliver trustworthy, on-device CED for MT, enabling private, low-cost error screening in real-world translation pipelines. All
datasets,
prompts, and scripts are publicly available at our GitHub repository.
Read the full abstract ▾
All authors Muskaan Chopra, Lorenz Sparrenberg, Sarthak Khanna, Rafet Sifa
Computation and Language
Assessing the Applicability of Natural Language Processing to Traditional Social Science Methodology: A Case Study in Identifying Strategic Signaling Patterns in Presidential Directives
C. LeMay, A. Lane, J. Seales et al. (+2 more)
Our research investigates how Natural
Language Processing (NLP) can be used to extract main topics from a larger corpus of written data, as applied to the case of identifying signaling themes in Presidential Directives (PDs) from the Reagan through Clinton administrations. Analysts and NLP both identified relevant documents, demonstrating the potential utility of NLPs in research involving large written corpuses. However, we also identified discrepancies between NLP and human-labeled results that indicate a need for more research to assess the validity of NLP in this use case. The research was conducted in 2023, and the rapidly evolving landscape of AIML means existing tools have improved and new tools have been developed; this research displays the inherent capabilities of a potentially dated AI tool in emerging social science applications.
Read the full abstract ▾
All authors C. LeMay, A. Lane, J. Seales, M. Winstead, S. Baty
Computation and Language
Contextual morphologically-guided tokenization for Latin encoder models
Marisa Hudspeth, Patrick J. Burns and Brendan O'Connor
Tokenization is a critical component of
language model pretraining, yet standard tokenization methods often prioritize information-theoretical goals like high compression and low fertility rather than linguistic goals like morphological
alignment. In fact, they have been shown to be suboptimal for morphologically rich
languages, where tokenization quality directly impacts downstream performance. In this work, we investigate morphologically-aware tokenization for Latin, a morphologically rich
language that is medium-resource in terms of pretraining data, but high-resource in terms of curated lexical resources – a distinction that is often overlooked but critical in discussions of low-resource
language modeling. We find that morphologically-guided tokenization improves overall performance on four downstream tasks. Performance gains are most pronounced for out of domain texts, highlighting our models' improved generalization ability. Our findings demonstrate the utility of linguistic resources to improve
language modeling for morphologically complex
languages. For low-resource
languages that lack large-scale pretraining data, the development and incorporation of linguistic resources can serve as a feasible alternative to improve LM performance.
Read the full abstract ▾
Computation and Language
Order Matters: Rethinking Prompt Construction in In-Context Learning
Warren Li, Yiqian Wang, Zihan Wang et al. (+1 more)
In-context learning (ICL) enables large
language models to perform new tasks by conditioning on a sequence of examples. Most prior work reasonably and intuitively assumes that which examples are chosen has a far greater effect on performance than how those examples are ordered, leading to a focus on example selection. We revisit this assumption and conduct a systematic comparison between the effect of selection and ordering. Through controlled experiments on both classification and generation tasks, using multiple open-source model families (0.5B to 27B parameters) and GPT-5, we find that the variance in performance due to different example orderings is comparable to that from using entirely different example sets. Furthermore, we show that strong orderings can be identified using only a development set, achieving performance close to an oracle that selects the best ordering based on test labels. Our findings highlight the equal and intertwined importance of example selection and ordering in
prompt design, calling for a reexamination of the assumptions held in ICL.
Read the full abstract ▾
All authors Warren Li, Yiqian Wang, Zihan Wang, Jingbo Shang
Computation and Language
Omnilingual ASR team: Gil Keren, Artyom Kozhevnikov, Yen Meng et al. (+29 more)
Automatic
speech recognition (ASR) has advanced in high-resource
languages, but most of the world's 7,000+
languages remain unsupported, leaving thousands of long-tail
languages behind. Expanding ASR coverage has been costly and limited by architectures that restrict
language support, making extension inaccessible to most–all while entangled with ethical concerns when pursued without community collaboration. To transcend these limitations, we introduce Omnilingual ASR, the first large-scale ASR system designed for extensibility. Omnilingual ASR enables communities to introduce unserved
languages with only a handful of data samples. It scales self-supervised pre-training to 7B parameters to learn robust
speech representations and introduces an encoder-decoder architecture designed for zero-shot generalization, leveraging a LLM-inspired decoder. This capability is grounded in a massive and diverse training corpus; by combining breadth of coverage with linguistic variety, the model learns representations robust enough to adapt to unseen
languages. Incorporating public resources with community-sourced recordings gathered through compensated local partnerships, Omnilingual ASR expands coverage to over 1,600
languages, the largest such effort to date–including over 500 never before served by ASR. Automatic evaluations show substantial gains over prior systems, especially in low-resource conditions, and strong generalization. We release Omnilingual ASR as a family of models, from 300M variants for low-power devices to 7B for maximum accuracy. We reflect on the ethical considerations shaping this design and conclude by discussing its societal impact. In particular, we highlight how open-sourcing models and tools can lower barriers for researchers and communities, inviting new forms of participation. Open-source artifacts are available at https://github.com/facebookresearch/omnilingual-asr.
Read the full abstract ▾
All authors Omnilingual ASR team: Gil Keren, Artyom Kozhevnikov, Yen Meng, Christophe Ropers, Matthew Setzler, Skyler Wang, Ife Adebara, Michael Auli, Can Balioglu, Kevin Chan, Chierh Cheng, Joe Chuang, Caley Droof, Mark Duppenthaler, Paul-Ambroise Duquenne, Alexander Erben, Cynthia Gao, Gabriel Mejia Gonzalez, Kehan Lyu, Sagar Miglani, Vineel Pratap, Kaushik Ram Sadagopan, Safiyyah Saleem, Arina Turkatenko, Albert Ventayol-Boada, Zheng-Xin Yong, Yu-An Chung, Jean Maillard, Rashel Moritz, Alexandre Mourachko, Mary Williamson, Shireen Yates
Computation and Language
SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification
Mohamed Elaraby and Jyoti Prakash Maheswari
Large
Language Models (LLMs) with extended context windows promise direct
reasoning over long documents, reducing the need for chunking or
retrieval. Constructing annotated resources for training and evaluation, however, remains costly. Synthetic data offers a scalable alternative, and we introduce SynClaimEval, a framework for evaluating synthetic data utility in long-context claim verification – a task central to
hallucination detection and fact-checking. Our framework examines three dimensions: (i) input characteristics, by varying context length and testing generalization to out-of-domain
benchmarks; (ii) synthesis logic, by controlling claim complexity and error type variation; and (iii) explanation quality, measuring the degree to which model explanations provide evidence consistent with predictions. Experiments across
benchmarks show that long-context synthesis can improve verification in base instruction-tuned models, particularly when augmenting existing human-written
datasets. Moreover, synthesis enhances explanation quality, even when verification scores do not improve, underscoring its potential to strengthen both performance and explainability.
Read the full abstract ▾
Computation and Language
Readability Measures and Automatic Text Simplification: In the Search of a Construct
Rémi Cardon and A. Seza Doğruöz
Readability is a key concept in the current era of abundant written information. To help making texts more readable and make information more accessible to everyone, a line of researched aims at making texts accessible for their target audience: automatic text simplification (ATS). Lately, there have been studies on the correlations between automatic evaluation metrics in ATS and human judgment. However, the correlations between those two aspects and commonly available readability measures (such as readability formulas or linguistic features) have not been the focus of as much
attention. In this work, we investigate the place of readability measures in ATS by complementing the existing studies on evaluation metrics and human judgment, on English. We first discuss the relationship between ATS and research in readability, then we report a study on correlations between readability measures and human judgment, and between readability measures and ATS evaluation metrics. We identify that in general, readability measures do not correlate well with automatic metrics and human judgment. We argue that as the three different angles from which simplification can be assessed tend to exhibit rather low correlations with one another, there is a need for a clear definition of the construct in ATS.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting
Renda Li, Hailang Huang, Fei Wei et al. (+3 more)
Reinforcement learning (RL) has demonstrated considerable potential for enhancing
reasoning in large
language models (LLMs). However, existing methods suffer from Gradient Starvation and Policy Degradation when training directly on samples with mixed difficulty. To mitigate this, prior approaches leverage Chain-of-Thought (CoT) data, but the construction of high-quality CoT annotations remains labor-intensive. Alternatively, curriculum learning strategies have been explored but frequently encounter challenges, such as difficulty mismatch, reliance on manual curriculum design, and catastrophic forgetting. To address these issues, we propose AdaCuRL, a Adaptive Curriculum Reinforcement Learning framework that integrates coarse-to-fine difficulty estimation with adaptive curriculum scheduling. This approach dynamically aligns data difficulty with model capability and incorporates a data revisitation mechanism to mitigate catastrophic forgetting. Furthermore, AdaCuRL employs adaptive reference and sparse KL strategies to prevent Policy Degradation. Extensive experiments across diverse
reasoning benchmarks demonstrate that AdaCuRL consistently achieves significant performance improvements on both LLMs and MLLMs.
Read the full abstract ▾
All authors Renda Li, Hailang Huang, Fei Wei, Feng Xiong, Yong Wang, Xiangxiang Chu
Computation and Language
BIG5-TPoT: Predicting BIG Five Personality Traits, Facets, and Items Through Targeted Preselection of Texts
Triet M. Le, Arjun Chandra, C. Anton Rytting et al. (+3 more)
Predicting an individual's personalities from their generated texts is a challenging task, especially when the text volume is large. In this paper, we introduce a straightforward yet effective novel strategy called targeted preselection of texts (TPoT). This method semantically filters the texts as input to a deep learning model, specifically designed to predict a Big Five personality trait, facet, or item, referred to as the BIG5-TPoT model. By selecting texts that are semantically relevant to a particular trait, facet, or item, this strategy not only addresses the issue of input text limits in large
language models but also improves the Mean Absolute Error and accuracy metrics in predictions for the Stream of Consciousness Essays
dataset.
Read the full abstract ▾
All authors Triet M. Le, Arjun Chandra, C. Anton Rytting, Valerie P. Karuzis, Vladimir Rife, William A. Simpson
Computation and Language
GSAP-ERE: Fine-Grained Scholarly Entity and Relation Extraction Focused on Machine Learning
Wolfgang Otto, Lu Gan, Sharmila Upadhyaya et al. (+2 more)
Research in Machine Learning (ML) and AI evolves rapidly. Information Extraction (IE) from scientific publications enables to identify information about research concepts and resources on a large scale and therefore is a pathway to improve understanding and reproducibility of ML-related research. To extract and connect fine-grained information in ML-related research, e.g. method training and data usage, we introduce GSAP-ERE. It is a manually curated fine-grained
dataset with 10 entity types and 18 semantically categorized relation types, containing mentions of 63K entities and 35K relations from the full text of 100 ML publications. We show that our
dataset enables fine-tuned models to automatically extract information relevant for downstream tasks ranging from knowledge graph (KG) construction, to monitoring the computational reproducibility of AI research at scale. Additionally, we use our
dataset as a test suite to explore prompting strategies for IE using Large
Language Models (LLM). We observe that the performance of state-of-the-art LLM prompting methods is largely outperformed by our best fine-tuned baseline model (NER: 80.6%, RE: 54.0% for the fine-tuned model vs. NER: 44.4%, RE: 10.1% for the LLM). This disparity of performance between supervised models and unsupervised usage of LLMs suggests
datasets like GSAP-ERE are needed to advance research in the domain of scholarly information extraction.
Read the full abstract ▾
All authors Wolfgang Otto, Lu Gan, Sharmila Upadhyaya, Saurav Karmakar, Stefan Dietze
Computation and Language
CARE-Bench: A Benchmark of Diverse Client Simulations Guided by Expert Principles for Evaluating LLMs in Psychological Counseling
Bichen Wang, Yixin Sun, Junzhe Wang et al. (+6 more)
The mismatch between the growing demand for psychological counseling and the limited availability of services has motivated research into the application of Large
Language Models (LLMs) in this domain. Consequently, there is a need for a robust and unified
benchmark to assess the counseling competence of various LLMs. Existing works, however, are limited by unprofessional client simulation, static question-and-answer evaluation formats, and unidimensional metrics. These limitations hinder their effectiveness in assessing a model's comprehensive ability to handle diverse and complex clients. To address this gap, we introduce CARE-Bench, a dynamic and interactive automated
benchmark. It is built upon diverse client profiles derived from real-world counseling cases and simulated according to expert guidelines. CARE-Bench provides a multidimensional performance evaluation grounded in established psychological scales. Using CARE-Bench, we evaluate several general-purpose LLMs and specialized counseling models, revealing their current limitations. In collaboration with psychologists, we conduct a detailed analysis of the reasons for LLMs' failures when interacting with clients of different types, which provides directions for developing more comprehensive, universal, and effective counseling models.
Read the full abstract ▾
All authors Bichen Wang, Yixin Sun, Junzhe Wang, Hao Yang, Xing Fu, Yanyan Zhao, Si Wei, Shijin Wang, Bing Qin
Computation and Language
Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque
Lukas Arana, Julen Etxaniz, Ander Salaberria et al. (+1 more)
Current
Multimodal Large
Language Models exhibit very strong performance for several demanding tasks. While commercial MLLMs deliver acceptable performance in low-resource
languages, comparable results remain unattained within the open science community. In this paper, we aim to develop a strong MLLM for a low-resource
language, namely Basque. For that purpose, we develop our own training and evaluation image-text
datasets. Using two different Large
Language Models as backbones, the Llama-3.1-Instruct model and a Basque-adapted variant called Latxa, we explore several data mixtures for training. We show that: i) low ratios of Basque
multimodal data (around 20%) are already enough to obtain solid results on Basque
benchmarks, and ii) contrary to expected, a Basque instructed backbone LLM is not required to obtain a strong MLLM in Basque. Our results pave the way to develop MLLMs for other low-resource
languages by openly releasing our resources.
Read the full abstract ▾
All authors Lukas Arana, Julen Etxaniz, Ander Salaberria, Gorka Azkune
Computation and Language
AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment
Ruibo Deng, Duanyu Feng and Wenqiang Lei
Offline preference optimization offers a simpler and more stable alternative to RLHF for aligning
language models. However, their effectiveness is critically dependent on ranking accuracy, a metric where further gains are highly impactful. This limitation arises from a fundamental problem that we identify and formalize as the Overfitting-Underfitting Dilemma: current margin designs cause models to apply excessive, wasteful gradients to correctly ranked samples (overfitting) while providing insufficient corrective signals for misranked ones (underfitting). To resolve this dilemma, we propose Adaptive Margin-attached Preference Optimization (AMaPO), a simple yet principled
algorithm. AMaPO employs an instance-wise adaptive margin, refined by Z-normalization and exponential scaling, which dynamically reallocates learning effort by amplifying gradients for misranked samples and suppressing them for correct ones. Extensive experiments on widely used
benchmarks demonstrate that AMaPO not only achieves better ranking accuracy and superior downstream
alignment performance, but targeted analysis also confirms that it successfully mitigates the core overfitting and underfitting issues.
Read the full abstract ▾
Computation and Language
Self-Correcting Large Language Models: Generation vs. Multiple Choice
Hossein A. Rahmani, Satyapriya Krishna, Xi Wang et al. (+2 more)
Large
language models have recently demonstrated remarkable abilities to self-correct their responses through iterative refinement, often referred to as self-consistency or self-reflection. However, the dynamics of this self-correction mechanism may differ substantially depending on whether the model is tasked with open-ended text generation or with selecting the most appropriate response from multiple predefined options. In this paper, we conduct a systematic investigation of these two paradigms by comparing performance trends and error-correction behaviors across various natural
language understanding and
reasoning tasks, covering
language models of different scales and families. Our experimental results reveal distinct patterns of improvement and failure modes: While open-ended generation often benefits from the flexibility of re-interpretation and compositional refinement, multiple-choice selection can leverage clearer solution boundaries but may be limited by the provided options. This contrast also reflects the dual demands faced by emerging agentic LLM applications: effective
agents must not only generate and refine open-ended plans or explanations, but also make reliable discrete choices when operating within constrained action spaces. Our findings, therefore, highlight that the design of self-correction mechanisms should take into account the interaction between task structure and output space, with implications for both knowledge-intensive
reasoning and decision-oriented applications of LLMs.
Read the full abstract ▾
All authors Hossein A. Rahmani, Satyapriya Krishna, Xi Wang, Mohammadmehdi Naghiaei, Emine Yilmaz
Computation and Language
MTQ-Eval: Multilingual Text Quality Evaluation for Language Models
Rhitabrat Pokharel and Ameeta Agrawal
The use of large
language models (LLMs) for evaluating outputs is becoming an increasingly effective and scalable approach. However, it remains uncertain whether this capability extends beyond task-specific evaluations to more general assessments of text quality, particularly in
multilingual contexts. In this study, we introduce, MTQ-Eval, a novel framework for
multilingual text quality evaluation that learns from examples of both high- and low-quality texts, adjusting its internal representations. To develop MTQ-Eval, we first automatically generate text quality preference data and then use it to train open-source base LLMs to align with ratings of high- and low-quality text. Our comprehensive evaluation across 115
languages demonstrates the improved performance of the proposed model. Upon further analysis, we find that this enhanced evaluation capability also leads to notable improvements in downstream tasks.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SE
Routesplain: Towards Faithful and Intervenable Routing for Software-related Tasks
Adam Štorek, Vikas Upadhyay, Marianne Menglin Liu et al. (+6 more)
LLMs now tackle a wide range of software-related tasks, yet we show that their performance varies markedly both across and within these tasks. Routing user queries to the appropriate LLMs can therefore help improve response quality while reducing cost. Prior work, however, has focused mainly on general-purpose LLM routing via black-box models. We introduce Routesplain, the first LLM router for software-related tasks, including
multilingual code generation and repair, input/output prediction, and computer science QA. Unlike existing routing approaches, Routesplain first extracts human-interpretable concepts from each query (e.g., task, domain,
reasoning complexity) and only routes based on these concepts, thereby providing intelligible, faithful rationales. We evaluate Routesplain on 16 state-of-the-art LLMs across eight software-related tasks; Routesplain outperforms individual models both in terms of accuracy and cost, and equals or surpasses all black-box baselines, with concept-level intervention highlighting avenues for further router improvements.
Read the full abstract ▾
All authors Adam Štorek, Vikas Upadhyay, Marianne Menglin Liu, Daniel W. Peterson, Anshul Mittal, Sujeeth Bharadwaj, Fahad Shah, Sujith Ravi, Dan Roth
Computation and Language
Spider4SSC & S2CLite: A text-to-multi-query-language dataset using lightweight ontology-agnostic SPARQL to Cypher parser
Martin Vejvar and Yasutaka Fujimoto
We present Spider4SSC
dataset and S2CLite parsing tool. S2CLite is a lightweight, ontology-agnostic parser that translates SPARQL queries into Cypher queries, enabling both in-situ and large-scale SPARQL to Cypher translation. Unlike existing solutions, S2CLite is purely rule-based (inspired by traditional programming
language compilers) and operates without requiring an RDF graph or external tools. Experiments conducted on the BSBM42 and Spider4SPARQL
datasets show that S2CLite significantly reduces query parsing errors, achieving a total parsing accuracy of 77.8% on Spider4SPARQL compared to 44.2% by the state-of-the-art S2CTrans. Furthermore, S2CLite achieved a 96.6% execution accuracy on the intersecting subset of queries parsed by both parsers, outperforming S2CTrans by 7.3%. We further use S2CLite to parse Spider4SPARQL queries to Cypher and generate Spider4SSC, a unified Text-to-Query
language (SQL, SPARQL, Cypher)
dataset with 4525 unique questions and 3 equivalent sets of 2581 matching queries (SQL, SPARQL and Cypher). We open-source S2CLite for further development on GitHub (github.com/vejvarm/S2CLite) and provide the clean Spider4SSC
dataset for download.
Read the full abstract ▾
Computation and Language
Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling
Shiyu Ji, Yixuan Wang, Yijun Liu et al. (+2 more)
Test-time scaling improves the inference performance of Large
Language Models (LLMs) but also incurs substantial computational costs. Although recent studies have reduced
token consumption through dynamic self-consistency, they remain constrained by the high latency of sequential requests. In this paper, we propose SeerSC, a dynamic self-consistency framework that simultaneously improves
token efficiency and latency by integrating System 1 and System 2
reasoning. Specifically, we utilize the rapid System 1 to compute the answer
entropy for given queries. This score is then used to evaluate the potential of samples for scaling, enabling dynamic self-consistency under System 2. Benefiting from the advance and accurate estimation provided by System 1, the proposed method can reduce
token usage while simultaneously achieving a significant decrease in latency through parallel generation. It outperforms existing methods, achieving up to a 47% reduction in
token consumption and a 43% reduction in inference latency without significant performance loss.
Read the full abstract ▾
All authors Shiyu Ji, Yixuan Wang, Yijun Liu, Qingfu Zhu, Wanxiang Che
Computation and Language
Arka Mukherjee and Shreya Ghosh
Contemporary
vision-
language models (VLMs) perform well on existing
multimodal reasoning benchmarks (78-85% accuracy on MMMU, MathVista). Yet, these results fail to sufficiently distinguish true scientific
reasoning articulation capabilities from pattern-matching. To address this gap, we introduce mmJEE-Eval, a
multimodal bilingual (English and Hindi)
benchmark comprising 1,460 questions from India's JEE Advanced examination (2019-2025) spanning pre-college Physics, Chemistry, and Mathematics domains. Our evaluation of 17 state-of-the-art models reveals that while frontier VLMs (GPT-5, Gemini 2.5 Pro/Flash) achieve 77-84% accuracy on held-out 2025 questions, open-source models plateau at 37-45% despite scaling to 400B parameters, a significant difference not observed on existing
benchmarks. While closed frontiers from Google and OpenAI show high problem-solving accuracies (up to 100% pass@3 scores), they fully collapse when the
reasoning load is increased meta-cognitively (GPT-5 fixes just 5.2% errors). Systematic ablations show mmJEE-Eval's difficulty stems from complexity and
reasoning depth rather than memorization. Effectively, our
benchmark segregates superior training and
reasoning methodologies where alternatives fail. We publicly release our code and data: https://mmjee-eval.github.io
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Not Everything That Counts Can Be Counted: A Case for Safe Qualitative AI
Stine Beltoft and Lukas Galke
Artificial intelligence (AI) and large
language models (LLM) are reshaping science, with most recent advances culminating in fully-automated scientific discovery pipelines. But qualitative research has been left behind. Researchers in qualitative methods are hesitant about AI adoption. Yet when they are willing to use AI at all, they have little choice but to rely on general-purpose tools like ChatGPT to assist with interview interpretation, data annotation, and topic modeling - while simultaneously acknowledging these system's well-known limitations of being biased, opaque, irreproducible, and
privacy-compromising. This creates a critical gap: while AI has substantially advanced quantitative methods, the qualitative dimensions essential for meaning-making and comprehensive scientific understanding remain poorly integrated. We argue for developing dedicated qualitative AI systems built from the ground up for interpretive research. Such systems must be transparent, reproducible, and
privacy-friendly. We review recent literature to show how existing automated discovery pipelines could be enhanced by robust qualitative capabilities, and identify key opportunities where safe qualitative AI could advance multidisciplinary and mixed-methods research.
Read the full abstract ▾
Computation and Language
AdaptDel: Adaptable Deletion Rate Randomized Smoothing for Certified Robustness
Zhuoqun Huang, Neil G. Marchant, Olga Ohrimenko et al. (+1 more)
We consider the problem of certified robustness for sequence classification against edit distance perturbations. Naturally occurring inputs of varying lengths (e.g., sentences in natural
language processing tasks) present a challenge to current methods that employ fixed-rate deletion mechanisms and lead to suboptimal performance. To this end, we introduce AdaptDel methods with adaptable deletion rates that dynamically adjust based on input properties. We extend the theoretical framework of randomized smoothing to variable-rate deletion, ensuring sound certification with respect to edit distance. We achieve strong empirical results in natural
language tasks, observing up to 30 orders of magnitude improvement to median cardinality of the certified region, over state-of-the-art certifications.
Read the full abstract ▾
All authors Zhuoqun Huang, Neil G. Marchant, Olga Ohrimenko, Benjamin I. P. Rubinstein
Computation and Language
Towards Explainable Khmer Polarity Classification
Marry Kong, Rina Buoy, Sovisal Chenda et al. (+1 more)
Khmer polarity classification is a fundamental natural
language processing task that assigns a positive, negative, or neutral label to a given Khmer text input. Existing Khmer models typically predict the label without explaining the rationale behind the prediction. This paper proposes an explainable Khmer polarity classifier by fine-tuning an instruction-based
reasoning Qwen-3 model. The notion of explainability in this paper is limited to self-explanations, which the model uses to rationalize its predictions. Experimental results show that the fine-tuned model not only predicts labels accurately but also provides
reasoning by identifying polarity-related keywords or phrases to support its predictions. In addition, we contribute a new Khmer polarity
dataset consisting of short- to medium-length casual, romanized, and mixed-code Khmer expressions. This
dataset was constructed using both heuristic rules and human curation and is publicly available through a gated Hugging Face repository (rinabuoy/khmerpolarity_nonreasoning). The fine-tuned Qwen-3 models are also made available in the same Hugging Face account.
Read the full abstract ▾
All authors Marry Kong, Rina Buoy, Sovisal Chenda, Nguonly Taing
Computation and Language
LiteraryTaste: A Preference Dataset for Creative Writing Personalization
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele et al. (+7 more)
People have different creative writing preferences, and large
language models (LLMs) for these tasks can benefit from adapting to each user's preferences. However, these models are often trained over a
dataset that considers varying personal tastes as a monolith. To facilitate developing personalized creative writing LLMs, we introduce LiteraryTaste, a
dataset of reading preferences from 60 people, where each person: 1) self-reported their reading habits and tastes (stated preference), and 2) annotated their preferences over 100 pairs of short creative writing texts (revealed preference). With our
dataset, we found that: 1) people diverge on creative writing preferences, 2) finetuning a
transformer encoder could achieve 75.8% and 67.7% accuracy when modeling personal and collective revealed preferences, and 3) stated preferences had limited utility in modeling revealed preferences. With an LLM-driven interpretability pipeline, we analyzed how people's preferences vary. We hope our work serves as a cornerstone for personalizing creative writing technologies.
Read the full abstract ▾
All authors John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yi Wang, Yuqian Sun, Tiffany Wang, Shm Garanganao Almeda, Brett A. Halperin, Yuwen Lu, Max Kreminski
Computation and Language
C$^3$TG: Conflict-aware, Composite, and Collaborative Controlled Text Generation
Yu Li, Zhe Yang, Yi Huang et al. (+2 more)
Recent advancements in large
language models (LLMs) have demonstrated remarkable text generation capabilities. However, controlling specific attributes of generated text remains challenging without architectural modifications or extensive fine-tuning. Current methods typically toggle a single, basic attribute but struggle with precise multi-attribute control. In scenarios where attribute requirements conflict, existing methods lack coordination mechanisms, causing interference between desired attributes. Furthermore, these methods fail to incorporate iterative optimization processes in the controlled generation pipeline. To address these limitations, we propose Conflict-aware, Composite, and Collaborative Controlled Text Generation (C$^3$TG), a two-phase framework for fine-grained, multi-dimensional text attribute control. During generation, C$^3$TG selectively pairs the LLM with the required attribute classifiers from the 17 available dimensions and employs weighted KL-divergence to adjust
token probabilities. The optimization phase then leverages an energy function combining classifier scores and penalty terms to resolve attribute conflicts through iterative feedback, enabling precise control over multiple dimensions simultaneously while preserving natural text flow. Experiments show that C$^3$TG significantly outperforms baselines across multiple metrics including attribute accuracy, linguistic fluency, and output diversity, while simultaneously reducing toxicity. These results establish C$^3$TG as an effective and flexible solution for multi-dimensional text attribute control that requires no costly model modifications.
Read the full abstract ▾
All authors Yu Li, Zhe Yang, Yi Huang, Xin Liu, Guilin Qi
Computation and Language · Cross-listed from cs.SD
End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering
Jiliang Hu, Zuchao Li, Baoyuan Qi et al. (+2 more)
Significant progress has been made in spoken question answering (SQA) in recent years. However, many existing methods, including large audio
language models, struggle with processing long audio. Follow the success of
retrieval augmented generation, a
speech-related retriever shows promising in help preprocessing long-form
speech. But the performance of existing
speech-related retrievers is lacking. To address this challenge, we propose CLSR, an end-to-end contrastive
language-
speech retriever that efficiently extracts question-relevant segments from long audio recordings for downstream SQA task. Unlike conventional
speech-text contrastive models, CLSR incorporates an intermediate step that converts acoustic features into text-like representations prior to
alignment, thereby more effectively bridging the gap between modalities. Experimental results across four cross-modal
retrieval datasets demonstrate that CLSR surpasses both end-to-end
speech related retrievers and pipeline approaches combining
speech recognition with text
retrieval, providing a robust foundation for advancing practical long-form SQA applications.
Read the full abstract ▾
All authors Jiliang Hu, Zuchao Li, Baoyuan Qi, Liu Guoming, Ping Wang
Computation and Language
POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translation
Xuanchen Li, Chenrui Cui, Tianrui Wang et al. (+9 more)
Speech Large
Language Models have achieved breakthroughs in
multilingual speech-to-text translation. However, existing approaches often overlook semantic commonalities across source
languages, leading to biased translation performance. In this work, we propose POTSA (Parallel Optimal Transport for
Speech Alignment), a new framework based on cross-lingual parallel
speech pairs and Optimal Transport, designed to bridge high- and low-resource translation gaps. First, we introduce a Bias Compensation module to coarsely align initial
speech representations. Second, we impose
token-level OT constraints on a Q-Former using parallel pairs to establish fine-grained representation consistency. Then, we apply a layer scheduling strategy to focus OT constraints on semantically beneficial layers. Experiments on FLEURS show our method achieves SOTA performance, with +1.29 BLEU over five common
languages and +2.93 BLEU on zero-shot
languages, using only 10 hours of parallel
speech per
language.
Read the full abstract ▾
All authors Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Yizhou Peng, Jin Li, Yuheng Lu, Yu Jiang, Nyima Tashi, Longbiao Wang, Jianwu Dang
Computation and Language · Cross-listed from cs.CV
Taming Object Hallucinations with Verified Atomic Confidence Estimation
Jiarui Liu, Weihao Xuan, Zhijing Jin et al. (+1 more)
Multimodal Large
Language Models (MLLMs) often suffer from
hallucinations, particularly errors in object existence, attributes, or relations, which undermine their reliability. We introduce TACO (Verified Atomic Confidence Estimation), a simple framework that mitigates
hallucinations through self-verification and confidence calibration without relying on external
vision experts. TACO decomposes responses into atomic queries, paraphrases them to reduce sensitivity to wording, and estimates confidence using self-consistency (black-box) or self-confidence (gray-box) aggregation, before refining answers with a
language model. Experiments on five
benchmarks (POPE, MME, HallusionBench, AMBER, and MM-Hal Bench) with two MLLMs (LLaVA-1.5-7B and CogVLM2) show that TACO consistently outperforms direct prompting and Visual Contrastive Decoding, reduces systematic biases, and improves confidence calibration, demonstrating its effectiveness in enhancing the faithfulness of MLLMs.
Read the full abstract ▾
All authors Jiarui Liu, Weihao Xuan, Zhijing Jin, Mona Diab
Computation and Language
Toward Honest Language Models for Deductive Reasoning
Jiarui Liu, Kaustubh Dhole, Yingheng Wang et al. (+7 more)
Deductive
reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current
language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive
reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two
datasets from graph structures, one for linear
algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ACNCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall
reasoning performance, underscoring the importance of training dynamics for enabling honest deductive
reasoning in
language models.
Read the full abstract ▾
All authors Jiarui Liu, Kaustubh Dhole, Yingheng Wang, Haoyang Wen, Sarah Zhang, Haitao Mao, Gaotang Li, Neeraj Varshney, Jingguo Liu, Xiaoman Pan
Computation and Language
Pretraining Finnish ModernBERTs
Akseli Reunamo, Laura-Maria Peltonen, Hans Moen et al. (+1 more)
This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing
languages relevant to Finland. Our models are competitive with, or superior to, existing
multilingual models. They outperform monolingual models on tasks that require a context longer than 512
tokens. We present empirical results on using different data in the final stage of training. The code and models are publicly released.
Read the full abstract ▾
All authors Akseli Reunamo, Laura-Maria Peltonen, Hans Moen, Sampo Pyysalo