Computation and Language
Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yorùbá
Opeyemi Osakuade and Simon King
Discrete
speech units (DSUs) are derived by quantising representations from models trained using self-supervised learning (SSL). They are a popular representation for a wide variety of spoken
language tasks, including those where prosody matters. DSUs are especially convenient for tasks where text and
speech are jointly modelled, such as text-to-
speech and
multimodal dialogue systems. But we have found that DSUs encode suprasegmental information less reliably than segmental structure, which we demonstrate in this work using lexical tone, though this limitation likely extends to other suprasegmental features such as prosody. Our investigations using the tone
languages Mandarin and Yorùbá show that the SSL latent representations themselves do encode tone, yet DSUs obtained using quantisation tend to prioritise phonetic structure, which makes lexical tone less reliably encoded. This remains true for a variety of quantisation methods, not only the most common, K-means. We conclude that current DSU quantisation strategies have limitations for suprasegmental features, which suggests a need for new, tone-aware (or prosody-aware) techniques in
speech representation learning. We point towards a potential form of the solution by performing K-means clustering once to encode phonetic information, then again on the residual representation, which better encodes lexical tone.
Read the full abstract ▾
Computation and Language
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba et al. (+2 more)
Cross-tokenizer
distillation (CTD), the transfer of knowledge from a teacher to a student
language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level
Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with–and on several
benchmarks surpasses–significantly more sophisticated CTD methods, across a range of
distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and
benchmarks remain elusive, underscoring that CTD is still an open problem.
Read the full abstract ▾
All authors Avyav Kumar Singh, Yen-Chen Wu, Alexandru Cioba, Alberto Bernacchia, Davide Buffelli
Computation and Language · Cross-listed from cs.IR
SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval
Roxana Petcu, Evangelos Kanoulas and Maarten de Rijke
Large
language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often require multi-step
reasoning over the retrieved information, with no clear or predetermined
reasoning path, they remain challenging. Recent approaches train models using reinforcement learning on the model's outcome, showing promise in improving how models handle complex information. We introduce SubSearch, a specialized framework that shifts from outcome-only supervision to intermediate reward signals that incentivize planning high-quality
reasoning. Unlike previous work on process reward modeling, which focuses on training a separate reward model with annotated trajectories by either human annotators or large LLM judges, SubSearch directly optimizes the generator using intrinsic process rewards, which we define as internally-derived rewards, eliminating the need for external supervision, and moving towards autonomous information-intensive
reasoning. Experiments on seven
benchmarks show that rewarding intermediate
reasoning steps with intrinsic rewards leads to more robust
reasoning traces in both QA and multi-hop QA
datasets over using only outcome rewards. SubSearch can help in building
reasoning traces that allow
agents to better integrate search engines for complex query answering, while offering a data-efficient alternative to supervised process modeling.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Quantong Qiu, Zhiyi Hong, Yi Yang et al. (+5 more)
The quadratic computational complexity of standard
attention mechanisms presents a severe scalability bottleneck for LLMs in long-context scenarios. While hybrid
attention mechanisms combining Full
Attention (FA) and Sparse
Attention (SA) offer a potential solution, existing methods typically rely on static allocation ratios that fail to accommodate the variable
retrieval demands of different tasks. Furthermore, head-level dynamic sparsity often introduces severe computational load imbalance and synchronization long-tails, which hinder hardware acceleration during autoregressive decoding. To bridge this gap, we introduce Flux
Attention, a context-aware framework that dynamically optimizes
attention computation at the layer level. By integrating a lightweight Layer Router into frozen pretrained LLMs, the proposed method adaptively routes each layer to FA or SA based on the input context. This layer-wise routing preserves high-fidelity information
retrieval while ensuring contiguous
memory access, translating theoretical computational reductions into practical wall-clock speedups. As a parameter-efficient approach, our framework requires only 12 hours of training on 8$\times$A800 GPUs. Extensive experiments across multiple long-context and mathematical
reasoning benchmarks demonstrate that Flux
Attention achieves a superior trade-off between performance and inference speed compared with baseline models, with speed improvements of up to $2.8\times$ and $2.0\times$ in the prefill and decode stages.
Read the full abstract ▾
All authors Quantong Qiu, Zhiyi Hong, Yi Yang, Haitian Wang, Kebin Liu, Qingqing Dang, Juntao Li, Min Zhang
Computation and Language
Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Qiyao Ma, Dechen Gao, Rui Cai et al. (+4 more)
Pluralistic
alignment has emerged as a critical frontier in the development of Large
Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While
benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel
benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model
benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our
benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.
Read the full abstract ▾
All authors Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou, Junshan Zhang, Zhe Zhao
Computation and Language · Cross-listed from cs.CV
Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir et al. (+4 more)
Recent advances in
vision-
language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural
benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic
alignment with reference annotations. To assess cultural
reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Read the full abstract ▾
All authors Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir, Qianqian Xie, Stavroula Golfomitsou, Konstantinos Arvanitis, Sophia Ananiadou
Computation and Language
Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction
Jackson Petty, Jaulie Goe and Tal Linzen
Low-resource
languages pose a challenge for machine translation with large
language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of
languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the
languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal
languages designed to model particular aspects of natural
language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal
language into another when given both the grammar and the source-
language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the
languages, and their written script. We note three key findings. First, LLMs' translation accuracy decreases markedly as a function of grammar size and sentence length. Second, differences in morphology and written representation between the source and target
languages can strongly diminish model performance. Third, we examine the types of errors committed by models and find they are most prone to recall the wrong words from the target
language vocabulary, hallucinate new words, or leave source-
language words untranslated.
Read the full abstract ▾
Computation and Language
OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligence
Jianhui Liu, Haoze Sun, Wenbo Li et al. (+11 more)
Spatial understanding is a fundamental cornerstone of human-level intelligence. Nonetheless, current research predominantly focuses on domain-specific data production, leaving a critical void: the absence of a principled, open-source engine capable of fully unleashing the potential of high-quality spatial data. To bridge this gap, we elucidate the design principles of a robust data generation system and introduce OpenSpatial – an open-source data engine engineered for high quality, extensive scalability, broad task diversity, and optimized efficiency. OpenSpatial adopts 3D bounding boxes as the fundamental primitive to construct a comprehensive data hierarchy across five foundational tasks: Spatial Measurement (SM), Spatial Relationship (SR), Camera Perception (CP), Multi-view Consistency (MC), and Scene-Aware
Reasoning (SAR). Leveraging this scalable infrastructure, we curate OpenSpatial-3M, a large-scale
dataset comprising 3 million high-fidelity samples. Extensive evaluations demonstrate that versatile models trained on our
dataset achieve state-of-the-art performance across a wide spectrum of spatial
reasoning benchmarks. Notably, the best-performing model exhibits a substantial average improvement of 19 percent, relatively. Furthermore, we provide a systematic analysis of how data attributes influence spatial perception. By open-sourcing both the engine and the 3M-scale
dataset, we provide a robust foundation to accelerate future research in spatial intelligence.
Read the full abstract ▾
All authors Jianhui Liu, Haoze Sun, Wenbo Li, Yanbing Zhang, Rui Yang, Zhiliang Zhu, Yijun Yang, Shenghe Zheng, Nan Jiang, Jiaxiu Jiang, Haoyang Huang, Tien-Tsin Wong, Nan Duan, Xiaojuan Qi
Computation and Language
Why teaching resists automation in an AI-inundated era: Human judgment, non-modular work, and the limits of delegation
Songhee Han
Debates about artificial intelligence (AI) in education often portray teaching as a modular and procedural job that can increasingly be automated or delegated to technology. This brief communication paper argues that such claims depend on treating teaching as more separable than it is in practice. Drawing on recent literature and empirical studies of large
language models and
retrieval-augmented generation systems, I argue that although AI can support some bounded functions, instructional work remains difficult to automate in meaningful ways because it is inherently interpretive, relational, and grounded in professional judgment. More fundamentally, teaching and learning are shaped by human cognition, behavior, motivation, and social interaction in ways that cannot be fully specified, predicted, or exhaustively modeled. Tasks that may appear separable in principle derive their instructional value in practice from ongoing contextual interpretation across learners, situations, and relationships. As long as educational practice relies on emergent understanding of human cognition and learning, teaching remains a form of professional work that resists automation. AI may improve access to information and support selected instructional activities, but it does not remove the need for human judgment and relational accountability that effective teaching requires.
Read the full abstract ▾
Computation and Language
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Nusrat Sultana, Abdullah Muhammad Moosa, Kazi Afzalur Rahman et al. (+1 more)
Large
language models (LLMs) have demonstrated strong capabilities in medical question answering; however, purely parametric models often suffer from knowledge gaps and limited factual grounding.
Retrieval-augmented generation (RAG) addresses this limitation by integrating external knowledge
retrieval into the
reasoning process. Despite increasing interest in RAG-based medical systems, the impact of individual
retrieval components on performance remains insufficiently understood. This study presents a systematic evaluation of
retrieval-augmented medical question answering using the MedQA USMLE
benchmark and a structured textbook-based knowledge corpus. We analyze the interaction between
language models,
embedding models,
retrieval strategies, query reformulation, and cross-encoder reranking within a unified experimental framework comprising forty configurations. Results show that
retrieval augmentation significantly improves zero-shot medical question answering performance. The best-performing configuration was dense
retrieval with query reformulation and reranking achieved 60.49% accuracy. Domain-specialized
language models were also found to better utilize retrieved medical evidence than general-purpose models. The analysis further reveals a clear tradeoff between
retrieval effectiveness and computational cost, with simpler dense
retrieval configurations providing strong performance while maintaining higher throughput. All experiments were conducted on a single consumer-grade GPU, demonstrating that systematic evaluation of
retrieval-augmented medical QA systems can be performed under modest computational resources.
Read the full abstract ▾
All authors Nusrat Sultana, Abdullah Muhammad Moosa, Kazi Afzalur Rahman, Sajal Chandra Banik
Computation and Language
ClickGuard: A Trustworthy Adaptive Fusion Framework for Clickbait Detection
Chhavi Dhiman, Naman Chawla, Riya Dhami et al. (+2 more)
The widespread use of clickbait headlines, crafted to mislead and maximize engagement, poses a significant challenge to online credibility. These headlines employ sensationalism, misleading claims, and vague
language, underscoring the need for effective detection to ensure trustworthy digital content. The paper introduces, ClickGuard: a trustworthy adaptive fusion framework for clickbait detection. It combines BERT
embeddings and structural features using a Syntactic-Semantic Adaptive Fusion Block (SSAFB) for dynamic integration. The framework incorporates a hybrid CNN-BiLSTM to capture patterns and dependencies. The model achieved 96.93% testing accuracy, outperforming state-of-the-art approaches. The model's trustworthiness is evaluated using LIME and Permutation Feature Importance (PFI) for interpretability and perturbation analysis. These methods assess the model's robustness and sensitivity to feature changes by measuring the average prediction variation. Ablation studies validated the SSAFB's effectiveness in optimizing feature fusion. The model demonstrated robust performance across diverse
datasets, providing a scalable, reliable solution for enhancing online content credibility by addressing syntactic-semantic modelling challenges. Code of the work is available at: https://github.com/palindromeRice/ClickBait_Detection_Architecture
Read the full abstract ▾
All authors Chhavi Dhiman, Naman Chawla, Riya Dhami, Gaurav Kumar, Ganesh Naik
Computation and Language
Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agent
Bingxuan Li, Simo Du and Yue Guo
Clinical expertise improves not only by acquiring medical knowledge, but by accumulating experience that yields reusable diagnostic patterns. Recent LLMs-based diagnostic
agents have shown promising progress in clinical
reasoning for decision support. However, most approaches treat cases independently, limiting experience reuse and continual adaptation. We propose SEA, a self-learning diagnostic
agent with cognitively inspired dual-
memory module. We design a reinforcement training framework tailored to our designed
agent for joint optimization of
reasoning and
memory management. We evaluate SEA in two complementary settings. On standard evaluation with MedCaseReasoning
dataset, SEA achieves 92.46% accuracy, outperforming the strongest baseline by +19.6%, demonstrating the benefit of jointly optimizing
reasoning and
memory. On the long-horizon with ER-Reason
dataset, SEA attains the best final accuracy (0.7214) and the largest improvement (+0.35 Acc@100), while baseline methods show limited or unstable gains. Expert evaluation further indicates that rules consolidated from SEA show strong clinical correctness, usefulness and trust, suggesting that the induced rules in dual-
memory module are reliable and practically meaningful. Overall, SEA improves both diagnostic
reasoning ability and continual learning by effectively transforming experience into reusable knowledge.
Read the full abstract ▾
Computation and Language
Efficient Learned Data Compression via Dual-Stream Feature Decoupling
Huidong Ma, Xinyan Shi, Hui Sun et al. (+4 more)
While Learned Data Compression (LDC) has achieved superior compression ratios, balancing precise probability modeling with system efficiency remains challenging. Crucially, uniform single-stream architectures struggle to simultaneously capture micro-syntactic and macro-semantic features, necessitating deep serial stacking that exacerbates latency. Compounding this, heterogeneous systems are constrained by device speed mismatches, where throughput is capped by Amdahl's Law due to serial processing. To this end, we propose a Dual-Stream Multi-Scale Decoupler that disentangles local and global contexts to replace deep serial processing with shallow parallel streams, and incorporate a Hierarchical Gated Refiner for adaptive feature refinement and precise probability modeling. Furthermore, we design a Concurrent Stream-Parallel Pipeline, which overcomes systemic bottlenecks to achieve full-pipeline parallelism. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both compression ratio and throughput, while maintaining the lowest latency and
memory usage. The code is available at https://github.com/huidong-ma/FADE.
Read the full abstract ▾
All authors Huidong Ma, Xinyan Shi, Hui Sun, Xiaofei Yue, Xiaoguang Liu, Gang Wang, Wentong Cai
Computation and Language · Cross-listed from cs.LG
On the Price of Privacy for Language Identification and Generation
Xiaoyu Li, Andi Han, Jiaojiao Jiang et al. (+1 more)
As large
language models (LLMs) are increasingly trained on sensitive user data, understanding the fundamental cost of
privacy in
language learning becomes essential. We initiate the study of differentially private (DP)
language identification and generation in the agnostic statistical setting, establishing
algorithms and matching lower bounds that precisely quantify the cost of
privacy. For both tasks, approximate $(\varepsilon, \delta)$-DP with constant $\varepsilon > 0$ recovers the non-private error rates: $\exp(-r(n))$ for identification (for any $r(n) = o(n)$) and $\exp(-\Omega(n))$ for generation. Under pure $\varepsilon$-DP, the exponents degrade by a multiplicative factor of $\min\{1, \varepsilon\}$, which we show is tight up to constants. Notably, for generation under pure DP with mild assumptions, the upper bound $\exp(-\min\{1,\varepsilon\} \cdot \Omega(n))$ matches the lower bound up to some constants, establishing an optimal rate. Our results show that the cost of
privacy in
language learning is surprisingly mild: absent entirely under approximate DP, and exactly a $\min\{1,\varepsilon\}$ factor in the exponent under pure DP.
Read the full abstract ▾
All authors Xiaoyu Li, Andi Han, Jiaojiao Jiang, Junbin Gao
Computation and Language · Cross-listed from cs.AI
How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent
Sungwoo Jung and Seonil Son
Agent harnesses – the stateful programs that wrap a
language model and decide what it sees at each step – are now known to change end-to-end performance on a fixed model by as much as six times. That raises a question asked less often than it should be: how much of an
agent's competence does the harness itself already carry, and how much genuinely still needs the LLM? We externalize a planning harness for noisy Collaborative Battleship into four progressively richer layers – posterior belief tracking, declarative planning, symbolic reflec tion, and an LLM-backed revision gate – under a common runtime, taking win rate as the primary metric and F1 as secondary, and pre-specifying heavy lifting as the single largest positive marginal to the primary metric. Across 54 games, declarative pla nning carries the heavy lifting ($+24.1$pp win rate over a belief-only harness, zero LLM calls); symbolic reflection is mechanistically real but calibration-sensitive, with signed board-level effects up to $\pm0.140$ F1 that cancel on aggregate; and LLM-backed revision ac tivates on only $4.3\%$ of turns with a bounded, non-monotonic effect. The contribution is methodological: once harness layers are made externally measurable, the LLM's role can be quantified as residual rather than assumed central.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang et al. (+1 more)
As large
language models (LLMs) evolve from static chatbots into autonomous
agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces. While safety guardrails are well-benchmarked for natural
language responses, their efficacy remains largely unexplored within multi-step tool-use trajectories. To address this gap, we introduce **TraceSafe-Bench**, the first comprehensive
benchmark specifically designed to assess mid-trajectory safety. It encompasses 12 risk categories, ranging from security threats (e.g.,
prompt injection,
privacy leaks) to operational failures (e.g.,
hallucinations, interface inconsistencies), featuring over 1,000 unique execution instances. Our evaluation of 13 LLM-as-a-guard models and 7 specialized guardrails yields three critical findings: 1) *Structural Bottleneck*: Guardrail efficacy is driven more by structural data competence (e.g., JSON parsing) than semantic safety
alignment. Performance correlates strongly with structured-to-text
benchmarks ($\rho=0.79$, $p<0.01$) while we detect no association with standard jailbreak robustness. 2) *Scale Is Not the Deciding Factor*: Trajectory detection accuracy does not scale monotonically with model size, and general-purpose LLMs consistently outperform specialized safety guardrails. 3) *Bounded Temporal Stability*: Detection accuracy improves with trajectory length within native context bounds as models process dynamic execution behavior, but degrades in extreme long-context regimes due to over-refusal.
Read the full abstract ▾
All authors Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang, Yun-Nung Chen
Computation and Language
LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics
Kosmas Pinitas and Ilias Maglogiannis
Predicting affect in unconstrained environments remains a fundamental challenge in human-centered AI. While deep neural
embeddings dominate contemporary approaches, they often lack interpretability and limit expert-driven refinement. We propose a novel framework that uses
Language Models (LMs) as semantic context conditioners over handcrafted affect descriptors to model changes in Valence and Arousal. Our approach begins with interpretable facial geometry and acoustic features derived from structured domain knowledge. These features are transformed into symbolic natural-
language descriptions encoding their affective implications. A pretrained LM processes these descriptions to generate semantic context
embeddings that act as high-level priors over affective dynamics. Unlike end-to-end black-box pipelines, our framework preserves feature transparency while leveraging the contextual abstraction capabilities of LMs. We evaluate the proposed method on the Aff-Wild2 and SEWA
datasets for affect change prediction. Experimental results show consistent improvements in accuracy for both Valence and Arousal compared to handcrafted-only and deep-
embedding baselines. Our findings demonstrate that semantic conditioning enables interpretable affect modelling without sacrificing predictive performance, offering a transparent and computationally efficient alternative to fully end-to-end architectures
Read the full abstract ▾
Computation and Language
Agent-Driven Corpus Linguistics: A Framework for Autonomous Linguistic Discovery
Jia Yu, Weiwei Yu, Pengfei Xiao et al. (+1 more)
Corpus linguistics has traditionally relied on human researchers to formulate hypotheses, construct queries, and interpret results - a process demanding specialized technical skills and considerable time. We propose
Agent-Driven Corpus Linguistics, an approach in which a large
language model (LLM), connected to a corpus query engine via a structured tool-use interface, takes over the investigative cycle: generating hypotheses, querying the corpus, interpreting results, and refining analysis across multiple rounds. The human researcher sets direction and evaluates final output. Unlike unconstrained LLM generation, every finding is anchored in verifiable corpus evidence. We treat this not as a replacement for the corpus-based/corpus-driven distinction but as a complementary dimension: it concerns who conducts the inquiry, not the epistemological relationship between theory and data. We demonstrate the framework by linking an LLM
agent to a CQP-indexed Gutenberg corpus (5 million
tokens) via the Model Context Protocol (MCP). Given only "investigate English intensifiers," the
agent identified a diachronic relay chain (so+ADJ > very > really), three pathways of semantic change (delexicalization, polarity fixation, metaphorical constraint), and register-sensitive distributions. A controlled baseline experiment shows that corpus grounding contributes quantification and falsifiability that the model cannot produce from training data alone. To test external validity, the
agent replicated two published studies on the CLMET corpus (40 million
tokens) - Claridge (2025) and De Smet (2013) - with close quantitative agreement.
Agent-driven corpus research can thus produce empirically grounded findings at machine speed, lowering the technical barrier for a broader range of researchers.
Read the full abstract ▾
All authors Jia Yu, Weiwei Yu, Pengfei Xiao, Fukun Xing
Computation and Language
Dynamic Context Evolution for Scalable Synthetic Data Generation
Ryan Lingo and Rajeev Chhajer
Large
language models produce repetitive output when prompted independently across many batches, a phenomenon we term cross-batch mode collapse: the progressive loss of output diversity when a
language model is prompted repeatedly without access to its prior generations. Practitioners have long mitigated this with ad hoc deduplication and seed rotation, but no principled framework exists. We introduce Dynamic Context
Evolution (DCE), comprising three mechanisms: (1) verbalized tail sampling (the model labels each idea with a guess about how obvious it is, and obvious ideas are discarded), which filters high-probability candidates via model self-assessment; (2) semantic
memory, which maintains a persistent
embedding index to reject near-duplicates across batches; and (3) adaptive
prompt evolution, which reconstructs the generation
prompt each batch using
memory state and rotating diversity strategies. In experiments across three domains (sustainable packaging concepts, educational exam questions, and creative writing
prompts) and two model families (gpt-5-mini and claude-haiku-4-5), a component ablation across 2-3 random seeds per method shows that DCE achieves 0.0 +/- 0.0% collapse versus 5.6 +/- 2.0% for naive prompting, while producing 17-18 HDBSCAN clusters per seed versus naive's volatile 2-17, indicating reliably richer conceptual structure. These results are validated with an independent
embedding model (all-MiniLM-L6-v2) and hold across sensitivity sweeps of the VTS threshold tau and dedup threshold delta. Deduplication and
prompt evolution are individually insufficient but jointly effective, at approximately $0.50 per 1,000 candidates using only standard API calls, with no fine-tuning or custom architectures required.
Read the full abstract ▾
Computation and Language
Language Bias under Conflicting Information in Multilingual LLMs
Robert Östling and Murathan Kurfal{\i}
Large
Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which
language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a
multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different
languages, for a range of
multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models and prompting
languages in which
languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. The
language preferences are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category. There is also a general tendency among models to prioritize information that matches the
language used for prompting. We hope to make users and developers of
multilingual LLMs aware of this category of biases, to spur further research on their causes and possible mitigation.
Read the full abstract ▾
Computation and Language
Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Ehsan Barkhordar, Abdulfattah Safa, Verena Blaschke et al. (+3 more)
Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study
language-of-study (LoS) bias: the tendency for reviewers to evaluate a paper differently based on the
language(s) it studies, rather than its scientific merit. Despite being explicitly flagged in reviewing guidelines, such biases are poorly understood. Prior work treats such comments as part of broader categories of weak or unconstructive reviews without defining them as a distinct form of bias. We present the first systematic characterization of LoS bias, distinguishing negative and positive forms, and introduce the human-annotated
dataset LOBSTER (
Language-Of-study Bias in ScienTific pEer Review) and an LLM-based detection pipeline achieving 87.37 macro F1. We analyze 15,645 reviews to estimate how negative and positive biases differ with respect to the LoS, and find that non-English papers face substantially higher bias rates than English-only ones, with negative bias consistently outweighing positive bias. Finally, we identify four subcategories of negative bias, and find that demanding unjustified cross-lingual generalization is the most dominant form. We publicly release all resources to support work on fairer reviewing practices in NLP and beyond.
Read the full abstract ▾
All authors Ehsan Barkhordar, Abdulfattah Safa, Verena Blaschke, Erika Lombart, Marie-Catherine de Marneffe, Gözde Gül Şahin
Computation and Language
Yale-DM-Lab at ArchEHR-QA 2026: Deterministic Grounding and Multi-Pass Evidence Alignment for EHR Question Answering
Elyas Irankhah and Samah Fodeh
We describe the Yale-DM-Lab system for the ArchEHR-QA 2026 shared task. The task studies patient-authored questions about hospitalization records and contains four subtasks (ST): clinician-interpreted question reformulation, evidence sentence identification, answer generation, and evidence-answer
alignment. ST1 uses a dual-model pipeline with Claude Sonnet 4 and GPT-4o to reformulate patient questions into clinician-interpreted questions. ST2-ST4 rely on Azure-hosted model ensembles (o3, GPT-5.2, GPT-5.1, and DeepSeek-R1) combined with few-shot prompting and voting strategies. Our experiments show three main findings. First, model diversity and ensemble voting consistently improve performance compared to single-model baselines. Second, the full clinician answer paragraph is provided as additional
prompt context for evidence
alignment. Third, results on the development set show that
alignment accuracy is mainly limited by
reasoning. The best scores on the development set reach 88.81 micro F1 on ST4, 65.72 macro F1 on ST2, 34.01 on ST3, and 33.05 on ST1.
Read the full abstract ▾
Computation and Language
Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring
Yongchao Wu and Aron Henriksson
Activation-based steering enables inference-time personalization of large
language models, but its effects in educational applications are not well understood. We study activation-based persona vectors representing seven character traits in short-answer generation and automated scoring on the ASAP-SAS
benchmark, across three
language models spanning dense and mixture-of-experts architectures. Persona steering lowers answer quality overall, with much larger effects on open-ended English
Language Arts (ELA)
prompts than on factual science
prompts. Interpretive and argumentative tasks are particularly sensitive, showing up to 11$\times$ larger degradation. On the scoring side, we observe predictable valence-aligned calibration shifts: “evil” and “impolite” scorers grade more harshly, while “good” and “optimistic” scorers grade more leniently. ELA tasks are 2.5-3$\times$ more susceptible to scorer personalization than science tasks, and the mixture-of-experts model shows roughly 6$\times$ larger calibration shifts than the dense models. To our knowledge, this is the first study to systematically examine the effects of activation-steered persona traits in educational generation and scoring. Our findings highlight the need for task- and architecture-aware calibration when deploying personalized models in educational settings.
Read the full abstract ▾
Computation and Language
STRIDE-ED: A Strategy-Grounded Stepwise Reasoning Framework for Empathetic Dialogue Systems
Hongru Ji, Yuyin Fan, Meng Zhao et al. (+3 more)
Empathetic dialogue requires not only recognizing a user's emotional state but also making strategy-aware, context-sensitive decisions throughout response generation. However, the lack of a comprehensive empathy strategy framework, explicit task-aligned multi-stage
reasoning, and high-quality strategy-aware data fundamentally limits existing approaches, preventing them from effectively modeling empathetic dialogue as a complex, multi-stage cognitive and decision-making process. To address these challenges, we propose STRIDE-ED, a STRategy-grounded, Interpretable, and DEep
reasoning framework that models Empathetic Dialogue through structured, strategy-conditioned
reasoning. To support effective learning, we develop a strategy-aware data refinement pipeline integrating LLM-based annotation, multi-model consistency-weighted evaluation, and dynamic sampling to construct high-quality training data aligned with empathetic strategies. Furthermore, we adopt a two-stage training paradigm that combines supervised fine-tuning with multi-objective reinforcement learning to better align model behaviors with target
emotions, empathetic strategies, and response formats. Extensive experiments demonstrate that STRIDE-ED generalizes across diverse open-source LLMs and consistently outperforms existing methods on both automatic metrics and human evaluations. Our data and code are publicly available at https://github.com/jicoder-nwpu/STRIDE-ED.
Read the full abstract ▾
All authors Hongru Ji, Yuyin Fan, Meng Zhao, Xianghua Li, Lianwei Wu, Chao Gao
Computation and Language · Cross-listed from cs.LG
Ryyan Akhtar, Payal Pahwa and Monika Arora
Large
language models often fail on tasks they seem to already understand. In our experiments, this appears to be less about missing knowledge and more about certain internal
circuits not being strongly activated during inference. We explore Selective
Neuron Amplification, which increases the influence of task relevant
neurons without changing the model's parameters. The method works at inference time and does not permanently alter the model. SNA helps mainly when the model is uncertain, while having low effect when the model is already confident. This suggests that some model failures are due to weak activation rather than lack of capability.
Read the full abstract ▾
Computation and Language
Multilingual Embedding Probes Fail to Generalize Across Learner Corpora
Laurits Lyngbaek and Ross Deans Kristensen-McLachlan
Do
multilingual embedding models encode a
language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from seven
embedding models (0.3-8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven
languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance (Quadratic Weighted Kappa $\approx0.7$), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic,
language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current
multilingual embeddings do not straightforwardly encode
language-general proficiency, with implications for representation-based approaches to proficiency-adaptive
language technology.
Read the full abstract ▾
Computation and Language
Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and English
Iza Škrjanec, Irene Elisabeth Winther, Vera Demberg et al. (+1 more)
Bilingual speakers show cross-lingual activation during reading, especially for words with shared surface form. Cognates (friends) typically lead to facilitation, whereas interlingual homographs (false friends) cause interference or no effect. We examine whether cross-lingual activation in bilingual
language models mirrors these patterns. We train Dutch-English causal
Transformers under four vocabulary-sharing conditions that manipulate whether (false) friends receive shared or
language-specific
embeddings. Using psycholinguistic stimuli from bilingual reading studies, we evaluate the models through surprisal and
embedding similarity analyses. The models largely maintain
language separation, and cross-lingual effects arise primarily when
embeddings are shared. In these cases, both friends and false friends show facilitation relative to controls. Regression analyses reveal that these effects are mainly driven by frequency rather than consistency in form-meaning mapping. Only when just friends share
embeddings are the qualitative patterns of bilinguals reproduced. Overall, bilingual
language models capture some cross-linguistic activation effects. However, their
alignment with human processing seems to critically depend on how lexical overlap is encoded, possibly limiting their explanatory adequacy as models of bilingual reading.
Read the full abstract ▾
All authors Iza Škrjanec, Irene Elisabeth Winther, Vera Demberg, Stefan L. Frank
Computation and Language
SemEval-2026 Task 3: Dimensional Aspect-Based Sentiment Analysis (DimABSA)
Liang-Chih Yu, Jonas Becker, Shamsuddeen Hassan Muhammad et al. (+14 more)
We present the SemEval-2026 shared task on Dimensional Aspect-Based Sentiment Analysis (DimABSA), which improves traditional ABSA by modeling sentiment along valence-arousal (VA) dimensions rather than using categorical polarity labels. To extend ABSA beyond consumer reviews to public-issue discourse (e.g., political, energy, and
climate issues), we introduce an additional task, Dimensional Stance Analysis (DimStance), which treats stance targets as aspects and reformulates stance detection as regression in the VA space. The task consists of two tracks: Track A (DimABSA) and Track B (DimStance). Track A includes three subtasks: (1) dimensional aspect sentiment regression, (2) dimensional aspect sentiment triplet extraction, and (3) dimensional aspect sentiment quadruplet extraction, while Track B includes only the regression subtask for stance targets. We also introduce a continuous F1 (cF1) metric to jointly evaluate structured extraction and VA regression. The task attracted more than 400 participants, resulting in 112 final submissions and 42 system description papers. We report baseline results, discuss top-performing systems, and analyze key design choices to provide insights into dimensional sentiment analysis at the aspect and stance-target levels. All resources are available on our GitHub repository.
Read the full abstract ▾
All authors Liang-Chih Yu, Jonas Becker, Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Lung-Hao Lee, Ying-Lung Lin, Jin Wang, Jan Philip Wahle, Terry Ruas, Natalia Loukachevitch, Alexander Panchenko, Ilseyar Alimova, Lilian Wanzare, Nelson Odhiambo, Bela Gipp, Kai-Wei Chang, Saif M. Mohammad
Computation and Language
IndoBERT-Sentiment: Context-Conditioned Sentiment Classification for Indonesian Text
Muhammad Apriandito Arya Saputra, Andry Alamsyah, Dian Puteri Ramadhani et al. (+2 more)
Existing Indonesian sentiment analysis models classify text in isolation, ignoring the topical context that often determines whether a statement is positive, negative, or neutral. We introduce IndoBERT-Sentiment, a context-conditioned sentiment classifier that takes both a topical context and a text as input, producing sentiment predictions grounded in the topic being discussed. Built on IndoBERT Large (335M parameters) and trained on 31,360 context-text pairs labeled across 188 topics, the model achieves an F1 macro of 0.856 and accuracy of 88.1%. In a head-to-head evaluation against three widely used general-purpose Indonesian sentiment models on the same test set, IndoBERT-Sentiment outperforms the best baseline by 35.6 F1 points. We show that context-conditioning, previously demonstrated for relevancy classification, transfers effectively to sentiment analysis and enables the model to correctly classify texts that are systematically misclassified by context-free approaches.
Read the full abstract ▾
All authors Muhammad Apriandito Arya Saputra, Andry Alamsyah, Dian Puteri Ramadhani, Thomhert Suprapto Siadari, Hanif Fakhrurroja
Computation and Language
Sell More, Play Less: Benchmarking LLM Realistic Selling Skill
Xuanbo Su, Wenhao Hu, Le Zhan et al. (+8 more)
Sales dialogues require multi-turn, goal-directed persuasion under asymmetric incentives, which makes them a challenging setting for large
language models (LLMs). Yet existing dialogue
benchmarks rarely measure deal progression and outcomes. We introduce SalesLLM
benchmark, a bilingual (ZH/EN)
benchmark derived from realistic applications covering Financial Services and Consumer Goods, built from 30,074 scripted configurations and 1,805 curated multi-turn scenarios with controllable difficulty and personas. We propose a fully automatic evaluation pipeline that combines (i) an LLM judge for sales-process progress, and (ii) fine-tuned BERT classifiers for end-of-dialogue buying intent. To improve simulation fidelity, we train a user model, CustomerLM, with SFT and DPO on 8,000+ crowdworker-involved sales conversations, reducing role inversion from 17.44% (GPT-4o) to 8.8%. SalesLLM
benchmark scores correlate strongly with human ratings (average Pearson r=0.86; inter-annotator agreement Krippendorff's alpha=0.86, 500 annotations per model). Across 15 mainstream LLMs, in Chinese the strongest models are competitive with typical (junior-to-intermediate) human salespeople – not with sales experts – while weaker ones fall below this baseline, and cross-lingual consistency remains poor. SalesLLM
benchmark serves as a scalable
benchmark for outcome-oriented sales
agents. Our code and data are released at https://github.com/Bairong-Xdynamics/Benchmarking-LLM-Realistic-Selling-Skill
Read the full abstract ▾
All authors Xuanbo Su, Wenhao Hu, Le Zhan, Yuting Xie, Kailin Lyu, Kaijie Chen, Ziwei Li, Yeqiang Wang, Haibo Su, Yunzhang Chen, Ling Huang