Computation and Language
MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge
Baochen Fu, Yuntao Du, Cheng Chang et al. (+6 more)
As real-world knowledge continues to evolve, the parametric knowledge acquired by
multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on
multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation
benchmark for
multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this
benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation
benchmark for
multimodal knowledge updating, advancing progress in this field.
Read the full abstract ▾
All authors Baochen Fu, Yuntao Du, Cheng Chang, Baihao Jin, Wenzhi Deng, Muhao Xu, Hongmei Yan, Weiye Song, Yi Wan
Computation and Language
Bridging National and International Legal Data: Two Projects Based on the Japanese Legal Standard XML Schema for Comparative Law Studies
Makoto Nakamura
This paper presents an integrated framework for computational comparative law by connecting two consecutive research projects based on the Japanese Legal Standard (JLS) XML schema. The first project establishes structural interoperability by developing a conversion pipeline from JLS to the Akoma Ntoso (AKN) standard, enabling Japanese statutes to be integrated into international LegalDocML-based legislative databases. Building on this foundation, the second project applies
multilingual embedding models and semantic textual similarity techniques to identify corresponding provisions across national legal systems. A prototype system combining
multilingual embeddings, FAISS
retrieval, and Cross-Encoder reranking generates candidate correspondences and visualizes them as cross-jurisdictional networks for exploratory comparative analysis.
Read the full abstract ▾
Computation and Language
Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing
Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji et al. (+2 more)
Training models for open-ended writing requires diverse
prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting
prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.
Read the full abstract ▾
All authors Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou, Biao Qin
Computation and Language
Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs
Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha et al. (+1 more)
Token-level Chain-of-Thought (CoT) prompting has become a standard way to elicit multi-step
reasoning in large
language models (LLMs), especially for mathematical word problems. However, generating long intermediate traces increases output length and inference cost, and can be inefficient when the model could arrive at the correct answer without extensive verbalization. This has motivated latent-space
reasoning approaches that shift computation into hidden representations and only emit a final answer. Yet, many latent
reasoning methods depend on a fixed number of latent refinement steps at inference, adding another hyperparameter that must be tuned across models and
datasets to balance accuracy and efficiency. We introduce AdaAnchor, a latent
reasoning framework that performs silent iterative computation by refining a set of latent anchor vectors attached to the input. AdaAnchor further incorporates an adaptive halting mechanism that monitors anchor stability across iterations and terminates refinement once the anchor dynamics converge, allocating fewer steps to easier instances while reserving additional refinement steps for harder ones under a shared maximum-step budget. Our empirical evaluation across three mathematical word-problem
benchmarks shows that AdaAnchor with adaptive halting yields accuracy gains of up to 5% over fixed-step latent refinement while reducing average latent refinement steps by 48-60% under the same maximum-step budget. Compared to standard
reasoning baselines, AdaAnchor achieves large reductions in generated
tokens (92-93%) by moving computation into silent latent refinement, offering a different accuracy-efficiency trade-off with substantially lower output-
token usage.
Read the full abstract ▾
All authors Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta
Computation and Language
Interpretable Predictability-Based AI Text Detection: A Replication Study
Adam Skurla, Dominik Macko and Jakub Simko
This paper replicates and extends the system used in the AuTexTification shared task for authorship attribution of machine-generated texts. Exact replication was not possible because of differences in data splits, model availability, and implementation details, which we document as a case study in reproducibility. We tested newer
multilingual language models (mDeBERTa-v3-base, Qwen, mGPT) and added 26 document-level stylometric features, using ablation, permutation importance, and SHAP analysis to assess feature influence. A single shared configuration was applied to both English and Spanish across Subtask 1 and Subtask 2. Averaged over three random seeds, the shared
multilingual configuration performs comparably to or better than the
language-specific baseline, with the clearest gains on model attribution (Subtask 2). The additional stylometric features yield small improvements, led by lexical diversity, but their contribution falls within seed variance once predictability-based probabilities are included, which remain the dominant signal. The study also shows that clear documentation is important for reliable replication and fair comparison of systems.
Read the full abstract ▾
Computation and Language
Kimi Team: Guangyu Chen, Yu Zhang, Jianlin Su et al. (+33 more)
Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose
Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax
attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the
memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the
memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead. Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T
tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.
Read the full abstract ▾
All authors Kimi Team: Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang, Guanduo Chen, Bohong Yin, Yutian Chen, Junjie Yan, Ming Wei, Y. Zhang, Fanqing Meng, Chao Hong, Xiaotong Xie, Shaowei Liu, Enzhe Lu, Yunpeng Tai, Yanru Chen, Xin Men, Haiqing Guo, Y. Charles, Haoyu Lu, Lin Sui, Jinguo Zhu, Zaida Zhou, Weiran He, Weixiao Huang, Xinran Xu, Yuzhi Wang, Guokun Lai, Yulun Du, Yuxin Wu, Zhilin Yang, Xinyu Zhou
Computation and Language · Cross-listed from cs.CV
MER-Bench: Benchmarking Multimodal Emotional Reappraisal for Image-Text Memes
Yiqi Nie, Fei Wang, Junjie Chen et al. (+5 more)
Multimodal Emotional Reappraisal (MER) aims to shift affective interpretation by rewriting and editing a situation without altering its underlying semantic context. Despite advances in controllable editing and sentiment transfer, MER remains challenging under coupled
multimodal constraints, especially in internet memes where images, embedded text, and layout jointly encode affective meaning. To bridge this gap, we present MER-Bench, a systematic
benchmark for MER on image-text memes, containing over 3,000 real-world memes with fine-grained annotations for
emotion transformation and structure-preserving
multimodal editing. MER-Bench spans diverse visual, textual, and structural settings and is paired with a structured MLLM-as-a-Judge evaluation suite, comprising eight fine-grained metrics and three aggregated objectives for emotional
alignment, content fidelity, and transformation strength. Extensive evaluation of 14 advanced
multimodal models reveals that: (1) current models struggle to jointly balance affect, semantics, structure, and text; (2) strong visual editing does not imply effective MER, with poor embedded text rendering and affect-aware rewriting remaining key bottlenecks. Our results identify MER as a critical capability gap in current
multimodal generation systems and highlight affective image-text
reasoning as a key challenge for future work. Project Page: https://github.com/one-seven17/MER-Bench
Read the full abstract ▾
All authors Yiqi Nie, Fei Wang, Junjie Chen, Kun Li, Yudi Cai, Dan Guo, Chenglong Li, Meng Wang
Computation and Language
Pretraining and Benchmarking Modern Encoders for Latvian
Arturs Znotins
Encoder-only
transformers remain essential for practical NLP tasks. While recent advances in
multilingual models have improved cross-lingual capabilities, low-resource
languages such as Latvian remain underrepresented in pretraining corpora, and few monolingual Latvian encoders currently exist. We address this gap by pretraining a suite of Latvian-specific encoders based on RoBERTa, DeBERTaV3, and ModernBERT architectures, including long-context variants, and evaluating them across a diverse set of Latvian diagnostic and linguistic
benchmarks. Our models are competitive with existing monolingual and
multilingual encoders while benefiting from recent architectural and efficiency advances. Our best model, lv-deberta-base (111M parameters), achieves the strongest overall performance, outperforming larger
multilingual baselines and prior Latvian-specific encoders. We release all pretrained models and evaluation resources to support further research and practical applications in Latvian NLP.
Read the full abstract ▾
Computation and Language
OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora
Jeffrey Flynt
Building and evaluating enterprise AI systems requires synthetic organizational corpora that are internally consistent, temporally structured, and cross-artifact traceable. Existing corpora either carry legal constraints or inherit
hallucination artifacts from the generating LLMs, silently corrupting results when timestamps or facts contradict across documents and reinforcing those errors during training. We present OrgForge, an open-source multi-
agent simulation framework that enforces a strict physics-cognition boundary: a deterministic Python engine maintains a SimEvent ground-truth bus while LLMs generate only surface prose. OrgForge simulates the organizational processes that produce documents, not the documents themselves. Engineers leave mid-sprint, triggering incident handoffs and CRM ownership lapses. Knowledge gaps emerge when under-documented systems break and recover through organic documentation and incident resolution. Customer emails fire only when simulation state warrants contact; silence is verifiable ground truth. A live CRM state machine extends the physics-cognition boundary to the customer boundary, producing cross-system causal cascades spanning engineering incidents, support escalation, deal risk flagging, and SLA-adjusted invoices. The framework generates fifteen interleaved artifact categories traceable to a shared immutable event log. Four graph-dynamic subsystems govern organizational behavior independently of any LLM. An
embedding-based ticket assignment system using the Hungarian
algorithm makes the simulation domain-agnostic. An empirical evaluation across ten incidents demonstrates a 0.46 absolute improvement in prose-to-ground-truth fidelity over chained LLM baselines, and isolates a consistent
hallucination failure mode in which chaining propagates fabricated facts faithfully across documents without correcting them.
Read the full abstract ▾
Computation and Language
Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI
Jinhu Qi, Yifan Li, Minghao Zhao et al. (+4 more)
Agentic AI systems increasingly act through tool-augmented, multi-step workflows whose failures (unsafe tool use, unauthorised actions, social harm) carry deployment-level consequences. Evaluation practice remains fragmented across isolated
benchmark slices, and "trustworthiness" is frequently invoked but rarely defined operationally. We argue the central limitation is twofold: (i) the absence of a measurable specification of what
agent trustworthiness means, and (ii) the lack of a principled notion of representativeness allowing assessment over a socio-technical scenario distribution rather than disconnected
benchmark instances. We address (i) by defining agentic trustworthiness as a five-property profile (Reliability, Robustness, Safety, Social-Ethical
Alignment, Operational Integrity) grounded in current AI risk frameworks, and (ii) with the
Holographic Agent Assessment Framework (HAAF), which measures this profile over a scenario
manifold through static policy analysis, sandbox simulation, social-ethical
alignment assessment, and distribution-aware sampling, connected through an iterative Trustworthy Optimization Factory that converts red-team diagnoses into blue-team interventions. Our contributions are: (1) an operational five-property definition of agentic trustworthiness; (2) a distribution-aware scenario-sampling framework that surfaces property-level trade-offs invisible to scalar leaderboards; and (3) a cross-family transfer experiment in which interventions designed from a single focal model generalise – without per-model or per-scenario tuning – to 13 systems from seven model families (Llama, Mistral, Kimi, GLM, Qwen, GPT, DeepSeek) on a 100-scenario suite, where all 13 systems improve and two reach a perfect risk-weighted profile, establishing HAAF's Factory as a model-agnostic deployment-readiness pipeline. Code: https://github.com/TonyQJH/haaf-pilot
Read the full abstract ▾
All authors Jinhu Qi, Yifan Li, Minghao Zhao, Wentao Zhang, Zijian Zhang, Yaoman Li, Irwin King
Computation and Language · Cross-listed from cs.AI
Why Agents Compromise Safety Under Pressure
Hengle Jiang and Ke Tang
Large
Language Model
agents deployed in complex environments frequently encounter a conflict between maximizing goal achievement and adhering to safety constraints. This paper identifies a new concept called Agentic Pressure, which characterizes the endogenous tension emerging when compliant execution becomes infeasible. We demonstrate that under this pressure
agents exhibit normative drift where they strategically sacrifice safety to preserve utility. Notably we find that advanced
reasoning capabilities accelerate this decline as models construct linguistic rationalizations to justify violation. Finally, we analyze the root causes and explore preliminary mitigation strategies, such as pressure isolation, which attempts to restore
alignment by decoupling decision-making from pressure signals.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CR
Rethinking LLM Watermark Detection in Black-Box Settings: A Non-Intrusive Third-Party Framework
Zhuoshang Wang, Yubing Ren, Yanan Cao et al. (+3 more)
While watermarking serves as a critical mechanism for LLM provenance, existing secret-key schemes tightly couple detection with injection, requiring access to keys or provider-side scheme-specific detectors for verification. This dependency creates a fundamental barrier for real-world governance, as independent
auditing becomes impossible without compromising model security or relying on the opaque claims of service providers. To resolve this dilemma, we introduce TTP-Detect, a pioneering black-box framework designed for non-intrusive, third-party watermark verification. By decoupling detection from injection, TTP-Detect reframes verification as a relative hypothesis testing problem. It employs a proxy model to amplify watermark-relevant signals and a suite of complementary relative measurements to assess the
alignment of the query text with watermarked distributions. Extensive experiments across representative watermarking schemes,
datasets and models demonstrate that TTP-Detect achieves superior detection performance and robustness against diverse attacks.
Read the full abstract ▾
All authors Zhuoshang Wang, Yubing Ren, Yanan Cao, Fang Fang, Xiaoxue Li, Li Guo
Computation and Language · Cross-listed from cs.LG
LLM as Graph Kernel: Rethinking Message Passing on Text-Rich Graphs
Ying Zhang, Hang Yu, Haipeng Zhang et al. (+1 more)
Text-rich graphs, which integrate complex structural dependencies with abundant textual information, are ubiquitous yet remain challenging for existing learning paradigms. Conventional methods and even LLM-hybrids compress rich text into static
embeddings or summaries before structural
reasoning, creating an information bottleneck and detaching updates from the raw content. We argue that in text-rich graphs, the text is not merely a node attribute but the primary medium through which structural relationships are manifested. We introduce RAMP, a Raw-text Anchored Message Passing approach that moves beyond using LLMs as mere feature extractors and instead recasts the LLM itself as a graph-native aggregation operator. RAMP exploits the text-rich nature of the graph via a novel dual-representation scheme: it anchors inference on each node's raw text during each iteration while propagating dynamically optimized messages from neighbors. It further handles both discriminative and generative tasks under a single unified generative formulation. Extensive experiments show that RAMP effectively bridges the gap between graph propagation and deep text
reasoning, achieving competitive performance and offering new insights into the role of LLMs as graph kernels for general-purpose graph learning.
Read the full abstract ▾
All authors Ying Zhang, Hang Yu, Haipeng Zhang, Peng Di
Computation and Language · Cross-listed from cs.CR
Fine-tuning RoBERTa for CVE-to-CWE Classification: A 125M Parameter Model Competitive with LLMs
Nikita Mosievskiy
We present a fine-tuned RoBERTa-base classifier (125M parameters) for mapping Common Vulnerabilities and Exposures (CVE) descriptions to Common Weakness Enumeration (CWE) categories. We construct a large-scale training
dataset of 234,770 CVE descriptions with AI-refined CWE labels using Claude Sonnet 4.6, and agreement-filtered evaluation sets where NVD and AI labels agree. On our held-out test set (27,780 samples, 205 CWE classes), the model achieves 87.4% top-1 accuracy and 60.7% Macro F1 – a +15.5 percentage-point Macro F1 gain over a TF-IDF baseline that already reaches 84.9% top-1, demonstrating the model's advantage on rare weakness categories. On the external CTI-Bench
benchmark (NeurIPS 2024), the model achieves 75.6% strict accuracy (95% CI: 72.8-78.2%) – statistically indistinguishable from Cisco Foundation-Sec-8B-
Reasoning (75.3%, 8B parameters) at 64x fewer parameters. We release the
dataset, model, and training code.
Read the full abstract ▾
Computation and Language
ExPosST: Explicit Positioning with Adaptive Masking for LLM-Based Simultaneous Machine Translation
Yuzhe Shang, Pengzhi Gao, Yazheng Yang et al. (+4 more)
Large
language models (LLMs) have recently demonstrated promising performance in simultaneous machine translation (SimulMT). However, applying decoder-only LLMs to SimulMT introduces a positional mismatch, which leads to a dilemma between decoding efficiency and positional consistency. Existing approaches often rely on specific positional encodings or carefully designed prompting schemes, and thus fail to simultaneously achieve inference efficiency, positional consistency, and broad model compatibility. In this work, we propose ExPosST, a general framework that resolves this dilemma through explicit position allocation. ExPosST reserves fixed positional slots for incoming source
tokens, enabling efficient decoding with KV cache across different positional encoding methods. To further bridge the gap between fine-tuning and inference, we introduce a policy-consistent fine-tuning strategy that aligns training with inference-time decoding behavior. Experiments across multiple
language pairs demonstrate that ExPosST effectively supports simultaneous translation under diverse policies.
Read the full abstract ▾
All authors Yuzhe Shang, Pengzhi Gao, Yazheng Yang, Jiayao Ma, Wei Liu, Jian Luan, Jinsong Su
Computation and Language
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
Jon-Paul Cacioli
Large
language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition
memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.
Read the full abstract ▾
Computation and Language
Decision-Level Ordinal Modeling for Multimodal Essay Scoring with Large Language Models
Han Zhang, Jiamin Su and Li liu
Automated essay scoring (AES) predicts multiple rubric-defined trait scores for each essay, where each trait follows an ordered discrete rating scale. Most LLM-based AES methods cast scoring as autoregressive
token generation and obtain the final score via decoding and parsing, making the decision implicit. This formulation is particularly sensitive in
multimodal AES, where the usefulness of visual inputs varies across essays and traits. To address these limitations, we propose Decision-Level Ordinal Modeling (DLOM), which makes scoring an explicit ordinal decision by reusing the
language model head to extract score-wise logits on predefined score
tokens, enabling direct optimization and analysis in the score space. For
multimodal AES, DLOM-GF introduces a gated fusion module that adaptively combines textual and
multimodal score logits. For text-only AES, DLOM-DA adds a distance-aware regularization term to better reflect ordinal distances. Experiments on the
multimodal EssayJudge
dataset show that DLOM improves over a generation-based SFT baseline across scoring traits, and DLOM-GF yields further gains when modality relevance is heterogeneous. On the text-only ASAP/ASAP++
benchmarks, DLOM remains effective without visual inputs, and DLOM-DA further improves performance and outperforms strong representative baselines.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.AS
SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
Jingyu Lu, Yuhan Wang, Fan Zhuo et al. (+7 more)
The rapid
evolution of end-to-end spoken dialogue systems demands transcending mere textual semantics to incorporate paralinguistic nuances and the spontaneous nature of human conversation. However, current methods struggle with two critical gaps: the modality gap, involving prosody and
emotion, and the colloquialness gap, distinguishing written scripts from natural
speech. To address these challenges, we introduce SDiaReward, an end-to-end multi-turn reward model trained on SDiaReward-
Dataset, a novel collection of episode-level preference pairs explicitly targeting these gaps. It operates directly on full multi-turn
speech episodes and is optimized with pairwise preference supervision, enabling joint assessment of modality and colloquialness in a single evaluator. We further establish ESDR-Bench, a stratified
benchmark for robust episode-level evaluation. Experiments demonstrate that SDiaReward achieves state-of-the-art pairwise preference accuracy, significantly outperforming general-purpose audio LLMs. Further analysis suggests that SDiaReward captures relative conversational expressiveness beyond superficial synthesis cues, improving generalization across domains and recording conditions. Code, data, and demos are available at https://github.com/MM-
Speech/SDiaReward/.
Read the full abstract ▾
All authors Jingyu Lu, Yuhan Wang, Fan Zhuo, Xize Cheng, Changhao Pan, Xueyi Pu, Yifu Chen, Chenyuhao Wen, Tianle Liang, Zhou Zhao
Computation and Language · Cross-listed from cs.HC
Customizing ChatGPT for Second Language Speaking Practice: Genuine Support or Just a Marketing Gimmick?
Fanfei Meng
ChatGPT, with its customization features and Voice Mode, has the potential for more engaging and peresonalized ESL (English as a Second
Language) education. This study examines the efficacy of customized ChatGPT conversational features in facilitating ESL speaking practices, comparing the performance of four versions of ChatGPT Voice Mode: uncustomized Standard mode, uncustomized Advanced mode, customized Standard mode, and customized Advanced mode. Customization was guided by
prompt engineering principles and grounded in relevant theories, including Motivation Theory, Culturally Responsive Teaching (CRT), Communicative
Language Teaching (CLT), and the Affective Filter Hypothesis. Content analysis found that customized versions generally provided more balanced feedback and emotional support, contributing to a positive and motivating learning environment. However, cultural responsiveness did not show significant improvement despite targeted customization efforts. These initial findings suggest that customization could enhance ChatGPT's capacity as a more effective
language tutor, with the standard model already capable of meeting the learning needs. The study underscores the importance of
prompt engineering and AI literacy in maximizaing AI's potential in
language learning.
Read the full abstract ▾
Computation and Language
Developing an English-Efik Corpus and Machine Translation System for Digitization Inclusion
Offiong Bassey Edet, Mbuotidem Sunday Awak, Emmanuel Oyo-Ita et al. (+2 more)
Low-resource
languages serve as invaluable repositories of human history, preserving cultural and intellectual diversity. Despite their significance, they remain largely absent from modern natural
language processing systems. While progress has been made for widely spoken African
languages such as Swahili, Yoruba, and Amharic, smaller indigenous
languages like Efik continue to be underrepresented in machine translation research. This study evaluates the effectiveness of state-of-the-art
multilingual neural machine translation models for English-Efik translation, leveraging a small-scale, community-curated parallel corpus of 13,865 sentence pairs. We fine-tuned both the mT5
multilingual model and the NLLB200 model on this
dataset. NLLB-200 outperformed mT5, achieving BLEU scores of 26.64 for English-Efik and 31.21 for Efik-English, with corresponding chrF scores of 51.04 and 47.92, indicating improved fluency and semantic fidelity. Our findings demonstrate the feasibility of developing practical machine translation tools for low-resource
languages and highlight the importance of inclusive data practices and culturally grounded evaluation in advancing equitable NLP.
Read the full abstract ▾
All authors Offiong Bassey Edet, Mbuotidem Sunday Awak, Emmanuel Oyo-Ita, Benjamin Okon Nyong, Ita Etim Bassey
Computation and Language
Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks
Zijian Yu, Kejun Xiao, Huaipeng Zhao et al. (+2 more)
In e-commerce, LLM
agents show promise for shopping tasks such as recommendations, budget management, and bundle deals, where accurately capturing user preferences from long-horizon conversations is critical. However, progress is limited by two key challenges: (1) the absence of
benchmarks for evaluating long-term preference-aware shopping tasks, and (2) the lack of fine-grained supervision for shopping
agent training. To fill the
benchmark gap, we introduce Shopping Companion Bench, a novel
benchmark comprising two shopping tasks that require cross-session preference
memory, grounded in a product pool of over 1.2 million real-world items. Our analysis further identifies two major sources of failure on this
benchmark: cascading errors caused by preference
hallucination, and insufficient verification of product attributes against user requirements. To address these failure modes, we design annotation-free, tool-wise rewards that provide process supervision for each tool call, alleviating reward sparsity in long-horizon tasks. Experimental results demonstrate that even state-of-the-art models such as GPT-5 achieve success rates below 70%, highlighting the difficulty of our
benchmark. Notably, our fine-tuned lightweight 4B model consistently outperforms strong baselines in both preference capture and task performance, suggesting the effectiveness of our reward design.
Read the full abstract ▾
All authors Zijian Yu, Kejun Xiao, Huaipeng Zhao, Tao Luo, Xiaoyi Zeng
Computation and Language
ContiGuard: A Framework for Continual Toxicity Detection Against Evolving Evasive Perturbations
Hankun Kang, Xin Miao, Jianhao Chen et al. (+5 more)
Toxicity detection mitigates the dissemination of toxic content (e.g., hateful comments, posts, and messages within online social actions) to safeguard a healthy online social environment. However, malicious users persistently develop evasive perturbations to disguise toxic content and evade detectors. Traditional detectors or methods are static over time and are inadequate in addressing these evolving evasion tactics. Thus, continual learning emerges as a logical approach to dynamically update detection ability against evolving perturbations. Nevertheless, disparities across perturbations hinder the detector's continual learning on perturbed text. More importantly, perturbation-induced noises distort semantics to degrade comprehension and also impair critical feature learning to render detection sensitive to perturbations. These amplify the challenge of continual learning against evolving perturbations. In this work, we present ContiGuard, the first framework tailored for continual learning of the detector on time-evolving perturbed text (termed continual toxicity detection) to enable the detector to continually update capability and maintain sustained resilience against evolving perturbations. Specifically, to boost the comprehension, we present an LLM-powered semantic enriching strategy, where we dynamically incorporate possible meaning and toxicity-related clues excavated by LLM into the perturbed text to improve the comprehension. To mitigate non-critical features and amplify critical ones, we propose a discriminability-driven feature learning strategy, where we strengthen discriminative features while suppressing the less-discriminative ones to shape a robust classification boundary for detection...
Read the full abstract ▾
All authors Hankun Kang, Xin Miao, Jianhao Chen, Jintao Wen, Mayi Xu, Weiyu Zhang, Wenpeng Lu, Tieyun Qian
Computation and Language
The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatments
Elmira Salari, Maria Claudia Nunes Delfino, Hazem Amamou et al. (+4 more)
This paper studies the impact of retrieved ideological texts on the outputs of large
language models (LLMs). While interest in understanding ideology in LLMs has recently increased, little
attention has been given to this issue in the context of
Retrieval-Augmented Generation (RAG). To fill this gap, we design an external knowledge source based on ideological loaded texts about COVID-19 treatments. Our corpus is based on 1,117 academic articles representing discourses about controversial and endorsed treatments for the disease. We propose a corpus linguistics framework, based on Lexical Multidimensional Analysis (LMDA), to identify the ideologies within the corpus. LLMs are tasked to answer questions derived from three identified ideological dimensions, and two types of contextual
prompts are adopted: the first comprises the user question and ideological texts; and the second contains the question, ideological texts, and LMDA descriptions. Ideological
alignment between reference ideological texts and LLMs' responses is assessed using cosine similarity for lexical and semantic representations. Results demonstrate that LLMs' responses based on ideological retrieved texts are more aligned with the ideology encountered in the external knowledge, with the enhanced
prompt further influencing LLMs' outputs. Our findings highlight the importance of identifying ideological discourses within the RAG framework in order to mitigate not just unintended ideological bias, but also the risks of malicious manipulation of such models.
Read the full abstract ▾
All authors Elmira Salari, Maria Claudia Nunes Delfino, Hazem Amamou, José Victor de Souza, Shruti Kshirsagar, Alan Davoust, Anderson Avila
Computation and Language · Cross-listed from cs.SD
VorTEX: Various overlap ratio for Target speech EXtraction
Ro-hoon Oh, Jihwan Seol and Bugeun Kim
Target
speech extraction (TSE) aims to recover a target speaker's voice from a mixture. While recent text-prompted approaches have shown promise, most approaches assume fully overlapped mixtures, limiting insight into behavior across realistic overlap ratios. We introduce
VorTEX (Various overlap ratio for Target
speech EXtraction), a text-prompted TSE architecture with a Decoupled Adaptive Multi-branch (DAM) Fusion block that separates primary extraction from auxiliary regularization pathways. To enable controlled analysis, we construct PORTE, a two-speaker
dataset spanning overlap ratios from 0% to 100%. We further propose Suppression Ratio on Energy (SuRE), a diagnostic metric that detects suppression behavior not captured by conventional measures. Experiments show that existing models exhibit suppression or residual interference under overlap, whereas
VorTEX achieves the highest separation fidelity across 20-100% overlap (e.g., 5.50 dB at 20% and 2.04 dB at 100%) while maintaining zero SuRE, indicating robust extraction without suppression-driven artifacts.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Universe Routing: Why Self-Evolving Agents Need Epistemic Control
Zhaohui Geoffrey Wang
A critical failure mode of current lifelong
agents is not lack of knowledge, but the inability to decide how to reason. When an
agent encounters "Is this coin fair?" it must recognize whether to invoke frequentist hypothesis testing or
Bayesian posterior inference - frameworks that are epistemologically incompatible. Mixing them produces not minor errors, but structural failures that propagate across decision chains. We formalize this as the universe routing problem: classifying questions into mutually exclusive belief spaces before invoking specialized solvers. Our key findings challenge conventional assumptions: (1) hard routing to heterogeneous solvers matches soft MoE accuracy while being 7x faster because epistemically incompatible frameworks cannot be meaningfully averaged; (2) a 465M-parameter router achieves a 2.3x smaller generalization gap than keyword-matching baselines, indicating semantic rather than surface-level
reasoning; (3) when expanding to new belief spaces, rehearsal-based continual learning achieves zero forgetting, outperforming EWC by 75 percentage points, suggesting that modular epistemic architectures are fundamentally more amenable to lifelong learning than regularization-based approaches. These results point toward a broader architectural principle: reliable self-evolving
agents may require an explicit epistemic control layer that governs
reasoning framework selection.
Read the full abstract ▾
Computation and Language
Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA
Renhao Pei, Siyao Peng, Verena Blaschke et al. (+2 more)
Large
Language Models (LLMs) are becoming a common way for humans to seek knowledge, yet their coverage and reliability vary widely. Especially for local
language varieties, there are large asymmetries, e.g., information in local Wikipedia that is absent from the standard variant. However, little is known about how well LLMs perform under such information asymmetry, especially on closely related
languages. We manually construct a novel challenge question-answering (QA)
dataset that captures knowledge conveyed on a local Wikipedia page, which is absent from their higher-resource counterparts-covering Mandarin Chinese vs. Cantonese and German vs. Bavarian. Our experiments show that LLMs fail to answer questions about information only in local editions of Wikipedia. Providing context from lead sections substantially improves performance, with further gains possible via translation. Our topical, geographic annotations, and stratified evaluations reveal the usefulness of local Wikipedia editions as sources of both regional and global information. These findings raise critical questions about inclusivity and cultural coverage of LLMs.
Read the full abstract ▾
All authors Renhao Pei, Siyao Peng, Verena Blaschke, Robert Litschko, Barbara Plank
Computation and Language
Vietnamese Automatic Speech Recognition: A Revisit
Thi Vu, Linh The Nguyen and Dat Quoc Nguyen
Automatic
Speech Recognition (ASR) performance is heavily dependent on the availability of large-scale, high-quality
datasets. For low-resource
languages, existing open-source ASR
datasets often suffer from insufficient quality and inconsistent annotation, hindering the development of robust models. To address these challenges, we propose a novel and generalizable data aggregation and preprocessing pipeline designed to construct high-quality ASR
datasets from diverse, potentially noisy, open-source sources. Our pipeline incorporates rigorous processing steps to ensure data diversity, balance, and the inclusion of crucial features like word-level timestamps. We demonstrate the effectiveness of our methodology by applying it to Vietnamese, resulting in a unified, high-quality 500-hour
dataset that provides a foundation for training and evaluating state-of-the-art Vietnamese ASR systems. Our project page is available at https://github.com/qualcomm-ai-research/PhoASR.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Yuzhe Tang
Large
language models (LLMs) achieve impressive scores on standard
benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a
benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense
reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models – four from the Claude family and four from the GPT family – using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing
stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect
reasoning deficits rather than
language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense
reasoning.
Read the full abstract ▾
Computation and Language
Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark
Wei Shao, Lemao Liu, Yinqiao Li et al. (+3 more)
Current online translation services require sending user text to cloud servers, posing a risk of
privacy leakage when the text contains sensitive information. This risk hinders the application of online translation services in
privacy-sensitive scenarios. One way to mitigate this risk for online translation services is introducing
privacy protection mechanisms targeting the inference stage of translation models. However, compared to subfields of NLP like text classification and summarization, the machine translation research community has limited exploration of
privacy protection during the inference stage. There is no clearly defined
privacy protection task for the inference stage, dedicated evaluation
datasets and metrics, and reference
benchmark methods. The absence of these elements has seriously constrained researchers' in-depth exploration of this direction. To bridge this gap, this paper proposes a novel "
Privacy-Preserving Machine Translation" (PPMT) task, aiming to protect the private information in text during the model inference stage. For this task, we constructed three
benchmark test
datasets, designed corresponding evaluation metrics, and proposed a series of
benchmark methods as a starting point for this task. The definition of
privacy is complex and diverse. Considering that named entities often contain a large amount of personal
privacy and commercial secrets, we have focused our research on protecting only the named entity's
privacy in the text. We expect this research work will provide a new perspective and a solid foundation for the
privacy protection problem in machine translation.
Read the full abstract ▾
All authors Wei Shao, Lemao Liu, Yinqiao Li, Guoping Huang, Shuming Shi, Linqi Song
Computation and Language
Learning Constituent Headedness
Zeyao Qi, Yige Chen, KyungTae Lim et al. (+2 more)
Headedness is widely used as an organizing device in syntactic analysis, yet constituency treebanks rarely encode it explicitly and most processing pipelines recover it procedurally via percolation rules. We treat this notion of constituent headedness as an explicit representational layer and learn it as a supervised prediction task over aligned constituency and dependency annotations, inducing supervision by defining each constituent head as the dependency span head. On aligned English and Chinese data, the resulting models achieve near-ceiling intrinsic accuracy and substantially outperform Collins-style rule-based percolation. Predicted heads yield comparable parsing accuracy under head-driven binarization, consistent with the induced binary training targets being largely equivalent across head choices, while increasing the fidelity of deterministic constituency-to-dependency conversion and transferring across resources and
languages under simple label-mapping interfaces.
Read the full abstract ▾
All authors Zeyao Qi, Yige Chen, KyungTae Lim, Haihua Pan, Jungyeul Park