Computation and Language
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
Manh Luong, Tamas Abraham, Junae Kim et al. (+6 more)
Existing
multimodal safety
benchmarks focus solely on visual inputs and cannot assess Omni Large
Language Models (LLMs) that process
vision, audio, and text. We introduce MCBench, a
benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of
reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal
reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for
multimodal safety.
Read the full abstract ▾
All authors Manh Luong, Tamas Abraham, Junae Kim, Amar Kaur, Rollin Omari, Gholamreza Haffari, Trang Vu, Lizhen Qu, Dinh Phung
Computation and Language
PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis
Lucas Tamic, Ilan Jaffeux-Cheniout and Xavier Marjou
While large
language models (LLMs) show strong performance in natural
language understanding and generation, their evaluation and adaptation to domain-specific constraints in telecommunications customer support remain limited. In addition, data sovereignty, regulatory constraints, and the handling of sensitive customer and network information complicate the use of externally hosted foundation models in this domain. We present a systematic study of parameter-efficient fine-tuning (PEFT) using Low-Rank Adaptation (LoRA) applied to Qwen2.5-3B to build a domain-specific conversational assistant. We introduce a combinatorial synthetic data generation approach based on a glossary of 52 industry-specific terms, producing approximately 30,000 training examples across 1,560 distinct problem scenarios via a generative pipeline powered by Gemini 2.0 Flash. We evaluate 16 LoRA configurations by varying hyperparameters and target modules. Our evaluation extends beyond standard metrics by incorporating energy consumption analysis and qualitative assessment using an LLM-as-a-judge framework with GPT-5.2 and Claude 4.5 Sonnet. Results show a clear divergence between quantitative and qualitative performance: models achieving the lowest validation loss do not necessarily obtain the best human-aligned rankings. The best validation loss (0.5024) ranks only 6th-7th in qualitative evaluation, while the worst loss (0.6807) ranks first according to both judges. This work contributes (1) a combinatorial method for synthetic
dataset construction, (2) insights into the impact of target module selection for LoRA injection, (3) evidence that validation loss alone is insufficient for selecting fine-tuning configurations in conversational AI, and (4) an energy-performance trade-off analysis for sustainable LLM deployment.
Read the full abstract ▾
Computation and Language
Generic Triple-Latent Compression with Gated Associative Retrieval
Liu Xiao
We study generic triple-latent sequence models that maintain a running
token state and compressed pair-
memory pathway to capture higher-order
token interactions without
benchmark-specific parsing. The triple-latent family improves a small
Transformer baseline on byte-level WikiText-2 and on a tokenizer-based MiniMind
language-model
benchmark, while a recall-focused gated key-value
retrieval extension improves associative recall but remains seed-sensitive and much slower in the current reference implementation.
Read the full abstract ▾
Computation and Language
Improving Heart-Focused Medical Question Answering in LLMs via Variance-Aware Rubric Rewards with GRPO
Arash Ahmadi, Parisa Masnadi Khiabani, Sarah Sharif et al. (+4 more)
Large
language models can support medical question answering, but
privacy, cost, and reliability still limit their clinical use in focused domains. Can rubric-based reinforcement learning bring a locally deployable model close to much larger systems? We study Group Relative Policy Optimization (GRPO) for heart-focused medical question answering with rubric supervision from RaR-Medicine. Rather than rely on binary rubric aggregation or a single overall score, our variance-aware reward framework builds continuous rewards from criterion-level outcomes. It preserves partial credit and reduces reward sparsity. We adapt a Qwen3-14B policy with low-rank adapters and quantized weights. On a held-out heart-related HealthBench subset, the best GRPO variant improves accuracy from 0.362 to 0.502 and F1 from 0.532 to 0.668 over the base model, close to GPT-OSS-120B. An independent clinician rated most responses clinically acceptable (61%) and patient ready (63%), both above the automated accuracy; fewer than 2% were unsafe. Taken together, the results support rubric-based reinforcement learning as a practical route for focused medical
language models.
Read the full abstract ▾
All authors Arash Ahmadi, Parisa Masnadi Khiabani, Sarah Sharif, Charles Nicholson, David Ebert, Aswathy K. Vaikom House, Mike Banad
Computation and Language
EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation
Shuyu Zhang, Lingfeng Pan, Qicheng Wang et al. (+6 more)
Speculative decoding accelerates Large
Language Model inference through draft-then-verify generation, yet lightweight draft models face coupled efficiency and quality limitations: large-vocabulary output projection is costly, while limited draft capacity and static parameters reduce acceptance under specialized or shifting inputs. Vocabulary
pruning lowers projection cost, but static variants miss locally important long-tail
tokens, while dynamic variants remain sensitive to preset selection policies and budgets. Moreover, limited draft capacity can leave the draft distribution misaligned even when the target
token is covered. Online
alignment improves draft quality, but full-parameter updates introduce substantial
memory and latency overhead. We introduce EvoSpec, which jointly adapts the active vocabulary and lightweight draft parameters from verification feedback. EvoSpec asynchronously retrieves semantic and statistical
token neighbors and performs curriculum-weighted online LoRA
alignment while preserving exact target-model verification. On Qwen3-8B/EAGLE-2, EvoSpec reaches a $2.18\times$ speedup over vanilla decoding and a $1.20\times$ gain over EAGLE-2, while improving specialized-domain coverage and using $27\%$ less auxiliary GPU adaptation
memory than full-parameter online adaptation.
Read the full abstract ▾
All authors Shuyu Zhang, Lingfeng Pan, Qicheng Wang, Yaqi Shi, Yueyang Tan, Ruyu Yan, Jiaqi Chen, Lixing Du, Lu Wang
Computation and Language · Cross-listed from cs.LG
Data-Driven Variational Basis Learning Beyond Neural Networks: A Non-Neural Framework for Adaptive Basis Discovery
Andrew Kiruluta
Classical representation systems such as
Fourier series, wavelets, and fixed dictionaries provide analytically tractable basis expansions, but they are not intrinsically adapted to the empirical structure of modern high-dimensional data. Neural networks overcome this limitation by learning features from data, yet they do so through layered nonlinear parameterizations that often sacrifice interpretability, explicit control over basis structure, and mathematical transparency. In this manuscript we develop a non-neural alternative that learns basis functions directly from data through variational optimization. The proposed framework, termed Data Driven Variational Basis Learning (DVBL), treats basis atoms as primary optimization variables and learns them jointly with sample-specific coefficients and, when appropriate, a latent linear
evolution operator. This yields a data-adaptive basis expansion that remains explicit, interpretable, and amenable to rigorous analysis. We formulate the model, establish existence of minimizers, prove blockwise descent properties for an alternating minimization
algorithm, give conditions for coefficient recovery and basis identifiability, and show how
manifold and dynamical regularization can be integrated without invoking neural architectures. We also discuss the conceptual novelty of the framework relative to classical dictionary learning, spectral methods, Koopman operator methods, and deep representation learning.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
ICLAD: In-Context Learning with Comparison-Guidance for Audio Deepfake Detection
Benjamin Chou, Yi Zhu and Surya Koppisetti
Audio deepfakes pose a significant security threat, yet current state-of-the-art (SOTA) detection systems do not generalize well to realistic in-the-wild deepfakes. We introduce a novel In-Context Learning paradigm with comparison-guidance for Audio Deepfake detection (ICLAD). The framework enables the use of audio
language models (ALMs) for training-free generalization to unseen deepfakes and provides textual rationales on the detection outcome. At the core of ICLAD is a pairwise comparative
reasoning strategy that guides the ALM to discover and filter
hallucinations and deepfake-irrelevant acoustic attributes. The ALM works alongside a specialized deepfake detector, whereby a routing mechanism feeds out-of-distribution samples to the ALM. On in-the-wild
datasets, ICLAD improves macro F1 over the specialized detector, with up to $2\times$ relative improvement. Further analysis demonstrates the flexibility of ICLAD and its potential for deployment on recent open-source ALMs.
Read the full abstract ▾
Computation and Language
Evaluating Adaptive Personalization of Educational Readings with Simulated Learners
Ryan T. Woo, Anmol Rao, Aryan Keluskar et al. (+1 more)
We present a framework for evaluating adaptive personalization of educational reading materials with theory-grounded simulated learners. The system builds a learning-objective and knowledge-component ontology from open textbooks, curates it in a browser-based Ontology Atlas, labels textbook chunks with ontology entities, and generates aligned reading-assessment pairs. Simulated readers learn from passages through a Construction-Integration-inspired
memory model with DIME-style reader factors, KREC-style misconception revision, and an open New Dale-Chall readability signal. Answers are produced by score-based option selection over the learner's explicit
memory state, while BKT drives adaptation. Across three sampled subject ontologies and matched cohorts of 50 simulated learners per condition, adaptive reading significantly improved outcomes in computer science, yielded smaller positive but inconclusive gains in inorganic chemistry, and was neutral to slightly negative in general biology.
Read the full abstract ▾
All authors Ryan T. Woo, Anmol Rao, Aryan Keluskar, Yinong Chen
Computation and Language · Cross-listed from cs.AI
CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction
Jianyou Wang, Youze Zheng, Longtian Bao et al. (+11 more)
Scientists have long sought to accurately predict outcomes of real-world events before they happen. Can AI systems do so more reliably? We study this question through clinical trial outcome prediction, a high-stakes open challenge even for domain experts. We introduce CT Open, an open-access, live platform that will run four challenge every year. Anyone can submit predictions for each challenge. CT Open evaluates those submissions on trials whose outcomes were not yet public at the time of submission but were made public afterwards. Determining if a trial's outcome is public on the internet before a certain date is surprisingly difficult. Outcomes posted on official registries may lag behind by years, while the first mention may appear in obscure articles. To address this, we propose a novel, fully automated decontamination pipeline that uses iterative LLM-powered web search to identify the earliest mention of trial outcomes. We validate the pipeline's quality and accuracy by human expert's annotations. Since CT Open's pipeline ensures that every evaluated trial had no publicly reported outcome when the prediction was made, it allows participants to use any methodology and any data source. In this paper, we release a training set and two time-stamped test
benchmarks, Winter 2025 and Summer 2025. We believe CT Open can serve as a central hub for advancing AI research on
forecasting real-world outcomes before they occur, while also informing biomedical research and improving clinical trial design. CT Open Platform is hosted at $\href{https://ct-open.net/}{https://ct-open.net/}$
Read the full abstract ▾
All authors Jianyou Wang, Youze Zheng, Longtian Bao, Hanyuan Zhang, Qirui Zheng, Yuhan Chen, Yang Zhang, Matthew Feng, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Ramamohan Paturi, Umber Dube, Leon Bergen
Computation and Language
Detecting Alarming Student Verbal Responses using Text and Audio Classifier
Christopher Ormerod and Gitit Kehat
This paper addresses a critical safety gap in the use Automated Verbal Response Scoring (AVRS). We present a novel hybrid framework for troubled student detection that combines a text classifier, trained to detect responses based on their content, and an audio classifier, trained to detect responses using prosodic markers. This approach overcomes key limitations of traditional AVRS systems by considering both content and prosody of responses, achieving enhanced performance in identifying potentially concerning responses. This system can expedite the review process by humans, which can be life-saving particularly when timely intervention may be crucial.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
Bhaskar Gurram
Automated evaluation of tool-using large
language model (LLM)
agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation. We present AgentProp-Bench, a diagnostic
benchmark of 14,750 execution traces from thirteen LLM
agents (nine proprietary, four open-weight) across four domains, and use it to audit three questions. First, substring-heuristic judging of
agent outputs agrees with human annotation only at chance level (Cohen's kappa = 0.049 against each of two annotators), while a three-LLM ensemble reaches moderate agreement (kappa = 0.432) and a single GPT-4o-mini judge is in fact the strongest (kappa = 0.567); dual-annotator agreement is almost perfect (kappa = 0.835). Second, under validated judging a parameter-level error propagates to a wrong final answer with human-calibrated probability approximately 0.62, replicated across proprietary and open-weight models, and a model's ability to reject corrupted inputs and to recover from them are statistically indepen1). Third, severalagents fabricate tool executions – asserting tool-derived renever obtained (up toilure invisible toend-to-end scores, and a lightweight runtime interceptor reduhallucination on everto 24 percentagepoints) at a tunable operating point that is net-positive on open-weight models. Ini-2.0-Flash, whichrarely emits an injectable tool call. All code, data, traces, and human labels are relehttps://github.com/bhaskargurram-ai/agenthallu-bench
Read the full abstract ▾
Computation and Language
The impact of postediting on AI generative translation in Yemeni context: Translating literary prose by ChatGPT
Nasim Al-wagieh and Mohammed Q. Shormani
This study examines the role of artificial intelligence in translation, focusing on ChatGPT, specifically ChatGPT-4, and the extent to which human postediting is required in literary translation. A mixed-method approach was adopted, involving 30 professional translators who evaluated and postedited AI-generated translations of selected Arabic and English literary texts. The results show that although AI improves translation speed and accessibility, it remains limited in handling cultural, stylistic, and figurative aspects of
language. Participants generally confirmed the necessity of human postediting, particularly in novels and drama. The findings indicate that emerging human-machine collaboration model rather than replacement of human translators. The study concludes that AI should be used as a supportive tool, while human expertise remains essential for ensuring translation quality and cultural appropriateness.
Read the full abstract ▾
Computation and Language
No-Worse Context-Aware Decoding: Preventing Neutral Regression in Context-Conditioned Generation
Yufei Tao and Ameeta Agrawal
Large
language models (LLMs) can answer questions and summarize documents when conditioned on external contexts (e.g., retrieved evidence), yet context use remains unreliable: models may overwrite an already-correct output (neutral regression) even when the context is non-informative. We formalize neutral regression as a do-no-harm requirement and quantify it by measuring accuracy drops on baseline-correct items under answer-consistent contexts. We propose No-Worse Context-Aware Decoding (NWCAD), a decode-time adapter built on a two-stream setup with a two-stage gate: it backs off to no-context decoding when the context is non-informative, and otherwise uses context-conditioned decoding with a CAD-style fallback under uncertainty. We evaluate NWCAD on
benchmarks that separate do-no-harm reliability from context utilization (accuracy gains on genuinely helpful contexts). NWCAD prevents neutral regression on baseline-correct items while preserving strong context-driven accuracy on helpful contexts.
Read the full abstract ▾
Computation and Language
CBRS: Cognitive Blood Request System with Bilingual Dataset and Dual-Layer Filtering for Multi-Platform Social Streams
Anik Saha, Mst. Fahmida Sultana Naznin, Zia Ul Hassan Abdullah et al. (+3 more)
Urgent blood donation seeking posts and messages on social media often go unnoticed due to the overwhelming volume of daily communications. Traditional app-based systems, reliant on manual input, struggle to reach users in low-resource settings, delaying critical responses. To address this, we introduce the Cognitive Blood Request System (CBRS), a multi-platform framework that efficiently filters and parses blood donation requests from social media streams using a cost-efficient dual-layered architecture. To do so, we curate a novel
dataset of 11K parsed blood donation request messages in Bengali, English, and transliterated Bengali, capturing the linguistic diversity of real social media communications. The inclusion of
adversarial negatives further enhances the robustness of our model. CBRS achieves an impressive 99% accuracy and precision in filtering, surpassing
benchmark methods. In the parsing task, our LoRA finetuned Llama-3.2-3B model achieves 92% zero-shot accuracy, surpassing the base model by 41.54% and exceeding the few-shot performance of GPT-4o-mini, Gemini-2.0-Flash, and other LLMs, while resulting in a 35X reduction in input
token usage. This work lays a robust foundation for scalable, inclusive information extraction in time-sensitive, object-focused tasks. Our code,
dataset, and trained models are publicly available at [https://github.com/aaniksahaa/CBRS](https://github.com/aaniksahaa/CBRS).
Read the full abstract ▾
All authors Anik Saha, Mst. Fahmida Sultana Naznin, Zia Ul Hassan Abdullah, Anisa Binte Asad, K. G. Subarno Bithi, A. B. M. Alim Al Islam
Computation and Language
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
Maitrey Mehta, Nishant Subramani, Zhichao Xu et al. (+2 more)
All
languages are equal; when it comes to tokenization, some are more equal than others.
Tokens are the hidden currency that dictates the cost and latency of contemporary LLM access. However, many
languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of
tokens to encode the same information in many
languages as they do for English. We observe that this issue, known as
token overfragmentation, persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target
language items missing from the model's vocabulary. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output
embeddings? We comprehensively study and advance the new research direction of interpretability-based vocabulary expansion along these two core decisions, making four key contributions: First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-
token efficiency tradeoff. Second, we strengthen the case for interpretability-based
embedding initialization by showing large gains (~20 pts) over baseline initialization methods for several
languages written in non-Latin scripts. Third, we identify the phenomenon of "subword detokenization," where models progressively merge fragmented subword
tokens into larger subwords across layers. Fourth, we propose FragMend, a method that leverages this phenomenon to push the efficiency ceiling of interpretability-based expansion, and validate its effectiveness.
Read the full abstract ▾
All authors Maitrey Mehta, Nishant Subramani, Zhichao Xu, Ashim Gupta, Vivek Srikumar
Computation and Language
IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language
Christina Chance, Rebecca Pattichis, Arjun Subramonian et al. (+4 more)
Reclaimed slur usage is a common and meaningful practice online for many marginalized communities. It serves as a source of solidarity, identity, and shared experience. However, contemporary automated and AI-based moderation tools for online content largely fail to distinguish between reclaimed and hateful uses of slurs, resulting in the suppression of marginalized voices. In this work, we use quantitative and qualitative methods to examine the attitudes of social media users in LGBTQIA+, Black, and women communities around reclaimed slurs targeting our focus groups including the f-word, n-word, and b-word. With social media users from these communities, we collect and analyze an annotated online slur usage corpus. The corpus includes annotators' perceptions of whether an online text containing a slur should be flagged as hate
speech, as well as contextual features of the slur usage. Across all communities and annotation questions, we observe low inter-annotator agreement, indicating substantial disagreement among in-group annotators. This is compounded by the fact that, absent clear contextual signals of identity and intent, even in-group members may disagree on how to interpret reclaimed slur usage online. Semi-structured interviews with annotators suggest that differences in lived experience and personal history contribute to this variation as well. We find poor
alignment between annotator judgments and automated hate
speech assessments produced by Perspective API. We further observe that certain features of a text such as whether the slur usage was derogatory and if the slur was targeted at oneself are more associated with whether annotators report the text as hate
speech. Together, these findings highlight the inherent subjectivity and contextual nature of how marginalized communities interpret slurs online.
Read the full abstract ▾
All authors Christina Chance, Rebecca Pattichis, Arjun Subramonian, James He, Shruti Narayanan, Saadia Gabriel, Kai-Wei Chang
Computation and Language
Migrant Voices, Local News: Insights on Bridging Community Needs with Media Content
David Alonso del Barrio, Paula Dolores Rescala, Victor Bros et al. (+1 more)
Research shows news consumption differs across demographics, yet little is known about non-mainstream audiences, especially in relation to local media. Our study addresses this gap by examining how French-speaking migrants in a mid-size European city engage with local news, and whether their needs are reflected in coverage. Eight community members participated in focus groups, whose insights guided the selection of natural
language processing methods (topic modeling, information
retrieval, sentiment analysis, and readability) applied to over 2000 hyper-local news articles. Results showed that while articles frequently covered local events, gaps remained in topics important to participants. Sentiment analysis revealed a generally positive tone, and readability measures indicated an intermediate-advanced French level, raising questions about accessibility for integration. Our work contributes to bridging the gap between local news platforms' content and diverse readers' needs, and could inform local media organizations about opportunities to expand their current news story coverage to appeal to more diverse audiences.
Read the full abstract ▾
All authors David Alonso del Barrio, Paula Dolores Rescala, Victor Bros, Daniel Gatica-Perez
Computation and Language
AdaExplore: Failure-Driven Adaptation and Diversity-Preserving Search for Efficient Kernel Generation
Weihua Du, Jingming Zhuo, Yixin Dong et al. (+9 more)
Recent large
language model (LLM)
agents have shown promise in using execution feedback for test-time adaptation. However, robust self-improvement remains far from solved: most approaches still treat each problem instance independently, without accumulating reusable knowledge. This limitation is particularly pronounced in domain-specific
languages such as Triton, which are underrepresented in LLM pretraining data. Their strict constraints and non-linear optimization landscape further make naive generation and local refinement unreliable. We propose AdaExplore, an
agent framework that enables self-improvement via accumulated execution feedback for performance-critical kernel code generation through two complementary stages: failure-driven adaptation and diversity-preserving search, jointly improving correctness and optimization performance without additional fine-tuning or external knowledge. In the adaptation stage, the
agent synthesizes tasks and converts recurring failures into a reusable
memory of validity rules, helping subsequent generations remain within the feasible set. In the search stage, the
agent organizes candidate kernels as a tree and alternates between small local refinements and larger structural regeneration, allowing it to explore the optimization landscape beyond local optima. Experiments on kernel runtime optimization
benchmarks validate these gains: AdaExplore achieves 3.11x and 1.62x speedups on KernelBench Level-2 and Level-3, respectively, within 100 steps, and continues to improve with additional computation.
Read the full abstract ▾
All authors Weihua Du, Jingming Zhuo, Yixin Dong, Andre Wang He, Weiwei Sun, Zeyu Zheng, Manupa Karunaratne, Ivan Fox, Tim Dettmers, Tianqi Chen, Yiming Yang, Sean Welleck
Computation and Language
Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning
Livia Qian and Gabriel Skantze
Backchannels (e.g., `yeah', `mhm', and `right') are short, non-interruptive feedback signals whose lexical form and prosody jointly convey pragmatic meaning. While prior computational research has largely focused on predicting backchannel timing, the relationship between lexico-prosodic form and meaning remains underexplored. We propose a two-stage framework: first, fine-tuning large
language models on dialogue transcripts to derive rich contextual representations; and second, learning a joint
embedding space for dialogue contexts and backchannel realizations. We evaluate
alignment with human perception via triadic similarity judgments (prosodic and cross-lexical) and a context-backchannel suitability task. Our results demonstrate that the learned projections substantially improve context-backchannel
retrieval compared to previous methods. In addition, they reveal that backchannel form is highly sensitive to extended conversational context and that the learned
embeddings align more closely with human judgments than raw WavLM features.
Read the full abstract ▾
Computation and Language
Spotlights and Blindspots: Evaluating Machine-Generated Text Detection
Kevin Stowe and Kailash Patil
With the rise of generative
language models, machine-generated text detection has become a critical challenge. A wide variety of models is available, but inconsistent
datasets, evaluation metrics, and assessment strategies obscure comparisons of model effectiveness. To address this, we evaluate 15 different detection models from six distinct systems, as well as seven trained models, across seven English-
language textual test sets and three creative human-written
datasets. We provide an empirical analysis of model performance, the influence of training and evaluation data, and the impact of key metrics. We find that no single system excels in all areas and nearly all are effective for certain tasks, and the representation of model performance is critically linked to
dataset and metric choices. We find high variance in model ranks based on
datasets and metrics, and overall poor performance on novel human-written texts in high-risk domains. Across
datasets and metrics, we find that methodological choices that are often assumed or overlooked are essential for clearly and accurately reflecting model performance.
Read the full abstract ▾
Computation and Language
Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
Yang Liu, Hongming Li, Melissa Xiaohui Qin et al. (+2 more)
We present SemanticQA, an evaluation suite designed to assess
language models (LMs) in semantic phrase processing tasks. The
benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic
reasoning, highlighting differences in
reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.
Read the full abstract ▾
All authors Yang Liu, Hongming Li, Melissa Xiaohui Qin, Qiankun Liu, Chao Huang
Computation and Language · Cross-listed from cs.IR
On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability
Yongkang Li, Panagiotis Eustratiadis, Yixing Fan et al. (+1 more)
Decoder-only large
language models (LLMs) are increasingly replacing BERT-style architectures as the backbone for dense
retrieval, achieving substantial performance gains and broad adoption. However, the robustness of these LLM-based retrievers remains underexplored. In this paper, we present the first systematic study of the robustness of state-of-the-art open-source LLM-based dense retrievers from two complementary perspectives: generalizability and stability. For generalizability, we evaluate
retrieval effectiveness across four
benchmarks spanning 30
datasets, using linear mixed-effects models to estimate marginal mean performance and disentangle intrinsic model capability from
dataset heterogeneity. Our analysis reveals that while instruction-tuned models generally excel, those optimized for complex
reasoning often suffer a “specialization tax,” exhibiting limited generalizability in broader contexts. For stability, we assess model resilience against both unintentional query variations~(e.g., paraphrasing, typos) and malicious
adversarial attacks~(e.g., corpus poisoning). We find that LLM-based retrievers show improved robustness against typos and corpus poisoning compared to encoder-only baselines, yet remain vulnerable to semantic perturbations like synonymizing. Further analysis shows that
embedding geometry (e.g., angular uniformity) provides predictive signals for lexical stability and suggests that scaling model size generally improves robustness. These findings inform future robustness-aware retriever design and principled benchmarking. Our code is publicly available at https://github.com/liyongkang123/Robust_LLM_Retriever_Eval.
Read the full abstract ▾
All authors Yongkang Li, Panagiotis Eustratiadis, Yixing Fan, Evangelos Kanoulas
Computation and Language · Cross-listed from cs.LG
S-GRPO: Unified Post-Training for Large Vision-Language Models
Yuming Yan, Kai Tang, Sihong Chen et al. (+4 more)
Current post-training methodologies for adapting Large
Vision-
Language Models (LVLMs) generally fall into two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Despite their prevalence, both approaches suffer from inefficiencies when applied in isolation. SFT forces the model's generation along a single expert trajectory, often inducing catastrophic forgetting of general
multimodal capabilities due to distributional shifts. Conversely, RL explores multiple generated trajectories but frequently encounters optimization collapse - a cold-start problem where an unaligned model fails to spontaneously sample any domain-valid trajectories in sparse-reward visual tasks. In this paper, we propose Supervised Group Relative Policy Optimization (S-GRPO), a unified post-training framework that integrates the guidance of imitation learning into the multi-trajectory exploration of preference optimization. Tailored for direct-generation visual tasks, S-GRPO introduces Conditional Ground-Truth Trajectory Injection (CGI). When a binary verifier detects a complete exploratory failure within a sampled group of trajectories, CGI injects the verified ground-truth trajectory into the candidate pool. By assigning a deterministic maximal reward to this injected anchor, S-GRPO enforces a positive signal within the group-relative advantage estimation. This mechanism reformulates the supervised learning objective as a high-advantage component of the policy gradient, compelling the model to dynamically balance between exploiting the expert trajectory and exploring novel visual concepts. Theoretical analysis and empirical results demonstrate that S-GRPO gracefully bridges the gap between SFT and RL, drastically accelerates convergence, and achieves superior domain adaptation while preserving the base model's general-purpose capabilities.
Read the full abstract ▾
All authors Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu
Computation and Language · Cross-listed from cs.CR
A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle
Zehao Lin, Xixuan Hao, Renyu Fu et al. (+5 more)
The emergence of writable, cross-session persistent
memory in LLM
agents introduces a qualitatively different threat landscape from conventional input-centric security concerns, characterized by three properties: persistence, statefulness, and propagation. To systematically characterize this landscape, we propose a
Memory Lifecycle Framework that organizes attacks, defenses, and their cross-phase dependencies along two axes: six lifecycle phases (Write, Store, Retrieve, Execute, Share & Propagate, Forget & Rollback) and four security objectives (Integrity, Confidentiality, Availability, Governance). This analysis in turn exposes the need for explicit security requirements at the system level, motivating Verifiable
Memory Governance (VMG), a framework of five architectural primitives for specifying and evaluating auditable, recoverable control over
memory state within declared system and threat-model boundaries. Our analysis motivates integrating provenance, versioning, and policy-aware retention with
retrieval and execution defenses to support auditable prevention, containment, and recovery across the lifecycle.
Read the full abstract ▾
All authors Zehao Lin, Xixuan Hao, Renyu Fu, Shaobo Cui, Kai Chen, Chunyu Li, Zhiyu Li, Feiyu Xiong
Computation and Language · Cross-listed from cs.CR
TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contexts
Hua-Rong Chu, Kuan-Chun Wang and Yao-Te Huang
Safety guardrails have become an active area of research in AI safety, aimed at ensuring the appropriate behavior of large
language models (LLMs). However, existing research lacks consideration of nuances across linguistic and cultural contexts, resulting in a gap between reported performance and in-the-wild effectiveness. To address this issue, this paper proposes an approach to optimize guardrail models for a designated linguistic context by leveraging a curated
dataset tailored to local linguistic characteristics, targeting the Taiwan linguistic context as a representative example of localized deployment challenges. The proposed approach yields TWGuard, a linguistic context-optimized guardrail model that achieves a huge gain (+0.289 in F1) compared to the foundation model and significantly outperforms the strongest baseline in practical use (-0.037 in false positive rate, a 94.9% reduction). Together, this work lays a foundation for regional communities to establish AI safety standards grounded in their own linguistic contexts, rather than accepting boundaries imposed by dominant
languages. The inadequacy of the latter is reconfirmed by our findings.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Learning to Reason with Insight for Informal Theorem Proving
Yunhe Li, Hao Shi, Bowen Deng et al. (+8 more)
Although most of the automated
theorem-proving approaches depend on formal proof systems, informal
theorem proving can align better with large
language models' (LLMs) strength in natural
language processing. In this work, we identify a primary bottleneck in informal
theorem proving as a lack of insight, namely the difficulty of recognizing the core techniques required to solve complex problems. To address this, we propose $\texttt{DeepInsight}$, a unified training framework designed to cultivate this essential
reasoning skill and enable LLMs to perform insightful
reasoning. Our framework consists of three components: (1) $\texttt{DeepInsightTheorem}$, a hierarchical
dataset that structures informal proofs by explicitly extracting core techniques and proof sketches alongside the final proof; (2) a Progressive Multi-Stage SFT strategy that mimics the human learning process, teaching the model proof writing, planning, and insight identification; and (3) $\texttt{InsightPO}$, a policy optimization method that assigns structured rewards over this insight hierarchy. Our experiments on challenging mathematical
benchmarks demonstrate that this insight-aware generation strategy significantly outperforms baselines. These results demonstrate that teaching models to identify and apply core techniques can substantially improve their mathematical
reasoning.
Read the full abstract ▾
All authors Yunhe Li, Hao Shi, Bowen Deng, Wei Wang, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Siyang Gao, Chao Wang, Shuang Qiu, Linqi Song
Computation and Language
No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
Hitesh Mehta, Arjit Saxena, Garima Chhikara et al. (+1 more)
This paper explores the response of Large
Language Models (LLMs) to user
prompts with different degrees of politeness and impoliteness. The Politeness Theory by Brown and Levinson and the Impoliteness Framework by Culpeper form the basis of experiments conducted across three
languages (English, Hindi, Spanish), five models (Gemini-Pro, GPT-4o Mini, Claude 3.7 Sonnet, DeepSeek-Chat, and Llama 3), and three interaction histories between users (raw, polite, and impolite). Our sample consists of 22,500 pairs of
prompts and responses of various types, evaluated across five levels of politeness using an eight-factor assessment framework: coherence, clarity, depth, responsiveness, context retention, toxicity, conciseness, and readability. The findings show that model performance is highly influenced by tone, dialogue history, and
language. While polite
prompts enhance the average response quality by up to ~11% and impolite tones worsen it, these effects are neither consistent nor universal across
languages and models. English is best served by courteous or direct tones, Hindi by deferential and indirect tones, and Spanish by assertive tones. Among the models, Llama is the most tone-sensitive (11.5% range), whereas GPT is more robust to
adversarial tone. These results indicate that politeness is a quantifiable computational variable that affects LLM behaviour, though its impact is
language- and model-dependent rather than universal. To support reproducibility and future work, we additionally release PLUM (Politeness Levels in Utterances,
Multilingual), a publicly available corpus of 1,500 human-validated
prompts across three
languages and five politeness categories, and provide a formal supplementary analysis of six falsifiable hypotheses derived from politeness theory, empirically assessed against the
dataset.
Read the full abstract ▾
All authors Hitesh Mehta, Arjit Saxena, Garima Chhikara, Rohit Kumar
Computation and Language · Cross-listed from cs.CV
VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects
Xiangbo Gao, Sicong Jiang, Bangya Liu et al. (+12 more)
As AI-assisted
video creation becomes increasingly practical, instruction-guided
video editing has become essential for refining generated or captured footage to meet professional requirements. Yet the field still lacks both a large-scale human-annotated
dataset with complete editing examples and a standardized evaluator for comparing editing systems. Existing resources are limited by small scale, missing edited outputs, or the absence of human quality labels, while current evaluation often relies on expensive manual inspection or generic
vision-
language model judges that are not specialized for editing quality. We introduce VEFX-
Dataset, a human-annotated
dataset containing 5,049
video editing examples across 9 major editing categories and 32 subcategories, each labeled along three decoupled dimensions: Instruction Following, Rendering Quality, and Edit Exclusivity. Building on VEFX-
Dataset, we propose VEFX-Reward, a reward model designed specifically for
video editing quality assessment. VEFX-Reward jointly processes the source
video, the editing instruction, and the edited
video, and predicts per-dimension quality scores via ordinal regression. We further release VEFX-Bench, a
benchmark of 300 curated
video-
prompt pairs for standardized comparison of editing systems. Experiments show that VEFX-Reward aligns more strongly with human judgments than generic VLM judges and prior reward models on both standard IQA/VQA metrics and group-wise preference evaluation. Using VEFX-Reward as an evaluator, we
benchmark representative commercial and open-source
video editing systems, revealing a persistent gap between visual plausibility, instruction following, and edit locality in current models. Our project page is https://xiangbogaobarry.github.io/VEFX-Bench/.
Read the full abstract ▾
All authors Xiangbo Gao, Sicong Jiang, Bangya Liu, Xinghao Chen, Minglai Yang, Siyuan Yang, Mingyang Wu, Jiongze Yu, Qi Zheng, Haozhi Wang, Jiayi Zhang, Jie Yang, Zihan Wang, Qing Yin, Zhengzhong Tu