Computation and Language · Cross-listed from cs.LG
WebATLAS: An LLM Agent with Experience-Driven Memory and Action Simulation
Jiali Cheng, Anjishnu Kumar, Roshan Lal et al. (+7 more)
Large
Language Model (LLM) web
agents often struggle with long-horizon web
navigation and web task completion in new websites, producing inefficient action sequences unless fine-tuned on environment-specific data. We show that experience-driven
memory, combined with look-ahead action simulation, is sufficient for LLM
agents to adapt to unseen web environments by remembering past failures and predicting the consequences of future actions. We introduce WebATLAS (Actor-Critic Task-completion with Look-ahead Action Simulation), a
memory-augmented LLM web
agent that learns a lightweight internal model of the environment from interaction experience and performs hypothetical action rollouts before acting in the real world. WebATLAS builds a persistent cognitive map via curiosity-driven exploration, stores interaction outcomes as experience-based
memory, and evaluates candidate actions in cognitive space using a planner–simulator–critic loop. This enables the
agent to reuse past experience, avoid previously unsuccessful behaviors, and generate more efficient plans. We evaluate WebATLAS on the WebArena-Lite
benchmark for autonomous web
navigation and demonstrate a success rate of 63%, outperforming the previous state-of-the-art at 53.9%. Unlike previous systems, our modular architecture requires no website-specific LLM fine-tuning. Ablation studies confirm that experience-driven
memory, look-ahead action simulation, and hierarchical replanning play complementary roles in enabling robust, training-free web
agents.
Read the full abstract ▾
All authors Jiali Cheng, Anjishnu Kumar, Roshan Lal, Rishi Rajasekaran, Hani Ramezani, Omar Zia Khan, Oleg Rokhlenko, Sunny Chiu-Webster, Gang Hua, Hadi Amiri
Computation and Language · Cross-listed from cs.AI
Critical Insights into Leading Conversational AI Models
Urja Kohli, Aditi Singh and Arun Sharma
Big
Language Models (LLMs) are changing the way businesses use software, the way people live their lives and the way industries work. Companies like Google, High-Flyer, Anthropic, OpenAI and Meta are making better LLMs. So, it's crucial to look at how each model is different in terms of performance, moral behaviour and usability, as these differences are based on the different ideas that built them. This study compares five top LLMs: Google's Gemini, High-Flyer's DeepSeek, Anthropic's Claude, OpenAI's GPT models and Meta's LLaMA. It performs this by analysing three important factors: Performance and Accuracy, Ethics and Bias Mitigation and Usability and Integration. It was found that Claude has good moral
reasoning, Gemini is better at
multimodal capabilities and has strong ethical frameworks. DeepSeek is great at
reasoning based on facts, LLaMA is good for open applications and ChatGPT delivers balanced performance with a focus on usage. It was concluded that these models are different in terms of how well they work, how easy they are to use and how they treat people ethically, making it a point that each model should be utilised by the user in a way that makes the most of its strengths.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation
Shu Zhao, Tianyi Shen, Nilesh Ahuja et al. (+2 more)
Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of
Multimodal Large
Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static
retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on
retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering
datasets to MRAG training
datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95%
retrieval times.
Read the full abstract ▾
All authors Shu Zhao, Tianyi Shen, Nilesh Ahuja, Omesh Tickoo, Vijaykrishnan Narayanan
Computation and Language
SALSA: Single-pass Autoregressive LLM Structured Classification
Ruslan Berdichevsky, Shai Nahum-Gefen and Elad Ben Zaken
Despite their impressive generalization capabilities, instruction-tuned Large
Language Models often underperform on text classification
benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-
token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output
token, and
prompts are constructed to elicit a single-
token response. During inference, the model's output is projected only onto the logits of the relevant class
tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse
benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.
Read the full abstract ▾
Computation and Language
Rule-Based Explanations for Retrieval-Augmented LLM Systems
Joel Rorseth, Parke Godfrey, Lukasz Golab et al. (+2 more)
If-then rules are widely used to explain machine learning models; e.g., "if employed = no, then loan application = rejected." We present the first proposal to apply rules to explain the emerging class of large
language models (LLMs) with
retrieval-augmented generation (RAG). Since RAG enables LLM systems to incorporate retrieved information sources at inference time, rules linking the presence or absence of sources can explain output provenance; e.g., "if a Times Higher Education ranking article is retrieved, then the LLM ranks Oxford first." To generate such rules, a brute force approach would probe the LLM with all source combinations and check if the presence or absence of any sources leads to the same output. We propose optimizations to speed up rule generation, inspired by Apriori-like
pruning from frequent itemset mining but redefined within the scope of our novel problem. We conclude with qualitative and quantitative experiments demonstrating our solutions' value and efficiency.
Read the full abstract ▾
All authors Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, Jarek Szlichta
Computation and Language · Cross-listed from cs.CV
RoboSVG: A Unified Framework for Interactive SVG Generation with Multi-modal Guidance
Jiuniu Wang, Gongjie Zhang, Quanhao Qian et al. (+3 more)
Scalable Vector Graphics (SVGs) are fundamental to digital design and
robot control, encoding not only visual structure but also motion paths in interactive drawings. In this work, we introduce RoboSVG, a unified
multimodal framework for generating interactive SVGs guided by textual, visual, and numerical signals. Given an input query, the RoboSVG model first produces
multimodal guidance, then synthesizes candidate SVGs through dedicated generation modules, and finally refines them under numerical guidance to yield high-quality outputs. To support this framework, we construct RoboDraw, a large-scale
dataset of one million examples, each pairing an SVG generation condition (e.g., text, image, and partial SVG) with its corresponding ground-truth SVG code. RoboDraw
dataset enables systematic study of four tasks, including basic generation (Text-to-SVG, Image-to-SVG) and interactive generation (PartialSVG-to-SVG, PartialImage-to-SVG). Extensive experiments demonstrate that RoboSVG achieves superior query compliance and visual fidelity across tasks, establishing a new state of the art in versatile SVG generation. The
dataset and source code of this project will be publicly available soon.
Read the full abstract ▾
All authors Jiuniu Wang, Gongjie Zhang, Quanhao Qian, Junlong Gao, Deli Zhao, Ran Xu
Computation and Language · Cross-listed from cs.AI
Do Stop Me Now: Detecting Boilerplate Responses with a Single Iteration
Yuval Kainan and Shaked Zychlinski
Large
Language Models (LLMs) often expend significant computational resources generating boilerplate responses, such as refusals, simple acknowledgements and casual greetings, which adds unnecessary cost and latency. To address this inefficiency, we propose a simple yet highly effective method for detecting such responses after only a single generation step. We demonstrate that the log-probability distribution of the first generated
token serves as a powerful signal for classifying the nature of the entire subsequent response. Our experiments, conducted across a diverse range of small, large, and
reasoning-specialized models, show that the first-
token log-probability vectors form distinctly separable clusters for different response types. Using a lightweight k-NN classifier, we achieve high accuracy in predicting whether a response will be a substantive answer or a form of boilerplate response, including user-specified refusals. The primary implication is a practical, computationally trivial technique, optimizing LLM inference by enabling early termination or redirection to a smaller model, thereby yielding significant savings in computational cost. This work presents a direct path toward more efficient and sustainable LLM deployment.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views
Anna Deichler and Jonas Beskow
We introduce Look and Tell, a
multimodal dataset for studying referential communication across
egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze,
speech, and
video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a
benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect
multimodal grounding. The
dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of
embodied agents that can understand and engage in situated dialogue.
Read the full abstract ▾
Computation and Language
Conjugate Relation Modeling for Few-Shot Knowledge Graph Completion
Zilong Wang, Qingtian Zeng, Hua Duan et al. (+3 more)
Few-shot Knowledge Graph Completion (FKGC) infers missing triples from limited support samples, tackling long-tail distribution challenges. Existing methods, however, struggle to capture complex relational patterns and mitigate data sparsity. To address these challenges, we propose a novel FKGC framework for conjugate relation modeling (CR-FKGC). Specifically, it employs a neighborhood aggregation encoder to integrate higher-order neighbor information, a conjugate relation learner combining an implicit conditional
diffusion relation module with a stable relation module to capture stable semantics and uncertainty offsets, and a
manifold conjugate decoder for efficient evaluation and inference of missing triples in
manifold space. Experiments on three
benchmarks demonstrate that our method achieves superior performance over state-of-the-art methods.
Read the full abstract ▾
All authors Zilong Wang, Qingtian Zeng, Hua Duan, Cheng Cheng, Minghao Zou, Ziyang Wang
Computation and Language
Culturally Grounded Physical Commonsense Reasoning in Italian and English: A Submission to the MRL 2025 Shared Task
Marco De Santis and Lisa Alazraki
This paper presents our submission to the MRL 2025 Shared Task on
Multilingual Physical
Reasoning Datasets. The objective of the shared task is to create manually-annotated evaluation data in the physical commonsense
reasoning domain, for
languages other than English, following a format similar to PIQA. Our contribution, FormaMentis, is a novel
benchmark for physical commonsense
reasoning that is grounded in Italian
language and culture. The data samples in FormaMentis are created by expert annotators who are native Italian speakers and are familiar with local customs and norms. The samples are additionally translated into English, while preserving the cultural elements unique to the Italian context.
Read the full abstract ▾
Computation and Language
Integrating Linguistics and AI: Morphological Analysis and Corpus development of Endangered Toto Language of West Bengal
Ambalika Guha, Sajal Saha, Debanjan Ballav et al. (+2 more)
Preserving linguistic diversity is necessary as every
language offers a distinct perspective on the world. There have been numerous global initiatives to preserve endangered
languages through documentation. This paper is a part of a project which aims to develop a trilingual (Toto-Bangla-English)
language learning application to digitally archive and promote the endangered Toto
language of West Bengal, India. This application, designed for both native Toto speakers and non-native learners, aims to revitalize the
language by ensuring accessibility and usability through Unicode script integration and a structured
language corpus. The research includes detailed linguistic documentation collected via fieldwork, followed by the creation of a morpheme-tagged, trilingual corpus used to train a Small
Language Model (SLM) and a
Transformer-based translation engine. The analysis covers inflectional morphology such as person-number-gender agreement, tense-aspect-mood distinctions, and case marking, alongside derivational strategies that reflect word-class changes. Script standardization and digital literacy tools were also developed to enhance script usage. The study offers a sustainable model for preserving endangered
languages by incorporating traditional linguistic methodology with AI. This bridge between linguistic research with technological innovation highlights the value of interdisciplinary collaboration for community-based
language revitalization.
Read the full abstract ▾
All authors Ambalika Guha, Sajal Saha, Debanjan Ballav, Soumi Mitra, Hritwick Chakraborty
Computation and Language
PerCoR: Evaluating Commonsense Reasoning in Persian via Multiple-Choice Sentence Completion
Morteza Alikhani, Mohammadtaha Bagherifard, Erfan Zinvandi et al. (+1 more)
We introduced PerCoR (Persian Commonsense
Reasoning), the first large-scale Persian
benchmark for commonsense
reasoning. PerCoR contains 106K multiple-choice sentence-completion problems drawn from more than forty news, cultural, and other web sources. We introduce a novel conjunction-based
segmentation strategy to generate coherent sentence-completion pairs, enabling broad topical and structural diversity. To create challenging distractors, we propose DRESS-AF (Distractor Ranking via
Embedding Similarity Scoring and
Adversarial Filtering), a generation-free
adversarial filtering method that selects distractors from the pool of gold continuations while maximising model confusion. Human annotators score 89% on PerCoR, while OpenAI-o3 achieves the highest performance at 92.18%, followed closely by Claude-Sonnet-3.7 (91.17%). The strongest open-source model, DeepSeek-R1, reaches 82.51%, underscoring both the
dataset's difficulty and the remaining performance gap in Persian commonsense
reasoning. We further show that DRESS-AF transfers to the English HellaSwag
benchmark, increasing its difficulty without hurting human solvability. The
dataset is available at https://huggingface.co/
datasets/MCINext/PerCoR.
Read the full abstract ▾
All authors Morteza Alikhani, Mohammadtaha Bagherifard, Erfan Zinvandi, Mehran Sarmadi
Computation and Language
Personal Care Utility (PCU): Building the Health Infrastructure for Everyday Insight and Guidance
Mahyar Abbasian and Ramesh Jain
Building on decades of success in digital infrastructure and biomedical innovation, we propose the Personal Care Utility (PCU) - a cybernetic system for lifelong health guidance. PCU is conceived as a global, AI-powered utility that continuously orchestrates
multimodal data, knowledge, and services to assist individuals and populations alike. Drawing on
multimodal agents, event-centric modeling, and contextual inference, it offers three essential capabilities: (1) trusted health information tailored to the individual, (2) proactive health
navigation and behavior guidance, and (3) ongoing interpretation of recovery and treatment response after medical events. Unlike conventional episodic care, PCU functions as an ambient, adaptive companion - observing, interpreting, and guiding health in real time across daily life. By integrating personal sensing, experiential computing, and population-level analytics, PCU promises not only improved outcomes for individuals but also a new substrate for public health and scientific discovery. We describe the architecture, design principles, and implementation challenges of this emerging paradigm.
Read the full abstract ▾
Computation and Language
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Dario Loi, Elena Maria Muià, Federico Siciliano et al. (+4 more)
We present AutoBench, a fully automated and self-sustaining framework for evaluating Large
Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.. Unlike static
benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established
benchmarks including MMLU-Pro and GPQA (respectively 78% and 63%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static
benchmarks for the continuous evaluation of evolving
language models.
Read the full abstract ▾
All authors Dario Loi, Elena Maria Muià, Federico Siciliano, Giovanni Trappolini, Vincenzo Crisà, Peter Kruger, Fabrizio Silvestri
Computation and Language · Cross-listed from cs.AI
ATOM: AdapTive and OptiMized dynamic temporal knowledge graph construction using LLMs
Yassir Lairgi, Ludovic Moncla, Khalid Benabdeslem et al. (+2 more)
In today's rapidly expanding data landscape, knowledge extraction from unstructured text is vital for real-time analytics, temporal inference, and dynamic
memory frameworks. However, traditional static knowledge graph (KG) construction often overlooks the dynamic and time-sensitive nature of real-world data, limiting adaptability to continuous changes. Moreover, recent zero- or few-shot approaches that avoid domain-specific fine-tuning or reliance on prebuilt ontologies often suffer from instability across multiple runs, as well as incomplete coverage of key facts. To address these challenges, we introduce ATOM (AdapTive and OptiMized), a few-shot and scalable approach that builds and continuously updates Temporal Knowledge Graphs (TKGs) from unstructured texts. ATOM splits input documents into minimal, self-contained "atomic" facts, improving extraction exhaustivity and stability. Then, it constructs atomic TKGs from these facts, employing a dual-time modeling that distinguishes between when information is observed and when it is valid. The resulting atomic TKGs are subsequently merged in parallel. Empirical evaluations demonstrate that ATOM achieves ~18% higher exhaustivity, ~33% better stability, and over ~90% latency reduction compared to baseline methods, demonstrating a strong scalability potential for dynamic TKG construction.
Read the full abstract ▾
All authors Yassir Lairgi, Ludovic Moncla, Khalid Benabdeslem, Rémy Cazabet, Pierre Cléau
Computation and Language · Cross-listed from eess.AS
UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models
Wenming Tu, Guanrou Yang, Ruiqi Yan et al. (+6 more)
Spoken dialogue models currently lack the ability for fine-grained
speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like
reasoning and question answering. To address this limitation, we introduce UltraVoice, the first large-scale
speech dialogue
dataset engineered for multiple fine-grained
speech style control. Encompassing over 830 hours of
speech dialogues, UltraVoice provides instructions across six key
speech stylistic dimensions:
emotion, speed, volume, accent,
language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained
speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench
benchmark, our fine-tuned models demonstrate substantial gains in core understanding,
reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the
dataset's utility extends to training controllable Text-to-
Speech (TTS) models, underscoring its high quality and broad applicability for expressive
speech synthesis. The complete
dataset and model checkpoints are available at: https://github.com/bigai-nlco/UltraVoice.
Read the full abstract ▾
All authors Wenming Tu, Guanrou Yang, Ruiqi Yan, Wenxi Chen, Ziyang Ma, Yipeng Kang, Kai Yu, Xie Chen, Zilong Zheng
Computation and Language
Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems
Kaushal Kumar Maurya and Ekaterina Kochmar
The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large
language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and
benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized
benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.
Read the full abstract ▾
Computation and Language
A Closed-Loop Personalized Learning Agent Integrating Neural Cognitive Diagnosis, Bounded-Ability Adaptive Testing, and LLM-Driven Feedback
Zhifeng Wang, Xinyue Zheng and Chunyan Zeng
As information technology advances, education is moving from one-size-fits-all instruction toward personalized learning. However, most methods handle modeling, item selection, and feedback in isolation rather than as a closed loop. This leads to coarse or opaque student models, assumption-bound adaptivity that ignores diagnostic posteriors, and generic, non-actionable feedback. To address these limitations, this paper presents an end-to-end personalized learning
agent, EduLoop-
Agent, which integrates a Neural Cognitive Diagnosis model (NCD), a Bounded-Ability Estimation Computerized Adaptive Testing strategy (BECAT), and large
language models (LLMs). The NCD module provides fine-grained estimates of students' mastery at the knowledge-point level; BECAT dynamically selects subsequent items to maximize relevance and learning efficiency; and LLMs convert diagnostic signals into structured, actionable feedback. Together, these components form a closed-loop framework of “Diagnosis–Recommendation–Feedback.” Experiments on the ASSISTments
dataset show that the NCD module achieves strong performance on response prediction while yielding interpretable mastery assessments. The adaptive recommendation strategy improves item relevance and personalization, and the LLM-based feedback offers targeted study guidance aligned with identified weaknesses. Overall, the results indicate that the proposed design is effective and practically deployable, providing a feasible pathway to generating individualized learning trajectories in intelligent education.
Read the full abstract ▾
Computation and Language
SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
Jinhan Chen, Jianchun Liu, Hongli Xu et al. (+2 more)
The growing
memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large
Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discarding less critical
tokens, existing
token-, block-, and sentence-level compression methods struggle to balance semantic coherence and
memory efficiency. To this end, we introduce SABlock, a \underline{s}emantic-aware KV cache eviction framework with \underline{a}daptive \underline{block} sizes. Specifically, SABlock first performs semantic
segmentation to align compression boundaries with linguistic structures, then applies segment-guided
token scoring to refine
token importance estimation. Finally, for each segment, a budget-driven search strategy adaptively determines the optimal block size that preserves semantic integrity while improving compression efficiency under a given cache budget. Extensive experiments on long-context
benchmarks demonstrate that SABlock consistently outperforms state-of-the-art baselines under the same
memory budgets. For instance, on Needle-in-a-Haystack (NIAH), SABlock achieves 99.9%
retrieval accuracy with only 96 KV entries, nearly matching the performance of the full-cache baseline that retains up to 8K entries. Under a fixed cache budget of 1,024, SABlock further reduces peak
memory usage by 46.28% and achieves up to 9.5x faster decoding on a 128K context length.
Read the full abstract ▾
All authors Jinhan Chen, Jianchun Liu, Hongli Xu, Xianjun Gao, Shilong Wang
Computation and Language
LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?
Ziyuan He, Yuxuan Wang, Jiaqi Li et al. (+2 more)
Large
language models (LLMs) are equipped with increasingly extended context windows recently, yet their long context understanding capabilities over long dependency tasks remain fundamentally limited and underexplored. This gap is especially significant in many real-world long-context applications that were rarely benchmarked. In this paper, we introduce LooGLE v2, a novel
benchmark designed to evaluate LLMs' long context ability in real-world applications and scenarios. Our
benchmark consists of automatically collected real-world long texts, ranging from 16k to 2M
tokens, encompassing domains in law, finance, game and code. Accordingly, we delicately design 10 types of domain-specific long-dependency tasks and generate 1,934 QA instances with various diversity and complexity in a scalable data curation pipeline for further practical needs. We conduct a comprehensive assessment of 6 locally deployed and 4 API-based LLMs. The evaluation results show that even the best-performing model achieves only a 59.2% overall score on our
benchmark. Despite the extensive context windows, popular LLMs are only capable of understanding a much shorter length of context than they claim to be, revealing significant limitations in their ability to handle real-world tasks with long dependencies and highlighting substantial room for model improvement in practical long-context understanding.
Read the full abstract ▾
All authors Ziyuan He, Yuxuan Wang, Jiaqi Li, Kexin Liang, Muhan Zhang
Computation and Language · Cross-listed from cs.AI
OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models
Hao Zheng, Zirui Pang, Ling li et al. (+5 more)
Advances in
Multimodal Large
Language Models (MLLMs) intensify concerns about data
privacy, making Machine Unlearning (MU), the selective removal of learned information, a critical necessity. However, existing MU
benchmarks for MLLMs are limited by a lack of image diversity, potential inaccuracies, and insufficient evaluation scenarios, which fail to capture the complexity of real-world applications. To facilitate the development of MLLMs unlearning and alleviate the aforementioned limitations, we introduce OFFSIDE, a novel
benchmark for evaluating misinformation unlearning in MLLMs based on football transfer rumors. This manually curated
dataset contains 15.68K records for 80 players, providing a comprehensive framework with four test sets to assess forgetting efficacy, generalization, utility, and robustness. OFFSIDE supports advanced settings like selective unlearning and corrective relearning, and crucially, unimodal unlearning (forgetting only text data). Our extensive evaluation of multiple baselines reveals key findings: (1) Unimodal methods (erasing text-based knowledge) fail on
multimodal rumors; (2) Unlearning efficacy is largely driven by catastrophic forgetting; (3) All methods struggle with "visual rumors" (rumors appear in the image); (4) The unlearned rumors can be easily recovered and (5) All methods are vulnerable to
prompt attacks. These results expose significant vulnerabilities in current approaches, highlighting the need for more robust
multimodal unlearning solutions. The code is available at https://github.com/zh121800/OFFSIDE
Read the full abstract ▾
All authors Hao Zheng, Zirui Pang, Ling li, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, Jiaheng Wei
Computation and Language
Text to Trust: Evaluating Fine-Tuning and LoRA Trade-offs in Language Models for Unfair Terms of Service Detection
Noshitha Padma Pratyusha Juttu, Sahithi Singireddy, Sravani Gona et al. (+1 more)
Large
Language Models (LLMs) have transformed text understanding, yet their adaptation to specialized legal domains remains constrained by the cost of full fine-tuning. This study provides a systematic evaluation of fine tuning, parameter efficient adaptation (LoRA, QLoRA), and zero-shot prompting strategies for unfair clause detection in Terms of Service (ToS) documents, a key application in legal NLP. We finetune BERT and DistilBERT, apply 4-bit Low-Rank Adaptation (LoRA) to models such as TinyLlama, LLaMA 3B/7B, and SaulLM, and evaluate GPT-4o and O-versions in zero-shot settings. Experiments on the CLAUDETTE-ToS
benchmark and the
Multilingual Scraper Corpus show that full fine-tuning achieves the strongest precision recall balance, while LoRA-based models provide competitive recall with up to 3x lower
memory cost. These findings highlight practical design trade-offs for efficient and domain-adapted LLMs, contributing open baselines for fine-tuning research in legal text processing.
Read the full abstract ▾
All authors Noshitha Padma Pratyusha Juttu, Sahithi Singireddy, Sravani Gona, Sujal Timilsina
Computation and Language · Cross-listed from cs.LG
Towards Scalable Oversight via Partitioned Human Supervision
Ren Yin, Takashi Ishida and Masashi Sugiyama
As artificial intelligence (AI) systems approach and surpass expert human performance across a broad range of tasks, obtaining high-quality human supervision for evaluation and training becomes increasingly challenging. Our focus is on tasks that require deep knowledge and skills of multiple domains, where this bottleneck is severe. Unfortunately, even the best human experts are knowledgeable only in a single narrow area, and will not be able to evaluate the correctness of advanced AI systems on such superhuman tasks. However, based on their narrow expertise, humans may provide a weak signal, i.e., a complementary label indicating an option that is incorrect. For example, a cardiologist could state that ”this is not related to any cardiovascular disease,” even if they cannot identify the true disease. Based on this weak signal, we propose a scalable oversight framework that enables us to evaluate frontier AI systems without the need to prepare the ground truth. We derive an unbiased
estimator of top-1 accuracy from complementary labels and quantify how many complementary labels are needed to match the variance of ordinary labels. We further introduce two
estimators to combine scarce ordinary labels with abundant complementary labels. We provide finite-sample deviation guarantees for both complementary-only and the mixed
estimators. Empirically, we show that we can evaluate the output of large
language models without the ground truth, if we have complementary labels. We further show that we can train an AI system with such weak signals: we show how we can design an agentic AI system automatically that can improve itself with this partitioned human supervision. Our code is available at https://github.com/R-Yin-217/Towards-Scalable-Oversight-via-Partitioned-Human-Supervision.
Read the full abstract ▾
Computation and Language
A Sociophonetic Analysis of Racial Bias in Commercial ASR Systems Using the Pacific Northwest English Corpus
Michael Scott, Siyu Liang, Alicia Wassink et al. (+1 more)
This paper presents a systematic evaluation of racial bias in four major commercial automatic
speech recognition (ASR) systems using the Pacific Northwest English (PNWE) corpus. We analyze transcription accuracy across speakers from four ethnic backgrounds (African American, Caucasian American, ChicanX, and Yakama) and examine how sociophonetic variation contributes to differential system performance. We introduce a heuristically-determined Phonetic Error Rate (PER) metric that links recognition errors to specific linguistically motivated variables derived from sociophonetic annotation. Our analysis of eleven sociophonetic features reveals that vowel quality variation, particularly resistance to the low-back merger and pre-nasal merger patterns, is systematically associated with differential error rates across ethnic groups, with the most pronounced effects for African American speakers across all evaluated systems. These findings demonstrate that acoustic modeling of dialectal phonetic variation, rather than lexical or syntactic factors, remains a primary source of bias in commercial ASR systems. The study establishes the PNWE corpus as a valuable resource for bias evaluation in
speech technologies and provides actionable guidance for improving ASR performance through targeted representation of sociophonetic diversity in training data.
Read the full abstract ▾
All authors Michael Scott, Siyu Liang, Alicia Wassink, Gina-Anne Levow
Computation and Language
The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR
Siyu Liang, Nicolas Ballier, Gina-Anne Levow et al. (+1 more)
How much audio is needed to fully observe a
multilingual ASR model's learned sub-
token inventory across
languages, and does data disparity in
multilingual pre-training affect how these
tokens are utilized during inference? We address this question by analyzing Whisper's decoding behavior during inference across 49
languages. By logging decoding candidate sub-
tokens and tracking their cumulative discovery over time, we study the utilization pattern of the model's sub-
token space. Results show that the total number of discovered
tokens remains largely independent of a
language's pre-training hours, indicating that data disparity does not strongly influence lexical diversity in the model's hypothesis space. Sub-
token discovery rates follow a consistent exponential saturation pattern across
languages, suggesting a stable time window after which additional audio yields minimal new sub-
token activation. We refer to this convergence threshold as acoustic saturation time (AST). Further analyses of rank-frequency distributions reveal Zipf-like patterns better modeled by a Zipf-Mandelbrot law, and mean sub-
token length shows a positive correlation with resource level. Additionally, those metrics show more favorable patterns for
languages in the Latin script than those in scripts such as Cyrillic, CJK, and Semitic. Together, our study suggests that sub-
token utilization during
multilingual ASR inference is constrained more by the statistical, typological, and orthographic structure of the
speech than by training data scale, providing an empirical basis for more equitable corpus construction and cross-lingual evaluation.
Read the full abstract ▾
All authors Siyu Liang, Nicolas Ballier, Gina-Anne Levow, Richard Wright
Computation and Language
Frustratingly Easy Task-aware Pruning for Large Language Models
Yuanhe Tian, Junjie Liu, Xican Yang et al. (+2 more)
Pruning provides a practical solution to reduce the resources required to run large
language models (LLMs) to benefit from their effective capabilities as well as control their cost for training and inference. Research on LLM
pruning often ranks the importance of LLM parameters using their magnitudes and calibration-data activations and removes (or masks) the less important ones, accordingly reducing LLMs' size. However, these approaches primarily focus on preserving the LLM's ability to generate fluent sentences, while neglecting performance on specific domains and tasks. In this paper, we propose a simple yet effective
pruning approach for LLMs that preserves task-specific capabilities while shrinking their parameter space. We first analyze how conventional
pruning minimizes loss perturbation under general-domain calibration and extend this formulation by incorporating task-specific feature distributions into the importance computation of existing
pruning algorithms. Thus, our framework computes separate importance scores using both general and task-specific calibration data, partitions parameters into shared and exclusive groups based on activation-norm differences, and then fuses their scores to guide the
pruning process. This design enables our method to integrate seamlessly with various foundation
pruning techniques and preserve the LLM's specialized abilities under compression. Experiments on widely used
benchmarks demonstrate that our approach is effective and consistently outperforms the baselines with identical
pruning ratios and different settings.
Read the full abstract ▾
All authors Yuanhe Tian, Junjie Liu, Xican Yang, Haishan Ye, Yan Song
Computation and Language
The Tonogenesis Continuum in Tibetan: A Computational Investigation
Siyu Liang and Zhaxi Zerong
Tonogenesis-the historical process by which segmental contrasts evolve into lexical tone-has traditionally been studied through comparative reconstruction and acoustic phonetics. We introduce a computational approach that quantifies the functional role of pitch at different stages of this sound change by measuring how pitch manipulation affects automatic
speech recognition (ASR) performance. Through analysis on the sensitivity to pitch-flattening from a set of closely related Tibetan
languages, we find evidence of a tonogenesis continuum: atonal Amdo dialects tolerate pitch removal the most, while fully tonal U-Tsang varieties show severe degradation, and intermediate Kham dialects fall measurably between these extremes. These gradient effects demonstrate how ASR models implicitly learn the shifting functional load of pitch as
languages transition from consonant-based to tone-based lexical contrasts. Our findings show that computational methods can capture fine-grained stages of sound change and suggest that traditional functional load metrics, based solely on minimal pairs, may overestimate pitch dependence in transitional systems where segmental and suprasegmental cues remain phonetically intertwined.
Read the full abstract ▾
Computation and Language
CHOIR: Collaborative Harmonization fOr Inference Robustness
Xiangjue Dong, Cong Wang, Maria Teleki et al. (+2 more)
Persona-assigned Large
Language Models (LLMs) can adopt diverse roles, enabling personalized and context-aware
reasoning. However, even minor demographic perturbations in personas, such as simple pronoun changes, can alter
reasoning trajectories, leading to divergent sets of correct answers. Instead of treating these variations as biases to be mitigated, we explore their potential as a constructive resource to improve
reasoning robustness. We propose CHOIR (Collaborative Harmonization fOr Inference Robustness), a test-time framework that harmonizes multiple persona-conditioned
reasoning signals into a unified prediction. CHOIR orchestrates a collaborative decoding process among counterfactual personas, dynamically balancing agreement and divergence in their
reasoning paths. Experiments on various
reasoning benchmarks demonstrate that CHOIR consistently enhances performance across demographics, model architectures, scales, and tasks - without additional training. Improvements reach up to 26.4% for individual demographic groups and 19.2% on average across five demographics. It remains effective even when base personas are suboptimal. By reframing persona variation as a constructive signal, CHOIR provides a scalable and generalizable approach to more reliable LLM
reasoning.
Read the full abstract ▾
All authors Xiangjue Dong, Cong Wang, Maria Teleki, Millennium Bismay, James Caverlee