Computation and Language · Cross-listed from cs.AI
SynBullying: A Multi LLM Synthetic Conversational Dataset for Cyberbullying Detection
Arefeh Kazemi, Hamza Qadeer, Joachim Wagner et al. (+3 more)
We introduce SynBullying, a synthetic multi-LLM conversational
dataset for studying and detecting cyberbullying (CB). SynBullying provides a scalable and ethically safe alternative to human data collection by leveraging large
language models (LLMs) to simulate realistic bullying interactions. The
dataset offers (i) conversational structure, capturing multi-turn exchanges rather than isolated posts; (ii) context-aware annotations, where harmfulness is assessed within the conversational flow considering context, intent, and discourse dynamics; and (iii) fine-grained labeling, covering various CB categories for detailed linguistic and behavioral analysis. We evaluate SynBullying across five dimensions, including conversational structure, lexical patterns, sentiment/toxicity, role dynamics, harm intensity, and CB-type distribution. We further examine its utility by testing its performance as standalone training data and as an augmentation source for CB classification.
Read the full abstract ▾
All authors Arefeh Kazemi, Hamza Qadeer, Joachim Wagner, Hossein Hosseini, Sri Balaaji Natarajan Kalaivendan, Brian Davis
Computation and Language
Chopping Trees: Semantic Similarity Based Dynamic Pruning for Tree-of-Thought Reasoning
Joongho Kim, Xirui Huang, Zarreen Reza et al. (+1 more)
Tree-of-Thought (ToT)
reasoning boosts the problem-solving abilities of Large
Language Models (LLMs) but is computationally expensive due to semantic redundancy, where distinct branches explore equivalent
reasoning paths. We introduce Semantic Similarity-Based Dynamic
Pruning (SSDP), a lightweight method that, to the best of our knowledge, is the first framework to integrate online semantic merging into parallelized tree search, enabling the clustering and
pruning of redundant steps in real time. Across
reasoning benchmarks, including GSM8K and MATH500, SSDP achieves up to a 2.3x speedup over state-of-the-art tree-search baselines while maintaining competitive accuracy (typically within 5% of the strongest baseline) and reducing the number of explored nodes by 85-90%, demonstrating a practical approach to efficient, scalable LLM
reasoning. The implementation of SSDP is publicly available at https://github.com/kimjoonghokim/SSDP.
Read the full abstract ▾
All authors Joongho Kim, Xirui Huang, Zarreen Reza, Gabriel Grand
Computation and Language
Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
Usha Bhalla, Alex Oesterling, Claudio Mayrink Verdun et al. (+2 more)
Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover
token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of
language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent
tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple
datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in
language models.
Read the full abstract ▾
All authors Usha Bhalla, Alex Oesterling, Claudio Mayrink Verdun, Himabindu Lakkaraju, Flavio P. Calmon
Computation and Language
Reasoning Up the Instruction Ladder for Controllable Language Models
Zishuo Zheng, Vidhisha Balachandran, Chan Young Park et al. (+2 more)
As large
language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single
prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a
reasoning task. The model must first "think" about the relationship between a given user
prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training
dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general
reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy
benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved
alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and
prompt injection, reducing absolute attack success rates by up to 20%. Our results establish
reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system
prompts produce predictable, controllable, and robust changes in model behavior.
Read the full abstract ▾
All authors Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar
Computation and Language · Cross-listed from cs.CY
Wayfinding through the AI wilderness: Mapping rhetorics of ChatGPT prompt writing on X (formerly Twitter) to promote critical AI literacies
Anuj Gupta and Ann Shivers-McNair
In this paper, we demonstrate how studying the rhetorics of ChatGPT
prompt writing on social media can promote critical AI literacies.
Prompt writing is the process of writing instructions for generative AI tools like ChatGPT to elicit desired outputs and there has been an upsurge of conversations about it on social media. To study this rhetorical activity, we build on four overlapping traditions of digital writing research in computers and composition that inform how we frame literacies, how we study social media rhetorics, how we engage iteratively and reflexively with methodologies and technologies, and how we blend computational methods with qualitative methods. Drawing on these four traditions, our paper shows our iterative research process through which we gathered and analyzed a
dataset of 32,000 posts (formerly known as tweets) from X (formerly Twitter) about
prompt writing posted between November 2022 to May 2023. We present five themes about these emerging AI literacy practices: (1) areas of communication impacted by
prompt writing, (2) micro-literacy resources shared for
prompt writing, (3)
market rhetoric shaping
prompt writing, (4) rhetorical characteristics of
prompts, and (5) definitions of
prompt writing. In discussing these themes and our methodologies, we highlight takeaways for digital writing teachers and researchers who are teaching and analyzing critical AI literacies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
QuantumBench: A Benchmark for Quantum Problem Solving
Shunya Minami, Tatsuya Ishigaki, Ikko Hamamura et al. (+6 more)
Large
language models are now integrated into many scientific workflows, accelerating data analysis, hypothesis generation, and design space exploration. In parallel with this growth, there is a growing need to carefully evaluate whether models accurately capture domain-specific knowledge and notation, since general-purpose
benchmarks rarely reflect these requirements. This gap is especially clear in
quantum science, which features non-intuitive phenomena and requires advanced mathematics. In this study, we introduce QuantumBench, a
benchmark for the
quantum domain that systematically examine how well LLMs understand and can be applied to this non-intuitive field. Using publicly available materials, we compiled approximately 800 questions with their answers spanning nine areas related to
quantum science and organized them into an eight-option multiple-choice
dataset. With this
benchmark, we evaluate several existing LLMs and analyze their performance in the
quantum domain, including sensitivity to changes in question format. QuantumBench is the first LLM evaluation
dataset built for the
quantum domain, and it is intended to guide the effective use of LLMs in
quantum research.
Read the full abstract ▾
All authors Shunya Minami, Tatsuya Ishigaki, Ikko Hamamura, Taku Mikuriya, Youmi Ma, Naoaki Okazaki, Hiroya Takamura, Yohichi Suzuki, Tadashi Kadowaki
Computation and Language
Detecting Data Contamination in LLMs via In-Context Learning
Michał Zawalski, Meriem Boubdir, Klaudia Bałazy et al. (+2 more)
We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large
language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen
datasets but may reduce it when the
dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen
datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and
dataset-agnostic, making it easy to integrate with
benchmark evaluations.
Read the full abstract ▾
All authors Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, Pablo Ribalta
Computation and Language
LLM-Centric RAG with Multi-Granular Indexing and Confidence Constraints
Xiaofan Guo, Yaxuan Luan, Yue Kang et al. (+2 more)
This paper addresses the issues of insufficient coverage, unstable results, and limited reliability in
retrieval-augmented generation under complex knowledge environments, and proposes a confidence control method that integrates multi-granularity
memory indexing with uncertainty estimation. The method builds a hierarchical
memory structure that divides knowledge representations into different levels of granularity, enabling dynamic indexing and
retrieval from local details to global context, and thus establishing closer semantic connections between
retrieval and generation. On this basis, an uncertainty estimation mechanism is introduced to explicitly constrain and filter low-confidence paths during the generation process, allowing the model to maintain information coverage while effectively suppressing noise and false content. The overall optimization objective consists of generation loss,
entropy constraints, and variance regularization, forming a unified confidence control framework. In the experiments, comprehensive sensitivity tests and comparative analyses were designed, covering hyperparameters, environmental conditions, and data structures, to verify the stability and robustness of the proposed method across different scenarios. The results show that the method achieves superior performance over existing models in QA accuracy,
retrieval recall, ranking quality, and factual consistency, demonstrating the effectiveness of combining multi-granularity indexing with confidence control. This study not only provides a new technical pathway for
retrieval-augmented generation but also offers practical evidence for improving the reliability and controllability of large models in complex contexts.
Read the full abstract ▾
All authors Xiaofan Guo, Yaxuan Luan, Yue Kang, Xiangchen Song, Jinxu Guo
Computation and Language
VISTA: Verification In Sequential Turn-based Assessment
Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier et al. (+1 more)
Hallucination–defined here as generating statements unsupported or contradicted by available evidence or conversational context–remains a major obstacle to deploying conversational AI systems in settings that demand factual reliability. Existing metrics either evaluate isolated responses or treat unverifiable content as errors, limiting their use for multi-turn dialogue. We introduce VISTA (Verification In Sequential Turn-based Assessment), a framework for evaluating conversational factuality through claim-level verification and sequential consistency tracking. VISTA decomposes each assistant turn into atomic factual claims, verifies them against trusted sources and dialogue history, and categorizes unverifiable statements (subjective, contradicted, lacking evidence, or abstaining). Across eight large
language models and four dialogue factuality
benchmarks (AIS, BEGIN, FAITHDIAL, and FADE), VISTA substantially improves
hallucination detection over FACTSCORE and LLM-as-Judge baselines. Human evaluation confirms that VISTA's decomposition improves annotator agreement and reveals inconsistencies in existing
benchmarks. By modeling factuality as a dynamic property of conversation, VISTA offers a more transparent, human-aligned measure of truthfulness in dialogue systems.
Read the full abstract ▾
All authors Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White
Computation and Language
Recursive numeral systems are highly regular and easy to process
Ponrawee Prasertsom, Andrea Silvi, Jennifer Culbertson et al. (+3 more)
Much recent work has shown how cross-linguistic variation is constrained by competing pressures from efficient communication. However, little
attention has been paid to the role of the systematicity of forms (regularity), a key property of natural
language. Here, we demonstrate the importance of regularity in explaining the shape of linguistic systems by looking at recursive numeral systems. Previous work has argued that these systems optimise the trade-off between lexicon size and average morphosyntatic complexity (Denić and Szymanik, 2024). However, showing that only natural-
language-like systems optimise this trade-off has proven elusive, and existing solutions rely on ad-hoc constraints to rule out unnatural systems (Yang and Regier, 2025). Drawing on the Minimum Description Length (MDL) approach, we argue that recursive numeral systems are better viewed as efficient with regard to their regularity and processing complexity. We show that our MDL-based measures of regularity and processing complexity better capture the key differences between attested, natural systems and theoretically possible ones, including "optimal" recursive numeral systems from previous work, and that the ad-hoc constraints naturally follow from regularity. Our approach highlights the need to incorporate regularity across sets of forms in studies attempting to measure efficiency in
language.
Read the full abstract ▾
All authors Ponrawee Prasertsom, Andrea Silvi, Jennifer Culbertson, Moa Johansson, Devdatt Dubhashi, Kenny Smith
Computation and Language
Quantitative Intertextuality from the Digital Humanities Perspective: A Survey
Siyu Duan
The connection between texts is referred to as intertextuality in literary theory, which served as an important theoretical basis in many digital humanities studies. Over the past decade, advancements in natural
language processing have ushered intertextuality studies into the quantitative age. Large-scale intertextuality research based on cutting-edge methods has continuously emerged. This paper provides a roadmap for quantitative intertextuality studies, summarizing their data, methods, and applications. Drawing on data from multiple
languages and topics, this survey reviews methods from statistics to deep learning. It also summarizes their applications in humanities and social sciences research and the associated platform tools. Driven by advances in computer technology, more precise, diverse, and large-scale intertext studies can be anticipated. Intertextuality holds promise for broader application in interdisciplinary research bridging AI and the humanities.
Read the full abstract ▾
Computation and Language
Dataset Creation and Baseline Models for Sexism Detection in Hausa
Fatima Adam Muhammad, Shamsuddeen Muhammad Hassan and Isa Inuwa-Dutse
Sexism reinforces gender inequality and social exclusion by perpetuating stereotypes, bias, and discriminatory norms. Noting how online platforms enable various forms of sexism to thrive, there is a growing need for effective sexism detection and mitigation strategies. While computational approaches to sexism detection are widespread in high-resource
languages, progress remains limited in low-resource
languages where limited linguistic resources and cultural differences affect how sexism is expressed and perceived. This study introduces the first Hausa sexism detection
dataset, developed through community engagement, qualitative coding, and data augmentation. For cultural nuances and linguistic representation, we conducted a two-stage user study (n=66) involving native speakers to explore how sexism is defined and articulated in everyday discourse. We further experiment with both traditional machine learning classifiers and pre-trained
multilingual language models and evaluating the effectiveness few-shot learning in detecting sexism in Hausa. Our findings highlight challenges in capturing cultural nuance, particularly with clarification-seeking and idiomatic expressions, and reveal a tendency for many false positives in such cases.
Read the full abstract ▾
Computation and Language
Elastic Architecture Search for Efficient Language Models
Shang Wang
As large pre-trained
language models become increasingly critical to natural
language understanding (NLU) tasks, their substantial computational and
memory requirements have raised significant economic and environmental concerns. Addressing these challenges, this paper introduces the Elastic
Language Model (ELM), a novel neural architecture search (NAS) method optimized for compact
language models. ELM extends existing NAS approaches by introducing a flexible search space with efficient
transformer blocks and dynamic modules for dimension and head number adjustment. These innovations enhance the efficiency and flexibility of the search process, which facilitates more thorough and effective exploration of model architectures. We also introduce novel knowledge
distillation losses that preserve the unique characteristics of each block, in order to improve the discrimination between architectural choices during the search process. Experiments on masked
language modeling and causal
language modeling tasks demonstrate that models discovered by ELM significantly outperform existing methods.
Read the full abstract ▾
Computation and Language
Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral
Ayoub Hammal, Pierre Zweigenbaum and Caio Corro
Several previous works concluded that the largest part of generation capabilities of large
language models (LLM) are learned (early) during pre-training. However, LLMs still require further
alignment to adhere to downstream task requirements and stylistic preferences, among other desired properties. As LLMs continue to scale in terms of size, the computational cost of
alignment procedures increase prohibitively. In this work, we propose a novel approach to circumvent these costs via proxy-based test-time
alignment, i.e. using guidance from a small aligned model. Our approach can be described as a
token-specific cascading method, where the
token-specific deferral rule is reduced to 0-1 knapsack problem. In this setting, we derive primal and dual approximations of the optimal deferral decision. We experimentally show the benefits of our method both in task performance and speculative decoding speed.
Read the full abstract ▾
Computation and Language
Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services
Jayden Serenari and Stephen Lee
With the increasing use of conversational AI systems, there is growing concern over
privacy leaks, especially when users share sensitive personal data in interactions with Large
Language Models (LLMs). Conversations shared with these models may contain Personally Identifiable Information (PII), which, if exposed, could lead to security breaches or identity theft. To address this challenge, we present the Local Optimizations for Pseudonymization with Semantic Integrity Directed Entity Detection (LOPSIDED) framework, a semantically-aware
privacy agent designed to safeguard sensitive PII data when using remote LLMs. Unlike prior work that often degrade response quality, our approach dynamically replaces sensitive PII entities in user
prompts with semantically consistent pseudonyms, preserving the contextual integrity of conversations. Once the model generates its response, the pseudonyms are automatically depseudonymized, ensuring the user receives an accurate,
privacy-preserving output. We evaluate our approach using real-world conversations sourced from ShareGPT, which we further augment and annotate to assess whether named entities are contextually relevant to the model's response. Our results show that LOPSIDED reduces semantic utility errors by a factor of 5 compared to baseline techniques, all while enhancing
privacy.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Semantic Frame Aggregation-based Transformer for Live Video Comment Generation
Anam Fatima, Yi Yu, Janak Kapuriya et al. (+2 more)
Live commenting on
video streams has surged in popularity on platforms like Twitch, enhancing viewer engagement through dynamic interactions. However, automatically generating contextually appropriate comments remains a challenging and exciting task.
Video streams can contain a vast amount of data and extraneous content. Existing approaches tend to overlook an important aspect of prioritizing
video frames that are most relevant to ongoing viewer interactions. This prioritization is crucial for producing contextually appropriate comments. To address this gap, we introduce a novel Semantic Frame Aggregation-based
Transformer (SFAT) model for live
video comment generation. This method not only leverages CLIP's visual-text
multimodal knowledge to generate comments but also assigns weights to
video frames based on their semantic relevance to ongoing viewer conversation. It employs an efficient weighted sum of frames technique to emphasize informative frames while focusing less on irrelevant ones. Finally, our comment decoder with a cross-
attention mechanism that attends to each modality ensures that the generated comment reflects contextual cues from both chats and
video. Furthermore, to address the limitations of existing
datasets, which predominantly focus on Chinese-
language content with limited
video categories, we have constructed a large scale, diverse,
multimodal English
video comments
dataset. Extracted from Twitch, this
dataset covers 11
video categories, totaling 438 hours and 3.2 million comments. We demonstrate the effectiveness of our SFAT model by comparing it to existing methods for generating comments from live
video and ongoing dialogue contexts.
Read the full abstract ▾
All authors Anam Fatima, Yi Yu, Janak Kapuriya, Julien Lalanne, Jainendra Shukla
Computation and Language
Overview of the MEDIQA-OE 2025 Shared Task on Medical Order Extraction from Doctor-Patient Consultations
Jean-Philippe Corbeil, Asma Ben Abacha, Jerome Tremblay et al. (+4 more)
Clinical documentation increasingly uses automatic
speech recognition and summarization, yet converting conversations into actionable medical orders for Electronic Health Records remains unexplored. A solution to this problem can significantly reduce the documentation burden of clinicians and directly impact downstream patient care. We introduce the MEDIQA-OE 2025 shared task, the first challenge on extracting medical orders from doctor-patient conversations. Six teams participated in the shared task and experimented with a broad range of approaches, and both closed- and open-weight large
language models (LLMs). In this paper, we describe the MEDIQA-OE task,
dataset, final leaderboard ranking, and participants' solutions.
Read the full abstract ▾
All authors Jean-Philippe Corbeil, Asma Ben Abacha, Jerome Tremblay, Phillip Swazinna, Akila Jeeson Daniel, Miguel Del-Agua, Francois Beaulieu
Computation and Language
Frame Semantic Patterns for Identifying Underreporting of Notifiable Events in Healthcare: The Case of Gender-Based Violence
Lívia Dutra, Arthur Lorenzi, Laís Berno et al. (+10 more)
We introduce a methodology for the identification of notifiable events in the domain of healthcare. The methodology harnesses semantic frames to define fine-grained patterns and search them in unstructured data, namely, open-text fields in e-medical records. We apply the methodology to the problem of underreporting of gender-based violence (GBV) in e-medical records produced during patients' visits to primary care units. A total of eight patterns are defined and searched on a corpus of 21 million sentences in Brazilian Portuguese extracted from e-SUS APS. The results are manually evaluated by linguists and the precision of each pattern measured. Our findings reveal that the methodology effectively identifies reports of violence with a precision of 0.726, confirming its robustness. Designed as a transparent, efficient, low-carbon, and
language-agnostic pipeline, the approach can be easily adapted to other health surveillance contexts, contributing to the broader, ethical, and explainable use of NLP in public health systems.
Read the full abstract ▾
All authors Lívia Dutra, Arthur Lorenzi, Laís Berno, Franciany Campos, Karoline Biscardi, Kenneth Brown, Marcelo Viridiano, Frederico Belcavello, Ely Matos, Olívia Guaranha, Erik Santos, Sofia Reinach, Tiago Timponi Torrent
Computation and Language · Cross-listed from cs.RO
RepV: Safety-Separable Latent Spaces for Scalable Neurosymbolic Plan Verification
Yunhao Yang, Neel P. Bhatt, Pranay Samineni et al. (+3 more)
As AI systems migrate to safety-critical domains, verifying that their actions comply with well-defined rules remains a challenge. Formal methods provide provable guarantees but demand hand-crafted temporal-logic specifications, offering limited expressiveness and accessibility. Deep learning approaches enable evaluation of plans against natural-
language constraints, yet their opaque decision process invites misclassifications with potentially severe consequences. We introduce RepV, a neurosymbolic verifier that unifies both views by learning a latent space where safe and unsafe plans are linearly separable. Starting from a modest seed set of plans labeled by an off-the-shelf model checker, RepV trains a lightweight projector that embeds each plan, together with a
language model-generated rationale, into a low-dimensional space; a frozen linear boundary then verifies compliance for unseen natural-
language rules in a single forward pass. Beyond binary classification, RepV provides a probabilistic guarantee on the likelihood of correct verification based on its position in the latent space. This guarantee enables a guarantee-driven refinement of the planner, improving rule compliance without human annotations. Empirical evaluations show that RepV improves compliance prediction accuracy by up to 15% compared to baseline methods while adding fewer than 0.2M parameters. Furthermore, our refinement framework outperforms ordinary fine-tuning baselines across various planning domains. These results show that safety-separable latent spaces offer a scalable, plug-and-play primitive for reliable neurosymbolic plan verification. Code and data are available at: https://repv-project.github.io/.
Read the full abstract ▾
All authors Yunhao Yang, Neel P. Bhatt, Pranay Samineni, Rohan Siva, Zhanyang Wang, Ufuk Topcu
Computation and Language
Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling
Hyunji Lee, Wenhao Yu, Hongming Zhang et al. (+4 more)
Hybrid models that combine state space models (SSMs) with
attention mechanisms have shown strong performance by leveraging the efficiency of SSMs and the high recall ability of
attention. However, the architectural design choices behind these hybrid models remain insufficiently understood. In this work, we analyze hybrid architectures through the lens of
memory utilization and overall performance, and propose a complementary method to further enhance their effectiveness. We first examine the distinction between sequential and parallel integration of SSM and
attention layers. Our analysis reveals several interesting findings, including that sequential hybrids perform better on shorter contexts, whereas parallel hybrids are more effective for longer contexts. We also introduce a data-centric approach of continually training on
datasets augmented with paraphrases, which further enhances recall while preserving other capabilities. It generalizes well across different base models and outperforms architectural modifications aimed at enhancing recall. Our findings provide a deeper understanding of hybrid SSM-
attention models and offer practical guidance for designing architectures tailored to various use cases. Our findings provide a deeper understanding of hybrid SSM-
attention models and offer practical guidance for designing architectures tailored to various use cases.
Read the full abstract ▾
All authors Hyunji Lee, Wenhao Yu, Hongming Zhang, Kaixin Ma, Jiyeon Kim, Dong Yu, Minjoon Seo
Computation and Language · Cross-listed from cs.AI
The Denario project: Deep knowledge AI agents for scientific discovery
Francisco Villaescusa-Navarro, Boris Bolliet, Pablo Villanueva-Domingo et al. (+33 more)
We present Denario, an AI multi-
agent system designed to serve as a scientific research assistant. Denario can perform many different tasks, such as generating ideas, checking the literature, developing research plans, writing and executing code, making plots, and drafting and reviewing a scientific paper. The system has a modular architecture, allowing it to handle specific tasks, such as generating an idea, or carrying out end-to-end scientific analysis using Cmbagent as a deep-research backend. In this work, we describe in detail Denario and its modules, and illustrate its capabilities by presenting multiple AI-generated papers generated by it in many different scientific disciplines such as astrophysics, biology, biophysics, biomedical informatics, chemistry, material science, mathematical physics, medicine, neuroscience and planetary science. Denario also excels at combining ideas from different disciplines, and we illustrate this by showing a paper that applies methods from
quantum physics and machine learning to astrophysical data. We report the evaluations performed on these papers by domain experts, who provided both numerical scores and review-like feedback. We then highlight the strengths, weaknesses, and limitations of the current system. Finally, we discuss the ethical implications of AI-driven research and reflect on how such technology relates to the philosophy of science. We publicly release the code at https://github.com/AstroPilot-AI/Denario. A Denario demo can also be run directly on the web at https://huggingface.co/spaces/astropilot-ai/Denario, and the full app will be deployed on the cloud.
Read the full abstract ▾
All authors Francisco Villaescusa-Navarro, Boris Bolliet, Pablo Villanueva-Domingo, Adrian E. Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Urbano L. França, ChangHoon Hahn, Yan-Fei Jiang, Raul Jimenez, Jun-Young Lee, Antonio Lerario, Osman Mamun, Thomas Meier, Anupam A. Ojha, Pavlos Protopapas, Shimanto Roy, David N. Spergel, Pedro Tarancón-Álvarez, Ujjwal Tiwari, Matteo Viel, Digvijay Wadekar, Chi Wang, Bonny Y. Wang, Licong Xu, Yossi Yovel, Shuwen Yue, Wen-Han Zhou, Qiyao Zhu, Jiajun Zou, Íñigo Zubeldia
Computation and Language · Cross-listed from cs.IR
Evaluating Perspectival Biases in Cross-Modal Retrieval
Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant et al. (+6 more)
Multimodal retrieval systems are expected to operate in a semantic space, agnostic to the
language or cultural origin of the query. In practice, however,
retrieval outcomes systematically reflect perspectival biases: deviations shaped by linguistic prevalence and cultural associations. We introduce the Cross-Cultural, Cross-Modal, Cross-lingual
Multimodal (3XCM)
benchmark to isolate these effects. Results from our studies indicate that, for image-to-text
retrieval, models tend to favor entries from prevalent
languages over those that are semantically faithful. For text-to-image
retrieval, we observe a consistent "tugging effect" in the joint
embedding space between semantic
alignment and
language-conditioned cultural association. When semantic representations are insufficiently resolved, particularly in low-resource
languages, similarity is increasingly governed by culturally familiar visual patterns, leading to systematic association bias in
retrieval. Our findings suggest that achieving equitable
multimodal retrieval necessitates targeted strategies that explicitly decouple
language from culture, rather than relying solely on broader data exposure. This work highlights the need to treat linguistic and cultural biases as distinct, measurable challenges in
multimodal representation learning.
Read the full abstract ▾
All authors Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant, Chompakorn Chaksangchaichot, Patawee Prakrankamanant, Witthawin Sripheanpol, Pak Lovichit, Sarana Nutanong, Ekapol Chuangsuwanich
Computation and Language · Cross-listed from cs.AI
CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments
Lingyue Fu, Xin Ding, Linyue Pan et al. (+9 more)
Current evaluation for Large
Language Model (LLM) code
agents predominantly focus on generating functional code in single-turn scenarios, which fails to evaluate the
agent's capability for continuous code optimization and multi-turn iterative development. To bridge this gap, we introduce CATArena, a framework designed to evaluate the evolutionary capabilities of code
agents via iterative tournaments.
Agents engage in multi-turn tournaments and continuously refine their code through self-reflection and peer-learning based on comprehensive execution feedback. For evaluation, we propose a dual-metric system to decouple static generation proficiency from evolutionary potential. Extensive experiments reveal that an
agent's evolutionary potential is not strictly correlated with its initial proficiency. Our analysis further reveals that current
agents struggle to concurrently leverage both peer-learning and self-reflection for effective performance gains. Furthermore, the results validate CATArena's high extensibility and resistance to variance tasks, establishing it as a continuous and reliable standard for assessing the evolutionary capability of LLM code
agents.
Read the full abstract ▾
All authors Lingyue Fu, Xin Ding, Linyue Pan, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu
Computation and Language · Cross-listed from cs.CR
Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
Shaked Zychlinski and Yuval Kainan
Large
Language Models (LLMs) are susceptible to jailbreak attacks where malicious
prompts are disguised using ciphers and character-level encodings to bypass safety guardrails. While these guardrails often fail to interpret the encoded content, the underlying models can still process the harmful instructions. We introduce CPT-Filtering, a novel, model-agnostic with negligible-costs and near-perfect accuracy guardrail technique that aims to mitigate these attacks by leveraging the intrinsic behavior of Byte-Pair Encoding (BPE) tokenizers. Our method is based on the principle that tokenizers, trained on natural
language, represent out-of-distribution text, such as ciphers, using a significantly higher number of shorter
tokens. Our technique uses a simple yet powerful artifact of using
language models: the average number of Characters Per
Token (CPT) in the text. This approach is motivated by the high compute cost of modern methods - relying on added modules such as dedicated LLMs or perplexity models. We validate our approach across a large
dataset of over 100,000
prompts, testing numerous encoding schemes with several popular tokenizers. Our experiments demonstrate that a simple CPT threshold robustly identifies encoded text with high accuracy, even for very short inputs. CPT-Filtering provides a practical defense layer that can be immediately deployed for real-time text filtering and offline data curation.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CV
Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
Ziyu Guo, Xinyan Chen, Renrui Zhang et al. (+7 more)
Recent
video generation models can produce high-fidelity, temporally coherent
videos, indicating that they may encode substantial world knowledge. Beyond realistic synthesis, they also exhibit emerging behaviors indicative of visual perception, modeling, and manipulation. Yet, an important question still remains: Are
video models ready to serve as zero-shot reasoners in challenging visual
reasoning scenarios? In this work, we conduct an empirical study to comprehensively investigate this question, focusing on the leading and popular Veo-3. We evaluate its
reasoning behavior across 12 dimensions, including spatial,
geometric, physical, temporal, and
embodied logic, systematically characterizing both its strengths and failure modes. To standardize this study, we curate the evaluation data into MME-CoF, a compact
benchmark that enables in-depth and thorough assessment of Chain-of-Frame (CoF)
reasoning. Our findings reveal that while current
video models demonstrate promising
reasoning patterns on short-horizon spatial coherence, fine-grained grounding, and locally consistent dynamics, they remain limited in long-horizon causal
reasoning, strict
geometric constraints, and abstract logic. Overall, they are not yet reliable as standalone zero-shot reasoners, but exhibit encouraging signs as complementary visual engines alongside dedicated
reasoning models. Project page: https://
video-cof.github.io
Read the full abstract ▾
All authors Ziyu Guo, Xinyan Chen, Renrui Zhang, Ruichuan An, Yu Qi, Dongzhi Jiang, Xiangtai Li, Manyuan Zhang, Hongsheng Li, Pheng-Ann Heng
Computation and Language
Gistify! Codebase-Level Understanding via Runtime Execution
Hyunji Lee, Minseon Kim, Chinmay Singh et al. (+10 more)
As coding
agents are increasingly deployed in large codebases, the need to automatically design challenging, codebase-level evaluation is central. We propose Gistify, a task where a coding LLM must create a single, minimal, self-contained file that can reproduce a specific functionality of a codebase. The coding LLM is given full access to a codebase along with a specific entrypoint (e.g., a python command), and the generated file must replicate the output of the same command ran under the full codebase, while containing only the essential components necessary to execute the provided command. Success on Gistify requires both structural understanding of the codebase, accurate modeling of its execution flow as well as the ability to produce potentially large code patches. Our findings show that current state-of-the-art models struggle to reliably solve Gistify tasks, especially ones with long executions traces.
Read the full abstract ▾
All authors Hyunji Lee, Minseon Kim, Chinmay Singh, Matheus Pereira, Atharv Sonwane, Isadora White, Elias Stengel-Eskin, Mohit Bansal, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Lucas Caccia
Computation and Language · Cross-listed from cs.LG
Defeating the Training-Inference Mismatch via FP16
Penghui Qi, Zichen Liu, Xiangxin Zhou et al. (+4 more)
Reinforcement learning (RL) fine-tuning of large
language models (LLMs) often suffers from instability due to the numerical mismatch between the training and inference policies. While prior work has attempted to mitigate this issue through algorithmic corrections or engineering
alignments, we show that its root cause lies in the floating point precision itself. The widely adopted BF16, despite its large dynamic range, introduces large rounding errors that breaks the consistency between training and inference. In this work, we demonstrate that simply reverting to FP16 effectively eliminates this mismatch. The change is simple, fully supported by modern frameworks with only a few lines of code change, and requires no modification to the model architecture or learning
algorithm. Our results suggest that using FP16 uniformly yields more stable optimization, faster convergence, and stronger performance across diverse tasks,
algorithms and frameworks. We hope these findings motivate a broader reconsideration of precision trade-offs in RL fine-tuning.
Read the full abstract ▾
All authors Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin
Computation and Language · Cross-listed from cs.LG
Remote Labor Index: Measuring AI Automation of Remote Work
Mantas Mazeika, Alice Gatti, Cristina Menghini et al. (+44 more)
AIs have made rapid progress on research-oriented
benchmarks of knowledge and
reasoning, but it remains unclear how these gains translate into economic value and automation. To measure this, we introduce the Remote Labor Index (RLI), a broadly multi-sector
benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end
agent performance in practical settings. AI
agents perform near the floor on RLI, with the highest-performing
agent achieving an automation rate of 2.5%. These results help ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts and enabling stakeholders to proactively navigate AI-driven labor automation.
Read the full abstract ▾
All authors Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, Oliver Ingebretsen, Aditya Mehta, Jean Mottola, John Teichmann, Kevin Yu, Zaina Shaik, Adam Khoja, Richard Ren, Jason Hausenloy, Long Phan, Ye Htet, Ankit Aich, Tahseen Rabbani, Vivswan Shah, Andriy Novykov, Felix Binder, Kirill Chugunov, Luis Ramirez, Matias Geralnik, Hernán Mesura, Dean Lee, Ed-Yeremai Hernandez Cardona, Annette Diamond, Summer Yue, Alexandr Wang, Bing Liu, Ernesto Hernandez, Dan Hendrycks