Computation and Language
DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models
Alexander Sheppert
Large
Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and
Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard
benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate
benchmarks for model selection in professional applications.
Read the full abstract ▾
Computation and Language
Berta: an open-source, modular tool for AI-enabled clinical documentation
Samridhi Vaid, Mike Weldon, Jesse Dunn et al. (+8 more)
Commercial AI scribes cost \$99-600 per physician per month, operate as opaque systems, and do not return data to institutional infrastructure, limiting organizational control over data governance, quality improvement, and clinical workflows. We developed Berta, an open-source modular scribe platform for AI-enabled clinical documentation, and deployed a customized implementation within Alberta Health Services (AHS) integrated with their existing Snowflake AI Data Cloud infrastructure. The system combines automatic speech recognition with large language models while retaining all clinical data within the secure AHS environment. During eight months (November 2024 to July 2025), 198 emergency physicians used the system in 105 urban and rural facilities, generating 22148 clinical sessions and more than 2800 hours of audio. The use grew from 680 to 5530 monthly sessions. Operating costs averaged less than \$30 per physician per month, a 70-95% reduction compared to commercial alternatives. AHS has since approved expansion to 850 physicians. This is the first provincial-scale deployment of an AI scribe integrated with existing health system infrastructure. By releasing Berta as open source, we provide a reproducible, cost-effective alternative that health systems can adapt to their own secure environments, supporting data sovereignty and informed evaluation of AI documentation technology.
Read the full abstract ▾
All authors Samridhi Vaid, Mike Weldon, Jesse Dunn, Sacha Davis, Kevin Lonergan, Henry Li, Jeffrey Franc, Mohamed Abdalla, Daniel C. Baumgart, Jake Hayward, J Ross Mitchell
Computation and Language
S-Path-RAG: Semantic-Aware Shortest-Path Retrieval Augmented Generation for Multi-Hop Knowledge Graph Question Answering
Rong Fu, Yemin Wang, Tianxiang Xu et al. (+5 more)
We present S-Path-RAG, a semantic-aware shortest-path
Retrieval-Augmented Generation framework designed to improve multi-hop question answering over large knowledge graphs. S-Path-RAG departs from one-shot, text-heavy
retrieval by enumerating bounded-length, semantically weighted candidate paths using a hybrid weighted $k$-shortest, beam, and constrained random-walk strategy, learning a differentiable path scorer together with a contrastive path encoder and lightweight verifier, and injecting a compact soft mixture of selected path latents into a
language model via cross-
attention. The system runs inside an iterative Neural-Socratic Graph Dialogue loop in which concise diagnostic messages produced by the
language model are mapped to targeted graph edits or seed expansions, enabling adaptive
retrieval when the model expresses uncertainty. This combination yields a
retrieval mechanism that is both
token-efficient and
topology-aware while preserving interpretable path-level traces for diagnostics and intervention. We validate S-Path-RAG on standard multi-hop KGQA
benchmarks and through ablations and diagnostic analyses. The results demonstrate consistent improvements in answer accuracy, evidence coverage, and end-to-end efficiency compared to strong graph- and LLM-based baselines. We further analyze trade-offs between semantic weighting, verifier filtering, and iterative updates, and report practical recommendations for deployment under constrained compute and
token budgets.
Read the full abstract ▾
All authors Rong Fu, Yemin Wang, Tianxiang Xu, Yongtai Liu, Weizhi Tang, Wangyu Wu, Xiaowen Ma, Simon Fong
Computation and Language · Cross-listed from eess.AS
ProKWS: Personalized Keyword Spotting via Collaborative Learning of Phonemes and Prosody
Jianan Pan, Yuanming Zhang and Kejie Huang
Current keyword spotting systems primarily use phoneme-level matching to distinguish confusable words but ignore user-specific pronunciation traits like prosody (intonation, stress, rhythm). This paper presents ProKWS, a novel framework integrating fine-grained phoneme learning with personalized prosody modeling. We design a dual-stream encoder where one stream derives robust phonemic representations through contrastive learning, while the other extracts speaker-specific prosodic patterns. A collaborative fusion module dynamically combines phonemic and prosodic information, enhancing adaptability across acoustic environments. Experiments show ProKWS delivers highly competitive performance, comparable to state-of-the-art models on standard
benchmarks and demonstrates strong robustness for personalized keywords with tone and intent variations.
Read the full abstract ▾
Computation and Language · Cross-listed from eess.AS
PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting
Jianan Pan and Kejie Huang
As advancements in technologies like Internet of Things (IoT), Automatic
Speech Recognition (ASR), Speaker Verification (SV), and Text-to-
Speech (TTS) lead to increased usage of intelligent voice assistants, the demand for
privacy and personalization has escalated. In this paper, we introduce a multi-task learning framework for personalized, customizable open-vocabulary Keyword Spotting (PCOV-KWS). This framework employs a lightweight network to simultaneously perform Keyword Spotting (KWS) and SV to address personalized KWS requirements. We have integrated a training criterion distinct from softmax-based loss, transforming multi-class classification into multiple binary classifications, which eliminates inter-category competition, while an optimization strategy for multi-task loss weighting is employed during training. We evaluated our PCOV-KWS system in multiple
datasets, demonstrating that it outperforms the baselines in evaluation results, while also requiring fewer parameters and lower computational resources.
Read the full abstract ▾
Computation and Language
NERdME: a Named Entity Recognition Dataset for Indexing Research Artifacts in Code Repositories
Genet Asefa Gesese, Zongxiong Chen, Shufan Jiang et al. (+4 more)
Existing scholarly information extraction (SIE)
datasets focus on scientific papers and overlook implementation-level details in code repositories. README files describe
datasets, source code, and other implementation-level artifacts, however, their free-form Markdown offers little semantic structure, making automatic information extraction difficult. To address this gap, NERdME is introduced: 200 manually annotated README files with over 10,000 labeled spans and 10 entity types. Baseline results using large
language models and fine-tuned
transformers show clear differences between paperlevel and implementation-level entities, indicating the value of extending SIE
benchmarks with entity types available in README files. A downstream entity-linking experiment was conducted to demonstrate that entities derived from READMEs can support artifact discovery and metadata integration.
Read the full abstract ▾
All authors Genet Asefa Gesese, Zongxiong Chen, Shufan Jiang, Mary Ann Tan, Zhaotai Liu, Sonja Schimmler, Harald Sack
Computation and Language
CodeScout: Contextual Problem Statement Enhancement for Software Agents
Manan Suri, Xiangci Li, Mehdi Shojaie et al. (+5 more)
Current AI-powered code assistance tools often struggle with poorly-defined problem statements that lack sufficient task context and requirements specification. Recent analysis of software engineering
agents reveals that failures on such underspecified requests are highly correlated with longer trajectories involving either over-exploration or repeated attempts at applying the same fix without proper
evolution or testing, leading to suboptimal outcomes across software development tasks. We introduce CodeScout, a contextual query refinement approach that systematically converts underspecified user requests into comprehensive, actionable problem statements through lightweight pre-exploration of the target codebase. Our key innovation is demonstrating that structured analysis before task execution can supplement existing agentic capabilities without requiring any modifications to their underlying scaffolds. CodeScout performs targeted context scoping, conducts multi-perspective analysis examining potential fixes and exploration opportunities, then synthesizes these insights into enhanced problem statements with reproduction steps, expected behaviors, and targeted exploration hints. This pre-exploration directly addresses the identified failure patterns by reducing non-converging
agent trajectories while clarifying user intent in natural
language space. We evaluate CodeScout using state-of-the-art agentic scaffolds and
language models on SWEBench-Verified, demonstrating a 20% improvement in resolution rates with up to 27 additional issues resolved compared to the default baseline method. Our results suggest that systematic query refinement through contextual analysis represents a promising direction for enhancing AI code assistance capabilities.
Read the full abstract ▾
All authors Manan Suri, Xiangci Li, Mehdi Shojaie, Songyang Han, Chao-Chun Hsu, Shweta Garg, Aniket Anand Deshmukh, Varun Kumar
Computation and Language
Designing Explainable Conversational Agentic Systems for Guaraní Speakers
Samantha Adorno, Akshata Kishore Moharir and Ratna Kandala
Although artificial intelligence (AI) and Human-Computer Interaction (HCI) systems are often presented as universal solutions, their design remains predominantly text-first, underserving primarily oral
languages and indigenous communities. This position paper uses Guaraní, an official and widely spoken
language of Paraguay, as a case study to argue that
language support in AI remains insufficient unless it aligns with lived oral practices. We propose an alternative to the standard "text-to-
speech" pipeline, proposing instead an oral-first multi-
agent architecture. By decoupling Guaraní natural
language understanding from dedicated
agents for conversation state and community-led governance, we demonstrate a technical framework that respects indigenous data sovereignty and diglossia. Our work moves beyond mere recognition to focus on turn-taking, repair, and shared context as the primary locus of interaction. We conclude that for AI to be truly culturally grounded, it must shift from adapting oral
languages to text-centric systems to treating spoken conversation as a first-class design requirement, ensuring digital ecosystems empower rather than overlook diverse linguistic practices.
Read the full abstract ▾
Computation and Language
Structured Multidimensional Representation Learning for Large Language Models
Alaa El Ichi, Khalide Jbilou, Mohamed El Guide et al. (+1 more)
Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural
language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the
embedding dimension. In this work, we introduce a structured spectral factorization of the
embedding space based on the L-product for third-order tensors. By reshaping
token representations into spectral tensor slices and performing
attention and feed-forward operations in the transform domain, we obtain a Tensor
Transformer architecture that decomposes the encoder into p independent spectral sub-
transformers while preserving standard
Transformer semantics. We prove that the proposed L-
Transformer is spectrally equivalent to p parallel
Transformers operating on reduceddimensional
embeddings, which yields approximately 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total
embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over
embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AG~News show that the proposed model can substantially reduce encoder parameters (up to 75% for p=4) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AG~News at moderate width we observe a small accuracy decrease in exchange for a 4 times encoder reduction; at BERT-base width (d=768), performance returns to parity.
Read the full abstract ▾
All authors Alaa El Ichi, Khalide Jbilou, Mohamed El Guide, Franck Dufrenois
Computation and Language
Cultural Perspectives and Expectations for Generative AI: A Global Survey Approach
Erin van Liemt, Renee Shelby, Andrew Smart et al. (+5 more)
There is a lack of empirical evidence about global attitudes around whether and how GenAI should represent cultures. This paper assesses understandings and beliefs about culture as it relates to GenAI from a large-scale global survey. We gathered data about what culture means to different groups, and about how GenAI should approach the representation of cultural artifacts, concepts, or values. We distill working definitions of culture directly from these communities to build an understanding of its conceptual complexities and how they relate to representations in Generative AI. We survey from across parts of Europe, North and South America, Asia, and Africa. We conclude with a set of recommendations for Culture and GenAI development. These include participatory approaches, prioritizing specific cultural dimensions beyond geography, such as religion and tradition, and a sensitivity framework for addressing cultural “redlines”.
Read the full abstract ▾
All authors Erin van Liemt, Renee Shelby, Andrew Smart, Sinchana Kumbale, Richard Zhang, Neha Dixit, Qazi Mamunur Rashid, Jamila Smith-Loud
Computation and Language
Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis
Hazem Amamou, Stéphane Gagnon, Alan Davoust et al. (+1 more)
Retrieval-Augmented Generation (RAG) was introduced to enhance the capabilities of Large
Language Models (LLMs) beyond their encoded prior knowledge. This is achieved by providing LLMs with an external source of knowledge, which helps reduce factual
hallucinations and enables access to new information not available during pretraining. However, inconsistent retrieved information can negatively affect LLM responses. The
Retrieval-Augmented Generation
Benchmark (RGB) was introduced to evaluate the robustness of RAG systems under such conditions. In this work, we use the RGB corpus to evaluate LLMs in four scenarios: noise robustness, information integration, negative rejection, and counterfactual robustness. We perform a comparative analysis between the RGB RAG baseline and GraphRAG, a knowledge graph based
retrieval system. We test three GraphRAG customizations to improve robustness. Results show improvements over the RGB baseline and provide insights for designing more reliable RAG systems for real world scenarios.
Read the full abstract ▾
All authors Hazem Amamou, Stéphane Gagnon, Alan Davoust, Anderson R. Avila
Computation and Language · Cross-listed from cs.CE
Autonomous Algorithm Discovery for Ptychography via Evolutionary LLM Reasoning
Xiangyu Yin, Ming Du, Junjing Deng et al. (+3 more)
Ptychography is a computational imaging technique widely used for high-resolution materials characterization, but high-quality reconstructions often require the use of regularization functions that largely remain manually designed. We introduce Ptychi-Evolve, an autonomous framework that uses large
language models (LLMs) to discover and evolve novel regularization
algorithms. The framework combines LLM-driven code generation with evolutionary mechanisms, including semantically-guided crossover and mutation. Experiments on three challenging
datasets (X-ray integrated
circuits, low-dose electron microscopy of apoferritin, and multislice imaging with crosstalk artifacts) demonstrate that discovered regularizers outperform conventional reconstructions, achieving up to +0.26 SSIM and +8.3~dB PSNR improvements. Besides, Ptychi-Evolve records
algorithm lineage and
evolution metadata, enabling interpretable and reproducible analysis of discovered regularizers.
Read the full abstract ▾
All authors Xiangyu Yin, Ming Du, Junjing Deng, Zhi Yang, Yimo Han, Yi Jiang
Computation and Language
FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisation
Hung Nguyen Huy, Mo El-Haj, Dawn Knight et al. (+1 more)
FreeTxt-Vi is a free and open source web based toolkit for creating and analysing bilingual Vietnamese English text collections. Positioned at the intersection of corpus linguistics and natural
language processing NLP it enables users to build explore and interpret free text data without requiring programming expertise. The system combines corpus analysis features such as concordancing keyword analysis word relation exploration and interactive visualisation with
transformer based NLP components for sentiment analysis and summarisation. A key contribution of this work is the design of a unified bilingual NLP pipeline that integrates a hybrid VnCoreNLP and Byte Pair Encoding BPE
segmentation strategy a fine tuned TabularisAI sentiment classifier and a fine tuned Qwen2.5 model for abstractive summarisation. Unlike existing text analysis platforms FreeTxt Vi is evaluated as a set of
language processing components. We conduct a three part evaluation covering
segmentation sentiment analysis and summarisation and show that our approach achieves competitive or superior performance compared to widely used baselines in both Vietnamese and English. By reducing technical barriers to
multilingual text analysis FreeTxt Vi supports reproducible research and promotes the development of
language resources for Vietnamese a widely spoken but underrepresented
language in NLP. The toolkit is applicable to domains including education digital humanities cultural heritage and the social sciences where qualitative text data are common but often difficult to process at scale.
Read the full abstract ▾
All authors Hung Nguyen Huy, Mo El-Haj, Dawn Knight, Paul Rayson
Computation and Language
The Fragility Of Moral Judgment In Large Language Models
Tom van Nuenen and Pratik S. Sachdeva
People increasingly use large
language models (LLMs) for everyday moral and interpersonal guidance, yet these systems cannot interrogate missing context and judge dilemmas as presented. We introduce a perturbation framework for testing the stability and manipulability of LLM moral judgments while holding the underlying moral conflict constant. Using 2,939 dilemmas from r/AmItheAsshole (January-March 2025), we generate three families of content perturbations: surface edits (lexical/structural noise), point-of-view shifts (voice and stance neutralization), and persuasion cues (self-positioning, social proof, pattern admissions, victim framing). We also vary the evaluation protocol (output ordering, instruction placement, and unstructured prompting). We evaluated all variants with four models (GPT-4.1, Claude 3.7 Sonnet, DeepSeek V3, Qwen2.5-72B) (N=129,156 judgments). Surface perturbations produce low flip rates (7.5%), largely within the self-consistency noise floor (4-13%), whereas point-of-view shifts induce substantially higher instability (24.3%). A large subset of dilemmas (37.9%) is robust to surface noise yet flips under perspective changes, indicating that models condition on narrative voice as a pragmatic cue. Instability concentrates in morally ambiguous cases; scenarios where no party is assigned blame are most susceptible. Persuasion perturbations yield systematic directional shifts. Protocol choices dominate all other factors: agreement between structured protocols is only 67.6% (kappa=0.55), and only 35.7% of model-scenario units match across all three protocols. These results show that LLM moral judgments are co-produced by narrative form and task scaffolding, raising reproducibility and equity concerns when outcomes depend on presentation skill rather than moral substance.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.RO
RACAS: Controlling Diverse Robots With a Single Agentic System
Dylan R. Ashley, Jan Przepióra, Yimeng Chen et al. (+5 more)
Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (
Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a
Memory Curator) communicate exclusively through natural
language to provide closed-loop
robot control. RACAS requires only a natural
language description of the
robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground
robot, a recently published novel multi-jointed robotic limb, and an
underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.
Read the full abstract ▾
All authors Dylan R. Ashley, Jan Przepióra, Yimeng Chen, Ali Abualsaud, Nurzhan Yesmagambet, Shinkyu Park, Eric Feron, Jürgen Schmidhuber
Computation and Language
Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
Patrick Ahrend, Tobias Eder, Xiyang Yang et al. (+2 more)
Chain-of-Thought (CoT) prompting improves LLM
reasoning but can increase
privacy risk by resurfacing personally identifiable information (PII) from the
prompt into
reasoning traces and outputs, even under policies that instruct the model not to restate PII. We study such direct, inference-time PII leakage using a model-agnostic framework that (i) defines leakage as risk-weighted,
token-level events across 11 PII types, (ii) traces leakage curves as a function of the allowed CoT budget, and (iii) compares open- and closed-source model families on a structured PII
dataset with a hierarchical risk taxonomy. We find that CoT consistently elevates leakage, especially for high-risk categories, and that leakage is strongly family- and budget-dependent. Increasing the
reasoning budget can either amplify or attenuate leakage depending on the base model. We then
benchmark lightweight inference-time gatekeepers: a rule-based detector, a TF-IDF + logistic regression classifier, a GLiNER-based NER model, and an LLM-as-judge, using risk-weighted F1, Macro-F1, and recall. No single method dominates across models or budgets, motivating hybrid, style-adaptive gatekeeping policies that balance utility and risk under a common, reproducible protocol.
Read the full abstract ▾
All authors Patrick Ahrend, Tobias Eder, Xiyang Yang, Zhiyi Pan, Georg Groh
Computation and Language
NOTAI.AI: Explainable Detection of Machine-Generated Text via Curvature and Feature Attribution
Oleksandr Marchenko Breneur, Adelaide Danilov, Aria Nourbakhsh et al. (+1 more)
We present NotAI.AI, an explainable AI-generated text detection system. Instead of returning only a binary label or confidence score, the system shows which signals influenced the prediction and lets users inspect an attribution-based sensitivity estimate obtained by subtracting selected local contributions. NotAI.AI combines sentence-level conditional probability
curvature, a neural detector score, and interpretable stylometric and readability features in an XGBoost meta-classifier. It explains predictions with TreeSHAP feature contributions and can turn the resulting evidence into a concise natural-
language explanation. We evaluate the system on a category-balanced subset of RAID containing human-written, clean AI-generated, and attacked AI-generated texts. The full model outperforms variants based on individual feature families, reaching 0.9685 F1 on the held-out within-subset test split. In an automatic evaluation, two model judges rate 94.5-98.6% of generated explanations as faithful to the supplied detector evidence. The web interface (https://notai-ai.vercel.app), source code (https://github.com/Oleksandr-MB/EMNLP2026DEMO_NotAI.AI), and demonstration
video(https://youtu.be/l8Nk8kdBTHQ) are publicly available.
Read the full abstract ▾
All authors Oleksandr Marchenko Breneur, Adelaide Danilov, Aria Nourbakhsh, Salima Lamsiyah
Computation and Language · Cross-listed from cs.IR
CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain
Hung Nguyen, Hans Moen and Pekka Marttinen
Extracting insights from Electronic Health Record (EHR) databases often requires SQL expertise, creating a barrier for clinical decision-making and research. A promising approach is to use Large
Language Models (LLMs) to translate natural
language questions into SQL through
Retrieval-Augmented Generation (RAG), where relevant question-SQL examples are retrieved to generate new queries via few-shot learning. However, adapting this method to the medical domain is non-trivial, as effective
retrieval requires examples that align with both the logical structure of the question and its referenced entities (e.g., drug names, procedure titles). Standard single-step RAG struggles to optimize both aspects simultaneously and often relies on near-exact matches to generalize effectively. This issue is especially severe in healthcare, as questions often contain noisy and inconsistent medical jargon. To address this, we present CBR-to-SQL, a framework inspired by Case-based
Reasoning theory that decomposes RAG's single-step
retrieval into two explicit stages: one that focuses on retrieving structurally relevant examples, and one that aligns entities with the target database schema. Evaluated on two clinical
benchmarks, CBR-to-SQL achieves competitive accuracies compared to fine-tuned methods. More importantly, it demonstrates considerably higher sample efficiency and robustness than the standard RAG approach, particularly under data scarcity and
retrieval perturbations.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment
Xiang Ma, Lexin Fang, Litian Xu et al. (+1 more)
Cross-modal
alignment is a crucial task in
multimodal learning aimed at achieving semantic consistency between
vision and
language. This requires that image-text pairs exhibit similar semantics. Traditional
algorithms pursue
embedding consistency to achieve semantic consistency, ignoring the non-semantic information present in the
embedding. An intuitive approach is to decouple the
embeddings into semantic and modality components, aligning only the semantic component. However, this introduces two main challenges: (1) There is no established standard for distinguishing semantic and modal information. (2) The modality gap can cause semantic
alignment deviation or information loss. To align the true semantics, we propose a novel cross-modal
alignment algorithm via Constrained Decoupling and Distribution Sampling (CDDS). Specifically, (1) A dual-path UNet is introduced to adaptively decouple the
embeddings, applying multiple constraints to ensure effective separation. (2) A distribution sampling method is proposed to bridge the modality gap, ensuring the rationality of the
alignment process. Extensive experiments on various
benchmarks and model backbones demonstrate the superiority of CDDS, outperforming state-of-the-art methods by 6.6% to 14.2%.
Read the full abstract ▾
All authors Xiang Ma, Lexin Fang, Litian Xu, Caiming Zhang
Computation and Language · Cross-listed from cs.SE
EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair
Jiaao Chen, Jingyuan Qi, Mingye Gao et al. (+3 more)
Function-calling
agents – large
language models that invoke tools and APIs – require high-quality, domain-specific training data spanning executable environments, backing databases, and diverse multi-turn trajectories. We introduce EigenData, an integrated, self-evolving platform that automates the full data lifecycle through a multi-
agent architecture. A top-level orchestrator, EigenCore, coordinates three specialized sub-systems: DatabaseAgent for realistic domain database construction, CodingAgent for verified executable environment generation with iterative test-debug loops, and DataAgent for multi-turn trajectory synthesis with self-evolving
prompt optimization. Cross-component feedback ensures consistency across all artifacts. We apply EigenData to audit and repair the Berkeley Function-Calling Leaderboard (BFCL-V3), identifying systematic errors in function schemas, implementations, and reference trajectories, automatically correcting them through coordinated schema refinement, code-level bug fixes, and trajectory modification, and introducing an outcome-aware evaluation protocol that assesses task success via database-state correctness rather than turn-level trajectory matching. We demonstrate that the repaired
benchmark, coupled with outcome-aware metrics, produces model rankings substantially better correlated with human judgments of functional correctness.
Read the full abstract ▾
All authors Jiaao Chen, Jingyuan Qi, Mingye Gao, Wei-Chen Wang, Hanrui Wang, Di Jin
Computation and Language · Cross-listed from cs.LG
POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
Zeju Qiu, Lixin Liu, Adrian Weller et al. (+2 more)
Efficient and stable training of large
language models (LLMs) remains a core challenge in modern machine learning systems. To address this challenge, Reparameterized Orthogonal Equivalence Training (POET), a spectrum-preserving framework that optimizes each weight
matrix through orthogonal equivalence transformation, has been proposed. Although POET provides strong training stability, its original implementation incurs high
memory consumption and computational overhead due to intensive
matrix multiplications. To overcome these limitations, we introduce POET-X, a scalable and
memory-efficient variant that performs orthogonal equivalence transformations with significantly reduced computational cost. POET-X maintains the generalization and stability benefits of POET while achieving substantial improvements in throughput and
memory efficiency. In our experiments, POET-X enables the pretraining of billion-parameter LLMs on a single Nvidia H100 GPU, and in contrast, standard optimizers such as AdamW run out of
memory under the same settings.
Read the full abstract ▾
All authors Zeju Qiu, Lixin Liu, Adrian Weller, Han Shi, Weiyang Liu
Computation and Language · Cross-listed from cs.AI
The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
Shangwen Sun, Alfredo Canziani, Yann LeCun et al. (+1 more)
We study two recurring phenomena in
Transformer language models: massive activations, in which a small number of
tokens exhibit extreme outliers in a few channels, and
attention sinks, in which certain
tokens attract disproportionate
attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same
tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern
Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model.
Attention sinks operate locally: they modulate
attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.
Read the full abstract ▾
All authors Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu
Computation and Language · Cross-listed from cs.LG
Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation
Helena Casademunt, Bartosz Cywiński, Khoi Tran et al. (+3 more)
Large
language models sometimes produce false or misleading responses. Two approaches to this problem are honesty elicitation – modifying
prompts or weights so that the model answers truthfully – and lie detection – classifying whether a given response is false. Prior work evaluates such methods on models specifically trained to lie or conceal information, but these artificial constructions may not resemble naturally-occurring dishonesty. We instead study open-weights LLMs from Chinese developers, which are trained to censor politically sensitive topics: Qwen3 models frequently produce falsehoods about subjects like Falun Gong or the Tiananmen protests while occasionally answering correctly, indicating they possess knowledge they are trained to suppress. Using this as a testbed, we evaluate a suite of elicitation and lie detection techniques. For honesty elicitation, sampling without a chat template, few-shot prompting, and fine-tuning on generic honesty data most reliably increase truthful responses. For lie detection, prompting the censored model to classify its own responses performs near an uncensored-model upper bound, and linear probes trained on unrelated data offer a cheaper alternative. The strongest honesty elicitation techniques also transfer to frontier open-weights models including DeepSeek R1. Notably, no technique fully eliminates false responses. We release all
prompts, code, and transcripts.
Read the full abstract ▾
All authors Helena Casademunt, Bartosz Cywiński, Khoi Tran, Arya Jakkli, Samuel Marks, Neel Nanda
Computation and Language
Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Siddharth Boppana, Annabel Ma, Max Loeffler et al. (+5 more)
We provide evidence of performative chain-of-thought (CoT) in
reasoning models, where a model becomes strongly confident in its final answer, but continues generating
tokens without revealing its internal belief. Our analysis compares activation probing, early forced answering, and a CoT monitor across two large models (DeepSeek-R1 671B & GPT-OSS 120B) and find task difficulty-specific differences: The model's final answer is decodable from activations far earlier in CoT than a monitor is able to say, especially for easy recall-based MMLU questions. We contrast this with genuine
reasoning in difficult multihop GPQA-Diamond questions. Despite this, inflection points (e.g., backtracking, 'aha' moments) occur almost exclusively in responses where probes show large belief shifts, suggesting these behaviors track genuine uncertainty rather than learned "
reasoning theater." Finally, probe-guided early exit reduces
tokens by up to 80% on MMLU and 30% on GPQA-Diamond with similar accuracy, positioning
attention probing as an efficient tool for detecting performative
reasoning and enabling adaptive computation.
Read the full abstract ▾
All authors Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo
Computation and Language
Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval
Artem Vazhentsev, Maria Marina, Daniil Moskovskiy et al. (+8 more)
Trustworthiness is a core research challenge for agentic AI systems built on Large
Language Models (LLMs). To enhance trust, natural
language claims from diverse sources, including human-written text, web content, and model outputs, are commonly checked for factuality by retrieving external knowledge and using an LLM to verify the faithfulness of claims to the retrieved evidence. As a result, such methods are constrained by
retrieval errors and external data availability, while leaving the models intrinsic fact-verification capabilities largely unused. We propose the task of fact-checking without
retrieval, focusing on the verification of arbitrary natural
language claims, independent of their source. To study this setting, we introduce a comprehensive evaluation framework focused on generalization, testing robustness to (i) long-tail knowledge, (ii) variation in claim sources, (iii) multilinguality, and (iv) long-form generation. Across 9
datasets, 18 methods and 3 models, our experiments indicate that logit-based approaches often underperform compared to those that leverage internal model representations. Building on this finding, we introduce INTRA, a method that exploits interactions between internal representations and achieves state-of-the-art performance with strong generalization. More broadly, our work establishes fact-checking without
retrieval as a promising research direction that can complement
retrieval-based frameworks, improve scalability, and enable the use of such systems as reward signals during training or as components integrated into the generation process.
Read the full abstract ▾
All authors Artem Vazhentsev, Maria Marina, Daniil Moskovskiy, Sergey Pletenev, Mikhail Seleznyov, Mikhail Salnikov, Elena Tutubalina, Vasily Konovalov, Irina Nikishina, Alexander Panchenko, Viktor Moskvoretskii
Computation and Language
NCTB-QA: A Large-Scale Bangla Educational Question Answering Dataset and Benchmarking Performance
Abrar Eyasir, Tahsin Ahmed and Muhammad Ibrahim
Reading comprehension systems for low-resource
languages face significant challenges in handling unanswerable questions. These systems tend to produce unreliable responses when correct answers are absent from context. To solve this problem, we introduce NCTB-QA, a large-scale Bangla question answering
dataset comprising 87,805 question-answer pairs extracted from 50 textbooks published by Bangladesh's National Curriculum and Textbook Board. Unlike existing Bangla
datasets, NCTB-QA maintains a balanced distribution of answerable (57.25%) and unanswerable (42.75%) questions. NCTB-QA also includes adversarially designed instances containing plausible distractors. We
benchmark three
transformer-based models (BERT, RoBERTa, ELECTRA) and demonstrate substantial improvements through fine-tuning. BERT achieves 313% relative improvement in F1 score (0.150 to 0.620). Semantic answer quality measured by BERTScore also increases significantly across all models. Our results establish NCTB-QA as a challenging
benchmark for Bangla educational question answering. This study demonstrates that domain-specific fine-tuning is critical for robust performance in low-resource settings.
Read the full abstract ▾
Computation and Language
DEBISS: a Corpus of Individual, Semi-structured and Spoken Debates
Klaywert Danillo Ferreira de Souza, David Eduardo Pereira, Cláudio E. C. Campelo et al. (+1 more)
The process of debating is essential in our daily lives, whether in studying, work activities, simple everyday discussions, political debates on TV, or online discussions on social networks. The range of uses for debates is broad. Due to the diverse applications, structures, and formats of debates, developing corpora that account for these variations can be challenging, and the scarcity of debate corpora in the state of the art is notable. For this reason, the current research proposes the DEBISS corpus: a collection of spoken and individual debates with semi-structured features. With a broad range of NLP task annotations, such as
speech-to-text, speaker diarization, argument mining, and debater quality assessment.
Read the full abstract ▾
All authors Klaywert Danillo Ferreira de Souza, David Eduardo Pereira, Cláudio E. C. Campelo, Larissa Lucena Vasconcelos
Computation and Language
FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
Ted Zadouri, Markus Hoehnerbach, Jay Shah et al. (+3 more)
Attention, as a core layer of the ubiquitous
Transformer architecture, is the bottleneck for large
language models and long-context applications. While FlashAttention-3 optimized
attention for Hopper GPUs through asynchronous execution and warp specialization, it primarily targets the H100 architecture. The AI industry has rapidly transitioned to deploying Blackwell-based systems such as the B200 and GB200, which exhibit fundamentally different performance characteristics due to asymmetric hardware scaling: tensor core throughput doubles while other functional units (shared
memory bandwidth, exponential units) scale more slowly or remain unchanged. We develop several techniques to address these shifting bottlenecks on Blackwell GPUs: (1) redesigned pipelines that exploit fully asynchronous MMA operations and larger tile sizes, (2) software-emulated exponential and conditional softmax rescaling that reduces non-matmul operations, and (3) leveraging tensor
memory and the 2-CTA MMA mode to reduce shared
memory traffic and atomic adds in the backward pass. We demonstrate that our method, FlashAttention-4, achieves up to 1.3$\times$ speedup over cuDNN 9.13 and 2.7$\times$ over Triton on B200 GPUs with BF16, reaching up to 1613 TFLOPs/s (71% utilization). Beyond algorithmic innovations, we implement FlashAttention-4 entirely in CuTe-DSL embedded in Python, achieving 20-30$\times$ faster compile times compared to traditional C++ template-based approaches while maintaining full expressivity.
Read the full abstract ▾
All authors Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, Vijay Thakkar, Tri Dao
Computation and Language · Cross-listed from cs.AI
Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry
Yifan Zhu, Mariah Bradford, Kenneth Lai et al. (+4 more)
Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in
multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich
multimodal communication under epistemic asymmetry. We present a
multimodal dataset of these interactions, annotated and temporally aligned across
speech, gesture, and action modalities to support
reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large
language models (LLMs), prompted to infer shared beliefs from
multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.
Read the full abstract ▾
All authors Yifan Zhu, Mariah Bradford, Kenneth Lai, Timothy Obiso, Videep Venkatesha, James Pustejovsky, Nikhil Krishnaswamy