Computation and Language
James Pustejovsky
Distributional and neural approaches to natural
language semantics have been built almost exclusively on conventional linear
algebra: vectors,
matrices, tensors, and the operations that accompany them. These methods have achieved remarkable empirical success, yet they face persistent structural limitations in compositional semantics, type sensitivity, and interpretability. I argue in this paper that
geometric algebra (GA) – specifically, Clifford
algebras – provides a mathematically superior foundation for semantic representation, and that a Functional
Geometric Algebra (FGA) framework extends GA toward a typed, compositional semantics capable of supporting inference, transformation, and interpretability while retaining full compatibility with distributional learning and modern neural architectures. I develop the formal foundations, identify three core capabilities that GA provides and linear
algebra does not, present a detailed worked example illustrating operator-level semantic contrasts, and show how GA-based operations already implicit in current
transformer architectures can be made explicit and extended. The central claim is not merely increased dimensionality but increased structural organization: GA expands an $n$-dimensional
embedding space into a $2^n$ multivector
algebra where base semantic concepts and their higher-order interactions are represented within a single, principled algebraic framework.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.CY
Three Models of RLHF Annotation: Extension, Evidence, and Authority
Steve Coyne
Preference-based
alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large
language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.
Read the full abstract ▾
Computation and Language
Why Are Some Emotions Harder for LLMs? Uncovering the Causal Mechanisms of Emotion Inference via Sparse Autoencoders
Bangzhao Shu, Arinjay Singh and Mai ElSherief
Large
language models (LLMs) are increasingly used in emotionally sensitive human-AI applications, where reliable
emotion detection is essential. However, their
emotion recognition abilities remain uneven: models often perform well on some
emotions while consistently struggling with others. Although recent work has explored
emotion mechanisms in LLMs, little is known about why models are weaker on some
emotions than others from a mechanistic interpretability perspective. In this work, we investigate
emotion-specific biases through the causal mechanisms of
emotion inference using sparse autoencoders (SAEs). We systematically identify causal sparse
emotion features that drive
emotion inference and analyze their sparse causal organization within and across
emotions. We show that some
emotions, such as surprise and fear, rely on highly concentrated feature sets, whereas disgust exhibits a more distributed sparse causal organization: its causal features are generally weaker, frequently co-activate with features for other
emotions, and are often overshadowed by causal features for anger. These representational differences provide a mechanistic explanation for why LLMs struggle more with certain
emotions. Finally, we conduct two intervention experiments: targeted steering of weaker causal features to mitigate
emotion-specific failures, and global optimization of a steering vector over the identified causal features to improve overall
emotion recognition performance.
Read the full abstract ▾
Computation and Language
Luminol-AIDetect: Fast Zero-shot Machine-Generated Text Detection based on Perplexity under Text Shuffling
Lucio La Cava and Andrea Tagarelli
Machine-generated text (MGT) detection requires identifying structurally invariant signals across generation models, rather than relying on model-specific fingerprints. In this respect, we hypothesize that while large
language models excel at local semantic consistency, their autoregressive nature results in a specific kind of structural fragility compared to human writing. We propose Luminol-AIDetect, a novel, zero-shot statistical approach that exposes this fragility through coherence disruption. By applying a simple randomized text-shuffling procedure, we demonstrate that the resulting shift in perplexity serves as a principled, model-agnostic discriminant, as MGT displays a characteristic dispersion in perplexity-under-shuffling that differs markedly from the more stable structural variability of human-written text. Luminol-AIDetect leverages this distinction to inform its decision process, where a handful of perplexity-based scalar features are extracted from an input text and its shuffled version, then detection is performed via density estimation and ensemble-based prediction. Evaluated across 8 content domains, 11
adversarial attack types, and 18
languages, Luminol-AIDetect demonstrates state-of-the-art performance, with gains up to 17x lower FPR while being cheaper than prior methods.
Read the full abstract ▾
Computation and Language
G-Loss: Graph-Guided Fine-Tuning of Language Models
Aditya Sharma, Vinti Agarwal and Rajesh Kumar
Traditional loss functions, including cross-
entropy, contrastive, triplet, and su pervised contrastive losses, used for fine-tuning pre-trained
language models such as BERT, operate only within local neighborhoods and fail to account for the global semantic structure. We present G-Loss, a graph-guided loss function that incorporates semi-supervised label propagation to use structural relationships within the
embedding manifold. G-Loss builds a document-similarity graph that captures global semantic relationships, thereby guiding the model to learn more discriminative and robust
embeddings. We evaluate G-Loss on five
benchmark datasets covering key downstream classification tasks: MR (sentiment analysis), R8 and R52 (topic categorization), Ohsumed (medical document classification), and 20NG (news categorization). In the majority of experimental setups, G-Loss converges faster and produces semantically coherent
embedding spaces, resulting in higher classification accuracy than models fine-tuned with traditional loss functions.
Read the full abstract ▾
Computation and Language
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Jiahang Lin, Shichun Liu, Chengjun Pan et al. (+8 more)
Harnesses are now central to coding-
agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory
tokens into a layered, drill-down evidence corpus that an evolving
agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness
evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-
evolution: on SWE-bench-verified it tops aggregate success at 12% fewer
tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than
benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term
memory rather than the system
prompt, suggesting factual harness structure transfers while prose-level strategy does not.
Read the full abstract ▾
All authors Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, Yu-Gang Jiang
Computation and Language
PSI-Bench: Interpretable and Clinically Meaningful Evaluation of Depression Patient Simulators
Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu et al. (+4 more)
Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified
prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically meaningful diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we
benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, and move through therapeutic stages and toward positive valence too quickly. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our
benchmark is strongly aligned with judgments of mental health professionals. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible
benchmark to guide future simulator design and evaluation.
Read the full abstract ▾
All authors Nguyen Khoi Hoang, Shuhaib Mehri, Tse-An Hsu, Yi-Jyun Sun, Quynh Xuan Nguyen Truong, Khoa D Doan, Dilek Hakkani-Tür
Computation and Language
MAIC-UI: Making Interactive Courseware with Generative UI
Shangqing Tu, Yanjia Li, Keyu Chen et al. (+7 more)
Creating interactive STEM courseware traditionally requires HTML/CSS/JavaScript expertise, leaving barriers for educators. While generative AI can produce HTML codes, existing tools generate static presentations rather than interactive simulations, struggle with long documents, and lack pedagogical accuracy mechanisms. Furthermore, full regeneration for modifications requires 200–600 seconds, disrupting creative flow. We present MAIC-UI, a zero-code authoring system that enables educators to create and rapidly edit interactive courseware from textbooks, PPTs, and PDFs. MAIC-UI employs: (1) structured knowledge analysis with multi-modal understanding to ensure pedagogical rigor; (2) a two-stage generate-verify-optimize pipeline separating content
alignment from visual refinement; and (3) Click-to-Locate editing with Unified Diff-based incremental generation achieving sub-10-second iteration cycles. A controlled lab study with 40 participants shows MAIC-UI reduces editing iterations (4.9 vs. 7.0) and significantly improves learnability and controllability compared to direct Text-to-HTML generation. A three-month classroom deployment with 53 high school students demonstrates that MAIC-UI fosters learning agency and reduces outcome disparities – the pilot class achieved 9.21-point gains in STEM subjects compared to -2.32 points in control classes. Our code is available at https://github.com/THU-MAIC/MAIC-UI.
Read the full abstract ▾
All authors Shangqing Tu, Yanjia Li, Keyu Chen, Sichen Zhang, Jifan Yu, Daniel Zhang-Li, Lei Hou, Juanzi Li, Yu Zhang, Huiqin Liu
Computation and Language · Cross-listed from cs.LG
Barriers to Universal Reasoning With Transformers (And How to Overcome Them)
Oliver Kraus, Yash Sarrof, Yuekun Yao et al. (+2 more)
Chain-of-Thought (CoT) has been shown to empirically improve
Transformers' performance, and theoretically increase their expressivity to Turing completeness. However, whether
Transformers can learn to generalize to CoT traces longer than those seen during training is understudied. We use recent theoretical frameworks for
Transformer length generalization and find that – under standard positional encodings and a finite alphabet –
Transformers with CoT cannot solve problems beyond $TC^0$, i.e. the expressivity benefits do not hold under the stricter requirement of length-generalizable learnability. However, if we allow the vocabulary to grow with problem size, we attain a length-generalizable simulation of Turing machines where the CoT trace length is linear in the simulated runtime up to a constant. Our construction overcomes two core obstacles to reliable length generalization: repeated copying and last-occurrence
retrieval. We assign each tape position a unique signpost
token, and log only value changes to enable recovery of the current tape symbol through counts circumventing both barriers. Further, we empirically show that the use of such signpost
tokens and value change encodings provide actionable guidance to improve length generalization on hard problems.
Read the full abstract ▾
All authors Oliver Kraus, Yash Sarrof, Yuekun Yao, Alexander Koller, Michael Hahn
Computation and Language
Subliminal Steering: Stronger Encoding of Hidden Signals
George Morgulis and John Hewitt
Subliminal learning describes a student
language model inheriting a behavioral bias by fine-tuning on seemingly innocuous data generated by a biased teacher model. Prior work has begun to characterize this phenomenon but leaves open questions about the scope of signals it can transfer, the mechanisms that explain it, and the precision with which a bias can be encoded. We tackle these problems by introducing subliminal steering, a variant of subliminal learning in which the teacher's bias is implemented not via a system
prompt, as in prior work, but through a steering vector trained to maximize the likelihood of a set of target samples. First, we show that subliminal steering transfers complex multi-word biases, whereas prior work focused on single-word preferences, demonstrating a large scope of subliminally transferable signals. Moreover, the transfer is reliable enough to appear in settings previously thought not to exhibit subliminal learning, including plain SGD (not just Adam), full fine-tuning (not just LoRA), and models such as Llama and Phi. Second, we provide mechanistic evidence that subliminal learning transfers not only the target behavioral bias, but also the steering vector itself, localized to the layers at which the teacher was steered. Finally, we show that the bias is encoded with such precision that a new steering vector trained on the subliminally-laden
dataset attains high cosine similarity with the original vector.
Read the full abstract ▾
Computation and Language
Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research
Taryn Wong, Zeerak Talat, Hanan Aldarmaki et al. (+1 more)
Critical analyses of
emotion recognition technology have raised ethical concerns around task validity and potential downstream impacts, urging researchers to ensure
alignment between their stated motivations and practice. However, these discussions have not adequately influenced or drawn from research on
speech emotion recognition (SER). We address this gap by conducting a systematic survey of SER research to uncover what stated motivations drive this work and if they align with the
datasets and
emotions studied. We find that while SER research identifies appealing goals, such as well-situated voice-activated systems or healthcare applications, commonly-used
datasets do not reflect these proposed deployment contexts, thus presenting a gap between motivations and research practices. We argue that such gaps engender ethical concerns, and that SER research should reassert itself with concrete use-cases to prevent misinterpretations, misuse, and downstream harms.
Read the full abstract ▾
All authors Taryn Wong, Zeerak Talat, Hanan Aldarmaki, Anjalie Field
Computation and Language
CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
Wei-Chun Chen, Yu-Xuan Chen, I-Fang Chung et al. (+1 more)
Accurate nutrient estimation from unstructured recipe text is an important yet challenging problem in dietary monitoring, due to ambiguous ingredient terminology and highly variable quantity expressions. We systematically evaluate models spanning a wide range of representational capacity, from lexical matching methods (TF-IDF with Ridge Regression), to deep semantic encoders (DeBERTa-v3), to generative
reasoning with large
language models (LLMs). Under the strict tolerance criteria defined by EU Regulation 1169/2011, our empirical results reveal a clear trade-off between predictive accuracy and computational efficiency. The TF-IDF baseline achieves moderate nutrient estimation performance with near-instantaneous inference, whereas the DeBERTa-v3 encoder performs poorly under task-specific data scarcity. In contrast, few-shot LLM inference (e.g., Gemini 2.5 Flash) and a hybrid LLM refinement pipeline (TF-IDF combined with Gemini 2.5 Flash) deliver the highest validation accuracy across all nutrient categories. These improvements likely arise from the ability of LLMs to leverage pre-trained world knowledge to resolve ambiguous terminology and normalize non-standard units, which remain difficult for purely lexical approaches. However, these gains come at the cost of substantially higher inference latency, highlighting a practical deployment trade-off between real-time efficiency and nutritional precision in dietary monitoring systems.
Read the full abstract ▾
All authors Wei-Chun Chen, Yu-Xuan Chen, I-Fang Chung, Ying-Jia Lin
Computation and Language · Cross-listed from cs.CV
Toward Multimodal Conversational AI for Age-Related Macular Degeneration
Ran Gu, Benjamin Hou, Mélanie Hébert et al. (+5 more)
Despite strong performance of deep learning models in retinal disease detection, most systems produce static predictions without clinical
reasoning or interactive explanation. Recent advances in
multimodal large
language models (MLLMs) integrate diagnostic predictions with clinically meaningful dialogue to support clinical decision-making and patient counseling. In this study, OcularChat, an MLLM, was fine-tuned from Qwen2.5-VL using simulated patient-physician dialogues to diagnose age-related macular degeneration (AMD) through visual question answering on color fundus photographs (CFPs). A total of 705,850 simulated dialogues paired with 46,167 CFPs were generated to train OcularChat to identify key AMD features and produce reasoned predictions. OcularChat demonstrated strong classification performance in AREDS, achieving accuracies of 0.954, 0.849, and 0.678 for the three diagnostic tasks: advanced AMD, pigmentary abnormalities, and drusen size, significantly outperforming existing MLLMs. On AREDS2, OcularChat remained the top-performing method on all tasks. Across three independent ophthalmologist graders, OcularChat achieved higher mean scores than a strong baseline model for advanced AMD (3.503 vs. 2.833), pigmentary abnormalities (3.272 vs. 2.828), drusen size (3.064 vs. 2.433), and overall impression (2.978 vs. 2.464) on a 5-point clinical grading rubric. Beyond strong objective performance in AMD severity classification, OcularChat demonstrated the ability to provide diagnostic
reasoning, clinically relevant explanations, and interactive dialogue, with high performance in subjective ophthalmologist evaluation. These findings suggest that MLLMs may enable accurate, interpretable, and clinically useful image-based diagnosis and classification of AMD.
Read the full abstract ▾
All authors Ran Gu, Benjamin Hou, Mélanie Hébert, Asmita Indurkar, Yifan Yang, Emily Y. Chew, Tiarnán D. L. Keenan, Zhiyong Lu
Computation and Language
Cross-Lingual Jailbreak Detection via Semantic Codebooks
Shirin Alanova, Bogdan Minko, Sabrina Sadiekh et al. (+1 more)
Safety mechanisms for large
language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in
multilingual deployment. Prior work shows that translating malicious
prompts into other
languages can substantially increase jailbreak success rates, exposing a structural cross-lingual security gap. We investigate whether such attacks can be mitigated through
language-agnostic semantic similarity without retraining or
language-specific adaptation. Our approach compares
multilingual query
embeddings against a fixed English codebook of jailbreak
prompts, operating as a training-free external guardrail for black-box LLMs. We conduct a systematic evaluation across four
languages, two translation pipelines, four safety
benchmarks, three
embedding models, and three target LLMs (Qwen, Llama, GPT-3.5). Our results reveal two distinct regimes of cross-lingual transfer. On curated
benchmarks containing canonical jailbreak templates, semantic similarity generalizes reliably across
languages, achieving near-perfect separability (AUC up to 0.99) and substantial reductions in absolute attack success rates under strict low-false-positive constraints. However, under distribution shift - on behaviorally diverse and heterogeneous unsafe
benchmarks - separability degrades markedly (AUC $\approx$ 0.60-0.70), and recall in the security-critical low-FPR regime drops across all
embedding models.
Read the full abstract ▾
All authors Shirin Alanova, Bogdan Minko, Sabrina Sadiekh, Evgeniy Kokuykin
Computation and Language
Backtranslation Augmented Direct Preference Optimization for Neural Machine Translation
Mehrdad Ghassabi, Spehr Rajabi, Hamidreza Baradaran Kashani et al. (+3 more)
Contemporary neural machine translation (NMT) systems are almost exclusively built by training on supervised parallel data. Despite the tremendous progress achieved, these systems still exhibit persistent translation errors. This paper proposes that a post-training paradigm based on reinforcement learning (RL) can effectively rectify such mistakes. We introduce a novel framework that requires only a general text corpus and an expert translator which can be either human or an AI system to provide iterative feedback. In our experiments, we focus specifically on English-to-German translation as a representative high-resource
language pair. Crucially, we implement this RL-based post-training using Direct Preference Optimization (DPO). Applying our DPO-driven framework to the gemma3-1b model yields a significant improvement in translation quality, elevating its COMET score from 0.703 to 0.747 on the English to German task. The results demonstrate that DPO offers an efficient and stable pathway for enhancing pre-trained NMT models through preference-based post-training.
Read the full abstract ▾
All authors Mehrdad Ghassabi, Spehr Rajabi, Hamidreza Baradaran Kashani, Sadra Hakim, Mahshid Keivandarian, Amirhossein Jahani Bahnamiri
Computation and Language
CORAL: Adaptive Retrieval Loop for Culturally-Aligned Multilingual RAG
Nayeon Lee, Jiwoo Song and Byeongcheol Kang
Multilingual retrieval-augmented generation (mRAG) is often implemented within a fixed
retrieval space, typically via query or document translation or
multilingual embedding vector representations. However, this approach may be inadequate for culturally grounded queries, in which
retrieval-condition misalignment may occur. Even strong retrievers and generators may struggle to produce culturally relevant answers when sourcing evidence from inappropriate linguistic or regional contexts. To this end, we introduce CORAL (COntext-aware
Retrieval with Agentic Loop, an adaptive
retrieval methodology for mRAG that enables iterative refinement of both the
retrieval space (corpora) and the
retrieval probe (query) based on the quality of the evidence. The overall process includes: (1) selecting corpora, (2) retrieving documents, (3) critiquing evidence for relevance and cultural
alignment, and (4) checking sufficiency. If the retrieved documents are insufficient to answer the query correctly, the system (5) reselects corpora and rewrites the query. Across two cultural QA
benchmarks, CORAL achieves up to a 3.58%p accuracy improvement on low-resource
languages relative to the strongest baselines.
Read the full abstract ▾
Computation and Language
Modeling Human-Like Color Naming Behavior in Context
Yuqing Zhang, Ecesu Ürker, Tessa Verhoef et al. (+2 more)
Modeling the emergence of human-like lexicons in computational systems has advanced through the use of interacting neural
agents, which simulate both learning and communicative pressures. The NeLLCom-Lex framework (Zhang et al., 2025) allows neural
agents to develop pragmatic color naming behavior and human-like lexicons through supervised learning (SL) from human data and reinforcement learning (RL) in referential games. Despite these successes, the lexicons that emerge diverge systematically from human color categories, producing highly non-convex regions in color space, which contrast with the convexity typical of human categories. To address this, we introduce two factors, upsampling rare color terms during SL and multi-listener RL interactions, and adopt a convexity measure to quantify
geometric coherence. We find that upsampling improves lexical diversity and system-level informativeness of the color lexicon, while many-listener setups promote more convex color categories. The combination of moderate upsampling and multiple listeners produces lexicons most similar to human systems.
Read the full abstract ▾
All authors Yuqing Zhang, Ecesu Ürker, Tessa Verhoef, Gemma Boleda, Arianna Bisazza
Computation and Language
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Huyen Nguyen, Haoxuan Zhang, Yang Zhang et al. (+2 more)
Reliable evaluation of large
language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven
datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher
alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated
benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and
datasets will be released in GitHub.
Read the full abstract ▾
All authors Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Haihua Chen, Junhua Ding
Computation and Language
Progressing beyond Art Masterpieces or Touristic Clichés: how to assess your LLMs for cultural alignment?
António Branco, João Silva, Nuno Marques et al. (+10 more)
Although the cultural (mis)
alignment of Large
Language Models (LLMs) has attracted increasing
attention – often framed in terms of cultural bias – until recently there has been limited work on the design and development of
datasets for cultural assessment. Here, we review existing approaches to such
datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a
dataset built according to these principles. We further present a series of contrastive experiments conducted with this
dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus.
Read the full abstract ▾
All authors António Branco, João Silva, Nuno Marques, Luis Gomes, Ricardo Campos, Raquel Sequeira, Sara Nerea, Rodrigo Silva, Miguel Marques, Rodrigo Duarte, Artur Putyato, Diogo Folques, Tiago Valente
Computation and Language · Cross-listed from cs.CR
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Alex Bogdan and Adrian de Valois-Franklin
We report a striking statistical regularity in frontier LLM outputs that enables a CPU-only scoring primitive running at 2.6 microseconds per
token, with estimated latency up to 100,000$\times$ (five orders of magnitude) below existing sampling-based detectors. Across six contemporary models from five independent vendors, two generation sizes, and five held-out domains,
token rank-frequency distributions converge to the same two-parameter Mandelbrot ranking distribution, with 34 of 36 model-by-domain fits exceeding $R^{2} = 0.94$ and 35 of 36 favoring Mandelbrot over Zipf by AIC. The shared family does not collapse the models into statistical duplicates. Fitted Mandelbrot parameters remain cleanly separable between models: the cross-model spread in $q$ (1.63 to 3.69) exceeds its per-model bootstrap standard deviation (0.03 to 0.10) by more than an order of magnitude, yielding tens of standard deviations of separation per few thousand output
tokens. Two capabilities follow. First, statistical model fingerprinting: text from a vendor-delivered LLM can be tested against its claimed model family without cryptographic watermarks or access to model internals, supporting provenance verification and silent-substitution audits. Second, a model-agnostic reference distribution for black-box output assessment, from which we derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs. Pilot results on FRANK, TruthfulQA, and HaluEval map where the primitive helps (lexical anomalies, unsupported entities) and where it structurally cannot (
reasoning errors in domain-appropriate vocabulary). We position the primitive as a first-pass triage layer in compound evaluation stacks, not as a replacement for sampling-based or source-conditioned verifiers.
Read the full abstract ▾
Computation and Language
WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition
Erfan Ramezani, Mohammad Mahdi Giahi, Mohammad Erfan Zarabadipour et al. (+2 more)
Real-time automatic
speech recognition (ASR) systems face a fundamental trade-off between transcription accuracy and computational efficiency, particularly when deploying large-scale
transformer models like Whisper. Existing streaming approaches either sacrifice accuracy through aggressive chunking or incur prohibitive
memory costs through unbounded context accumulation. We present WhisperPipe, a novel streaming architecture that achieves bounded
memory consumption while maintaining transcription quality through three key innovations a hybrid Voice Activity Detection (VAD) pipeline combining Silero VAD with energy-based filtering to reduce false activations by 34%, a dynamic buffering mechanism with overlapping context windows that prevents information loss at segment boundaries, and an adaptive processing strategy that balances latency and accuracy based on
speech characteristics. Evaluated on 2.5 hours of diverse audio data, WhisperPipe demonstrates a median end-to-end latency of 89ms (90th percentile: 142ms) while consuming 48% less peak GPU
memory and 80.9% lower average GPU utilization compared to baseline Whisper implementations. The system maintains stable
memory usage over extended sessions, with zero growth rate across 150-minute continuous operation. Comparative analysis against related work shows that WhisperPipe achieves competitive accuracy (WER within 2% of offline Whisper) while operating at 3-5x lower latency than existing streaming solutions. The architecture's modular design enables deployment across resource-constrained environments, from edge devices to cloud infrastructure. Our results demonstrate that careful architectural design can reconcile the competing demands of real-time responsiveness and model sophistication in production ASR systems.
Read the full abstract ▾
All authors Erfan Ramezani, Mohammad Mahdi Giahi, Mohammad Erfan Zarabadipour, Amir Reza Yosefian, Hamid Ghadiri
Computation and Language · Cross-listed from eess.AS
Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models
Chun-Yi Kuan, Wei-Ping Huang and Hung-yi Lee
Recent audio-aware large
language models (ALLMs) have demonstrated strong capabilities across diverse audio understanding and
reasoning tasks, but they still frequently produce hallucinated or overly confident outputs. While uncertainty estimation has been extensively studied in text-only LLMs, it remains largely unexplored for ALLMs, where audio-conditioned generation introduces additional challenges such as perceptual ambiguity and cross-modal grounding. In this work, we present the first systematic empirical study of uncertainty estimation in ALLMs. We
benchmark five representative methods, including predictive
entropy, length-normalized
entropy, semantic
entropy, discrete semantic
entropy, and P(True), across multiple models and diverse evaluation settings spanning general audio understanding,
reasoning,
hallucination detection, and unanswerable question answering. Our results reveal two key findings. First, semantic-level and verification-based methods consistently outperform
token-level baselines on general audio
reasoning benchmarks. Second, on trustworthiness-oriented
benchmarks, the relative effectiveness of uncertainty methods becomes notably more model- and
benchmark-dependent, indicating that conclusions drawn from general
reasoning settings do not straightforwardly transfer to
hallucination and unanswerable-question scenarios. We further explore uncertainty-based adaptive inference as a potential downstream application. We hope this study provides a foundation for future research on reliable, uncertainty-aware audio-
language systems.
Read the full abstract ▾
Computation and Language
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
David Hartmann, Manuel Tonneau, Angelie Kraft et al. (+7 more)
Perspective API closes at the end of 2026, removing the de facto standard for toxicity measurement and exposing researchers' dependence on a tool they did not control. Drawing on this case, we argue that a research field must build and govern its own measurement infrastructure rather than borrow it. Surveying 241 papers that use or study Perspective, we show what depending on it cost the research community: claims reaching past what the tool could support, results that shifted when its model was silently retrained, and disparities researchers could measure but not explain. These failures were amplified throughout the LLM lifecycle, where Perspective supplied the labels, filtered the corpora, and graded the systems trained on each, rewarding errors rather than catching them. To keep the instrument open to study after shutdown, we release Perspective scores for 5.9 million text snippets from 77
datasets. We further specify ten requirements for measurement infrastructure a field owns, and argue that what blocks such infrastructure is not technical capability but the value the field places on infrastructure work.
Read the full abstract ▾
All authors David Hartmann, Manuel Tonneau, Angelie Kraft, LK Seiling, Dimitri Staufer, Pieter Delobelle, Jan Fillies, Anna Ricarda Luther, Jan Batzner, Mareike Lisker
Computation and Language
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Fan Jiang, Yu Zhao, Chenyang Lyu et al. (+5 more)
We present Marco-MoE, a suite of fully open
multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5% of the total parameters are activated per input
token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T
tokens. Our models surpass similarly-sized competitors on English and
multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-Instruct variants, which surpass the performance of competing models possessing $3$–$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related
languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable
language expansion without the interference typical of dense models. To support the community, we disclose our full training
datasets, recipes, and model weights.
Read the full abstract ▾
All authors Fan Jiang, Yu Zhao, Chenyang Lyu, Tianqi Shi, Yichao Du, Feihu Jiang, Longyue Wang, Weihua Luo
Computation and Language
From Chatbots to Confidants: A Cross-Cultural Study of LLM Adoption for Emotional Support
Natalia Amat-Lefort, Mert Yazan, Amanda Cercas Curry et al. (+1 more)
Large
Language Models (LLMs) are increasingly used not only for instrumental tasks, but as always-available and non-judgmental confidants for emotional support. Yet what drives adoption and how users perceive emotional support interactions across countries remains unknown. To address this gap, we present the first large-scale cross-cultural study of LLM use for emotional support, surveying 4,641 participants across seven countries (USA, UK, Germany, France, Spain, Italy, and The Netherlands). Our results show that adoption rates vary dramatically across countries (from 20% to 59%). Using mixed models that separate cultural effects from demographic composition, we find that: Being aged 25-44, religious, married, and of higher socioeconomic status are predictors of positive perceptions (trust, usage, perceived benefits), with socioeconomic status being the strongest. English-speaking countries consistently show more positive perceptions than Continental European countries. We further collect a corpus of 731 real
multilingual prompts from user interactions, showing that users mainly seek help for loneliness, stress, relationship conflicts, and mental health struggles. Our findings reveal that LLM emotional support use is shaped by a complex sociotechnical landscape and call for a broader research agenda examining how these systems can be developed, deployed, and governed to ensure safe and informed access.
Read the full abstract ▾
All authors Natalia Amat-Lefort, Mert Yazan, Amanda Cercas Curry, Flor Miriam Plaza-del-Arco
Computation and Language
From World-Gen to Quest-Line: A Dependency-Driven Prompt Pipeline for Coherent RPG Generation
Dominik Borawski, Marta Szulc, Robert Chudy et al. (+2 more)
Large
Language Models (LLMs) have shown strong potential for narrative generation, but their use in complex, multi-layered role-playing game (RPG) worlds is still limited by issues of coherence, controllability, and structural consistency. This paper explores a dependency-aware, multi-stage
prompt pipeline for procedural RPG content generation that models narrative dependencies through structured intermediate representations. The approach decomposes generation into sequential stages: world building, non-player character creation, player character creation, campaign-level quest planning, and quest expansion. Each stage conditions on structured JSON outputs from previous stages. By enforcing schemas and explicit data flow, the pipeline reduces narrative drift, limits
hallucinations, and supports scalable creation of interconnected narrative elements. The system is evaluated qualitatively through human-centered analysis across multiple independent runs. Outputs are assessed using criteria such as structural completeness, internal consistency, narrative coherence, diversity, and actionability. Results show that the pipeline consistently generates logically sound and structurally valid RPG content, without quality degradation as complexity increases. Separating high-level campaign planning from detailed quest expansion improves both global structure and local storytelling. These findings suggest that dependency-aware
prompt pipelines with structured intermediate representations are an effective design pattern for LLM-based procedural content generation. This approach may also generalize to other domains requiring sequential
reasoning over evolving contextual states.
Read the full abstract ▾
All authors Dominik Borawski, Marta Szulc, Robert Chudy, Małgorzata Giedrowicz, Piotr Mironowicz
Computation and Language · Cross-listed from cs.SD
PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech
Venkata Pushpak Teja Menta
Standard text-to-
speech (TTS) evaluation measures intelligibility (WER, CER) and overall naturalness (MOS, UTMOS) but does not quantify accent. A synthesiser may score well on all four yet sound non-native on features that are phonemic in the target
language. For Indic
languages, these features include retroflex articulation, aspiration, vowel length, and the Tamil retroflex approximant (letter zha). We present PSP, the Phoneme Substitution Profile, an interpretable, per-phonological-dimension accent
benchmark for Indic TTS. PSP decomposes accent into six complementary dimensions: retroflex collapse rate (RR), aspiration fidelity (AF), vowel-length fidelity (LF), Tamil-zha fidelity (ZF), Frechet Audio Distance (FAD), and prosodic signature divergence (PSD). The first four are measured via forced
alignment plus native-speaker-centroid acoustic probes over Wav2Vec2-XLS-R layer-9
embeddings; the latter two are corpus-level distributional distances. In this v1 we
benchmark four commercial and open-source systems (ElevenLabs v3, Cartesia Sonic-3, Sarvam Bulbul, Indic Parler-TTS) on Hindi, Telugu, and Tamil pilot sets, with a fifth system (Praxy Voice) included on all three
languages, plus an R5->R6 case study on Telugu. Three findings: (i) retroflex collapse grows monotonically with phonological difficulty Hindi < Telugu < Tamil (~1%, ~40%, ~68%); (ii) PSP ordering diverges from WER ordering – commercial WER-leaders do not uniformly lead on retroflex or prosodic fidelity; (iii) no single system is Pareto-optimal across all six dimensions. We release native reference centroids (500 clips per
language), 1000-clip
embeddings for FAD, 500-clip prosodic feature
matrices for PSD, 300-utterance golden sets per
language, scoring code under MIT, and centroids under CC-BY. Formal MOS-correlation is deferred to v2; v1 reports five internal-consistency signals plus a native-audio sanity check.
Read the full abstract ▾
Computation and Language
An Investigation of Linguistic Biases in LLM-Based Recommendations
Nitin Venkateswaran, Jason Ang, Deep Adhikari et al. (+1 more)
We investigate linguistic biases in LLM-based restaurant and product recommendations given
prompts varying across Southern American English (AE), Indian English (IE), and Code-Switched Hindi-English dialects, using the Yelp Open
dataset (Yelp Inc., 2023) and Walmart product reviews
dataset (PromptCloud,2020). We add lists of restaurant and product names balanced by cuisine type and product category to the
prompts given to the LLM, and we zero-shot
prompt the LLMs in a cold-start setting to select the top-20 restaurant and product recommendations from these lists for each of the dialect-varied
prompts. We
prompt LLMs using different list samples across 20 seeds for better generalization, and aggregate per cuisine-type and per category response counts for each seed, question/
prompt, and LLM model. We run mixed-effects regression models for each model family and topic (restaurant/product) with the aggregate response counts as the dependent, and conduct likelihood ratio tests for the fixed effects with post-hoc pairwise testing of estimated marginal means differences, to investigate group-level differences in recommendation counts by model size and dialect type. Results show that dialect plays a role in the type of restaurant selected across the models tested with the mistral-small-3.1 model and both the llama-3.1 family models tested showing more sensitivity to Indian English and Code-Switched
prompts. In terms of product recommendations, the llama-3.1-70B-model is particularly sensitive to Code-Switched
prompts in four out of seven categories, and more beauty and home category recommendations are seen when using the Indian English and Code-Switched
prompts for larger and smaller models, respectively. No broad trends are seen in the model-size based differences, with differing recommendations based on model sizes conditioned by the type of dialect.
Read the full abstract ▾
All authors Nitin Venkateswaran, Jason Ang, Deep Adhikari, Tarun Krishna Dasari
Computation and Language
Benchmarking Logistic Regression, SVM, and LightGBM Against BiLSTM with Attention for Sentiment Analysis on Indonesian Product Reviews
Razin Hafid Hamdi, Ivana Margareth Hutabarat, Hanna Gresia Sinaga et al. (+3 more)
Sentiment analysis of product reviews on e-commerce platforms plays a critical role in automatically understanding customer satisfaction and providing actionable insights for sellers seeking to improve product quality. This paper presents a comprehensive benchmarking study comparing a Machine Learning (ML) approach via the PyCaret AutoML framework against a Deep Learning (DL) approach based on a Bidirectional Long Short-Term
Memory (BiLSTM) architecture with an
Attention mechanism for binary sentiment classification on Indonesian product reviews. The
dataset comprises 19,728 samples balanced equally between positive and negative reviews. For the ML approach, three prominent
algorithms were evaluated via 10-fold stratified cross-validation: Logistic Regression (LR), Support Vector Machine (SVM) with a linear kernel, and Light Gradient Boosting Machine (LightGBM). Logistic Regression achieved the best ML performance with an accuracy of 97.26% and an F1-score of 97.26%. The BiLSTM with
Attention model, evaluated on 3,946 held-out test samples, achieved an accuracy of 97.24% and an F1-score of 97.24%. These comparative results demonstrate that traditional ML
algorithms with proper preprocessing and feature extraction can compete closely with, and even marginally outperform, more complex sequential DL architectures on high-dimensional
datasets, while simultaneously offering greater computational efficiency.
Read the full abstract ▾
All authors Razin Hafid Hamdi, Ivana Margareth Hutabarat, Hanna Gresia Sinaga, Luluk Muthoharoh, Ardika Satria, Martin C.T. Manullang
Computation and Language
Navigating Global AI Regulation: A Multi-Jurisdictional Retrieval-Augmented Generation System
Courtney Ford, Ojas Rane and Susan Leavy
Navigating AI regulation across jurisdictions is increasingly difficult for policymakers, legal professionals, and researchers. To address this, we present a multi-jurisdictional
Retrieval-Augmented Generation system for global AI regulation. Our corpus includes 242 documents across 68 jurisdictions, ranging from formal legislation like the EU AI Act to unstructured policy documents such as national AI strategies. The system makes three technical contributions: type-specific chunking that preserve legal structure across heterogenous documents; conditional
retrieval routing with entity detection and metadata for legal citations; and priority-based re-ranking to boost enacted legislation over policy and secondary sources. Evaluation of 50 queries reveals strong performance across both single-entity and multi-jurisdictional questions, achieving 0.87 average faithfulness and 0.84 average answer relevancy. Single-entity queries achieve 0.86 average faithfulness and 0.92 average answer relevancy, while multi-jurisdictional comparison queries achieve 0.88 average faithfulness and 0.75 average answer relevancy. These findings highlight the effectiveness of domain-specific
retrieval strategies for navigating complex, heterogenous regulatory corpora.
Read the full abstract ▾