Computation and Language · Cross-listed from cs.LG
Omitted Variable Bias in Language Models Under Distribution Shift
Victoria Lin, Louis-Philippe Morency and Eli Ben-Michael
Despite their impressive performance on a wide variety of tasks, modern
language models remain susceptible to distribution shifts, exhibiting brittle behavior when evaluated on data that differs in distribution from their training data. In this paper, we describe how distribution shifts in
language models can be separated into observable and unobservable components, and we discuss how established approaches for dealing with distribution shift address only the former. Importantly, we identify that the resulting omitted variable bias from unobserved variables can compromise both evaluation and optimization in
language models. To address this challenge, we introduce a framework that maps the strength of the omitted variables to bounds on the worst-case generalization performance of
language models under distribution shift. In empirical experiments, we show that using these bounds directly in
language model evaluation and optimization provides more principled measures of out-of-distribution performance, improves true out-of-distribution performance relative to standard distribution shift adjustment methods, and further enables inference about the strength of the omitted variables when target distribution labels are available.
Read the full abstract ▾
Computation and Language · Cross-listed from q-bio.OT
PREFER: An Ontology for the PREcision FERmentation Community
Txell Amigó, Shawn Zheng Kai Tan, Angel Luu Phanthanourak et al. (+12 more)
Precision fermentation relies on microbial
cell factories to produce sustainable food, pharmaceuticals, chemicals, and biofuels. Specialized laboratories such as biofoundries are advancing these processes using high-throughput bioreactor platforms, which generate vast
datasets. However, the lack of community standards limits data accessibility and interoperability, preventing integration across platforms. In order to address this, we introduce PREFER, an open-source ontology designed to establish a unified standard for bioprocess data. Built in
alignment with the widely adopted Basic Formal Ontology (BFO) and connecting with several other community ontologies, PREFER ensures consistency and cross-domain compatibility and covers the whole precision fermentation process. Integrating PREFER into high-throughput bioprocess development workflows enables structured metadata that supports automated cross-platform execution and high-fidelity data capture. Furthermore, PREFER's standardization has the potential to bridge disparate data silos, generating machine-actionable
datasets critical for training predictive, robust machine learning models in synthetic biology. This work provides the foundation for scalable, interoperable bioprocess systems and supports the transition toward more data-driven bioproduction.
Read the full abstract ▾
All authors Txell Amigó, Shawn Zheng Kai Tan, Angel Luu Phanthanourak, Sebastian Schulz, Pasquale D. Colaianni, Dominik M. Maszczyk, Ester Milesi, Ivan Schlembach, Mykhaylo Semenov Petrov, Marta Reventós Montané, Lars K. Nielsen, Jochen Förster, Bernhard Ø. Palsson, Suresh Sudarsan, Alberto Santos
Computation and Language
Reinforced Fast Weights via Next-Sequence Prediction
Hee Seung Hwang, Xindi Wu, Sanghyuk Chun et al. (+2 more)
Fast weight architectures offer a promising alternative to standard
transformers for long-context modeling by replacing the KV-cache with a recurrently updated fixed-size
memory. Despite this architectural shift, they are typically trained with the same next-
token prediction (NTP) objective as standard
transformers. This creates a mismatch: fast weight models rely on an evolving
memory state to support future predictions, while NTP provides only
token-level supervision for the immediate next
token. We address this mismatch with next-sequence prediction (NSP), a sequence-level extension of NTP that trains models to produce coherent multi-
token continuations from their recurrent
memory state. To optimize this sequence-level objective, we propose ReFINE (Reinforced Fast weIghts via Next sEquence prediction). ReFINE selects informative
token positions based on prediction
entropy, generates multi-
token rollouts, assigns sequence-level rewards, and optimizes the model with Group Relative Policy Optimization (GRPO). ReFINE is applicable throughout the training lifecycle of pre-trained
language models: mid-training, post-training, and test-time training. Experiments on LaCT-760M and DeltaNet-1.3B show that ReFINE consistently outperforms NTP-based supervised fine-tuning in needle-in-a-haystack
retrieval, long-context question answering, and diverse tasks in LongBench. ReFINE provides an effective and versatile framework for improving long-context modeling in fast weight architectures.
Read the full abstract ▾
All authors Hee Seung Hwang, Xindi Wu, Sanghyuk Chun, Olga Russakovsky, Zhiwei Deng
Computation and Language
Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents
Wenxuan Ding, Nicholas Tomlin and Greg Durrett
LLM
agents are deployed in environments where they must interact to acquire information. In these scenarios, the
agent must reason about inherent cost-uncertainty tradeoffs in how to act, such as when to stop exploring and commit to an answer. For instance, on a programming task, an
agent might run the code it generates, or it might generate tests for that code snippet; the cost of writing and running a test is nonzero, but typically lower than the cost of running buggy code. In this work, we show that we can induce LLM
agents to explicitly reason about balancing these cost-uncertainty tradeoffs, then act more optimally in their environments. We formalize multiple tasks, including
retrieval-augmented QA and a file reading coding task, as sequential decision-making problems under uncertainty. Each problem has latent environment state that impacts the
agent's performance. We introduce a framework called Calibrate-Then-Act (CTA), where we pass the
agent an inferred prior about this environment state to enable it to act more optimally. This information qualitatively changes
agent behavior, and adds environment sensitivity to the
agent which is not learned via standard RL training. Our results on a synthetic task, QA, and file reading show that making cost-benefit tradeoffs explicit with CTA helps
agents discover more optimal decision-making strategies.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.SD
Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens
Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds et al. (+3 more)
Current audio
language models are predominantly text-first, either extending pre-trained text LLM backbones or relying on semantic-only audio
tokens, limiting general audio modeling. This paper presents a systematic empirical study of native audio foundation models that apply next-
token prediction to audio at scale, jointly modeling semantic content, acoustic details, and text to support both general audio generation and cross-modal capabilities. We provide comprehensive empirical insights for building such models: (1) We systematically investigate design choices – data sources, text mixture ratios, and
token composition – establishing a validated training recipe. (2) We conduct the first scaling law study for discrete audio models via IsoFLOP analysis on 64 models spanning $3{\times}10^{18}$ to $3{\times}10^{20}$ FLOPs, finding that optimal data grows 1.6$\times$ faster than optimal model size. (3) We apply these lessons to train SODA (Scaling Open Discrete Audio), a suite of models from 135M to 4B parameters on 500B
tokens, comparing against our scaling predictions and existing models. SODA serves as a flexible backbone for diverse audio/text tasks – we demonstrate this by fine-tuning for voice-preserving
speech-to-
speech translation, using the same unified architecture.
Read the full abstract ▾
All authors Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun, William Held, Diyi Yang
Computation and Language
Align Once, Benefit Multilingually: Enforcing Multilingual Consistency for LLM Safety Alignment
Yuyan Bu, Xiaohao Liu, ZhaoXing Ren et al. (+2 more)
The widespread deployment of large
language models (LLMs) across linguistic communities necessitates reliable
multilingual safety
alignment. However, recent efforts to extend
alignment to other
languages often require substantial resources, either through large-scale, high-quality supervision in the target
language or through pairwise
alignment with high-resource
languages, which limits scalability. In this work, we propose a resource-efficient method for improving
multilingual safety
alignment. We introduce a plug-and-play Multi-Lingual Consistency (MLC) loss that can be integrated into existing monolingual
alignment pipelines. By improving collinearity between
multilingual representation vectors, our method encourages directional consistency at the
multilingual semantic level in a single update. This allows simultaneous
alignment across multiple
languages using only
multilingual prompt variants without requiring additional response-level supervision in low-resource
languages. We validate the proposed method across different model architectures and
alignment paradigms, and demonstrate its effectiveness in enhancing
multilingual safety with limited impact on general model utility. Further evaluation across
languages and tasks indicates improved cross-lingual generalization, suggesting the proposed approach as a practical solution for
multilingual consistency
alignment under limited supervision.
Read the full abstract ▾
All authors Yuyan Bu, Xiaohao Liu, ZhaoXing Ren, Yaodong Yang, Juntao Dai
Computation and Language
Quecto-V1: Empirical Analysis of 8-bit Quantized Small Language Models for On-Device Legal Retrieval
Subrit Dikshit
The rapid proliferation of Large
Language Models (LLMs) has revolutionized Natural
Language Processing (NLP) but has simultaneously created a "resource divide." State-of-the-art legal intelligence systems typically rely on massive parameter counts (7B+) and cloud-based inference, rendering them inaccessible to practitioners in resource-constrained environments and posing significant data sovereignty risks. This paper introduces Quecto-V1, a domain-specific Small
Language Model (SLM) engineered to democratize access to Indian legal intelligence. Built upon a custom configuration of the GPT-2 architecture (124 million parameters), Quecto-V1 was trained from scratch exclusively on a corpus of Indian statutes, including the Indian Penal Code (IPC), the Code of Criminal Procedure (CrPC), and the Constitution of India. Unlike generalist models, which prioritize broad world knowledge, our approach maximizes "lexical density" within the legal domain. Furthermore, we address the deployment bottleneck by applying post-training 8-bit quantization (GGUF format), compressing the model to a
memory footprint of under 150 MB. Our empirical analysis demonstrates that Quecto-V1 achieves high fidelity in retrieving statutory definitions and penal provisions, outperforming general-purpose SLMs in domain-specific exact match tasks while running entirely offline on consumer-grade CPUs. We further present an ablation study showing that 8-bit quantization yields a 74% reduction in model size with less than 3.5% degradation in
retrieval accuracy compared to full-precision baselines. These findings suggest that for specialized, high-stakes domains like law, domain-specific training coupled with aggressive quantization offers a viable,
privacy-preserving alternative to monolithic cloud models.
Read the full abstract ▾
Computation and Language
AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models
Adib Sakhawat and Fardeen Sadab
Evaluating the social intelligence of Large
Language Models (LLMs) increasingly requires moving beyond static text generation toward dynamic,
adversarial interaction. We introduce the
Adversarial Resource Extraction Game (AREG), a
benchmark that operationalizes persuasion and resistance as a multi-turn, zero-sum negotiation over financial resources. Using a round-robin tournament across frontier models, AREG enables joint evaluation of offensive (persuasion) and defensive (resistance) capabilities within a single interactional framework. Our analysis provides evidence that these capabilities are weakly correlated ($\rho = 0.33$) and empirically dissociated: strong persuasive performance does not reliably predict strong resistance, and vice versa. Across all evaluated models, resistance scores exceed persuasion scores, indicating a systematic defensive advantage in
adversarial dialogue settings. Further linguistic analysis suggests that interaction structure plays a central role in these outcomes. Incremental commitment-seeking strategies are associated with higher extraction success, while verification-seeking responses are more prevalent in successful defenses than explicit refusal. Together, these findings indicate that social influence in LLMs is not a monolithic capability and that evaluation frameworks focusing on persuasion alone may overlook asymmetric behavioral vulnerabilities.
Read the full abstract ▾
Computation and Language
Who can we trust? LLM-as-a-jury for Comparative Assessment
Mengjie Qian, Guangzhi Sun, Mark J.F. Gales et al. (+1 more)
Large
language models (LLMs) are increasingly applied as automatic evaluators for natural
language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and evaluation aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-asa-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on
benchmark NLG evaluation
datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminators strongly correlate with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
Read the full abstract ▾
All authors Mengjie Qian, Guangzhi Sun, Mark J.F. Gales, Kate M. Knill
Computation and Language
ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models
Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain et al. (+1 more)
Current state-of-the-art multi-vector models are obtained through a small Knowledge
Distillation (KD) training step on top of strong single-vector models, leveraging the large-scale pre-training of these models. In this paper, we study the pre-training of multi-vector models and show that large-scale multi-vector pre-training yields much stronger multi-vector models. Notably, a fully ColBERT-pre-trained model, ColBERT-Zero, trained only on public data, outperforms GTE-ModernColBERT as well as its base model, GTE-ModernBERT, which leverages closed and much stronger data, setting new state-of-the-art for model this size. We also find that, although performing only a small KD step is not enough to achieve results close to full pre-training, adding a supervised step beforehand allows to achieve much closer performance while skipping the most costly unsupervised phase. Finally, we find that aligning the fine-tuning and pre-training setups is crucial when repurposing existing models. To enable exploration of our results, we release various checkpoints as well as code used to train them.
Read the full abstract ▾
All authors Antoine Chaffin, Luca Arnaboldi, Amélie Chatelain, Florent Krzakala
Computation and Language
Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
Melkamu Abay Mersha and Jugal Kalita
Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret. Existing explainability methods rely on final-layer attributions, capture either local
token-level attributions or global
attention patterns without unification, and lack context-awareness of inter-
token dependencies and structural components. They also fail to capture how relevance evolves across layers and how structural components shape decision-making. To address these limitations, we proposed the Context-Aware Layer-wise Integrated Gradients (CA-LIG) Framework, a unified hierarchical attribution framework that computes layer-wise Integrated Gradients within each
Transformer block and fuses these
token-level attributions with class-specific
attention gradients. This integration yields signed, context-sensitive attribution maps that capture supportive and opposing evidence while tracing the hierarchical flow of relevance through the
Transformer layers. We evaluate the CA-LIG Framework across diverse tasks, domains, and
transformer model families, including sentiment analysis and long and multi-class document classification with BERT, hate
speech detection in a low-resource
language setting with XLM-R and AfroLM, and image classification with Masked Autoencoder
vision Transformer model. Across all tasks and architectures, CA-LIG provides more faithful attributions, shows stronger sensitivity to contextual dependencies, and produces clearer, more semantically coherent visualizations than established explainability methods. These results indicate that CA-LIG provides a more comprehensive, context-aware, and reliable explanation of
Transformer decision-making, advancing both the practical interpretability and conceptual understanding of deep neural models.
Read the full abstract ▾
Computation and Language
CitiLink-Summ: Summarization of Discussion Subjects in European Portuguese Municipal Meeting Minutes
Miguel Marques, Ana Luísa Fernandes, Ana Filipa Pacheco et al. (+10 more)
Municipal meeting minutes are formal records documenting the discussions and decisions of local government, yet their content is often lengthy, dense, and difficult for citizens to navigate. Automatic summarization can help address this challenge by producing concise summaries for each discussion subject. Despite its potential, research on summarizing discussion subjects in municipal meeting minutes remains largely unexplored, especially in low-resource
languages, where the inherent complexity of these documents adds further challenges. A major bottleneck is the scarcity of
datasets containing high-quality, manually crafted summaries, which limits the development and evaluation of effective summarization models for this domain. In this paper, we present CitiLink-Summ, a new corpus of European Portuguese municipal meeting minutes, comprising 100 documents and 2,322 manually hand-written summaries, each corresponding to a distinct discussion subject. Leveraging this
dataset, we establish baseline results for automatic summarization in this domain, employing state-of-the-art generative models (e.g., BART, PRIMERA) as well as large
language models (LLMs), evaluated with both lexical and semantic metrics such as ROUGE, BLEU, METEOR, and BERTScore. CitiLink-Summ provides the first
benchmark for municipal-domain summarization in European Portuguese, offering a valuable resource for advancing NLP research on complex administrative texts.
Read the full abstract ▾
All authors Miguel Marques, Ana Luísa Fernandes, Ana Filipa Pacheco, Rute Rebouças, Inês Cantante, José Isidro, Luís Filipe Cunha, Alípio Jorge, Nuno Guimarães, Sérgio Nunes, António Leal, Purificação Silvano, Ricardo Campos
Computation and Language · Cross-listed from cs.AI
Creating a digital poet
Vered Tohar, Tsahi Hayat and Amir Leshem
Can a machine write good poetry? Any positive answer raises fundamental questions about the nature and value of art. We report a seven-month poetry workshop in which a large
language model was shaped into a digital poet through iterative in-context expert feedback, without retraining. Across sessions, the model developed a distinctive style and a coherent corpus, supported by quantitative and qualitative analyses, and it produced a pen name and author image. In a blinded authorship test with 50 humanities students and graduates (three AI poems and three poems by well-known poets each), judgments were at chance: human poems were labeled human 54% of the time and AI poems 52%, with 95% confidence intervals including 50%. After the workshop, a commercial publisher released a poetry collection authored by the model. These results show that workshop-style prompting can support long-horizon creative shaping and renew debates on creativity and authorship.
Read the full abstract ▾
Computation and Language
Utility-Preserving De-Identification for Math Tutoring: Investigating Numeric Ambiguity in the MathEd-PII Benchmark Dataset
Zhuqian Zhou, Kirk Vanacore, Bakhtawar Ahtisham et al. (+7 more)
Large-scale sharing of dialogue data is key to advancing the science of teaching and learning, yet rigorous de-identification remains a major barrier. In mathematics tutoring transcripts, numeric expressions frequently resemble structured identifiers (e.g., dates or IDs), leading generic Personally Identifiable Information (PII) detection systems to over-redact core instructional content and reduce data utility. This work asks how to detect PII while preserving educational utility, focusing on this "numeric ambiguity" problem. We introduce MathEd-PII, the first
benchmark dataset for PII detection in math tutoring dialogues, built with human-in-the-loop LLM annotation. Using density-based
segmentation, we show that false PII redactions cluster in math-dense regions, confirming numeric ambiguity as a key failure mode. We then compare four detection strategies: a Presidio baseline and three LLM-based approaches with basic, math-aware, and segment-aware prompting. Domain-aware prompting, including both math-aware (F1: 0.802) and segment-aware versions (F1: 0.821), substantially outperforms the baseline (F1: 0.379) while reducing numeric false positives, demonstrating that de-identification must incorporate domain context to preserve analytic utility. This work provides a new
benchmark and evidence that utility-preserving de-identification for tutoring data requires domain-aware modeling.
Read the full abstract ▾
All authors Zhuqian Zhou, Kirk Vanacore, Bakhtawar Ahtisham, Jinsook Lee, Doug Pietrzak, Daryl Hedley, Jorge Dias, Chris Shaw, Ruth Schäfer, René F. Kizilcec
Computation and Language
Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification
Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić et al. (+1 more)
This paper introduces ParlaCAP, a large-scale
dataset for analyzing parliamentary agenda setting across Europe, and proposes a cost-effective method for building domain-specific policy topic classifiers. Applying the Comparative Agendas Project (CAP) schema to the
multilingual ParlaMint corpus of over 8 million
speeches from 28 parliaments of European countries and autonomous regions, we follow a teacher-student framework in which a high-performing large
language model (LLM) annotates in-domain training data and a
multilingual encoder model is fine-tuned on these annotations for scalable data annotation. We show that this approach produces a classifier tailored to the target domain. Agreement between the LLM and human annotators is comparable to inter-annotator agreement among humans, and the resulting model outperforms existing CAP classifiers trained on manually-annotated but out-of-domain data. In addition to the CAP annotations, the ParlaCAP
dataset offers rich speaker and party metadata, as well as sentiment predictions coming from the ParlaSent
multilingual transformer model, enabling comparative research on political
attention and representation across countries. We illustrate the analytical potential of the
dataset with three use cases, examining the distribution of parliamentary
attention across policy topics, sentiment patterns in parliamentary
speech, and gender differences in policy
attention.
Read the full abstract ▾
All authors Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić, Nikola Ljubešić
Computation and Language
Optimizing Soft Prompt Tuning via Structural Evolution
Zhenzhen Huang, Chaoning Zhang, Haoyu Bian et al. (+8 more)
Soft
prompt tuning leverages continuous
embeddings to capture task-specific information in large pre-trained
language models (LLMs), achieving competitive performance in few-shot settings. However, soft
prompts rely on high-dimensional, implicit representations and lack explicit semantics and traceable training behaviors, which limits their interpretability. To address this limitation, we propose a soft
prompt tuning optimization method based on topological morphological
evolution. Specifically, we employ persistent homology from topological data analysis (TDA) to quantify the structural representations of soft
prompts in continuous parameter space and their training process
evolution. Quantitative analysis shows that topologically stable and compact soft
prompts achieve better downstream performance. Based on this empirical observation, we construct a loss function for optimizing soft
prompt tuning, termed Topological Soft
Prompt Loss (TSLoss). TSLoss guides the model to learn structurally stable adaptations by quantifying inter-parameter connectivity and redundancy. Extensive experiments show that training with TSLoss accelerates convergence and improves tuning performance, providing an interpretable method to understand and optimize soft
prompt tuning from structural and topological perspectives.
Read the full abstract ▾
All authors Zhenzhen Huang, Chaoning Zhang, Haoyu Bian, Songbo Zhang, Chi-lok Andy Tai, Jiaquan Zhang, Caiyan Qin, Jingjing Qu, Yalan Ye, Yang Yang, Heng Tao Shen
Computation and Language
From Growing to Looping: A Unified View of Iterative Computation in LLMs
Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile et al. (+2 more)
Looping, reusing a block of layers across depth, and depth growing, training shallow-to-deep models by duplicating middle layers, have both been linked to stronger
reasoning, but their relationship remains unclear. We provide a mechanistic unification: looped and depth-grown models exhibit convergent depth-wise signatures, including increased reliance on late layers and recurring patterns aligned with the looped or grown block. These shared signatures support the view that their gains stem from a common form of iterative computation. Building on this connection, we show that the two techniques are adaptable and composable: applying inference-time looping to the middle blocks of a depth-grown model improves accuracy on some
reasoning primitives by up to $2\times$, despite the model never being trained to loop. Both approaches also adapt better than the baseline when given more in-context examples or additional supervised fine-tuning data. Additionally, depth-grown models achieve the largest
reasoning gains when using higher-quality, math-heavy cooldown mixtures, which can be further boosted by adapting a middle block to loop. Overall, our results position depth growth and looping as complementary, practical methods for inducing and scaling iterative computation to improve
reasoning.
Read the full abstract ▾
All authors Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald, Stefan Bauer
Computation and Language
Learning to Learn from Language Feedback with Social Meta-Learning
Jonathan Cook, Diego Antognini, Martin Klissarov et al. (+2 more)
Large
language models (LLMs) often struggle to learn from corrective feedback within a conversational context. They are rarely proactive in soliciting this feedback, even when faced with ambiguity, which can make their dialogues feel static, one-sided, and lacking the adaptive qualities of human conversation. To address these limitations, we draw inspiration from social meta-learning (SML) in humans - the process of learning how to learn from others. We formulate SML as a finetuning methodology, training LLMs to solicit and learn from
language feedback in simulated pedagogical dialogues, where static tasks are converted into interactive social learning problems. SML effectively teaches models to use conversation to solve problems they are unable to solve in a single turn. This capability generalises across domains; SML on math problems produces models that better use feedback to solve coding problems and vice versa. Furthermore, despite being trained only on fully-specified problems, these models are better able to solve underspecified tasks where critical information is revealed over multiple turns. When faced with this ambiguity, SML-trained models make fewer premature answer attempts and are more likely to ask for the information they need. This work presents a scalable approach to developing AI systems that effectively learn from
language feedback.
Read the full abstract ▾
All authors Jonathan Cook, Diego Antognini, Martin Klissarov, Claudiu Musat, Edward Grefenstette
Computation and Language
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling
Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu et al. (+1 more)
Existing Multi-
Agent Systems (MAS) typically rely on homogeneous model configurations, failing to exploit the diverse expertise inherent in different post-trained architectures. We propose Team-of-Thoughts, a heterogeneous MAS framework that treats diverse models as specialized tools within an orchestrator-driven paradigm. Team-of-Thoughts introduces two novel components: (1) Orchestrator Calibration, which identifies models with superior coordination and synthesis capabilities, and (2)
Agent Self-Assessment, a protocol where tool
agents profile their own domain-specific strengths to guide selection. At inference, the orchestrator dynamically activates the most compatible
agents based on these profiles to maximize capability coverage. Across five mathematical
reasoning and code generation
benchmarks, Team-of-Thoughts consistently outperforms individual models and existing MAS baselines. Notably, on AIME24 and LiveCodeBench, Team-of-Thoughts achieves 96.00% and 77.91% accuracy, respectively, significantly improving over homogeneous role-play baselines (80.00% and 65.93%).
Read the full abstract ▾
All authors Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao
Computation and Language
Dialects of Translationese Shape Language Model Learning
Jenny Kunz
Machine-translated data is widely used in
multilingual NLP, particularly where native text is scarce. However, translated text differs systematically from native text. This phenomenon is known as translationese, and it reflects both traces of the source
language and characteristic properties of translation itself. In this paper, we study how training on machine-translated data affects small English
language models, focusing on how translationese from different source
languages shapes linguistic acceptability judgments and
language modeling for different domains. We train models on English text translated from 24 typologically and resource-diverse source
languages, enabling a systematic analysis of how source
language and corpus properties influence what models learn. Our results show that the source
language has a clear impact on model behavior: general perplexity is more driven by the lexical diversity of the translated corpus, but grammatical performance is strongly correlated to typological similarity to English if trained on enough data. Even translation quality is a strong predictor of
language modeling performance.
Read the full abstract ▾
Computation and Language
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap et al. (+1 more)
The rapid advancement of large
language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and
multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic
benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of
reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and
languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves
reasoning accuracy, with substantial gains across subjects and
languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third,
multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual
reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in
multilingual educational settings and offers actionable insights for improving
reasoning robustness and
language adaptability.
Read the full abstract ▾
All authors Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap, Nachiket Tapas
Computation and Language
TabAgent: Distilling LLM Agent Components into Cheap Classifiers
Ido Levy, Segev Shlomov, Avi Yaeli et al. (+5 more)
Agentic systems, AI architectures that autonomously execute multi-step workflows to achieve complex goals, are often built using repeated large
language model (LLM) calls for closed-set decision tasks such as routing, shortlisting, gating, and verification. While convenient, this design makes deployments slow and expensive due to cumulative latency and
token usage. We propose TabAgent, a framework for replacing generative decision components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces. TabAgent (i) extracts structured schema, state, and dependency features from trajectories (TabSchema), (ii) augments coverage with schema-aligned synthetic supervision (TabSynth), and (iii) scores candidates with a lightweight classifier (TabHead). On the long-horizon AppWorld
benchmark, TabAgent maintains task-level success while eliminating shortlist-time LLM calls, reducing latency by approximately 95% and inference cost by 85-91%. Beyond tool shortlisting, TabAgent generalizes to other agentic decision heads, establishing a paradigm for learned discriminative replacements of generative bottlenecks in production
agent architectures.
Read the full abstract ▾
All authors Ido Levy, Segev Shlomov, Avi Yaeli, Nir Mashkif, Asaf Adi, Yinon Goldshtein, Eilam Shapira, Leshem Choshen
Computation and Language
Label-Consistent Data Generation for Aspect-Based Sentiment Analysis Using LLM Agents
Mohammad H.A. Monfared, Lucie Flek and Akbar Karimi
We propose an agentic data augmentation method for Aspect-Based Sentiment Analysis (ABSA) that uses iterative generation and verification to produce high quality synthetic training examples. To isolate the effect of agentic structure, we also develop a closely matched prompting-based baseline using the same model and instructions. Both methods are evaluated across three ABSA subtasks (Aspect Term Extraction (ATE), Aspect Sentiment Classification (ATSC), and Aspect Sentiment Pair Extraction (ASPE)), four SemEval
datasets, and two encoder-decoder models: T5-Base and Tk-Instruct. Our results show that the agentic augmentation outperforms raw prompting in label preservation of the augmented data, especially when the tasks require aspect term generation. In addition, when combined with real data, agentic augmentation provides higher gains, consistently outperforming prompting-based generation. These benefits are most pronounced for T5-Base, while the more heavily pretrained Tk-Instruct exhibits smaller improvements. As a result, augmented data helps T5-Base achieve comparable performance with its counterpart.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.IR
Variable-Length Semantic IDs for Recommender Systems
Kirill Khrylchenko
Generative models are increasingly used in recommender systems, both for modeling user behavior as event sequences and for integrating large
language models into recommendation pipelines. A key challenge in this setting is the extremely large cardinality of item spaces, which makes training generative models difficult and introduces a vocabulary gap between natural
language and item identifiers. Semantic identifiers (semantic IDs), which represent items as sequences of low-cardinality
tokens, have recently emerged as an effective solution to this problem. However, existing approaches generate semantic identifiers of fixed length, assigning the same description length to all items. This is inefficient, misaligned with natural
language, and ignores the highly skewed frequency structure of real-world catalogs, where popular items and rare long-tail items exhibit fundamentally different information requirements. In parallel, the emergent communication literature studies how
agents develop discrete communication protocols, often producing variable-length messages in which frequent concepts receive shorter descriptions. Despite the conceptual similarity, these ideas have not been systematically adopted in recommender systems. In this work, we bridge recommender systems and emergent communication by introducing variable-length semantic identifiers for recommendation. We propose a discrete variational autoencoder with Gumbel-Softmax reparameterization that learns item representations of adaptive length under a principled probabilistic framework, avoiding the instability of REINFORCE-based training and the fixed-length constraints of prior semantic ID methods.
Read the full abstract ▾
Computation and Language
Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents
Nivya Talokar, Ayush K Tarun, Murari Mandal et al. (+2 more)
LLM-based
agents execute real-world workflows via tools and
memory. These affordances enable ill-intended adversaries to also use these
agents to carry out complex misuse scenarios. Existing
agent misuse
benchmarks largely test single-
prompt instructions, leaving a gap in measuring how
agents end up helping with harmful or illegal tasks over multiple turns. We introduce STING (Sequential Testing of Illicit N-step Goal execution), an automated red-teaming framework that constructs a step-by-step illicit plan grounded in a benign persona and iteratively probes a target
agent with adaptive follow-ups, using judge
agents to track phase completion. We further introduce an analysis framework that models multi-turn red-teaming as a time-to-first-jailbreak random variable, enabling analysis tools like discovery curves, hazard-ratio attribution by attack
language, and a new metric: Restricted Mean Jailbreak Discovery. Across AgentHarm scenarios, STING yields substantially higher illicit-task completion than single-turn prompting and chat-oriented multi-turn baselines adapted to tool-using
agents. In
multilingual evaluations across six non-English settings, we find that attack success and illicit-task completion do not consistently increase in lower-resource
languages, diverging from common chatbot findings. Overall, STING provides a practical way to evaluate and stress-test
agent misuse in realistic deployment settings, where interactions are inherently multi-turn and often
multilingual.
Read the full abstract ▾
All authors Nivya Talokar, Ayush K Tarun, Murari Mandal, Maksym Andriushchenko, Antoine Bosselut
Computation and Language
MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
Zexue He, Yu Wang, Churan Zhi et al. (+11 more)
Existing evaluations of
agents with
memory typically assess memorization and action in isolation. One class of
benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how
memory is used to guide future decisions. Another class focuses on
agents acting in single-session tasks without the need for long-term
memory. However, in realistic settings, memorization and action are tightly coupled:
agents acquire
memory while interacting with the environment, and subsequently rely on that
memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking
agent memory in multi-session
Memory-
Agent-Environment loops. The
benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where
agents must learn from earlier actions and feedback by distilling experiences into
memory, and subsequently use that
memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web
navigation, preference-constrained planning, progressive information search, and sequential formal
reasoning, and reveals that
agents with near-saturated performance on existing long-context
memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for
agents with
memory. MemoryArena is now released at https://memoryarena.github.io/.
Read the full abstract ▾
All authors Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang, Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, Alex Pentland
Computation and Language
MultiCW: A Large-Scale Balanced Benchmark Dataset for Training Robust Check-Worthiness Detection Models
Martin Hyben, Sebastian Kula, Jan Cegin et al. (+3 more)
Large
Language Models (LLMs) are beginning to reshape how media professionals verify information, yet automated support for detecting check-worthy claims a key step in the fact-checking process remains limited. We introduce the Multi-Check-Worthy (MultiCW)
dataset, a balanced
multilingual benchmark for check-worthy claim detection spanning 16
languages, 7 topical domains, and 2 writing styles. It consists of 123,722 samples, evenly distributed between noisy (informal) and structured (formal) texts, with balanced representation of check-worthy and non-check-worthy classes across all
languages. To probe robustness, we also introduce an equally balanced out-of-distribution evaluation set of 27,761 samples in 4 additional
languages. To provide baselines, we
benchmark 3 common fine-tuned
multilingual transformers against a diverse set of 15 commercial and open LLMs under zero-shot settings. Our findings show that fine-tuned models consistently outperform zero-shot LLMs on claim classification and show strong out-of-distribution generalization across
languages, domains, and styles. MultiCW provides a rigorous
multilingual resource for advancing automated fact-checking and enables systematic comparisons between fine-tuned models and cutting-edge LLMs on the check-worthy claim detection task.
Read the full abstract ▾
All authors Martin Hyben, Sebastian Kula, Jan Cegin, Jakub Simko, Ivan Srba, Robert Moro
Computation and Language
Aladdin-FTI @ AMIYA Three Wishes for Arabic NLP: Fidelity, Diglossia, and Multidialectal Generation
Jonathan Mutal, Perla Al Almaoui, Simon Hengchen et al. (+1 more)
Arabic dialects have long been under-represented in Natural
Language Processing (NLP) research due to their non-standardization and high variability, which pose challenges for computational modeling. Recent advances in the field, such as Large
Language Models (LLMs), offer promising avenues to address this gap by enabling Arabic to be modeled as a pluricentric
language rather than a monolithic system. This paper presents Aladdin-FTI, our submission to the AMIYA shared task. The proposed system is designed to both generate and translate dialectal Arabic (DA). Specifically, the model supports text generation in Moroccan, Egyptian, Palestinian, Syrian, and Saudi dialects, as well as bidirectional translation between these dialects, Modern Standard Arabic (MSA), and English. The code and trained model are publicly available.
Read the full abstract ▾
All authors Jonathan Mutal, Perla Al Almaoui, Simon Hengchen, Pierrette Bouillon
Computation and Language · Cross-listed from nlin.AO
Lyapunov Spectral Analysis of Speech Embedding Trajectories in Psychosis
Jelena Vasic, Branislav Andjelic, Ana Mancic et al. (+5 more)
We analyze
speech embeddings from structured clinical interviews of psychotic patients and healthy controls by treating
language production as a high-dimensional dynamical process. Lyapunov exponent (LE) spectra are computed from word-level and answer-level
embeddings generated by two distinct large
language models, allowing us to assess the stability of the conclusions with respect to different
embedding presentations. Word-level
embeddings exhibit uniformly contracting dynamics with no positive LE, while answer-level
embeddings, in spite of the overall contraction, display a number of positive LEs and higher-dimensional attractors. The resulting LE spectra robustly separate psychotic from healthy
speech, while differentiation within the psychotic group is not statistically significant overall, despite a tendency of the most severe cases to occupy distinct dynamical regimes. These findings indicate that nonlinear dynamical invariants of
speech embeddings provide a physics-inspired probe of disordered cognition whose conclusions remain stable across
embedding models.
Read the full abstract ▾
All authors Jelena Vasic, Branislav Andjelic, Ana Mancic, Dusica Filipovic Djurdjevic, Ljiljana Mihic, Aleksandar Kovacevic, Nadja P. Maric, Aleksandra Maluckov
Computation and Language
Are LLMs Ready to Replace Bangla Annotators?
Md. Najib Hasan, Touseef Hasan and Souvika Sarkar
Large
Language Models (LLMs) are increasingly used as automated annotators to scale
dataset creation, yet their reliability as unbiased annotators–especially for low-resource and identity-sensitive settings–remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate
speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic
benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality–smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource
languages and underscore the need for careful evaluation before deployment.
Read the full abstract ▾