Computation and Language
A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization
Yangqiaoyu Zhou, Mohammad Alqudah, Kwei-Herng Lai et al. (+3 more)
Enterprise AI
agents route user queries to specialized skills by matching queries against natural
language skill descriptions. When two skills share overlapping descriptions, the routing LLM misroutes queries, a failure we term skill collision. As
agents scale to dozens of skills, manually tuning descriptions to maintain routing accuracy becomes a significant engineering bottleneck. We deploy an automated description optimization pipeline on a production enterprise group chat
agent (9 skills, 372 regression cases). The pipeline produces descriptions averaging 79.2% F1, matching manually tuned descriptions at 79.4% F1 (average per-skill difference -0.20%, within the 0.78% multi-seed noise floor), while reducing per-skill engineering effort from 120 minutes to 3.8 minutes (32 times speedup). We then examine which pipeline components actually drive this match. Systematic ablation on both the production system and ToolBench (16k tools) reveals that a single LLM rewrite using any available false-positive and false-negative cases captures most of the available improvement. Other design choices we tested (iteration budget, feedback signal composition, dual editing of confused pairs, and training set size) each affect final F1 by less than 0.5%. Description optimization addresses skill collisions caused by overlapping descriptions but cannot resolve cases where two skills intended scopes genuinely overlap. We identify a diagnostic (a large train-validation F1 gap) that flags the latter cases for architectural rather than text-level intervention.
Read the full abstract ▾
All authors Yangqiaoyu Zhou, Mohammad Alqudah, Kwei-Herng Lai, Aaron Halfaker, Yingqi Xiong, Yaar Harari
Computation and Language · Cross-listed from cs.LG
From Search to Synthesis: Training LLMs as Zero-Shot Workflow Generators
Gan Luo, Zihan Qin, Bin Dong et al. (+1 more)
Large
language models (LLMs) excel across a wide range of tasks, yet their instance-specific solutions often lack the structural consistency needed for reliable deployment. Workflows that encode recurring algorithmic patterns at the task level provide a principled framework, offering robustness across instance variations, interpretable traces for debugging, and reusability across problem instances. However, manually designing such workflows requires significant expertise and effort, limiting their broader application. While automatic workflow generation could address this bottleneck, existing methods either produce instance-specific solutions without learning task-level patterns, or cannot generalize beyond their training configurations. We present MetaFlow, which casts workflow generation as a meta-learning problem: given a task and an operator set, the model learns to compose solution strategies. MetaFlow trains in two stages: supervised fine-tuning on synthetic workflow data, followed by reinforcement learning with verifiable rewards (RLVR) that uses execution feedback across problem instances in the task to improve end-to-end success. The resulting model produces effective workflows for trained tasks and exhibits strong generalization to untrained tasks and novel operator sets. Across
benchmarks in question answering, code generation, and mathematical
reasoning, MetaFlow achieves performance comparable to state-of-the-art baselines on in-domain tasks with single inference, while demonstrating remarkable zero-shot generalization capabilities on out-of-domain tasks and operator sets.
Read the full abstract ▾
All authors Gan Luo, Zihan Qin, Bin Dong, Wotao Yin
Computation and Language · Cross-listed from cs.RO
Kaier Liang, Hengde Dai and Cristian-Ioan Vasile
Enabling
robots to follow natural
language commands to complete zero-shot long-horizon tasks remains challenging. It requires extracting implicit temporal and logical constraints from natural
language commands and executing multiple sub-tasks accordingly. Recent zero-shot object
navigation methods use
vision-
language models (VLMs) to guide frontier-based exploration in unknown environments, but they are limited to single-target tasks. Real-world commands such as "Clean either the chair or the couch, then turn on the tv." require navigating to multiple targets in a temporally constrained order, which no existing zero-shot system can handle. We present ViTL, a framework that addresses this gap at two levels. At the task level, we use a large
language model (LLM) to compile natural
language commands into Linear Temporal Logic (LTL) formulas, which are then converted into Deterministic Finite Automata~(DFA) that coordinate multi-channel value maps and trigger dynamic replanning when new objects are detected. At the
navigation level, we introduce directional score: rather than producing a direction-agnostic value across the entire field of view, we label frontier directions on the observation image and extract per-direction scores from the VLM. Experiments on Habitat-Matterport 3D (HM3D) show that the full framework enables zero-shot long-horizon completion of natural
language navigation tasks with temporal constraints, and that directional score improves single-target
navigation accuracy and efficiency over the baseline.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.AI
Self-Evolving World Models for LLM Agent Planning
Xuan Zhang, Wenxuan Zhang, See-Kiong Ng et al. (+1 more)
World models offer a principled way to equip long-horizon LLM
agents with foresight: predictions of action consequences before execution. However, unreliable foresight can be ignored, misused, or even degrade downstream decision-making. In this paper, we introduce WorldEvolver, a self-evolving world model framework that revises its deployment-time context while keeping the downstream
agent and all model parameters frozen. WorldEvolver integrates three modules: (i) Episodic
Memory, which exploits real action transitions through
retrieval-based simulation; (ii) Semantic
Memory, which extracts persistent heuristic rules from prediction-observation mismatches; and (iii) Selective Foresight, which filters low-confidence predictions before integrating them into
agent reasoning context. We evaluate WorldEvolver on ALFWorld and ScienceWorld, measuring world model prediction accuracy on Word2World and downstream
agent success rate on AgentBoard. Extensive experiments show that WorldEvolver achieves the highest prediction accuracy across three backbones and leads other world model baselines on downstream
agent success rate, demonstrating that test-time
memory revision enhances both predictive fidelity and planning performance.
Read the full abstract ▾
All authors Xuan Zhang, Wenxuan Zhang, See-Kiong Ng, Yang Deng
Computation and Language
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Lei Bai, Zongsheng Cao, Yang Chen et al. (+50 more)
We introduce
Agents-A1, a 35B Mixture-of-Experts Agentic Model that reaches trillion-parameter-level performance by scaling the
agent horizon. We investigate
agent-horizon scaling from two perspectives: scaling long-horizon trajectories and scaling heterogeneous
agent abilities. To support this goal, we build a long-horizon knowledge-action infrastructure that connects external knowledge, actions, observations, and verifier outcomes, producing agentic trajectories with an average length of 45K
tokens. Based on this, we train
Agents-A1 with a three-stage recipe. First, we perform full-domain supervised fine-tuning to align the base model with broad agentic behaviors. Second, we train domain-level teacher models to capture specialized expertise in each domain. Third, we propose a multi-teacher domain-routed on-policy
distillation with salient vocabulary
alignment to improve knowledge transfer efficiency across different domains, unifying six heterogeneous domains into one deployable student model.
Agents-A1 achieves strong and broad performance for long-horizon
agent benchmarks. Compared with 1T-parameter model such as Kimi-K2.6 and DeepSeek-V4-pro,
Agents-A1 achieves leading results on SEAL-0 (56.4), IFBench (80.6), HiPhO (46.4), FrontierScience-Olympiad (79.0), and MolBench-Bind (56.8), and remains highly competitive on SciCode (44.3), HLE (47.6) and BrowseComp (75.5). We hope this work provides the community with a practical path for scaling the horizon using a 35B
agent that can reach or match the performance of 1T models on long-horizon tasks.
Read the full abstract ▾
All authors Lei Bai, Zongsheng Cao, Yang Chen, Zhiyao Cui, Shangheng Du, Yue Fan, Shiyang Feng, Zijie Guo, Haonan He, Liang He, Xiaohan He, Shuyue Hu, Yusong Hu, Songtao Huang, Yichen Jiang, Hao Li, Xin Li, Dahua Lin, Weihao Lin, Fenghua Ling, Dongrui Liu, Zhuo Liu, Wenjie Lou, Runmin Ma, Chunjiang Mu, Haoyang Peng, Tianshuo Peng, Jinxin Shi, Luohe Shi, Boyuan Sun, Zelin Tan, Shengji Tang, Yan Teng, Qianyi Wang, Xiaosong Wang, Yiming Wu, Yi Xie, Xiangchao Yan, Jingqi Ye, Peng Ye, Fangchen Yu, Jiakang Yuan, Bihao Zhan, Bo Zhang, Chen Zhang, Shufei Zhang, Shuaiyu Zhang, Wenlong Zhang, Yiqun Zhang, Junpeng Zhao, Zhijie Zhong, Bowen Zhou, Yuhao Zhou
Computation and Language
Uncertainty-Aware Generation and Decision-Making Under Ambiguity
Nico Daheim and Iryna Gurevych
With rapidly improving capabilities, Large
Language Models (LLMs) are increasingly used in many complex real-world tasks. Beyond requiring in-depth knowledge and
reasoning skills, many of these tasks exhibit a high degree of subjectivity and require that the outputs of the model can be trusted. While a lot of progress has been made to train better models, decision-making
algorithms have received less
attention. In this work, we present and evaluate various uncertainty-aware decision-making
algorithms based on
Bayesian decision theory and risk-averse decision making on the tasks of tutoring and automatic peer reviewing. Concretely, we take uncertainty over tutoring strategies and review scores into account when generating a tutor response or review and use conformal prediction to provide guarantees over strategy and score. We find empirically that these
algorithms can improve the utility of the generations but need to be carefully implemented when ambiguity is high. For example, risk-averse rules can degrade performance by optimizing for generic outputs, while
Bayesian methods tend to perform better. Our work uses techniques from decision theory to improve LLM-based decision-making and outlines open challenges for the community.
Read the full abstract ▾
Computation and Language · Cross-listed from cs.LG
Attractor States Emerge in Multi-Turn LLM Conversations
Ting-Wen Ko and Jonas Geiping
Large
language models (LLMs) are increasingly used in open-ended multi-
agent settings, but the long-run dynamics of model–model interaction remain poorly understood. We study whether open-ended LLM discussions exhibit attractor-like behavior, i.e. topic-independent stable sets of behaviors which conversations settle into. Across 7 LLMs and 20 controversial topics, we compare self-play and mixed-play dyadic debates, tracking trajectories in representation space, discourse traits, and stances. We find self-play trajectories to be model-specific attractors that draw their conversation partners asymmetrically in mixed-play debates, influencing the other models' stylistic choices and behavior. For example, Claude Haiku is a strong attractor of other models in latent space, corresponding to other models taking on its traits like metacommentary, and models like GPT-4.1 nano are especially malleable. Our results suggest that open-ended LLM interactions are partially predictable from model-specific attractors, but shaped by structured and asymmetric partner influence. Overall, our analysis sheds some light on the complex behavior of open-ended multi-
agent interaction, which we hope is helpful in designing, predicting, and monitoring autonomous agentic systems in the real world.
Read the full abstract ▾
Computation and Language
Morphing into Hybrid Attention Models
Disen Lan, Jianbin Zheng, Yuxi Ren et al. (+5 more)
Hybrid
attention models improve long-context efficiency by retaining only a subset of full-
attention layers and replacing the remaining layers with linear
attention. However, the effectiveness of
Transformer-to-hybrid conversion critically depends on which layers preserve full
attention. Existing hybrid layer selection methods typically rely on heuristic strategies such as fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking the interdependent layer effect under a global hybrid configuration. In this work, we formulate hybrid layer selection as a budget-constrained subset optimization problem. We further propose FlashMorph (Fast LAyer Selection for Hybrid MORPHing), an effective, efficient and scalable layer selection method for
Transformer-to-hybrid conversion. FlashMorph first constructs a morphable model by equipping each full-
attention layer with a converted linear-
attention branch. It then freezes all model weights and jointly optimizes layerwise gates on synthetic long-context
retrieval data, with a linearization regularization that encourages the model to rely on linear
attention for efficiency. The learned gates are discretized under a preset full-
attention budget to instantiate the hybrid architecture, followed by standard logits
distillation and long-context finetuning. Extensive experiments show that FlashMorph discovers more effective hybrid configurations, preserves strong long-context recall and general
benchmark performance while substantially reducing layer selection cost compared with existing layer selection methods, demonstrating its effectiveness, efficiency, and scalability.
Read the full abstract ▾
All authors Disen Lan, Jianbin Zheng, Yuxi Ren, Xin Xia, Xuanda Wang, Xuefeng Xiao, Xipeng Qiu, Yu Cheng
Computation and Language
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Shanshan Wang, Derek F. Wong, Jingming Yao et al. (+1 more)
Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large
language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.
Read the full abstract ▾
All authors Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. Chao
Computation and Language
TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech
Sathvik Manikantan Napa Ugandhar, Hao Zhang, Alison Gunzler et al. (+5 more)
With the proliferation of
speech AI
agents, understanding emotional entrainment in conversational interaction has become increasingly important. Emotional entrainment is shaped by social relationships and conversational context, influencing affective coordination over time. We introduce DyadEE, a
dataset for emotional entrainment detection in dyadic
speech interactions, containing both emotionally entrained conversations and synthetic interactions where entrainment is disrupted through partner swapping and
emotion resynthesis. We further propose TRACE, a window-level framework that models dyadic interaction as ordered sequences of acoustic
embeddings derived from
emotion fine-tuned Whisper representations, treating each sample as an interaction trace rather than pooled utterances. Experimental results on DyadEE show that incorporating conversational context and relationship information improves emotional entrainment detection, with TRACE achieving the best accuracy of 97.01%.
Read the full abstract ▾
All authors Sathvik Manikantan Napa Ugandhar, Hao Zhang, Alison Gunzler, Yuzhe Wang, Thomas Thebaud, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velázquez
Computation and Language
Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflicts
Bo Wang, Heyan Huang, Yaolin Li et al. (+5 more)
Retrieval-augmented generation (RAG) improves
language models by grounding generation in external context. However, it can be fragile when the retrieved context conflicts with the model's parametric knowledge. Such conflicts span a reliability spectrum, ranging from reliable and partially reliable evidence to
adversarial context. Existing remedies often handle such heterogeneous conflicts with regime-agnostic supervision, which can conflate incompatible learning signals across reliability regimes. To disentangle these signals, we propose RAPS-DA, a regime-aware peer specialization framework that addresses conflict at two complementary granularities. At the sample level, conflicts are divided into three regimes, including Grounding, Arbitration, and Resistance, with one same-scale peer specialist trained per regime from a shared base model. Each sample is then hard-routed to its regime-matched peer for on-policy reverse-KL supervision. At the
token level, a dual-layer selector uses inter-teacher disagreement, student-teacher divergence, and student
entropy to filter uninformative or unstable
tokens, upweight confidently misaligned ones, and gradually focus supervision on high-conflict
tokens as the student matures. Gains stem from specialization at a fixed model scale, not from a stronger teacher, and the peer specialists exist only during training, so the deployed student requires no regime labels or peer access. Experiments on five conflict scenarios and two out-of-distribution
benchmarks show RAPS-DA surpasses all prompting, decoding, fine-tuning, RL, and single-teacher baselines.
Read the full abstract ▾
All authors Bo Wang, Heyan Huang, Yaolin Li, Yanghao Zhou, Jiahao Teng, Ziyi Yang, Ge Shi, Chong Feng
Computation and Language
SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation
Zhuhan Bao, Rui Yang, Bohao Yang et al. (+16 more)
Background. The widespread deployment of ambient digital scribes is driving large-scale capture of clinician-patient dialogues. Human coding of clinical communication data remains costly, inconsistent, and difficult to scale, motivating AI-driven communication coding systems. However, evaluating these systems requires real-world dialogues and human-coded labels, both hard to obtain at scale. Methods. We developed SIMAX (Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation), a framework for generating controlled clinical dialogue data with reference behavioral annotations. SIMAX generates clinician-patient dialogues from predefined clinical scenarios, personas and voice conditions, and target communication behaviors. Behaviors are controlled using two codebooks: the Global Codebook for overall communication quality and the WISER Codebook for specific countable behaviors. We evaluated SIMAX using automated and human quality assessments and an example communication coding system. Results. SIMAX generated 3,388 simulated dialogues across three specialties, multiple visit stages, persona characteristics, and accent conditions. Automated assessment showed mean UTMOS and WV-MOS scores of 3.03 and 2.61, WER and CER of 0.07 and 0.05, and CLAP cosine similarity of 0.41, suggesting reasonable
speech naturalness, high transcription fidelity, and positive text-audio correspondence. Human evaluation showed a median MOS of 4.67 and a median clinical realism score of 3.00. Downstream evaluation suggests that SIMAX can assess how a communication coding system responds to behavioral targets and reveal insufficient sensitivity in some dimensions. Conclusions. SIMAX generates controlled and reproducible simulated clinician-patient dialogues, providing a data foundation for developing, validating, and refining communication coding systems.
Read the full abstract ▾
All authors Zhuhan Bao, Rui Yang, Bohao Yang, Zhiyi Liu, Sicheng Shu, Ruio Heerschap, Le Li, Doris Yang, Elisabeth Bond, Haoyuan Wang, Nicoleta Economou-Zavlanos, Joshua M. Biro, Matthew McDermott, Nan Liu, Anand Chowdhury, Kai Sun, Kathryn Pollak, Ed Hammond, Chuan Hong
Computation and Language · Cross-listed from cs.CY
Situation Perception: A Necessary Primitive to Artificial Superintelligence
Ziqin Yuan and Jaymari Chua
Current large
language models are extraordinary statistical engines. They compress vast amounts of text into useful patterns and can explain science, write code, imitate
reasoning, and participate in philosophical conversation. Yet pattern mastery is not the same as general intelligence. A human infant begins with little explicit knowledge, but gradually discovers object permanence, cause and effect, other minds, bodily agency, and the persistence of the physical world. We make an argument that the path to artificial superintelligence (ASI) depends on a missing capacity we call situation perception: the ability to construct, revise, and act within internal simulations of possible worlds across latent time. perception requires at least three core components: abstract prediction, long-term compressed
memory, and active learning guided by objectives. In this work, we analyse why modern large
language models remain incomplete, and propose the appropriate tests for measuring progress and consequences of machines that can simulate futures, pursue self-directed goals, and possibly judge their own creators.
Read the full abstract ▾
Computation and Language
Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval
Aivin V. Solatorio, Olivier Dupriez and Rafael Macalaba
We study
retrieval over catalogs of structured metadata, where each record is a small schema whose fields answer different kinds of query.
Embedding a record with a text encoder first serializes its fields into a string, which forces a choice of field order. We show this choice, usually treated as an implementation detail, silently controls
retrieval quality once the encoder is fine-tuned. A standard fine-tune loses 7.4 nDCG@10 points when the index is rebuilt under a different field order, because it reads absolute position instead of the field labels. We propose permutation-invariant fine-tuning ($\textbf{PI-FT}$), which serializes each record under a freshly sampled field order with random field dropout, so meaning binds to the labels rather than to position. The change is about two lines in the data loader; it costs negligible in-distribution accuracy and cuts the order-change penalty to 0.2 points. We study this in the discovery of development statistics, a catalog of nearly 10,000 indicators that should be searchable in many
languages by a model small enough to self-host. As AI assistants and
agents increasingly mediate access to public data and statistics, this
retrieval step decides whether an answer is grounded in the right indicator or series, making discoverability a precondition for disseminating data through AI. Because usage logs cannot provide training signal for indicators no one has searched, we generate the queries instead. $\textbf{DevDataBench}$ is a fully LLM-generated
benchmark of grounded, facet-targeted queries across 15
languages, covering every indicator for both training and evaluation. A fine-tuned 118M-parameter CPU encoder outperforms every zero-shot baseline, including $\texttt{text-embedding-3-large}$ (0.707 vs.\ 0.556 nDCG@10), with the largest gains in low-resource
languages. We release the
benchmark, pipeline, models, and a reusable PI-FT framework.
Read the full abstract ▾
Computation and Language
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Wenhan Ma, Jianyu Wei, Liang Zhao et al. (+10 more)
Modern large
language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy
Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of
MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.
Read the full abstract ▾
All authors Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, Fuli Luo
Computation and Language · Cross-listed from cs.CY
Uncovering Salience-Driven Dynamics in Consumer Confidence with Generative Social Simulation
Yixu Huang, Yunlu Yin, Jiayu Lin et al. (+6 more)
Consumer confidence is typically modeled as a persistent macroeconomic index, yet its movements arise from households that interpret economic information through heterogeneous constraints, exposures, prior beliefs, and
attention. We introduce ConsumerSim, a generative Human–Environment response framework that reconstructs Consumer Confidence Index (CCI) dynamics from a microdata-calibrated synthetic population, time-stamped macroeconomic, financial, policy, and news signals, survey-like response generation, post-stratified belief expansion, and behavioral inertia
alignment. Across U.S., EU27, and Japanese official CCI target series, ConsumerSim ranks first among persistence, time-series, regression, and information-augmented baselines on the reported reconstruction metrics, with clear gains around high-salience shocks. Its reconstructed signal also improves short-horizon prediction of real activity, most consistently for housing outcomes. Mechanism analyses show that CCI movements concentrate around salient events; subgroup trajectories often align in direction while differing in magnitude; and signal sensitivity varies across income, homeownership, education, and political-
alignment groups. Population-expansion and ablation results indicate that representative aggregation, situational signals, persona heterogeneity, and inertia are necessary for both accuracy and diagnosis. The findings support a behavioral view of consumer confidence as an interpretable Human–Environment response process rather than a purely aggregate time series.
Read the full abstract ▾
All authors Yixu Huang, Yunlu Yin, Jiayu Lin, Xinnong Zhang, Jia Wang, Siyuan Wang, Xuanjing Huang, Liyin Jin, Zhongyu Wei
Computation and Language · Cross-listed from cs.DB
MaDI-Bench: An End-to-End Data Integration Benchmark
Aaron Steiner, Ralph Peeters and Christian Bizer
Data integration is the process of combining data from multiple, heterogeneous sources into a consistent, unified representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, blocking, entity matching, and data fusion. Existing table-based
benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration
benchmarks hinders research on data integration methods that address the integration process as a whole and account for the interdependencies among the different tasks. This paper fills this gap by introducing the Mannheim Data Integration
Benchmark (MaDI-Bench), the first
benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and data fusion pipeline, and (ii) a generic method for deriving task variants that mitigates rapid
benchmark saturation as data integration systems advance. We validate the
benchmark using human-engineered pipelines, a best-of-breed pipeline, an LLM workflow, and a pipeline written by a coding
agent. The validation demonstrates the utility of the
benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All
benchmark artifacts are available for public download.
Read the full abstract ▾
Computation and Language
OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
Karl El Hajal and Mathew Magimai.-Doss
We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised
speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction.
Read the full abstract ▾
Computation and Language
REAR: Test-time Preference Realignment through Reward Decomposition
Fuxiang Zhang, Pengcheng Wang, Chenran Li et al. (+6 more)
Aligning large
language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference
alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of
token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS
algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scalable test-time realignment for preference
alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.
Read the full abstract ▾
All authors Fuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li, Yuxin Chen, Lang Feng, Chenfeng Xu, Masayoshi Tomizuka, Bo An
Computation and Language
DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information
Roland Roller, Vera Czehmann, Derya Erman et al. (+13 more)
Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis. However, responsible data sharing requires the detection and removal of personally identifiable and sensitive information to protect individual
privacy. To support the development and evaluation of automatic de-identification systems, we present DialogPII, a
multilingual dataset of synthetic dialogs and
speech-derived transcripts for personal information detection. DialogPII covers eight interaction scenarios (emergency calls, medical anamnesis interviews, therapy sessions, insurance communication, customer support, clinical interviews regarding an AI-supported dashboard, police reports, and group therapy discussions), 19 entity types, and 11
languages (English, Arabic, Finnish, French, German, Hindi, Italian, Polish, Portuguese, Spanish, and Turkish). Dialogs were generated semi-automatically using large
language models, manually curated for plausibility and diversity, and localized to country- and city-specific contexts. All dialogs were additionally converted to
speech via text-to-
speech synthesis, transcribed with Whisper, and annotated through automatic projection and manual correction, yielding aligned written and
speech-derived resources across all
languages. We further release baseline
multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and
benchmark experiments with
transformer-based sequence labeling models.
Read the full abstract ▾
All authors Roland Roller, Vera Czehmann, Derya Erman, Luke Flanagan, Ibrahim Baroud, Frédéric Blain, Viviana Cotik, Eletta Giusto, Akhil Juneja, Mariana Neves, Maria Słowińska, Christine Hovhannisyan, Aaron Louis Eidt, Lisa Raithel, Sebastian Möller, Maija Poikela
Computation and Language · Cross-listed from cs.LG
When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding
Aaryam Sharma
Speculative decoding accelerates
language model inference by using a fast drafter to propose candidate
tokens that are then verified by a larger target model. Existing theory largely studies the
stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and
entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy
token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.
Read the full abstract ▾
Computation and Language
Multi-Agentic System Leveraging Open-Source LLMs to Mitigate Disinformation Threats
Sebastian Kula and Martin Tamajka
In contemporary societies, the threat of disinformation has reached alarming levels, exacerbated by the proliferation of electronic communication, social media, and advancements in artificial intelligence. As a result, there is an urgent need to develop effective countermeasures to mitigate this menace. However, the sheer scale of the problem renders manual fact-checking and human-based verification inadequate, underscoring the necessity for automated methods to detect and debunk disinformation. This article proposes a novel approach based on a multi-
agent system that emulates the decision-making processes of human annotators engaged in disinformation detection tasks. By incorporating a consensus mechanism, diversity in cognition and diversity in knowledge, and also hierarchical structure, inspired by human annotators' behavior, the proposed method achieves superior results compared to individual Large
Language Models (LLMs), including GPT 4 and GPT 3.5. The system leverages open models (e.g., LLaMA, Kimi, Qwen, Deepseek and LLaMA-Nemotron) to ensure greater transparency. The evaluation of the proposed method encompasses
datasets in
languages with varying resource availability, including English (high-resource), Polish (medium-resource), Slovak (low-resource) and Bulgarian (low-resource). Experiments were conducted on tasks such as direct disinformation detection, identification of texts worthy of verification, and detection of texts containing verifiable factual claims.
Read the full abstract ▾
Computation and Language
Grounding LLM Reasoning under Incomplete Graph Evidence
Jiaqi Li and Fanghui Song
Knowledge graphs can guide large
language models (LLMs)
reasoning, but the graph seen by a system is usually a retrieved, linked, temporally scoped, and incomplete evidence state rather than a complete account of truth. We develop a theoretical perspective on grounding observable LLM trajectories under such incomplete graph evidence.The evidence state induces entity anchors, typed relation residuals, path energies, and support regions, while the
language model supplies a prior over candidate trajectories. We show that, under open-world incompleteness, no hard rule based only on the observed state can both reject every false unsupported trajectory and retain every true-but-unobserved one.We then characterize soft grounding as a KL-regularized deformation of the LLM prior: finite slack preserves support for unsupported but non-contradicted trajectories, whereas hard conditioning appears as an infinite-penalty limit.The framework also yields stability bounds under evidence perturbations and clarifies the constraint regimes appropriate for GraphRAG, KGQA, graph
agents, constrained decoding, and faithful generation. The claims are evidence-relative: KG compatibility is treated as declared support, not factual truth.
Read the full abstract ▾
Computation and Language
Comparing Human and Automatic Recognition of Dutch Dysarthric Continuous Speech: A Case Study
Yuanyuan Zhang, Dimme de Groot, Jorge Martinez et al. (+1 more)
In our goal to develop personalised dysarthric
speech recognition (DSR) models, this study compared the recognition performances of human listeners and those of three state-of-the-art, off-the-shelf ASR systems (Whisper-large-V3, Google Chirp 3, and Omnilingual) on the recognition of Dutch continuous read and spontaneous
speech from a single speaker with severe dysarthria. Results showed that both humans listeners and the three off-the-shelf ASR systems exhibit word error rates (WER) exceeding 70% on average, indicating that DSR is highly challenging for both humans and ASR systems. Fine-tuning on the dysarthric
speech significantly reduced WER. Although overall WERs are still quite high (>23%), the personalised DSR models outperformed the human listeners, and performance is getting closer to being useful for supporting day-to-day communication of dysarthric speakers. Future research should focus on improving personalized DSR on spontaneous
speech and longer utterances in the case of read
speech, with a specific focus on particular phonemes.
Read the full abstract ▾
All authors Yuanyuan Zhang, Dimme de Groot, Jorge Martinez, Odette Scharenborg
Computation and Language
CaresAI at CT-DEB26: Detecting Dosing Errors In Clinical Trials Using Domain-Specific Transformer Embeddings and Classification Models
Leon Hamnett, Favour Igwezeke, Joseph Itopa Abubakar et al. (+1 more)
Medication errors, particularly dosing errors in clinical trials (CT), can lead to patient harm, adverse drug events and worse patient outcomes. Dosing errors are preventable, and early identification can improve trial integrity and mitigate subsequent clinical and financial burden. This study aims to detect dosing errors within CT protocols by evaluating text representations of trial information using
transformer-based
language models trained on biomedical corpora. CT textual data was encoded using several models, including ClinicalBERT, PubMedBERT, BioBERT, and MedCPT, and integrated with categorical features. These text
embeddings were used as input to classical machine learning models and neural network architectures within an experimental framework. Performance was primarily assessed using ROC-AUC with respect to predicting dosage error. Under a logistic regression baseline, BioBERT consistently outperformed alternative encoders, achieving an ROC-AUC of 0.794, a 3.95% improvement over the ClinicalBERT baseline. Combining multiple
embeddings did not yield improvements, indicating that domain
alignment outweighs representational stacking. Gradient boosting models, support vector classifiers, logistic regression, and residual neural networks achieved the strongest performance for predicting dosage error, achieving ROC-AUCs: 0.821 to 0.853. Overall, the integration of domain-specific
transformer embeddings with structured metadata enables discrimination of trials meeting a predefined elevated dosing error risk criterion, advancing safety monitoring and supporting informed regulatory decision-making.
Read the full abstract ▾
All authors Leon Hamnett, Favour Igwezeke, Joseph Itopa Abubakar, Mary Adetutu Adewunmi
Computation and Language · Cross-listed from cs.AI
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
Buğra Alperen Ulu{\i}rmak and Rifat Kurban
This paper presents a systematic survey and conceptual synthesis of the shared measurement problem underlying large
language model (LLM) evaluation and AI safety:
benchmark scores, reward signals, and safety metrics can improve while the capabilities and
alignment properties they are meant to represent remain uncertain. Synthesizing 373 primary studies published between 2018 and 2026, the survey organizes evidence on
benchmark validity, contamination, dynamic evaluation, LLM-as-a-judge protocols,
adversarial safety testing, reward and proxy optimization, mechanistic interpretability, and AI governance into an eight-stream evidence taxonomy. Building on this synthesis, we introduce EvalSafetyGap, a conceptual framework that unifies
benchmark-validity and
alignment-failure research as a shared proxy-target divergence problem under optimization pressure, formalized through a Goodhart-inspired Instability Decomposition and an
Alignment Trilemma. An exploratory ten-model public-evidence audit illustrates the framework by showing why capability, behavioral robustness, and governance disclosure should be reported as separate evidence layers rather than collapsed into a single safety score. The survey closes with a research agenda for dynamic and contamination-resistant
benchmarks, pre-specified multi-attempt threat models, version-locked evaluation, transparent source reporting, and validated mechanistic safety indicators, offering researchers, model developers, and AI auditors a shared vocabulary for measurement-aware LLM safety evaluation.
Read the full abstract ▾
Computation and Language
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
Yinan Zhou, Haokun Lin, Yichen Wu et al. (+7 more)
Large
multimodal models have achieved strong
reasoning on complex visual tasks, but their inference efficiency is often restricted by long chains of thought. A promising solution is to pair a small draft model with a large target model, enabling cooperative inference employing a routing signal that adaptively routes queries to either the draft or target model based on their difficulties for optimal efficiency and accuracy. Yet, the remaining bottleneck is to establish a reliable query difficulty signal under
multimodal settings. Existing approaches designed for
language models either rely on post-hoc
token probabilities, which fall short in
multimodal scenarios, or depend on supervised fine-tuning, which is a data-sensitive strategy. Both paradigms perform routing only after a complete output, and ignore whether the target model can actually solve the routed instances. To address this, we propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models. Our Draft Rating Learning (DRL) equips the draft model with an internal confidence
estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query, thereby prioritizing the allocation of samples it excels at rather than the hardest ones. These ratings enable fine-grained, instance-level Proactive Routing and substantially accelerate inference without compromising overall performance. Extensive experiments across multiple
multimodal reasoning benchmarks validate our effectiveness and efficiency.
Read the full abstract ▾
All authors Yinan Zhou, Haokun Lin, Yichen Wu, Yuxin Chen, Teng Wang, Caifeng Shan, Zhenan Sun, Chen Ma, Li Zhu, Ying Shan
Computation and Language · Cross-listed from cs.CV
SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation
Filippo Ruffini, Marco Salmé, Rosa Sicilia et al. (+2 more)
Current evaluation protocols for
Vision-
Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as
vision shortcut. We introduce SHOVIR, a
benchmark for evaluating
vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray
datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and
datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.
Read the full abstract ▾
All authors Filippo Ruffini, Marco Salmé, Rosa Sicilia, Valerio Guarrasi, Paolo Soda
Computation and Language
Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector
Elys Allesiardo, Antoine Caubrière and Valentin Vielzeuf
This paper offers an in-depth analysis of non-sequential
multimodal sentence-level
embeddings, with a particular focus on the SONAR model. We demonstrate that certain
embedding dimensions are sensitive to perturbations and can serve as indicators of decoding anomalies. By leveraging the consistency between successive encoding and decoding, we successfully build an accurate detector. Additionally, we explore modifying specific dimensions of interest to attempt to correct them. This work underscores the importance of understanding and analyzing the
embeddings themselves to enhance the reliability of
multimodal representations.
Read the full abstract ▾
Computation and Language
DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning
Xinxin Chen, Yuchen Li, Zihan Wang et al. (+3 more)
Current
multimodal fusion approaches, particularly those based on static Mixture-of-Experts (MoE) architectures, often struggle to provide the adaptive and efficient collaborative
reasoning required by complex real-world applications. We introduce the Dynamic
Agent-based Interaction Network (DAIN), which reconceptualizes
multimodal fusion as a dynamic, multi-
agent collaborative process. DAIN employs a context-aware Meta-Controller that dynamically schedules sparse activation of specialized interaction
agents and orchestrates compressed inter-
agent communication for consensus-building. The framework is guided by a multi-objective loss function that jointly optimizes task accuracy,
agent specialization, and operational efficiency through sparse activation and communication regularization. Comprehensive evaluations across five diverse
benchmarks – ADNI, MIMIC-IV, MM-IMDB, CMU-MOSI, and ENRICO – establish DAIN as a new state-of-the-art, delivering significant performance improvements including a 2.6% accuracy gain on ADNI. Ablation studies verify the critical roles of both dynamic scheduling and
agent communication. Furthermore, DAIN offers enhanced interpretability by exposing context-dependent
agent roles and collaboration patterns while maintaining computational efficiency through sample-wise sparse
agent activation. Our work demonstrates the promise of dynamic,
agent-based paradigms for
multimodal reasoning.
Read the full abstract ▾
All authors Xinxin Chen, Yuchen Li, Zihan Wang, Haoyu Zhang, Ruixin Liu, Mingyuan Zhao