Audio and Speech Processing · Cross-listed from cs.CL
$\tau$-Multilingual: Benchmarking Voice Agents Across Languages
Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela et al. (+3 more)
English-only
benchmarks expose only a narrow slice of voice-
agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated
language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release
language packs, validated judges, and tools for community-built multilingual voice-
agent evaluation.
Read the full abstract ▾
All authors Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan, Keshav Dhandhania, Victor Barres
Audio and Speech Processing
Distributional Metrics for Evaluating Spoken Conversational Systems
Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová et al. (+4 more)
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational behaviour using human conversations as a reference. CDS describes
speech rate, syllabic rhythm, and turn interaction through eight interpretable features plus a separate two-feature semantic baseline. We compare conversations with two reference scales: one based on conversational success within human dialogue and another contrasting human and synthetic dialogue. Using listener judgments from out-of-domain goal–oriented dialogues, we examine system ranking, preferences between conversations, and ranking stability. Composite CDS recovers five of six listener system comparisons while individual features show strong correlation with listener preferences between conversations. We examine how many minutes and conversations are required before rankings stabilize. These findings support distributional comparisons as a complement to specific interactional metrics to evaluate conversations and conversational models while showing their interpretable value.
Read the full abstract ▾
All authors Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová, Maike Züfle, Éva Székely, Nicholas Sanders, Ondřej Klejch
Audio and Speech Processing
Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis
Biel Tura Vecino, Yoach Lacombe, Julian Weber et al. (+4 more)
Classifier-free Guidance (CFG) is widely adopted in text-to-
speech (TTS) systems to enhance generation quality and conditioning fidelity by interpolating between conditioned and unconditioned predictions. A common unconditional technique is to use an empty representation, in the form of a fixed null vector. In this work, we propose replacing this representation with a learnable unconditional
embedding, optimized to represent a meaningful unconditional state. Objective and subjective evaluations demonstrate that learnable null
embeddings consistently outperform fixed null
embeddings across speaker similarity,
speech stability, and expressiveness, while exhibiting greater robustness to larger guidance scales. We further show that learning a distinct unconditional
embedding for each of the TTS conditioning modalities allows fine-grained control over speaker and text guidance, showcasing the trade-off between similarity and quality, and stability and expressiveness in the generated
speech.
Read the full abstract ▾
All authors Biel Tura Vecino, Yoach Lacombe, Julian Weber, Zbigniew Łatka, Haitong Zhang, Logan Hart, Eren Gölge
Audio and Speech Processing
SPADE: A Multilingual Dataset for Speech Partial Deepfake Detection and Localization
Yuan Tseng, Aishwarya Fursule, Andrew Zijun Ma et al. (+4 more)
Recent improvements in voice-cloning
speech generation systems raise concerns about misuse by malicious actors to impersonate others and spread misinformation. Detecting such tampering is difficult, since deepfakes in the wild may be created by different generative models in a wide range of
languages. Furthermore, the
speech audio may also only be partially modified, presenting a different and potentially more challenging task than detecting fully-synthetic
speech waveforms. To enable further research in this direction, we propose a multilingual
dataset for detection and localization of partially edited
speech samples. Our
dataset includes
speech in 12
languages, generated by up to five systems per
language, and includes both a training set as well as an evaluation
benchmark. To showcase the utility of our proposed
dataset, we train localization models of existing architectures and study generalization across three axes: across different
languages, across different
speech synthesis and editing systems, and across different acoustic environments. Our results show that localization models almost always generalize poorly to
speech edited by systems not seen during training. On the other hand, generalization to edited
speech in unseen
languages still degrades performance but to a lesser extent. We also augment our testing sets with noise to evaluate generalization across acoustic environments, and find that performance of localization models degrade significantly when tested on different acoustic conditions. All together, our results imply that existing deepfake
speech detection methods are insufficient for reliably detecting edit-based
speech deepfakes in various scenarios unseen during training. SPADE is publicly available on HuggingFace.
Read the full abstract ▾
All authors Yuan Tseng, Aishwarya Fursule, Andrew Zijun Ma, Vamshi Nallaguntla, Anderson Avila, Shruti Kshirsagar, David Harwath
Audio and Speech Processing
Qwen-Audio-Agent Technical Report
Chong Deng, Yunjie Ji, Yuxiang Kong et al. (+5 more)
We present Qwen-Audio-
Agent, a harness that combines full-duplex voice interaction with asynchronous task execution through a foreground-background architecture. A Frontend
Agent manages dialogue and selects between direct tool use and delegation, while a Backend
Agent carries out delegated tasks in a separate context. An Orchestration Runtime maintains task state, coordinates requests for user input and authorization, and schedules the return of results to the conversation. The runtime separates
speech interruption from task cancellation and execution completion from result delivery, allowing conversation to continue while delegated work proceeds. Environmental events and persistent
memory provide context within and across sessions. Independent adapters support integration with different frontend models, backend
agents, and clients. We instantiate the architecture in desktop assistance, intelligent cockpits, and voice customer service. On an in-house cockpit
benchmark of 134 cases, mixed execution achieves a task success rate of 91.04%, compared with 72.39% and 80.60% for the direct and all delegated configurations, respectively. In a separate latency evaluation on matched successful turns, mixed execution reduces mean task execution latency by 26.73% and 30.91% relative to these baselines, respectively. These results support the complementary use of direct tool calls for immediate operations and backend delegation for multi-step tasks.
Read the full abstract ▾
All authors Chong Deng, Yunjie Ji, Yuxiang Kong, Xiangang Li, Xu Li, Binbin Zhang, Haina Zhu, Jianheng Zhuo
Audio and Speech Processing
Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
Lujia Bao, Qian Chen, Luyao Cheng et al. (+15 more)
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy
Distillation (M$^{2}$-OPD) to transfer
language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio
reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex
speech-to-text adaptation of $\tau$-Voice. On
speech-to-
speech Full-Duplex-Bench v1.5, the response rate to background
speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground–background coordination and
memory.
Read the full abstract ▾
All authors Lujia Bao, Qian Chen, Luyao Cheng, Chong Deng, Yuxiang Kong, Xiangang Li, Xu Li, Jiaqing Liu, Chao-Hong Tan, Haoyu Wang, Wen Wang, Xilou Wang, Haoxiang Xu, Junhao Xu, Liang Yi, Binbin Zhang, Qinglin Zhang, Qiquan Zhang
Audio and Speech Processing · Cross-listed from cs.SD
Fast Time-Varying Exponentiated Convolution Methods for Generative Direction Dependent Reverberation
Yuancheng Luo
Spherical harmonic encoded acoustic sound-fields capture directional characteristics of room impulse responses that are useful for accurate spatial audio reproduction. However, high costs of multi-microphone measurements and numerical simulations motivate alternative data-set augmentation and synthetic data generation methods that supplement small collections. This paper introduces time-varying exponentiated convolution methods that transform both Gaussian noise and impulse responses into reverberation and modified spectral-decay fields respectively. We derive two recursive and fast convolution
algorithms that extend into the spherical harmonic domain, model smooth reverberation time distributions with non-stationary Gaussian processes, and realize an optimal filter design. Experiments evaluate computational performance, and validate out-of-distribution generated impulse responses.
Read the full abstract ▾
Audio and Speech Processing
XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts
Ben Heritage, Luca Resti, Mónica Villanueva Aylagas et al. (+3 more)
Localising artifacts in synthetic
speech remains challenging, as most evaluation methods yield only global quality scores. This paper presents XSQ-AST, a framework that combines the SQ-AST
speech quality model with WhisperX phoneme
alignment and multiple saliency methods to produce temporally localised artifact diagnostics without model retraining. Saliency maps are projected onto continuous distributions via kernel density estimation and onto phoneme boundaries via phoneme-discretised saliency maps. A 40-participant listening test validated the framework across five perceptual dimensions.
Attention Rollout,
Attention Flow and an adapted GradCAM produced temporal distributions that correlated with listener highlights, with different methods best suited to different artifact types. An AUC-ROC analysis confirmed discrimination above chance.
Read the full abstract ▾
All authors Ben Heritage, Luca Resti, Mónica Villanueva Aylagas, Timothy Mehlenbacher, Konrad Tollmar, James Alfred Walker
Audio and Speech Processing
Morpho-VITS: Variational Inference with Morphological Modeling for End-to-End Speech Synthesis of a Tonal Bantu Language
Antoine Nzeyimana
Text-to-
speech models for Bantu tonal
languages are challenged by a tonal system that is rooted in both the lexis (i.e., the inventory of words, stems, and affixes) and the grammar (i.e., morpho-syntax). To complicate matters, the standard writing systems of these
languages often omit tone markings and syllable duration information, which must be disambiguated by the reader based on context. Motivated by linguistic descriptions of Bantu
language tone systems, we propose an end-to-end text-to-
speech model that augments the text encoding mechanism with a morpho-syntactic prior. We replace the standard phoneme encoder in the VITS architecture with a morpheme sequence encoder and a phoneme-to-morpheme
attention network. We posit that, by using this explicit morphological modeling, we can capture the information required to produce the correct tone. Experiments conducted on the Kinyarwanda
language, a tonal and morphologically complex Bantu
language, reveal substantial TTS improvement from this morphological modeling. Specifically, the proposed method significantly improves the naturalness, intonation, and intelligibility of the produced synthetic voices.
Read the full abstract ▾
Audio and Speech Processing
StreamTN: A Low-Latency Streaming Chinese Text Normalization Model for Streaming TTS in Dialogue Systems
Wenhao Li, Jinrui Liang, Haoyu Zhang et al. (+11 more)
Text-to-
Speech (TTS) is an essential module that provides spoken responses in a spoken dialogue system (SDS) centered on a large
language model (LLM). To ensure accurate TTS synthesis, responses generated by an LLM must be converted into TTS-readable formats via a Text Normalization (TN) module, imposing strict low-latency requirements in real-time SDS scenarios. Existing TN solutions are largely rule-based, rely on manual engineering, and generalize poorly to unseen patterns. Although an LLM itself can perform TN through
prompt engineering, it faces key limitations: high first-
token latency due to non-streaming processing,
hallucination risks, and degraded intelligence or
reasoning when the core LLM module is fine-tuned solely for TN. To address these challenges, we propose StreamTN, a lightweight LLM-based Chinese streaming TN model. Built on Qwen3-0.6B, StreamTN employs a dual-track streaming framework in which input
tokens and output
tokens are processed on two parallel tracks, enabling low-latency real-time inference without complex prompting. Moreover, task-specific fine-tuning yields superior TN performance and fewer
hallucinations than rule-based systems and general-purpose LLMs. We also introduce a TN
benchmark that spans diverse text scenarios, providing a comprehensive evaluation standard for
speech generation in spoken dialogue systems. Experiments demonstrate the effectiveness of StreamTN in accuracy and inference latency.
Read the full abstract ▾
All authors Wenhao Li, Jinrui Liang, Haoyu Zhang, Jingbin Hu, Xiaming Ren, Hanke Xie, Huakang Chen, Chengyou Wang, Dake Guo, Linhan Ma, Su Feng, Houdun Liu, Yunxiang Chen, Lei Xie
Audio and Speech Processing
Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang et al. (+4 more)
Large Audio
Language Models (LALMs) can follow diverse instructions to synthesize
speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent
reasoning models have shown that intermediate "thinking"
tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend
reasoning to the audio
token space by training a LALM with reinforcement learning to reason over its own
speech output. The model first generates a draft
speech as a form of audio-
token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass
speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15% on the InstructTTSEval
benchmark, demonstrating the model's reflective ability.
Read the full abstract ▾
All authors Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang, Yun-Shao Tsai, Ho-Lam Chung, Xuanjun Chen, Hung-yi Lee
Audio and Speech Processing
P2Flow: Phoneme-aware Progressive Flow Matching for Extreme Speech Super-Resolution
Ningyuan Yang, Yize Li, Pu Zhao et al. (+5 more)
Generative models have recently demonstrated considerable promise in
speech super-resolution (SSR). Nevertheless, the majority of existing work has concentrated on standard or versatile SSR configurations, leaving the extreme setting with severely limited spectral inputs largely unexplored. In this regime, current approaches exhibit marked performance degradation, underscoring the need for dedicated solutions. To bridge this gap, we introduce P2Flow, a phoneme-aware progressive flow matching (FM) framework designed for extreme SSR with three main strategies. First, our model leverages phonetic information to reconstruct missing spectral components. Furthermore, it employs a progressive architectural design that hierarchically restores distinct frequency regions. Finally, we incorporate post-training of the vocoder to enhance overall waveform fidelity. Extensive experiments are conducted on the TIMIT and VCTK
datasets under both 1 kHz to 16 kHz and 2 kHz to 16 kHz settings, demonstrating that P2Flow yields state-of-the-art results across multiple evaluation metrics.
Read the full abstract ▾
All authors Ningyuan Yang, Yize Li, Pu Zhao, Diego A. Cuji, Kanad Sarkar, Ryan M. Corey, Xue Lin, Andrew C. Singer
Audio and Speech Processing
Performance Analysis and Design of an Optomechanical Microphone Using an Integrated Photonic Waveguide Interferometer
Xiaoyu Niu, Yuqi Meng, Zihuan Liu et al. (+2 more)
We present an optomechanical microphone based on a diaphragm-integrated photonic waveguide Mach-Zehnder interferometer. Acoustic pressure deforms the MEMS diaphragm, inducing strain in the sensing waveguide and changing its optical path length. We analytically evaluate the optical and mechanical transduction mechanisms and key figures of merit, including signal-to-noise ratio, dynamic range, acoustic overload pressure, and minimum detectable pressure. Two design cases are considered: a MEMS microphone and a measurement microphone. The results indicate competitive performance but no substantial overall advantage over state-of-the-art microphones in conventional applications. The architecture may nevertheless offer advantages for high-temperature and other harsh-environment sensing applications.
Read the full abstract ▾
All authors Xiaoyu Niu, Yuqi Meng, Zihuan Liu, Ehsan Vatankhah, Neal Hall
Audio and Speech Processing · Cross-listed from cs.SD
MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification
Yu Zheng, Jinghan Peng, ChangHao Zhang et al. (+2 more)
In this paper, we propose MECT, a speaker verification model that integrates the Mixture-of-Experts (MoE) mechanism into a CNN-
Transformer backbone with optimized block structure and stacking scheme. Specifically, we investigated four MoE variants that span utterance-level and frame-level granularity with dense and sparse routing strategies. The MoE mechanism proves to be effective over the baseline without MoE with only a small increase in parameters. We further scale MECT to a series of model sizes, all maintaining compact parameters and low computational complexity. In particular, MECT-B2 achieves state-of-the-art performance on VoxCeleb1 and delivers strong results on CN-Celeb, demonstrating its effectiveness across diverse
datasets. In addition, we establish a streaming inference paradigm through causal retraining, which maintains strong performance at a chunk size of 100ms.
Read the full abstract ▾
All authors Yu Zheng, Jinghan Peng, ChangHao Zhang, Jian Liu, Weiqiang Wang
Audio and Speech Processing
AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Models
Natarajan Balaji Shankar, Zilai Wang, Zihan Wang et al. (+3 more)
Attention encoder-decoder (AED)
Speech Foundation Models achieve strong ASR performance but can generate acoustically unsupported text when inputs contain no
speech, weak acoustic evidence, or unreliable transcription. We propose AURA: Activation-editing with Uncertainty-Routed Adaptation, an ultra-efficient representation-editing method that freezes the pretrained model and applies sparse scale-and-shift edits to decoder cross-
attention heads. AURA dynamically routes edits using cross-
attention uncertainty features that capture over-concentration, diffuse
attention, and abrupt frame shifts. We evaluate AURA on four
datasets spanning non-
speech hallucination and
speech grounding stressors, including imperfect-label child
speech, imperfect-label adult
speech, and disfluent
speech. On non-
speech audio, AURA reduces
hallucination rate from 89.18% to 1.94% without prior
hallucination-head identification. On imperfect-label corpora, AURA approaches LoRA WER while using roughly 500x fewer trainable parameters. Sensitivity analysis and qualitative cross-
attention examples are consistent with AURA's uncertainty-routed editing behavior, supporting dynamic activation editing as a practical path for grounding AED
speech models.
Read the full abstract ▾
All authors Natarajan Balaji Shankar, Zilai Wang, Zihan Wang, Mohan Shi, Kaiyuan Zhang, Abeer Alwan