Peer Revenue Publish or profit

Papers

Fresh from arXiv, typeset properly. Highlighted words trade on the market.

Electrical Engineering and Systems Science Audio and Speech Processing

eess.AS Watch Audio and Speech Processing

Thursday 17 September

Audio and Speech Processing

State-Space-Based FIR Filtering on a Quantum Computer

Roope Salmi, Davide Rocchesso and Vesa Välimäki

Many signal processing tasks require intensive computations. Quantum computing promises to accelerate certain tasks, but algorithms must be designed around the limitations of quantum mechanics. This paper provides a quantum implementation of finite impulse response (FIR) filters, which are a widely used tool in classical signal processing. The filter can be parallelized and composed as part of larger quantum algorithms, with potential for speedups using quantum amplitude estimation and future fault-tolerant quantum hardware. To accomplish these properties, we introduce a state-space framework wherein sample-based signal processing is performed with unitary quantum circuits. We present the quantum delay gate as the quantum analog of the delay line in discrete-time signal processing. Unitary circuits intrinsically describe lossless systems, but we also implement lowpass and other bounded filters by emulating a projection operator. The FIR filter implementation is tested and demonstrated with a quantum circuit simulator.
Read the full abstract

Audio and Speech Processing

Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognition

Yan Jia, Kai Huang, Junjie Chen et al. (+3 more)

In this paper, we propose an alignment-path distillation framework for streaming automatic speech recognition (ASR) with large language models (LLMs). Interleaved streaming ASR-LLMs use forced alignments (FA) from alignment models, such as those trained with connectionist temporal classification (CTC), to construct speech-text training sequences. However, alignments obtained from a separate acoustic model may be inconsistent with those learned by LLM-based ASR. This motivates us to transfer alignment information from a non-streaming ASR-LLM to improve streaming recognition. Specifically, we extract monotonic alignment paths from a non-streaming teacher's soft text-audio attention and use them to construct interleaved training sequences. The framework also includes logit and hidden-state distillation to learn from the teacher's output distributions and internal representations. Experimental results show that, without logit or hidden-state distillation, training with teacher-derived alignment paths achieves a 5.2% relative error rate reduction compared with training using forced alignments. When both models use logit and hidden-state distillation, teacher-derived alignments yield a 3.9% relative error rate reduction, with similar mean emission latency but higher flicker. The complete framework achieves a 16.6% relative error rate reduction compared with training using forced alignments without logit or hidden-state distillation.
Read the full abstract

All authors Yan Jia, Kai Huang, Junjie Chen, Feng-Long Xie, Xu Tang, Yao Hu

Audio and Speech Processing · Cross-listed from cs.CL

Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition

Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello et al. (+4 more)

SpeechLLMs have shown strong potential for emotion recognition, yet they read the predicted emotion off a generative decoder not suited for classification: it can emit labels outside the target set and favors frequent classes. We propose a discriminative adaptation that reads the final prompt token's hidden state through a classification head, producing a label in one forward pass without modifying the backbone. Because this readout starts from the hidden state the model would otherwise decode, it gives a controlled comparison of generative and discriminative inference in an otherwise identical speechLLM. We keep the head a single linear layer, trading little accuracy for interpretability: each emotion becomes one direction in the LLM output token space, revealing associated tokens. On IEMOCAP, across two speechLLM architectures, it improves Macro F1 and removes hallucinations, with largest gains on realistic ASR transcripts. Our analysis reveals that these emotion directions encode indirect associations mirroring biases in web-scale text.
Read the full abstract

All authors Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke

Audio and Speech Processing · Cross-listed from cs.CL

VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D et al. (+3 more)

Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. We first validate evaluation methods against human judgements: Gemini-as-a-judge best approximates human ratings but is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference. Using this validated setup, we benchmark proprietary and open-weight models across input modality, language, and domain. We observe that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively. VākQA is publicly released.
Read the full abstract

All authors Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala

Audio and Speech Processing

Foreground Voice Activity Detection: Learning Speaker Selectivity from Supervision

Guangzhao Yang, Muhammad Huzaifah, Yu Pan et al. (+2 more)

Voice activity detection (VAD) fronts most voice-agent pipelines, yet production detectors treat all human speech, background talkers included, as valid activity; in crowded settings this floods recognition, stalls turn-taking, and triggers false barge-in. We formalize Foreground VAD (FVAD): a frame-synchronous, enrollment-free task in which only the dominant speaker, defined by sustained presence rather than instantaneous loudness, is positive, and which reduces to conventional VAD when a single speaker is present. We show that foreground selectivity is largely governed by training supervision: the crucial ingredient is an augmentation recipe pairing foreground-only labels with competing-speaker mixing, generated fully automatically without human annotation. To quantify selectivity we introduce the Background False-Alarm Rate (BG-FAR), gated by foreground F1, and build a controlled benchmark, Mix-Interference, complemented by an adapted VOiCES for real-world far-field evaluation. Across equal-size backbones, Mamba and LSTM perform on par while a longer-context attention model is no better, suggesting that training supervision plays a substantially larger role than temporal modeling capacity in achieving foreground selectivity. The resulting lightweight streaming model, Mamba-FVAD, outperforms commercial VADs and enrollment-based speaker-aware systems in foreground selectivity while staying competitive on conventional VAD, at 1-2 ms per-frame CPU latency.
Read the full abstract

All authors Guangzhao Yang, Muhammad Huzaifah, Yu Pan, Jinya Sakurai, Ningjie Bai

Audio and Speech Processing

Consensus-Guided Shared-Specific Tri-View Learning for Speech Emotion Recognition

Bing Huang, Yujian Ma, Xikun Lu et al. (+2 more)

Speech emotion recognition (SER) benefits from heterogeneous acoustic representations, but views derived from the same utterance contain both overlapping emotional evidence and representation-dependent cues. Direct fusion may therefore propagate redundant information or obscure complementary details. To address this issue, we propose Tri-view Consensus-Guided Fusion (TriCGF) for jointly modeling spectrogram, Mel-frequency cepstral coefficients, and HuBERT representations. TriCGF organizes each view into common and view-specific components before fusion. Cross-view Consensus Learning aggregates the common components into a global reference, while View-wise Gated Integration adaptively combines this reference with each view-specific component. A soft difference regularizer further discourages excessive information overlap. Under speaker-independent evaluation, TriCGF achieves 74.19% weighted accuracy (WA) and 75.17% unweighted accuracy (UA) on IEMOCAP, and 94.36% WA and 94.28% UA on EmoDB, outperforming representative SER methods on both datasets.
Read the full abstract

All authors Bing Huang, Yujian Ma, Xikun Lu, Xianquan Jiang, Jinqiu Sang

Audio and Speech Processing

Multimodal Conversational Context for LLM-Based ASR: Data Construction, Training, and Benchmark

Longhao Li, Jian Tang, Yuxiang Kong et al. (+4 more)

Conversational context provides semantic and acoustic cues across turns for automatic speech recognition (ASR), but relying on historical transcripts can propagate recognition errors and discard pronunciation and speaker information. We present a multimodal conversational-context framework for LLM-based ASR that integrates a scenario-controlled data pipeline, scalable multimodal context training, and systematic evaluation. We construct dialogues around entities and their confusable forms and interleave historical user speech with assistant text responses for supervised fine-tuning. We also introduce MM-ContextASR Bench, which evaluates contextual understanding and entity error correction across five scenarios. Experiments with Qwen3-Omni and Step-Audio-2-mini reveal limitations in handling irrelevant and erroneous history and show that our data construction and training improve context utilization, with multimodal context achieving the highest overall entity recall on both models. Further experiments on accent, dialect, and target-speaker ASR demonstrate the value of historical speech. The benchmark data and evaluation code are publicly available at https://github.com/llh666521/MM-ContextASR.
Read the full abstract

All authors Longhao Li, Jian Tang, Yuxiang Kong, Jie Chen, Binbin Zhang, Lei Xie, Xiangang Li

Wednesday 16 September

Audio and Speech Processing · Cross-listed from cs.CL

FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech

Puneet Mathur and Dinesh Manocha

Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic voice activity detection lacks this semantic information, while cascaded ASR-based endpointing introduces transcription dependence and additional processing stages. We formulate semantic endpoint detection as a causal audio-language reasoning task and introduce FD-VAD, an ASR-free streaming endpointer that maps bounded causal audio windows directly to Continue/Stop decisions. FD-VAD combines a frozen speech encoder with a lightweight modality adapter and a parameter-efficiently adapted language model, using a last-chunk training objective for streaming inference. We further introduce confidence-gated endpoint commitment to control interruption versus delay and boundary-focused hard-negative sampling to improve decisions around ambiguous turn boundaries. Across in-domain and conversational evaluations, FD-VAD outperforms strong streaming and non-streaming semantic turn classifiers, and achieves the highest EOT recall among qualifying systems on TurnBench dev set $0.853$ (at FP<=0.10) in a zero-shot setting. These results show that semantic endpointing can be performed directly from streaming audio without intermediate ASR or dialogue state tracking.
Read the full abstract

Audio and Speech Processing

OneVoice: An Intermediate Representation for Agentic Speech Pipelines

Vipul Charugundla, Dancheng Liu and Jinjun Xiong

Agentic speech systems must exchange more than text, including speaker identity, timing, phonetic information, and behavioral annotations. Yet these signals are often produced in incompatible tool-specific formats, making agent-to-agent handoff fragile. We present OneVoice, a lightweight, JSON-native intermediate representation that provides a shared semantic structure for speech pipelines. OneVoice organizes heterogeneous speech evidence into validated session records with stable identifiers, layered transcripts, explicit timing relationships, and provenance information. We evaluate OneVoice in two complementary multi-agent workflows covering event aggregation and acoustic-phonetic temporal linking across three language models. Compared with implicit agent-defined handoffs, OneVoice substantially improves aggregation reliability and the preservation of temporal relationships, demonstrating the value of an explicit speech-specific representation for agent communication.
Read the full abstract

Audio and Speech Processing

Cross-Lingual Parkinson's Disease Severity Assessment Using Pre-trained Speech Embeddings: A Multi-Class Evaluation

Simon Pals and Cristian Tejedor-Garcia

Parkinson's disease (PD) often manifests through speech impairments, facilitating accessible, non-invasive, and cost-effective severity assessment for early diagnosis and progression tracking. Despite advances in speech foundation models (SFMs), their cross-lingual generalization for PD severity multi-class classification remains underexplored due to limited labeled data, a lack of explainable methods and variability across languages and datasets. In this work, we evaluate pre-trained embeddings from four state-of-the-art open-source SFMs across three datasets in zero-shot and k-shot cross-lingual settings for multi-class PD severity assessment. Our results show that pre-trained speech embeddings enable meaningful cross-lingual transfer, although performance is sensitive to dataset properties, preprocessing, and adaptation strategy. Misclassifications under these conditions related to inter-speaker variability and atypical speech patterns highlight the need for more robust feature extraction and modeling for PD severity assessment while emphasizing the importance of explainability for reliable clinical insights.
Read the full abstract

Audio and Speech Processing

Inverse Problems in Musical Instrument Modeling: A Structured Taxonomy and Review

Xinmeng Luan and Gary Scavone

Inverse problems arise in a wide range of applications in musical acoustics, including physics-based sound synthesis, musical instrument modeling, design, and optimization. However, these problems are inherently challenging due to their ill-conditioned nature and strong sensitivity to measurement noise. This paper presents a structured taxonomy and systematic review of inverse problems in musical instrument modeling, providing a unified framework for their classification and analysis. We categorize inverse problems in musical instrument modeling into ten distinct tasks: mechanical parameter estimation; geometric parameter estimation; loss estimation; boundary condition estimation; modal parameter estimation; excitation and articulatory parameter estimation; sound matching or model parameter fitting; instrument design and optimization; field reconstruction, characterization and separation; physical model identification and discovery.
Read the full abstract

Audio and Speech Processing

PersianVox: A Prosody-Aware Approach for Speech Dataset Generation from In-the-Wild Data

Saeedreza Zouashkiani, Soheil Khalesi, Saman Soleimani Roudi et al. (+2 more)

Advancement of zero-shot text-to-speech synthesis is currently hindered for low-resource languages by the scarcity of large-scale, high-fidelity speech datasets. Traditional alignment-based methods require rare verbatim transcripts, while standard in-the-wild pipelines often rely on single-model automatic speech recognition and silence-based segmentation, leading to transcription errors and truncated prosody. To address these challenges for the Persian language, this paper introduces PersianVox, a fully automated pipeline designed to generate high-quality speech corpora from unlabeled web data. Our approach integrates a novel prosody-aware segmentation strategy that utilizes acoustic turn-detection to preserve linguistic completeness and optimize utterance duration for long-context modeling. Furthermore, we employ a dual-model agreement mechanism, leveraging two distinct model architectures to filter unreliable transcriptions without ground truth. This pipeline yields a 2,400-hour multi-speaker dataset, the largest open-source speech resource available for Persian to date. Additionally, we provide the first comparative benchmark of speech quality assessment methods for Persian, releasing a human-annotated subset to facilitate future research.
Read the full abstract

All authors Saeedreza Zouashkiani, Soheil Khalesi, Saman Soleimani Roudi, Sajjad Amini, Shahrokh Ghaemmaghami

Audio and Speech Processing

Decaf: A privacy preserving speech codec using speaker disentanglement and canonical voice conversion

Md Shakhrul Iman Siam, Dushyant Sharma, Stanislav Yu. Kruchinin et al. (+1 more)

We present DECAF, a privacy preserving neural speech codec that obfuscates a speaker's voice while preserving linguistic content while maintaining automatic speech recognition (ASR) performance at very low bitrates, inspired by decaffeination. At the transmitter end, speech is encoded into speaker independent content embeddings, which are compressed using residual vector quantization and transmitted without any speaker related information. At the receiver, a canonical speaker embedding, shared a priori between endpoints, is used for waveform reconstruction, enabling deterministic and consistent obfuscation of a speaker's voice. The proposed framework leverages an information bottleneck applied to self supervised representations, along with a separate speaker embedding branch, to achieve effective speaker content disentanglement. We further incorporate a CTC-based auxiliary objective, encouraging content representations that are well aligned with downstream ASR tasks. We show that DECAF operating at a bit rate of 0.5 kbps achieves an Equal Error Rate (EER) of up to 43.5% for a speaker verification system, while maintaining competitive ASR performance, yielding a relative reduction in word error rate of 33.2% compared to a state of the art method.
Read the full abstract

All authors Md Shakhrul Iman Siam, Dushyant Sharma, Stanislav Yu. Kruchinin, Peter Skala

Audio and Speech Processing

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis

Zitao Liang and Chang Gao

Compact acoustic models face a challenging quality-capacity trade-off. We investigate two factors in this regime: encoder context and Mel-spectrogram supervision. A receptive-field-scaling study shows that expanding self-attention beyond 15 phonemes provides no consistent gains in pitch, energy, or duration prediction. Guided by this finding, we introduce a fixed-receptive-field convolutional encoder that reduces the respective prediction errors by 36.0%, 17.3%, and 3.4%. We further show that directly transferring image-domain gradient-variance supervision restores fine-scale variation but degrades predicted quality, motivating a Mel-specific formulation with axis-specific gradients, overlapping local statistics, and log-domain variance matching. GrainSpeech contains only 264.8K parameters and achieves 17.9x real-time Mel generation on a microcontroller (MCU), while attaining UTMOS scores comparable to substantially larger models with less than 1.5% of their parameters. Source code and demos are available at https://github.com/lab-emi/GrainSpeech.
Read the full abstract

Audio and Speech Processing

Absolute Quality Ratings of Speech Enhancement Systems by Listeners of Different Ages and Degrees of Hearing Loss

Matteo Torcoli, Chih-Wei Wu, Andrea Esposito et al. (+6 more)

Speech Enhancement (SE) supports listening, particularly for older adults with age-related hearing loss. Yet, enhanced Speech Quality (SQ) is commonly evaluated by young normal-hearing listeners, and how their ratings translate to older adults remains under-explored. We compared absolute SQ ratings from 40 younger normal-hearing listeners (20-30 years) and 67 older listeners (60-95 years) with diverse audiometric profiles, after screening. Test materials comprised natural dialogues with realistic backgrounds. SQ differences between SE systems that were clear for younger listeners were smaller or inseparable in older groups, regardless of hearing status. Hearing loss severity was associated with lower absolute ratings, but did not strongly modulate the contraction in separable SQ differences. A small, audiometrically mixed subgroup of older listeners showed younger-like rating patterns, suggesting that peripheral audiology alone cannot explain the contraction.
Read the full abstract

All authors Matteo Torcoli, Chih-Wei Wu, Andrea Esposito, Phillip A. Williams, Katrien Cambier, William Wolcott, Antonio Curci, Nicholas S. Reed, Mark Laureyns

Audio and Speech Processing · Cross-listed from cs.SD

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

Giorgia Adorni, Michela Papandrea, Battista Rimoldi et al. (+1 more)

Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essential, yet reliable performance comparison remains challenging. This difficulty stems from differences in system architecture, supported conditioning information, and access mode, as well as heterogeneous and fragmented metrics that cannot be applied uniformly across systems. To address these challenges, we introduce TTM-Bench, a framework that defines a common protocol for systematic, reproducible performance benchmarking of contemporary TTM systems. It evaluates performance along two dimensions: musical-content alignment, quantified by interpretable semantic, genre, and musical-descriptor agreement scores against a common musical specification and summarized by an aggregate score; and computational efficiency, characterized by generation latency and real-time factor, alongside resource use for local models and cost for hosted services. We demonstrate the framework through a preliminary comparative case study, illustrating the complementary evidence captured by these dimensions. The results show that higher musical-content alignment does not systematically coincide with lower computational demands, highlighting the importance of assessing TTM performance through distinct, interpretable measures rather than a reductive overall indicator.
Read the full abstract

All authors Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi

Audio and Speech Processing

Mask-Based Speech Enhancement for Spatial Audio: A Comparison of Ambisonics, Beamforming, and Microphone Channels

Sheli Hendel, Boaz Rafaely and Dorothea Kolossa

Mask-based speech enhancement is widely used for suppressing noise and interference, but its performance in spatial audio algorithms with multichannel output has not been studied extensively. In such settings, speech enhancement must improve speech quality while preserving spatial cues that are essential for localization, spatial awareness, and spatial release from masking. In this work, we systematically compare time frequency masking applied to three signal representations: microphone signals, beamformer outputs, and Ambisonics signals. Performance is evaluated in terms of speech quality, intelligibility, binaural cue preservation, and reverberation preservation. Results reveal a clear trade-off between enhancement and spatial fidelity: beamformer-domain masking achieves the highest speech enhancement scores, while Ambisonics-domain masking better preserves the spatial attributes of the residual interference. All methods preserve the target's localization cues.
Read the full abstract

Audio and Speech Processing · Cross-listed from cs.SD

VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval

Aaron Yee, Fengjie Lu, Jiarui Hai et al. (+5 more)

Speech retrieval has become increasingly important as spoken content continues to grow across meetings, lectures, podcasts, and videos. Existing benchmarks and models have advanced semantic search over spoken content, but largely focus on what is said while overlooking who says it. In many real-world scenarios, however, users need to retrieve speech based jointly on semantic content and a target speaker, where the speaker may be specified naturally through a reference speech utterance rather than a predefined identity. To address this gap, we introduce VoiceTrace-Bench, a benchmark for hybrid speech retrieval in which each query combines text specifying what to retrieve with reference speech specifying who to retrieve. This setting requires models to integrate complementary semantic and speaker information directly from heterogeneous query inputs. Motivated by the joint audio-text modeling capabilities of audio-language models (ALMs), we develop VoiceTrace, a two-stage retrieval framework consisting of VoiceTrace-Emb, an embedding model that learns unified representations for efficient large-scale retrieval, and VoiceTrace-Reranker, a reranking model that jointly examines each query–candidate pair for fine-grained relevance estimation. Experiments show that VoiceTrace achieves state-of-the-art performance on established semantic speech retrieval benchmarks, while substantially outperforming cascade-based approaches on VoiceTrace-Bench, demonstrating its effectiveness for both conventional semantic retrieval and the new hybrid retrieval setting.
Read the full abstract

All authors Aaron Yee, Fengjie Lu, Jiarui Hai, Chenang Jiang, Helin Wang, Siwei Tu, Weitao You, Lingyun Sun

Audio and Speech Processing

Reviving Etter method for autoregressive inpainting: Generalization, evaluation, implementation

Ondřej Mokrý, Matěj Hrdlička and Pavel Rajmic

Audio inpainting aims to restore missing segments in an audio waveform, as encountered in dropouts and packet losses. This paper revisits the autoregressive (AR) interpolation method proposed by Etter, which combines forward and backward AR prediction through a structured linear system, yet lacks a widely used full implementation for audio signals. We provide an open implementation and propose two practical extensions: A formulation that allows high-order AR models even when the gap length is shorter than the model order, and a causal variant tailored to packet loss concealment. The method is evaluated on musical excerpts with gaps up to 80 ms, using SNR and perceptually motivated objective difference grades, complemented by a listening test. The results show that Etter inpainting matches the performance of most AR- and sparsity-based baseline methods, with the exception of the iterative, gap-wise Janssen approach.
Read the full abstract

Audio and Speech Processing · Cross-listed from cs.HC

Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers

Zhixing Chen and Cheng-Zhi Anna Huang

Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.
Read the full abstract

Audio and Speech Processing

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

Pei-Jun Liao, Hung-Shin Lee, Wenze Ren et al. (+3 more)

Multi-encoder fusion extends Large Audio-Language Models (LALMs) beyond speech-centric recognition, but selecting encoders via intuition or exhaustive search often introduces redundant representations and inflates an already constrained compute budget. We propose CUES (Correlation-gUided Encoder Selection), a lightweight heuristic that estimates complementarity through task- and category-level Pearson correlations between encoders' performance profiles, scoring a candidate set from single-encoder evaluations alone–without fusion training during selection. Evaluated on the XARES-LLM benchmark with a frozen SmolLM2-135M backbone (LoRA-adapted) via five-fold cross-validation, CUES consistently identifies the same configuration per track from held-out development splits alone, without using test data for selection. For the broad Track~A suite, CUES selects a cross-family trio (Whisper-medium, mHuBERT-147, and Dasheng-base), achieving a 4.3% relative gain over Whisper-medium (0.771 vs. 0.739). For Track~B text generation, it re-anchors on a focused, speech-only pair (mHuBERT-147 and WavLM-base-plus) and actively abstains from adding a divergent encoder, outperforming mHuBERT-147 by 6.3% (0.589 vs. 0.554). Rather than a failure to scale, this divergence is consistent with a diversity–interference trade-off that CUES navigates per track from correlation signals alone: across the evaluated pool, added cross-family diversity tends toward an inverted-U on broad audio tasks but toward steady degradation on text generation, which favors a focused, speech-anchored set.
Read the full abstract

All authors Pei-Jun Liao, Hung-Shin Lee, Wenze Ren, Kuo-Hsuan Hung, Hung-yi Lee, Hsin-Min Wang

Audio and Speech Processing

Task-oriented neural FOA encoding for SELD from irregular microphone arrays

Jiachen Liu, Yin Cao, Ming Wu et al. (+1 more)

Sound event localization and detection (SELD) systems often rely on first-order Ambisonics (FOA) input, whereas obtaining useful FOA representations from irregular microphone arrays remains challenging. This paper proposes a two-stage SELD framework that learns a task-oriented, FOA-compatible representation from microphone-array signals. A neural residual encoder first refines conventional FOA encoding through a signal-dependent correction. A teacher–student scheme then transfers event and spatial knowledge from theoretical FOA representations through frame-level permutation-invariant knowledge distillation. Experiments on synthetic scenes with tetrahedral and 12-channel Benchmark arrays, together with real stationary-source recordings from the LOCATA dataset, show that teacher guidance consistently improves downstream SELD performance and substantially reduces localization error. Signal-level analysis further shows that lower FOA reconstruction error does not necessarily correspond to better SELD performance, indicating that the distilled representation is optimized primarily for task-relevant spatial information rather than strict FOA reconstruction.
Read the full abstract

All authors Jiachen Liu, Yin Cao, Ming Wu, Jun Yang

Audio and Speech Processing

SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement

Guo-Ruei Tseng, Hung-Shin Lee, Hsin-Min Wang et al. (+1 more)

Lightweight audio-visual speech enhancement (AVSE) models face a critical trade-off between computational efficiency and cross-modal alignment accuracy. While simple concatenation lacks relational expressiveness, dense cross-attention incurs computational overhead and is prone to unreliable cross-modal correspondence under strong acoustic interference. We propose Sparse Graph-Guided Mamba (SG-Mamba), a lightweight AVSE framework that integrates a sparse heterogeneous graph with a linear-complexity Mamba backbone. The graph explicitly models modality-specific relations through content-adaptive attention and cross-frame audio-visual connections, while Mamba captures long-range temporal context. We further introduce an audio skip connection to preserve spectral detail without sacrificing noise suppression. Evaluated on LRS3, SG-Mamba achieves competitive or superior performance against strong lightweight baselines and reaches 13.091 dB SI-SDR under noise-only condition. It also remains robust in cluttered multi-speaker conditions with a competitive cost of 3.45 G MACs (or 6.90 G FLOPs). Results on VoxCeleb2 further suggest that explicit structural priors improve robustness, generalizability, and computational efficiency in lightweight AVSE.
Read the full abstract

All authors Guo-Ruei Tseng, Hung-Shin Lee, Hsin-Min Wang, Berlin Chen

Audio and Speech Processing

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Mohan Shi, Zilai Wang, Natarajan Balaji Shankar et al. (+3 more)

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.
Read the full abstract

All authors Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

Tuesday 15 September

Audio and Speech Processing

Counting Closures in Spanish Trills: A Multi-Corpus Acoustic Study

Mateo Cámara and Maria F. Alcala-Durand

The Spanish trill /r/ is canonically described as a short sequence of lingual closures, yet large-scale acoustic evidence across corpora is scarce, and automatic counters locating envelope peaks tend to conflate each closure with its release. We present a closure-based detector that locates closures gated by a quality filter and cross-checked against an independent autocorrelation-based period estimator. Applied to 3,560 well-formed (voiced, periodic) trill tokens from 356 speakers across six Spanish corpora, the detector yields a median of two closures and an inter-closure period near 36ms, matching the descriptive literature on all corpora. At the speaker level, phonotactic context is the only factor with a robust, medium effect: onset trills (word-initial and post-/n,l,s/) show more closures than intervocalic rr. We find no robust evidence of a sex effect once closures are counted directly. We report reference values and release a reproducible measurement pipeline for Spanish trills.
Read the full abstract

Audio and Speech Processing

Optimal transport of image sources for interpolation of room impulse responses with moving sources

Jesper Brunnström, Filip Elvander and Isabel Haasler

In geometrical acoustics, room impulse responses (RIRs) can be represented by a set of image sources in free space. For a fixed source position, the image sources allow for computing RIRs at arbitrary receiver positions. However, for a moving physical source, the image sources also move, making interpolation more difficult. In this paper we develop an interpolation method for image source positions of a moving source, given image source positions at the start and end of the trajectory. The method exploits the fact that each image source moves the same distance as the physical source. A statistical model is developed to derive cost functions and an appropriate dummy cost used in the proposed partial optimal transport (POT) approach. Through simulated experiments, POT using the proposed cost functions is shown to be effective compared to the alternatives.
Read the full abstract

Audio and Speech Processing · Cross-listed from cs.SD

Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech

Shuhei Kato

Text-to-speech that reads raw text has no lexicon: a rare word is read as guessed, and a native Japanese listener accepts a word only if its reading and pitch accent are both right. A known remedy installs a reading-and-accent channel into a released model, but it needs many recordings. This paper removes the recordings: the frozen backbone reads a sentence containing a common word it already says correctly, and that output serves as the teacher for the same sentence with the word replaced by an annotated reading with its pitch accent. Screened raters judged the tag right on 0.80 to 0.93 of unseen difficult words on four backbones spanning autoregressive, diffusion, and encoder-decoder synthesis; plain kana, which cannot express an accent, got 0.38 to 0.60. On words needing no edit, naturalness is non-inferior on one backbone; on the other three, listeners prefer the unedited rendition by 0.19 to 0.26.
Read the full abstract

Audio and Speech Processing

Differentiable and Severity-invariant Discrete Tokens for Dysarthric Speech Recognition

Huimeng Wang, Xurong Xie, Mengzhe Geng et al. (+5 more)

This paper proposes novel differentiable and severity-invariant (DSI) discrete token approaches that are not only tightly integrated with downstream dysarthric speech recognition tasks, but also minimise discrete token diversity across speech impairment severity groups. Experiments conducted on the UASpeech and TORGO corpora suggest that Conformer models trained using the DSI tokens outperform the comparable baseline HuBERT discrete/continuous features by statistically significant WER reductions of 2.22%/0.78% absolute (9.14%/3.41% relative) and 1.78%/1.06% absolute (18.43%/11.86% relative) on the two tasks, respectively. After system combination, the lowest WERs of 18.90% and 6.38% were obtained on UASpeech and TORGO. Phoneme-specific T-SNE visualizations show that severity-invariant regularization reduces severity-dependent variation by producing greater overlap and less distinct boundaries among severity-group distributions.
Read the full abstract

All authors Huimeng Wang, Xurong Xie, Mengzhe Geng, Haoning Xu, Jiajun Deng, Youjun Chen, Chengxi Deng, Xunying Liu

Audio and Speech Processing

The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS

Qian Chen, Xiangang Li, Xiang Lv et al. (+2 more)

Speech synthesis systems are commonly narrated as a sequence of larger models, better tokenizers, and broader data. This technical retrospective offers a different account of the CosyVoice lineage, from CosyVoice through CosyVoice 2 and CosyVoice 3 to Qwen-Audio-3.0-TTS: progress came from repeatedly relocating the system's dominant bottleneck. Across the lineage, a stable decomposition separates an autoregressive language model that plans speech from a flow-matching model that renders acoustics. What changes is the contract between them. CosyVoice establishes supervised semantic tokens as a content-aligned interface; CosyVoice 2 makes that interface causally available for streaming and removes the utterance-level speaker embedding from the language model; CosyVoice 3 improves the learnability and coverage of the interface through multitask supervision, scaling, and differentiable reward optimization; and Qwen-Audio-3.0-TTS reduces token rate, conditions its renderer on continuous language-model hidden states instead of token embeddings, and progressively aligns the coupled system. We formalize this history through four interface dimensions—representation, ownership, availability, and gradient reach—and separate within-paper evidence from cross-paper comparison. The resulting synthesis connects discrete autoregressive, continuous non-autoregressive, hybrid, and continuous autoregressive speech-generation paradigms, and yields practical principles for diagnosing and training modular speech generators.
Read the full abstract

All authors Qian Chen, Xiangang Li, Xiang Lv, Han Zhao, Tianyu Zhao

Audio and Speech Processing

Segmental Posterior Decoding for Audio Moment Retrieval

Seungdeok Choi, Seongmin Choi, Inhan Choi et al. (+3 more)

Audio moment retrieval (AMR) identifies temporal segments in long recordings that best match a free-form text query. Existing systems largely rely on fixed-slot DETR decoders that assign proposal-level confidence scores without explicitly normalizing over competing explanations of the full timeline. We propose segmental posterior decoding, which defines a globally normalized distribution over temporal segmentations and scores each candidate moment by its exact segment marginal posterior computed through forward-backward inference. We further expand the training segmentation space by treating a foreground span and its adjacent subdivisions as distinct hypotheses, thereby increasing competition among alternative segmentations. On CASTELLA, our method achieves 41.15% R1@0.7 and 34.68% mAP, outperforming the same network decoded with DETR slot confidence by 10.91 and 9.20 percentage points, respectively.
Read the full abstract

All authors Seungdeok Choi, Seongmin Choi, Inhan Choi, Junho Kim, Jeong-gyu Ban, Yong-Hwa Park