Sound · Cross-listed from eess.AS
MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces
Guobin Ma, Yuepeng Jiang, Dake Guo et al. (+15 more)
Continuous latent representations underlie recent
diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: existing systems resort to generator-side machinery such as sentence-level timestamps, auxiliary
alignment objectives, or semi-autoregressive decoding, while the latents they predict are still produced by autoencoders trained purely for reconstruction. We address this difficulty from the representation side. We propose MSVAE (Music Semantic Variational Autoencoder), which structures its continuous latent around two complementary semantic streams: work-level compositional content and lyric-level linguistic-vocal content. Partitioned dual-stream
alignment makes both streams explicit by aligning disjoint latent subspaces with frozen lead-sheet and ASR references, while a semantic-first gated residual posterior derives its base from a frozen music-SSL encoder and admits acoustic information only through gated, bounded corrections. MSVAE achieves competitive stereo reconstruction and stronger MARBLE probing than continuous-autoencoder baselines, and improves downstream lyrics-to-song and text-to-music generation. Notably, with only an untimed phoneme prefix and no lyric timing, MSVAE yields the lowest phoneme error rate, whereas all baseline representations require sentence-level
alignment to produce intelligible vocals.
Read the full abstract ▾
All authors Guobin Ma, Yuepeng Jiang, Dake Guo, Huakang Chen, Yuan Zhou, Lingxin Xiong, Wenkai Cheng, Jingbin Hu, Hanke Xie, Chunbo Hao, Wenhao Li, Ziqian Ning, Jixun Yao, Jian Zhao, Zhonglin Jiang, Yong Chen, Lei Xie, Pengcheng Zhu
Sound
ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding
Ashutosh Chaubey, Jinxi Guo, Minh Tran et al. (+2 more)
Temporal grounding is fundamental to audio understanding, yet large audio
language models (LALMs) remain weak at it on long audio, where
hallucinations compound and annotated data is scarce. Existing
benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified
benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or
token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an
attention-density reward encouraging generation to attend to its own claimed time spans, and (iii)
token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding
benchmarks, with fewer
hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
Read the full abstract ▾
All authors Ashutosh Chaubey, Jinxi Guo, Minh Tran, Michael Andersland, Ozlem Kalinli
Sound
SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning
Chenlong Wei, Hanling Wang, Mingjin Lou et al. (+6 more)
Dynamic spatial audio
reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping
speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The
benchmark covers localization, distance estimation, and motion
reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial
reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22{\deg} to 48.01{\deg}. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29{\deg} azimuth error, compared with 27.14% and 70.74{\deg} for Spatial-Omni.
Read the full abstract ▾
All authors Chenlong Wei, Hanling Wang, Mingjin Lou, Yufan Shi, Yandong Sun, Sheldon Yu, Xiaohui Zhu, Xiaobo Jin, Yiwei Wang
Sound
SpectralBridge: Adding High-Frequency Capability to Large Audio-Language Models
Sergei Polezhaev, Saba Anjum and Michael Gormish
Large audio-
language models (LALMs) have become highly capable on a wide range of tasks spanning
speech, music, and environmental sound. However, many modern LALMs resample input audio to 16 kHz mono, making all frequencies above 8 kHz unobservable. We present SpectralBridge, a lightweight extension that gives such a model access to the 8-16 kHz band by joining low- and upper-band representations with time-aligned paired cross-frequency
attention. Applied to Audio Flamingo Next, the method trains only a small adapter and rank-8 LoRA modules, while leaving the original audio encoder, audio projector, and base
language model frozen. SpectralBridge improves both zero-shot transfer and downstream fine-tuning: it raises zero-shot balanced accuracy on the ESC-50 audio classification
dataset from 75.5 to 90.5, and when fine-tuned on Insect-10, a 10-way insect species task we construct from InsectSet459 recordings, under the same budget as a LoRA fine-tuned baseline, it reaches 55.4 balanced accuracy against the baseline's 26.2. On a
benchmark where the answer depends only on the 8-16 kHz band, it gains 30 mAP over the frozen model. Erasing the upper-band stream from the trained model removes these gains, showing that they come from the high-frequency signal itself rather than from extra
tokens or the adapted
prompt format.
Read the full abstract ▾
Sound
PitchIn: Natural-Language Vocal Coaching from Unstructured Community Feedback
Hayeon Bang, Hounsu Kim, Joonyong Park et al. (+2 more)
Effective vocal coaching connects performance issues with corrective advice, but recordings paired with expert diagnoses and advice are scarce. Online singing communities provide a potential source of supervision, but observations and advice are scattered across unstructured comments. We introduce PitchIn, audio-
language models for natural-
language vocal coaching, and the PeerCoach
Dataset: 14,445 instruction examples covering Focused QA, Overall Feedback, and Recognition QA. We compare target combinations on Qwen2.5-Omni and train PitchIn on Focused QA and Overall Feedback using Qwen2.5-Omni and MOSS-Music. On VocalCoachBench, the proportion of requested diagnosis-correction pairs matching an expert issue with valid advice rises from 21.3% to 61.0% for Qwen2.5-Omni and from 31.4% to 51.2% for MOSS-Music. Direct training on source comments yields only 19.8% for Qwen2.5-Omni. For Qwen, Focused QA alone reaches 65.6%, but adding Overall Feedback improves
alignment with expert reviews and human preference. Both models improve expert-review
alignment over pretrained baselines and are significantly preferred for usefulness (92.0% and 67.3%, respectively). Demo examples are available at https://hayeonbang.github.io/PitchIn.
Read the full abstract ▾
All authors Hayeon Bang, Hounsu Kim, Joonyong Park, Jung-Jae Yu, Juhan Nam
Sound
UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition
Zhe Sun, Liang An, Jiayu Yao et al. (+9 more)
Large audio-
language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing UA-Bench, to our knowledge the first large-scale
benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0%–13.1% leaf accuracy, against a random reference of 7.7%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.
Read the full abstract ▾
All authors Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang, Yujun Cai, Hongli Cao, Yiwei Wang, Xiaoyan Wang
Sound
What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective
Wang Xiang, Bang Zeng and Ming Li
Target speaker extraction (TSE) aims to recover a target speaker's
speech from a
speech mixture using an enrollment utterance. Recent codec-based generative approaches formulate TSE as conditional
speech token generation and achieve promising perceptual quality. However, most existing systems rely on unified codec
tokens that entangle linguistic content, speaker characteristics, and acoustic details, forcing a single model to jointly handle target selection, content semantic, and acoustic reconstruction. In this paper, we investigate whether disentangled semantic-acoustic codec representations are better suited to TSE. We first show that simply replacing unified codec
tokens with semantic-acoustic
tokens is insufficient and can even degrade performance when both remain handled by a unified generative model. Motivated by this observation, we propose a task-aware discriminative-generative framework, where a discriminative branch extracts target-related semantic information, while a generative branch predicts coarse acoustic
tokens and refines them under semantic guidance. This specialization separates what is said from how it sounds while coordinating both factors in a factorized codec space. Experiments on Libri2Mix show that our method reduces dWER from 0.388 to 0.118 compared with a unified-codec baseline, while maintaining competitive perceptual quality and speaker consistency. Ablation studies confirm that the gains arise from task-aware semantic-acoustic modeling rather than representation disentanglement alone.
Read the full abstract ▾
Sound
Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors
Lukáš Prokeš, Vojtěch Staněk, Anton Firc et al. (+1 more)
Deepfake
speech detectors are typically evaluated as binary classifiers on fixed
benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal
embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector
embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three
datasets. The monitor raises alerts for
dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake
speech detectors should be treated not only as a binary classification, but also as a reliability problem.
Read the full abstract ▾
All authors Lukáš Prokeš, Vojtěch Staněk, Anton Firc, Kamil Malinka
Sound
Predicting Audio Popularity: A Comparative Study of Metadata and Acoustic Features Across Music and Sound Effects
Ikram El Hadi, Nihad Elbari, Kawtar Tlemçani et al. (+2 more)
This study addresses the prediction of audio popularity on an online sharing platform by leveraging both metadata and acoustic feature representations. Two distinct categories of audio content are considered: music and sound effects. Popularity is formulated as a supervised classification task based on two target variables: the number of downloads and the average user rating. We conduct a comparative evaluation of multiple machine learning models, including Random Forest, XGBoost, and a Neural Network model. Experiments are performed across different feature sets, including metadata, platform-provided descriptors, and eGeMAPS acoustic features. Results indicate that metadata consistently provide the most informative representation for predicting popularity. Nevertheless, acoustic features alone achieve performance significantly above chance level, demonstrating that intrinsic properties of the audio signal contribute to popularity prediction.
Read the full abstract ▾
All authors Ikram El Hadi, Nihad Elbari, Kawtar Tlemçani, Théo Mariotte, Meysam Shamsi
Sound
PRB-WM: Physics-Regularized Beta World Models for Predictive Link Adaptation on Real Time-Varying Underwater Acoustic Channels
Shengjie Zhong, Zhongliang Zhao, Jingxuan Chen et al. (+5 more)
Adaptive modulation and coding (AMC) improves the spectral efficiency of underwater acoustic (UWA) links, but its feedback is inevitably stale: sound propagates slowly, and by the time channel state information (CSI) arrives, the fast time-varying multipath channel has moved on. Predictive link adaptation therefore needs more than a point forecast: it needs a distribution over each candidate action's consequences. We propose PRB-WM, a physics-regularized Beta world model for predictive per-subcarrier link adaptation on real time-varying UWA channels. A persistence-anchored
stochastic latent-space dynamics model learns how the channel evolves; a counterfactual outcome head evaluates, for every candidate modulation and coding scheme (MCS), a Beta-binomial distribution of bit errors on shared imagined futures; and a chance-constrained receding-horizon model predictive controller maximizes expected spectral efficiency subject to a bounded probability of violating the bit error rate (BER) target. To the best of our knowledge, PRB-WM is the first world model for UWA communications. In closed-loop replay of six measured sea-trial links from the public UWA Channel Library against a 24-method pool spanning the full methodological spectrum, PRB-WM attains a mean goodput 87% above the strongest baseline, ranks first on all six time-varying links at the longest lookahead, and is the only method whose realized frame risk decreases systematically with lookahead, meeting the BER-violation contract where no baseline does. Ablations confirm that rollout sampling dispersion is essential to the chance constraint and that the physics anchor is necessary for trainability.
Read the full abstract ▾
All authors Shengjie Zhong, Zhongliang Zhao, Jingxuan Chen, Dianrun Huang, Zichen Zhao, Yang Liu, Dapeng O. Wu, Tony Q. S. Quek
Sound
LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR
Yuxuan He, Genshun Wan, Pengcheng Li et al. (+3 more)
Code-switching (CS)
speech, characterized by
language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic
speech recognition (ASR) due to rapid
language transitions. We propose LLM-HB, a
language-aware large
language model-guided hotword biasing framework for code-switching ASR (CS-ASR). The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary
language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting. Experiments on the ASRU2019 code-switching challenge test set demonstrate that LLM-HB achieves a 20.30% relative reduction in mixed error rate (MER) over the baseline, reaching 5.85% MER under a 15-distractor hotword setting. To the best of our knowledge, this is the first work to integrate LLM-guided biasing into code-switching ASR. We further release the extracted ASRU2019 hotword list to support future research.
Read the full abstract ▾
All authors Yuxuan He, Genshun Wan, Pengcheng Li, Gongping Huang, Jianqing Gao, Yanmin Qian
Sound
MPASCNet: Magnitude-Phase-Aware Adaptive Spectral Compression for Compressed Music Restoration
Xinlong Deng, Yu Xia, Qiuqiang Kong et al. (+2 more)
Compressed music restoration aims to recover high-fidelity audio from lossy codecs such as MP3 by restoring missing spectral content and reducing coding artifacts. Recently, generative
adversarial networks and band-split models have made progress in compressed music restoration. However, existing methods either operate without explicitly modeling phase or rely on predefined frequency partitions, which may limit restoration performance and the use of input-dependent frequency information. To address these issues, we propose MPASCNet, a Magnitude-Phase-Aware Adaptive Spectral Compression network. Instead of using fixed bands, MPASCNet combines magnitude and phase features to guide cross-
attention, adaptively compressing full-resolution spectral features into compact
tokens. These
tokens are then processed by a spectro-temporal backbone to model spectral-temporal dependencies. Finally, separate decoder branches reconstruct the full-resolution magnitude and wrapped phase spectra. Experiments on MP3 restoration show that MPASCNet outperforms the Apollo and AEROMamba baselines across multiple restoration metrics while using only 6.49 M parameters. The code is available at https://github.com/xinghour/MPASCNet.
Read the full abstract ▾
All authors Xinlong Deng, Yu Xia, Qiuqiang Kong, Yin Cao, Jie Jiang
Sound
Behavior-Conditioned Animal Vocalization Generation: Event-Aware Modeling and Multidimensional Evaluation
Xinlong Deng, Yong Tian, Jian Guan et al. (+3 more)
Behavior-conditioned animal vocalization generation aims to synthesize vocalizations from natural-
language descriptions of animal behaviors, social interactions, and contexts. This task remains largely unexplored, with limited behavioral text supervision and substantial variation in vocalization duration. We construct the Behavior-Vocalization
Dataset (BeVo) by integrating public bioacoustic data from six species with natural-
language behavioral conditions derived from species-specific ethological and bioacoustic literature, totaling 31,191 events. We then adapt a rectified-flow-based generator with Event-Aware variable-length modeling, which encodes vocalizations at their actual durations and uses masks to prevent
attention to padded latent positions and exclude these positions from the training loss. We evaluate vocalization discriminability, acoustic distribution fidelity, and
language-aligned representation fidelity. Experiments show that task-specific adaptation improves all evaluated metrics compared with its zero-shot baseline, while Event-Aware modeling further increases vocalization-anchor macro accuracy from 59.86% to 61.99% and reduces Frechet Audio Distance from 1.40 to 1.22. Conditioning analyses further show that behavioral conditions improve target-anchor discriminability, while swapping behavior shifts generations toward the corresponding vocalization anchors. Demos are available on our project page.
Read the full abstract ▾
All authors Xinlong Deng, Yong Tian, Jian Guan, Qiuqiang Kong, Jie Jiang, Yin Cao
Sound · Cross-listed from cs.CV
SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
Kaisi Guan, Ying Ba, Wenjing Wang et al. (+4 more)
Recent years have witnessed rapid progress in joint audio–visual generation. However, audio–visual synchronization remains a challenge, as generated
speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation
benchmarks, and synchronization-aware evaluation models for generated audio–visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a
dataset containing 80K professional synchronization ratings for clips from eight audio–visual generators. A taxonomy-guided construction pipeline covers four sound-event families:
Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out
benchmark of 1K clips, each independently rated by multiple professionals. Using this
benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic–temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-$N$ selection and reinforcement learning based post-training. We will release the data,
benchmark, model, and code at https://github.com/WeChatCV/SyncReward.
Read the full abstract ▾
All authors Kaisi Guan, Ying Ba, Wenjing Wang, Xihua Wang, Chen Li, Jing Lyu, Ruihau Song
Sound
What Matters in Preference Post-Training of Continuous Autoregressive Speech Language Models
Yingda Shen, Yao Qian, Dekun Chen et al. (+6 more)
Continuous autoregressive
speech language models combine an autoregressive (AR) backbone with a
diffusion or flow-matching head. Preference post-training requires choosing how to construct pairs and which modules to update. Controlled interventions show that a perturbed acoustic patch changes subsequent backbone states, yet larger representation changes do not necessarily reduce
speech quality. We construct preference pairs through noise or inference-configuration perturbations and formulate a preference update based on reference-relative flow-matching errors. Updating the backbone with a fixed head yields stronger reward improvements than head-only adaptation in our evaluated setting. Across flow-time and module ablations, the lowest training loss does not consistently produce the best
speech. These findings support preference post-training that considers both acoustic sampling and autoregressive conditioning, with training choices validated on generated utterances.
Read the full abstract ▾
All authors Yingda Shen, Yao Qian, Dekun Chen, Yufei Liu, Junan Zhang, Chong Jing, Han Wang, Yuxiang Wang, Zhizheng Wu
Sound · Cross-listed from eess.AS
FOAdapter: A Plug-and-Play Spatial Tokenizer for Low-Bitrate FOA Speech Coding
Kirak Kim, Yoonjeong Park, Juhan Nam et al. (+2 more)
Spatially encoded
speech conveys not only linguistic information but also spatial cues. This enables listeners to distinguish and localize speakers, contributing to more natural and realistic auditory experiences in applications such as remote conferencing and immersive media. Neural
speech codecs (NSCs) enable high-quality reconstruction of monaural
speech at increasingly low bitrates while preserving linguistic information, but their ability to effectively preserve spatial cues remains limited. We propose FOAdapter, a plug-and-play spatial tokenizer that leverages a pretrained monaural
speech codec to efficiently represent both linguistic and spatial information in first-order ambisonic (FOA)
speech at low bitrates. Comprehensive evaluation demonstrates that FOAdapter consistently preserves spatial information across all evaluated codecs. Source code and audio examples are available on our demo website.
Read the full abstract ▾
All authors Kirak Kim, Yoonjeong Park, Juhan Nam, Sungyoung Kim, Minje Kim
Sound
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
Liumeng Xue, Yuang Cao, Jingbin Hu et al. (+12 more)
Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware
speech systems must recognize and generate these events while preserving lexical accuracy and perceptual quality. Yet differences in annotation, resources, and evaluation complicate comparisons, and aggregate scores can obscure specific weaknesses. We review ten system descriptions from the NVVSpeech Challenge 2026, covering understanding and generation in Chinese and English across 16 event categories. Our analysis examines supervision construction, task adaptation, and inference-time or preference-based feedback. Results reveal persistent recognition difficulties with breath and gasp and trade-offs between event recognition and lexical accuracy, as well as between generation intelligibility and acoustic quality, with human listening providing complementary evaluation evidence. These findings identify priorities for advancing NVV-aware
speech understanding, controllable generation, and comparable evaluation.
Read the full abstract ▾
All authors Liumeng Xue, Yuang Cao, Jingbin Hu, Qirui Zhan, Huakang Chen, Yibin Zheng, Yunxiang Chen, Houdun Liu, Su Feng, Haibin Wu, Zheng Lian, Wei Xue, Bengu Wu, Hui Bu, Lei Xie
Sound
Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection
Jianqiao Cui, Haoxian Zhu, Hanbiao Meng et al. (+1 more)
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a
stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band
Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on
benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen
languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
Read the full abstract ▾
All authors Jianqiao Cui, Haoxian Zhu, Hanbiao Meng, Yanghao Zhang
Sound
Prime-EVC: One-Step Expressive Emotional Voice Conversion via Emotion-Primed Flow Matching
Yu Pan, Xiaojuan Cai, Liang Zhang et al. (+7 more)
Recent
diffusion- and conditional flow matching (CFM)-based emotional voice conversion (EVC) systems achieve strong conversion quality. However, they typically rely on multiple sampling steps, because the conditioning specifies which emotion to impose while its acoustic realization is left to the sampler. In this paper, we propose
Prime-EVC, a one-step expressive EVC architecture based on emotion-primed flow matching. Our key idea is to reallocate the emotional conversion between the conditioning and the sampler: a FiLM-fused speaker-emotion style, an emotional variance adaptor, and direct emotion and spectro-temporal supervision assemble an emotion-rich conditioning upstream, rendering the residual conversion left to the CFM decoder attainable within one Euler step. Experiments on the ESD corpus demonstrate that single-step
Prime-EVC surpasses five representative EVC systems on every quality metric and exceeds 4- and 100-step DurFlex-EVC in subjective naturalness and emotion while running 3.4–50 times faster than both.
Read the full abstract ▾
All authors Yu Pan, Xiaojuan Cai, Liang Zhang, Yang Hou, Tianjiao Zhang, Yuguang Yang, Jixun Yao, Hiroshi Koide, Lei Ma, Jianjun Zhao