Sound
SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training
Eunseo Choi, Hyunku Kang and Chanwoo Kim
Speech emotion recognition (SER) faces two fundamental challenges: scarcity of labeled data and inter-speaker variability, both of which hinder generalization of emotion recognition systems. While prior
adversarial approaches address speaker variability, they fall short in leveraging powerful pre-trained representations. We propose SISER (Speaker-Invariant
Speech Emotion Recognition), integrating wav2vec 2.0 as a feature encoder and ECAPA-TDNN as a speaker discriminator within an
entropy-based
adversarial training scheme. wav2vec 2.0 provides rich self-supervised representations that alleviate dependency on large labeled
datasets, while ECAPA-TDNN enables suppression of speaker identity via a stronger
adversarial signal than shallow classifiers. Evaluated on IEMOCAP, SISER achieves a UA of 60.63%, outperforming the baseline (51.15%) and wav2vec 2.0 without speaker suppression (56.46%), with ablation emphasizing that the choice of speaker classifier architecture is a key factor.
Read the full abstract ▾
Sound · Cross-listed from eess.AS
U-PAST: A Phase-Aware Audio Spectrogram Transformer-U-Net for Single-Channel Speech Enhancement
Cao Duong Ly and Jörn Anemüller
Convolutional neural networks (CNNs), used widely and successfully in audio enhancement, capture long-range time-frequency dependencies only indirectly, through successive convolution and pooling. Here, we present U-PAST, a hybrid
transformer-U-Net architecture that addresses this limitation through self-
attention dependency-modeling in the complex spectrogram domain. U-PAST tokenizes a complex STFT representation, similarly to the magnitude spectrogram tokenization of the Audio Spectrogram
Transformer (AST), applies a multi-layer
transformer encoder, and reconstructs the enhanced complex spectrogram with a U-Net-style decoder. We evaluate four architectural variants with between 1.17M and 2.40M parameters on the DNS Challenge, VoiceBank-DEMAND, and LibriMix corpora under matched, acoustic mismatch, and two-
dataset mismatch conditions. U-PAST attains the best SI-SDR of any evaluated model under acoustic mismatch and closely trails substantially larger convolutional and time-domain baselines by 0.26 dB to 0.63 dB SI-SDR under the remaining three conditions while achieving the strongest perceptual (DNSMOS) quality under
dataset mismatch. The largest evaluated configuration, U-PAST-H (2.40M parameters), is consistently the strongest variant of the family, offering an attractive performance-to-cost trade-off at a small parameter footprint.
Read the full abstract ▾
Sound
Cleaner Speech, Weaker Generalization: Revisiting Pitt-Derived Benchmarks for Alzheimer's Disease Detection
Luqi Sun, Shreeram Suresh Chandra, Lin Zhang et al. (+5 more)
Speech-based Alzheimer's disease (AD) detection increasingly relies on
speech-enhanced and curated versions of the Pitt Corpus, where
speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of
speech preprocessing and
dataset curation across widely used
benchmarks for
speech-based AD detection. We evaluate the
speech quality of different
datasets, the cross-
dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-
language models (LALMs). Experimental results show that across multiple supervised
speech models,
speech-enhanced
datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced
datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that
speech preprocessing and
dataset curation can substantially influence downstream AD detection behavior, indicating that “cleaner”
speech datasets are not necessarily more reliable for real-world AD detection.
Read the full abstract ▾
All authors Luqi Sun, Shreeram Suresh Chandra, Lin Zhang, You-Jin Li, Brian MacWhinney, Yu Tsao, Emily Mower Provost, Berrak Sisman
Sound
XVAE-WMT: Explainable Wavelet-Temporal Variational Autoencoder for Blind Source Separation of Heart and Lung Sounds
Yasaman Torabi, Shahram Shirani and James P. Reilly
The separation of cardiovascular sounds is a critical task in biomedical signal processing. In this paper, we introduce XVAE-WMT1, an unsupervised explainable generative AI
algorithm combining a variational autoencoder (VAE) with explainable AI (XAI), wavelet-based inputs, a post-hoc output mask, and temporal consistency (TC) loss. Unlike existing supervised and VAE-based methods that rely on Short-Time
Fourier Transform (STFT) and ignore latent interpretability, XVAE-WMT requires no paired clean recordings and integrates a Continuous Wavelet Transform (CWT) front-end for superior time-frequency localization. We assessed the latent space interpretability via different metrics, with SHAP (SHapley Additive exPlanations) enabling dimensionality reduction to the top 75% of latent features while preserving separation quality. Evaluated across two
datasets using Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifacts Ratio (SAR), XVAE-WMT attains 26.8 dB SDR, 32.8 dB SIR, and 28.6 dB SAR.
Read the full abstract ▾
Sound · Cross-listed from cs.CV
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Jiashu Zhu, Yanhao Zheng, Ruitian Tian et al. (+7 more)
Recent
video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-
video generation system centered on a 7B generator. Conditioned on a first frame and a text
prompt, the generator jointly denoises modality-specialized audio and
video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal
Attention, whose
token- and head-wise output gates modulate each active cross-modal
attention-head output. A unified Audio-
Video Data System constructs and filters temporally coherent clips, produces structured
multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-
video pre-training stages followed by High-Quality Finetuning. Audio-
Video Reinforcement Learning further post-trains the generator with Modality-Aware
Multimodal Feedback that routes
video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-
video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-
video generation and provide an accessible foundation for future research in unified audio-
video generative modeling.
Read the full abstract ▾
All authors Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
Sound · Cross-listed from cs.CL
Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
Joonyong Park, Shinnosuke Takamichi, David M. Chan et al. (+3 more)
Neural audio codecs (NACs) convert
speech into discrete
token sequences, and prior work has reported that these sequences follow
language-like statistical laws. This paper analyzes the
token statistics of 13 NACs spanning multi-codebook residual vector quantization (RVQ), single-codebook VQ, and non-VQ designs, evaluated on three corpora under clean, white-noise, and real-world DEMAND-noise conditions. Zipf and Heaps parameters, unigram
entropy, codebook occupancy, and Jensen-Shannon divergence (JSD) are estimated from matched
token samples with explicit fit-validity safeguards and family-conditional $n$-gram orders. Corpus identity explains little variance in any metric, whereas acoustic condition and quantizer meta-category dominate in a metric-dependent way, and unigram
entropy is the metric most strongly associated with meta-category. Clean-to-noise JSD computed at a common unigram order is associated with mel-cepstral distortion most clearly under DEMAND noise. The collapse and explosion degradation signatures previously reported for RVQ codecs concentrate in RVQ
cells under white and DEMAND noise, respectively; explosion also occurs in non-VQ codecs, and single-codebook VQ codecs shift in occupancy and distribution shape without either signature. These results provide architecture-conditioned conventions for applying
language-statistical analysis to NAC
tokens.
Read the full abstract ▾
All authors Joonyong Park, Shinnosuke Takamichi, David M. Chan, Shunsuke Kando, Yuki Saito, Hiroshi Saruwatari
Sound · Cross-listed from cs.CL
When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models
Joonyong Park and Jerry Li
Codec-based text-to-
speech (TTS) models make
language-model post-training applicable to
speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing
alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective
speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward
speech post-training.
Read the full abstract ▾
Sound
CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations
Gabriel Meseguer-Brocal, Yuexuan Kong and Romain Hennequin
Joint-
Embedding Predictive Architecture (JEPA) has shown strong performance in learning rich representations through self-supervised prediction in latent space. However, it typically relies on teacher–student architecture with an EMA to stabilise training, and can tend to yield uninformative representations. Contrastive learning is stable to train and produces strong global representations, but remains limited on local tasks by the global nature of its objective. In this work, we combine both into CoJEPA: a single shared backbone jointly trained with a JEPA objective on masked sequence
tokens and a contrastive objective on the class
token. The contrastive gradient provides stability, removing the need for an EMA teacher entirely, while JEPA enriches the sequence
tokens via local predictions that contrastive learning alone cannot provide. Crucially, no extra parameters are added to the backbone: the same model is guided towards richer representations purely through the design of its training signal. CoJEPA takes the best of both worlds, outperforming or matching both individual methods across global and local MIR tasks, with a particularly strong advantage on tonal and harmonic understanding, and without any task-specific architectural changes. CoJEPA shows that combining objectives with complementary inductive biases can substitute for scale, encouraging future work to invest in smarter training objectives over ever-larger models.
Read the full abstract ▾
Sound
MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI
Laura Ibáñez-Martínez, Roser Batlle-Roca, Xavier Serra et al. (+1 more)
Generative music systems are increasingly presented as tools that democratize music creation, yet their practical suitability for musicians remains underexplored. Prior work includes openness-focused evaluation frameworks, such as MusGO (Music-Generative Open AI), as well as qualitative studies of musicians' experiences with generative systems. However, these approaches do not support systematic comparison or early-stage discovery of models for creative use. Motivated by such limitations, we introduce MusGU+, a musician-centered evaluation framework organized around three dimensions: Adaptability, Usability, and Controllability. Together, these capture whether a model can be feasibly trained or fine-tuned on personal data, integrated into real-world music workflows, and controlled in musically meaningful ways. We evaluate 10 representative generative music systems and present an interactive discovery tool that enables musicians to explore and filter models according to these criteria. While MusGO remains valuable for promoting responsible research practices, MusGU+ supports informed selection and practical adoption of generative systems by musicians.
Read the full abstract ▾
All authors Laura Ibáñez-Martínez, Roser Batlle-Roca, Xavier Serra, Martín Rocamora
Sound
Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
Chanhee Cho, Junhyuk Choi and Bugeun Kim
Whisper exposes
speech through a fixed 1500-
token encoder interface, now a default representation for ASR decoders and Whisper-based
speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th
token after the convolutional stem or encoder
transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and
attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio
tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR
benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-
token interface carries more capacity than downstream tasks require.
Read the full abstract ▾
Sound
Decoupled Latent Flow Matching for Few-Step Joint Vocal-Accompaniment Separation
Lishi Zuo, Youzhi Tu, Lu Yi et al. (+4 more)
Generative modeling provides a flexible way to model mixture-conditioned source distributions, but iterative
diffusion and flow matching models are costly for long music signals. This paper studies joint vocal-accompaniment separation through latent flow matching, where a pretrained variational autoencoder (VAE) maps mixtures and sources into a compact latent space and a flow matching model generates vocal and accompaniment latents jointly. The proposed framework decouples semantic separation from acoustic velocity prediction through a Separation Encoder and a Velocity Decoder. To reduce sampling cost, we further apply latent
adversarial post-training inspired by Flow2GAN for few-step generation. Experiments show that latent
adversarial refinement can improve perceptual and separation metrics under a reduced sampling budget.
Read the full abstract ▾
All authors Lishi Zuo, Youzhi Tu, Lu Yi, Zezhong Jin, Chongxin Gan, Man-Wai Mak, KongAik Lee
Sound
Playability-Aware Audio-to-Tablature Guitar Transcription via Diffusion Models
Riccardo Simionato and Louis Bigo
Guitar tablature transcription requires not only accurate pitch detection but also assigning each note to a specific string-fret position, as the same pitch can be played at multiple fretboard positions. Existing approaches treat this as a standard classification problem, ignoring the musical and physical constraints that govern playable fingering sequences. We propose Noise2Fret, a
diffusion model for audio-to-tablature transcription that generates tablature through a continuous latent representation of discrete fret and string targets, conditioned on spectral and audio features. To bridge the gap between pitch accuracy and physical playability, we introduce five auxiliary losses encoding Pitch-Class Distance, Positional Distance, Circle-of-Fifths Distance, String Similarity, and Hand-Span Feasibility directly into the training objective. Experiments on GuitarSet and GOAT
datasets demonstrate that the model outperforms baselines while remaining computationally more efficient, and that the auxiliary losses yield consistent gains over the standard training objective.
Read the full abstract ▾
Sound
Vocal Music under Phoneme-Conditional Analysis
Hayoon Kim and Kyogu Lee
The vocal music of each
language carries a distinctive sonic identity, even without instrumental accompaniment. We ask whether these differences are measurable and traceable to specific phonemes. To tackle this question, we introduce phoneme-conditional analysis, which isolates the acoustic effect of typologically distinctive phonemes by comparing marker syllables against matched non-marker controls within the same song, holding singer, melody, and genre constant. Across nine typologically diverse
languages and thousands of songs, we measure effects along five acoustic dimensions. Song-level profiles built from these effects identify the
language of an unaccompanied vocal at 85.5% balanced accuracy in a nine-way classification with folds grouped by artist; whether the separability arises by accumulation of the phoneme-local effects themselves is left open. Our findings suggest that phonological structure leaves systematic and measurable traces in how each
language is sung.
Read the full abstract ▾
Sound · Cross-listed from cs.HC
LipCoder: Voice-Enabled Coding Toolkit
Hayoon Kim, Sungho Lee, Juhwi Kim et al. (+2 more)
AI-assisted programming environments have accelerated software development, giving rise to new paradigms like vibe coding. However, their benefits remain largely inaccessible to visually impaired programmers, as existing screen readers and assistive tools offer limited support for these emerging workflows. We introduce LipCoder, a voice-centric programming toolkit designed to deliver editor-level functionality through auditory and
speech-based interfaces. LipCoder offers features comprising
speech feedback and earcon cues for comprehension and validation, as well as natural
language input for navigation and modification. In an exploratory evaluation, 5 visually impaired programmers performed a series of coding tasks comparing LipCoder with a baseline of VSCode, Copilot, and VoiceOver. Quantitative trends and qualitative feedback point to directions for auditory-first design that may broaden accessibility in
speech-driven coding environments.
Read the full abstract ▾
All authors Hayoon Kim, Sungho Lee, Juhwi Kim, Bongwon Suh, Kyogu Lee
Sound
Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning
Fengji Ma, Yan Rong, Xu Li et al. (+3 more)
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual
multimodal language models. We attribute this failure to two structural problems. The first is data poverty, as no public corpus jointly provides long clips, paragraph captions, and verbatim-transcript fidelity. The second is generation-mode failure, evidenced by a 44.8 to 46.4 percentage-point gap between right-audio and shuffled-audio multiple-choice question (MCQ) accuracy. We address both within Self-Check Captioning (SCC), a unified framework that instantiates audio-grounded question answering as the verification primitive at every lifecycle stage. SCC yields three artifacts. Long-paragraph Audio Caption 50k (LACap-50k) is a 50,222-clip audio-visual corpus with 491.5-word captions and a post-hoc automatic
speech recognition (ASR) audit. Layer-
Curvature Supervised Fine-Tuning (LC-SFT) is the first on-policy supervised fine-tuning method to weight
tokens by intermediate-layer evidence, motivated by our identification of Late-Layer Semantic-
Entropy Collapse (SEC). SCC-Verifier arbitrates among caption rollouts via audio-grounded self-answering at inference. Across multiple
benchmarks, our system attains state-of-the-art among open-source captioners and is competitive with proprietary baselines. We release LACap-50k to fill the resource gap for long-paragraph detailed audio captioning research.
Read the full abstract ▾
All authors Fengji Ma, Yan Rong, Xu Li, Chen Zhang, Pengfei Wan, Li Liu
Sound
Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection
Yassine El Kheir, Xin Wang, Wanqing Ge et al. (+3 more)
Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large
Language Models (ALLMs) show promise, the modality gap between continuous audio
embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text
tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD
benchmarks, yielding over 16.2% absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.
Read the full abstract ▾
All authors Yassine El Kheir, Xin Wang, Wanqing Ge, Tim Polzehl, Sebastian Moeller, Junichi Yamagishi
Sound
SPHERE: Automatic Music Upmixing via Audio Language Model Post-Training with Spatial Heuristic Rewards
Zixun Guo, Calvin Murdock, Sanjeel Parekh et al. (+4 more)
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio
language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into
language models, without task-specific architectures.
Read the full abstract ▾
All authors Zixun Guo, Calvin Murdock, Sanjeel Parekh, W Owen Brimijoin, Simon Dixon, Joshua Reiss, Ishwarya Ananthabhotla
Sound · Cross-listed from physics.app-ph
Arctic Dispersion Interruption Phenomenon and Sound Source Depth Estimation
Weng Jinbao, Yang Yanming, Chen Benqing et al. (+2 more)
The sound speed profile in the deep Arctic Ocean causes the surface layer to form a normal mode waveguide. When the depth of the sound source or receiver is near a node of the eigenfunction, the source cannot excite the mode, or the receiver cannot detect it, resulting in the modal amplitude at the receiver being approximately zero. This manifests as a dispersion interruption in the dispersion structure, which can be observed through time-frequency analysis of the received acoustic signal. Based on the interruption frequency identified from the received signal, combined with the relationship between node depth and modal frequency calculated from the ocean sound speed profile, the source depth can be estimated if the receiver depth is known. The phenomenon of dispersion interruption and the method of estimating source depth have been validated through simulations and experiments.
Read the full abstract ▾
All authors Weng Jinbao, Yang Yanming, Chen Benqing, Xu Dewei, Zhou Hongtao
Sound
Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems
Randy Frans Fela and Pejman Mowlaee
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean
speech, and
benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy
speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $\rho=-0.068$, PESQ $\rho=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.
Read the full abstract ▾
Sound
Ouroboros: Self-Referential Backdoor Attacks on Speech Enhancement via Clean Audio Triggers
Yunjie Zhou, Yuheng Huang and Diqun Yan
Speech enhancement models are widely deployed as frontend modules in real-time
speech services, yet their vulnerability to backdoor attacks remains unexplored. Existing backdoor methods are confined to classification tasks and rely on active trigger injection, an assumption incompatible with the passive processing nature of
speech enhancement models. In this paper, we propose Ouroboros, a novel backdoor attack framework that leverages the ideal clean outputs of
speech enhancement models as natural triggers, enabling inference-time activation without any external trigger injection. Extensive evaluations show Ouroboros achieves near-perfect attack success rates with minimal performance degradation on diverse models and
datasets. Physical-world validations confirm that naturally recorded, unaltered clean audio can reliably activate the backdoor. Moreover, Ouroboros generalizes to targeted content-tampering attacks and remains effective against common filtering and finetuning defenses.
Read the full abstract ▾
Sound
Parallel Time-Band Mixing with Learned Observation-Adding for Robust ASR Front-Ends
Xingyu Shen, Runze Wang, Wei-Ping Zhu et al. (+1 more)
Speech enhancement is often used as a front-end for robust ASR, yet recurrent temporal and cross-band modules introduce sequential dependencies that reduce parallel efficiency. In this paper, we present a sequence-parallel band-split enhancement front-end built on a Parallel Time-Band Mixer (PTBM) block that eliminates within-block recurrent unrolling. PTBM integrates intra-band temporal mixing and per-frame cross-band
attention within a unified parallel architecture, enabling efficient contextual modeling across both time and frequency dimensions. The system retains the mask-plus-residual reconstruction interface and introduces learned Observation-Adding (LOA) to suppress ASR-sensitive artifacts without development-set tuning. Experiments on DNS Challenge and CHiME-4 with frozen Whisper back-ends show that the proposed front-end consistently reduces word error rate relative to recurrent band-split baselines while requiring only 0.96 M parameters and 0.58 GMAC/s for the front-end network.
Read the full abstract ▾
All authors Xingyu Shen, Runze Wang, Wei-Ping Zhu, Benoit Champagne
Sound · Cross-listed from cs.CL
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Yan Zhou, Yun Hong and Yang Feng
Natural-
language instructions enable flexible control of synthesized
speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion control underexplored. We study two complementary multi-emotion TTS tasks: emotion trajectory, which spans several ordered affective stages, and emotion blending, in which multiple emotions coexist throughout an utterance. These tasks expose a supervision mismatch: supervised fine-tuning (SFT) does not explicitly evaluate emotion features, while single-emotion rewards provide neither structure-aware feedback for trajectory completion nor pair-aware feedback for blending. We introduce HybridEmo, a post-training framework that initializes both tasks with SFT and then aligns the
speech-
token policy through Group Relative Policy Optimization using a sample-aware hybrid reward. For trajectory samples, segment-aligned consistency combines average and weakest-stage evidence to preserve the correctness and completeness of prescribed stages. For blending samples, a GMM-based reward combines frame-level support from the union of target-emotion anchors in an offline emotion space with an utterance-level weaker-target margin. Both branches share an ASR reward and are routed within a unified policy. On MultiEmo-Test, HybridEmo significantly improves trajectory correctness and blending intensity, without a noticeable degradation in speaker similarity. Human evaluation prefers HybridEmo to CosyVoice 3 and EmoVoice-0.5B, with nearly balanced preferences against Qwen3-TTS.
Read the full abstract ▾
Sound
VIBE: Video Instruction-aligned Background music gEneration
Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj et al. (+5 more)
Current
video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in
Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-
video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and
diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality,
multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual
alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and
multimodal alignment.
Read the full abstract ▾
All authors Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj, Gouthaman KV, Sreyan Ghosh, Ramani Duraiswami, Lie Lu, Dinesh Manocha