Sound · Cross-listed from cs.LG
CircuitTree: Composing Circuits into A Syllable Transcriber
Jiankun Wei, Ewan Dunbar and Gerald Penn
The goal of
circuit discovery is to decompose a monolithic neural network into tiny modular subnetworks which are interpretable in the sense that each does an identifiable subtask. We develop a practical method for composing arbitrarily many
circuits into a single model. We propose a greedy, density-scaled composition
algorithm that overcomes limitations of previous attempts to compose
circuits by naive union, then generalize it to $N$
circuits via CircuitTree, a new data structure built on a $\Theta(N\log N)$ tree-compose schedule. We compose
circuits pruned from HuBERT for classifying individual phonemes into a syllable classifier, a capability none of the 39 phoneme
circuits has individually. Chaining CircuitTrees into a hierarchy-of-
circuits pipeline, we transcribe open syllables at 64.8% accuracy, within 1.2 points of a 66.0% ceiling that shares no circuitry at all, with 99.5% of
circuit components shared.
Read the full abstract ▾
Sound · Cross-listed from eess.SP
Robust Prediction of Internal Wave-Affected Multi-Scale Sound Speed Distribution Using Lightweight Kolmogorov-Arnold Networks with Hybrid Basis Functions
Wei Huang, Junpeng Lu, Tianhe Xu et al. (+2 more)
The underwater sound speed distribution directly governs acoustic propagation paths, rendering it critically important for underwater acoustic communication and target localization. Conventional sound speed profile (SSP) prediction methods provide a good way to estimate the underwater sound speed distribution without on-site data measurement, thus breaking through the coverage area constraints of sonar observation equipment and making the model universal in most marine areas. However, underwater sound speed exhibits multi-scale variations, such as diurnal, quarterly, and intermittent fluctuations caused by ocean processes such as internal waves. This makes it difficult for the fixed structure models in existing methods to have good generalization ability for multi-scale sound speed distribution patterns. To tackle this problem, we proposed a lightweight hybrid basis-function empowered Kolmogorov-Arnold network (LHBF-KAN) model for multi-scale sound speed prediction. We aim to construct a multi-branch representation layer in which different basis functions respond to distinct temporal patterns, from slowly varying background trends to rapid fluctuations induced by dynamic ocean processes, allowing the model to naturally accommodate the inherently multi-scale
evolution of sound speed at different depths. To prevent the multi-branch structure from increasing model size, a
pruning strategy is further introduced to suppress branches with consistently low contribution during training, yielding a compact architecture, suitable for deployment on resource constrained underwater platforms.
Read the full abstract ▾
All authors Wei Huang, Junpeng Lu, Tianhe Xu, Hao Zhang, Feng Yin
Sound · Cross-listed from eess.AS
Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners
Xilin Jiang, Shun Zhang, Tejas Jayashankar et al. (+2 more)
We introduce Conversational Voice Aesthetic Model, a
speech large
language model for describing the voice aesthetics of real or synthetic
speech responses in natural conversational contexts. Given a context and a response
speech, CVAM describes salient moments that characterize the voice and predicts nine categorical attributes spanning gender, pitch, pacing, emotion, and delivery. The key challenge lies in perceptual fields such as emotion and delivery, which are inherently subjective and lack definitive ground truth. Therefore, we collect ~10 human annotations for each of 3k real and synthetic responses derived from the CANDOR corpus. CVAM is supervised finetuned on synthesized aesthetic descriptions and labels, then optimized with Group Relative Policy Optimization on human judgments. Experiments show that CVAM better agrees with human listeners than Gemini 3.1 Pro and open-source
speech LLMs, and outperforms single-human-vs.-rest agreement. Together, we demonstrate the importance of grounding voice aesthetics in human perception and propose a principled framework for human
alignment.
Read the full abstract ▾
All authors Xilin Jiang, Shun Zhang, Tejas Jayashankar, Yinghao Aaron Li, Osama Hanna
Sound
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
Tristan Wu, Daniel Chin, Liwei Lin et al. (+2 more)
Audio generation models can translate natural-
language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression, and their outputs do not readily support direct edits to notes, timbral parameters, or modulation relationships. We present AutoSynth, which represents MIDI performance events, fixed synthesizer parameters, and variable-length modulation routes as a unified sequence for a synthesizer, and learns their dependencies with an audio-conditioned autoregressive model. A single model supports both tasks. Given reference audio, the model directly predicts a synthesizer program; given text, it uses a pretrained audio generation model and converts the generated audio into a program. Training consists of two stages: supervised learning on large-scale audio-program pairs automatically constructed from a small set of native presets, followed by group-relative policy optimization with a mixed reward combining semantic similarity, pitch-related features, acoustic similarity, and sound usefulness. The pipeline requires neither paired text-target-program annotations nor a differentiable synthesizer. Experiments show that AutoSynth produces complete, editable synthesizer programs and achieves competitive results in both synthesizer inversion and text-driven generation. Audio demos and source code are available at https://auto-synth.github.io/.
Read the full abstract ▾
All authors Tristan Wu, Daniel Chin, Liwei Lin, Junan Zhang, Gus Xia
Sound · Cross-listed from cs.CR
Detection-Guided Adaptive Purification with Diffusion Models for Robust Audio Deepfake Detection
Muhammed Salih Kayhan and Qiben Yan
Audio deepfake detectors remain vulnerable to
adversarial perturbations that suppress the acoustic cues used for detection, allowing manipulated utterances to evade the detector. Although existing defenses can improve robustness, they require retraining the detector or introduce additional distortion.
Diffusion-based purification instead leaves the pretrained detector unchanged, but existing methods use the same purification strength for all inputs, creating a trade-off between removing
adversarial perturbations and preserving the subtle spoofing cues needed for detection. In this paper, we propose Detection-Guided Adaptive Purification (DGAP), a
diffusion-based defense that adjusts purification strength per input. Building on the observation that a light purification perturbs the detector score of an
adversarial input far more than that of a benign one, the framework uses the resulting score shift as a reference-free indicator of
adversarial manipulation. Inputs with small shifts are passed unchanged, whereas flagged inputs undergo stronger purification before final detection. We evaluate the framework against three
adversarial attack settings across three deepfake detectors, and compare it with nine existing defenses. Our results show that DGAP achieves the best defense performance across all detectors while leaving benign inputs nearly unaffected, and remains effective under the defense-aware adaptive attack.
Read the full abstract ▾
Sound
Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs
Pooneh Mousavi, Mirco Ravanelli and Cem Subakan
Large audio-
language models (LALMs) fuse an audio encoder into a large
language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for
speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.
Read the full abstract ▾
Sound
BEANS-Next and ROOTS: Broadening Audio-Language Capabilities for Bioacoustics
Christos Plachouras, David Robinson, Marius Miron et al. (+17 more)
Bioacoustics and ethology encompass a wide range of audio understanding tasks, many of which stand to benefit from recent advances in large audio-
language models. However, progress in the field has so far been assessed on a narrow set of tasks, primarily centered on label-centric biological category recognition, such as species and call-type classification. In this work, we introduce BEANS-Next, a
benchmark grounded in a taxonomy of bioacoustics tasks spanning acoustic perception, biological category recognition, scene understanding, and in-context learning. Using BEANS-Next, we show that existing models exhibit limited performance beyond the task families emphasized by existing evaluations, constraining their usefulness for broader bioacoustic applications. To support progress on this broader task space, we also introduce ROOTS, a large-scale training resource built from expanded curated real-world data and previously underused behavioral and acoustic metadata, supplemented by audio-derived information and scalable synthetic generation where labeling is insufficient. We demonstrate that training on this
dataset yields substantial progress across all task groups of BEANS-Next, moving audio-
language models closer to their potential as general-purpose assistants for bioacoustics and ethology. To accelerate progress in the field, we open-source our
benchmark,
dataset, and data pipelines.
Read the full abstract ▾
All authors Christos Plachouras, David Robinson, Marius Miron, Gagan Narula, Paul Laisné, Anthony L. T. Fine, Benno Weck, Ellen Gilsenan-McMahon, Diane Kim, Laura Hay Mack, Maddie Cusimano, Sara Keen, Lukas Rauch, Benjamin Hoffman, Emmanuel Chemla, Emmanouil Benetos, Johan Pauwels, Milad Alizadeh, Matthieu Geist, Olivier Pietquin
Sound
SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding
Yu Chen, Ruihang Liu, Yangguang Xu et al. (+5 more)
Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing
benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce SAVU-Bench, a real-world
benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce SAVU-Diag, a scene-linked diagnostic set that decomposes
reasoning questions into their prerequisite grounding and
alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most
reasoning errors co-occur with failures on these prerequisite tasks, though
reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial
reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.
Read the full abstract ▾
All authors Yu Chen, Ruihang Liu, Yangguang Xu, Xinyue Jiang, Mohammed Bennamoun, Farid Boussaid, Xinyuan Qian, Qiuhong Ke
Sound
Unsupervised Maneuver-Aware Acoustic Fault Detection for Autonomous Drones
Ali M Ali, Ziyi Tang, Nurdaulet Nazarbay et al. (+1 more)
This paper presents a maneuver-aware acoustic fault detection framework for autonomous drones that integrates Noise2Noise-inspired deep learning denoising with maneuver-conditioned reconstruction. A key practical constraint motivating this work is that labeled faulty-flight data are difficult and potentially unsafe to collect; the proposed framework therefore follows an unsupervised learning paradigm in which only nominal flight recordings are required for training. In flight environments, acoustic signals acquired from unmanned aerial vehicles are subject to variability arising both from environmental noise and from structured, maneuver-dependent aerodynamic effects. To address these challenges simultaneously, a two-stage learning architecture is developed. In the first stage, a Noise2Noise-inspired denoising model attenuates
stochastic acoustic noise while preserving fault-relevant spectral-temporal structures, without requiring clean reference signals. In the second stage, a maneuver-Conditioned Convolutional AutoEncoder (maneuver-CCAE) is trained using maneuver-related labels including drone type and flight direction to model nominal acoustic behavior under varying operating conditions. Fault detection is subsequently performed using reconstruction error as an anomaly score. Experimental results demonstrate that the proposed maneuver-aware conditioning raises the area under the ROC curve (AUC) from $\AUCaeOnly$ (unconditioned baseline) to $\AUCfull$ (full model), validating the critical role of maneuver-dependent modeling. The complete framework is deployed on an NVIDIA Jetson Orin Nano Super embedded platform within a ROS2 pipeline, achieving an end-to-end fault detection latency of approximately $20\,\text{ms}$ per audio segment with a TensorRT half-precision (FP16) backend, confirming real-time viability for onboard
UAV health monitoring.
Read the full abstract ▾
All authors Ali M Ali, Ziyi Tang, Nurdaulet Nazarbay, Hashim A. Hashim
Sound
Steerspeech: Activation Steering For Emotion Control In Generated Speech
Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin et al. (+1 more)
Pretrained text-to-
speech (TTS) models can generate expressive
speech, but reliable inference-time emotion control remains challenging:
prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete
speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through
estimator to backpropagate expert supervision through sampled
tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.
Read the full abstract ▾
All authors Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin, Salar Rahili
Sound
MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent
Zekai Liu, Zhilin Wang, Xuzheng He et al. (+2 more)
Text-to-music systems produce increasingly convincing audio, yet evaluation reveals little about whether the result matches user intent. A global text-audio relevance score can overlook the implicit intent in underspecified
prompts and mask failures in specific requirements, such as instrumentation, structure, rhythm, or mood progression. To bridge this gap, we formulate text-to-music intent
alignment as satisfying a per-request rubric of independently verifiable items covering both a request's explicit requirements and its implied musical intent. Scoring items individually makes evaluation diagnostic by intent source and musical dimension, rather than a single opaque score. We instantiate this as MuRA-Bench, a
benchmark of real-world platform requests curated by music experts. We further propose MIRA (Musical Intent Refinement
Agent), a test-time
agent that first grounds a request's intent into rubrics, then searches over
prompt revisions for a black-box generator under a bounded budget, iteratively generating music, verifying it against the rubrics, and using this feedback to guide a trajectory-aware tree search. Experiments across open-source and commercial backends show that MIRA improves intent
alignment, enabling an open-source generator to achieve performance comparable to representative commercial systems (e.g. Suno and Mureka). Project page: https://mirareview.github.io/.
Read the full abstract ▾
All authors Zekai Liu, Zhilin Wang, Xuzheng He, Yu Cheng, Yang Yang
Sound
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
William Chen, Prem Seetharaman, Ke Chen et al. (+10 more)
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and
video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated
benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced
speech editing and standard image,
video, and audio editing
benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.
Read the full abstract ▾
All authors William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte, Siddharth Srinivasan Iyer, Mamshad Nayeem Rizve, Zhiwen Cao, Shinji Watanabe, Yuanjun Xiong, Jianming Zhang, Zeyu Jin, Justin Salamon
Sound
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
Marcel Gibier, Thomas Thebaud, Olivier Boëffard et al. (+1 more)
Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a
language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then
benchmark six audio-
language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.
Read the full abstract ▾
All authors Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre
Sound · Cross-listed from eess.AS
DuRe-ST: Dual-Relation Spectro-Temporal Modeling for Speech Deepfake Detection
Shaole Li, Siqing Qin, Youzhi Tu et al. (+1 more)
Previous
speech deepfake detectors can adaptively capture spectro-temporal dependencies through graph
attention, yet they largely overlook the co-variation between spectral and temporal representations. To address this gap, we construct a normalized affinity graph from their joint covariance and apply
polynomial graph filtering to capture higher-order covariance-induced dependencies. We first develop Cov-ST to isolate the contribution of covariance-based relational modeling. Although it improves detection performance, its sensitivity to the
polynomial order suggests limited robustness when covariance relations are modeled alone. We therefore propose DuRe-ST, which jointly exploits covariance-induced and graph-
attention-induced relations to capture complementary second-order co-variation and adaptive spectro-temporal dependencies. Experiments show that DuRe-ST achieves an average relative EER reduction of 25.9% over XLSR-AASIST on the ASVspoof
benchmarks and 28.4% across four cross-
dataset benchmarks with only 4-8k additional trainable back-end parameters. It further outperforms the strongest publicly available comparison models by 2.2-13.2% in relative EER on four
benchmarks, while remaining smaller than the publicly available models considered.
Read the full abstract ▾
All authors Shaole Li, Siqing Qin, Youzhi Tu, Kong Aik Lee
Sound
A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine
Amirhossein Hajavi, Hanhee Lee, Pushya Jain et al. (+4 more)
Language-queried audio source separation (LASS) enables extracting any sound source using natural
language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.
Read the full abstract ▾
All authors Amirhossein Hajavi, Hanhee Lee, Pushya Jain, Sky Qiao, Emmanuel Ko, Yuanhao Yu, Irina Kezele
Sound · Cross-listed from cs.CL
Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for Robust, Mixed-Dialect and Code-Switched Arabic ASR
Ibrahim Almajai
We describe the Itgan systems for the three ASR subtasks of NADI 2026, namely robust country-level ASR (1.1), mixed-dialect ASR (1.2), and Tunisian code-switched ASR (1.3). All three share one recipe, Whisper adapted with LoRA on consumer GPUs, and each was carried by a different addition to it. On 1.1, where the dialect label is given at test time, per-dialect specialists continued from a pooled adapter gave the largest gain, and the submitted system reached 57.1% country-average WER. A post-evaluation linear probe on frozen encoder features routes utterances without the label and recovers 44% of what oracle routing gives. On 1.2 the choice of base model mattered more than adapter capacity, and system combination helped only once we added a decorrelated member, reaching 46.7% WER. On 1.3 our system placed second at 14.49% WER with the lowest CER among the leading submissions, 5.38%. Its last 0.60 WER points came without further training, mostly from an exact weight-space average of independently trained runs, with ROVER voting adding the remainder. Every comparison carries a paired-bootstrap test, and we report eight directions that did not work.
Read the full abstract ▾
Sound · Cross-listed from eess.AS
Toward Part-Aware Choral Transcription with singing voice assignment
Hanyu Meng, Zhanhong He, Zixun Guo et al. (+1 more)
Single-instrument automatic music transcription (AMT) has advanced substantially, yet choral applications require soprano, alto, tenor, and bass (SATB) to be transcribed as separate parts. Recent note-level choral AMT instead produces a single merged note track, limiting rehearsal, education, and score reconstruction. To address this limitation, we introduce Part-aware Choral Transcription (PawCT), to our knowledge the first end-to-end neural framework that identifies active SATB parts from choral audio and transcribes each into a separate note-level track. PawCT combines part-specific onset, offset, and frame prediction with part-presence estimation, union-level supervision, and structured training targets using a range prior (RP) based on SATB pitch ranges and its ordered-continuity (OC) extension, which adds within-part melodic continuity and cross-part pitch ordering. On YouChorale, PawCT-RP-OC achieves a macro part-aware note F1 of 0.225 at a 50-ms onset tolerance, outperforming an adapted choral baseline (0.165) by 36.4% relative and a two-stage post-hoc assignment pipeline (0.175). Its part-agnostic variant, PagCT, achieves a 50-ms onset F1 of 0.382, compared with 0.237 for the previous state-of-the-art choral AMT model. Cross-
dataset evaluations on CSD and Cantoria further assess performance under
dataset shift. These results demonstrate the benefit of jointly modeling note transcription and vocal-part assignment. Code and demos are available at https://hanyu-meng.github.io/Paw_Choral_AMT_Demo/.
Read the full abstract ▾
All authors Hanyu Meng, Zhanhong He, Zixun Guo, Yaolong Ju
Sound · Cross-listed from eess.AS
Training-Free Instruction TTS Gender Bias Calibration Using Model-Adaptive Steering
Kuan-Yu Chen, Yi-Cheng Lin, Jeng-Lin Li et al. (+1 more)
Instruction text-to-
speech (ITTS) systems systematically encode acoustic gender skews from descriptive style
prompts, such as occupations or personas, even when demographic attributes are left unspecified. Calibrating the implicit gender distribution of the synthesized voices, across heterogeneous architectures and without model retraining or overriding explicit user
prompts, is an open problem. In this work, we propose model-adaptive steering, a training-free bias calibration method that steers post-encoder conditioning representations using a group deviation vector paired with a coarse-to-fine strength search on a development set. A deterministic lexical gate bypasses intervention whenever explicit gender keywords are detected, preserving intended
prompt semantics. Evaluated on a 12,800-
prompt held-out
benchmark across four ITTS models (three architectures), the proposed method reduces aggregate calibration error from 11.5–28.1 to 0.8–5.9 percentage points (e.g., shifting Parler-TTS Mini from 78.1% and PromptTTS++ from 27.3% female to 50.8–52.4%). Output rates stay within 0.9 pp across anchor set sizes at fixed operating points, while UTMOS decreases by at most 0.08 and WER degrades by at most 3.3 pp.
Read the full abstract ▾
All authors Kuan-Yu Chen, Yi-Cheng Lin, Jeng-Lin Li, Jian-Jiun Ding
Sound
Backdooring Acoustic Foundation Models for Physically Realizable Triggers
Zebin Yun, Eyal Ronen and Mahmood Sharif
Acoustic foundation models (AFMs) have democratized acoustic applications, enabling powerful models for tasks ranging from
speech recognition to speaker verification with minimal resources. However, the security of applications based on AFMs remains largely underexplored. Our work addresses this gap by proposing the Foundation Acoustic model Backdoor (FAB) attack, demonstrating that state-of-the-art AFMs are susceptible to backdooring under practical settings. Despite making minimal assumptions about adversary capabilities (e.g., no access to pre-training data), we show that FAB preserves benign performance while inducing backdoors that survive fine-tuning and cause significant degradation across diverse downstream tasks when activated. Notably, FAB utilizes task-agnostic, physically realizable, inconspicuous, and sync-free triggers (e.g., a background siren). We evaluate FAB using two leading AFMs, nine downstream tasks, and four different triggers. We further demonstrate its effectiveness against established defenses and across both digital and physical domains. While extensive end-to-end fine-tuning can mitigate FAB, such a defense is resource-intensive and task-specific. Our work highlights critical risks to AFMs and calls for advanced defenses.
Read the full abstract ▾
Sound
Tracing Inputs, Verifying Outputs: Validating Attribution in Music Generation
Taejun Kim, Wonil Kim, Jongmin Jung et al. (+10 more)
How can we verify whose music contributed to an AI-generated output? This paper demonstrates how input-based attribution can provide verifiable evidence of which audio sources were used in a generation and whether they shaped the output. To do so, we condition the generation solely on audio without any text input, then trace the inputs behind each output, and establish their musical effect. In
prompt adherence tests and controlled input swaps, the stems generated by our generator, MixAudio, follow the
prompt audio in timbre and the context audio in harmony. Yet these outputs may still reproduce training data not supplied as inputs. We therefore audit memorization with our musical version identification model, musicDNA, and find few reproductions outside the input records. On human-judged cases within the flagged pool, it achieves higher precision and recall than the other tested memorization detectors. The two evaluations suggest that input records and output analysis provide complementary evidence for attribution, on which rights-holder reporting and compensation can draw as the AI music economy takes shape. Audio examples are available at https://neutune.github.io/attr2027demo/
Read the full abstract ▾
All authors Taejun Kim, Wonil Kim, Jongmin Jung, Hyeongseok Wi, Sangeun Kum, Keunhyoung Luke Kim, Taehyoung Kim, Dongjoo Moon, Seungsoon Park, Taewan Kim, Virginie Berger, Juhan Nam, Jongpil Lee