Sound
ScorePrompts: Natural-Language Exploration of Symbolic Music Scores through Analysis
Emmanouil Karystinaios and Gerhard Widmer
We present ScorePrompts, an interactive system in which users upload a score, receive natural-
language descriptions of its musical structure, ask questions about specific passages, and inspect the corresponding analysis results in staff notation. Specialist MIR components first estimate harmony, tonality, cadences, formal boundaries, texture, and note-level roles, organizing their outputs at note, beat, measure, and piece levels. A schema-constrained
language model converts these results into descriptions rather than inferring musical structure directly from raw MusicXML. For questions such as "What changes in measures 14-18?", a deterministic router selects the relevant measures and analytical levels and returns a concise response together with the underlying results and caveats. Verovio renders the score and links the returned information to cited measures and note-level attributes. The interface also exposes intermediate tables and disagreements between analytical levels. ScorePrompts is intended for exploratory score analysis and explanation, not score editing. The demo shows how existing analysis models, constrained
language generation, Q&A
retrieval, and notation-based visualization can provide natural-
language access to symbolic music analysis while keeping intermediate results inspectable.
Read the full abstract ▾
Sound
Learned Continuous Synthesis of Quadratic Difference Tone Spectra
Esteban Gutiérrez, Behzad Haki, Christopher Haworth et al. (+2 more)
Quadratic difference tones (QDTs) are a species of auditory distortion product in which a "phantom" pure tone, absent from the acoustic signal, is clearly audible to listeners. Exploiting this phenomenon, one can synthesize harmonically rich tones for musical purposes, a technique called Quadratic Difference Tone Spectrum (QDTS) synthesis. Previous works have introduced numerical methods to synthesize QDTS based on the distortion function, which links a target QDTS and an overtone-structured carrier signal. While accurate, these methods were
stochastic and discontinuous, making them difficult to control for musical purposes and effectively limiting them to stationary signals. This paper proposes a neural network-based approach that learns an approximate inverse of the distortion mapping in an autoencoder-like configuration, producing a continuous approximation that addresses prior limitations. Experimental results show that, although slightly less numerically precise, the method is sufficient for perceptual and musical applications. We also implement a real-time version in Max and evaluate its performance. Various sound examples demonstrate its expressive and musical potential. The source code, audio examples, tutorials, and software accompanying this work are available at https://cordutie.github.io/projects/qdts.html
Read the full abstract ▾
All authors Esteban Gutiérrez, Behzad Haki, Christopher Haworth, Xavier Serra, Rodrigo Cádiz
Sound · Cross-listed from cs.CL
LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection
Xiao Wei, Yuqin Lin, Yaru Cao et al. (+6 more)
Speech-based automatic detection of Alzheimer's disease (AD) provides a non-invasive and scalable approach to early cognitive screening. AD affects both lexical-semantic organization and
speech production, including atypical pauses and word elongations. However, existing methods have yet to fully integrate these paralinguistic cues with linguistic content. We propose LLM-Anchored Paralinguistic Enrichment (LAPE), which enriches LLM-derived linguistic representations with paralinguistic cues through three coordinated innovations. The first is prosodic event textualization, which enables the LLM to model pauses and elongations jointly with lexical content by encoding them as explicit markers with bounded duration-aware repetition. The second is lexico-prosodic unitization and chunking, which preserves event identity and magnitude in both modalities by pooling only consecutive word units. The third is text-anchored paralinguistic fusion, which integrates local and utterance-level
speech features by using NormGate to normalize and dynamically scale them relative to text. We evaluate LAPE on ADReSS and ADReSSo using participant-level cross-validation and leave-one-subject-out evaluation. LAPE achieves state-of-the-art performance across all four primary settings. Code will be released upon acceptance.
Read the full abstract ▾
All authors Xiao Wei, Yuqin Lin, Yaru Cao, Jinyu Li, Bin Wen, Kai Li, Yueying Chen, Longbiao Wang, Jianwu Dang
Sound
Project Qualia: Recovering Experiential Music Structure from Session Co-occurrence Data
Nizam Mohammed, Abu B. S. Rahman and Dimuthu D. K. Arachchige
This report presents results from Project Qualia, an ongoing effort to determine whether experiential similarity between songs, a structure not captured by genre or metadata taxonomies, can be recovered from real listening behavior. We constructed a large-scale
dataset of listening sessions, comprising 1.29 billion scrobbles collected from 9,396 users via the Last.fm API and reduced through a preprocessing pipeline to 531.6 million training scrobbles across 28.6 million sessions. On this corpus, we trained a skip-gram Word2Vec model (Song2Vec), treating each session as a sentence and each track as a
token. As anticipated, the resulting
embedding space was dominated by artist identity, a consequence of single-artist runs within sessions. To test for a subtler, artist-independent signal, we developed an artist-residual procedure: subtracting each artist's centroid from its tracks'
embeddings and evaluating whether the remainder retained structure. Mean cross-artist cosine similarity fell from 0.2487 in raw
embedding space to 0.0005 in residual space, yet 4,577 cross-artist track pairs retained cosine similarity $\ge 0.70$ in residual space, forming coherent genre- and era-based clusters, including trip-hop, 1990s grunge, 2020 mainstream pop, and cross-composer classical piano pairs at cosine similarity up to 0.95. These results confirm that the training data contains experiential structure independent of artist identity, establishing an empirical basis for an architecture designed to learn this experiential layer directly.
Read the full abstract ▾
Sound · Cross-listed from eess.SP
Bridging Echolocation Gaps in Automated Beaked Whale Tracking
Clair Ma, Thomas Kropfreiter, Lauren Baggett et al. (+2 more)
Passive acoustic monitoring (PAM) is an effective and widely used tool for tracking marine mammals, particularly beaked whales, which are infrequently observed visually because of their deep-diving behavior. However, the large data sets generated by PAM methods often require time-consuming hand labeling to identify whale trajectories in the recorded audio. Automated multi-target tracking (MTT) methods could significantly reduce human workload, but current methods have difficulty forming continuous tracks because of the irregularity of beaked whale echolocation clicks. More precisely, regular sequences of clicks are often interrupted by longer pauses that occur when whales face away from the sensors or stop clicking. Consequently, the probability of detection is difficult to model accurately, and MTT trajectories become fragmented at these pauses. In this paper, we propose a multistage target-estimation method aimed at bridging large gaps in click sequences by combining belief propagation-based MTT with track smoothing and stitching. We validate our method using acoustic recordings of clicks from goose-beaked whales (Ziphius cavirostris), and demonstrate that it improves track estimates and reduces fragmentation in the presence of consecutive missed detections. When evaluated with the generalized optimal subpattern assignment (GOSPA) metric, our method outperforms existing MTT reference methods through reductions in missed-target errors.
Read the full abstract ▾
All authors Clair Ma, Thomas Kropfreiter, Lauren Baggett, Simone Baumann-Pickering, Florian Meyer
Sound
Sparse Weight and Edge Circuit Discovery in Transformer-based Acoustic Models
Jiankun Wei, Ewan Dunbar and Gerald Penn
Transformer-based foundation models are powerful but opaque, motivating Mechanistic Interpretation methods to uncover the black-box by identifying small computation subgraphs responsible for a task. DiscoGP is a joint weight-and-edge
circuit discovery framework originally developed for text decoders. We extend DiscoGP to
speech encoders and present, to our knowledge, the first
circuit discovery study for modern
speech foundation models. Across HuBERT and Wav2Vec 2.0 on several
speech classification tasks, we find that the discovered
circuits are extremely compact, yet often match or even exceed the performance of the full pretrained encoder with the same downstream head. Through ablations, we show that these
circuits reflect pretrained computation rather than random structure or task-head artifacts. We also introduce a
memory-efficient DiscoGP variant that reduces the GPU
memory cost of edge-
circuit discovery at runtime from quartic to cubic. Overall, our results broaden Mechanistic Interpretation beyond text decoders and show that
circuit-level analysis can reveal both explanatory structure and unexpected functional behavior in
speech encoders.
Read the full abstract ▾
Sound · Cross-listed from cs.CL
Do speech foundation models really learn words?
Robin Huo and Ewan Dunbar
Self-supervised
speech foundation models are now used in a wide array of downstream applications, including traditional
speech recognition and as the basis for
tokens in
speech-aware
language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.
Read the full abstract ▾
Sound
Unifying Score and Performance for Fine-Grained Music Understanding in Audio-Language Models
Milan Liessens Dujardin, Song-Ze Yu and Kevin Miao
Large audio
language models (LALMs) have shown promising progress in broad music-understanding tasks such as tagging,
retrieval, and captioning. Music understanding that requires finer hearing over both the content and how it is realized within a performance through dynamics, phrasing, articulation, time, and other performance techniques, however, remains at an earlier stage. Existing audio-
language model (ALM) training pipelines typically rely on coarse, weakly grounded captions and therefore provide little support for learning these subtle nuances in music, limiting their ability to serve real-world applications in education or artistic practice. We therefore introduce MuNo-SP (Music Notation unifying Score and Performance), a text-based representation that jointly encodes score content and performance information. Building on MuNo-SP, we develop an automatic training-data generation pipeline that uses aligned scores and performances to produce long-form auditory analyses and musically informed question-answer pairs. We use this pipeline to construct MAESTROCaps, a classical piano
dataset comprising 148 long-form performance analyses and 31,080 question-answer pairs derived from 148 aligned score-performance pairs. In a human evaluation, MuNo-SP analyses were preferred by majority vote over MIDI-only analyses for eight of nine excerpts. MuNo-SP also performed strongly on a
benchmark of score-performance understanding, suggesting that integrating score and performance information enables more reliable and musically informative LALM supervision than a MIDI-only baseline.
Read the full abstract ▾
Sound
TimeCues Studio: A Workspace for Music Annotation and Algorithm Prototyping
Sapir Caduri and Yoav Goldberg
Multimedia applications require precise music annotation-labeled positions, segments, or loops-placed by hand or algorithmically. Machine-learning
algorithms are scalable and effective but need annotated training data, scarce for many tasks. TimeCues Studio is an open-source workspace where
algorithm-development teams annotate a music corpus, compare detection
algorithms against those annotations, and prototype new ones. Unlike existing tools built for a single track at a time, TimeCues targets teams annotating whole collections, tightly integrated with
algorithm development. Annotators place several marker types-each supporting ambiguity-aware labeling-on a grid-locked timeline that visualizes many music features, including separated audio stems. The same timeline drives an
algorithm-comparison engine with bundled baselines, a Python sandbox for prototyping new models, and an ambiguity-aware evaluator that honors the structured fields. The same visualization suits solo annotators on music-sync projects. TimeCues is MIT-licensed and deploys via one Docker Compose command.
Read the full abstract ▾
Sound · Cross-listed from eess.AS
Pushing the Boundaries of Streaming Multi-Speaker ASR: A Systematic Study of Architectural Trade-offs
Taejin Park, Ivan Medennikov, Kunal Dhawan et al. (+3 more)
Streaming multi-speaker ASR is a challenging task that must balance accuracy, latency, and efficiency while handling overlapping
speech and maintaining coherent long-context modeling over extended conversations in an online fashion. We present a unified framework that categorizes streaming multi-speaker ASR into four architectural strategies based on how diarization and ASR are integrated. Using a shared pair of open-source streaming ASR and diarization models as a common foundation, we derive four multi-speaker ASR systems that differ in whether they employ multiple model instances, fine-tuning, or both. We evaluate these systems across multi-speaker accuracy, single-speaker accuracy degradation,
memory footprint, and training complexity. Through this systematic architectural analysis, we clarify the design space for streaming multi-speaker ASR and provide practical guidance for selecting the most suitable approach under diverse deployment constraints.
Read the full abstract ▾
All authors Taejin Park, Ivan Medennikov, Kunal Dhawan, Weiqing Wang, Jagadeesh Balam, Boris Ginsburg
Sound
NOPE-HYPE: A Structured Simulation Workflow for Robust Speech-to-Text Across Diverse Acoustic Environments
Niramay M. Patel, Bibek Behera and Raksha Sharma
Robust
speech-to-text translation systems should perform reliably across diverse acoustic conditions, yet practical pipelines lack controllable tools for systematic environment exploration. Large
speech models remain sensitive to unseen acoustic conditions, as training data rarely cover the full range of real environments.We present NOPEHYPE, a structured training workflow that combines a controllable environment simulator, coverage-optimal environment reduction on Power Spectral Density (PSD) templates, and a small, interpretable hyperparameter search over simulator knobs. We show that simulator-generated noise achieves performance comparable to balanced realnoise training across Whisper and SeamlessM4T models, provide principled environment prototype sets, and identify practical default simulator configurations from a structured 27-run hyperparameter sweep.
Read the full abstract ▾
Sound
Orukeet: Multilingual ASR with Frozen Gabor Kernels
Nathan Roll, Irene Yi, Büşra Marşan et al. (+6 more)
Orukeet replaces half of an adapted Parakeet encoder's temporal filters with 12,288 fitted Gabor kernels, freezes these replacements, and trains the remaining parameters on multilingual and multi-accent data. Final adaptation and checkpoint selection use LibriSpeech test-other. Across 20,146 FLEURS recordings in 25
languages, pooled word error rate (WER) falls from Parakeet's 11.01% to Orukeet's 9.85%, a 10.6% relative reduction. Orukeet has lower WER on 23 of the 25
languages. Orukeet outperforms Parakeet on 61 out of 74 tested splits, including LibriSpeech test-clean (1.46% vs. 1.53% WER), test-other (2.86% vs. 3.14%), and FLEURS English (3.82% vs. 4.28%). All comparisons decode the same audio with matched NeMo settings. The fitted kernels are stored as ordinary convolution weights, retaining Parakeet's architecture and inference operators.
Read the full abstract ▾
All authors Nathan Roll, Irene Yi, Büşra Marşan, Vianney Grenez, Gabriel Stein, Momcilo Mrkaic, Pavle Padjin, Vladimir Zeljkovic, Calbert Graham
Sound
Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations
Soumyadeep Roy
Voice-cloning fraud increasingly relies on surgical injection: a genuine conversation in which only one or two sentences are replaced by synthetic
speech. Utterance-level deepfake detectors emit a single real/fake label per clip and cannot report where the synthetic
speech lies. We formalise this as Temporal Deepfake Localisation in Multi-Speaker Conversations (TDLMC), show that equal error rate and min-DCF are ill-posed once a file contains both classes, and propose temporal metrics for this regime. Our contribution is a training-free five-stage pipeline that wraps a frozen binary detector and adds segment-level output with no retraining, using a two-threshold hysteresis finitestate-machine decoder to turn noisy window scores into coherent intervals. On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone, and its false-alarm rate on genuine
speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI). Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision. Evaluated across three frozen detectors under one decoder whose constants are selected on a held-out calibration split, and with a controlled analysis attributing the residual false-alarm rate to a backbone domain gap rather than to the decoder, this provides the first zero-shot baseline and a reusable
benchmark for TDLMC.
Read the full abstract ▾
Sound
Deterministic Prompting for Speaker-Stable Low-Resource Greek TTS
Georgios Syllas, Efthymios Georgiou, Kosmas Kritsis et al. (+1 more)
Modern TTS systems approach human quality for high-resource
languages but degrade when clean
speech data is scarce. Modern Greek exemplifies this, lacking the curated corpora behind state-of-the-art synthesis. We propose a data curation recipe that transforms audiobook recordings into TTS-ready data via WhisperX
alignment and filtering. Then we fine-tune Parler-TTS (880M), a
prompt-based multilingual model whose pre-training encodes phonetic priors transferable to Greek. During development, we find that LLM-generated style
prompts introduce speaker drift at inference. Replacing them with deterministic
prompts resolves this, and a speaker-specific LoRA stage trained on 3.5 h of single-speaker data anchors identity while updating ~5% of parameters. Our system achieves WER 10.7% (2.9 above the ASR floor), MOS-I 4.00 (vs. 4.36 human
speech), and near-human speaker consistency (MOS-C 4.24 vs. 4.30), showing that robust single-speaker Greek TTS is achievable with limited curated data.
Read the full abstract ▾
All authors Georgios Syllas, Efthymios Georgiou, Kosmas Kritsis, Alexandros Potamianos
Sound · Cross-listed from eess.AS
Over-Tightening-Aware Pseudo-Labeling for Tight-Boundary Speaker Diarization
Shota Horiguchi, Takanori Ashihara, Marc Delcroix et al. (+2 more)
Training speaker diarization models on loose labels, such as
speech segments with padded boundaries or filled pauses, often results in similarly loose model outputs. To obtain tighter boundaries, pseudo-labeling based on the averaged outputs of causal and anticausal models has been proposed. However, since the pseudo-labels are estimation-based, they can suffer from over-tightening, which increases missed detections that can propagate as unrecoverable errors to downstream tasks. This paper carefully analyzes the causes of over-tightening and proposes three approaches to address them: (i) removing pause filling rather than padding, (ii) introducing a burn-in phase to mitigate missed detections near the beginning of causal and anticausal predictions, and (iii) making pseudo-label-based co-training aware of the non-causal model used for final inference. Experimental results show that the proposed method reduces missed detections caused by over-tightening and improves both diarization accuracy and downstream multi-talker ASR performance.
Read the full abstract ▾
All authors Shota Horiguchi, Takanori Ashihara, Marc Delcroix, Naohiro Tawara, Alexis Plaquet
Sound
Robust Rank Aggregation for Multimodal Speech-Based Alzheimer's Disease Detection
Zemin Jin and Tomoko Matsui
Speech-based Alzheimer's disease (AD) detection has recently benefited from
multimodal foundation-model representations that integrate complementary acoustic and linguistic information. However, conventional probability averaging over these complementary classifiers is unreliable, because their posterior probabilities exhibit mismatched scales: identical values may reflect different confidence levels across models. We propose a robust rank aggregation framework that aggregates normalized prediction ranks instead of posterior probabilities. Each subject is scored by its percentile within a fixed training-cohort distribution of out-of-fold predictions; since rank ordering is invariant to monotonic transformations, this avoids probability-scale mismatch while preserving classifier confidence ordering. A confidence-gated Random Forest further corrects residual errors using clinically interpretable linguistic features, overriding the rank prediction only when the two disagree and the RF is highly confident, without additional deep model training or explicit posterior-probability calibration. On ADReSS2020 and ADReSSo2021, the method achieves accuracies of 95.83% and 90.14%, respectively, comparing favorably with previously reported results.
Read the full abstract ▾
Sound · Cross-listed from eess.AS
SpeechAnnotator: A Context-Aware Multi-Agent Framework and Benchmark for Multidimensional Speech Annotation
Qirui Zhan, Shuiyuan Wang, Jingbin Hu et al. (+10 more)
Recent controllable
speech generation requires training data with fine-grained annotations of speaker traits, prosody, emotion, paralinguistic cues, acoustic scenes, and context. Existing workflows often rely on manual correction, paid hosted
multimodal services, or fixed processing chains, which limits large-scale data processing through annotation cost, external-service dependence, or weak cross-stage recovery. We introduce SpeechAnnotator, a locally deployable, context-aware multi-
agent framework built entirely from open-source models and tools. Supporting frontend modules first obtain speaker-aware segments and final segment transcripts, while prior evidence extractors attach heterogeneous segment-level cues. Three specialist
agents then collaborate through shared state: the Planning
Agent converts local audio evidence, speaker history, neighboring segments, and recording-level context into field-specific contracts; the Labeling
Agent performs contract-guided
multimodal prediction for directly observable attributes; and the Review
Agent runs a bounded review loop that checks evidence support and cross-segment consistency, triggering relabeling only for unsupported or inconsistent fields. To address the fragmentation of existing evaluation resources across isolated tasks and narrow-domain test sets, we introduce SpeechAnnotator-Bench (SA-Bench), containing 8.87 hours of human-annotated audio across nine source formats, together with SpeechAnnotator-Eval (SA-Eval), which separates Timeline-Eval for speaker-aware timeline recovery, Closed-Eval for finite-set attributes, and Open-Eval for open-ended attributes. Experiments and ablations show that SpeechAnnotator provides a locally deployable alternative to commercial audio-capable systems, while the bounded review loop improves multidimensional annotation through evidence- and context-aware field-level recovery.
Read the full abstract ▾
All authors Qirui Zhan, Shuiyuan Wang, Jingbin Hu, Haoyu Zhang, Xiaming Ren, Jinrui Liang, Chaoren Yu, Bengu Wu, Yunxiang Chen, Houdun Liu, Su Feng, Liumeng Xue, Lei Xie
Sound · Cross-listed from eess.AS
NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding
Yuang Cao, Bingshen Mu, Zhennan Lin et al. (+7 more)
Human
speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information. Existing approaches typically represent NVVs as transcript-level tags, providing limited supervision for their waveform-time boundaries. We present NVV-Locator for fine-grained NVV temporal grounding. We first unify 26 NVV categories across public resources and construct large-scale timestamp-supervised training data through dual-LLM verification, transcript-guided forced
alignment, and energy-based boundary refinement. We further introduce NVV-TimeBench, an expert-refined
benchmark with 667 utterances and 1,094 events. NVV-Locator uses a non-autoregressive slot-filling architecture to jointly predict lexical timestamps, NVV categories, and event boundaries. On NVV-TimeBench, it achieves 71.0% Micro F1, 70.2% Macro F1, 80.4% Macro mIoU, and 59.6 ms Macro mMAE, outperforming the evaluated large audio model counterparts. Evaluation on an external corpus further demonstrates the cross-corpus generalization of NVV-Locator.
Read the full abstract ▾
All authors Yuang Cao, Bingshen Mu, Zhennan Lin, Guojian Li, Haoyue Zhan, Jie Liu, Chuan Xie, Qiang Zhang, Liumeng Xue, Lei Xie
Sound · Cross-listed from eess.AS
Source-Adaptive Data Curation for Bilingual NVV-Aware ASR
Yuang Cao, Qirui Zhan, Jingbin Hu et al. (+7 more)
Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic
speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical
tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and
multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.
Read the full abstract ▾
All authors Yuang Cao, Qirui Zhan, Jingbin Hu, Ziyu Zhang, Yunxiang Chen, Houdun Liu, Su Feng, Bengu Wu, Lei Xie, Liumeng Xue
Sound
Arti-JEPA: Adapting Video World Model to Real-Time MRI of the Vocal Tract for Speech-Production Analysis
Hong Nguyen, Sean Foley, Christina Hagedorn et al. (+4 more)
Real-time MRI (rtMRI) captures the dynamics of the entire vocal tract during
speech, but labeled data are scarce and the modality - single-slice, grayscale, low-resolution - differs substantially from the natural
videos that
video foundation models are trained on. We introduce Arti-JEPA, a joint
embedding predictive architecture to model vocal tract rtMRI by continuing its self-supervised objective on about 62h of unlabelled vocal-tract
videos, and evaluate the frozen representation on three tasks: cross-domain phoneme prediction (on typical speakers), fluent-vs-disfluent classification (a corpus containing stuttered
speech), and characterizing pre/post-operative transfer (after partial glossectomy). Three key findings emerge. (1) A temporal
video prior decisively outperforms per-frame image encoders, and latent prediction (V-JEPA) is at least as strong as pixel reconstruction (VideoMAE), with the edge on fine-grained phonemes. (2) Domain adaptation is task-dependent: it roughly doubles cross-domain phoneme prediction $\kappa$ (to 0.352) but does not help binary stuttering classification. (3) Arti-JEPA was able to recover phoneme signal from pre/post glossectomy
speech — an in-domain probe decodes patients at least as well as a typical speaker, indicating that the residual transfer gap is cross-speaker/domain misalignment, not surgical signal loss, and post-operative decoding does not fall below performance on pre-operative
speech. Together, these position a frozen, domain-adapted rtMRI encoder as a reusable measurement tool for articulatory and clinical
speech science.
Read the full abstract ▾
All authors Hong Nguyen, Sean Foley, Christina Hagedorn, Yijing Lu, Sudarsana Reddy Kadiri, Dani Byrd, Shrikanth Narayanan
Sound · Cross-listed from cs.CL
StreamAlign: Streaming Text-Aligned Speech Tokenization
Kang-wook Kim, Jinyoung Park, Jinsoo Kim et al. (+3 more)
Text-aligned
speech tokenization methods have emerged to better align
speech tokens with LLM
token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automatic
speech recognition (ASR), leading to two key limitations: (i) the need for complete utterances before tokenization, precluding real-time streaming, and (ii) vocabulary mismatch between ASR and LLMs, which reduces acoustic granularity from the subword to the word level. We introduce StreamAlign, a text-aligned
speech tokenization framework that enables streaming tokenization for real-time
speech-text joint modeling. StreamAlign performs online
speech-text
alignment by combining character-level RNN-Transducer
alignment with word-level ASR guidance, mitigating ASR-LLM vocabulary mismatch while preserving recognition accuracy. A proactive word boundary classifier anticipates word completion at chunk boundaries, reducing tokenization latency from 560 ms to 270 ms. On LibriSpeech, StreamAlign achieves the lowest WER and highest UTMOS among evaluated tokenizers. Furthermore, StreamAlign-SLM, a spoken
language model trained on StreamAlign units, outperforms other end-to-end spoken
language models in
speech continuation while achieving the strongest overall consistency on SALMon and spoken StoryCloze.
Read the full abstract ▾
All authors Kang-wook Kim, Jinyoung Park, Jinsoo Kim, Sehun Lee, Sang Hoon Woo, Gunhee Kim
Sound
Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models
Yupei Li, Qiyang Sun, Mohamed Mady et al. (+4 more)
Large Audio
Language Models (LALMs) have shown strong performance on audio
reasoning benchmarks, but accuracy alone cannot distinguish true
reasoning from superficial pattern matching, often overestimating
reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the
reasoning process itself is essential for improving LALMs'
reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio
reasoning introduces unique challenges absent in text-based settings, perceptual
hallucination and cross-modal
alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio
Reasoning Integrity Assessment), a lightweight, annotation-free gold
reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio
reasoning quality across perceptual grounding,
reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the
reasoning process observable. Experiments on 9 models across 2
benchmarks identify three
reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Read the full abstract ▾
All authors Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang, Zhengwei Gong, Berrak Sisman, Björn Schller
Sound · Cross-listed from cs.MA
Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference
Yaohan Guan, Yen-Ju Lu, Yuzhe Wang et al. (+5 more)
Inferring speaker relationships from spoken conversations is an important step towards socially aware
speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and
multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-
agent reasoning framework that organizes inference through structured interaction among LLM
agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-
Agent Debate as a task-specific adaptation of standard multi-
agent debate for speaker relationship inference, assigning
agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-
Agent Compete, a competition-based protocol that compares
agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction
dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-
agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among
agents, while acoustic cues are not yet fully captured by current models.
Read the full abstract ▾
All authors Yaohan Guan, Yen-Ju Lu, Yuzhe Wang, Junhyeok Lee, Jesus Villalba, Laureano Moro Velazquez, Thomas Thebaud, Najim Dehak