Audio and Speech Processing
Towards Real-Time Generative Speech Restoration with Flow-Matching
Tsun-An Hsieh and Sebastian Braun
Generative models have shown robust performance on
speech enhancement and restoration tasks, but most prior approaches operate offline with high latency, making them unsuitable for streaming applications. In this work, we investigate the feasibility of a low-latency, real-time generative
speech restoration system based on flow-matching (FM). Our method tackles diverse real-world tasks, including denoising, dereverberation, and generative restoration. The proposed causal architecture without time-downsampling achieves introduces an total latency of only 20 ms, suitable for real-time communication. In addition, we explore a broad set of architectural variations and sampling strategies to ensure effective training and efficient inference. Notably, our flow-matching model maintains high enhancement quality with only 5 number of function evaluations (NFEs) during sampling, achieving similar performance as when using ~20 NFEs under the same conditions. Experimental results indicate that causal FM-based models favor few-step reverse sampling, and smaller backbones degrade with longer reverse trajectories. We further show a side-by-side comparison of FM to typical
adversarial-loss-based training for the same model architecture.
Read the full abstract ▾
Audio and Speech Processing
Adaptive Deterministic Flow Matching for Target Speaker Extraction
Tsun-An Hsieh and Minje Kim
Generative target speaker extraction (TSE) methods often produce more natural outputs than predictive models. Recent work based on
diffusion or flow matching (FM) typically relies on a small, fixed number of reverse steps with a fixed step size. We introduce Adaptive Discriminative Flow Matching TSE (AD-FlowTSE), which extracts the target
speech using an adaptive step size. We formulate TSE within the FM paradigm but, unlike prior FM-based
speech enhancement and TSE approaches that transport between the mixture (or a normal prior) and the clean-
speech distribution, we define the flow between the background and the source, governed by the mixing ratio (MR) of the source and background that creates the mixture. This design enables MR-aware initialization, where the model starts at an adaptive point along the background-source trajectory rather than applying the same reverse schedule across all noise levels. Experiments show that AD-FlowTSE achieves strong TSE with as few as a single step, and that incorporating auxiliary MR estimation further improves target
speech accuracy. Together, these results highlight that aligning the transport path with the mixture composition and adapting the step size to noise conditions yields efficient and accurate TSE.
Read the full abstract ▾
Signal Processing
Stochastic Geometry Analysis of Asymmetric Uplink Interference for Urban UAV-RC Networks
Donggu Lee, Sung Joon Maeng and Ismail Guvenc
Uncrewed aerial vehicles (
UAVs) have emerged as a flexible platform for providing coverage over challenging environments, particularly for public safety and surveillance missions in urban areas. However, deploying the
UAVs in dense urban areas introduces unique challenges, most notably asymmetric uplink (UL, remote controller to
UAV) interference due to a higher chance of line-of-sight (LoS) interference at the
UAV. In this letter, we propose a
stochastic geometry framework to tractably analyze the large-scale asymmetric interference in urban areas. We incorporate a log-Gaussian Cox process (LGCP) model to capture the spatial correlation of the interference field in both UL and downlink (DL) as a function of the
UAV altitude and the two-dimensional (2-D) distance between the remote controller and
UAV. To quantify the UL and the DL interference asymmetry, we also define the interference asymmetry ratio characterizing the interference disparity between the UL and the DL. Our numerical results demonstrate that the interference asymmetry ratio increases as the
UAV altitude and 2-D distance increase, highlighting that the UL interference worsens.
Read the full abstract ▾
Systems and Control
Safe Payload Transfer with Ship-Mounted Cranes: A Robust Model Predictive Control Approach
Ersin Das, William A. Welch, Patrick Spieler et al. (+14 more)
Ensuring safe real-time control of ship-mounted cranes in unstructured transportation environments requires handling multiple safety constraints while maintaining effective payload transfer performance. Unlike traditional crane systems, ship-mounted cranes are consistently subjected to significant external disturbances affecting underactuated crane dynamics due to the ship's dynamic motion response to harsh sea conditions, which can lead to robustness issues. To tackle these challenges, we propose a robust and safe model predictive control (MPC) framework and demonstrate it on a 5-DOF crane system, where a Stewart platform simulates the external disturbances that ocean surface motions would have on the supporting ship. The crane payload transfer operation must avoid obstacles and accurately place the payload within a designated target area. We use a robust zero-order control barrier function (R-ZOCBF)-based safety constraint in the nonlinear MPC to ensure safe payload positioning, while time-varying bounding boxes are utilized for collision avoidance. We introduce a new optimization-based online robustness parameter adaptation scheme to reduce the conservativeness of R-ZOCBFs. Experimental trials on a crane prototype demonstrate the overall performance of our safe control approach under significant perturbing motions of the crane base. While our focus is on crane-facilitated transfer, the methods more generally apply to safe robotically-assisted parts mating and parts insertion.
Read the full abstract ▾
All authors Ersin Das, William A. Welch, Patrick Spieler, Keenan Albee, Aurelio Noca, Jeffrey Edlund, Jonathan Becktor, Thomas Touma, Jessica Todd, Sriramya Bhamidipati, Stella Kombo, Maira Saboia, Anna Sabel, Grace Lim, Rohan Thakker, Amir Rahmani, Joel W. Burdick
Signal Processing · Cross-listed from cs.IT
Unlocking Off-the-Grid Sparse Recovery with Unlimited Sensing: Simultaneous Super-Resolution in Time and Amplitude
Ruiming Guo and Ayush Bhandari
The recovery of Dirac impulses, or spikes, from filtered measurements is a classical problem in signal processing. As the spikes lie in the continuous domain while measurements are discrete, this task is known as super-resolution or off-the-grid sparse recovery. Despite significant theoretical and algorithmic advances over the past decade, these developments often overlook critical challenges at the analog-digital interface. In particular, when spikes exhibit strong-weak amplitude disparity, conventional digital acquisition may result in clipping of strong components or loss of weak ones beneath the quantization noise floor. This motivates a broader perspective: super-resolution must simultaneously resolve both amplitude and temporal structure. Under a fixed bit budget, such information loss is unavoidable. In contrast, the emerging theory and practice of the Unlimited Sensing Framework (USF) demonstrate that these fundamental limitations can be overcome. Building on this foundation, we demonstrate that modulo encoding within USF enables digital super-resolution by enhancing measurement precision, thereby unlocking temporal super-resolution beyond conventional limits. We develop new theoretical results that extend to non-bandlimited kernels commonly encountered in practice and introduce a robust
algorithm for off-the-grid sparse recovery. To demonstrate practical impact, we instantiate our framework in the context of time-of-flight imaging. Both numerical simulations and hardware experiments validate the effectiveness of our approach under low-bit quantization, enabling super-resolution in amplitude and time.
Read the full abstract ▾
Signal Processing · Cross-listed from cs.LG
A Primer on Kolmogorov-Arnold Networks (KANs) for Probabilistic Time Series Forecasting
Cristian J. Vaca-Rubio, Roberto Pereira, Luis Blanco et al. (+2 more)
This work introduces Probabilistic Kolmogorov-Arnold Network (P-KAN), a novel probabilistic extension of Kolmogorov-Arnold Networks (KANs) for time series
forecasting. By replacing scalar weights with spline-based functional connections and directly parameterizing predictive distributions, P-KANs offer expressive yet parameter-efficient models capable of capturing nonlinear and heavy-tailed dynamics. We evaluate P-KANs on satellite traffic
forecasting, where uncertainty-aware predictions enable dynamic thresholding for resource allocation. Results show that P-KANs consistently outperform Multi Layer Perceptron (MLP) baselines in both accuracy and calibration, achieving superior efficiency-risk trade-offs while using significantly fewer parameters. We build up P-KANs on two distributions, namely Gaussian and Student-t distributions. The Gaussian variant provides robust, conservative forecasts suitable for safety-critical scenarios, whereas the Student-t variant yields sharper distributions that improve efficiency under stable demand. These findings establish P-KANs as a powerful framework for probabilistic
forecasting with direct applicability to satellite communications and other resource-constrained domains.
Read the full abstract ▾
All authors Cristian J. Vaca-Rubio, Roberto Pereira, Luis Blanco, Engin Zeydan, Màrius Caus
Systems and Control · Cross-listed from physics.optics
Ultra High Sensitivity Soil Moisture Detection Using Photonic Crystal Cavity with SIW Technology
Justin Jose and Nikhil Kumar
Soil nutrients and water content are two crucial factors that significantly affect agricultural production yields. Hence, monitoring and measuring the water content and soil type are critical requirements. This study proposes a two-dimensional structure of photonic
crystals centered around a symmetrical cross-shaped slot. The cross-slots act as resonators, and the photonic
crystals surrounding the slots tune the resonance frequency of the resonators to enhance mode confinement within the resonator. The various resonant modes are located in the 2.1 GHz, 5.2 GHz, and 8.1 GHz bands, which correspond to the S band, C band, and X band, respectively. These bands are used to compare the absorption, whereas the upper resonant mode is of the order of 20 GHz. Band structure analysis was performed using the Plane Wave Method (PWM). The resonant frequency is computed using a 3D electromagnetic (EM) simulation software that utilizes the Finite Element Method (FEM) and lies in the radiation mode region of the band structure of the photonic
crystal. Varying the incident angle had a negligible effect on the absorption characteristics of the sensor, allowing it to produce accurate sensing results regardless of the incident angle. The sensor's sensitivity is maximized using this design, which results in a sensitivity of 85.4 % in the 2.1 GHz resonant frequency, which is much higher than that of a single column of photonic
crystal-based SIW, resulting in 50.6 % of sensitivity at 2.1 GHz, at which there is a frequency shift of the order of GHz. In contrast, in the proposed design, the frequency shift is on the order of MHz, resulting in ultra-high sensitivity.
Read the full abstract ▾
Audio and Speech Processing · Cross-listed from cs.SD
SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models
Chih-Kai Yang, Yen-Ting Piao, Tzu-Wen Hsu et al. (+8 more)
Knowledge editing enables targeted updates without retraining, but prior work focuses on textual or visual facts, leaving abstract auditory perceptual knowledge underexplored. We introduce SAKE, the first
benchmark for editing perceptual auditory attribute knowledge in large audio-
language models (LALMs), which requires modifying acoustic generalization rather than isolated facts. We evaluate eight diverse editing methods on three LALMs across reliability, generality, locality, and portability, under single and sequential edits. Results show that most methods enforce edits reliably but struggle with auditory generalization, intra-attribute locality, and
multimodal knowledge propagation, and often exhibit forgetting or degeneration in sequential editing. Additionally, fine-tuning the modality connector emerges as a more robust and balanced baseline compared with directly editing the LLM backbones. SAKE reveals key limitations of current methods and provides a foundation for developing auditory-specific LALM editing techniques.
Read the full abstract ▾
All authors Chih-Kai Yang, Yen-Ting Piao, Tzu-Wen Hsu, Szu-Wei Fu, Zhehuai Chen, Ke-Han Lu, Sung-Feng Huang, Chao-Han Huck Yang, Yu-Chiang Frank Wang, Yun-Nung Chen, Hung-yi Lee
Audio and Speech Processing · Cross-listed from cs.SD
Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations
Bo-Han Feng, Chien-Feng Liu, Yu-Hsuan Li Liang et al. (+9 more)
Large audio-
language models (LALMs) extend text-based LLMs with auditory understanding, offering new opportunities for
multimodal applications. While their perception,
reasoning, and task performance have been widely studied, their safety
alignment under paralinguistic variation remains underexplored. This work systematically investigates the role of speaker
emotion. We construct a
dataset of malicious
speech instructions expressed across multiple
emotions and intensities, and evaluate several state-of-the-art LALMs. Our results reveal substantial safety inconsistencies: different
emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk. These findings highlight an overlooked vulnerability in LALMs and call for
alignment strategies explicitly designed to ensure robustness under emotional variation, a prerequisite for trustworthy deployment in real-world settings.
Read the full abstract ▾
All authors Bo-Han Feng, Chien-Feng Liu, Yu-Hsuan Li Liang, Chih-Kai Yang, Szu-Wei Fu, Zhehuai Chen, Ke-Han Lu, Sung-Feng Huang, Chao-Han Huck Yang, Yu-Chiang Frank Wang, Yun-Nung Chen, Hung-yi Lee
Audio and Speech Processing
SAC: Neural Speech Codec with Semantic-Acoustic Dual-Stream Quantization
Wenxi Chen, Xinsheng Wang, Ruiqi Yan et al. (+9 more)
Speech codecs that convert continuous
speech signals into discrete
tokens have become essential for
speech language models. However, existing codecs struggle to balance high-quality reconstruction with semantically rich representations, limiting their effectiveness in both generative and understanding tasks. In this work, we propose SAC, a neural
speech codec with semantic-acoustic dual-stream quantization. By disentangling semantic and acoustic modeling into two dedicated streams, SAC enables each to be optimized for its respective role. Comprehensive evaluations show that SAC achieves strong reconstruction performance across diverse bitrates under both clean and noisy conditions, with particularly high scores on UTMOS and WER, indicating superior naturalness and intelligibility. Moreover, SAC substantially surpasses prior codecs in semantic representation, approaching the level of continuous self-supervised
embeddings. When used as a tokenizer for LLM-based text-to-
speech, SAC enables a single-stage autoregressive (AR) TTS model that clearly outperforms state-of-the-art AR systems. Our disentanglement analysis further validates the effectiveness of the dual-stream design, offering new potential for controllable
speech generation.
Read the full abstract ▾
All authors Wenxi Chen, Xinsheng Wang, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Hanlin Wen, Shunshun Yin, Ming Tao, Xie Chen
Audio and Speech Processing · Cross-listed from cs.SD
Schrödinger Bridge Mamba for One-Step Speech Enhancement
Jing Yang, Sirui Wang, Chao Wu et al. (+2 more)
We present Schrödinger Bridge Mamba (SBM), a novel model for efficient
speech enhancement by integrating the Schrödinger Bridge (SB) training paradigm and the Mamba architecture. Experiments of joint denoising and dereverberation tasks demonstrate SBM outperforms strong generative and discriminative methods on multiple metrics with only one step of inference while achieving a competitive real-time factor for streaming feasibility. Ablation studies reveal that the SB paradigm consistently yields improved performance across diverse architectures over conventional mapping. Furthermore, Mamba exhibits a stronger performance under the SB paradigm compared to Multi-Head Self-
Attention (MHSA) and Long Short-Term
Memory (LSTM) backbones. These findings highlight the synergy between the Mamba architecture and the SB trajectory-based training, providing a high-quality solution for real-world
speech enhancement. Demo page: https://sbmse.github.io
Read the full abstract ▾
All authors Jing Yang, Sirui Wang, Chao Wu, Lei Guo, Fan Fan
Audio and Speech Processing
Audio dequantization using instantaneous frequency
Vojtěch Kovanda and Pavel Rajmic
We present a dequantization method that employs a phase-aware regularizer, originally successfully applied in an audio inpainting problem. The method promotes a temporal continuity of sinusoidal components in time-frequency representation of the audio signal, and avoids energy loss artifacts commonly encountered with l1-based regularization approaches. The proposed method is called the Phase-Aware Audio Dequantizer (PHADQ). The method is evaluated against the state-of-the-art using the SDR and PEMO-Q ODG objective metrics, and a~subjective MUSHRA-like test.
Read the full abstract ▾
Audio and Speech Processing · Cross-listed from cs.AI
End-to-end Listen, Look, Speak and Act
Siyin Wang, Wenyi Yu, Xianzhao Chen et al. (+4 more)
Human interaction is inherently
multimodal and full-duplex: we listen while watching, speak while acting, and fluidly adapt to turn-taking and interruptions. Realizing these capabilities is essential for building models simulating humans. We present ELLSA (End-to-end Listen, Look, Speak and Act), which, to our knowledge, is the first full-duplex, end-to-end model that simultaneously perceives and generates across
vision, text,
speech, and action within a single architecture, enabling interaction patterns previously out of reach, yielding more natural, human-like behaviors. At its core is a novel SA-MoE architecture (Self-
Attention Mixture-of-Experts) that routes each modality to specialized experts and fuses them through a unified
attention backbone. This provides a generalizable solution for joint
multimodal perception and concurrent generation, leveraging strong pre-trained components while enabling efficient modality integration and mitigating modality interference. On
speech-interaction and
robot-manipulation
benchmarks, ELLSA matches modality-specific baselines, while uniquely supporting advanced
multimodal and full-duplex behaviors such as dialogue and action turn-taking, defective instruction rejection, speaking-while-acting, context-grounded visual question answering, and action barge-ins. We contend that ELLSA represents a step toward more natural and general interactive intelligence, contributing to the broader pursuit of artificial general intelligence. All data, code and model checkpoints will be released at https://github.com/bytedance/SALMONN/tree/ELLSA.
Read the full abstract ▾
All authors Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang
Systems and Control · Cross-listed from cs.RO
Adaptive Invariant Extended Kalman Filter for Legged Robot State Estimation
Kyung-Hwan Kim, DongHyun Ahn, Dong-hyun Lee et al. (+2 more)
State estimation is crucial for legged
robots as it directly affects control performance and locomotion stability. In this paper, we propose an Adaptive Invariant Extended Kalman Filter to improve proprioceptive state estimation for legged
robots. The proposed method adaptively adjusts the noise level of the contact foot model based on online covariance estimation, leading to improved state estimation under varying contact conditions. It effectively handles small slips that traditional slip rejection fails to address, as overly sensitive slip rejection settings risk causing filter divergence. Our approach employs a contact detection
algorithm instead of contact sensors, reducing the reliance on additional hardware. The proposed method is validated through real-world experiments on the quadruped
robot LeoQuad, demonstrating enhanced state estimation performance in dynamic locomotion scenarios.
Read the full abstract ▾
All authors Kyung-Hwan Kim, DongHyun Ahn, Dong-hyun Lee, JuYoung Yoon, Dong Jin Hyun
Systems and Control
A Control-Theoretic Approach to Dynamic Payment Routing for Success Rate Optimization
Aniket Agrawal and Harsharanga Patil
This paper introduces a control-theoretic framework for dynamic payment routing, implemented within JUSPAY's Payment Orchestrator to maximize transaction success rate. The routing system is modeled as a closed-loop feedback controller continuously sensing gateway performance, computing corrective actions, and dynamically routes transactions across gateway to ensure operational resilience. The system leverages concepts from control theory, reinforcement learning, and multi-armed bandit optimization to achieve both short-term responsiveness and long-term stability. Rather than relying on explicit PID regulation, the framework applies generalized feedback-based adaptation, ensuring that corrective actions remain proportional to observed performance deviations and the computed gateway score gradually converges toward the success rate. This hybrid approach unifies control theory and adaptive decision systems, enabling self-regulating transaction routing that dampens instability, and improves reliability. Live production results show an improvement of up to 1.15% in success rate over traditional rule-based routing, demonstrating the effectiveness of feedback-based control in payment systems.
Read the full abstract ▾
Systems and Control · Cross-listed from math.OC
Local integral input-to-state stability for non-autonomous infinite-dimensional systems
Yongchun Bi, Panyu Deng, Jun Zheng et al. (+1 more)
In this paper, we prove comparison principles for nonlinear differential equations with time-varying coefficients and develop Lyapunov analytical tools for the integral input-to-state stability (iISS) analysis of nonlinear non-autonomous infinite-dimensional systems, which involve nonlinearities satisfying a superlinear growth, {bringing} difficulties to the iISS {analysis.} Specifically, our approach starts by establishing several forms of comparison principles for a wide range of ordinary differential equations having time-varying coefficients and superlinear terms, paving the way to conduct iISS assessment for general nonlinear non-autonomous infinite-dimensional systems within the Lyapunov stability framework. Then, by using the comparison principles, we prove a local {iISS} {(LiISS)} Lyapunov
theorem for the nonlinear non-autonomous infinite-dimensional systems in the framework of
Banach spaces. {Furthermore,} we provide sufficient conditions of the existence of a local iISS Lyapunonv functional (LiISS-LF) and construct LiISS-LFs for the systems in the framework of
Hilbert spaces. Finally, we preset two examples to illustrate the proposed {Lyapunov} method for the LiISS analysis: one is to show how to obtain the LiISS of a nonlinear finite-dimensional system with time-varying coefficients and superlinear terms under linear state feedback control law while another one is to show how to employ the interpolation inequalities to handle superliner terms and establish the LiISS-LF for a class of multi-dimensional
parabolic equations with space-time-varying coefficients. To demonstrate the validity of the results, numerical experiments are also conducted to verify the LiISS of these two classes of systems.
Read the full abstract ▾
All authors Yongchun Bi, Panyu Deng, Jun Zheng, Guchuan Zhu
Systems and Control
Linear State Estimation in Presence of Bounded Uncertainties: A Comparative Analysis
Ayan Das, Anushka Sharma and Anamitra Pal
A variety of
algorithms have been proposed to address the power system state estimation problem in the presence of uncertainties in the data. However, less emphasis has been given to handling perturbations in the model. In the context of linear state estimation (LSE), which is the focus of this paper, perturbations in the model come from variations in the line parameters. Since the actual values of the line parameters can be different from the values stored in a power utility's database, we investigate three approaches in this paper to estimate the states in the presence of bounded uncertainties in the data and the model. The first approach is based on interval arithmetic, the second is based on convex optimization, and the third is based on generalized linear fractional programming. The three
algorithms are applied to multiple IEEE test systems and compared in terms of their speed and accuracy. The results indicate that the first two
algorithms are extremely fast and give expected results, while the third suffers from scalability issues and is unsuitable for LSE.
Read the full abstract ▾
Systems and Control · Cross-listed from math.OC
Geometric Control Theory Over Networks: Minimal Node Cardinality Disturbance Decoupling Problems
Luca Claude Gino Lebon and Claudio Altafini
In this paper we show how to formulate and solve disturbance decoupling problems over networks while choosing a minimal number of input and output nodes. Feedback laws that isolate and eliminate the impact of disturbance nodes on specific target nodes to be protected are provided using state, output, and dynamical feedback. For that, we leverage the fact that when reformulated in terms of sets of nodes rather than subspaces, the controlled and conditional invariance properties admit a simple graphical interpretation. For state and dynamical feedback, the minimal input and output cardinality solutions can be computed exactly in
polynomial time, via min-cut/max-flow
algorithms.
Read the full abstract ▾