Image and Video Processing
CLIP Based Region-Aware Feature Fusion for Automated BBPS Scoring in Colonoscopy Images
Yujia Fu, Zhiyu Dong, Tianwen Qian et al. (+3 more)
Accurate assessment of bowel cleanliness is essential for effective colonoscopy procedures. The Boston Bowel Preparation Scale (BBPS) offers a standardized scoring system but suffers from subjectivity and inter-observer variability when performed manually. In this paper, to support robust training and evaluation, we construct a high-quality colonoscopy
dataset comprising 2,240 images from 517 subjects, annotated with expert-agreed BBPS scores. We propose a novel automated BBPS scoring framework that leverages the CLIP model with adapter-based transfer learning and a dedicated fecal-feature extraction branch. Our method fuses global visual features with stool-related textual priors to improve the accuracy of bowel cleanliness evaluation without requiring explicit
segmentation. Extensive experiments on both our
dataset and the public NERTHU
dataset demonstrate the superiority of our approach over existing baselines, highlighting its potential for clinical deployment in computer-aided colonoscopy analysis.
Read the full abstract ▾
All authors Yujia Fu, Zhiyu Dong, Tianwen Qian, Chenye Zheng, Danian Ji, Linhai Zhuo
Audio and Speech Processing · Cross-listed from cs.SD
EnvSSLAM-FFN: Lightweight Layer-Fused System for ESDD 2026 Challenge
Xiaoxuan Guo, Hengyan Huang, Jiayi Zhou et al. (+5 more)
Recent advances in generative audio models have enabled high-fidelity environmental sound synthesis, raising serious concerns for audio security. The ESDD 2026 Challenge therefore addresses environmental sound deepfake detection under unseen generators (Track 1) and black-box low-resource detection (Track 2) conditions. We propose EnvSSLAM-FFN, which integrates a frozen SSLAM self-supervised encoder with a lightweight FFN back-end. To effectively capture spoofing artifacts under severe data imbalance, we fuse intermediate SSLAM representations from layers 4-9 and adopt a class-weighted training objective. Experimental results show that the proposed system consistently outperforms the official baselines on both tracks, achieving Test Equal Error Rates (EERs) of 1.20% and 1.05%, respectively.
Read the full abstract ▾
All authors Xiaoxuan Guo, Hengyan Huang, Jiayi Zhou, Renhe Sun, Jian Liu, Haonan Cheng, Long Ye, Qin Zhang
Signal Processing
A Tutorial to Multirate Extended Kalman Filter Design for Monitoring of Agricultural Anaerobic Digestion Plants
Simon Hellmann, Terrance Wilms, Stefan Streif et al. (+1 more)
In many applications of biotechnology, measurements are available at different sampling rates, e.g., due to online sensors and offline lab analysis. Offline measurements typically involve time delays that may be unknown a priori due to the underlying laboratory procedures. This multirate (MR) setting poses a challenge to Kalman filtering, where conventionally measurement data is assumed to be available on an equidistant time grid and without delays. This tutorial paper derives the MR version of an extended Kalman filter (EKF) based on sample state augmentation, and applies it to the anaerobic digestion (AD) process in a simulative agricultural setting. The performance of the MR-EKF is investigated for various scenarios including varying delay lengths, measurement noise levels, plant-model mismatch (PMM), and initial state error. Provided with an adequate tuning, the MR-EKF can reliably estimate the process state and, thus, appropriately fuse the delayed offline measurements and smooth the noisy online measurements. Because of the sample state augmentation approach, the delay length of offline measurements does not critically effect the performance of the state estimation, provided that observability is not lost during the delays. Poor state initialization and PMM affect convergence more than measurement noise levels. Furthermore, selecting an appropriate tuning was found to be critically important for successful application of the MR-EKF for which a systematic approach is presented. This tutorial provides implementation guidance for practitioners seeking to successfully apply state estimation for multirate systems. Thus, it contributes to the development of demand-driven operation of biogas plants, which may aid in stabilizing a renewable electricity grid.
Read the full abstract ▾
All authors Simon Hellmann, Terrance Wilms, Stefan Streif, Soeren Weinrich
Signal Processing · Cross-listed from cs.IT
RIS-Empowered OTFS Modulation With Faster-than-Nyquist Signaling in High-Mobility Wireless Communications
Chaorong Zhang, Benjamin K. Ng, Hui Xu et al. (+2 more)
High-mobility wireless communication systems suffer from severe Doppler spread and multi-path delay, which degrade the reliability and spectral efficiency of conventional modulation schemes. Orthogonal time frequency space (OTFS) modulation offers strong robustness in such environments by representing symbols in the delay-Doppler (DD) domain, while faster-than-Nyquist (FTN) signaling can further enhance spectral efficiency through intentional symbol packing. Meanwhile, reconfigurable intelligent surfaces (RIS) provide a promising means to improve link quality via passive beamforming. Motivated by these advantages, we propose a novel RIS-empowered OTFS modulation with FTN signaling (RIS-OTFS-FTN) scheme. First, we establish a unified DD-domain input-output relationship that jointly accounts for RIS passive beamforming, FTN-induced inter-symbol interference, and DD-domain channel characteristics. Based on this model, we provide comprehensive analytical performance for the frame error rate, spectral efficiency, and peak-to-average power ratio (PAPR), etc. Furthermore, a practical RIS phase adjustment strategy with quantized phase selection is designed to maximize the effective channel gain. Extensive Monte Carlo simulations under a standardized extended vehicular A (EVA) channel model validate the theoretical results and provide key insights into the trade-offs among spectral efficiency, PAPR, input back-off (IBO), and error performance, with some interesting insights.The proposed RIS-OTFS-FTN scheme demonstrates notable performance gains in both reliability and spectral efficiency, offering a viable solution for future high-mobility and spectrum-constrained wireless systems.
Read the full abstract ▾
All authors Chaorong Zhang, Benjamin K. Ng, Hui Xu, Chan-Tong Lam, Halim Yanikomeroglu
Image and Video Processing
Branch Learning in MRI: More Data, More Models, More Training
Yuyang Li, Yipin Deng, Zijian Zhou et al. (+1 more)
We investigated two complementary strategies for multicontrast cardiac MR reconstruction: physics-consistent data-space augmentation (DualSpaceCMR) and parameter-efficient capacity scaling via VQPrompt and Moero. DualSpaceCMR couples image-level transforms with kspace noise and motion simulations while preserving forwardmodel consistency. VQPrompt adds a lightweight bottleneck
prompt; Moero embeds a sparse mixture of experts within a deep unrolled network with histogram-based routing. In the multivendor, multisite CMRxRecon25
benchmark, we evaluate fewshot and out-of-distribution generalization. On small
datasets, k-space motion-plus-noise improves reconstruction; on the large
benchmark it degrades performance, revealing sensitivity to augmentation ratio and schedule. VQPrompt produces modest and consistent gains with negligible
memory overhead. Moero continues to improve after early plateaus and maintains baseline-like fewshot and out-of-distribution behavior despite mild overfitting, but sparse routing lowers PyTorch throughput and makes wall clock time the main bottleneck. These results motivate scale-aware augmentation and suggest
prompt-based capacity scaling as a practical path, while efficiency improvements are crucial for sparse expert models.
Read the full abstract ▾
All authors Yuyang Li, Yipin Deng, Zijian Zhou, Peng Hu
Audio and Speech Processing
LP-CFM: Perceptual Invariance-Aware Conditional Flow Matching for Speech Modeling
Doyeop Kwak, Youngjoon Jang and Joon Son Chung
The goal of this paper is to provide a new perspective on
speech modeling by incorporating perceptual invariances such as amplitude scaling and temporal shifts. Conventional generative formulations often treat each
dataset sample as a fixed representative of the target distribution. From a generative standpoint, however, such samples are only one among many perceptually equivalent variants within the true
speech distribution. To address this, we propose Linear Projection Conditional Flow Matching (LP-CFM), which models targets as projection-aligned elongated Gaussians along perceptually equivalent variants. We further introduce Vector Calibrated Sampling (VCS) to keep the sampling process aligned with the line-projection path. In neural vocoding experiments across model sizes, data scales, and sampling steps, the proposed approach consistently improves over the conventional optimal transport CFM, with particularly strong gains in low-resource and few-step scenarios. These results highlight the potential of LP-CFM and VCS to provide more robust and perceptually grounded generative modeling of
speech.
Read the full abstract ▾
Audio and Speech Processing · Cross-listed from cs.CL
SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Maxime Poli, Mahi Luthra, Youssef Benchekroun et al. (+8 more)
The parallel advances in
language modeling and
speech representation learning have raised the prospect of learning
language directly from
speech without textual intermediates. This requires extracting semantic representations directly from
speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised
speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken
language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-
distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream
language modeling
benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between
speech unit quality (ABX, PNMI) and
language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr.
Read the full abstract ▾
All authors Maxime Poli, Mahi Luthra, Youssef Benchekroun, Yosuke Higuchi, Martin Gleize, Jiayi Shen, Robin Algayres, Yu-An Chung, Mido Assran, Juan Pino, Emmanuel Dupoux
Audio and Speech Processing · Cross-listed from cs.CV
TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak et al. (+2 more)
The objective of this paper is to jointly synthesize interactive
videos and conversational
speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational
speech generation. However, these works are typically studied in isolation, overlooking the
multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational
speech in a synchronized manner. TAVID integrates face and
speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and
speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.
Read the full abstract ▾
All authors Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Joon Son Chung, Shinji Watanabe
Systems and Control
Data-based Moving Horizon Estimation under Irregularly Measured Data
Tobias M. Wolff, Isabelle Krauss, Victor G. Lopez et al. (+1 more)
In this work, we introduce a sample- and data-based moving horizon estimation framework for linear systems. We perform state estimation in a sample-based fashion in the sense that we assume to have only few, irregular output measurements available. This setting is encountered in applications where measuring is expensive or time-consuming. Furthermore, the state estimation framework does not rely on a standard mathematical model, but on an implicit system representation based on measured data. We prove sample-based practical robust exponential stability of the proposed
estimator under mild assumptions. Furthermore, we apply the proposed scheme to estimate the states of a gastrointestinal tract absorption system.
Read the full abstract ▾
All authors Tobias M. Wolff, Isabelle Krauss, Victor G. Lopez, Matthias A. Müller
Image and Video Processing · Cross-listed from cs.CV
Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
Binfeng Wang, Di Wang, Haonan Guo et al. (+2 more)
Unified hyperspectral image (HSI) restoration aims to recover diverse degradations within a single model. However, current methods often rely on impractical explicit priors or opaque black-box representations that overfit to training distributions, hampering generalization to unseen scenarios. To bridge this gap, we propose Degradation-Aware Metric Prompting (DAMP), a novel framework that characterizes multi-dimensional degradations through interpretable spatial-spectral metrics. These metrics serve as Degradation
Prompts (DP), enabling the model to capture shared characteristics across tasks and adapt to unknown corruptions. Central to our framework is the Degradation-Adaptive Mixture-of-Experts (DAMoE), where Spatial-Spectral Adaptive Modules (SSAMs) serve as experts that utilize learnable fusion coefficients to specialize in distinct degradation degrees. By using DP as a gating router, DAMoE dynamically activates specialized experts tailored to the specific degradation profile. Extensive experiments on natural and remote sensing HSI
datasets demonstrate that DAMP achieves state-of-the-art performance and exhibits exceptional zero-shot generalization on unseen restoration tasks. Code is publicly available at \href{DAMP}{https://github.com/MiliLab/DAMP}.
Read the full abstract ▾
All authors Binfeng Wang, Di Wang, Haonan Guo, Ying Fu, Jing Zhang
Systems and Control
Inference in Latent Force Models Using Optimal State Estimation
Tobias M. Wolff, Victor G. Lopez, Matthias A. Müller et al. (+1 more)
Latent force models, a class of hybrid modeling approaches, integrate physical knowledge of system dynamics with a latent force - an unknown, unmeasurable input modeled as a Gaussian process. In this work, we introduce two optimal state estimation frameworks to reconstruct the latent forces and to estimate the states. In contrast to state-of-the-art approaches, the designed
estimators enable the consideration of system-inherent constraints. Finally, the performance of the novel frameworks is investigated in several numerical examples. In particular, we demonstrate the performance of the new framework in a real-world biomedical example - the hypothalamic-pituitary-thyroid axis - using hormone measurements.
Read the full abstract ▾
All authors Tobias M. Wolff, Victor G. Lopez, Matthias A. Müller, Thomas Beckers
Image and Video Processing · Cross-listed from cs.LG
Unified Multimodal Brain Decoding via Cross-Subject Soft-ROI Fusion
Xuanyu Hu
Multimodal brain decoding aims to reconstruct semantic information that is consistent with visual stimuli from
brain activity signals such as fMRI, and then generate readable natural
language descriptions. However,
multimodal brain decoding still faces key challenges in cross-subject generalization and interpretability. We propose a BrainROI model and achieve leading-level results in
brain-captioning evaluation on the NSD
dataset. Under the cross-subject setting, compared with recent state-of-the-art methods and representative baselines, metrics such as BLEU-4 and CIDEr show clear improvements. Firstly, to address the heterogeneity of functional
brain topology across subjects, we design a new fMRI encoder. We use multi-atlas soft functional parcellations (soft-ROI) as a shared space. We extend the discrete ROI Concatenation strategy in MINDLLM to a voxel-wise gated fusion mechanism (Voxel-gate). We also ensure consistent ROI mapping through global label
alignment, which enhances cross-subject transferability. Secondly, to overcome the limitations of manual and black-box prompting methods in stability and transparency, we introduce an interpretable
prompt optimization process. In a small-sample closed loop, we use a locally deployed Qwen model to iteratively generate and select human-readable
prompts. This process improves the stability of
prompt design and preserves an auditable optimization trajectory. Finally, we impose parameterized decoding constraints during inference to further improve the stability and quality of the generated descriptions.
Read the full abstract ▾
Signal Processing
Sum-Rate Maximization for Uplink Segmented Waveguide-Enabled Pinching-Antenna Systems
Songnan Gu, Hao Jiang, Chongjun Ouyang et al. (+2 more)
A multiuser uplink transmission framework based on the segmented waveguide-enabled pinching-
antenna system (SWAN) is proposed under two operating protocols: segment selection (SS) and segment aggregation (SA). For each protocol, the achievable uplink sum-rate is characterized for both time-division multiple access (TDMA) and non-orthogonal multiple access (NOMA). Low-complexity placement methods for the pinching
antennas (PAs) are developed for both protocols and for both multiple-access schemes. Numerical results validate the effectiveness of the proposed methods and show that SWAN achieves higher sum-rate performance than conventional pinching-
antenna systems, while SA provides additional performance gains over SS.
Read the full abstract ▾
All authors Songnan Gu, Hao Jiang, Chongjun Ouyang, Yuanwei Liu, Dong In Kim
Systems and Control
Sliding Mode Control for a Parabolic-Elliptic PDE System with Boundary Perturbation
Moussa Labbadi and Ilyasse Lamrani
In this paper, we address the robustness of
parabolic-
elliptic systems under boundary control. A sliding mode control strategy is proposed to reject matched perturbations. The stability analysis establishes finite-time convergence of the sliding
manifold and exponential stability of the closed-loop system. Since the closed-loop system is discontinuous, we also prove its well-posedness. A numerical example is provided to validate the effectiveness of the proposed approach.
Read the full abstract ▾
Systems and Control
Finite-Time Control Based on Differential Flatness for Wheeled Mobile Robots with Experimental Validation
Imtiaz Ur Rehman, Moussa Labbadi, Amine Abadi et al. (+1 more)
A robust tracking control strategy is designed to empower wheeled mobile
robots (WMRs) to track predetermined routes while operating in diverse fields and encountering disturbances like strong winds or uneven path conditions, which affect tracking performance. Ensuring the applicability of this tracking method in real-world scenarios is essential. To accomplish this, the WMR model is initially transformed into a linear canonical form by leveraging the differential flatness of its kinematic model, facilitating controller design. Subsequently, a novel integral nonlinear hyperplane-based sliding mode control (INH-SMC) technique is proposed for WMR under disturbances. The stability of the technique is analyzed and verified. Finally, its practical viability is demonstrated through a comparative real-world indoor experiment on a TurtleBot3 WMR subjected to disturbances, confirming the feasibility and efficacy of the proposed approach.
Read the full abstract ▾
All authors Imtiaz Ur Rehman, Moussa Labbadi, Amine Abadi, Lew Lew Yan Voon
Systems and Control
Robust safety design for strict-feedback nonlinear systems via observer-based linear time varying feedback
Imtiaz Ur Rehman, Moussa Labbadi, Amine Abadi et al. (+1 more)
This paper develops a robust safety-critical control method for nonlinear strictfeedback systems with mismatched disturbances. Using a state transformation and a linear time-varying disturbance observer, the system is converted into a form that enables safe control design. The approach ensures forward invariance of the safety set and also applies to disturbancefree systems. Safety is proven for all cases, and a numerical example illustrates the results.
Read the full abstract ▾
All authors Imtiaz Ur Rehman, Moussa Labbadi, Amine Abadi, Lew Lew Yan Voon
Audio and Speech Processing · Cross-listed from cs.SD
Aliasing-Free Neural Audio Synthesis
Yicheng Gu, Junan Zhang, Chaoren Wang et al. (+3 more)
In neural audio synthesis, neural vocoders and codecs are models that reconstruct waveforms from acoustic and latent representations, which are essential to the resulting audio quality. While current models are capable of generating perceptually natural
speech, they still struggle with high-fidelity music and singing voice synthesis, as severe aliasing artifacts are introduced by non-linear activation functions and upsampling layers in existing architectures. Although various anti-aliasing techniques have been proposed in digital signal processing, their integration into neural vocoders and codecs remains under-explored. This paper incorporates differentiable anti-aliasing techniques into the activation and upsampling modules to bridge this gap, and thus presents Pupu-Vocoder and Pupu-Codec. We build a test signal
benchmark to evaluate the anti-aliased modules, and validate our proposed models on
speech, singing voice, music, and audio. Experimental results show that Pupu-Vocoder and Pupu-Codec outperform existing systems on singing voice, music, and audio, while achieving comparable performance on
speech. Demos, codes, and checkpoints are available at: VocodexElysium.github.io/AliasingFreeNeuralAudioSynthesis/.
Read the full abstract ▾
All authors Yicheng Gu, Junan Zhang, Chaoren Wang, Jerry Li, Zhizheng Wu, Lauri Juvela
Systems and Control
Joint Design of Embedded Index Coding and Beamforming for MIMO-based Distributed Computing via Multi-Agent Reinforcement Learning
Heekang Song and Wan Choi
In distributed computing systems, reducing the communication load during the data shuffling phase is a critical challenge, as excessive inter-node transmissions are a major performance bottleneck. One promising approach to alleviate this burden is Embedded Index Coding (EIC), which exploits cached data at user nodes to encode transmissions more efficiently. However, most prior work on EIC has focused on minimizing code length in wired, error-free environments-an objective often suboptimal for wireless multiple-input multiple-output (
MIMO) systems, where channel conditions and spatial multiplexing gains must be considered. This paper investigates the joint design of EIC and transmit beamforming in
MIMO systems to minimize total transmission time, an NP-hard problem. We first present a conventional optimization method that determines the optimal EIC via exhaustive search. To address its prohibitive complexity and adapt to dynamic wireless environments, we propose a novel, low-complexity multi-
agent reinforcement learning (MARL) framework. The proposed framework enables decentralized
agents to act on local observations while effectively managing the hybrid action space of discrete EIC selection and continuous beamforming design. Simulation results demonstrate that the proposed MARL-based approach achieves near-optimal performance with significantly reduced complexity, underscoring its effectiveness and practicality for real-world wireless systems.
Read the full abstract ▾
Signal Processing · Cross-listed from cs.AR
Designing Spatial Architectures for Sparse Attention: STAR Accelerator via Cross-Stage Tiling
Huizheng Wang, Taiquan Wei, Hongbin Wang et al. (+6 more)
Large
language models (LLMs) rely on self-
attention for contextual understanding, demanding high-throughput inference and large-scale
token parallelism (LTPP). Existing dynamic sparsity accelerators falter under LTPP scenarios due to stage-isolated optimizations. Revisiting the end-to-end sparsity acceleration flow, we identify an overlooked opportunity: cross-stage coordination can substantially reduce redundant computation and
memory access. We propose
STAR, a cross-stage compute- and
memory-efficient
algorithm-hardware co-design tailored for
Transformer inference under LTPP.
STAR introduces a leading-zero-based sparsity prediction using log-domain add-only operations to minimize prediction overhead. It further employs distributed sorting and a sorted updating FlashAttention mechanism, guided by a coordinated tiling strategy that enables fine-grained stage interaction for improved
memory efficiency and latency. These optimizations are supported by a dedicated
STAR accelerator architecture, achieving up to 9.2$\times$ speedup and 71.2$\times$ energy efficiency over A100, and surpassing SOTA accelerators by up to 16.1$\times$ energy and 27.1$\times$ area efficiency gains. Further, we deploy
STAR onto a multi-core spatial architecture, optimizing dataflow and execution orchestration for ultra-long sequence processing. Architectural evaluation shows that, compared to the baseline design, Spatial-
STAR achieves a 20.1$\times$ throughput improvement.
Read the full abstract ▾
All authors Huizheng Wang, Taiquan Wei, Hongbin Wang, Zichuan Wang, Xinru Tang, Zhiheng Yue, Shaojun Wei, Yang Hu, Shouyi Yin
Image and Video Processing · Cross-listed from cs.CV
Generative Latent Coding for Ultra-Low Bitrate Image Compression
Zhaoyang Jia, Jiahao Li, Bin Li et al. (+2 more)
Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better
alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.
Read the full abstract ▾
All authors Zhaoyang Jia, Jiahao Li, Bin Li, Houqiang Li, Yan Lu
Audio and Speech Processing · Cross-listed from cs.SD
Spectral or spatial? Leveraging both for speaker extraction in challenging data conditions
Aviad Eisenberg, Sharon Gannot and Shlomo E. Chazan
This paper presents a robust multi-channel speaker extraction
algorithm designed to handle inaccuracies in reference information. While existing approaches often rely solely on either spatial or spectral cues to identify the target speaker, our method integrates both sources of information to enhance robustness. A key aspect of our approach is its emphasis on stability, ensuring reliable performance even when one of the features is degraded or misleading. Given a noisy mixture and two potentially unreliable cues, a dedicated network is trained to dynamically balance their contributions-or disregard the less informative one when necessary. We evaluate the system under challenging conditions by simulating inference-time errors using a simple direction of arrival (DOA)
estimator and a noisy spectral enrollment process. Experimental results demonstrate that the proposed model successfully extracts the desired speaker even in the presence of substantial reference inaccuracies.
Read the full abstract ▾
Audio and Speech Processing · Cross-listed from cs.CL
Fun-Audio-Chat Technical Report
Tongyi Fun Team, Qian Chen, Luyao Cheng et al. (+10 more)
Recent advancements in joint
speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between
speech tokens (25Hz) and text
tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio
Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution
Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via
token grouping), while the
Speech Refined Head generates high-quality
tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding,
reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on
Speech-to-Text and
Speech-to-
Speech tasks, ranking top among similar-scale models on Spoken QA
benchmarks. They also achieve competitive to superior performance on Audio Understanding,
Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo, at https://github.com/FunAudioLLM/Fun-Audio-Chat .
Read the full abstract ▾
All authors Tongyi Fun Team, Qian Chen, Luyao Cheng, Chong Deng, Xiangang Li, Jiaqing Liu, Chao-Hong Tan, Wen Wang, Junhao Xu, Jieping Ye, Qinglin Zhang, Qiquan Zhang, Jingren Zhou
Signal Processing
Target Classification for Integrated Sensing and Communication in Industrial Deployments
Luca Barbieri, Marcus Henninger, Paolo Tosi et al. (+4 more)
Integrated Sensing and Communication (
ISAC) systems enable cellular networks to jointly operate as communication technology and sense the environment. While opportunities and potential performance have been largely investigated in simulations, few experimental works have showcased Automatic Target Recognition (ATR) effectiveness in a real-world deployment based on cellular
radio units. To bridge this gap, this paper presents an initial study investigating the feasibility of ATR for
ISAC. Our ATR solution uses a Deep Learning (DL)-based detector to infer the target class directly from the radar images generated by the
ISAC system. The DL detector is evaluated with experimental data from a
ISAC testbed based on commercially available mmWave
radio units in the ARENA 2036 industrial research campus located in Stuttgart, Germany. Experimental results demonstrate accurate classification performance, demonstrating the feasibility of ATR
ISAC with cellular hardware in our setup. We finally provide insights about the open generalization challenges, that will fuel future work on the topic.
Read the full abstract ▾
All authors Luca Barbieri, Marcus Henninger, Paolo Tosi, Artjom Grudnitsky, Mattia Brambilla, Monica Nicoli, Silvio Mandelli
Audio and Speech Processing
QuarkAudio Technical Report
Chengwei Liu, Haoyin Yan, Shaofei Xue et al. (+5 more)
Many existing audio processing and generation models rely on task-specific architectures, resulting in fragmented development efforts and limited extensibility. It is therefore promising to design a unified framework capable of handling multiple tasks, while providing robust instruction and audio understanding and high-quality audio generation. This requires a compatible paradigm design, a powerful backbone, and a high-fidelity audio reconstruction module. To meet these requirements, this technical report introduces QuarkAudio, a decoder-only autoregressive (AR) LM-based generative framework that unifies multiple tasks. The framework includes a unified discrete audio tokenizer, H-Codec, which incorporates self-supervised learning (SSL) representations into the tokenization and reconstruction process. We further propose several improvements to H-Codec, such as a dynamic frame-rate mechanism and extending the audio sampling rate to 48 kHz. QuarkAudio unifies tasks by using task-specific conditional information as the conditioning sequence of the decoder-only LM, and predicting discrete target audio
tokens in an AR manner. The framework supports a wide range of audio processing and generation tasks, including
speech restoration (SR), target speaker extraction (TSE),
speech separation (SS), voice conversion (VC), and
language-queried audio source separation (LASS). In addition, we extend downstream tasks to universal free-form audio editing guided by natural
language instructions (including
speech semantic editing and audio event editing). Experimental results show that H-Codec achieves high-quality audio reconstruction with a low frame rate, improving both the efficiency and performance of downstream audio generation, and that QuarkAudio delivers competitive or comparable performance to state-of-the-art task-specific or multi-task systems across multiple tasks.
Read the full abstract ▾
All authors Chengwei Liu, Haoyin Yan, Shaofei Xue, Xiaotao Liang, Xiaofu Chen, Bin Gong, Zheng Xue, Gang Song
Signal Processing
Net 582-Gb/s C-band and 4$\times$526-Gb/s O-band IMDD Transmis-sion Using Ultra-broadband InP-DHBT-based Electrical Mixer
Masanori Nakamura, Teruo Jyo, Munehiko Nagatani et al. (+11 more)
We successfully transmitted a net 582-Gb/s probabilistically shaped PAM12 C-band signal over 11-km dispersion-shifted fibre and net 4$\times$526-Gb/s uniform PAM8 O-band signals over 2-km four-core fibre using a single-carrier 216-GBd IMDD system based on a 150-GHz bandwidth InP-DHBT electrical mixer and a thin-film lithium-niobate modulator.
Read the full abstract ▾
All authors Masanori Nakamura, Teruo Jyo, Munehiko Nagatani, Hitoshi Wakita, Miwa Mutoh, Yuta Shiratori, Hiroki Taniguchi, Akira Masuda, Shuto Yamamoto, Fukutaro Hamaoka, Etsushi Yamazaki, Hiroyuki Takahashi, Takayuki Kobayashi, Yutaka Miyamoto
Audio and Speech Processing
SpatialNet with Binaural Loss Function for Correcting Binaural Signal Matching Outputs under Head Rotations
Dor Shamay and Boaz Rafaely
Binaural reproduction is gaining increasing
attention with the rise of devices such as virtual reality headsets, smart glasses, and head-tracked headphones. Achieving accurate binaural signals with these systems is challenging, as they often employ arbitrary microphone arrays with limited spatial resolution. The Binaural Signals Matching with Magnitude Least-Squares (BSM-MagLS) method was developed to address limitations of earlier BSM formulations, improving reproduction at high frequencies and under head rotation. However, its accuracy still degrades as head rotation increases, resulting in spatial and timbral artifacts, particularly when the virtual listener's ear moves farther from the nearest microphones. In this work, we propose the integration of deep learning with BSM-MagLS to mitigate these degradations. A post-processing framework based on the SpatialNet network is employed, leveraging its ability to process spatial information effectively and guided by both signal-level loss and a perceptually motivated binaural loss derived from a theoretical model of human binaural hearing. The effectiveness of the approach is investigated in a simulation study with a six-microphone semicircular array, showing its ability to perform robustly across head rotations. These findings are further studied in a listening experiment across different reverberant acoustic environments and head rotations, demonstrating that the proposed framework effectively mitigates BSM-MagLS degradations and provides robust correction across substantial head rotations.
Read the full abstract ▾
Audio and Speech Processing · Cross-listed from cs.CV
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Jingqi Tian, Yiheng Du, Haoji Zhang et al. (+6 more)
Audio-Visual
Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source
entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional
Alignment via Disentangled Audio Semantics for Audio-Visual
Segmentation. To mitigate multi-source
entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype
memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross
attention with delayed modality interaction, improving the robustness of
multimodal alignment. Extensive experiments on the AVS-Objects and VPO
benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual
segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/
Read the full abstract ▾
All authors Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang, Isaac Ning Lee, Xulong Bai, Tianrui Zhu, Jingxuan Niu, Yansong Tang
Image and Video Processing · Cross-listed from cs.CV
Attention Fusion for Bridge Deck Delamination Detection
Alireza Moayedikia and Amirhossein Moayedikia
Subsurface delaminations in reinforced concrete bridge decks escape conventional visual inspection, and the two principal sensing techniques used to find them are individually incomplete: Ground Penetrating Radar (GPR) penetrates deeply but degrades near the surface, while Infrared Thermography (IRT) resolves shallow defects but cannot reach deeper structure. This paper presents a framework for fusing the two modalities through hierarchical
attention: temporal self-
attention over GPR A-scans, channel-spatial
attention over IRT patches, and cross-modal multi-head
attention with learnable modality
embeddings, coupled with decomposed aleatoric/epistemic uncertainty estimation. Beyond the architecture itself, which is lightweight at approximately 0.53M parameters with a closed-form accounting of where capacity resides, we contribute an elementary formal analysis. Two-
token cross-modal
attention is shown to be exactly a bank of per-sample learned gates; a gradient-allocation proposition quantifies how class imbalance starves
attention parameters of minority-class signal and how loss reweighting trades that starvation for gradient variance; and closed-form metric floors under majority-class collapse anchor a diagnostic divergence between ranking metrics (AUC) and thresholded metrics (F1). The analysis suggests that adaptively weighted fusion, precisely because its feature-selection policy is learned, may be distinctively vulnerable to the severe class imbalance typical of operational bridge decks; establishing whether and when this occurs is deferred to empirical evaluation.
Read the full abstract ▾
Signal Processing · Cross-listed from cs.IT
Generative Spectrum Cartography: Unified Reconstruction and Active Sensing via Diffusion Models
Yuntong Gu, Xiangming meng, Zhiyuan Lin et al. (+2 more)
High-fidelity spectrum cartography is important for spectrum monitoring and wireless situational awareness, especially in satellite-based wide-area sensing scenarios where measurements are sparse, noisy, and often low-bit quantized. In such settings, two coupled challenges arise: accurate reconstruction from severely incomplete measurements and efficient allocation of additional sensing resources under a limited sensing budget. Existing methods usually address these problems separately, and, for reconstruction, they often rely on priors that are insufficiently expressive under sparse and quantized measurements. This paper proposes Generative Spectrum Cartography (GSC), a
diffusion-based posterior inference framework for spectrum cartography with uncertainty-aware active sensing. Specifically, spectrum map recovery is formulated as a
Bayesian inverse problem under a learned
diffusion model prior, and closed-form posterior mean updates are derived for both linear and quantized measurement models. By
embedding these updates into the reverse
diffusion process, GSC enables gradient-free and measurement-consistent posterior sampling without relying on computationally costly likelihood-gradient guidance. The resulting posterior samples are further used to estimate spatial uncertainty and to guide diversity-aware selection of additional measurement locations for active sensing. Experiments on simulated electromagnetic maps and a high-fidelity simulated satellite monitoring scenario show that GSC achieves higher PSNR, lower LPIPS, and more efficient sensing than representative baseline methods under sparse, noisy, and low-bit quantized measurements.
Read the full abstract ▾
All authors Yuntong Gu, Xiangming meng, Zhiyuan Lin, Sheng Wu, Linling Kuang
Image and Video Processing
Neural Compression of 360-Degree Equirectangular Videos using Quality Parameter Adaptation
Daichi Arai, Yuichi Kondo, Kyohei Unno et al. (+2 more)
This study proposes a practical approach for compressing 360-degree equirectangular
videos using pretrained neural
video compression (NVC) models. Without requiring additional training or changes in the model architectures, the proposed method extends quantization parameter adaptation techniques from traditional
video codecs to NVC, utilizing the spatially varying sampling density in equirectangular projections. We introduce latitude-based adaptive quality parameters through rate-distortion optimization for NVC. The proposed method utilizes vector bank interpolation for latent modulation, enabling flexible adaptation with arbitrary quality parameters and mitigating the limitations caused by rounding errors in the adaptive quantization parameters. Experimental results demonstrate that applying this method to the DCVC-RT framework yields BD-Rate savings of 5.2% in terms of the weighted spherical peak signal-to-noise ratio for JVET class S1 test sequences, with only a 0.3% increase in processing time.
Read the full abstract ▾
All authors Daichi Arai, Yuichi Kondo, Kyohei Unno, Yasuko Sugito, Yuichi Kusakabe