Audio and Speech Processing
Monaural Continuous-Radius Regional Speech Extraction with Cross-Radius Consistency Learning
Biao Dong, Jie Chen, Jianwei Fang et al. (+3 more)
Source-to-microphone distance enables speaker-independent
speech extraction without prior enrollment. We propose a monaural continuous-radius regional
speech extraction method that directly models the cumulative
speech target within a queried radius. To realize continuous region control, the query radius is encoded as a continuous scalar and injected into a time-frequency extraction network, allowing a single model to operate over arbitrary radii within the trained range. Exploiting the nested structure of target-speaker sets across query radii, we introduce cross-radius consistency learning to stabilize predictions for adjacent radii sharing the same nonempty target-speaker set. Experiments on measured RIRs show that continuous-radius conditioning improves selective extraction over discrete conditioning. The proposed method achieves 29.25~dB SI-SDR, 4.40~dB SI-SDRi, 62.01~dB attenuation, and 0.46% RCE, outperforming fixed-threshold and local-range baselines. It also remains effective with more speakers and additive noise.
Read the full abstract ▾
All authors Biao Dong, Jie Chen, Jianwei Fang, Wei Xiao, Jiqing Han, Yongjun He
Audio and Speech Processing · Cross-listed from cs.SD
Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
Shiao Zhu, Lianbo Liu, Sizhen Lyu et al. (+3 more)
Natural-
language style descriptions provide an interpretable interface between large
language models (LLMs) and controllable text-to-
speech (TTS). However, using descriptions as pseudo-labels compresses target acoustics into text, and descriptive fidelity need not imply effective control of a particular synthesizer. We empirically show that
speech-text
alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance. We therefore propose
Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model. Given dialogue history and response text, the planner generates candidate instructions and is optimized with group-relative policy optimization (GRPO), using the teacher-forced likelihood of target
speech tokens as the reward. On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves higher
speech-style and emotion similarity to target
speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines. LLM-based expressive
speech evaluation further shows gains over all baselines in contextual appropriateness and reference consistency.
Read the full abstract ▾
All authors Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang, Sheng Li, Takahiro Shinozaki
Audio and Speech Processing · Cross-listed from cs.SD
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
Junchuan Zhao, Chenglin Xu, Wei Zeng et al. (+3 more)
Instruction-based text-to-
speech (TTS) offers control over voice characteristics and
speech expression through interfaces including voice cloning and text-based voice design. Voice cloning reproduces a reference voice, whereas text-based voice design creates a voice from a natural-
language description. However, neither interface directly enables users to modify the timbre of a given reference and synthesize
speech with the modified voice. Meanwhile, utterance-level expressive instructions leave changes across individual text segments underspecified. We introduce EDICT, a framework that unifies global timbre editing and local expressive control by using an edited acoustic reference to anchor voice identity across segments. To enable synthesis with an instruction-edited voice, EDICT combines reference audio with structured timbre edits to generate an edited reference in codec-
token space. This representation serves as a shared voice anchor for a frozen TTS backbone, allowing segment-specific natural-
language instructions to guide expression. To accommodate instruction changes while supporting acoustic continuity, EDICT rebuilds the KV cache at each segment boundary, refreshing instruction conditioning while retaining bounded acoustic context from previously generated
speech. Evaluations on our proposed TimbreEdit-Bench and IntraTTS-Bench demonstrate improved timbre editing and a favorable balance between local instruction adherence, speaker consistency, and transition quality. Audio demos are available.
Read the full abstract ▾
All authors Junchuan Zhao, Chenglin Xu, Wei Zeng, Haoyang Li, Yiwen Guo, Ye Wang
Signal Processing
An Embodied Multiagent Framework Based on Token Communications for Cooperative ISAC
Jiahe Guo, Jun Du, Chunxiao Jiang et al. (+1 more)
The emerging low-altitude economy demands unmanned aerial vehicle (
UAV)-enabled integrated sensing and communication (
ISAC) for reliable connectivity and environmental awareness. In particular, embodied
UAV agents offer a promising means of supporting autonomous operations through a closed loop linking perception, decision-making, and physical actions. However, each
UAV has access only to local observations, and effective cooperation requires exchanging local states and intentions. Directly sharing such information can incur substantial signaling overhead and hinder timely coordination in dynamic environments. To deal with this problem, this paper investigates a cooperative
ISAC network of embodied
UAV agents and formulates a joint
token communications (TokCom) and physical control problem to minimize total propulsion energy subject to communication and sensing rate requirements. Then, we propose a state–intent TokCom (SI-TokCom) framework driven by multi-
agent embodied policy learning. Specifically, separate pretrained codebooks enable compact exchanges of local states and intentions, while
UAV agents jointly learn to select and compose
tokens and determine physical actions based on local observations and received
tokens. Simulation results show that SI-TokCom achieves 98.9% and 99.1% of the centralized baseline's communication and sensing rates, respectively. Compared with the local baseline, it improves the corresponding rates by 5.0% and 43.6%, respectively, with essentially unchanged propulsion energy. These results highlight the potential of TokCom for communication-efficient cooperation among embodied
UAV agents in
ISAC systems.
Read the full abstract ▾
All authors Jiahe Guo, Jun Du, Chunxiao Jiang, Jintao Wang
Systems and Control
Causal-fate dynamics of unrealized influence
Yiwei Liu, Luwei Yang and Shunbo Lei
Many dynamical systems generate influences whose consequences are not fully exhausted in the realized trajectory at the moment they arise. Such consequences are often treated as absent, delayed or statically stored, leaving unclear how unrealized influence retains future relevance as the system evolves. Here we formulate causal-fate dynamics, in which generated influence may be realized, remain latent, or be transformed by subsequent dynamics, and give an exact finite-transport representation when the relevant maps are specified. A connectome-constrained Caenorhabditis elegans model first motivates the biological hypothesis that unresolved inter-neuronal influence may persist and contribute to later propagation; it does not establish such a mechanism in living animals. We next examine operational Internet routing, where a dynamically updated cross-observer history retains predictive information beyond the current local route state. We then use the representation to construct a
Transformer architecture that explicitly transports and selectively realizes latent contextual influence while retaining
language-modeling function. The three studies distinguish a model-motivated scientific hypothesis, an observational phenomenon compatible with future-relevant history and an executable construction for carrying unrealized influence through subsequent computation.
Read the full abstract ▾
Systems and Control · Cross-listed from physics.app-ph
Time-Domain Analysis of Surface Acoustic Wave Magnetoelectric Sensors
Mohsen Samadi, Henrik Wolframm, Felix Weisheit et al. (+4 more)
The transient response of surface acoustic wave sensors is important for understanding their dynamic behavior, propagation delay, and response to time-varying signals. In this work, the time-domain propagation characteristics of a magnetoelectric SAW sensor based on a multilayer Love-wave delay line are investigated using complementary numerical and experimental approaches. Time-domain analysis is employed to determine the delay time and to extract the phase and group velocities, while frequency-domain analysis provides an independent characterization of the wave dispersion. Both numerically and experimentally, the phase and group velocities are found to differ, reflecting the dispersion of the Love wave in the multilayer structure. At the operating frequency of 146 MHz, the simulated group velocity obtained from wave-field tracking agrees within 0.4% with the value determined from the frequency-domain dispersion relation. The simulated delay time and group velocity also show good agreement with experiment, with deviations of 2.7% in delay time and 0.9% in group velocity. These results demonstrate the consistency of the time- and frequency-domain analyses and provide a quantitative characterization of the transient propagation behavior of SAW sensors.
Read the full abstract ▾
All authors Mohsen Samadi, Henrik Wolframm, Felix Weisheit, Dirk Meyners, Eckhard Quandt, Michael Höft, Martina Gerken
Signal Processing
Unrolled Time-Varying Graph Signal Restoration under Spatiotemporal Smoothness Priors
Hayate Kojima, Hikari Noguchi, Koki Yamada et al. (+1 more)
In this paper, we propose restoration methods for time-varying graph signals using deep
algorithm unrolling (DAU). Time-varying graph signals, such as signals obtained from sensor networks, are nonuniformly distributed in space and observed as time series. Since these observed signals often contain noise and missing values, their restoration needs to consider both spatial and temporal relationships. Our approach is based on an optimization problem that models signal properties using a spatiotemporal regularizer that combines a Sobolev operator for spatial smoothness and a multi-tap FIR filter for temporal smoothness. We then unroll the iterative conjugate gradient method to solve this problem and learn the regularization parameters and filter coefficients in each iteration. Our method can be applied to supervised batch, supervised online, and unsupervised batch settings to learn these parameters. Experiments on several synthetic and real-world
datasets show that the supervised batch method achieves the lowest RMSEs in almost all cases. The supervised online and the unsupervised batch methods outperform the existing methods of their own settings, and are often competitive with the existing batch methods.
Read the full abstract ▾
All authors Hayate Kojima, Hikari Noguchi, Koki Yamada, Yuichi Tanaka
Signal Processing
gr-ntnisac: An Open GNU Radio Testbed for NTN-ISAC
Soham Dhiren Desai
A passive receiver can sense nearby objects from the echoes of a 5G downlink. When the transmitter is a low-Earth-orbit (LEO) satellite, one transmitter covers a wide area, but the receiver must handle tens of kHz of Doppler and a direct-path delay that changes within every observation interval. Such a receiver should be tested where the correct answer is known before it is tried on a satellite. This paper presents gr-ntnisac, an open GNU
Radio testbed that applies an emulated LEO channel to a standard 5G New
Radio downlink, sends it through two cabled USRPs, and scores the receiver against the delay and Doppler commanded for every path. The same receiver code runs in simulation, on recorded captures, and on hardware. As a first use, we measure cancellation of the direct path, the strong satellite signal that masks weak echoes. Depth is 34.8 dB on hardware and 37.6 dB in a matched simulation without
radios, so the
radios are not the main limit. Controlled simulation shows that the emulator's 8-tap fractional-delay filter and a simulated rotating blade at almost the direct path's delay each limit depth to about 40 dB; without either, depth reaches 71.0 dB. A diagnostic canceller that also removes the per-subcarrier response and per-symbol phase reaches 72.7 dB on the same captures. On a conducted capture, a payload-aided receiver detects a -25 dB moving path in 76.9 percent of dwells.
Read the full abstract ▾
Audio and Speech Processing
Toward accurate speaker inventories for online speaker diarization
Youngki Kwon, Hee-Soo Heo, Minjae Lee et al. (+1 more)
An online speaker diarizer maintains a growing inventory of speakers, and applications read it directly: a live transcript enumerates them as they appear. Existing systems either fix the inventory's size in advance, so speakers beyond that size are merged, or grow it on distance alone, so noise and turn boundaries split speakers. Scored on speaker count over nine public
datasets, both failures appear where DER reports neither: on VoxSRC-23 the fixed-capacity systems return 44-46% too few speakers and the unbounded tracker with the lowest DER 63% too many. We build on that
embedding-based tracker and change only what triggers a registration. A candidate speaker pool keeps unregistered
embeddings apart and registers a speaker only once one candidate has gathered enough mutually compatible
embeddings; commit-gated label assignment holds an output while its candidate is undecided, so no speaker appears in the output before it is confirmed. Across the nine
datasets the macro speaker-count error falls from 225% to 21% of the reference and macro DER from 17.59% to 15.95%, at a mean added latency of 0.109s over all outputs; on VoxSRC-23 the tracker obtains both the lowest DER and the most accurate speaker count of the systems compared.
Read the full abstract ▾
All authors Youngki Kwon, Hee-Soo Heo, Minjae Lee, Bong-Jin Lee
Audio and Speech Processing
Using the enrollment gallery as evidence: affine-invariant score calibration for target speaker tagging
Hee-Soo Heo, Minjae Lee, Youngki Kwon et al. (+1 more)
Target speaker tagging (TST) assigns enrolled speaker identities to diarized segments of multi-speaker recordings. The enrollment utterances of the other registered speakers are an attractive source of evidence for calibrating each verification decision: they share the deployment domain of the test material and require no external cohort set. We show, however, that they cannot serve as the cohort of conventional score normalization. Enrollments tend to cluster by recording session, so the per-speaker cohort statistics reflect enrollment proximity to the rest of the gallery rather than impostor behavior, and normalization then rejects entire speakers. We propose gallery affinity verification, a cohort-free calibration that exploits the gallery through two affine-invariant terms, score dispersion and enrollment-profile deviance, and is therefore immune to such per-speaker shifts. On a synthetic
benchmark and an in-house meeting corpus, it improves tagging accuracy with and without conventional score normalization and adds further gains when combined with it.
Read the full abstract ▾
All authors Hee-Soo Heo, Minjae Lee, Youngki Kwon, Bong-Jin Lee
Systems and Control
Reference-Filter-Driven Transition Probabilities for IMM-Based Satellite Maneuver Detection
Euiseok Han, Sangheon Choi and Seung-Hyun Kong
Maneuver detection of non-cooperative satellites is an essential task of space situational awareness. The interacting multiple model (IMM) filter has been applied to detect unannounced maneuvers, but the standard IMM assumes a fixed Markov transition probability
matrix (TPM) that must be tuned without knowledge of the maneuver rate of the target. To avoid this tuning, adaptive-TPM approaches update the TPM online from statistics computed inside the IMM. However, these statistics already depend on the TPM through the mixing step, and this dependence forms a feedback loop. Under the dynamics mismatch of orbit tracking, the loop can keep the maneuver probability high in the absence of a maneuver or suppress its rise when a maneuver occurs. In this paper, the TPM is driven by the normalized innovation squared (NIS) of a reference coast filter that is not mixed with the IMM, and the NIS is mapped continuously to the coast-to-maneuver transition probability. The reference-driven IMM (RD-IMM) is evaluated with four space-based optical sensors tracking a geosynchronous target. RD-IMM detects small burns that the fixed-TPM IMM fails to detect, with few false declarations in maneuver-free runs, and avoids both failures of the closed-loop adaptation. Moreover, RD-IMM reduces the position error after a small in-track burn by more than 50% compared with the single-filter approaches.
Read the full abstract ▾
Image and Video Processing · Cross-listed from cond-mat.mtrl-sci
Localization of Candidate Kikuchi Regions in RHEED Images: Visibility and Annotation Boundaries
Lumou Weng, Hanshan Huang, Zhenhan Zhang et al. (+1 more)
Kikuchi lines and bands in reflection high-energy electron diffraction (RHEED) carry information on
crystal geometry and electron scattering, but are often obscured by intense diffraction streaks. We combine multiscale convolution with spatial detail from skip connections and independent supervision that allows overlapping regions to localize streaks and candidate Kikuchi regions separately. Using manual polygon annotations made before model-assisted editing, the Kikuchi intersection-over-union (IoU) on 29 laboratory test images from separate growth batches was $0.6290 \pm 0.0137$. We then performed supervised adaptation to public chalcogenide images and stratified images by Kikuchi visibility using the median ratings of three observers who rated the images separately, two of whom rated with predictions hidden. With all 244 training images, IoU for the clear-feature group of 10 test images was $0.5014 \pm 0.0336$, whereas ambiguous images gave lower values. With 100 training images, clear-group IoU was $0.5171 \pm 0.0073$; expanding the training set also reduced responses on images without target features. Reported uncertainties are sample standard deviations across four laboratory runs or three adaptation runs. Public-image adaptation used reference masks of mixed provenance, including prediction-derived drafts. The method converts visual cues into inspectable spatial regions, providing a basis for analysis of line positions, intersections, and local intensity.
Read the full abstract ▾
All authors Lumou Weng, Hanshan Huang, Zhenhan Zhang, Gan Wang
Audio and Speech Processing
Randomized Scores and Diverse Timbres: Augmenting Automatic Music Transcription with Online-Generated Data
Haiwen Xia, Chao Zhang and Qiuqiang Kong
Automatic music transcription (AMT) is limited by the scarcity of audio recordings paired with precise symbolic annotations. Synthetic data can provide supervision at scale, but it remains unclear whether effective transfer depends on realistic score structure or broad timbral coverage. We study these factors separately through an online sampler–renderer pipeline. A unified corruption sampler ranges from unmodified MIDI clips through partial corruption to deeply randomized note-event distributions. The renderer converts these events to audio while independently controlling instrument and timbral coverage. A fixed transcription model is trained jointly on offline recordings and newly rendered examples. Controlled ablations reveal an asymmetry between the two factors: moderate corruption of the note-event distribution does not impair transfer and can improve it, whereas broader renderer-side timbral support consistently improves out-of-domain generalization under a fixed note-event distribution. Finally, online-rendered examples complement real and existing synthetic data in a strong combined-data regime. These results suggest that synthetic AMT data should prioritize coverage of note-level attributes and their timbral realizations over realistic joint score structure.
Read the full abstract ▾
Signal Processing
Integrity and Credibility in Navigation: From Error Characterization to Operational Assuran
Penggao Yan, Baoshan Song, Yuan Li et al. (+2 more)
Navigation integrity and
estimator credibility use error and uncertainty information to address different questions. This perspective distinguishes actual positioning error from reported statistical characterizations, formulates integrity as control of hazardous unwarned-use risk under the requirement of a specified operation, and describes credibility as the justification of an uncertainty or risk claim by its evidence usage and modeling process. Three hypothetical scenarios show how deteriorated accuracy, a contradicted uncertainty report, and a hazardous event can lead to different credibility and integrity judgments. The resulting assessment asks what happened, why the reported claim should be believed, and whether the information is sufficient for the intended operation.
Read the full abstract ▾
All authors Penggao Yan, Baoshan Song, Yuan Li, Ronghe Jin, Li-Ta Hsu
Signal Processing
2D Coprime Pilots for Delay-Doppler Sensing in OFDM-ISAC Systems
Guanping Shang, Zeyan Zhuang, Anzheng Tang et al. (+2 more)
Integrated Sensing and Communication (
ISAC) is envisioned to endow future 6G systems with seamless sensing capabilities. To support efficient sensing with minimum communication overhead, sparse pilots embedded within communication frames have emerged as a promising solution. Along this line of research, existing studies have achieved engaging results in maximizing the unambiguous sensing region. However, jointly maximizing the sensing region and sensing accuracy remains challenging due to the lack of a unified performance metric and an effective pilot design framework. This paper jointly optimizes the unambiguous sensing region and sensing accuracy for estimating delay-Doppler (DD) parameters in Orthogonal Frequency Division Multiplexing (OFDM)-
ISAC systems, where sensing mutual information (SMI) is adopted as a unified performance metric to characterize the overall sensing capability. Specifically, the joint optimization is formulated as an SMI maximization problem by systematically resolving sensing ambiguity and optimizing sensing accuracy. In particular, based on the generalized Bezout identity, we derive a 2D (timefrequency) coprime condition, which, as far as the authors know, is the first necessary and sufficient condition to achieve the unique estimation of DD parameters in the literature. Under this unambiguous condition, we further propose an Adam-Guided Iterative Refinement (AGIR)
algorithm to optimize the sensing accuracy. Numerical results demonstrate the advantage of the proposed framework over existing designs, owing to the freedom offered by the 2D coprime condition in optimizing the sensing accuracy.
Read the full abstract ▾
All authors Guanping Shang, Zeyan Zhuang, Anzheng Tang, Shenghui Song, Chi-Ying Tsui
Audio and Speech Processing · Cross-listed from cs.SD
MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness
Jinbo Hu, Hang Su, Lichun Fan et al. (+10 more)
Large audio-
language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-
language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-
language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-
language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and
reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural
benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.
Read the full abstract ▾
All authors Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li, Zhanchen Dai, Yiru Zhang, Chang Liu, Peng Wang, Junnan Wu, Jian Luan, Cong Zou, Heng Qu
Audio and Speech Processing
SmoothConv and DuplexConv: Complementary Mandarin Multi-Party Conversational Speech Corpora for Speech Interaction
Chengyou Wang, Mingchen Shao, Chunjiang He et al. (+8 more)
Recent advances in large audio
language models (LALMs) have driven the development of natural and intelligent
speech interaction systems. Such systems need to model complex conversational behaviors, including turn-taking, overlapping
speech, and speaker coordination. Multi-party conversations provide a realistic setting for studying these behaviors, yet existing Mandarin conversational corpora often lack synchronized participant-level
speech tracks and comprehensive annotations. In this work, we introduce SmoothConv and DuplexConv, two complementary Mandarin multi-party conversational
speech corpora totaling 2,100 hours. SmoothConv provides human-verified conversations for reliable analysis and evaluation, while DuplexConv offers large-scale automatically annotated conversations through a scalable pipeline for model training. Both corpora provide synchronized participant-level
speech tracks and multi-dimensional fine-grained annotations. We further release the SmoothConv
Benchmark and evaluate these resources on
speech separation, multi-speaker automatic
speech recognition (MSASR), and turn detection tasks. Experimental results demonstrate the utility of the proposed resources for multi-party
speech interaction modeling. The
datasets,
benchmark, and related resources are publicly available.
Read the full abstract ▾
All authors Chengyou Wang, Mingchen Shao, Chunjiang He, Zeyu Zhu, Jierui Guo, Bingshen Mu, Zikai Liu, Hanke Xie, Yuhang Dai, Zhou Zhu, Lei Xie
Image and Video Processing · Cross-listed from cs.CV
A Unified Score Matching Paradigm for Video Anomaly Detection and Anticipation
Congqi Cao, Zhenhe Liang, Hanwen Zhang et al. (+4 more)
Video anomaly detection (VAD) is a fundamental and safety-critical task in computer
vision. Recent generative approaches detect anomalies from a distributional perspective, but remain limited by local anomaly modes. Meanwhile,
video anomaly anticipation (VAA), as a proactive extension beyond post-hoc detection, introduces additional challenges. In particular, the contrastive inference paradigm in VAD, which relies on ground-truth frames, is not applicable to VAA, hindering its development. To address these challenges, we propose a unified score-driven framework, termed Uni-DSM, based on denoising score matching (DSM), which models anomaly patterns through likelihood estimation and score functions over the learned data distribution. Within this unified framework, we adopt a shared noise-conditioned score
transformer backbone with scene-dependent
embeddings and motion-aware weighting for distribution-level modeling. Instead of introducing separate architectures, Uni-DSM unifies VAD and VAA through different inference and supervision paradigms built upon the same score-based formulation. For VAD, we instantiate an autoregressive denoising score matching (ADSM) mechanism, which progressively accumulates anomalous evidence via autoregressive denoising, enabling enhanced perception of local modes beyond visual cues. For VAA, we extend the same architecture by incorporating a lightweight auxiliary decoder and a novel self-distilled denoising score matching (SDSM) mechanism. By constructing supervision from output discrepancies instead of relying on unavailable future ground truth, our method achieves efficient training suitable or early anomaly anticipation. Extensive experiments on multiple
benchmark datasets demonstrate state-of-the-art performance in both VAD and VAA while maintaining high efficiency, establishing a unified and scalable pipeline from anomaly detection to anticipation.
Read the full abstract ▾
All authors Congqi Cao, Zhenhe Liang, Hanwen Zhang, Yifan Zhao, Qinyi Lv, Lingtong Min, Yanning Zhang
Image and Video Processing · Cross-listed from cs.CV
Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery
Jingbo Yue, Bruce Coburn, Jinge Ma et al. (+2 more)
Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses
multimodal large
language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery
prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted
retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.
Read the full abstract ▾
All authors Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
Signal Processing
SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech
Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung et al. (+1 more)
Cochlear implants (CIs) restore
speech access but weaken cues needed for vocal emotion recognition. Prior CI-oriented enhancement requires parallel normal/strong recordings and intensity labels. We propose SEER, a
retrieval-based framework that learns which same-emotion reference helps each source remain recognizable after CI processing. A source-conditioned retriever learns CI-aware utility from sampled emotional voice conversion outcomes, while uncertainty-guided exploration avoids exhaustive pair evaluation; neither parallel recordings nor intensity labels are required. SEER improves Source macro-F1 at N8 by 7.30 points on RAVDESS and 11.66 points on ESD, with significant ESD gains across N4/N8/N16. Sixteen-listener RAVDESS gains are significant across all conditions. Exhaustive analysis finds an aggregate benefit from stronger references but little effect from matching gender or content.
Read the full abstract ▾
All authors Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung, Chi-Chun Lee
Systems and Control
A User-Triggered UAV Dispatching System for Precise and Timely Mountain Search Missions
Mingyang Wang, Yi Hong, Tungchak Lee et al. (+4 more)
Smartphones carried by hikers routinely record routes, locations, and requests for assistance, and these records can provide precise clues for mountain search. However, they are rarely transformed in a direct and timely way into unmanned aerial vehicle (
UAV) missions that support search and rescue. This paper presents a user-triggered
UAV dispatching system that integrates mobile information collection, ground station mission management, and
UAV search into a single workflow. In our system, a mobile application is designed to continuously track user records and synchronize these records to the cloud database, thereby initiating all subsequent search events and
UAV-dispatch missions. A control platform is developed to retrieve records from the cloud database for operator verification and mission scheduling. A
UAV prototype is implemented to execute the approved mission, follow waypoints, report status, relay phone records when available, and record downward-facing
video for later inspection. Simulations are conducted for
UAV-dispatching missions using a terrain-following flight altitude of $80$~m above ground level and demonstrate successful route generation, ordered waypoint traversal, and return to the launch site for landing of the
UAV dispatching. Local software tests verified timeout detection with a 1s threshold and phone-record transfer from the mobile application through the
UAV to the ground station. Outdoor trials showed that the
UAV could fly autonomously and capture downward-facing imagery in which a person, a nearby footpath, and surrounding vegetation were clearly discernible, providing visual cues for operator assessment and further inspection of the search area.
Read the full abstract ▾
All authors Mingyang Wang, Yi Hong, Tungchak Lee, Ashwin Sundar, Kevin Hung, Qubeijian Wang, Yalin Liu
Audio and Speech Processing
Towards Automated Clinical Behavioral Coding with Large Language Models: A Case study Using BOSCC recordings of Children
Jordan Prescott, Aditya Kommineni, Tiantian Feng et al. (+8 more)
Autism spectrum disorder (ASD) is a neurodevelopmental condition characterized by differences in social communication and by restricted interests and repetitive behaviors. Treatment interventions often target social-communication skills, creating a need for reliable measures of behavioral change. The Brief Observation of Social Communication Change (BOSCC) is a validated treatment-response measure based on brief play and social-communication interactions between a child and trained examiner. The BOSCC coding process is resource-intensive and requires trained experts, motivating the automation of coding in order to improve scalability and accessibility. In this work, we evaluate general-purpose large
language models (LLMs) for predicting
speech-related BOSCC codes from different input representations. We compare transcript, diarized-transcript, and targeted audio conditions across 163 in-house recordings. LLMs are able to perform well in assessing verbal exchange, but do not perform as well when identifying atypical
speech patterns. Additionally, performance varies considerably across scoring decisions, with no consistent pattern across diagnosis groups. An audit of model predictions indicates that applying the BOSCC coding criteria and interpreting ambiguous
speech evidence remain challenges.
Read the full abstract ▾
All authors Jordan Prescott, Aditya Kommineni, Tiantian Feng, Megan Micheletti, Alyssa Viggiano, Luis Angeles, Anfeng Xu, Lynn Perry, Catherine Lord, Daniel Messinger, Shrikanth Narayanan
Image and Video Processing · Cross-listed from cs.CV
Skeleton-Guided Progressive Test-Time Adaptation for Thin Curvilinear Structures
Boa Jang, JunGyu Lee, Gwanho Lee et al. (+2 more)
Accurate segmentation of thin curvilinear structures is vital for various real-world applications, from vessel analysis to road extraction. Yet their intricate geometry makes even minor pixel-wise errors enough to break the global
topology, and this structural fragility turns severe domain shifts into catastrophic failures. The difficulty is most acute under cross-modality gaps, where the imaging process itself differs fundamentally between source and target. While test-time adaptation (TTA) offers a practical source-free remedy, existing methods adapt feature statistics and confidence, neither of which constrains connectivity, and thus degrade under such extreme gaps. To address this, we propose Skeleton-Guided Progressive Test-Time Adaptation (SGP-TTA). Progressive Batch Normalization (ProgBN) shifts normalization from frozen source statistics toward current target estimates under a sample-count schedule, so that the source-target balance follows the stage of adaptation rather than a fixed coefficient. Consensus Skeleton Recall (CSR) then derives a structural target from geometrically aligned multi-view predictions and updates only the BN affine parameters to preserve connected structures. Extensive experiments show that SGP-TTA consistently outperforms existing TTA methods in topological connectivity, with the largest margins under cross-modality shift. The project page is available at https://boa-jang.github.io/SGP-TTA.
Read the full abstract ▾
All authors Boa Jang, JunGyu Lee, Gwanho Lee, Jinwook Choi, Young-Gon Kim
Audio and Speech Processing · Cross-listed from cs.SD
Cross-Lingual Speaker Verification with Self-Supervised Pre-Trained Models
Jinghan Peng, Yu Zheng, Weiqiang Wang et al. (+1 more)
Speaker verification (SV) performance degrades under
language mismatch due to the
entanglement of speaker identity with
language-specific acoustic cues. To address this problem, we leverage large-scale self-supervised pre-trained models (PTMs) to learn
language-agnostic speaker representations. We utilize PTMs as robust front-end feature extractors, capitalizing on their rich acoustic and linguistic knowledge acquired from vast, diverse audio data. These generalized features are then used to train a downstream speaker
embedding network, effectively disentangling speaker identity from
language-specific characteristics. We validate our approach on the TidyVoice2026
benchmark, which
benchmarks SV under
language mismatch. Our proposed system (team T02) achieves equal error rates (EERs) of 2.21% on tv26_eval-A and 2.99% on tv26_eval-U.
Read the full abstract ▾
All authors Jinghan Peng, Yu Zheng, Weiqiang Wang, Jian Liu
Systems and Control
A Latent Space Optimization Approach for Symbolic Discovery of Dynamical Models
Tongjia Liu and Ilias Mitrai
In this paper, we propose a hybrid latent-space
Bayesian Optimization (BO) and global optimization framework for solving symbolic regression tasks to discover dynamical models from data. Symbolic regression can discover equations from data without fixing the functional form of the expression a priori. The proposed framework uses the fact that if the functional form of the expression is fixed, the symbolic regression task reduces to a parameter estimation problem, which must be solved to global optimality. Motivated by this structure, first, we train a VAE to map the discrete space of expression trees into a continuous latent space. Then, we use BO to search this latent space while assessing a candidate expression's prediction error by solving the parameter estimation problem to global optimality. We evaluate our framework across two case studies: static reaction rate law discovery and dynamic concentration identification in a continuous stirred-tank reactor (CSTR). The results show that our framework identifies the true governing equations faster than MINLP formulations without solver timeouts and achieves higher sample efficiency than evolutionary baselines under tight evaluation budgets.
Read the full abstract ▾
Image and Video Processing
Adapting Appearance-Based Gaze Estimation to Narrow-Range, Long-Duration Screen Viewing
Jordan Prescott, Kleanthis Avramidis and Shrikanth Narayanan
Appearance-based gaze estimation offers a low-cost alternative to infrared eye tracking for screen-based behavioral and clinical applications, however existing models are typically developed for wide ranges of gaze angle and head pose. Prolonged screen viewing presents a distinct regime in which gaze remains near the screen center, head motion is limited, and calibration drift accumulates over time. In this work, we
benchmark six published
estimators along with a proposed differential-gaze model on 140 long-duration facial
video recordings with synchronized eye tracking under subject-disjoint evaluation and a common budget of calibration frames. Differential estimation was the only static method that significantly improved upon a baseline predictor of each recording's mean gaze, and was further boosted by addition of temporal context. It also yielded the strongest agreement with reference fixations and saccades, demonstrating that low angular error alone could not validate eye movement reconstruction. These findings establish differential estimation as a promising foundation for reliable gaze tracking in long-duration, narrow-range settings.
Read the full abstract ▾