Audio and Speech Processing · Cross-listed from cs.CV
AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs
Yaoting Wang, Ziyi Zhang, Wenming Tu et al. (+10 more)
Recent advances in Omni-
Multimodal Large
Language Models (Omni-MLLMs) have enabled strong integration of
vision, audio, and
language. However, their audio-visual intelligence (AVI) remains insufficiently evaluated due to the lack of systematic and comprehensive
benchmarks. We introduce AVI-Bench, a cognitively inspired
benchmark that evaluates Omni-MLLMs across three stages, perception, understanding, and
reasoning, through cross-modal tasks requiring joint audio-visual interpretation. This design enables fine-grained diagnosis of model capabilities and failure modes. To further assess robustness beyond familiar domains, we propose AVI-Bench-PriSe, an extension that probes models' primitive audio-visual sensation using unfamiliar, low-semantic stimuli, testing generalization beyond common training distributions. Extensive experiments on both open-source and closed-source models reveal substantial limitations in current Omni-MLLMs. Based on these findings, we present a four-level AVI taxonomy. Overall, AVI-Bench provides a principled evaluation framework to guide the development of more robust and generalizable AVI. Project website: https://fudancvl.github.io/AVI-Bench/
Read the full abstract ▾
All authors Yaoting Wang, Ziyi Zhang, Wenming Tu, Shaoxuan Xu, Wenjie Du, Cheng Liang, Weijun Wang, Yuanchao Li, Guangyao Li, Hao Fei, Yuanchun Li, Henghui Ding, Yunxin Liu
Signal Processing
Gravity-Aware Hierarchical Routing for Lightweight SensorLLM on Human Activity Recognition
Hao Li, Mingrui Zheng, Yasuyuki Tahara et al. (+1 more)
Recent studies on sensor-
language alignment have shown that two-stage frameworks can improve the semantic modeling ability of wearable-sensor human activity recognition (HAR), where SensorLLM-style methods first perform motion-to-
language alignment and then fine-tune the model for downstream tasks. However, our experiments reveal a consistent failure mode when the Stage 2 backbone is compressed to a compact model such as TinyLlama: recognition of dynamic activities remains relatively strong, while the discrimination of low-motion static classes such as standing, sitting, and lying degrades substantially. To address this issue, we propose a
gravity-aware hierarchical routing head as a lightweight post-
alignment adaptation built on top of an already aligned model, rather than a new large-scale pretraining framework. The method uses the per-channel mean and std from the Chronos tokenizer state to extract statistical cues related to posture and
gravity direction, and adaptively combines a static expert and a full expert through soft routing, together with a load-balancing loss for stable training. On the MHealth
dataset, this design significantly improves macro-F1 with minimal parameter overhead, and the gains are concentrated mainly on static classes while preserving strong performance on dynamic activities. As a first arXiv disclosure, the current paper reports results on a single
dataset only, with the goal of highlighting the core method and laying the groundwork for broader evaluation in future work.
Read the full abstract ▾
All authors Hao Li, Mingrui Zheng, Yasuyuki Tahara, Yuichi Sei
Image and Video Processing · Cross-listed from cs.CV
Hand Trajectory Fusion for Egocentric Natural Language Query Grounding
Enmin Zhong, Carlos R. del-Blanco, Fernando Jaureguizar et al. (+1 more)
Egocentric Natural
Language Query (NLQ) grounding asks a model to localize, in a long first-person
video, the temporal interval that answers a free-form text query. Existing methods fuse
video appearance with the query but ignore hand motion, despite the fact that roughly 41% of Ego4D NLQ queries are answered at a moment of hand–object manipulation or their immediate outcomes.We propose a hand-trajectory encoder for converting a sequence of hand skeletons into highly-semantic hand kinematic features, which are then aligned and combined with pretrained
video–text features through a cross-
attention fusion strategy with adaptive gating. On the Ego4D NLQ v2 validation split, the clearest gains appear for Hand-Object Interaction queries (+2.54 R1@IoU=0.3) and Quantity/State queries (+4.32 R1@IoU=0.3), indicating that hand trajectory provides grounding cues beyond appearance alone.
Read the full abstract ▾
All authors Enmin Zhong, Carlos R. del-Blanco, Fernando Jaureguizar, Narciso García
Image and Video Processing
AtlasGS: Brain MRI Spatial Resolution Harmonization With Shared Gaussian Geometry
Yifan Gao, Peiran Xu, Yimeng He et al. (+5 more)
Splatting (GS)-based shared geometry framework adopts a two-stage training strategy, in which an explicit, subject-specific Gaussian scaffold encoding anatomical geometry is first learned from the isotropic structural scan and then reused to fit appearance for target modalities acquired with sparse slices. Experiments on the UK Biobank, GBM, and ABCD
datasets for through-plane super-resolution across multiple modalities (T2-weighted, FLAIR, DWI, ASL), degradation factors ($\times 3$, $\times 5$, $\times 7$), and pathological abnormalities (glioblastoma) demonstrate state-of-the-art reconstruction fidelity. The shared Gaussian geometry enables arbitrary-view generation for target modalities with strong structural consistency and further shows potential for self-supervised in-plane super-resolution. This work establishes explicit geometry-guided representations as a novel, flexible, and interpretable pathway toward retrospective multi-contrast MRI harmonization and reliable clinical reference construction. Source code is available at: https://github.com/yfgao76/AtlasGS
Read the full abstract ▾
All authors Yifan Gao, Peiran Xu, Yimeng He, Haoran Li, Ziyang Long, Yufeng Wang, Ju Dong Yang, Debiao Li
Systems and Control
Power System CBFs
Abdallah Alalem B. Albustami, Ahmad F. Taha and Taylor T. Johnson
Control barrier functions (CBFs) have become a standard tool in safety critical-control systems. CBFs convert state constraints into real time control conditions that certify forward invariance (meaning that once the system starts in a safe region, it remains there for all future times) and minimally modify a nominal controller only when safety is at risk. In power systems, CBF based methods have been proposed for frequency and voltage safety, but they largely remain disconnected from three key features that are central to power system operation: differential algebraic equation (DAE) models that capture network power flow constraints, safety specifications involving algebraic variables such as bus voltages, and formal verification of the resulting closed loop system. This paper closes this gap by developing a CBF framework for power system DAE models that supports safety constraints on both dynamic and algebraic variables. The framework provides real time safety filtering through an optimization layer that wraps around an existing controller and minimally modifies its command to enforce safety. In addition, it provides formal verification (i.e., a mathematical guarantee that all admissible trajectories satisfy the prescribed safety constraints) through an offline reachability based certificate of safe operation. The result is a unified filter and verify methodology for enforcing and certifying frequency and voltage safety in power systems while preserving the DAE structure of the underlying model.
Read the full abstract ▾
Signal Processing · Cross-listed from cs.LG
ERP-XTTN: Interpretable Prototype-Guided Cross-Attention for Cross-Subject ERP Classification
Charlotte Genevier Wyman and Leanne Hirshfield
Interpretable
brain-computer interface classifiers that generalize across subjects without calibration remain an open challenge. We evaluated whether prototype-based cross-
attention can provide competitive, inherently interpretable ERP classification across paradigms under deployment-compatible conditions. We propose ERP-XTTN (ERP Cross-
Attention), a cross-
attention architecture that routes input
EEG peaks to fixed difference-wave prototypes via query-key-only cross-
attention with no value projection. Classification is based directly on prototype similarity and a separate measure of component amplitude, so that prototype content contributes to every decision by construction. Prototypes are derived automatically from prominent extrema in the training-fold grand-average difference wave. We evaluated across three public sources (BNCI Horizon 2020, HRI Cursor, and ERP CORE) encompassing eight ERP components (ERN, LRP, ErrP, N170, P300, N2pc, MMN, N400). Evaluations used LOSO cross-validation with causal filtering at a three-channel montage, compared against EEGNet,
EEG-Deformer, EPMN, and xDAWN with Riemannian geometry. The mean performance gap between the best baseline and ERP-XTTN was 0.025 AUROC. Prototype interventions confirmed that decisions depend on prototype content rather than on the routing
attention pattern alone. False positives morphologically resembled true positives more than true negatives did, so classification errors are neurophysiologically explicable. ERP-XTTN generalizes across diverse ERP morphologies under causal, calibration-free conditions, while retaining competitive performance and decisions that depend directly on physiological prototype content. Unlike post-hoc explanation methods for black-box models, the basis of each decision is directly observable in the trained model. To our knowledge, this is the first epoch-level LOSO
benchmark on ERP CORE.
Read the full abstract ▾
Systems and Control · Cross-listed from math.OC
Koopman operator learning for predictive control via Khatri-Rao kernel regression
Mircea Lazar
This paper develops a data-driven realization of the generalized Koopman operator (GeKo) [1], in which states and inputs are lifted independently and the dynamics are expressed as a tensor bilinear system. The first contribution is a time-sequenced multi-step Khatri-Rao kernel regression formulation that exposes the operator to evolved snapshots along trajectories rather than only single one-step pairs, which reduces compounded prediction error. Secondly, we develop a kernel- and input-agnostic structured SVD reduction that compresses the lifted state and input spaces while preserving the Khatri-Rao realization. We instantiate the framework with random
Fourier features and describe a complete predictive-control pipeline, including a multi-step roll-out diagnostic that guides the choice of MPC horizon. The framework is validated on the chaotic Lorenz system, where the learned reduced-order GeKo model reduces the long-horizon prediction error by up to a factor of five relative to one-step and lifted linear baselines fitted on the same data, and stabilizes an unstable
equilibrium.
Read the full abstract ▾
Audio and Speech Processing
A Comparison of Generative and Discriminative Methods for Speech Enhancement: Robustness, Complexity, and Hallucination
Shrishti Saha Shetu, Emanuël A. P. Habets and Andreas Brendel
In this study, we conduct a comprehensive comparative analysis of generative and discriminative deep learning-based
speech enhancement methods, specifically in noise reduction tasks. Our investigation focuses on evaluating their effectiveness under high and low signal-to-noise ratio conditions, considering both matched and mismatched training scenarios. We further investigate the impact of training data volume, model convergence speed, and interpret the performance differences in terms of objective results for the considered training paradigms. Additionally, we compare the complexity-performance trade-off and the practical viability of these approaches. To further strengthen the evaluation, we study the
hallucination characteristics of generative approaches in terms of word error rate and phoneme similarity. The insights derived from this study provide empirical evidence to assist researchers and practitioners in understanding whether the perceptual gains of different approaches justify their computational cost in practical applications.
Read the full abstract ▾
Image and Video Processing
Depth from Dual Differential Defocus and Stereo Consensus
Junjie Luo, Wei Xu, Dylan Chu et al. (+2 more)
We introduce D^3S Consensus, a physics-based, closed-form
algorithm that unifies depth-from-defocus (DfD) and stereo to achieve highly accurate depth estimation throughout an extended working range beyond the depth-of-field (DoF) of cameras. Given a pair of dual-defocus stereo images, the method estimates an overdetermined set of depth using a novel DfD theory, Dual Differential Defocus (D^3), and (S)tereo in a coupled fashion. It then picks the most confident depth prediction from the set by enforcing consensus between these physically independent cues to reject unreliable estimates. Analysis shows that D^3S achieves a comparable working range under the same error tolerance with 10x smaller baseline than previous triangulation-based depth estimation systems. This enables compact passive binocular rangefinders with substantially smaller form factors than conventional stereo and DfD designs. We demonstrate the first D^3S prototype with only 4 mm baseline and 12 mm EFL. It generates up to 900 x 1800-pixel depth maps with 1-cm mean absolute error over 0.3-1.64 m from a snapshot acquisition. This has surpassed the reported accuracy of certain commercially available stereo cameras with much larger form factors.
Read the full abstract ▾
All authors Junjie Luo, Wei Xu, Dylan Chu, Emma Alexander, Qi Guo
Signal Processing
Global Unknown Estimation: A Statistical Framework for Wireless Distributed Learning
Yicheng Qu, Ali Bereyhi and Ben Liang
Over-the-air computation (AirComp) is widely used for model aggregation in wireless distributed learning. Although it enhances communication efficiency, we believe the AirComp aggregation has limited effectiveness due to the difference between its target problem and that of distributed learning. In this paper, we develop a rigorous formulation for optimal model aggregation in wireless distributed learning. Using this formulation, we show that AirComp aggregation generally assumes a mismatched statistical model for local parameters. We then propose a statistical framework for model aggregation, called global unknown estimation (GUE). It captures the statistical relation between the local and global model parameters, allowing to interpret model aggregation as an inference task. We validate the efficiency of GUE through numerical experiments. Our results show that, in the low SNR regime, GUE can reduce the required power for model aggregation by approximately 15 dB compared to AirComp aggregation. Remarkably, this gain is obtained without additional computational overhead
Read the full abstract ▾
Systems and Control · Cross-listed from quant-ph
Towards Efficient Synthesis of Quantum Graph States by Fusing Graph Motifs
Tingxiang Ji, Hansika Weerasena, Demitry Farfurnik et al. (+1 more)
Photonic graph states with advanced
topologies can enable measurement-based
quantum computing, distributed
quantum sensing, and
quantum interconnects. However, the efficient generation of photonic graph states is limited by the probabilistic nature of photonic entangling operations and the exponential dependence of generation rate on resource cost. In this work, we study photonic graph state synthesis as a cost-aware decomposition problem, exploiting local Clifford (LC) equivalence to identify more synthesis-friendly representations of the target graph state before decomposition. Specifically, we propose Cost-aware Fusion-based Decomposition (CFD), a three-stage heuristic framework that decomposes a target graph state into ring,
star, and linear motifs, and assembles them via Type-I fusion operations to minimize fusion overhead and physical-
qubit consumption. We further show that selecting the LC-equivalent graph state with the minimum number of edges provides a highly effective proxy for near-optimal synthesis: in many cases it matches the best generation rate observed within the LC equivalence class under CFD, and in most remaining cases it remains close to it. Numerical evaluations on graph state orbit data and 2D and 3D
lattice graph states demonstrate that CFD achieves up to 84.6% reduction in resource overhead compared to baseline constructions, and yields improvements in photonic generation rate spanning multiple orders of magnitude. These results suggest that combining structure-aware motif decomposition with LC equivalence is a practical and scalable strategy for photonic graph state synthesis.
Read the full abstract ▾
All authors Tingxiang Ji, Hansika Weerasena, Demitry Farfurnik, Jianqing Liu
Systems and Control
Package-Embedded Coupled Inductor Arrays for High-Performance Computing Power Delivery
Rami Rasheedi, Salma Abdelzaher and Inna Partin-Vaisband
A novel power delivery framework, comprising a package-embedded inductor
topology and an inductance-island methodology, is introduced to maximize both inductance and current densities in vertical power delivery (VPD). The framework leverages multiple multi-phase converters, a common strategy in high-performance computing systems, to enhance efficiency and scalability. The proposed
topology employs an array of tightly coupled spiral square inductors sharing a common
magnetic rod, serving multiple converters operating in the same conversion phase. The array is optimized to maximize coupling and minimize conversion losses, achieving superior inductance and current densities of 250 nH/mm^2 and 10 A/mm^2, respectively. At the system level, the inductance-island methodology partitions the power delivery network into multiple islands, each dedicated to a converter phase and supplying a portion of the load current, thereby enabling scalable and efficient distribution. To validate the framework, the inductor array is designed and simulated in ANSYS Maxwell 3D and Mechanical, exhibiting an average quality factor of 23.6 and efficiency of 97.4% at 2 A load current, 6 V input, and 10 MHz switching frequency. The inductor array netlist is extracted from ANSYS and co-designed in Cadence Virtuoso with a distributed dual-phase power conversion system, ensuring joint optimization of passive and active components. The co-designed converter achieves a significant efficiency gain of 5.65% on average and up to 11.04% at 40 A load over a similar converter with uncoupled inductors, demonstrating the practical benefits of the approach.
Read the full abstract ▾
Systems and Control
Terminal Time and Angle-Constrained Nonlinear Intercept Guidance
Shivam Bajpai and Abhinav Sinha
This paper considers the problem of simultaneously controlling an interceptor's impact time and impact angle using its lateral acceleration as the sole control input. With a single control input, the nonlinear engagement kinematics is inherently underactuated, which complicates guidance law synthesis. To overcome this challenge, a hierarchical sliding mode-based guidance law is developed to concurrently regulate the two terminal constraints. The proposed architecture consists of a two-layer sliding
manifold. The first layer comprises two sub-sliding surfaces corresponding to the impact time and impact angle error dynamics, respectively, while the second layer introduces a composite sliding
manifold that combines the two individual sub-surfaces. Then, a variable-gain adaptive guidance law is designed to ensure time and angle-constrained interception against a stationary target, which is further extended to intercept a constant velocity target. Simulations are conducted for various engagement scenarios to attest to the efficacy of the proposed approach.
Read the full abstract ▾
Systems and Control
Closed-Form PI and PID Tuning of All-Pole Plants up to Third Order for Monotonic Minimum-Settling Step Responses
Senol Gulgonul
A unified, closed-form analytical PI/PID tuning method is presented for all-pole plants up to third order that yields a strictly monotonic (zero-overshoot) step response with minimum settling time. The design target is the binomial closed loop p^n/(s+p)^n, which is monotonic with robustness depending only on the order n. Because a fixed PI/PID cannot assign the closed-loop poles and the controller zeros independently, realizing this target exactly requires the controller zeros to be cancelled, which forces the controller numerator to divide the plant denominator. It follows that an exact, real-gained solution exists for any stable plant precisely up to second order with a PI controller and third order with a PID controller; beyond that the residual binomial factor acquires a complex pair of damping sqrt(3)/2, which a generic plant does not contain. Explicit gains are derived for first-order plants (PI), second-order plants with real and complex poles (PI and PID), and third-order plants with three real poles or one real pole plus a complex pair (PID). The freedom of the coincident designs is shown to be bounded: a quadratic nonnegativity condition gives the exact window of the design pole for strict monotonicity, which collapses at the pole-ratio-2 changeover for real poles and is nonempty for damping ratios above approximately 0.443 for complex poles. Monotonicity guarantees Mt = 1, hence Ms <= 2, phase margin >= 60 degrees, and gain margin >= 6 dB, tightening to universal constants for the binomial family. Load-disturbance attenuation obeys IAEd = 1/Ki, making the cost of cancellation explicit, and comparisons with SIMC, the CHR zero-overshoot rule, and deadbeat-fitted explicit formulas quantify the trade: at matched maximum sensitivity the proposed design settles faster than SIMC on the third-order example, with markedly lower controller gains and peak control effort.
Read the full abstract ▾
Systems and Control
Fairness as an Investment: Dynamic Participation and Long-Run Profit in Virtual Power Plants
Liudong Chen and Bolun Xu
We show that incorporating
fairness constraints into virtual power plant (VPP) operations can incentivize consumer participation and thus improve the aggregator's long-run profitability. VPPs rely on sustained participation from heterogeneous consumers to provide a variety of grid services whose timing and frequency are often uncertain. As a result, consumers' willingness and ability to provide flexibility evolve over time, creating a dynamic link between past participation and future resource availability. We develop a dynamic aggregation framework to study how
fairness in service allocation affects future participation and long-run profitability. By linking current dispatch decisions to future resource availability, we show that fairer allocations can strengthen consumer engagement, expand aggregate availability, and create additional value during high-price and high-demand events. To balance
fairness and operational efficiency, we introduce a slack-augmented allocation mechanism that preserves most of the participation benefits from
fairness while avoiding unnecessary reductions in service procurement. We derive conditions under which the resulting availability gains outweigh the short-run cost of redistribution and validate the approach using real-world consumer behavior and electricity
market data from Norway.
Read the full abstract ▾
Signal Processing
Short-Acquisition Contrast-Free Super-Resolution Microvascular Imaging in Rabbit Kidney
Zhengchang Kou, Yuning Zhao, Mingrui Liu et al. (+2 more)
Ultrasound localization microscopy (ULM) enables micrometer-scale microvascular imaging by localizing and tracking intravascular microbubbles, but its dependence on exogenous contrast
agents and long acquisition times limits clinical translation. This study presents a high-frame-rate contrast-free super-resolution ultrasound microvascular imaging method based on high-frequency ultrafast ultrasound and nonlinear beamforming of backscatter signals from native blood flow. Using only 125 milliseconds of in vivo ultrafast data per image, the proposed method achieved an imaging frame rate of 8 frames/s in a rabbit kidney model. The reconstructed microvascular images resolved vessels with a global spatial resolution of 22.2 um over a field of view of 23.04 x 15.18 mm2, where the wavelength of ultrasound was 67.5 um. This corresponds to a three-fold improvement over conventional power Doppler imaging under the same acquisition duration. Compared with conventional flow imaging, the proposed method provided improved microvascular contrast and finer vessel delineation without microbubble injection. These results demonstrate a practical pathway toward high frame rate, contrast-free super-resolution ultrasound imaging for microvascular assessment.
Read the full abstract ▾
All authors Zhengchang Kou, Yuning Zhao, Mingrui Liu, Rita J. Miller, Michael L. Oelze
Signal Processing
A data-driven filter bank framework for IMU-based heave motion estimation
Aybars Tokta
In this study, we address the IMU-based heave motion estimation problem for inertial
navigation systems. Unlike existing approaches, we propose a data-driven framework in which a bank of IIR filters, each associated with a specific frequency range, is optimized using a synthetically generated
dataset of realistic heave-acceleration tuples. The synthetic heave signal generation pipeline starts by synthesizing random wave signals from established wave energy spectra and then processing them through heave response amplitude operators reported in the literature. The corresponding vertical acceleration measurements are obtained by double-differentiating the heave signals and corrupting them with realistic low- and high-frequency disturbances observed in real IMU recordings. A
Fourier-transform-based method is used to estimate the mean peak period and select the appropriate filter. Simulation results from both offline and real-time tests demonstrate that the proposed method is robust to varying sea regimes and provides accurate heave estimation, with a maximum RMSE not exceeding the larger of 5 cm or 5% of the significant heave height.
Read the full abstract ▾
Systems and Control
Corridor Design and Separation Definition in Advanced Air Mobility: Systematic Literature Review
Evgenii Vinogradov, Debashisha Mishra, Mariam Ali Askar Alobeidli et al. (+4 more)
Advanced Air Mobility (AAM) uses electric vertical take-off and landing (eVTOL) vehicles to address urban congestion and emissions. However, corridor design, operation management, and separation standards remain underexamined for safe high-density operations. This paper applies the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to systematically review relevant literature from IEEE Xplore and Web of Science, focusing on publications from 2010 to 2024. A Context, Intervention, Mechanism, and Outcome (CIMO) framework guided the development of research questions. After screening 2,039 journal and conference papers, 62 articles met the inclusion criteria. The findings reveal a lack of integrated corridor design approaches, limited operational strategies, and reliance on standards originally designed for conventional aviation. A unified corridor design and separation definition frameworks and taxonomies are proposed to address these shortcomings, informing future investigations and operational frameworks for safe, efficient eVTOL operation deployment in urban settings.
Read the full abstract ▾
All authors Evgenii Vinogradov, Debashisha Mishra, Mariam Ali Askar Alobeidli, Jamal Khaled Al Ali, Ahmed Saleh Alshehhi, Jennifer Simonjan, Enrico Natalizio
Audio and Speech Processing · Cross-listed from cs.SD
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Hui Li, Yangfan Gao, Junlin Shang et al. (+4 more)
Audio tokenizers serve as the discrete interface between continuous audio and Audio
Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation. Reconstruction-oriented codecs preserve acoustic fidelity but lack rich semantics, while semantic-aware tokenizers typically rely on separate semantic and acoustic streams, introducing redundancy or misalignment. We propose EntangleCodec, a unified discrete audio tokenizer that learns caption-aligned semantic-acoustic representations before quantization. By aligning audio with rich captions rather than ASR transcripts, EntangleCodec captures linguistic content, speaker identity,
emotion, prosody, and acoustic scenes within a compact
token stream. A flow-matching
diffusion decoder further enables high-quality reconstruction across
speech, music, and general audio. EntangleCodec achieves reconstruction quality competitive with specialized codecs, outperforms all codec-based baselines on audio understanding by up to +7.4% on MMAR, and supports both TTS and TTA generation in a unified framework. Furthermore, EntangleCodec-based audio
language models demonstrate strong scaling behavior: even at 0.6B parameters, the model surpasses specialized continuous-representation LLMs with over 13B parameters across three
benchmarks using \textbf{22$\times$} fewer parameters; scaling to 8B further establishes new state-of-the-art results on MMAR, highlighting that representation quality is as critical as model scale in audio
language modeling. Code and model weights are available at https://github.com/luckyerr/EntangleCodec.
Read the full abstract ▾
All authors Hui Li, Yangfan Gao, Junlin Shang, Changhao Jiang, Tao Gui, Qi Zhang, Xuanjing Huang
Audio and Speech Processing · Cross-listed from cs.LG
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang et al. (+2 more)
Multimodal systems often benefit from combining information across
language, sound, and visual streams, but this benefit is not guaranteed. A modality that is useful for one input may become distracting for another, and local feature responses within the same modality can disagree with evidence from other sources. This work investigates how to adjust
multimodal representations before they are merged by a downstream predictor. We develop a compact calibration module that compares each modality with the others at the summary level, extracts cues of cross-source support and conflict, and converts these cues into instance-wise and dimension-wise modulation signals. The calibration is applied to the original modality features rather than to already fused representations, enabling the model to suppress misleading components, preserve weak but useful evidence, and emphasize responses that are better supported by the current
multimodal context. The module is designed as a plug-in component and can be attached to different fusion backbones without changing their prediction heads. Across five
benchmarks covering sentiment understanding, action recognition, audio-visual event detection, and audio-visual
emotion classification, the proposed pre-combination calibration strategy improves performance under both sequence-based and convolutional fusion settings. Additional analyses under modality removal, synthetic corruption, training dynamics, and feature-level visualization show that calibrating signals before fusion can reduce interference from unreliable modalities and produce more stable
multimodal optimization.
Read the full abstract ▾
All authors Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang, Xinpei Wang, Weitong Chen
Image and Video Processing
Learning to Refine: Spectral-Decoupled Iterative Refinement Framework for Precipitation Nowcasting
Yunlong Zhou, Chen Zhao, Danyang Peng et al. (+2 more)
Accurate precipitation nowcasting is vital for disaster mitigation, but deep learning methods face a key trade-off: regression models produce over-smoothed, spectrally decaying predictions that blur convective details and violate
turbulence power laws;
diffusion models generate realistic yet unanchored
hallucinations lacking physical grounding. We propose Spectral-Decoupled Iterative Refinement (SDIR), a deterministic framework that reformulates nowcasting as progressive frequency-decoupled refinement. SDIR first extracts a stable low-frequency synoptic skeleton, then iteratively refines high-frequency textures under physical constraints, eliminating both blurring and
hallucinations. It features a dual-path design: the Synoptic Frequency-Guided Former (SFG-Former) with Scale-Adaptive
Transformers for global structure, and the
Fourier Residual Refiner (FR-Refiner) with Scale-Conditioned
Fourier Neural Operators for fine residuals. A Physically Consistent Power Spectral Density (PCPSD) loss with dynamic masking enforces a
turbulence-consistent spectral distribution. Experiments on three
benchmarks show SDIR significantly outperforms SOTA methods in spatial accuracy while achieving spectral fidelity competitive with
diffusion-based methods, enabling reliable high-resolution operational nowcasting. Code link: https://github.com/RuntimeWarning/SDIR.
Read the full abstract ▾
All authors Yunlong Zhou, Chen Zhao, Danyang Peng, Fanfan Ji, Xiao-Tong Yuan
Systems and Control · Cross-listed from cs.RO
Permissive Safety Through Trusted Inference: Verifiable Belief-Space Neural Safety Filters for Assured Interactive Robotics
Haimin Hu
Autonomous
robots that interact with people must make safe and efficient decisions under human-induced uncertainty, such as their preferences, goals, competency, and willingness to cooperate. Safety filters are a popular approach for ensuring safety in interactive robotics, since their modular design separates safety from performance, allowing
robots to operate safely around people with minimal impact on task efficiency. While traditional safety filters typically operate only in the physical space, neglecting the
robot's ability to learn and adapt online, the recently proposed belief-space safety filter (BeliefSF) reasons about
robot safety in closed-loop with runtime inference that actively reduces the
robot's uncertainty online, thereby reducing conservativeness in filtering. However, providing formal safety guarantees for
robots deploying BeliefSF remains a significant challenge due to errors in runtime inference and neural approximation of safety filters required to handle the high dimensionality of belief spaces. In this paper, we propose an algorithmic approach to certify high-probability safety of BeliefSF using conformal prediction, while explicitly accounting for the reliability of the
robot's runtime inference module. Our method leverages the structure of belief-space safety filtering by focusing verification on a region where inference is expected to be reliable. It preserves the simplicity and sample complexity of standard conformal prediction, yet can certify a substantially less conservative safety filter. Through a simulated human-vehicle interaction
benchmark, we show that our approach verifies a significantly more permissive belief-space safety filter than a standard conformal prediction baseline.
Read the full abstract ▾