Audio and Speech Processing · Cross-listed from cs.CL
Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding
Yuchen Wang, Haonan Wang, Yu Guo et al. (+2 more)
Decoding natural
language from non-invasive
EEG signals is a promising yet challenging task. However, current state-of-the-art models remain constrained by three fundamental issues: Semantic Bias, where outputs collapse into generic linguistic templates; Signal Neglect, where models rely heavily on LLM priors to hallucinate fluent text even in the absence of meaningful signals; and the "BLEU Trap", where high-frequency stopwords inflate n-gram metrics, masking a lack of true semantic fidelity. To resolve these challenges, we move beyond conventional end-to-end pipelines and propose SemKey, a novel multi-stage framework that enforces signal-grounded generation through four decoupled semantic objectives: sentiment, topic, length, and surprisal. We extract these semantic anchors from
EEG embeddings directly, then unify them with an Active
Retrieval Decoding mechanism, compelling the LLM to ground its
token generation in the neural signals rather than defaulting to linguistic priors. Furthermore, we break the BLEU Trap by establishing a comprehensive evaluation protocol using rigorous
retrieval and distribution-based metrics such as Fréchet Distance. Extensive experiments demonstrate that SemKey effectively mitigates
hallucinations on noise inputs and achieves SOTA performance on these robust protocols. Code will be released upon acceptance at https://github.com/xmed-lab/SemKey.
Read the full abstract ▾
All authors Yuchen Wang, Haonan Wang, Yu Guo, Honglong Yang, Xiaomeng Li
Signal Processing
ZUNA: Flexible EEG Superresolution with Position-Aware Diffusion Autoencoders
Christopher Warner, Jonas Mago, JR Huml et al. (+2 more)
We present ZUNA, a 380M-parameter masked
diffusion autoencoder trained to perform masked channel infilling and superresolution for arbitrary electrode numbers and positions in
EEG signals. The ZUNA architecture tokenizes multichannel
EEG into short temporal windows and injects spatiotemporal structure via a 4D rotary positional encoding over (x,y,z,t), enabling inference on arbitrary channel subsets and positions. We train ZUNA on an aggregated and harmonized corpus spanning 208 public
datasets containing approximately 2 million channel-hours using a combined reconstruction and heavy channel-dropout objective. We show that ZUNA substantially improves over ubiquitous spherical-spline interpolation methods, with the gap widening at higher dropout rates. Crucially, compared to other deep learning methods in this space, ZUNA's performance generalizes across
datasets and channel positions allowing it to be applied directly to novel
datasets and problems. Despite its generative capabilities, ZUNA remains computationally practical for deployment. We release Apache-2.0 weights and an MNE-compatible preprocessing/inference stack to encourage reproducible comparisons and downstream use in
EEG analysis pipelines.
Read the full abstract ▾
All authors Christopher Warner, Jonas Mago, JR Huml, Mohamed Osman, Beren Millidge
Signal Processing
Wavenumber-domain signal processing for holographic MIMO: Foundations, methods, and future directions
Zijian Zhang and Linglong Dai
Holographic multiple-input multiple-output (H-
MIMO) systems represent a paradigm shift in wireless communications by enabling quasi-continuous apertures. Unlike conventional
MIMO systems, H-
MIMO with subwavelength
antenna spacing operates in both far-field and near-field regimes, where classical discrete
Fourier transform (DFT) representations fail to sufficiently capture the channel characteristics. To address this challenge, this article provides an overview of the emerging wavenumber-domain signal processing framework. Specifically, by leveraging spatial
Fourier plane-wave decomposition to model H-
MIMO channels, the wavenumber domain offers a unified and physically consistent basis for characterizing subwavelength-level spatial correlation and spherical wave propagation. This article first introduces the concept of H-
MIMO and the wavenumber representation of H-
MIMO channels. Next, it elaborates on wavenumber-domain signal processing technologies reported in the literature, including multiplexing, channel estimation, and waveform designs. Finally, it highlights open challenges and outlines future research directions in wavenumber-domain signal processing for next-generation wireless systems.
Read the full abstract ▾
Image and Video Processing · Cross-listed from cs.CV
Spectral Collapse in Diffusion Inversion
Nicolas Bourriez, Alexandre Verine and Auguste Genovesio
Conditional
diffusion inversion provides a powerful framework for unpaired image-to-image translation. However, we demonstrate through an extensive analysis that standard deterministic inversion (e.g. DDIM) fails when the source domain is spectrally sparse compared to the target domain (e.g., super-resolution, sketch-to-image). In these contexts, the recovered latent from the input does not follow the expected isotropic Gaussian distribution. Instead it exhibits a signal with lower frequencies, locking target sampling to oversmoothed and texture-poor generations. We term this phenomenon spectral collapse. We observe that
stochastic alternatives attempting to restore the noise variance tend to break the semantic link to the input, leading to structural drift. To resolve this structure-texture trade-off, we propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the ODE dynamics to enforce the theoretical Gaussian noise magnitude within the null-space of the structural gradient. Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) demonstrate that OVG effectively restores photorealistic textures while preserving structural fidelity.
Read the full abstract ▾
Image and Video Processing · Cross-listed from cs.CV
NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
Xiaoxu Peng, Dong Zhou, Jianwen Zhang et al. (+3 more)
Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to
adversarial threats. These risks range from localized physical patches to imperceptible global perturbations. Existing defense methods for VLMs remain limited and often fail to reconcile robustness with clean-sample performance. To bridge these gaps, we propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle. Specifically, we first employ NutNet++ as a sentinel, which is a unified detection-purification mechanism. It identifies benign samples, local patches, and global perturbations through three-way classification. Subsequently, localized threats are purified via efficient grayscale masking, while global perturbations trigger Expert-guided
Adversarial Prompt Tuning (EAPT). Instead of the costly parameter updates of full-model fine-tuning, EAPT generates "corrective driving
prompts" via gradient-based latent optimization and discrete projection. These
prompts refocus the VLM's
attention without requiring exhaustive full-model retraining. Evaluated on the Dolphins
benchmark, our NutVLM yields a 4.89% improvement in overall metrics (e.g., Accuracy,
Language Score, and GPT Score). These results validate NutVLM as a scalable security solution for intelligent transportation. Our code is available at https://github.com/PXX/NutVLM.
Read the full abstract ▾
All authors Xiaoxu Peng, Dong Zhou, Jianwen Zhang, Guanghui Sun, Anh Tu Ngo, Anupam Chattopadhyay
Image and Video Processing
Data-driven registration and modeling of brain deformation for image-guided neurosurgery
Tiago Assis, Colin P. Galvin, Joshua P. Castillo et al. (+12 more)
Accurate compensation of
brain deformation is critical for reliable image-guided neurosurgery.
Surgical manipulation and tumor resection induce tissue motion, causing preoperative planning images to become misaligned with the intraoperative anatomy. In this review, we examine data-driven methods developed between 2020 and 2025 for
brain deformation registration and modeling, with a particular focus on learning-based approaches. A comprehensive literature search was conducted in PubMed, IEEE Xplore, Scopus, and Web of Science using predefined inclusion and exclusion criteria for computational methods addressing
brain deformation in neurosurgical imaging, resulting in 46 eligible studies. We provide a unified analysis of methodological strategies, including deep learning-based image registration, direct deformation field regression, synthesis-driven
multimodal alignment, resection-aware architectures for handling missing correspondences, and hybrid models integrating biomechanical priors. We also examine
dataset utilization, evaluation metrics, validation protocols, and the assessment of uncertainty and generalization across studies. While learning-based methods demonstrate promising accuracy and computational efficiency, current approaches remain limited by out-of-distribution robustness, standardized benchmarking, interpretability, and readiness for clinical deployment. Our review highlights these gaps and outlines future directions toward more robust, generalizable, and clinically translatable solutions for neurosurgical guidance. By organizing recent advances and critically assessing evaluation practices, this work provides a comprehensive reference for researchers and clinicians working on data-driven registration and modeling of
brain deformation.
Read the full abstract ▾
All authors Tiago Assis, Colin P. Galvin, Joshua P. Castillo, Nazim Haouchine, Marta Kersten-Oertel, Zeyu Gao, Mireia Crispin-Ortuzar, Stephen J. Price, Thomas Santarius, Yangming Ou, Sarah Frisken, Nuno C. Garcia, Alexandra J. Golby, Reuben Dorent, Ines P. Machado
Audio and Speech Processing · Cross-listed from cs.SD
Gencho: Room Impulse Response Generation from Reverberant Speech and Text via Diffusion Transformers
Jackie Lin, Jiaqi Su, Nishit Anand et al. (+3 more)
Blind room impulse response (RIR) estimation is a core task for capturing and transferring acoustic properties; yet existing methods often suffer from limited modeling capability and degraded performance under unseen conditions. Moreover, emerging generative audio applications call for more flexible impulse response generation methods. We propose Gencho, a
diffusion-
transformer-based model that predicts complex spectrogram RIRs from reverberant
speech. A structure-aware encoder leverages isolation between early and late reflections to encode the input audio into a robust representation for conditioning, while the
diffusion decoder generates diverse and perceptually realistic impulse responses from it. Gencho integrates modularly with standard
speech processing pipelines for acoustic matching. Results show richer generated RIRs than non-generative baselines while maintaining strong performance in standard RIR metrics. We further demonstrate its application to text-conditioned RIR generation, highlighting Gencho's versatility for controllable acoustic simulation and generative audio tasks.
Read the full abstract ▾
All authors Jackie Lin, Jiaqi Su, Nishit Anand, Zeyu Jin, Minje Kim, Paris Smaragdis
Systems and Control
Real-time Load Current Monitoring of Overhead Lines Using GMR Sensors
Md Mahfuzur Rahman Chy, Md Rifat Al Amin Khan, Md Sultan Mahamud et al. (+2 more)
Non-contact current monitoring has emerged as a prominent research focus owing to its non-intrusive characteristics and low maintenance requirements. However, while they offer high sensitivity, contactless sensors necessitate sophisticated design methodologies and thorough experimental validation. In this study, a
Giant Magneto-Resistance (GMR) sensor is employed to monitor the instantaneous currents of a three-phase 400-volt overhead line, and its performance is evaluated against that of a conventional contact-based
Hall effect sensor. A mathematical framework is developed to calculate current from the measured
magnetic field signals. Furthermore, a MATLAB-based dashboard is implemented to enable real-time visualization of current measurements from both sensors under linear and non-linear load conditions. The GMR current sensor achieved a relative accuracy of 64.64% to 91.49%, with most phases above 80%. Identified improvements over this are possible, indicating that the sensing method has potential as a basis for calculating phase currents.
Read the full abstract ▾
All authors Md Mahfuzur Rahman Chy, Md Rifat Al Amin Khan, Md Sultan Mahamud, Anwarul Islam Sifat, Fiona J. Stevens McFadden
Signal Processing
AI-Driven Cardiorespiratory Signal Processing: Separation, Clustering, and Anomaly Detection
Yasaman Torabi
This research applies artificial intelligence (AI) to separate, cluster, and analyze cardiorespiratory sounds. We recorded a new
dataset (HLS-CMDS) and developed several AI models, including generative AI methods based on large
language models (LLMs) for guided separation, explainable AI (XAI) techniques to interpret latent representations, variational autoencoders (VAEs) for waveform separation, a chemistry-inspired non-negative
matrix factorization (NMF)
algorithm for clustering, and a
quantum convolutional neural network (QCNN) designed to detect abnormal physiological patterns. The performance of these AI models depends on the quality of the recorded signals. Therefore, this thesis also reviews the biosensing technologies used to capture biomedical data. It summarizes developments in microelectromechanical systems (MEMS) acoustic sensors and
quantum biosensors, such as
quantum dots and nitrogen-vacancy centers. It further outlines the transition from electronic integrated
circuits (EICs) to photonic integrated
circuits (PICs) and early progress toward integrated
quantum photonics (IQP) for chip-based biosensing. Together, these studies show how AI and next-generation sensors can support more intelligent diagnostic systems for future healthcare.
Read the full abstract ▾
Systems and Control
EExApp: GNN-Based Reinforcement Learning for Radio Unit Energy Optimization in 5G O-RAN
Jie Lu, Peihao Yan and Huacheng Zeng
With over 3.5 million 5G base stations deployed globally, their collective energy consumption (projected to exceed 131 TWh annually) raises significant concerns over both operational costs and environmental impacts. In this paper, we present EExAPP, a deep reinforcement learning (DRL)-based xApp for 5G Open
Radio Access Network (O-RAN) that jointly optimizes
radio unit (RU) sleep scheduling and distributed unit (DU) resource slicing. EExAPP uses a dual-actor-dual-critic Proximal Policy Optimization (PPO) architecture, with dedicated actor-critic pairs targeting energy efficiency and quality-of-service (QoS) compliance. A
transformer-based encoder enables scalable handling of variable user equipment (UE) populations by encoding all-UE observations into fixed-dimensional representations. To coordinate the two optimization objectives, a bipartite Graph
Attention Network (GAT) is used to modulate actor updates based on both critic outputs, enabling adaptive trade-offs between power savings and QoS. We have implemented EExAPP and deployed it on a real-world 5G O-RAN testbed with live traffic, commercial RU and smartphones. Extensive over-the-air experiments and ablation studies confirm that EExAPP significantly outperforms existing methods in reducing the energy consumption of RU while maintaining QoS. The source code is available at https://github.com/EExApp/EExApp.
Read the full abstract ▾
Systems and Control · Cross-listed from cs.CY
Genocide by Algorithm in Gaza: Artificial Intelligence, Countervailing Responsibility, and the Corruption of Public Discourse
Branislav Radeljic
The accelerating militarization of artificial intelligence has transformed the ethics, politics, and governance of warfare. This article interrogates how AI-driven targeting systems function as epistemic infrastructures that classify, legitimize, and execute violence, using Israel's conduct in Gaza as a paradigmatic case. Through the lens of responsibility, the article examines three interrelated dimensions: (a) political responsibility, exploring how states exploit AI to accelerate warfare while evading accountability; (b) professional responsibility, addressing the complicity of technologists, engineers, and defense contractors in the weaponization of data; and (c) personal responsibility, probing the moral agency of individuals who participate in or resist algorithmic governance. This is complemented by an examination of the position and influence of those participating in public discourse, whose narratives often obscure or normalize AI-enabled violence. The Gaza case reveals AI not as a neutral instrument but as an active participant in the reproduction of colonial hierarchies and the normalization of atrocity. Ultimately, the paper calls for a reframing of technological agency and accountability in the age of automated warfare. It concludes that confronting algorithmic violence demands a democratization of AI ethics, one that resists technocratic fatalism and centers the lived realities of those most affected by high-tech militarism.
Read the full abstract ▾
Signal Processing
Digital-Twin-Aided Dynamic Spectrum Sharing and Resource Management in Integrated Satellite-Terrestrial Networks
Hung Nguyen-Kha, Vu Nguyen Ha, Ti Nguyen et al. (+4 more)
The explosive growth in wireless service demand has prompted the
evolution of integrated satellite-terrestrial networks (ISTNs) to overcome the limitations of traditional terrestrial networks (TNs) in terms of coverage, spectrum efficiency, and deployment cost. Particularly, leveraging LEO satellites and dynamic spectrum sharing (DSS), ISTNs offer promising solutions but face significant challenges due to diverse terrestrial environments, user and satellite mobility, and long propagation LEO-to-ground distance. To address these challenges, digitial-twin (DT) has emerged as a promising technology to offer virtual replicas of real-world systems, facilitating prediction for resource management. In this work, we study a time-window-based DT-aided DSS framework for ISTNs, enabling joint long-term and short-term resource decisions to reduce system congestion. Based on that, two optimization problems are formulated, which aim to optimize resource management using DT information and to refine obtained solutions with actual real-time information, respectively. To efficiently solve these problems, we proposed
algorithms using compressed-sensing-based and successive convex approximation techniques. Simulation results using actual traffic data and the London 3D map demonstrate the superiority in terms of congestion minimization of our proposed
algorithms compared to
benchmarks. Additionally, it shows the adaptation ability and practical feasibility of our proposed solutions.
Read the full abstract ▾
All authors Hung Nguyen-Kha, Vu Nguyen Ha, Ti Nguyen, Eva Lagunas, Joel Grotz, Symeon Chatzinotas, Björn Ottersten
Signal Processing
Foundation Model-Aided Hierarchical Deep Reinforcement Learning for Blockage-Aware Link in RIS-Assisted Networks
Mohammad Ghassemi, Han Zhang, Ali Afana et al. (+2 more)
Reconfigurable intelligent surface (RIS) technology has the potential to significantly enhance the spectral efficiency (SE) of 6G wireless networks. However, practical deployment remains constrained by challenges in accurate channel estimation and control optimization under dynamic conditions. This paper presents a foundation model-aided hierarchical deep reinforcement learning (FM-HDRL) framework designed for joint beamforming and phase-shift optimization in RIS-assisted wireless networks. To implement this, we first fine-tune a pre-trained large wireless model (LWM) to translate raw channel data into low-dimensional, context-aware channel state information (CSI)
embeddings. Next, these
embeddings are combined with user location information and blockage status to select the optimal communication path. The resulting features are then fed into an HDRL model, assumed to be implemented at a centralized controller, which jointly optimizes the base station (BS) beamforming vectors and the RIS phase-shift configurations to maximize SE. Simulation results demonstrate that the proposed FM-HDRL framework consistently outperforms baseline methods in terms of convergence speed, spectral efficiency, and scalability. According to the simulation results, our proposed method improves 7.82% SE compared to the FM-aided deep reinforcement learning (FM-DRL) approach and a substantial enhancement of about 48.66% relative to the beam sweeping approach.
Read the full abstract ▾
All authors Mohammad Ghassemi, Han Zhang, Ali Afana, Akram Bin Sediq, Melike Erol-Kantarci
Systems and Control
Dynamic Passivity Multipliers for Plug-and-Play Stability Certificates of Converter-Dominated Grids
Andrey Gorbunov, Youhong Chen, Petr Vorobev et al. (+2 more)
Ensuring small-signal stability in power systems with a high share of inverter-based resources (IBRs) is hampered by two factors: (i) device and network parameters are often uncertain or completely unknown, and (ii) brute-force enumeration of all
topologies is computationally intractable. These challenges motivate plug-and-play (PnP) certificates that verify stability locally yet hold globally. Passivity is an attractive property because it guarantees stability under feedback and network interconnections; however, strict passivity rarely holds for practical controllers such as Grid Forming Inverters (GFMs) employing P-Q droop. This paper extends the passivity condition by constructing a dynamic, frequency-dependent multiplier that enables PnP stability certification of each component based solely on its admittance, without requiring any modification to the controller design. The multiplier is parameterised as a linear filter whose coefficients are tuned under a passivity goal. Numerical results for practical droop gains confirm the PnP rules, substantially enlarging the certified stability region while preserving the decentralised, model-agnostic nature of passivity-based PnP tests.
Read the full abstract ▾
All authors Andrey Gorbunov, Youhong Chen, Petr Vorobev, Jin Ma, Gregor Verbic
Systems and Control
Shaping Energy Exchange with Gyroscopic Interconnections: a Geometric Approach
Jasper Juchem and Mia Loccufier
Gyroscopic interconnections enable redistribution of energy among degrees of freedom while preserving passivity and total energy, and they play a central role in controlled Lagrangian methods and IDA-PBC. Yet their quantitative effect on transient energy exchange and subsystem performance is not well characterised. We study a conservative mechanical system with constant skew-symmetric velocity coupling. Its dynamics are integrable and evolve on invariant two-
tori, whose projections onto subsystem phase planes provide
geometric description of energy exchange. When the ratio of normal-mode frequencies is rational, these projections become closed resonant Lissajous curves, enabling structured analysis of subsystem trajectories. To quantify subsystem behaviour, we introduce the inscribed-radius metric: the radius of the largest origin-centred circle contained in a projected trajectory. This gives a lower bound on attainable subsystem energy and acts as an internal performance measure. We derive resonance conditions and develop an efficient method to compute or certify the inscribed radius without time-domain simulation. Our results show that low-order resonances can strongly restrict energy depletion through phase-locking, whereas high-order resonances recover conservative bounds. These insights lead to an explicit interconnection-shaping design framework for both energy absorption and containment control strategies, while taking responsiveness into account.
Read the full abstract ▾
Systems and Control
An Actor-Critic-Identifier Control Design for Increasing Energy Efficiency of Automated Electric Vehicles
Hamed Faghihian and Arman Sargolzaei
Electric vehicles (EVs) are increasingly deployed, yet range limitations remain a key barrier. Improving energy efficiency via advanced control is therefore essential, and emerging vehicle automation offers a promising avenue. However, many existing strategies rely on indirect surrogates because linking power consumption to control inputs is difficult. We propose a neural-network (NN) identifier that learns this mapping online and couples it with an actor-critic reinforcement learning (RL) framework to generate optimal control commands. The resulting actor-critic-identifier architecture removes dependence on explicit models relating total power, recovered energy, and inputs, while maintaining accurate speed tracking and maximizing efficiency. Update laws are derived using Lyapunov stability analysis, and performance is validated in simulation. Compared to a traditional controller, the method increases total energy recovery by 12.84%, indicating strong potential for improving EV energy efficiency.
Read the full abstract ▾
Systems and Control · Cross-listed from cs.RO
Agile asymmetric multi-legged locomotion: contact planning via geometric mechanics and spin model duality
Jackson Habala, Gabriel B. Margolis, Tianyu Wang et al. (+8 more)
Legged
robot research is presently focused on bipedal or quadrupedal
robots, despite capabilities to build
robots with many more legs to potentially improve locomotion performance. This imbalance is not necessarily due to hardware limitations, but rather to the absence of principled control frameworks that explain when and how additional legs improve locomotion performance. In multi-legged systems, coordinating many simultaneous contacts introduces a severe curse of dimensionality that challenges existing modeling and control approaches. As an alternative, multi-legged
robots are typically controlled using low-dimensional gaits originally developed for bipeds or quadrupeds. These strategies fail to exploit the new
symmetries and control opportunities that emerge in higher-dimensional systems. In this work, we develop a principled framework for discovering new control structures in multi-legged locomotion. We use
geometric mechanics to reduce contact-rich locomotion planning to a graph optimization problem, and propose a
spin model duality framework from statistical mechanics to exploit
symmetry breaking and guide optimal gait reorganization. Using this approach, we identify an asymmetric locomotion strategy for a hexapod
robot that achieves a forward speed of 0.61 body lengths per cycle (a 50% improvement over conventional gaits). The resulting asymmetry appears at both the control and hardware levels. At the control level, the body orientation oscillates asymmetrically between fast clockwise and slow counterclockwise turning phases for forward locomotion. At the hardware level, two legs on the same side remain unactuated and can be replaced with rigid parts without degrading performance. Numerical simulations and robophysical experiments validate the framework and reveal novel locomotion behaviors that emerge from
symmetry reforming in high-dimensional
embodied systems.
Read the full abstract ▾
All authors Jackson Habala, Gabriel B. Margolis, Tianyu Wang, Pratyush Bhatt, Juntao He, Naheel Naeem, Zhaochen Xu, Pulkit Agrawal, Daniel I. Goldman, Di Luo, Baxi Chong
Signal Processing
WiLoc: Massive Measured Dataset of Wi-Fi Channel State Information with Application to Machine-Learning Based Localization
Yuning Zhang, Lei Chu, Omer Gokalp Serbetci et al. (+2 more)
Localization is a key component of the wireless ecosystem. Machine learning (ML)-based localization using channel state information (CSI) is one of the most popular methods for achieving high-accuracy localization with low cost. However, to be accurate and robust, ML-based
algorithms need to be trained and tested with large amounts of data, covering not only many user equipment (UE)/target locations, but also many different access points (APs) locations to which the UEs connect, in a variety of different environment types. This paper presents a massive-sized CSI
dataset, WiLoc (Wi-Fi Localization), and makes it publicly available. WiLoc is obtained by a series of precision measurement campaigns that span three months, and it is massive in all the above-mentioned three dimensions: > 12 million UE locations, > 3,000 APs, covering 16 buildings for indoor localization, and > 30 streets for outdoor use. The paper describes the
dataset structure, measurement environments, measurement protocols, and the
dataset validations. Comprehensive case studies validate the advantages of large
datasets in ML-driven localization strategies for both "standard" and transfer learning. We envision this
dataset, which is by far the largest of its kind, to become a standard resource for researchers in the field of ML-based localization.
Read the full abstract ▾
All authors Yuning Zhang, Lei Chu, Omer Gokalp Serbetci, Jorge Gomez-Ponce, Andreas F. Molisch
Audio and Speech Processing · Cross-listed from cs.SD
NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control
Yufan Wen, Zhaocheng Liu, YeGuo Hua et al. (+4 more)
Synthesizing coherent soundtracks for long-form
videos remains a formidable challenge, currently stalled by three critical impediments: computational scalability, temporal coherence, and, most critically, a pervasive semantic blindness to evolving narrative logic. To bridge these gaps, we propose NarraScore, a hierarchical framework predicated on the core insight that
emotion serves as a high-density compression of narrative logic. Uniquely, we repurpose frozen
Vision-
Language Models (VLMs) as continuous affective sensors, distilling high-dimensional visual streams into dense, narrative-aware Valence-Arousal trajectories. Mechanistically, NarraScore employs a Dual-Branch Injection strategy to reconcile global structure with local dynamism: a Global Semantic Anchor ensures stylistic stability, while a
surgical Token-Level Affective Adapter modulates local tension via direct element-wise residual injection. This minimalist design bypasses the bottlenecks of dense
attention and architectural cloning, effectively mitigating the overfitting risks associated with data scarcity. Experiments demonstrate that NarraScore achieves state-of-the-art consistency and narrative
alignment with negligible computational overhead, establishing a fully autonomous paradigm for long-
video soundtrack generation.
Read the full abstract ▾
All authors Yufan Wen, Zhaocheng Liu, YeGuo Hua, Ziyi Guo, Lihua Zhang, Chun Yuan, Jian Wu