Machine Learning
Attention Isn't All You Need for Emotion Recognition:Domain Features Outperform Transformers on the EAV Dataset
Anmol Guragain
We present a systematic study of
multimodal emotion recognition using the EAV
dataset, investigating whether complex
attention mechanisms improve performance on small
datasets. We implement three model categories: baseline
transformers (M1), novel factorized
attention mechanisms (M2), and improved CNN baselines (M3). Our experiments show that sophisticated
attention mechanisms consistently underperform on small
datasets. M2 models achieved 5 to 13 percentage points below baselines due to overfitting and destruction of pretrained features. In contrast, simple domain-appropriate modifications proved effective: adding delta MFCCs to the audio CNN improved accuracy from 61.9% to 65.56% (+3.66pp), while frequency-domain features for
EEG achieved 67.62% (+7.62pp over the paper baseline). Our
vision transformer baseline (M1) reached 75.30%, exceeding the paper's ViViT result (74.5%) through domain-specific pretraining, and
vision delta features achieved 72.68% (+1.28pp over the paper CNN). These findings demonstrate that for small-scale
emotion recognition, domain knowledge and proper implementation outperform architectural complexity.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Towards Latent Diffusion Suitable For Text
Nesta Midavaine, Christian A. Naesseth and Grigory Bartosh
Language diffusion models aim to improve sampling speed and coherence over autoregressive LLMs. We introduce Neural Flow
Diffusion Models for
language generation, an extension of NFDM that enables the straightforward application of continuous
diffusion models to discrete state spaces. NFDM learns a multivariate forward process from the data, ensuring that the forward process and generative trajectory are a good fit for
language modeling. Our model substantially reduces the likelihood gap with autoregressive models of the same size, while achieving sample quality comparable to that of previous latent
diffusion models.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
SSC-UNet: UNet with Self-Supervised Contrastive Learning for Phonocardiography Noise Reduction
Lizy Abraham, Siobhan Coughlan, Kritika Rajain et al. (+3 more)
Congenital Heart Disease (CHD) remains a significant global health concern affecting approximately 1% of births worldwide. Phonocardiography has emerged as a supplementary tool to diagnose CHD cost-effectively. However, the performance of these diagnostic models highly depends on the quality of the phonocardiography, thus, noise reduction is particularly critical. Supervised UNet effectively improves noise reduction capabilities, but limited clean data hinders its application. The complex time-frequency characteristics of phonocardiography further complicate finding the balance between effectively removing noise and preserving pathological features. In this study, we proposed a self-supervised phonocardiography noise reduction model based on Noise2Noise to enable training without clean data. Augmentation and contrastive learning are applied to enhance its performance. We obtained an average SNR of 12.98 dB after filtering under 10~dB of hospital noise. Classification sensitivity after filtering was improved from 27% to 88%, indicating its promising pathological feature retention capabilities in practical noisy environments.
Read the full abstract ▾
All authors Lizy Abraham, Siobhan Coughlan, Kritika Rajain, Changhong Li, Saji Philip, Adam James
Machine Learning
Time-Series Anomaly Classification for Launch Vehicle Propulsion Systems: Fast Statistical Detectors Enhancing LSTM Accuracy and Data Quality
Sean P. Engelstad, Sameul R. Darr, Matthew Taliaferro et al. (+1 more)
Supporting Go/No-Go decisions prior to launch requires assessing real-time telemetry data against redline limits established during the design qualification phase. Family data from ground testing or previous flights is commonly used to detect initiating failure modes and their timing; however, this approach relies heavily on engineering judgment and is more error-prone for new launch vehicles. To address these limitations, we utilize Long-Term Short-Term
Memory (LSTM) networks for supervised classification of time-series anomalies. Although, initial training labels derived from simulated anomaly data may be suboptimal due to variations in anomaly strength, anomaly settling times, and other factors. In this work, we propose a novel statistical detector based on the Mahalanobis distance and forward-backward detection fractions to adjust the supervised training labels. We demonstrate our method on digital twin simulations of a ground-stage propulsion system with 20.8 minutes of operation per trial and O(10^8) training timesteps. The statistical data relabeling improved precision and recall of the LSTM classifier by 7% and 22% respectively.
Read the full abstract ▾
All authors Sean P. Engelstad, Sameul R. Darr, Matthew Taliaferro, Vinay K. Goyal
Machine Learning
Data-Driven Reduced-Complexity Modeling of Fluid Flows: A Community Challenge
Oliver T. Schmidt, Aaron Towne, Adrian Lozano-Duran et al. (+2 more)
We introduce a community challenge designed to facilitate direct comparisons between data-driven methods for compression,
forecasting, and sensing of complex aerospace flows. The challenge is organized into three tracks that target these complementary capabilities: compression (compact representations for large
datasets),
forecasting (predicting future flow states from a finite history), and sensing (inferring unmeasured flow states from limited measurements). Across these tracks, multiple challenges span diverse flow
datasets and use cases, each emphasizing different model requirements. The challenge is open to anyone, and we invite broad participation to build a comprehensive and balanced picture of what works and where current methods fall short. To support fair comparisons, we provide standardized success metrics, evaluation tools, and baseline implementations, with one classical and one machine-learning baseline per challenge. Final assessments use blind tests on withheld data. We explicitly encourage negative results and careful analyses of limitations. Outcomes will be disseminated through an AIAA Journal Virtual Collection and invited presentations at AIAA conferences.
Read the full abstract ▾
All authors Oliver T. Schmidt, Aaron Towne, Adrian Lozano-Duran, Scott T. M. Dawson, Ricardo Vinuesa
Machine Learning · Cross-listed from cs.AI
Neuro-Symbolic Compliance: Integrating LLMs and SMT Solvers for Automated Financial Legal Analysis
Yung-Shen Hsia, Fang Yu and Jie-Hong Roland Jiang
Financial regulations are increasingly complex, hindering automated compliance-especially the maintenance of logical consistency with minimal human oversight. We introduce a Neuro-Symbolic Compliance Framework that integrates Large
Language Models (LLMs) with Satisfiability Modulo Theories (SMT) solvers to enable formal verifiability and optimization-based compliance correction. The LLM interprets statutes and enforcement cases to generate SMT constraints, while the solver enforces consistency and computes the minimal factual modification required to restore legality when penalties arise. Unlike transparency-oriented methods, our approach emphasizes logic-driven optimization, delivering verifiable, legally consistent
reasoning rather than post-hoc explanation. Evaluated on 87 enforcement cases from Taiwan's Financial Supervisory Commission (FSC), the system attains 86.2% correctness in SMT code generation, improves
reasoning efficiency by over 100x, and consistently corrects violations-establishing a preliminary foundation for optimization-based compliance applications.
Read the full abstract ▾
Machine Learning
MixDPO: Modeling Preference Strength for Pluralistic Alignment
Saki Imai, Pedram Heydari, Anthony Sicilia et al. (+3 more)
Preference based
alignment objectives implicitly assume that all human preferences are expressed with equal strength. In practice, however, preference strength varies across individuals and contexts – a phenomenon established in behavioral economics and discrete choice theory. This mismatch limits the ability of existing objectives to faithfully capture heterogeneous human judgments. Inspired by this literature, we introduce Mixed Logit Direct Preference Optimization (MixDPO), a generalization of Direct Preference Optimization that models variation in preference strength. MixDPO enables
alignment objectives to capture heterogeneity in how strongly preferences are expressed across training examples. We evaluate MixDPO on three preference
datasets using two open-weight
language models. Across
datasets, MixDPO improves aggregate
alignment performance (+11.2 points on Pythia-2.8B) while preserving subgroup level preferences, with the largest gains appearing in settings with higher inferred preference heterogeneity. MixDPO makes preference heterogeneity explicit through learned strength distributions. We release our code for reproducibility.
Read the full abstract ▾
All authors Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani
Machine Learning · Cross-listed from cs.CY
Performance of models for monitoring sustainable development goals from remote sensing: A three-level meta-regression
Jonas Klingwort, Nina M. Leach and Joep Burger
Machine learning (ML) is a tool to exploit remote sensing data for the monitoring and implementation of the United Nations' Sustainable Development Goals (SDGs). In this paper, we report on a meta-analysis to evaluate the performance of ML applied to remote sensing data to monitor SDGs. Specifically, we aim to 1) estimate the average performance; 2) determine the degree of heterogeneity between and within studies; and 3) assess how study features influence model performance. Using PRISMA guidelines, a search was performed across multiple academic databases to identify potentially relevant studies. A random sample of 200 was screened by three reviewers, resulting in 86 trials within 20 studies with 14 study features. Overall accuracy was the most reported performance metric. It was analyzed using double arcsine transformation and a three-level random effects model. The average overall accuracy of the best model was 0.90 [0.86, 0.92]. There was considerable heterogeneity in model performance, 64% of which was between studies. The only significant feature was the prevalence of the majority class, which explained 61% of the between-study heterogeneity. None of the other thirteen features added value to the model. The most important contributions of this paper are the following two insights. 1) Overall accuracy is the most popular performance metric, yet arguably the least insightful. Its sensitivity to class imbalance makes it necessary to normalize it, which is far from common practice. 2) The field needs to standardize the reporting. Reporting of the confusion
matrix for independent test sets is the most important ingredient for between-study comparisons of ML classifiers. These findings underscore the need for robust and comparable evaluation metrics in machine learning applications to ensure reliable and actionable insights for effective SDG monitoring and policy formulation.
Read the full abstract ▾
Machine Learning
Parent-Guided Adaptive Reliability (PGAR): A Behavioural Meta-Learning Framework for Stable and Trustworthy AI
Anshum Rankawat
Parent-Guided Adaptive Reliability (PGAR) is a lightweight behavioural meta-learning framework that adds a supervisory "parent" layer on top of a standard learner to improve stability, calibration, and recovery under disturbances. PGAR computes three reflex-level signals (incident detection, overconfidence correction, and recovery
memory) and fuses them into a bounded reliability index in [0,1]. This index continuously modulates the learner's effective learning rate, reducing update magnitude during instability and restoring it as reliability improves. We provide a Lyapunov-based proof sketch establishing bounded adaptation of the reliability dynamics under mild assumptions (smooth loss, descent direction, and bounded reflex outputs). Empirical evaluations on representative learning tasks show improved calibration, reduced loss variance, and faster recovery compared to standard optimizers, while retaining computational simplicity. PGAR functions as a plug-in reliability layer for existing optimization and learning pipelines, supporting interpretable reliability traces in safety-relevant settings.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking
Kaiyuan Deng, Bo Hui, Gen Li et al. (+4 more)
The widespread adoption of text-to-image (T2I)
diffusion models has raised concerns about their potential to generate copyrighted, inappropriate, or sensitive imagery. As a practical solution, machine unlearning aims to erase unwanted concepts without retraining from scratch. While most existing methods are effective for single-concept unlearning, they often struggle when removing multiple concepts, causing significant challenges in unlearning effectiveness, generation quality, and sensitivity to hyperparameters and
datasets. We take a unique perspective on multi-concept unlearning by leveraging model sparsity and propose the Forget It All (FIA) framework. FIA first introduces Contrastive Concept Saliency to quantify each weight connection's contribution to a target concept. It then identifies Concept Sensitive
Neurons by combining temporal and spatial information, ensuring that only
neurons consistently responsive to the target concept are selected. Finally, FIA constructs masks from the identified
neurons and fuses them into a unified multi-concept mask, where Concept Agnostic
Neurons that broadly support general content generation are preserved while concept-specific
neurons are pruned to remove the targets. FIA is training-free and requires minimal hyperparameter tuning for new tasks, enabling plug-and-play use. Extensive experiments across three distinct unlearning tasks demonstrate that FIA achieves more reliable multi-concept unlearning, improving forgetting effectiveness while maintaining generation fidelity and quality. Code is available at https://github.com/kaiyuan02415/Forget-It-All
Read the full abstract ▾
All authors Kaiyuan Deng, Bo Hui, Gen Li, Jie Ji, Minghai Qin, Geng Yuan, Xiaolong Ma
Machine Learning · Cross-listed from hep-ph
A universal vision transformer for fast calorimeter simulations
Luigi Favaro, Andrea Giammanco and Claudius Krause
The high-dimensional complex nature of detectors makes fast calorimeter simulations a
prime application for modern generative machine learning.
Vision transformers (ViTs) can emulate the Geant4 response with unmatched accuracy and are not limited to regular geometries. Starting from the CaloDREAM architecture, we demonstrate the robustness and scalability of ViTs on regular and irregular geometries, and multiple detectors. Our results show that ViTs generate electromagnetic and hadronic showers with minimal deviations from Geant4 in multiple evaluation metrics, while maintaining the generation time in the $\mathcal{O}(10-100)$ ms on a single GPU. Furthermore, we show that pretraining on a large
dataset and fine-tuning on the target geometry leads to reduced training costs and higher data efficiency, or altogether improves the fidelity of generated showers.
Read the full abstract ▾
Machine Learning
Explainable Admission-Level Predictive Modeling for Prolonged Hospital Stay in Elderly Populations: Challenges in Low- and Middle-Income Countries
Daniel Sierra-Botero, Ana Molina-Taborda, Leonardo Espinosa-Leal et al. (+3 more)
Prolonged length of stay (pLoS) is a significant factor associated with the risk of adverse in-hospital events. We develop and explain a predictive model for pLos using admission-level patient and hospital administrative data. The approach includes a feature selection method by selecting non-correlated features with the highest information value. The method uses features weights of evidence to select a representative within cliques from graph theory. The prognosis study analyzed the records from 120,354 hospital admissions at the Hospital Alma Mater de Antioquia between January 2017 and March 2022. After a cleaning process the
dataset was split into training (67%), test (22%), and validation (11%) cohorts. A logistic regression model was trained to predict the pLoS in two classes: less than or greater than 7 days. The performance of the model was evaluated using accuracy, precision, sensitivity, specificity, and AUC-ROC metrics. The feature selection method returns nine interpretable variables, enhancing the models' transparency. In the validation cohort, the pLoS model achieved a specificity of 0.83 (95% CI, 0.82-0.84), sensitivity of 0.64 (95% CI, 0.62-0.65), accuracy of 0.76 (95% CI, 0.76-0.77), precision of 0.67 (95% CI, 0.66-0.69), and AUC-ROC of 0.82 (95% CI, 0.81-0.83). The model exhibits strong predictive performance and offers insights into the factors that influence prolonged hospital stays. This makes it a valuable tool for hospital management and for developing future intervention studies aimed at reducing pLoS.
Read the full abstract ▾
All authors Daniel Sierra-Botero, Ana Molina-Taborda, Leonardo Espinosa-Leal, Alexander Karpenko, Alejandro Hernandez, Olga Lopez-Acevedo
Machine Learning
When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learning
Gal Fybish and Teo Susnjak
Machine learning models are increasingly used in high-stakes domains where their predictions can actively shape the environments in which they operate, a phenomenon known as performative prediction. This dynamic, in which the deployment of the model influences the very outcome it seeks to predict, can lead to unintended consequences, including feedback loops, performance issues, and significant societal risks. While the literature in the field has grown rapidly in recent years, a socio-technical synthesis that systemises the phenomenon concepts and provides practical guidance has been lacking. This Systematisation of Knowledge (SoK) addresses this gap by providing a comprehensive review of the literature on performative predictions. We provide an overview of the primary mechanisms through which performativity manifests, present a typology of associated risks, and survey the proposed solutions offered in the literature. Our primary contribution is the “Performative Strength vs. Impact
Matrix" assessment framework. This practical tool is designed to help practitioners assess the potential influence and severity of performativity on their deployed predictive models and select the appropriate level of algorithmic or human intervention.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.mtrl-sci
SpectraFormer: an Attention-Based Raman Unmixing Tool for Accessing the Graphene Buffer-Layer Signature on SiC
Dmitriy Poteryayev, Pietro Novelli, Annalisa Coriolano et al. (+7 more)
Raman spectroscopy is a key tool for
graphene characterization, yet its application to
graphene grown on silicon carbide (SiC) is strongly limited by the intense and variable second-order Raman response of the substrate. This limitation is critical for buffer layer
graphene, a semiconducting interfacial phase, whose vibrational signatures are overlapped with the SiC background and challenging to be reliably accessed using conventional reference-based subtraction, due to strong spatial and experimental variability of the substrate signal. Here we present SpectraFormer, a
transformer-based deep learning model that reconstructs the SiC Raman substrate contribution directly from post-growth partially masked spectroscopic data without relying on explicit reference measurements. By learning global correlations across the entire Raman shift range, the model captures the statistical structure of the SiC background and enables accurate reconstruction of its contribution in mixed spectra. Subtraction of the reconstructed substrate signal reveals weak vibrational features associated with ZLG that are inaccessible through conventional analysis methods. The extracted spectra are validated by ab initio vibrational calculations, allowing assignment of the resolved features to specific modes and confirming their physical consistency. By leveraging a state-of-the-art
attention-based deep learning architecture, this approach establishes a robust, reference-free framework for Raman analysis of
graphene on SiC and provides a foundation, compatible with real-time data acquisition, to its integration into automated, closed-loop AI-assisted growth optimization.
Read the full abstract ▾
All authors Dmitriy Poteryayev, Pietro Novelli, Annalisa Coriolano, Riccardo Dettori, Valentina Tozzini, Fabio Beltram, Massimiliano Pontil, Antonio Rossi, Stiven Forti, Camilla Coletti
Machine Learning · Cross-listed from cs.CR
Large Language Models for Detecting Cyberattacks on Smart Grid Protective Relays
Ahmad Mohammad Saber, Saeed Jafari, Zhengmao Ouyang et al. (+3 more)
This paper presents a large
language model (LLM)-based framework that adapts and fine-tunes compact LLMs for detecting cyberattacks on
transformer current differential relays (TCDRs), which can otherwise cause false tripping of critical power
transformers. The core idea is to textualize multivariate time-series current measurements from TCDRs, across phases and input/output sides, into structured natural-
language prompts that are then processed by compact, locally deployable LLMs. Using this representation, we fine-tune DistilBERT, GPT-2, and DistilBERT+LoRA to distinguish cyberattacks from genuine fault-induced disturbances while preserving relay dependability. The proposed framework is evaluated against a broad set of state-of-the-art machine learning and deep learning baselines under nominal conditions, complex cyberattack scenarios, and measurement noise. Our results show that LLM-based detectors achieve competitive or superior cyberattack detection performance, with DistilBERT detecting up to 97.62% of attacks while maintaining perfect fault detection accuracy. Additional evaluations demonstrate robustness to
prompt formulation variations, resilience under combined time-synchronization and false-data injection attacks, and stable performance under realistic measurement noise levels. The
attention mechanisms of LLMs further enable intrinsic interpretability by highlighting the most influential time-phase regions of relay measurements. These results demonstrate that compact LLMs provide a practical, interpretable, and robust solution for enhancing cyberattack detection in modern digital substations. We provide the full
dataset used in this study for reproducibility.
Read the full abstract ▾
All authors Ahmad Mohammad Saber, Saeed Jafari, Zhengmao Ouyang, Paul Budnarain, Amr Youssef, Deepa Kundur
Machine Learning
Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization
Matthew Landers, Taylor W. Killian, Thomas Hartvigsen et al. (+1 more)
Reinforcement learning in discrete combinatorial action spaces requires searching over exponentially many joint actions to simultaneously select multiple sub-actions that form coherent combinations. Existing approaches either simplify policy learning by assuming independence across sub-actions, which often yields incoherent or invalid actions, or attempt to learn action structure and control jointly, which is slow and unstable. We introduce Structured Policy Initialization (
SPIN), a two-stage framework that first pre-trains an Action Structure Model (ASM) to capture the
manifold of valid actions, then freezes this representation and trains lightweight policy heads for control. On challenging discrete DM Control
benchmarks,
SPIN improves average return by up to 39% over the state of the art while reducing time to convergence by up to 12.8$\times$.
Read the full abstract ▾
All authors Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab
Machine Learning · Cross-listed from cs.DS
Learning Multinomial Logits in $O(n \log n)$ time
Flavio Chierichetti, Mirko Giacchini, Ravi Kumar et al. (+4 more)
A Multinomial Logit (MNL) model is composed of a finite universe of items $[n]=\{1,\dots,n\}$, each assigned a positive weight. A query specifies an admissible subset—called a slate—and the model chooses one item from that slate with probability proportional to its weight. This query model is also known as the Plackett–Luce model or conditional sampling oracle in the literature. Although MNLs have been studied extensively, a basic computational question remains open: given query access to slates, how efficiently can we learn weights so that, for every slate, the induced choice distribution is within total variation distance $\varepsilon$ of the ground truth? This question is central to MNL learning and has direct implications for modern recommender system interfaces. We provide both adaptive and non-adaptive
algorithms for this task. Our first adaptive
algorithm makes $O(\frac{n\log n}{\varepsilon^2})$ queries and learns every choice distribution in total variation distance $\varepsilon$. We also provide a second adaptive
algorithm that with $O(\frac{n \log n}{\varepsilon^3})$ queries provides a stronger guarantee: each choice distribution is approximated to a multiplicative $1\pm\varepsilon$ error (except for subsets of mass smaller than $\varepsilon$). We complement these
algorithms with lower bounds of $\Omega(\frac{n\log n}{\varepsilon^2})$ for total variation distance and $\Omega(\frac{n \log n}{\varepsilon^3})$ for the multiplicative error guarantee, showing the optimality of our adaptive
algorithms. Our non-adaptive
algorithm makes $O\left(\frac{n^{2}}{\varepsilon^{3}}\log n \log\frac{n}{\varepsilon}\right)$ queries to obtain guarantees under the multiplicative error notion. We complement this result with an $\Omega(\frac{n^2 \log n}{\varepsilon^2})$ lower bound for any non-adaptive
algorithm, showing that our
algorithm is almost tight in $n$ (within a $\log n$ factor).
Read the full abstract ▾
All authors Flavio Chierichetti, Mirko Giacchini, Ravi Kumar, Silvio Lattanzi, Alessandro Panconesi, Erasmo Tani, Andrew Tomkins