Machine Learning
Hybrid Mamba-Attention Neural Architecture for Channel Estimation
Dianxin Luan, Chengsi Liang, Jie Huang et al. (+5 more)
This paper proposes a hybrid Mamba-
attention neural architecture to achieve improved channel estimation for orthogonal frequency-division multiplexing (OFDM) waveforms, particularly for configurations with a large number of subcarriers. By integrating a customized Mamba module, the proposed framework handles large-scale subcarrier channel estimation efficiently while capturing long-distance dependencies among these subcarriers effectively. Unlike the conventional Mamba structure, this paper implements a bidirectional selective scan to enable information propagation from both directions, because channel gains at different subcarriers are inherently non-causal. In addition, by integrating Mamba to reduce the reliance on quadratic-complexity self-
attention, the proposed solution achieves lower space complexity than fully
transformer architectures. Simulation results based on the 3GPP TS 36.101 channel demonstrate that compared to other baseline neural networks, the proposed method achieves superior channel estimation performance with fewer tunable parameters and exhibits good generalization across previously unseen channels.
Read the full abstract ▾
All authors Dianxin Luan, Chengsi Liang, Jie Huang, Zheng Lin, Kaitao Meng, John Thompson, Cheng-Xiang Wang, Ozgur Akan
Machine Learning · Cross-listed from cs.CV
Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals
Pascaline André, Charles Heitz, Evangelia Christodoulou et al. (+10 more)
Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24
segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (
segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. Finally, we derived practical implications of this study in the form of a decision tree which shall prove useful to the community. This paves the way for future consensus guidelines on reporting performance uncertainty.
Read the full abstract ▾
All authors Pascaline André, Charles Heitz, Evangelia Christodoulou, Annika Reinke, Carole H. Sudre, Michela Antonelli, Patrick Godau, M. Jorge Cardoso, Antoine Gilson, Sophie Tezenas du Montcel, Gaël Varoquaux, Lena Maier-Hein, Olivier Colliot
Machine Learning
The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generation
Junichiro Niimi
Large
Language Models (LLMs) generate fluent text, yet whether they truly understand the world or merely produce plausible texts about it remains contested. We propose an architectural principle, the mouth is not the
brain, that explicitly separates world models from
language models. Our architecture comprises three components: a DBM that captures domain structure as an energy-based world model, an adapter that projects latent belief states into
embedding space, and a frozen GPT-2 that provides linguistic competence without domain knowledge. We instantiate this framework in the consumer review domain using Amazon smartphone reviews. Experiments demonstrate that (1) world model conditioning achieves lower cross-
entropy loss and higher semantic similarity than architectural baselines including direct projection and full fine-tuning, while qualitative analysis reveals that soft
prompt conditioning resolves a trade-off that
prompt-based approaches cannot: simple
prompts lack expressiveness while detailed
prompts cause output collapse in small LLMs; (2) the DBM's energy function distinguishes coherent from incoherent
market configurations, assigning higher energy to implausible brand-price combinations; and (3) interventions on specific attributes propagate causally to generated text with intervened outputs exhibiting distributions statistically consistent with naturally occurring samples sharing the target configuration. These findings suggest that even small-scale
language models can achieve consistent, controllable generation when connected to an appropriate world model, providing empirical support for separating linguistic competence from world understanding.
Read the full abstract ▾
Machine Learning
The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations
Olha Sirikova and Alvin Chan
Comparing neural network representations is essential for understanding and validating models in scientific applications. Existing methods, however, often provide a limited view. We propose the Triangle of Similarity, a framework that combines three complementary perspectives: static representational similarity (CKA/Procrustes), functional similarity (Linear Mode Connectivity or Predictive Similarity), and sparsity similarity (robustness under
pruning). Analyzing a range of CNNs,
Vision Transformers, and
Vision-
Language Models using both in-distribution (ImageNetV2) and out-of-distribution (CIFAR-10) testbeds, our initial findings suggest that: (1) architectural family is a primary determinant of representational similarity, forming distinct clusters; (2) CKA self-similarity and task accuracy are strongly correlated during
pruning, though accuracy often degrades more sharply; and (3) for some model pairs,
pruning appears to regularize representations, exposing a shared computational core. This framework offers a more holistic approach for assessing whether models have converged on similar internal mechanisms, providing a useful tool for model selection and analysis in scientific research.
Read the full abstract ▾
Machine Learning
CUROCKET: Optimizing ROCKET for GPU
Ole Stüven, Keno Moenck and Thorsten Schüppstuhl
ROCKET (RandOm Convolutional KErnel Transform) is a feature extraction
algorithm created for Time Series Classification (TSC), published in 2019. It applies convolution with randomly generated kernels on a time series, producing features that can be used to train a linear classifier or regressor like Ridge. At the time of publication, ROCKET was on par with the best state-of-the-art
algorithms for TSC in terms of accuracy while being significantly less computationally expensive, making ROCKET a compelling
algorithm for TSC. This also led to several subsequent versions, further improving accuracy and computational efficiency. The currently available ROCKET implementations are mostly bound to execution on CPU. However, convolution is a task that can be highly parallelized and is therefore suited to be executed on GPU, which speeds up the computation significantly. A key difficulty arises from the inhomogeneous kernels ROCKET uses, making standard methods for applying convolution on GPU inefficient. In this work, we propose an
algorithm that is able to efficiently perform ROCKET on GPU and achieves up to 11 times higher computational efficiency per watt than ROCKET on CPU. The code for CUROCKET is available in this repository https://github.com/oleeven/CUROCKET on github.
Read the full abstract ▾
Machine Learning
SFO: Learning PDE Operators via Spectral Filtering
Noam Koren, Rafael Moschopoulos, Kira Radinsky et al. (+1 more)
Partial differential equations (PDEs) govern complex systems, yet neural operators often struggle to efficiently capture the long-range, nonlocal interactions inherent in their solution maps. We introduce Spectral Filtering Operator (SFO), a neural operator that parameterizes integral kernels using the Universal Spectral Basis (USB), a fixed, global orthonormal basis derived from the eigenmodes of the
Hilbert matrix in spectral filtering theory. Motivated by our theoretical finding that the discrete Green's functions of shift-invariant PDE discretizations exhibit spatial Linear Dynamical System (LDS) structure, we prove that these kernels admit compact approximations in the USB. By learning only the spectral coefficients of rapidly decaying
eigenvalues, SFO achieves a highly efficient representation. Across six
benchmarks, including reaction-
diffusion, fluid dynamics, and 3D electromagnetics, SFO achieves state-of-the-art accuracy, reducing error by up to 40% relative to strong baselines while using substantially fewer parameters.
Read the full abstract ▾
All authors Noam Koren, Rafael Moschopoulos, Kira Radinsky, Elad Hazan
Machine Learning · Cross-listed from cs.HC
Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde et al. (+2 more)
Agentic
benchmarks increasingly rely on LLM-simulated users to scalably evaluate
agent performance, yet the robustness, validity, and
fairness of this approach remain unexamined. Through a user study with participants across the United States, India, Kenya, and Nigeria, we investigate whether LLM-simulated users serve as reliable proxies for real human users in evaluating
agents on {\tau}-Bench retail tasks. We find that user simulation lacks robustness, with
agent success rates varying up to 9 percentage points across different user LLMs. Furthermore, evaluations using simulated users exhibit systematic miscalibration, underestimating
agent performance on challenging tasks and overestimating it on moderately difficult ones. African American Vernacular English (AAVE) speakers experience consistently worse success rates and calibration errors than Standard American English (SAE) speakers, with disparities compounding significantly with age. We also find simulated users to be a differentially effective proxy for different populations, performing worst for AAVE and Indian English speakers. Additionally, simulated users introduce conversational artifacts and surface different failure patterns than human users. These findings demonstrate that current evaluation practices risk misrepresenting
agent capabilities across diverse user populations and may obscure real-world deployment challenges.
Read the full abstract ▾
All authors Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, Seraphina Goldfarb-Tarrant
Machine Learning · Cross-listed from eess.AS
Recovering Performance in Speech Emotion Recognition from Discrete Tokens via Multi-Layer Fusion and Paralinguistic Feature Integration
Esther Sun, Abinay Reddy Naini and Carlos Busso
Discrete
speech tokens offer significant advantages for storage and
language model integration, but their application in
speech emotion recognition (SER) is limited by paralinguistic information loss during quantization. This paper presents a comprehensive investigation of discrete
tokens for SER. Using a fine-tuned WavLM-Large model, we systematically quantify performance degradation across different layer configurations and k-means quantization granularities. To recover the information loss, we propose two key strategies: (1)
attention-based multi-layer fusion to recapture complementary information from different layers, and (2) integration of openSMILE features to explicitly reintroduce paralinguistic cues. We also compare mainstream neural codec tokenizers (SpeechTokenizer, DAC, EnCodec) and analyze their behaviors when fused with acoustic features. Our findings demonstrate that through multi-layer fusion and acoustic feature integration, discrete
tokens can close the performance gap with continuous representations in SER tasks.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CY
Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models
Zhining Liu, Tianyi Wang, Xiao Lin et al. (+9 more)
Despite substantial efforts toward improving the moral
alignment of
Vision-
Language Models (VLMs), it remains unclear whether their ethical judgments are stable in realistic settings. This work studies moral robustness in VLMs, defined as the ability to preserve moral judgments under textual and visual perturbations that do not alter the underlying moral context. We systematically probe VLMs with a diverse set of model-agnostic
multimodal perturbations and find that their moral stances are highly fragile, frequently flipping under simple manipulations. Our analysis reveals systematic vulnerabilities across perturbation types, moral domains, and model scales, including a sycophancy trade-off where stronger instruction-following models are more susceptible to persuasion. We further show that lightweight inference-time interventions can partially restore moral stability. These results demonstrate that moral
alignment alone is insufficient and that moral robustness is a necessary criterion for the responsible deployment of VLMs.
Read the full abstract ▾
All authors Zhining Liu, Tianyi Wang, Xiao Lin, Penghao Ouyang, Gaotang Li, Ze Yang, Hui Liu, Sumit Keswani, Vishwa Pardeshi, Huijun Zhao, Wei Fan, Hanghang Tong
Machine Learning · Cross-listed from eess.AS
PC-MCL: Patient-Consistent Multi-Cycle Learning with multi-label bias correction for respiratory sound classification
Seung Gyu Jeong and Seong-Eun Kim
Automated respiratory sound classification supports the diagnosis of pulmonary diseases. However, many deep models still rely on cycle-level analysis and suffer from patient-specific overfitting. We propose PC-MCL (Patient-Consistent Multi-Cycle Learning) to address these limitations by utilizing three key components: multi-cycle concatenation, a 3-label formulation, and a patient-matching auxiliary task. Our work resolves a multi-label distributional bias in respiratory sound classification, a critical issue inherent to applying multi-cycle concatenation with the conventional 2-label formulation (crackle, wheeze). This bias manifests as a systematic loss of normal signal information when normal and abnormal cycles are combined. Our proposed 3-label formulation (normal, crackle, wheeze) corrects this by preserving information from all constituent cycles in mixed samples. Furthermore, the patient-matching auxiliary task acts as a multi-task regularizer, encouraging the model to learn more robust features and improving generalization. On the ICBHI 2017
benchmark, PC-MCL achieves an ICBHI Score of 65.37%, outperforming existing baselines. Ablation studies confirm that all three components are essential, working synergistically to improve the detection of abnormal respiratory events.
Read the full abstract ▾
Machine Learning
E2PL: Effective and Efficient Prompt Learning for Incomplete Multi-view Multi-Label Class Incremental Learning
Jiajun Chen, Yue Wu, Kai Huang et al. (+6 more)
Multi-view multi-label classification (MvMLC) is indispensable for modern web applications aggregating information from diverse sources. However, real-world web-scale settings are rife with missing views and continuously emerging classes, which pose significant obstacles to robust learning. Prevailing methods are ill-equipped for this reality, as they either lack adaptability to new classes or incur exponential parameter growth when handling all possible missing-view patterns, severely limiting their scalability in web environments. To systematically address this gap, we formally introduce a novel task, termed incomplete multi-view multi-label class incremental learning (IMvMLCIL), which requires models to simultaneously address heterogeneous missing views and dynamic class expansion. To tackle this task, we propose E2PL, an Effective and Efficient
Prompt Learning framework for IMvMLCIL. E2PL unifies two novel
prompt designs: task-tailored
prompts for class-incremental adaptation and missing-aware
prompts for the flexible integration of arbitrary view-missing scenarios. To fundamentally address the exponential parameter explosion inherent in missing-aware
prompts, we devise an efficient prototype tensorization module, which leverages atomic tensor decomposition to elegantly reduce the
prompt parameter complexity from exponential to linear w.r.t. the number of views. We further incorporate a dynamic contrastive learning strategy explicitly model the complex dependencies among diverse missing-view patterns, thus enhancing the model's robustness. Extensive experiments on three
benchmarks demonstrate that E2PL consistently outperforms state-of-the-art methods in both effectiveness and efficiency. The codes and
datasets are available at https://anonymous.4open.science/r/code-for-E2PL.
Read the full abstract ▾
All authors Jiajun Chen, Yue Wu, Kai Huang, Wen Xi, Yangyang Wu, Xiaoye Miao, Mengying Zhu, Meng Xi, Guanjie Cheng
Machine Learning
Physics-Encoded Inverse Modeling for Arctic Snow Depth Estimation
Akila Sampath, Vandana P. Janeja and Jianwu Wang
Accurate estimation of unobserved quantities in time-varying inverse problems remains challenging when observations are sparse and only indirectly related to the target variable. In Arctic
climate applications, snow depth over sea ice is not directly available in commonly used reanalysis products and must instead be inferred from related physical and environmental variables. To address this challenge, we introduce Physics-Encoded Inverse Modeling (PhysE-Inv), a framework that combines sequential deep learning with a physics-encoded parameter estimation module for inverse estimation under sparse observational conditions. PhysE-Inv uses an LSTM encoder-decoder to capture temporal dependencies and incorporates contrastive learning to improve the consistency of learned representations. The framework learns structured physics-encoded parameters that are integrated with observational inputs to estimate snow depth proxies. Under the proposed proxy evaluation framework, PhysE-Inv outperforms all evaluated baselines, achieving an average MSE reduction of 24.4% compared with baseline models and a 17.3% improvement over the strongest baseline under the parameter estimation setting. These results demonstrate the potential of physics-encoded modeling approaches for estimating unobserved quantities in data-scarce domains where direct observations are limited.
Read the full abstract ▾
Machine Learning
Attention-Based Variational Framework for Joint and Individual Components Learning with Applications in Brain Network Analysis
Yifei Zhang, Meimei Liu and Zhengwu Zhang
Brain organization is increasingly characterized through multiple imaging modalities, most notably structural connectivity (SC) and functional connectivity (FC). Integrating these inherently distinct yet complementary data sources is essential for uncovering the cross-modal patterns that drive behavioral phenotypes. However, effective integration is hindered by the high dimensionality and non-linearity of connectome data, complex non-linear SC-FC coupling, and the challenge of disentangling shared information from modality-specific variations. To address these issues, we propose the Cross-Modal Joint-Individual Variational Network (CM-JIVNet), a unified probabilistic framework designed to learn factorized latent representations from paired SC-FC
datasets. Our model utilizes a multi-head
attention fusion module to capture non-linear cross-modal dependencies while isolating independent, modality-specific signals. Validated on Human Connectome Project Young Adult (HCP-YA) data, CM-JIVNet demonstrates superior performance in cross-modal reconstruction and behavioral trait prediction. By effectively disentangling joint and individual feature spaces, CM-JIVNet provides a robust, interpretable, and scalable solution for large-scale
multimodal brain analysis.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
AnyView: Synthesizing Any Novel View in Dynamic Scenes
Basile Van Hoorick, Dian Chen, Shun Iwase et al. (+7 more)
Modern generative
video models excel at producing convincing, high-quality outputs, but struggle to maintain multi-view and spatiotemporal consistency in highly dynamic real-world environments. In this work, we introduce $\textbf{AnyView}$, a
diffusion-based
video generation framework for $\textit{dynamic view synthesis}$ with minimal inductive biases or
geometric assumptions. We leverage multiple data sources with various levels of supervision, including
monocular (2D), multi-view static (3D) and multi-view dynamic (4D)
datasets, to train a generalist spatiotemporal implicit representation capable of producing zero-shot novel
videos from arbitrary camera locations and trajectories. We evaluate AnyView on standard
benchmarks, showing competitive results with the current state of the art, and propose $\textbf{AnyViewBench}$, a challenging new
benchmark tailored towards $\textit{extreme}$ dynamic view synthesis in diverse real-world scenarios. In this more dramatic setting, we find that most baselines drastically degrade in performance, as they require significant overlap between viewpoints, while AnyView maintains the ability to produce realistic, plausible, and spatiotemporally consistent
videos when prompted from $\textit{any}$ viewpoint. Results, data, code, and models can be viewed at: https://tri-ml.github.io/AnyView/
Read the full abstract ▾
All authors Basile Van Hoorick, Dian Chen, Shun Iwase, Pavel Tokmakov, Muhammad Zubair Irshad, Igor Vasiljevic, Swati Gupta, Fangzhou Cheng, Sergey Zakharov, Vitor Campagnolo Guizilini
Machine Learning
A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs
Dayal Singh Kalra, Jean-Christophe Gagnon-Audet, Andrey Gromov et al. (+4 more)
Understanding the
curvature evolution of the loss landscape is fundamental to analyzing the training dynamics of neural networks. The most commonly studied measure, Hessian sharpness ($\lambda_{\max}^H$) – the largest
eigenvalue of the loss Hessian – determines local training stability and interacts with the learning rate throughout training. Despite its significance in analyzing training dynamics, direct measurement of Hessian sharpness remains prohibitive for Large
Language Models (LLMs) due to high computational cost. We analyze $\textit{critical sharpness}$ ($\lambda_c$), a computationally efficient measure requiring fewer than $10$ forward passes given the update direction $\Delta \mathbf{\theta}$. Critically, this measure captures well-documented Hessian sharpness phenomena, including progressive sharpening and Edge of Stability. Using this measure, we provide the first demonstration of these sharpness phenomena at scale, up to $7$B parameters, spanning both pre-training and mid-training of OLMo-2 models. We further introduce $\textit{relative critical sharpness}$ ($\lambda_c^{1\to 2}$), which quantifies the
curvature of one loss landscape while optimizing another, to analyze the transition from pre-training to fine-tuning and guide data mixing strategies. Critical sharpness provides practitioners with a practical tool for diagnosing
curvature dynamics and informing data composition choices at scale. More broadly, our work shows that scalable
curvature measures can provide actionable insights for large-scale training.
Read the full abstract ▾
All authors Dayal Singh Kalra, Jean-Christophe Gagnon-Audet, Andrey Gromov, Ishita Mediratta, Kelvin Niu, Alexander H Miller, Michael Shvartsman
Machine Learning
Latent Diffusion for Internet of Things Attack Data Generation in Intrusion Detection
Estela Sánchez-Carballo, Francisco M. Melgarejo-Meseguer and José Luis Rojo-Álvarez
Intrusion Detection Systems (IDSs) are a key component for protecting Internet of Things (IoT) environments. However, in Machine Learning-based (ML-based) IDSs, performance is often degraded by the strong class imbalance between benign and attack traffic. Although data augmentation has been widely explored to mitigate this issue, existing approaches typically rely on simple oversampling techniques or generative models that struggle to simultaneously achieve high sample fidelity, diversity, and computational efficiency. To address these limitations, we propose the use of a Latent
Diffusion Model (LDM) for attack data augmentation in IoT intrusion detection and provide a comprehensive comparison against state-of-the-art baselines. Experiments were conducted on three representative IoT attack types, specifically Distributed Denial-of-Service (DDoS), Mirai, and Man-in-the-Middle, evaluating both downstream IDS performance and intrinsic generative quality using distributional, dependency-based, and diversity metrics. Results show that balancing the training data with LDM-generated samples substantially improves IDS performance, achieving F1-scores of up to 0.99 for DDoS and Mirai attacks and consistently outperforming competing methods. Additionally, quantitative and qualitative analyses demonstrate that LDMs effectively preserve feature dependencies while generating diverse samples and reduce sampling time by approximately 25% compared to
diffusion models operating directly in data space. These findings highlight latent
diffusion as an effective and scalable solution for synthetic IoT attack data generation, substantially mitigating the impact of class imbalance in ML-based IDSs for IoT scenarios.
Read the full abstract ▾
Machine Learning
Auto-Regressive Masked Diffusion Models
Mahdi Karami and Ali Ghodsi
Masked
diffusion models (MDMs) have emerged as a promising approach for
language modeling, yet they face a performance gap compared to autoregressive models (ARMs) and require more training iterations. In this work, we present the Auto-Regressive Masked
Diffusion (ARMD) model, an architecture designed to close this gap by unifying the training efficiency of autoregressive models with the parallel generation capabilities of
diffusion-based models. Our key insight is to reframe the masked
diffusion process as a block-wise causal model. This perspective allows us to design a strictly causal, permutation-equivariant architecture that computes all conditional probabilities across multiple denoising steps in a single, parallel forward pass. The resulting architecture supports efficient, autoregressive-style decoding and a progressive permutation training scheme, allowing the model to learn both canonical left-to-right and random
token orderings. Leveraging this flexibility, we introduce a novel strided parallel generation strategy that accelerates inference by generating
tokens in parallel streams while maintaining global coherence. Empirical results demonstrate that ARMD achieves state-of-the-art performance on standard
language modeling
benchmarks, outperforming established
diffusion baselines while requiring significantly fewer training steps. Furthermore, it establishes a new
benchmark for parallel text generation, effectively bridging the performance gap between parallel and sequential decoding.
Read the full abstract ▾
Machine Learning
3D Molecule Generation from Rigid Motifs via SE(3) Flows
Roman Poletukhin, Marcel Kollovieh, Eike Eberhard et al. (+1 more)
Three-dimensional molecular structure generation is typically performed at the level of individual atoms, yet molecular graph generation techniques often consider fragments as their structural units. Building on the advances in frame-based
protein structure generation, we extend these fragmentation ideas to 3D, treating general molecules as sets of rigid-body motifs. Utilising this representation, we employ SE(3)-equivariant generative modelling for de novo 3D molecule generation from rigid motifs. In our evaluations, we observe results competitive with state-of-the-art bond-implicit methods, surpassing them in atom stability on GEOM-Drugs, while yielding a 2x to 10x reduction in generation steps and 3.5x compression in molecular representations compared to standard atom-based methods. We release our code at https://github.com/rpoletukhin/MotiFlow.
Read the full abstract ▾
All authors Roman Poletukhin, Marcel Kollovieh, Eike Eberhard, Stephan Günnemann
Machine Learning
Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles
Anton Zamyatin, Patrick Indri, Sagar Malhotra et al. (+1 more)
In resource-constrained and low-latency settings, uncertainty estimates must be efficiently obtained. Deep Ensembles provide robust epistemic uncertainty (EU) but require training multiple full-size models. BatchEnsemble aims to deliver ensemble-like EU at far lower parameter and
memory cost by applying learned rank-1 perturbations to a shared base network. We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline in terms of accuracy, calibration and out-of-distribution (OOD) detection on CIFAR10/10C/SVHN. A controlled study on MNIST finds members are near-identical in function and parameter space, indicating limited capacity to realize distinct predictive modes. Thus, BatchEnsemble behaves more like a single model than a true ensemble.
Read the full abstract ▾
All authors Anton Zamyatin, Patrick Indri, Sagar Malhotra, Thomas Gärtner
Machine Learning · Cross-listed from cs.CV
Reward-Forcing: Autoregressive Video Generation with Reward Feedback
Jingran Zhang, Ning Li, Yuanhao Ban et al. (+2 more)
While most prior work in
video generation relies on bidirectional architectures, recent efforts have sought to adapt these models into autoregressive variants to support near real-time generation. However, such adaptations often depend heavily on teacher models, which can limit performance, particularly in the absence of a strong autoregressive teacher, resulting in output quality that typically lags behind their bidirectional counterparts. In this paper, we explore an alternative approach that uses reward signals to guide the generation process, enabling more efficient and scalable autoregressive generation. By using reward signals to guide the model, our method simplifies training while preserving high visual fidelity and temporal consistency. Through extensive experiments on standard
benchmarks, we find that our approach performs comparably to existing autoregressive models and, in some cases, surpasses similarly sized bidirectional models by avoiding constraints imposed by teacher architectures. For example, on VBench, our method achieves a total score of 84.92, closely matching state-of-the-art autoregressive methods that score 84.31 but require significant heterogeneous
distillation.
Read the full abstract ▾
All authors Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui
Machine Learning
Group-realizable multi-group learning by minimizing empirical risk
Navid Ardeshir, Samuel Deng, Daniel Hsu et al. (+1 more)
The sample complexity of multi-group learning is shown to improve in the group-realizable setting over the agnostic setting, even when the family of groups is infinite so long as it has finite VC dimension. The improved sample complexity is obtained by empirical risk minimization over the class of group-realizable concepts, which itself could have infinite VC dimension. Implementing this approach is also shown to be computationally intractable, and an alternative approach is suggested based on improper learning.
Read the full abstract ▾
All authors Navid Ardeshir, Samuel Deng, Daniel Hsu, Jingwen Liu
Machine Learning
What Does a Cosine of 0.8 Mean? Monotone Calibration of Embedding Similarity
Nicolas Tacheny
Text
embedding models turn sentences into vectors, and the cosine between two vectors is the usual measure of their similarity. Search, semantic caching and deduplication often accept a pair when this cosine reaches a fixed threshold. When people judge one pair of sentences more similar than another, the cosine usually agrees, but its value means different things for different models, because each model packs its scores into its own narrow range. The same threshold can therefore keep most pairs with one model and few with another. We propose to calibrate the score instead of changing the model. An increasing function fitted on sentence pairs rated by people translates the cosine of each model into the human rating scale and never reverses the order of two pairs. We fit three such functions for six models, ranging from a simple S-shaped curve to a free step function. On new pairs, calibrated scores agree with human ratings and the models agree with one another. One threshold selects pairs of similar quality whatever the model. The scale obtained belongs to the human ratings used to fit it, so its values shift on two
datasets rated differently while the models stay comparable.
Read the full abstract ▾
Machine Learning
The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning
Calarina Muslimani, Yunshu Du, Kenta Kawamoto et al. (+3 more)
The success of RL is fundamentally tied to having a reward function that accurately reflects the task objective. Yet, designing reward functions is time-consuming and prone to misspecification. To address this, our first goal is to understand how to support RL practitioners in specifying reward functions. We leverage the Trajectory
Alignment Coefficient (TAC), a metric that evaluates how closely a reward function's induced preferences match those of a domain expert. To test whether TAC provides meaningful support, we evaluated its impact in a user study where practitioners designed reward functions for Lunar Lander. We found that iterative TAC feedback allowed participants to design more performant reward functions while reporting lower cognitive workload than unguided reward design. However, the study also underscored that manual reward design, even with TAC, remains labor-intensive. This motivated our second goal: to learn a reward model that maximizes TAC directly. Specifically, we propose Soft-TAC, a differentiable approximation of TAC that can be used as a loss function to train reward models from human preferences. Our theoretical analysis guarantees that under Soft-TAC optimization: (1) the optimal reward function belongs exclusively to the set of human-aligned reward functions, and (2) this optimal reward remains invariant whether trained on clean or noisy preference data. Empirically, we first evaluate reward models trained with Soft-TAC across three robotic tasks, demonstrating that they consistently match or outperform those trained with the Cross-
Entropy loss. Next, we evaluate Soft-TAC in the GT7 driving simulator, and find that Soft-TAC-trained reward models more successfully capture preference-specific driving styles. Ultimately, this work demonstrates that TAC can serve as both a practical tool for guiding reward design and a reward learning objective in complex domains.
Read the full abstract ▾
All authors Calarina Muslimani, Yunshu Du, Kenta Kawamoto, Kaushik Subramanian, Peter Stone, Peter Wurman
Machine Learning
GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints
Andy Zhu, Rongzhe Wei, Yupu Gu et al. (+1 more)
Machine unlearning in Mixture-of-Experts (MoE) large
language models presents a critical yet under-explored challenge. Current unlearning methods applied to MoE architectures often exploit dynamic routing as an optimization shortcut: rather than genuinely erasing knowledge from expert parameters, they manipulate routers to redirect queries away from the originally assigned experts. This not only causes severe utility degradation but also leaves hazardous knowledge intact. Consequently, adversaries can bypass the router to recover sensitive information directly from dormant experts. In this study, we propose
Geometric Routing Invariance Preservation (GRIP), an
algorithm-agnostic framework that resolves these failure modes by enforcing hard
geometric constraints on router updates. By projecting router gradient updates into the null space of the retain set's routing
matrix, GRIP suppresses routing manipulation without freezing the router entirely, thereby directing the unlearning pressure into the expert parameters themselves across all relevant experts. GRIP offers two complementary variants: training-time
stochastic projection and a post-training closed-form analytical correction. Extensive experiments on two MoE models across hazardous knowledge removal and copyright unlearning
benchmarks demonstrate that GRIP restores routing stability from 0.21 to >0.94, improves retain accuracy by up to 89%, and reduces white-box
adversarial knowledge recovery from 11% to just 3% while in line with dense-architecture unlearning under black-box
prompt attack, establishing
geometric constraints as a principled solution for genuine unlearning in sparse MoE architectures.
Read the full abstract ▾
All authors Andy Zhu, Rongzhe Wei, Yupu Gu, Pan Li
Machine Learning
Embedding -based Crop Type Classification in the Groundnut Basin of Senegal
Madeline C. Lisaius, Srinivasan Keshav, Andrew Blake et al. (+1 more)
Crop type maps from satellite remote sensing are important tools for food security, local livelihood support and
climate change mitigation in smallholder regions of the world, but most satellite-based methods are not well suited to smallholder conditions. To address this gap, we establish a four-part criteria for a useful
embedding-based approach consisting of 1) performance, 2) plausibility, 3) transferability and 4) accessibility and evaluate geospatial foundation model (FM)
embeddings -based approaches using TESSERA and AlphaEarth against current baseline methods for a region in the groundnut basin of Senegal. We find that the TESSERA -based approach to land cover and crop type mapping fulfills the selection criteria best, and in one temporal transfer example shows 28% higher accuracy compared to the next best method. These results indicate that TESSERA
embeddings are an effective approach for crop type classification and mapping tasks in Senegal.
Read the full abstract ▾
All authors Madeline C. Lisaius, Srinivasan Keshav, Andrew Blake, Clement Atzberger
Machine Learning
Variance Reduction for Stochastic Switching in Functionally Constrained Federated Optimization
Antesh Upadhyay, Sang Bin Moon, Zhankun Luo et al. (+1 more)
Federated learning (FL) is increasingly used in safety-critical and
fairness-sensitive applications, where learning objectives must be optimized subject to functional constraints. This setting is challenging due to costly projection onto the resulting feasible set,
stochastic constraint information, partial client participation, and limited communication at edge devices. We propose a
federated stochastic switching framework based on primal-only updates, avoiding dual-variable tuning for functionally constrained
federated optimization. For convex objectives, the framework further supports compression with error feedback to control compression-induced bias. We establish high-probability convergence and feasibility guarantees under
stochastic gradients and partial client participation, and extend the analysis to weakly convex objectives through proximal-stationarity guarantees. Our bounds isolate the effects of
stochastic gradients, constraint-evaluation noise, and partial participation, and show that direct switching can retain a nonvanishing constraint-estimation error even with exact local constraint evaluations. To overcome this limitation, we develop variance-reducing constraint
estimators that yield vanishing feasibility and stationarity guarantees with fixed per-round client participation and constraint-evaluation batch sizes. Experiments on Neyman-Pearson classification,
fairness-constrained learning on the ADULT
dataset, and constrained deep reinforcement learning demonstrate the effectiveness of the proposed framework.
Read the full abstract ▾
All authors Antesh Upadhyay, Sang Bin Moon, Zhankun Luo, Abolfazl Hashemi
Machine Learning · Cross-listed from cs.CL
LLM-Based Adversarial Persuasion Attacks on Fact-Checking Systems
João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva et al. (+1 more)
Automated fact-checking (AFC) systems are susceptible to
adversarial attacks, enabling false claims to evade detection. Existing
adversarial frameworks typically rely on injecting noise or altering semantics, yet no existing framework exploits the
adversarial potential of persuasion techniques against AFC systems, which are widely used in disinformation campaigns to manipulate audiences. In this paper, we introduce a novel class of persuasive
adversarial attacks on AFCs by employing an LLM to rephrase claims using persuasion techniques. Considering $15$ techniques grouped into $5$ categories, we study the effects of persuasion on both claim verification and evidence
retrieval using a decoupled evaluation strategy. Experiments on the FEVER and FEVEROUS
benchmarks show that persuasion attacks can substantially degrade both verification performance and evidence
retrieval. Our analysis identifies persuasion techniques as a potent class of
adversarial attacks, highlighting the need for more robust AFC systems.
Read the full abstract ▾
All authors João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, Carolina Scarton
Machine Learning
Multigrade Neural Network Approximation
Shijun Zhang, Zuowei Shen and Yuesheng Xu
We study multigrade deep learning (MGDL) as a principled framework for structured error refinement in deep neural networks. While the approximation power of neural networks is now relatively well understood, training very deep architectures remains challenging due to highly nonconvex and often ill-conditioned optimization landscapes. In contrast, for relatively shallow networks, most notably certain one-hidden-layer ReLU models, training admits convex reformulations with global guarantees under appropriate settings, motivating learning paradigms that improve stability while scaling to depth. MGDL builds on this insight by training deep networks grade by grade: previously learned grades are frozen, and each newly added grade-wise subnetwork is composed on top of the previously learned grades and trained to fit the residual left by the current approximation, yielding a structured and interpretable hierarchical refinement process. We develop an operator-theoretic foundation for MGDL and prove that, for any continuous target function defined on a hypercube, there exists a fixed-width multigrade ReLU scheme whose residuals are pointwise nonincreasing in magnitude and converge uniformly to zero, with strict $L^p$-norm decay at every nontrivial grade for $p\in [1,\infty)$. To the best of our knowledge, this work provides the first rigorous constructive approximation guarantee showing that a grade-wise residual refinement scheme can achieve vanishing error in a fixed-width multigrade ReLU architecture.
Read the full abstract ▾
Machine Learning
Theory of Minimal Weight Perturbations in Deep Networks and its Applications for Low-Rank Activated Backdoor Attacks
Bethan Evans and Jared Tanner
The minimal norm weight perturbations of DNNs required to achieve a specified change in output are derived and the factors determining its size are discussed. These single-layer exact formulae are contrasted with more generic multi-layer Lipschitz constant based robustness guarantees; both are observed to be of the same order which indicates similar efficacy in their guarantees. These results are applied to precision-modification-activated
backdoor attacks, establishing provable compression thresholds below which such attacks cannot succeed, and show empirically that low-rank compression can reliably activate latent
backdoors while preserving full-precision accuracy. These expressions reveal how back-propagated margins govern layer-wise sensitivity and provide certifiable guarantees on the smallest parameter updates consistent with a desired output shift.
Read the full abstract ▾
Machine Learning
Provably Learning Attention with Queries
Satwik Bhattamishra, Kulin Shah, Michael Hahn et al. (+1 more)
We study the problem of learning
Transformer-based sequence models with black-box access to their outputs. In this setting, a learner may adaptively query the oracle with any sequence of vectors and observe the output of the target function. We begin with studying the learnability of the simplest formulation, that is, learning a single-head
attention-based regressor with queries. We show that for a model with width $d$, there is an elementary
algorithm to learn the parameters of single-head
attention with $O(d^2)$ queries. Further, we show that if there exists an
algorithm to learn ReLU feedforward networks (FFNs), then the single-head
algorithm can be easily adapted to learn one-layer
Transformers with single-head
attention. Next, we show that, in the common regime where the head dimension $r \ll d$, single-head
attention-based models can be learned with $O(rd)$ queries via compressed sensing arguments. We also study robustness to noisy oracle access, proving that under mild norm and margin conditions, the parameters can be estimated to $\varepsilon$ accuracy with a
polynomial number of queries even when outputs are only provided up to additive tolerance. Finally, we consider the learnability of multi-head
attention and show that they are not identifiable from queries, and hence, learnability in the same sense is not feasible without additional assumptions. We discuss potential approaches to learn multi-head
attention-based models under certain structural assumptions.
Read the full abstract ▾
All authors Satwik Bhattamishra, Kulin Shah, Michael Hahn, Varun Kanade