Machine Learning
PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training
Yanyi Li, Yimu Zhang and Cong Fang
Activations have become the primary
memory bottleneck in large-batch LLM training. However, existing compression methods fail to exploit the spectral structure of activations, resulting in slow convergence or limited compression. To address this, we bridge the relationship between the
algorithm's fast convergence and the requirements for subspace projection, and show that an effective compression should yield an unbiased estimate of the original activation with low variance. We propose Principal-Random Subspace for LLM Activation Compression (PRAC), which novelly decomposes activations into two components: a principal subspace captured via SVD to retain dominant information, and a random subspace sampled from the orthogonal complement to approximate the tail. By introducing a precise scaling factor, we prove that PRAC yields an unbiased gradient
estimator with minimum variance under certain conditions. Extensive experiments on pre-training and fine-tuning tasks demonstrate that PRAC achieves up to 36% total
memory reduction with negligible performance degradation and minimal computational cost.
Read the full abstract ▾
Machine Learning
Physics-informed neural particle flow for the Bayesian update step
Domonkos Csuzdi, Tamás Bécsi and Olivér Törő
The
Bayesian update step poses significant computational challenges in high-dimensional nonlinear estimation. While log-homotopy particle flow filters offer an alternative to
stochastic sampling, existing formulations usually yield stiff differential equations. Conversely, existing deep learning approximations typically treat the update as a black-box task or rely on asymptotic relaxation, neglecting the exact
geometric structure of the finite-horizon probability transport. In this work, we propose a physics-informed neural particle flow, which is an amortized inference framework. To construct the flow, we couple the log-homotopy trajectory of the prior to posterior density function with the continuity equation describing the density
evolution. This derivation yields a governing partial differential equation (PDE), referred to as the master PDE. By
embedding this PDE as a physical constraint into the loss function, we train a neural network to approximate the transport velocity field. This approach enables purely unsupervised training, eliminating the need for ground-truth posterior samples. We demonstrate that the neural parameterization acts as an implicit regularizer, mitigating the numerical stiffness inherent to analytic flows and reducing online computational complexity. Experimental validation on
multimodal benchmarks and a challenging nonlinear scenario confirms better mode coverage and robustness compared to state-of-the-art baselines.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
Q-Tag: Watermarking Quantum Circuit Generative Models
Yang Yang, Yuzhu Long, Han Fang et al. (+4 more)
Quantum cloud platforms have become the most widely adopted and mainstream approach for accessing
quantum computing resources, due to the scarcity and operational complexity of
quantum hardware. In this service-oriented paradigm,
quantum circuits, which constitute high-value intellectual property, are exposed to risks of unauthorized access, reuse, and misuse. Digital watermarking has been explored as a promising mechanism for protecting
quantum circuits by
embedding ownership information for tracing and verification. However, driven by recent advances in generative artificial intelligence, the paradigm of
quantum circuit design is shifting from individually and manually constructed
circuits to automated synthesis based on
quantum circuit generative models (QCGMs). In such generative settings, protecting only individual output
circuits is insufficient, and existing post hoc,
circuit-centric watermarking methods are not designed to integrate with the generative process, often failing to simultaneously ensure stealthiness, functional correctness, and robustness at scale. These limitations highlight the need for a new watermarking paradigm that is natively integrated with
quantum circuit generative models. In this work, we present the first watermarking framework for QCGMs, which embeds ownership signals into the generation process while preserving
circuit fidelity. We introduce a symmetric sampling strategy that aligns watermark encoding with the model's Gaussian prior, and a synchronization mechanism that counteracts
adversarial watermark attack through latent drift correction. Empirical results confirm that our method achieves high-fidelity
circuit generation and robust watermark detection across a range of perturbations, paving the way for scalable, secure copyright protection in AI-powered
quantum design.
Read the full abstract ▾
All authors Yang Yang, Yuzhu Long, Han Fang, Zhaoyun Chen, Zhonghui Li, Weiming Zhang, Guoping Guo
Machine Learning · Cross-listed from cs.CL
Assessing Deanonymization Risks with Stylometry-Assisted LLM Agent
Boyang Zhang and Yang Zhang
The rapid advancement of large
language models (LLMs) has enabled powerful authorship inference capabilities, raising growing concerns about unintended deanonymization risks in textual data such as news articles. In this work, we introduce an LLM
agent designed to evaluate and mitigate such risks through a structured, interpretable pipeline. Central to our framework is the proposed $\textit{SALA}$ (Stylometry-Assisted LLM Analysis) method, which integrates quantitative stylometric features with LLM
reasoning for robust and transparent authorship attribution. Experiments on large-scale news
datasets demonstrate that $\textit{SALA}$, particularly when augmented with a database module, achieves high inference accuracy in various scenarios. Finally, we propose a guided recomposition strategy that leverages the
agent's
reasoning trace to generate rewriting
prompts, effectively reducing authorship identifiability while preserving textual meaning. Our findings highlight both the deanonymization potential of LLM
agents and the importance of interpretable, proactive defenses for safeguarding author
privacy.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
MoDora: Tree-Based Semi-Structured Document Analysis System
Bangrui Xu, Qihang Yao, Zirui Tang et al. (+8 more)
Semi-structured documents integrate diverse interleaved data elements (e.g., tables, charts, hierarchical paragraphs) arranged in various and often irregular layouts. These documents are widely observed across domains and account for a large portion of real-world data. However, existing methods struggle to support natural
language question answering over these documents due to three main technical challenges: (1) The elements extracted by techniques like OCR are often fragmented and stripped of their original semantic context, making them inadequate for analysis. (2) Existing approaches lack effective representations to capture hierarchical structures within documents (e.g., associating tables with nested chapter titles) and to preserve layout-specific distinctions (e.g., differentiating sidebars from main content). (3) Answering questions often requires retrieving and aligning relevant information scattered across multiple regions or pages, such as linking a descriptive paragraph to table
cells located elsewhere in the document. To address these issues, we propose MoDora, an LLM-powered system for semi-structured document analysis. First, we adopt a local-
alignment aggregation strategy to convert OCR-parsed elements into layout-aware components, and conduct type-specific information extraction for components with hierarchical titles or non-text elements. Second, we design the Component-Correlation Tree (CCTree) to hierarchically organize components, explicitly modeling inter-component relations and layout distinctions through a bottom-up cascade summarization process. Finally, we propose a question-type-aware
retrieval strategy that supports (1) layout-based grid partitioning for location-based
retrieval and (2) LLM-guided
pruning for semantic-based
retrieval. Experiments show MoDora outperforms baselines by 5.97%-61.07% in accuracy. The code is at https://github.com/weAIDB/MoDora.
Read the full abstract ▾
All authors Bangrui Xu, Qihang Yao, Zirui Tang, Xuanhe Zhou, Yeye He, Shihan Yu, Qianqian Xu, Bin Wang, Guoliang Li, Conghui He, Fan Wu
Machine Learning
RhythmBERT: A Self-Supervised Language Model Based on Latent Representations of ECG Waveforms for Heart Disease Detection
Xin Wang, Burcu Ozek, Aruna Mohan et al. (+3 more)
Electrocardiogram (ECG) analysis is crucial for diagnosing heart disease, but most self-supervised learning methods treat ECG as a generic time series, overlooking physiologic semantics and rhythm-level structure. Existing contrastive methods utilize augmentations that distort morphology, whereas generative approaches employ fixed-window
segmentation, which misaligns cardiac cycles. To address these limitations, we propose RhythmBERT, a generative ECG
language model that considers ECG as a
language paradigm by encoding P, QRS, and T segments into symbolic
tokens via autoencoder-based latent representations. These discrete
tokens capture rhythm semantics, while complementary continuous
embeddings retain fine-grained morphology, enabling a unified view of waveform structure and rhythm. RhythmBERT is pretrained on approximately 800,000 unlabeled ECG recordings with a masked prediction objective, allowing it to learn contextual representations in a label-efficient manner. Evaluations show that despite using only a single lead, RhythmBERT achieves comparable or superior performance to strong 12-lead baselines. This generalization extends from prevalent conditions such as atrial fibrillation to clinically challenging cases such as subtle ST-T abnormalities and myocardial infarction. Our results suggest that considering ECG as structured
language offers a scalable and physiologically aligned pathway for advancing cardiac analysis.
Read the full abstract ▾
All authors Xin Wang, Burcu Ozek, Aruna Mohan, Amirhossein Ravari, Or Zilbershot, Fatemeh Afghah
Machine Learning
Latent Matters: Learning Deep State-Space Models
Alexej Klushyn, Richard Kurle, Maximilian Soelch et al. (+2 more)
Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic
Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.
Read the full abstract ▾
All authors Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, Patrick van der Smagt
Machine Learning
Learning Disease-Sensitive Latent Interaction Graphs From Noisy Cardiac Flow Measurements
Viraj Patel, Marko Grujic, Philipp Aigner et al. (+4 more)
Cardiac blood flow patterns contain rich information about disease severity and clinical interventions, yet current imaging and computational methods fail to capture underlying relational structures of coherent flow features. We propose a physics-informed, latent relational framework to model cardiac
vortices as interacting nodes in a graph. Our model combines a neural relational inference architecture with physics-inspired interaction energy and birth-death dynamics, yielding a latent graph sensitive to disease severity and intervention level. We first develop the method using computational fluid dynamics simulations of aortic coarctation, where learned interaction graphs reveal increasingly structured
vortex interactions as vessel narrowing progresses. The resulting graph
entropy exhibits a strong monotonic relationship with coarctation severity ($R^2=0.78$, Spearman $|\rho|=0.96$). We then evaluate the framework on fluid-structure interaction simulations of intracranial aneurysms using multiple
geometric severity descriptors. Among these, aneurysm volume produces the most informative latent representation, with non-interaction graph
entropy demonstrating a strong monotonic relationship with severity (Spearman $|\rho| = 0.91$) and generalising to several alternative morphological measures. Finally, we apply the approach to ultrasound-derived flow fields of a left ventricle under varying levels of left ventricular assist device support, where the latent graph captures the progressive loss of coherent
vortex interactions under mechanical assistance, demonstrating cross-modal generalisation to imaging data. Across all
datasets, latent interaction graphs and graph
entropy provide interpretable markers of disease severity and intervention, linking haemodynamic organisation to clinically relevant physiological changes.
Read the full abstract ▾
All authors Viraj Patel, Marko Grujic, Philipp Aigner, Theodor Abart, Marcus Granegger, Deblina Bhattacharjee, Katharine Fraser
Machine Learning · Cross-listed from math.ST
Low-degree Lower bounds for clustering in moderate dimension
Alexandra Carpentier and Nicolas Verzelen
We study the fundamental problem of clustering $n$ points into $K$ groups drawn from a mixture of isotropic Gaussians in $\mathbb{R}^d$. Specifically, we investigate the requisite minimal distance $\Delta$ between mean vectors to partially recover the underlying partition. While the minimax-optimal threshold for $\Delta$ is well-established, a significant gap exists between this information-theoretic limit and the performance of known
polynomial-time procedures. Although this gap was recently characterized in the high-dimensional regime ($n \leq dK$), it remains largely unexplored in the moderate-dimensional regime ($n \geq dK$). In this manuscript, we address this regime by establishing a new low-degree
polynomial lower bound for the moderate-dimensional case when $d \geq K$. We show that while the difficulty of clustering for $n \leq dK$ is primarily driven by dimension reduction and spectral methods, the moderate-dimensional regime involves more delicate phenomena leading to a "non-parametric rate". We provide a novel non-spectral
algorithm matching this rate, shedding new light on the computational limits of the clustering problem in moderate dimension.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SubspaceAD: Training-Free Few-Shot Anomaly Detection via Subspace Modeling
Camile Lendering, Erkut Akdag and Egor Bondarev
Detecting visual anomalies in industrial inspection often requires training with only a few normal images per category. Recent few-shot methods achieve strong results employing foundation-model features, but typically rely on
memory banks, auxiliary
datasets, or multi-modal tuning of
vision-
language models. We therefore question whether such complexity is necessary given the feature representations of
vision foundation models. To answer this question, we introduce SubspaceAD, a training-free method, that operates in two simple stages. First, patch-level features are extracted from a small set of normal images by a frozen DINOv2 backbone. Second, a Principal Component Analysis (PCA) model is fit to these features to estimate the low-dimensional subspace of normal variations. At inference, anomalies are detected via the reconstruction residual with respect to this subspace, producing interpretable and statistically grounded anomaly scores. Despite its simplicity, SubspaceAD achieves state-of-the-art performance across one-shot and few-shot settings without training,
prompt tuning, or
memory banks. In the one-shot anomaly detection setting, SubspaceAD achieves image-level and pixel-level AUROC of 97.1% and 97.5% on the MVTec-AD
dataset, and 93.2% and 98.2% on the VisA
dataset, respectively, surpassing prior state-of-the-art results. Code and demo are available at https://github.com/CLendering/SubspaceAD.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
RQ-Reg: A Residual-Quantization-Based Framework for Continuous Value Prediction in Recommender Systems
Runpeng Cui, Zhipeng Sun, Chi Lu et al. (+1 more)
Predicting continuous values such as watch-time and gross merchandise value (GMV) is a core problem in industrial recommendation systems. Its inherent difficulty stems from the highly complex and long-tailed distributions of the target signals, which are hard to model accurately. Existing regression methods typically rely on fixed parametric assumptions on the target distribution: overly simple assumptions underfit real-world data, whereas more intricate ones tend to sacrifice scalability and generalization. To address these limitations, we propose a sequence modeling framework based on residual quantization (RQ), in which the target continuous value is decomposed into a sequence of quantization codes that represent progressively finer approximations. The model autoregressively predicts these codes from coarse to fine granularity, with each step refining the residual error left by the previous one. To further improve the quality of the learned representations, we introduce an ordinal-aware representation learning objective that aligns the RQ code
embedding space with the ordinal structure of target values, thereby yielding continuous representations of quantization codes and more accurate predictions. We conduct comprehensive experiments on public
benchmarks for watch-time and lifetime value (LTV) prediction, together with a large-scale online A/B test for GMV prediction on an industrial short-
video recommendation platform. Across all settings, the proposed method shows competitive performance among existing state-of-the-art approaches and generalizes well across diverse continuous value prediction scenarios.
Read the full abstract ▾
All authors Runpeng Cui, Zhipeng Sun, Chi Lu, Peng Jiang
Machine Learning
Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Zeyuan Liu, Jeonghye Kim, Xufang Luo et al. (+2 more)
Exploration remains the key bottleneck for large
language model
agents trained with reinforcement learning. While prior methods exploit pretrained knowledge, they fail in environments requiring the discovery of novel states. We propose Exploratory
Memory-Augmented On- and Off-Policy Optimization (EMPO$^2$), a hybrid RL framework that leverages
memory for exploration and combines on- and off-policy updates to make LLMs perform well with
memory while also ensuring robustness without it. On ScienceWorld and WebShop, EMPO$^2$ achieves 128.6% and 11.3% improvements over GRPO, respectively. Moreover, in out-of-distribution tests, EMPO$^2$ demonstrates superior adaptability to new tasks, requiring only a few trials with
memory and no parameter updates. These results highlight EMPO$^2$ as a promising framework for building more exploratory and generalizable LLM-based
agents.
Read the full abstract ▾
All authors Zeyuan Liu, Jeonghye Kim, Xufang Luo, Dongsheng Li, Yuqing Yang
Machine Learning · Cross-listed from stat.ML
Regular Fourier Features for Nonstationary Gaussian Processes
Arsalan Jawaid, Abdullah Karatas and Jörg Seewig
Simulating a Gaussian process requires sampling from a high-dimensional Gaussian distribution, which scales cubically with the number of sample locations. Spectral methods address this challenge by exploiting the
Fourier representation and treating the spectral density as a probability distribution suitable for Monte Carlo approximation. Although this probabilistic interpretation is valid for stationary processes, it is overly restrictive for the nonstationary case, where spectral densities are generally not probability measures. To avoid this limitation, we propose regular
Fourier features for harmonizable processes with one-dimensional inputs. Our method discretizes the spectral representation directly, preserving the correlation structure among spectral weights without requiring probability assumptions. Assuming finite spectral support, this yields an efficient low-rank approximation that is positive semi-definite by construction and consistent under mild regularity conditions. When the spectral density is unknown, the framework also extends to kernel learning from data, which we explore as a proof of concept. We demonstrate the approximation on locally stationary and harmonizable mixture kernels, the latter with a complex-valued spectral density. As a feasibility study, we then apply the kernel-learning extension to real and synthetic data, where it matches competitive baselines.
Read the full abstract ▾
Machine Learning
Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
Bum Jun Kim, Shohei Taniguchi, Makoto Kawano et al. (+2 more)
Training divergence in
transformers wastes compute, yet practitioners discover instability only after expensive runs begin. They therefore need an expected probability of failure for a
transformer before training starts. Our study of Residual Koopman Spectral Profiling (RKSP) provides such an estimate. From a single forward pass at initialization, RKSP extracts Koopman spectral features by applying whitened dynamic mode decomposition to layer-wise residual snapshots. Our central diagnostic, the near-unit spectral mass, quantifies the fraction of modes concentrated near the unit circle, which captures instability risk. For predicting divergence across extensive configurations, this
estimator achieves an AUROC of 0.995, outperforming the best gradient baseline. We further make this diagnostic actionable through Koopman Spectral Shaping (KSS), which reshapes spectra during training. We empirically validate that our method works in practice: RKSP predicts divergence at initialization, and when RKSP flags high risk, turning on KSS successfully prevents divergence. In the challenging high learning rate regime without normalization layers, KSS reduces the divergence rate from 66.7% to 12.5% and enables learning rates that are 50% to 150% higher. These findings generalize to WikiText-103
language modeling,
vision transformers on CIFAR-10, and pretrained
language models, including GPT-2 and LLaMA-2 up to 7B, as well as emerging architectures such as MoE, Mamba-style SSMs, and KAN.
Read the full abstract ▾
All authors Bum Jun Kim, Shohei Taniguchi, Makoto Kawano, Yusuke Iwasawa, Yutaka Matsuo
Machine Learning · Cross-listed from stat.ML
Kernel Integrated $R^2$: A Measure of Dependence
Pouya Roudaki, Shakeel Gavioli-Akilagun, Florian Kalinke et al. (+2 more)
We introduce kernel integrated $R^2$, a new measure of statistical dependence that combines the local normalization principle of the recently introduced integrated $R^2$ with the flexibility of reproducing kernel
Hilbert spaces (RKHSs). The proposed measure extends integrated $R^2$ from scalar responses to responses taking values on general spaces equipped with a characteristic kernel, allowing to measure dependence of multivariate, functional, and structured data, while remaining sensitive to tail behaviour and oscillatory dependence structures. We establish that (i) this new measure takes values in $[0,1]$, (ii) equals zero if and only if independence holds, and (iii) equals one if and only if the response is almost surely a measurable function of the
covariates. Two
estimators are proposed: a graph-based method using $K$-nearest neighbours and an RKHS-based method built on conditional mean
embeddings. We prove consistency and derive convergence rates for the graph-based
estimator, showing its adaptation to intrinsic dimensionality. Numerical experiments on simulated data and a real data experiment in the context of dependency testing for media annotations demonstrate competitive power against state-of-the-art dependence measures, particularly in settings involving non-linear and structured relationships.
Read the full abstract ▾
All authors Pouya Roudaki, Shakeel Gavioli-Akilagun, Florian Kalinke, Mona Azadkia, Zoltán Szabó
Machine Learning
Scaling Laws of Global Weather Models
Yuejiang Yu, Langwen Huang, Alexandru Calotoiu et al. (+1 more)
Data-driven models are revolutionizing weather
forecasting. To optimize training efficiency and model performance, this paper analyzes empirical scaling laws within this domain. We investigate the relationship between model performance (validation loss) and three key factors: model size ($N$),
dataset size ($D$), and compute budget ($C$). Across a range of models, we find that Aurora exhibits the strongest data-scaling behavior: increasing the training
dataset by 10x reduces validation loss by up to 3.2x. GraphCast demonstrates the highest parameter efficiency, yet suffers from limited hardware utilization. Our compute-optimal analysis indicates that, under fixed compute budgets, allocating resources to more total training data yields greater performance gains than increasing model size. Furthermore, we analyze model shape and uncover scaling behaviors that differ fundamentally from those observed in
language models: weather
forecasting models consistently favor increased width over depth. These findings suggest that future weather models should prioritize wider architectures and larger effective training
datasets to maximize predictive performance.
Read the full abstract ▾
All authors Yuejiang Yu, Langwen Huang, Alexandru Calotoiu, Torsten Hoefler
Machine Learning · Cross-listed from cs.CV
pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation
Shentong Mo, Xufang Luo and Dongsheng Li
Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and
segmentation. Typically,
prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts
prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized
prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific
prompt tokens and utilizes a dynamic
token dispatching mechanism at various
prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and
segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.
Read the full abstract ▾
Machine Learning
MSINO: Curvature-Aware Sobolev Optimization for Manifold Neural Networks
Suresan Pareth
We introduce
Manifold Sobolev Informed Neural Optimization (MSINO), a
curvature aware training framework for neural networks defined on Riemannian
manifolds. The method replaces standard Euclidean derivative supervision with a covariant Sobolev loss that aligns gradients using parallel transport and improves stability via a Laplace Beltrami smoothness regularization term. Building on classical results in Riemannian optimization and Sobolev theory on
manifolds, we derive geometry dependent constants that yield (i) a Descent Lemma with a
manifold Sobolev smoothness constant, (ii) a Sobolev Polyak Lojasiewicz inequality giving linear convergence guarantees for Riemannian gradient descent and
stochastic gradient descent under explicit step size bounds, and (iii) a two step Newton Sobolev method with local quadratic contraction in
curvature controlled neighborhoods. Unlike prior Sobolev training in Euclidean space, MSINO provides training time guarantees that explicitly track
curvature and transported Jacobians. Applications include surface imaging, physics informed learning settings, and robotics on Lie groups such as SO(3) and SE(3). The framework unifies value and gradient based learning with
curvature aware convergence guarantees for neural training on
manifolds.
Read the full abstract ▾
Machine Learning
Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks
Wenquan Ma, Yang Sui, Jiaye Teng et al. (+3 more)
Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize $\eta_t = \mathcal{O}(1/t)$ under non-convex training regimes, where $t$ denotes iterations. This rigid decay of the stepsize potentially impedes optimization and may not align with practical scenarios. In this paper, we derive the generalization bounds under the homogeneous neural network regimes, proving that this regime enables slower stepsize decay of order $\Omega(1/\sqrt{t})$ under mild assumptions. We further extend the theoretical results from several aspects, e.g., non-Lipschitz regimes. This finding is broadly applicable, as homogeneous neural networks encompass fully-connected and convolutional neural networks with ReLU and LeakyReLU activations.
Read the full abstract ▾
All authors Wenquan Ma, Yang Sui, Jiaye Teng, Bohan Wang, Jing Xu, Jingqin Yang
Machine Learning · Cross-listed from stat.ML
Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks
Katerina Papagiannouli, Dario Trevisan and Giuseppe Pio Zitto
We study wide
Bayesian neural networks focusing on the rare but statistically dominant fluctuations that govern posterior concentration, beyond Gaussian-process limits. Large-deviation theory provides explicit variational objectives-rate functions-on predictors, providing an emerging notion of complexity and feature learning directly at the functional level. We show that the posterior output rate function is obtained by a joint optimization over predictors and internal kernels, in contrast with fixed-kernel (NNGP) theory. Numerical experiments demonstrate that the resulting predictions accurately describe finite-width behavior for moderately sized networks, capturing non-Gaussian tails, posterior deformation, and data-dependent kernel selection effects.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
SIGMA: A Semantic-Grounded Instruction-Driven Generative Multi-Task Recommender at AliExpress
Yang Yu, Lei Kou, Huaikuan Yi et al. (+6 more)
With the rapid
evolution of Large
Language Models (LLMs), generative recommendation is gradually reshaping the paradigm of recommender systems. However, most existing methods remain confined to the interaction-driven next-item prediction paradigm, struggling to keep pace with the latest evolving trends or address the diverse recommendation tasks along with business-specific requirements in real-world scenarios. To this end, we present SIGMA, a Semantic-Grounded Instruction-Driven Generative Multi-Task Recommender deployed at AliExpress. Specifically, we first ground item entities in a unified latent space capturing both general semantics and collaborative signals. Building upon this, we introduce a hybrid item tokenization method for both precise modeling and efficient generation. Moreover, we construct a large-scale multi-task supervised fine-tuning
dataset empowering SIGMA to fulfill various recommendation demands via instruction-following. Finally, we design a three-step item generation procedure integrated with an adaptive probabilistic fusion mechanism to calibrate the output distributions based on task-specific requirements for recommendation accuracy and diversity. Extensive offline experiments and online A/B tests demonstrate the effectiveness of SIGMA across various real-world recommendation tasks.
Read the full abstract ▾
All authors Yang Yu, Lei Kou, Huaikuan Yi, Bin Chen, Yayu Cao, Lei Shen, Chao Zhang, Bing Wang, Xiaoyi Zeng
Machine Learning
CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation with Non-linearity Retained at Inference
Hung-Hsuan Chen
Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning (PEFT). However, it faces a “linear ceiling”: increasing the rank yields diminishing returns in expressive capacity due to linear constraints. We introduce CeRA (Capacity-enhanced Rank Adaptation), a weight-level parallel adapter that injects SiLU gating and dropout to induce non-linearity during inference, thereby placing it in a different function class from adapters whose non-linearity exists during training and collapses to an affine map at inference time. On both the basic arithmetic (GSM8K) and the complex MATH
benchmark, CeRA is markedly more parameter-efficient. Across a full rank $\times$ learning rate sweep, CeRA at rank 64 achieves the highest MATH pass@1 of any configuration in the grid (23.6%), matching or exceeding both a rank-512 LoRA (22.4%) and DoRA (19.8%) while using only 1/8 of the parameter budget. With the rank and learning rate fixed, CeRA equals or outperforms LoRA in 10 of 12 matched settings. Spectrally, CeRA's learned updates utilize the singular-value spectrum more broadly than linear adapters, which exhibit rank collapse at high rank, although a scale-matched control shows that this difference stems mostly from output scale and partially from non-linearity. Additionally, dropout appears to contribute to regularization rather than rank expansion. We release the code for reproducibility: https://github.com/hhchen1105/cera.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.IR
PSQE: A Theoretical-Practical Approach to Pseudo Seed Quality Enhancement for Unsupervised Multimodal Entity Alignment
Yunpeng Hong, Chenyang Bu, Jie Zhang et al. (+3 more)
Multimodal Entity
Alignment (MMEA) aims to identify equivalent entities across different data modalities, enabling structural data integration that in turn improves the performance of various large
language model applications. To lift the requirement of labeled seed pairs that are difficult to obtain, recent methods shifted to an unsupervised paradigm using pseudo-
alignment seeds. However, unsupervised entity
alignment in
multimodal settings remains underexplored, mainly because the incorporation of
multimodal information often results in imbalanced coverage of pseudo-seeds within the knowledge graph. To overcome this, we propose PSQE (Pseudo-Seed Quality Enhancement) to improve the precision and graph coverage balance of pseudo seeds via
multimodal information and clustering-resampling. Theoretical analysis reveals the impact of pseudo seeds on existing contrastive learning-based MMEA models. In particular, pseudo seeds can influence the attraction and the repulsion terms in contrastive learning at once, whereas imbalanced graph coverage causes models to prioritize high-density regions, thereby weakening their learning capability for entities in sparse regions. Experimental results validate our theoretical findings and show that PSQE as a plug-and-play module can improve the performance of baselines by considerable margins.
Read the full abstract ▾
All authors Yunpeng Hong, Chenyang Bu, Jie Zhang, Yi He, Di Wu, Xindong Wu
Machine Learning
A Data-Driven Approach to Support Clinical Renal Replacement Therapy
Alice Balboni, Luis Escobar, Andrea Manno et al. (+5 more)
This study investigates a data-driven machine learning approach to predict membrane fouling in critically ill patients undergoing Continuous Renal Replacement Therapy (CRRT). Using time-series data from an ICU, 16 clinically selected features were identified to train predictive models. To ensure interpretability and enable reliable counterfactual analysis, the researchers adopted a tabular data approach rather than modeling temporal dependencies directly. Given the imbalance between fouling and non-fouling cases, the ADASYN oversampling technique was applied to improve minority class representation. Random Forest, XGBoost, and LightGBM models were tested, achieving balanced performance with 77.6% sensitivity and 96.3% specificity at a 10% rebalancing rate. Results remained robust across different
forecasting horizons. Notably, the tabular approach outperformed LSTM recurrent neural networks, suggesting that explicit temporal modeling was not necessary for strong predictive performance. Feature selection further reduced the model to five key variables, improving simplicity and interpretability with minimal loss of accuracy. A Shapley value-based counterfactual analysis was applied to the best-performing model, successfully identifying minimal input changes capable of reversing fouling predictions. Overall, the findings support the viability of interpretable machine learning models for predicting membrane fouling during CRRT. The integration of prediction and counterfactual analysis offers practical clinical value, potentially guiding therapeutic adjustments to reduce fouling risk and improve patient management.
Read the full abstract ▾
All authors Alice Balboni, Luis Escobar, Andrea Manno, Fabrizio Rossi, Maria Cristina Ruffa, Gianluca Villa, Giordano D'Aloisio, Antonio Consolo
Machine Learning · Cross-listed from cs.AI
OmniGAIA: Towards Native Omni-Modal AI Agents
Xiaoxi Li, Wenxiang Jiao, Jiarui Jin et al. (+10 more)
Human intelligence naturally intertwines omni-modal perception – spanning
vision, audio, and
language – with complex
reasoning and tool usage to interact with the world. However, current multi-modal LLMs are primarily confined to bi-modal interactions (e.g.,
vision-
language), lacking the unified cognitive capabilities required for general AI assistants. To bridge this gap, we introduce OmniGAIA, a comprehensive
benchmark designed to evaluate omni-modal
agents on tasks necessitating deep
reasoning and multi-turn tool execution across
video, audio, and image modalities. Constructed via a novel omni-modal event graph approach, OmniGAIA synthesizes complex, multi-hop queries derived from real-world data that require cross-modal
reasoning and external tool integration. Furthermore, we propose OmniAtlas, a native omni-modal foundation
agent under tool-integrated
reasoning paradigm with active omni-modal perception. Trained on trajectories synthesized via a hindsight-guided tree exploration strategy and OmniDPO for fine-grained error correction, OmniAtlas effectively enhances the tool-use capabilities of existing open-source models. This work marks a step towards next-generation native omni-modal AI assistants for real-world scenarios.
Read the full abstract ▾
All authors Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li, Hao Wang, Shijian Wang, Guanting Dong, Jiajie Jin, Yinuo Wang, Yuan Lu, Ji-Rong Wen, Zhicheng Dou, Zhouchen Lin
Machine Learning · Cross-listed from q-bio.NC
SPD Learn: A Geometric Deep Learning Python Library for Neural Decoding Through Trivialization
Bruno Aristimunha, Ce Ju, Antoine Collas et al. (+5 more)
Implementations of symmetric positive definite (SPD)
matrix-based neural networks for neural decoding remain fragmented across research codebases and Python packages. Existing implementations often employ ad hoc handling of
manifold constraints and non-unified training setups, which hinders reproducibility and integration into modern deep-learning workflows. To address this gap, we introduce SPD Learn, a unified and modular Python package for
geometric deep learning with SPD
matrices. SPD Learn provides core SPD operators and neural-network layers, including numerically stable spectral operators, and enforces Stiefel/SPD constraints via trivialization-based parameterizations. This design enables standard backpropagation and optimization in unconstrained Euclidean spaces while producing
manifold-constrained parameters by construction. The package also offers reference implementations of representative SPDNet-based models and interfaces with widely used
brain computer interface/neuroimaging toolkits and modern machine-learning libraries (e.g., MOABB, Braindecode, Nilearn, and SKADA), facilitating reproducible benchmarking and practical deployment.
Read the full abstract ▾
All authors Bruno Aristimunha, Ce Ju, Antoine Collas, Florent Bouchard, Ammar Mian, Bertrand Thirion, Sylvain Chevallier, Reinmar Kobler
Machine Learning · Cross-listed from stat.ML
Unsupervised Continual Learning for Amortized Bayesian Inference
Aayush Mishra, Šimon Kucharský and Paul-Christian Bürkner
Amortized
Bayesian Inference (ABI) enables efficient posterior estimation using generative neural networks trained on simulated data, but often suffers from performance degradation under model misspecification. While self-consistency (SC) training on unlabeled empirical data can enhance network robustness, current approaches are limited to static, single-task settings and fail to handle sequentially arriving data or distribution shifts. We propose a continual learning framework for ABI that decouples simulation-based pre-training from unsupervised sequential SC fine-tuning on real-world data. To address the challenge of catastrophic forgetting, we introduce two adaptation strategies: (1) SC with episodic replay, utilizing a
memory buffer of past observations, and (2) SC with elastic weight consolidation, which regularizes updates to preserve task-critical parameters. Across three diverse case studies, our methods significantly mitigate forgetting and yield posterior estimates that outperform standard simulation-based training, achieving estimates closer to MCMC reference, providing a viable path for trustworthy ABI across a range of different tasks.
Read the full abstract ▾
Machine Learning
Fair feature attribution for multi-output prediction: a Shapley-based perspective
Umberto Biccari, Alain Ibáñez de Opakua, José María Mato et al. (+3 more)
In this article, we provide an axiomatic characterization of feature attribution for multi-output predictors within the Shapley framework. While SHAP explanations are routinely computed independently for each output coordinate, the theoretical necessity of this practice has remained unclear. By extending the classical Shapley axioms to vector-valued cooperative games, we establish a
rigidity theorem showing that any attribution rule satisfying efficiency,
symmetry, dummy player, and additivity must necessarily decompose component-wise across outputs. Consequently, any joint-output attribution rule must relax at least one of the classical Shapley axioms. This result identifies a previously unformalized structural constraint in Shapley-based interpretability, clarifying the precise scope of
fairness-consistent explanations in multi-output learning. Numerical experiments on a biomedical
benchmark illustrate that multi-output models can yield computational savings in training and deployment, while producing SHAP explanations that remain fully consistent with the component-wise structure imposed by the Shapley axioms.
Read the full abstract ▾
All authors Umberto Biccari, Alain Ibáñez de Opakua, José María Mato, Óscar Millet, Roberto Morales, Enrique Zuazua
Machine Learning
MEDNA-DFM: A Dual-View FiLM-MoE Model for Explainable DNA Methylation Prediction
Yi He, Yina Cao, Jixiu Zhai et al. (+3 more)
Accurate computational identification of DNA methylation is essential for understanding epigenetic regulation. Although deep learning excels in this binary classification task, its "black-box" nature impedes biological insight. We address this by introducing a high-performance model MEDNA-DFM, alongside mechanism-inspired signal purification
algorithms. Our investigation demonstrates that MEDNA-DFM effectively captures conserved methylation patterns, achieving robust distinction across diverse species. Validation on external independent
datasets confirms that the model's generalization is driven by conserved intrinsic motifs (e.g., GC content) rather than phylogenetic proximity. Furthermore, applying our developed
algorithms extracted motifs with significantly higher reliability than prior studies. Finally, empirical evidence from a Drosophila 6mA case study prompted us to propose a "sequence-structure synergy" hypothesis, suggesting that the GAGG core motif and an upstream A-tract element function cooperatively. We further validated this hypothesis via in silico mutagenesis, confirming that the ablation of either or both elements significantly degrades the model's recognition capabilities. This work provides a powerful tool for methylation prediction and demonstrates how explainable deep learning can drive both methodological innovation and the generation of biological hypotheses.
Read the full abstract ▾
All authors Yi He, Yina Cao, Jixiu Zhai, Di Wang, Junxiao Kong, Tianchi Lu
Machine Learning
Decentralized Ranking Aggregation via Gossip: Convergence and Robustness
Kerrian Le Caillec, Anna Van Elst, Igor Colin et al. (+1 more)
The concept of ranking aggregation plays a central role in preference analysis, and numerous
algorithms for calculating median rankings, often originating in social choice theory, have been documented in the literature, offering theoretical guarantees in a centralized setting, i.e., when all the ranking data to be aggregated can be brought together in a single computing unit. For many technologies (e.g. peer-to-peer networks, IoT, multi-
agent systems), extending the ability to calculate consensus rankings with guarantees of convergence and resilience to potential contamination in a decentralized setting, when preference data is initially distributed across a communicating network, remains a major methodological challenge. Indeed, in recent years, the literature on decentralized computation has mainly focused on computing or optimizing statistics such as arithmetic means using gossip
algorithms. The purpose of this article is precisely to study how to achieve reliable and resilient consensus on collective rankings in a decentralized setting, thereby raising new questions, robustness to corrupted nodes, and scalability through reduced communication costs in particular. The approach proposed and analyzed here relies on the robustness guarantees offered by random gossip communication, which allows autonomous
agents to compute a global ranking consensus using local interactions only, without coordination or a central authority.
Read the full abstract ▾
All authors Kerrian Le Caillec, Anna Van Elst, Igor Colin, Stephan Clémençon