Machine Learning
How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
Sumin Park and Noseong Park
Finding the optimal configuration of Sparse Mixture-ofExperts (SMoE) that maximizes semantic differentiation among experts is essential for exploiting the full potential of MoE architectures. However, existing SMoE frameworks either heavily rely on hyperparameter tuning or overlook the importance of diversifying semantic roles across experts when adapting the expert pool size. We propose Mixture-of-Experts for Adaptive Semantic Specialization (MASS), a semanticaware MoE framework for adaptive expert expansion and dynamic routing. MASS introduces two key advancements: (i) a gradient-based semantic drift detector that
prompts targeted expert expansion when the existing expert pool lacks capacity to capture the full semantic diversity of the data, and (ii) an integration of adaptive routing strategy that dynamically adjusts expert usage based on
token-level routing confidence mass. We first demonstrate that MASS reliably converges to the point of optimal balance between cost-performance trade-off with notably improved sematic specialization in a highly controlled synthetic setup. Further empirical results on real-world
datasets across
language and
vision domains show that MASS consistently outperforms a range of strong MoE baselines, demonstrating its domain robustness and enhanced expert specialization.
Read the full abstract ▾
Machine Learning
Concurrent training methods for Kolmogorov-Arnold networks: Disjoint datasets and FPGA implementation
Andrew Polar and Michael Poluektov
The present paper introduces concurrency-driven enhancements to the training
algorithm for the Kolmogorov-Arnold networks (KANs) that is based on the Newton-Kaczmarz (NK) method. Prior research shows that KANs trained using the NK-based approach outperform classical neural networks (multilayer perceptrons - MLPs) both in terms of accuracy and training time. Up to now, the fundamental limitation of the
algorithm has been the sequential computation of the updates - each update depends on the results of the previous step, obstructing parallelisation; even though parallelisation of some parts of the
algorithm, such as the evaluation of the basis functions, has already been proposed and tested. However, substantial acceleration is achievable. Three complementary concurrency-driven novel strategies are proposed in the present paper: (i) a pre-training procedure tailored to the NK updates' structure, (ii) training on disjoint subsets of data together with models' merging, and (iii) a division-free customisation of the
algorithm for field-programmable gate arrays (FPGAs), which is implemented and tested directly on the device. Computational experiments are used to assess the scalability and to compare the authors' KAN implementation with popular MLP packages. All presented experimental results are fully reproducible, with the complete source codes available online.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Gabliteration: Adaptive Multi-Directional Neural Weight Modification for Selective Behavioral Alteration in Large Language Models
Gökdeniz Gülmez
We present Gabliteration, a novel neural weight modification technique that advances beyond traditional abliteration methods by implementing adaptive multi-directional projections with regularized layer selection. Our approach addresses the fundamental limitation of existing methods that compromise model quality while attempting to modify specific behavioral patterns. Through dynamic layer optimization, regularized projection
matrices, and adaptive scaling mechanisms, we achieve theoretically superior weight modification while minimizing quality degradation in unrelated domains. We validate our method through the gabliterated-v1 model series (0.6B to 4B parameters) available on Hugging Face, demonstrating practical applicability across multiple model scales.
Read the full abstract ▾
Machine Learning · Cross-listed from econ.TH
Structural Reinforcement Learning for Heterogeneous Agent Macroeconomics
Yucheng Yang, Chiyuan Wang, Andreas Schaab et al. (+1 more)
We present a new approach to formulating and solving heterogeneous
agent models with aggregate risk. We replace the cross-sectional distribution with low-dimensional prices as state variables and let
agents learn
equilibrium price dynamics directly from simulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which treats prices via simulation while exploiting
agents' structural knowledge of their own individual dynamics. Our SRL method yields a general and highly efficient global solution method for heterogeneous
agent models that sidesteps the Master equation and handles problems traditional methods struggle with, in particular nontrivial
market-clearing conditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which we solve globally within minutes.
Read the full abstract ▾
All authors Yucheng Yang, Chiyuan Wang, Andreas Schaab, Benjamin Moll
Machine Learning · Cross-listed from cs.CV
Application of deep learning approaches for medieval historical documents transcription
Maksym Voloshchuk, Bohdana Zarembovska and Mykola Kozlenko
Handwritten text recognition and optical character recognition solutions show excellent results with processing data of modern era, but efficiency drops with Latin documents of medieval times. This paper presents a deep learning method to extract text information from handwritten Latin-
language documents of the 9th to 11th centuries. The approach takes into account the properties inherent in medieval documents. The paper provides a brief introduction to the field of historical document transcription, a first-sight analysis of the raw data, and the related works and studies. The paper presents the steps of
dataset development for further training of the models. The explanatory data analysis of the processed data is provided as well. The paper explains the pipeline of deep learning models to extract text information from the document images, from detecting objects to word recognition using classification models and
embedding word images. The paper reports the following results: recall, precision, F1 score, intersection over union, confusion
matrix, and mean string distance. The plots of the metrics are also included. The implementation is published on the GitHub repository.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning
Zijun Gao, Zhikun Xu, Xiao Ye et al. (+1 more)
Large
language models (LLMs) often solve challenging math exercises yet fail to apply the concept right when the problem requires genuine understanding. Popular Reinforcement Learning with Verifiable Rewards (RLVR) pipelines reinforce final answers but provide little fine-grained conceptual signal, so models improve at pattern reuse rather than conceptual applications. We introduce CORE (Concept-Oriented REinforcement), an RL training framework that turns explicit concepts into a controllable supervision signal. Starting from a high-quality, low-contamination textbook resource that links verifiable exercises to concise concept descriptions, we run a sanity probe showing LLMs can restate definitions but fail concept-linked quizzes, quantifying the conceptual
reasoning gap. CORE then (i) synthesizes concept-aligned quizzes, (ii) injects brief concept snippets during rollouts to elicit concept-primed trajectories, and (iii) reinforces conceptual
reasoning via trajectory replacement after group failures, a lightweight forward-KL constraint that aligns unguided with concept-primed policies, or standard GRPO directly on concept-aligned quizzes. Across several models, CORE delivers consistent gains over vanilla and SFT baselines on both in-domain concept-exercise suites and diverse out-of-domain math
benchmarks. CORE unifies direct training on concept-aligned quizzes and concept-injected rollouts under outcome regularization. It provides fine-grained conceptual supervision that bridges problem-solving competence and genuine conceptual
reasoning, while remaining
algorithm- and verifier-agnostic.
Read the full abstract ▾
All authors Zijun Gao, Zhikun Xu, Xiao Ye, Ben Zhou
Machine Learning
Generative Modeling through Koopman Spectral Analysis: An Operator-Theoretic Perspective
Yuanchao Xu, Fengyi Li, Masahiro Fujisawa et al. (+3 more)
We propose Koopman Spectral Wasserstein Gradient Descent (KSWGD), a particle-based generative modeling framework that learns the Langevin generator via Koopman theory and integrates it with Wasserstein gradient descent. Our key insight is that this spectral structure of the underlying distribution can be directly estimated from trajectory data via the Koopman operator, eliminating the need for explicit knowledge of the target potential. Additionally, we prove that KSWGD maintains an approximately constant dissipation rate, thereby establishing linear convergence and overcoming the vanishing-gradient phenomenon that hinders existing kernel-based particle methods. We further provide a Feynman–Kac interpretation that clarifies the method's probabilistic foundation. Experiments on compact
manifolds, metastable multi-well systems, and high-dimensional
stochastic partial differential equations demonstrate that KSWGD consistently outperforms baselines in both convergence speed and sample quality.
Read the full abstract ▾
All authors Yuanchao Xu, Fengyi Li, Masahiro Fujisawa, Xiaoyuan Cheng, Youssef Marzouk, Isao Ishikawa
Machine Learning
Hyperbolic Graph Embeddings: a Survey and an Evaluation on Anomaly Detection
Souhail Abdelmouaiz Sadat, Mohamed Yacine Touahria Miliani, Khadidja Hab El Hames et al. (+2 more)
This survey reviews
hyperbolic graph
embedding models, and evaluate them on anomaly detection, highlighting their advantages over Euclidean methods in capturing complex structures. Evaluating models like HGCAE, \textit{\(\mathcal{P}\)-VAE}, and HGCN demonstrates high performance, with \textit{\(\mathcal{P}\)-VAE} achieving an F1-score of 94% on the
Elliptic dataset and HGCAE scoring 80% on Cora. In contrast, Euclidean methods like DOMINANT and GraphSage struggle with complex data. The study emphasizes the potential of
hyperbolic spaces for improving anomaly detection, and provides an open-source library to foster further research in this field.
Read the full abstract ▾
All authors Souhail Abdelmouaiz Sadat, Mohamed Yacine Touahria Miliani, Khadidja Hab El Hames, Hamida Seba, Mohammed Haddad
Machine Learning
Controllable Probabilistic Forecasting with Stochastic Decomposition Layers
John S. Schreck, William E. Chapman, Charlie Becker et al. (+6 more)
AI weather prediction ensembles with latent noise injection and optimized with the continuous ranked probability score (CRPS) have produced both accurate and well-calibrated predictions with far less computational cost compared with
diffusion-based methods. However, current CRPS ensemble approaches vary in their training strategies and noise injection mechanisms, with most injecting noise globally throughout the network via conditional normalization. This structure increases training expense and limits the physical interpretability of the
stochastic perturbations. We introduce
Stochastic Decomposition Layers (SDL) for converting deterministic machine learning weather models into probabilistic ensemble systems. Adapted from StyleGAN's hierarchical noise injection, SDL applies learned perturbations at three decoder scales through latent-driven modulation, per-pixel noise, and channel scaling. When applied to WXFormer via transfer learning, SDL requires less than 2% of the computational cost needed to train the baseline model. Each ensemble member is generated from a compact latent tensor (5 MB), enabling perfect reproducibility and post-inference spread adjustment through latent rescaling. Evaluation on 2022 ERA5 reanalysis shows ensembles with spread-skill ratios approaching unity and rank histograms that progressively flatten toward uniformity through medium-range forecasts, achieving calibration competitive with operational IFS-ENS. Multi-scale experiments reveal hierarchical uncertainty: coarse layers modulate synoptic patterns while fine layers control mesoscale variability. The explicit latent parameterization provides interpretable uncertainty quantification for operational
forecasting and
climate applications.
Read the full abstract ▾
All authors John S. Schreck, William E. Chapman, Charlie Becker, David John Gagne II, Dhamma Kimpara, Nihanth Cherukuru, Judith Berner, Kirsten J. Mayer, Negin Sobhani
Machine Learning · Cross-listed from eess.SP
RIS-Enabled Smart Wireless Environments: Fundamentals and Distributed Optimization
George C. Alexandropoulos, Kostantinos D. Katsanos, George Stamatelis et al. (+1 more)
This chapter overviews the concept of Smart Wireless Environments (SWEs) motivated by the emerging technology of Reconfigurable Intelligent Surfaces (RISs). The operating principles and state-of-the-art hardware architectures of programmable metasurfaces are first introduced. Subsequently, key performance objectives and use cases of RIS-enabled SWEs, including spectral and energy efficiency, physical-layer security, integrated sensing and communications, as well as the emerging paradigm of over-the-air computing, are discussed. Focusing on the recent trend of Beyond-Diagonal (BD) RISs, two distributed designs of respective SWEs are presented. The first deals with a multi-user Multiple-Input Single-Output (MISO) system operating within the area of influence of a SWE comprising multiple BD-RISs. A hybrid distributed and fusion machine learning framework based on multi-branch
attention-based convolutional Neural Networks (NNs), NN parameter sharing, and neuroevolutionary training is presented, which enables online mapping of channel realizations to the BD-RIS configurations as well as the multi-user transmit precoder. Performance evaluation results showcase that the distributedly optimized RIS-enabled SWE achieves near-optimal sum-rate performance with low online computational complexity. The second design focuses on the wideband interference MISO broadcast channel, where each base station exclusively controls one BD-RIS to serve its assigned group of users. A cooperative optimization framework that jointly designs the base station transmit precoders as well as the tunable capacitances and switch
matrices of all metasurfaces is presented. Numerical results demonstrating the superior sum-rate performance of the designed RIS-enabled SWE for multi-
cell MISO networks over
benchmark schemes, considering non-cooperative configuration and conventional diagonal metasurfaces, are presented.
Read the full abstract ▾
All authors George C. Alexandropoulos, Kostantinos D. Katsanos, George Stamatelis, Ioannis Gavras
Machine Learning · Cross-listed from cs.CV
Eff-GRot: Efficient and Generalizable Rotation Estimation with Transformers
Fanis Mathioulakis, Gorjan Radevski and Tinne Tuytelaars
We introduce Eff-GRot, an approach for efficient and generalizable rotation estimation from RGB images. Given a query image and a set of reference images with known orientations, our method directly predicts the object's rotation in a single forward pass, without requiring object- or category-specific training. At the core of our framework is a
transformer that performs a comparison in the latent space, jointly processing rotation-aware representations from multiple references alongside a query. This design enables a favorable balance between accuracy and computational efficiency while remaining simple, scalable, and fully end-to-end. Experimental results show that Eff-GRot offers a promising direction toward more efficient rotation estimation, particularly in latency-sensitive applications.
Read the full abstract ▾
Machine Learning
Label-Informed Outlier Detection Based on Granule Density
Baiyang Chen, Zhong Yuan, Dezhong Peng et al. (+3 more)
Outlier detection, crucial for identifying unusual patterns with significant implications across numerous applications, has drawn considerable research interest. Existing semi-supervised methods typically treat data as purely numerical and} in a deterministic manner, thereby neglecting the heterogeneity and uncertainty inherent in complex, real-world
datasets. This paper introduces a label-informed outlier detection method for heterogeneous data based on Granular Computing and Fuzzy Sets, namely Granule Density-based Outlier Factor (GDOF). Specifically, GDOF first employs label-informed fuzzy granulation to effectively represent various data types and develops granule density for precise density estimation. Subsequently, granule densities from individual attributes are integrated for outlier scoring by assessing attribute relevance with a limited number of labeled outliers. Experimental results on various real-world
datasets show that GDOF stands out in detecting outliers in heterogeneous data with a minimal number of labeled outliers. The integration of Fuzzy Sets and Granular Computing in GDOF offers a practical framework for outlier detection in complex and diverse data types. All relevant
datasets and source codes are publicly available for further research. This is the author's accepted manuscript of a paper published in IEEE Transactions on Fuzzy Systems. The final version is available at https://doi.org/10.1109/TFUZZ.2024.3514853
Read the full abstract ▾
All authors Baiyang Chen, Zhong Yuan, Dezhong Peng, Hongmei Chen, Xiaomin Song, Huiming Zheng
Machine Learning
Gaussian-Mixture-Model Q-Functions for Policy Iteration in Reinforcement Learning
Minh Vu and Konstantinos Slavakis
Unlike their conventional use as
estimators of probability density functions in reinforcement learning (RL), this paper introduces a novel function-approximation role for Gaussian mixture models (GMMs) as direct surrogates for Q-function losses. The proposed
estimators, termed GMM-QFs, possess substantial representational capacity, as they are shown to be universal approximators over a broad class of functions. They are further embedded within Bellman residuals, where their learnable parameters–a fixed number of mixing weights, together with Gaussian mean vectors and covariance
matrices–are inferred from data via optimization on a product Riemannian
manifold. This
geometric perspective on the parameter space naturally introduces Riemannian optimization into the policy-evaluation step of standard policy-iteration (PI) frameworks. Moreover, rigorous theoretical analysis establishes performance bounds on Q-function estimation error under the proposed PI scheme. Supporting numerical tests show that GMM-QFs deliver competitive performance and, in some cases, outperform state-of-the-art approaches across a range of
benchmark RL tasks, all while maintaining a significantly smaller computational footprint than deep-learning methods.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search
Kaican Li, Lewei Yao, Jiannan Wu et al. (+7 more)
The ability for AI
agents to "think with images" requires a sophisticated blend of
reasoning and perception. However, current open
multimodal agents still largely fall short on the
reasoning aspect crucial for real-world tasks like analyzing documents with dense charts/diagrams and navigating maps. To address this gap, we introduce O3-Bench, a new
benchmark designed to evaluate
multimodal reasoning with interleaved
attention to visual details. O3-Bench features challenging problems that require
agents to piece together subtle visual information from distinct image areas through multi-step
reasoning. The problems are highly challenging even for frontier systems like OpenAI o3, which only obtains 40.8% accuracy on O3-Bench. To make progress, we propose InSight-o3, a multi-
agent framework consisting of a visual
reasoning agent (vReasoner) and a visual search
agent (vSearcher) for which we introduce the task of generalized visual search – locating relational, fuzzy, or conceptual regions described in free-form
language, beyond just simple objects or figures in natural images. We then present a
multimodal LLM purpose-trained for this task via reinforcement learning. As a plug-and-play
agent, our vSearcher empowers frontier
multimodal models (as vReasoners), significantly improving their performance on a wide range of
benchmarks. This marks a concrete step towards powerful o3-like open systems. Our code and
dataset can be found at https://github.com/m-Just/InSight-o3 .
Read the full abstract ▾
All authors Kaican Li, Lewei Yao, Jiannan Wu, Tiezheng Yu, Jierun Chen, Haoli Bai, Lu Hou, Lanqing Hong, Wei Zhang, Nevin L. Zhang
Machine Learning
PIPCFR: Pseudo-outcome Imputation with Post-treatment Variables for Individual Treatment Effect Estimation
Zichuan Lin, Xiaokai Huang, Jiate Liu et al. (+4 more)
The estimation of individual treatment effects (ITE) focuses on predicting the outcome changes that result from a change in treatment. A fundamental challenge in observational data is that while we need to infer outcome differences under alternative treatments, we can only observe each individual's outcome under a single treatment. Existing approaches address this limitation either by training with inferred pseudo-outcomes or by creating matched instance pairs. However, recent work has largely overlooked the potential impact of post-treatment variables on the outcome. This oversight prevents existing methods from fully capturing outcome variability, resulting in increased variance in counterfactual predictions. This paper introduces Pseudo-outcome Imputation with Post-treatment Variables for Counterfactual Regression (PIPCFR), a novel approach that incorporates post-treatment variables to improve pseudo-outcome imputation. We analyze the challenges inherent in utilizing post-treatment variables and establish a novel theoretical bound for ITE risk that explicitly connects post-treatment variables to ITE estimation accuracy. Unlike existing methods that ignore these variables or impose restrictive assumptions, PIPCFR learns effective representations that preserve informative components while mitigating bias. Empirical evaluations on both real-world and simulated
datasets demonstrate that PIPCFR achieves significantly lower ITE errors compared to existing methods.
Read the full abstract ▾
All authors Zichuan Lin, Xiaokai Huang, Jiate Liu, Yuxuan Han, Jia Chen, Xiapeng Wu, Deheng Ye
Machine Learning
Is Your Conditional Diffusion Model Actually Denoising?
Daniel Pfrommer, Zehao Dou, Christopher Scarvelis et al. (+2 more)
We study the inductive biases of
diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized "denoising" process upon which
diffusion models are formulated, inducing disagreement between popular sampling
algorithms (e.g. DDPM, DDIM). We introduce Schedule Deviation, a rigorous measure which captures the rate of deviation from a standard denoising process, and provide a methodology to compute it. Crucially, we demonstrate that the deviation from an idealized denoising process occurs irrespective of the model capacity or amount of training data. We posit that this phenomenon occurs due to the difficulty of bridging distinct denoising flows across different parts of the conditioning space and show theoretically how such a phenomenon can arise through an inductive bias towards smoothness.
Read the full abstract ▾
All authors Daniel Pfrommer, Zehao Dou, Christopher Scarvelis, Max Simchowitz, Ali Jadbabaie
Machine Learning · Cross-listed from cs.AI
Counterfactual Basis Extension and Representational Geometry: An MDL-Constrained Model of Conceptual Growth
Chainarong Amornbunchornvej
Concept learning becomes possible only when existing representations fail to account for experience. Most models of learning and inference, however, presuppose a fixed representational basis within which belief updating occurs. In this paper, I address a prior question: under what structural conditions can the representational basis itself expand in a principled and selective way? I propose a
geometric framework in which conceptual growth is modeled as admissible basis extension evaluated under a Minimum Description Length (MDL) criterion. Experience, whether externally observed or internally simulated, is represented as vectors relative to a current conceptual subspace. Residual components capture systematic representational failure, and candidate conceptual extensions are restricted to low-rank, admissible transformations. I show that any MDL-accepted extension can be chosen so that its novel directions lie entirely within the residual span induced by experience, while extensions orthogonal to this span strictly increase description length and are therefore rejected. This yields a conservative account of imagination and conceptual innovation. Internally generated counterfactual representations contribute to learning only insofar as they expose or amplify structured residual error, and cannot introduce arbitrary novelty. I further distinguish representational counterfactuals–counterfactuals over an
agent's conceptual basis–from causal or value-level counterfactuals, and show how MDL provides a normative selection principle governing representational change. Overall, the framework characterizes conceptual development as an error-driven, geometry-constrained process of basis extension, clarifying both the role and the limits of imagination in learning and theory change.
Read the full abstract ▾
Machine Learning
A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
Zhiquan Tan and Yinrong Hong
Large
language models (LLMs) trained via KL-regularized reinforcement learning demonstrate strong instruction following, self-correction, and
reasoning abilities. Yet their theoretical underpinnings remain limited. We exploit the closed-form energy-based model (EBM) structure of the optimal KL-regularized policy to provide a unified variational analysis of LLMs. For instruction-tuned models, under natural assumptions on reward potentials and pretraining
symmetry, we prove that the transition kernel satisfies detailed balance with respect to a scalar potential encoding response quality. This yields monotonic KL convergence to a high-quality stationary distribution, bounded hitting times to superior states, and exponential mixing governed by the spectral gap. For
reasoning models trained with verifiable rewards (RLVR), we show the objective is equivalent to expected KL minimization toward an optimal
reasoning distribution, with the suboptimality gap reducing to the Bernoulli KL between target and current accuracies along the natural gradient flow. This helps explain empirical
entropy-accuracy trade-offs.
Read the full abstract ▾
Machine Learning
ML Inference Scheduling with Predictable Latency
Haidong Zhao and Nikolaos Georgantas
Machine learning (ML) inference serving systems can schedule requests to improve GPU utilization and to meet service level objectives (SLOs) or deadlines. However, improving GPU utilization may compromise latency-sensitive scheduling, as concurrent tasks contend for GPU resources and thereby introduce interference. Given that interference effects introduce unpredictability in scheduling, neglecting them may compromise SLO or deadline satisfaction. Nevertheless, existing interference prediction approaches remain limited in several respects, which may restrict their usefulness for scheduling. First, they are often coarse-grained, which ignores runtime co-location dynamics and thus restricts their accuracy in interference prediction. Second, they tend to use a static prediction model, which may not effectively cope with different workload characteristics. In this paper, we evaluate the potential limitations of existing interference prediction approaches, finding that coarse-grained methods can lead to noticeable deviations in prediction accuracy and that static models degrade considerably under changing workloads.
Read the full abstract ▾
Machine Learning
Generating Risky Samples with Conformity Constraints via Diffusion Models
Han Yu, Hao Zou, Xingxuan Zhang et al. (+4 more)
Although neural networks achieve promising performance in many tasks, they may still fail when encountering some examples and bring about risks to applications. To discover risky samples, previous literature attempts to search for patterns of risky samples within existing
datasets or inject perturbation into them. Yet in this way the diversity of risky samples is limited by the coverage of existing
datasets. To overcome this limitation, recent works adopt
diffusion models to produce new risky samples beyond the coverage of existing
datasets. However, these methods struggle in the conformity between generated samples and expected categories, which could introduce label noise and severely limit their effectiveness in applications. To address this issue, we propose RiskyDiff that incorporates the
embeddings of both texts and images as implicit constraints of category conformity. We also design a conformity score to further explicitly strengthen the category conformity, as well as introduce the mechanisms of
embedding screening and risky gradient guidance to boost the risk of generated samples. Extensive experiments reveal that RiskyDiff greatly outperforms existing methods in terms of the degree of risk, generation quality, and conformity with conditioned categories. We also empirically show the generalization ability of the models can be enhanced by augmenting training data with generated samples of high conformity.
Read the full abstract ▾
All authors Han Yu, Hao Zou, Xingxuan Zhang, Zhengyi Wang, Yue He, Kehan Li, Peng Cui
Machine Learning · Cross-listed from stat.ML
Unsupervised Feature Selection via Robust Autoencoder and Adaptive Graph Learning
Feng Yu, MD Saifur Rahman Mazumder, Ying Su et al. (+1 more)
Effective feature selection is essential for high-dimensional data analysis and machine learning. Unsupervised feature selection (UFS) aims to simultaneously cluster data and identify the most discriminative features. Most existing UFS methods linearly project features into a pseudo-label space for clustering, but they suffer from two critical limitations: (1) an oversimplified linear mapping that fails to capture complex feature relationships, and (2) an assumption of uniform cluster distributions, ignoring outliers prevalent in real-world data. To address these issues, we propose the Robust Autoencoder-based Unsupervised Feature Selection (RAEUFS) model, which leverages a deep autoencoder to learn nonlinear feature representations while inherently improving robustness to outliers. We further develop an efficient optimization
algorithm for RAEUFS. Extensive experiments demonstrate that our method outperforms state-of-the-art UFS approaches in both clean and outlier-contaminated data settings.
Read the full abstract ▾
All authors Feng Yu, MD Saifur Rahman Mazumder, Ying Su, Oscar Contreras Velasco
Machine Learning · Cross-listed from cs.SD
Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis
Pengchao Feng, Yao Xiao, Ziyang Ma et al. (+5 more)
Recent advances in text-to-
speech (TTS) have yielded remarkable improvements in naturalness and intelligibility. Building on these achievements, research has increasingly shifted toward enhancing the expressiveness of generated
speech, such as dialectal and emotional TTS. However, cross-style synthesis combining both dialect and
emotion remains challenging and largely unexplored, mainly due to the scarcity of dialectal data with emotional labels. To address this, we propose Hierarchical Expressive Vector (HE-Vector), a two-stage method for Emotional Dialectal TTS. In the first stage, we construct different task vectors to model dialectal and emotional styles independently, and then enhance single-style synthesis by adjusting their weights, a method we refer to as Expressive Vector (E-Vector). For the second stage, we hierarchically integrate these vectors to achieve controllable emotionally expressive dialect synthesis without requiring jointly labeled data, corresponding to Hierarchical Expressive Vector (HE-Vector). Experimental results demonstrate that HE-Vectors achieve superior performance in dialect synthesis, and promising results in synthesizing emotionally expressive dialectal
speech in a zero-shot setting.
Read the full abstract ▾
All authors Pengchao Feng, Yao Xiao, Ziyang Ma, Zhikang Niu, Shuai Fan, Yao Li, Sheng Wang, Xie Chen
Machine Learning
Fusion of Multiscale Features Via Centralized Sparse-attention Network for EEG Decoding
Xiangrui Cai, Shaocheng Ma, Lei Cao et al. (+3 more)
Electroencephalography (
EEG) signal decoding is a key technology that translates
brain activity into executable commands, laying the foundation for direct
brain-machine interfacing and intelligent interaction. To address the inherent spatiotemporal heterogeneity of
EEG signals, this paper proposes a multi-branch parallel architecture, where each temporal scale is equipped with an independent spatial feature extraction module. To further enhance multi-branch feature fusion, we propose a Fusion of Multiscale Features via Centralized Sparse-
attention Network (
EEG-CSANet), a centralized sparse-
attention network. It employs a main-auxiliary branch architecture, where the main branch models core spatiotemporal patterns via multiscale self-
attention, and the auxiliary branch facilitates efficient local interactions through sparse cross-
attention. Experimental results show that
EEG-CSANet achieves state-of-the-art (SOTA) performance across five public
datasets (BCIC-IV-2A, BCIC-IV-2B, HGD, SEED, and SEED-VIG), with accuracies of 88.54%, 91.09%, 97.15%, 96.03%, and 90.56%, respectively. Such performance demonstrates its strong adaptability and robustness across various
EEG decoding tasks. Moreover, extensive ablation studies are conducted to enhance the interpretability of
EEG-CSANet. In the future, we hope that
EEG-CSANet could serve as a promising baseline model in the field of
EEG signal decoding. The source code is publicly available at: https://github.com/Xiangrui-Cai/
EEG-CSANet
Read the full abstract ▾
All authors Xiangrui Cai, Shaocheng Ma, Lei Cao, Jie Li, Tianyu Liu, Yilin Dong
Machine Learning
Improving Pattern Recognition of Scheduling Anomalies through Structure-Aware and Semantically-Enhanced Graphs
Ning Lyu, Junjie Jiang, Lu Chang et al. (+3 more)
This paper proposes a structure-aware driven scheduling graph modeling method to improve the accuracy and representation capability of anomaly identification in scheduling behaviors of complex systems. The method first designs a structure-guided scheduling graph construction mechanism that integrates task execution stages, resource node states, and scheduling path information to build dynamically evolving scheduling behavior graphs, enhancing the model's ability to capture global scheduling relationships. On this basis, a multi-scale graph semantic aggregation module is introduced to achieve semantic consistency modeling of scheduling features through local adjacency semantic integration and global
topology alignment, thereby strengthening the model's capability to capture abnormal features in complex scenarios such as multi-task concurrency, resource competition, and stage transitions. Experiments are conducted on a real scheduling
dataset with multiple scheduling disturbance paths set to simulate different types of anomalies, including structural shifts, resource changes, and task delays. The proposed model demonstrates significant performance advantages across multiple metrics, showing a sensitive response to structural disturbances and semantic shifts. Further visualization analysis reveals that, under the combined effect of structure guidance and semantic aggregation, the scheduling behavior graph exhibits stronger anomaly separability and pattern representation, validating the effectiveness and adaptability of the method in scheduling anomaly detection tasks.
Read the full abstract ▾
All authors Ning Lyu, Junjie Jiang, Lu Chang, Chihui Shao, Feng Chen, Chong Zhang
Machine Learning
Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments
Xue Yang, Michael Schukat, Junlin Lu et al. (+3 more)
Reinforcement learning (RL) excels in various applications but struggles in dynamic environments where the underlying Markov decision process evolves. Continual reinforcement learning (CRL) enables RL
agents to continually learn and adapt to new tasks, but balancing stability (preserving prior knowledge) and plasticity (acquiring new knowledge) remains challenging. Existing methods primarily address the stability-plasticity dilemma through mechanisms where past knowledge influences optimization but rarely affects the
agent's behavior directly, which may hinder effective knowledge reuse and efficient learning. In contrast, we propose demonstration-guided continual reinforcement learning (DGCRL), which stores prior knowledge in an external, self-evolving demonstration repository that directly guides RL exploration and adaptation. For each task, the
agent dynamically selects the most relevant demonstration and follows a curriculum-based strategy to accelerate learning, gradually shifting from demonstration-guided exploration to fully self-exploration. Extensive experiments on 2D
navigation and MuJoCo locomotion tasks demonstrate its superior average performance, enhanced knowledge transfer, mitigation of forgetting, and training efficiency. The additional sensitivity analysis and ablation study further validate its effectiveness.
Read the full abstract ▾
All authors Xue Yang, Michael Schukat, Junlin Lu, Patrick Mannion, Karl Mason, Enda Howley
Machine Learning · Cross-listed from cs.CV
PMPGuard: Catching Pseudo-Matched Pairs in Remote Sensing Image-Text Retrieval
Pengxiang Ouyang, Qing Ma, Zheng Wang et al. (+1 more)
Remote sensing (RS) image-text
retrieval faces significant challenges in real-world
datasets due to the presence of Pseudo-Matched Pairs (PMPs), semantically mismatched or weakly aligned image-text pairs, which hinder the learning of reliable cross-modal
alignments. To address this issue, we propose a novel
retrieval framework that leverages Cross-Modal Gated
Attention and a Positive-Negative Awareness
Attention mechanism to mitigate the impact of such noisy associations. The gated module dynamically regulates cross-modal information flow, while the awareness mechanism explicitly distinguishes informative (positive) cues from misleading (negative) ones during
alignment learning. Extensive experiments on three
benchmark RS
datasets, i.e., RSICD, RSITMD, and RS5M, demonstrate that our method consistently achieves state-of-the-art performance, highlighting its robustness and effectiveness in handling real-world mismatches and PMPs in RS image-text
retrieval tasks.
Read the full abstract ▾
All authors Pengxiang Ouyang, Qing Ma, Zheng Wang, Cong Bai
Machine Learning
From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Ryotaro Kawata, Yujin Song, Alberto Bietti et al. (+4 more)
Transformers can implement both generalizable
algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study how the choice of pretraining data distribution steers a shallow
transformer toward one behavior or the other. Focusing on a minimal trigger-output prediction task – copying the
token immediately following a special trigger upon its second occurrence – we present a rigorous analysis of gradient-based training of a single-layer
transformer. In both the infinite and finite sample regimes, we prove a transition in the learned mechanism: if input sequences exhibit sufficient diversity, measured by a low “max-sum” ratio of trigger-to-trigger distances, the trained model implements an induction head and generalizes to unseen contexts; by contrast, when this ratio is large, the model resorts to a positional shortcut and fails to generalize out-of-distribution (OOD). We also reveal a trade-off between the pretraining context length and OOD generalization, and derive the optimal pretraining distribution that minimizes computational cost per sample. Finally, we validate our theoretical predictions with controlled synthetic experiments, demonstrating that broadening context distributions robustly induces induction heads and enables OOD generalization. Our results shed light on the algorithmic biases of pretrained
transformers and offer conceptual guidelines for data-driven control of their learned behaviors.
Read the full abstract ▾
All authors Ryotaro Kawata, Yujin Song, Alberto Bietti, Naoki Nishikawa, Taiji Suzuki, Samuel Vaiter, Denny Wu
Machine Learning
ARC: Leveraging Compositional Representations for Cross-Problem Learning on VRPs
Han-Seul Jeong, Youngjoon Park, Hyungseok Song et al. (+1 more)
Vehicle Routing Problems (VRPs) with diverse real-world attributes have driven recent interest in cross-problem learning approaches that efficiently generalize across problem variants. We propose ARC (Attribute Representation via Compositional Learning), a cross-problem learning framework that learns disentangled attribute representations by decomposing them into two complementary components: an Intrinsic Attribute
Embedding (IAE) for invariant attribute semantics and a Contextual Interaction
Embedding (CIE) for attribute-combination effects. This disentanglement is achieved by enforcing analogical consistency in the
embedding space to ensure the semantic transformation of adding an attribute (e.g., a length constraint) remains invariant across different problem contexts. This enables our model to reuse invariant semantics across trained variants and construct representations for unseen combinations. ARC achieves state-of-the-art performance across in-distribution, zero-shot generalization, few-shot adaptation, and real-world
benchmarks.
Read the full abstract ▾
All authors Han-Seul Jeong, Youngjoon Park, Hyungseok Song, Woohyung Lim
Machine Learning
The Procrustean Bed of Time Series: The Optimization Bias in Point-wise Loss Functions
Rongyao Cai, Yuxi Wan, Kexin Zhang et al. (+6 more)
Intuitively, a more deterministic time series should be easier to forecast. However, point-wise loss functions (e.g., MSE and MAE), serving as differentiable surrogates for the ideal optimization target, score each timestamp independently and therefore disregard temporal dependence. This mismatch induces a systematic optimization bias that cannot be eliminated merely by improving model expressiveness or optimizer. To formalize this issue, we define the Expectation of Optimization Bias (EOB) as the Kullback–Leibler divergence between the true joint distribution and the factorized i.i.d. surrogate induced by the point-wise paradigm. Under covariance-stationary Gaussian assumptions, we derive closed-form expressions for the
stochastic component of EOB, establishing it as an irreducible lower bound on the total bias in linear systems, and further extend it to nonlinear regimes through a Gaussian mixture model lower bound. Crucially, we prove this bias is governed intrinsically by two data properties, i.e., sequence length and Structural Signal-to-Noise Ratio (SSNR), regardless of specific model architecture, optimizer, or point-wise loss forms. This theory motivates a principled debiasing program based on sequence length reduction and structural orthogonalization, which we instantiate through DFT/DWT combined with a novel harmonized $\ell_p$ norm. Extensive experiments validate the predicted SSNR–horizon dynamics, resolve the classic trigonometric fitting failure as an objective-induced pathology, and demonstrate substantial plug-and-play gains. Notably, on iTransformer, our proposed objective reduces average MSE/MAE by 5.2%/5.0% in
forecasting across 11
datasets and by 27.4%/19.4% in imputation across 9
datasets.
Read the full abstract ▾
All authors Rongyao Cai, Yuxi Wan, Kexin Zhang, Ming Jin, Zhiqiang Ge, Daoyi Dong, Hang Yu, Yong Liu, Qingsong Wen
Machine Learning
The Interaction Bottleneck of Deep Neural Networks: Discovery, Proof, and Modulation
Huiqi Deng, Qihan Ren, Zhuofan Chen et al. (+5 more)
Understanding what kinds of cooperative structures deep neural networks (DNNs) can represent remains a fundamental yet insufficiently understood problem. In this work, we treat interactions as the fundamental units of such structure and investigate a largely unexplored question: how DNNs encode interactions under different levels of contextual complexity, and how these microscopic interaction patterns shape macroscopic representation capacity. To quantify this complexity, we use multi-order interactions [57], where each order reflects the amount of contextual information required to evaluate the joint interaction utility of a variable pair. This formulation enables a stratified analysis of cooperative patterns learned by DNNs. Building on this formulation, we develop a comprehensive study of interaction structure in DNNs. (i) We empirically discover a universal interaction bottleneck: across architectures and tasks, DNNs easily learn low-order and high-order interactions but consistently under-represent mid-order ones. (ii) We theoretically explain this bottleneck by proving that mid-order interactions incur the highest contextual variability, yielding large gradient variance and making them intrinsically difficult to learn. (iii) We further modulate the bottleneck by introducing losses that steer models toward emphasizing interactions of selected orders. Finally, we connect microscopic interaction structures with macroscopic representational behavior: low-order-emphasized models exhibit stronger generalization and robustness, whereas high-order-emphasized models demonstrate greater structural modeling and fitting capability. Together, these results uncover an inherent representational bias in modern DNNs and establish interaction order as a powerful lens for interpreting and guiding deep representations.
Read the full abstract ▾
All authors Huiqi Deng, Qihan Ren, Zhuofan Chen, Zhenyuan Cui, Wen Shen, Peng Zhang, Hongbin Pei, Quanshi Zhang