Machine Learning
A New Family of Poisson Non-negative Matrix Factorization Methods Using the Shifted Log Link
Eric Weine, Peter Carbonetto, Rafael A. Irizarry et al. (+1 more)
Poisson non-negative
matrix factorization (NMF) is a widely used method to find interpretable "parts-based" decompositions of count data. While many variants of Poisson NMF exist, existing methods assume that the "parts" in the decomposition combine additively. This assumption may be natural in some settings, but not in others. Here we introduce Poisson NMF with the shifted-log link function to relax this assumption. The shifted-log link function has a single tuning parameter, and as this parameter varies the model changes from assuming that parts combine additively (i.e., standard Poisson NMF) to assuming that parts combine more multiplicatively. We provide an
algorithm to fit this model by maximum likelihood, and also an approximation that substantially reduces computation time for large, sparse
datasets (computations scale with the number of non-zero entries in the data
matrix). We illustrate these new methods on a variety of real
datasets. Our examples show how the choice of link function in Poisson NMF can substantively impact the results, and how in some settings the use of a shifted-log link function may improve interpretability compared with the standard, additive link.
Read the full abstract ▾
All authors Eric Weine, Peter Carbonetto, Rafael A. Irizarry, Matthew Stephens
Machine Learning
A Dual Pipeline Machine Learning Framework for Automated Multi Class Sleep Disorder Screening Using Hybrid Resampling and Ensemble Learning
Md Sultanul Islam Ovi, Muhsina Tarannum Munfa, G.M.M Miftahul Alam Adib et al. (+1 more)
Accurate classification of sleep disorders, particularly insomnia and sleep apnea, is important for reducing long term health risks and improving patient quality of life. However, clinical sleep studies are resource intensive and are difficult to scale for population level screening. This paper presents a Dual Pipeline Machine Learning Framework for multi class sleep disorder screening using the Sleep Health and Lifestyle
dataset. The framework consists of two parallel processing streams: a statistical pipeline that targets linear separability using Mutual Information and Linear Discriminant Analysis, and a wrapper based pipeline that applies Boruta feature selection with an autoencoder for non linear representation learning. To address class imbalance, we use the hybrid SMOTETomek resampling strategy. In experiments, Extra Trees and K Nearest Neighbors achieved an accuracy of 98.67%, outperforming recent baselines on the same
dataset. Statistical testing using the Wilcoxon Signed Rank Test indicates that the improvement over baseline configurations is significant, and inference latency remains below 400 milliseconds. These results suggest that the proposed dual pipeline design supports accurate and efficient automated screening for non invasive sleep disorder risk stratification.
Read the full abstract ▾
All authors Md Sultanul Islam Ovi, Muhsina Tarannum Munfa, G.M.M Miftahul Alam Adib, Syed Sabbir Hasan
Machine Learning
Detecting Autism Spectrum Disorder with Deep Eye Movement Features
Zhanpei Huang, Taochen chen, Fangqing Gu et al. (+1 more)
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by deficits in social communication and behavioral patterns. Eye movement data offers a non-invasive diagnostic tool for ASD detection, as it is inherently discrete and exhibits short-term temporal dependencies, reflecting localized gaze focus between fixation points. These characteristics enable the data to provide deeper insights into subtle behavioral markers, distinguishing ASD-related patterns from typical development. Eye movement signals mainly contain short-term and localized dependencies. However, despite the widespread application of stacked
attention layers in
Transformer-based models for capturing long-range dependencies, our experimental results indicate that this approach yields only limited benefits when applied to eye movement data. This may be because discrete fixation points and short-term dependencies in gaze focus reduce the utility of global
attention mechanisms, making them less efficient than architectures focusing on local temporal patterns. To efficiently capture subtle and complex eye movement patterns, distinguishing ASD from typically developing (TD) individuals, a discrete short-term sequential (DSTS) modeling framework is designed with Class-aware Representation and Imbalance-aware Mechanisms. Through extensive experiments on several eye movement
datasets, DSTS outperforms both traditional machine learning techniques and more sophisticated deep learning models.
Read the full abstract ▾
All authors Zhanpei Huang, Taochen chen, Fangqing Gu, Yiqun Zhang
Machine Learning
Learning Reconstructive Embeddings in Reproducing Kernel Hilbert Spaces via the Representer Theorem
Enrique Feito-Casares, Francisco M. Melgarejo-Meseguer and José-Luis Rojo-Álvarez
Motivated by the growing interest in representation learning approaches that uncover the latent structure of high-dimensional data, this work proposes new
algorithms for reconstruction-based
manifold learning within Reproducing-Kernel
Hilbert Spaces (RKHS). Each observation is first reconstructed as a linear combination of the other samples in the RKHS, by optimizing a vector form of the Representer
Theorem for their autorepresentation property. A separable operator-valued kernel extends the formulation to vector-valued data while retaining the simplicity of a single scalar similarity function. A subsequent kernel-
alignment task projects the data into a lower-dimensional latent space whose Gram
matrix aims to match the high-dimensional reconstruction kernel, thus transferring the auto-reconstruction geometry of the RKHS to the
embedding. Therefore, the proposed
algorithms represent an extended approach to the autorepresentation property, exhibited by many natural data, by using and adapting well-known results of Kernel Learning Theory. Numerical experiments on both simulated (concentric circles and swiss-roll) and real (cancer molecular activity and IoT network intrusions)
datasets provide empirical evidence of the practical effectiveness of the proposed approach.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SceneFoundry: Generating Interactive Infinite 3D Worlds
ChunTeng Chen, YiChen Hsu, YiWen Liu et al. (+5 more)
The ability to automatically generate large-scale, interactive, and physically realistic 3D environments is crucial for advancing robotic learning and
embodied intelligence. However, existing generative approaches often fail to capture the functional complexity of real-world interiors, particularly those containing articulated objects with movable parts essential for manipulation and
navigation. This paper presents SceneFoundry, a
language-guided
diffusion framework that generates apartment-scale 3D worlds with functionally articulated furniture and semantically diverse layouts for robotic training. From natural
language prompts, an LLM module controls floor layout generation, while
diffusion-based posterior sampling efficiently populates the scene with articulated assets from large-scale 3D repositories. To ensure physical usability, SceneFoundry employs differentiable guidance functions to regulate object quantity, prevent articulation collisions, and maintain sufficient walkable space for robotic
navigation. Extensive experiments demonstrate that our framework generates structurally valid, semantically coherent, and functionally interactive environments across diverse scene types and conditions, enabling scalable
embodied AI research. project page: https://anc891203.github.io/SceneFoundry-Demo/
Read the full abstract ▾
All authors ChunTeng Chen, YiChen Hsu, YiWen Liu, WeiFang Sun, TsaiChing Ni, ChunYi Lee, Min Sun, YuanFu Yang
Machine Learning · Cross-listed from cs.CL
EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis
Xiaoshuai Song, Haofei Chang, Guanting Dong et al. (+3 more)
Large
language models (LLMs) are expected to be trained to act as
agents in various real-world environments, but this process relies on rich and varied tool-interaction sandboxes. However, access to real systems is often restricted; LLM-simulated environments are prone to
hallucinations and inconsistencies; and manually built sandboxes are hard to scale. In this paper, we propose EnvScaler, an automated framework for scalable tool-interaction environments via programmatic synthesis. EnvScaler comprises two components. First, SkelBuilder constructs diverse environment skeletons through topic mining, logic modeling, and quality evaluation. Then, ScenGenerator generates multiple task scenarios and rule-based trajectory validation functions for each environment. With EnvScaler, we synthesize 191 environments and about 7K scenarios, and apply them to Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Qwen3 series models. Results on three
benchmarks show that EnvScaler significantly improves LLMs' ability to solve tasks in complex environments involving multi-turn, multi-tool interactions. We release our code and data at https://github.com/RUC-NLPIR/EnvScaler.
Read the full abstract ▾
All authors Xiaoshuai Song, Haofei Chang, Guanting Dong, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou
Machine Learning
Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers
Mohamed Amine Hallam and Kuo-Kun Tseng
Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with
token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies–element-wise addition, concatenation with projection, and scalar gated fusion–under identical
Transformer architectures, data splits, and random seeds. Experiments on three text classification
datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than
stochastic, we perform paired-seed analysis and cross-
dataset comparison across sequence-length regimes. Additional experiments on the ArXiv
dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence
Transformers and should be treated as an explicit modeling decision rather than a fixed default.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Simplify-This: A Comparative Analysis of Prompt-Based and Fine-Tuned LLMs
Eilam Cohen, Itamar Bul, Danielle Inbar et al. (+1 more)
Large
language models (LLMs) enable strong text generation, and in general there is a practical tradeoff between fine-tuning and
prompt engineering. We introduce Simplify-This, a comparative study evaluating both paradigms for text simplification with encoder-decoder LLMs across multiple
benchmarks, using a range of evaluation metrics. Fine-tuned models consistently deliver stronger structural simplification, whereas prompting often attains higher semantic similarity scores yet tends to copy inputs. A human evaluation favors fine-tuned outputs overall. We release code, a cleaned derivative
dataset used in our study, checkpoints of fine-tuned models, and
prompt templates to facilitate reproducibility and future work.
Read the full abstract ▾
All authors Eilam Cohen, Itamar Bul, Danielle Inbar, Omri Loewenbach
Machine Learning
Tensor-DTI: Enhancing Biomolecular Interaction Prediction with Contrastive Embedding Learning
Manel Gil-Sorribes, Júlia Vilalta-Mor, Isaac Filella-Mercè et al. (+4 more)
Accurate drug-target interaction (DTI) prediction is essential for computational drug discovery, yet existing models often rely on single-modality predefined molecular descriptors or sequence-based
embeddings with limited representativeness. We propose Tensor-DTI, a contrastive learning framework that integrates
multimodal embeddings from molecular graphs,
protein language models, and binding-site predictions to improve interaction modeling. Tensor-DTI employs a siamese dual-encoder architecture, enabling it to capture both chemical and structural interaction features while distinguishing interacting from non-interacting pairs. Evaluations on multiple DTI
benchmarks demonstrate that Tensor-DTI outperforms existing sequence-based and graph-based models. We also conduct large-scale inference experiments on CDK2 across billion-scale chemical libraries, where Tensor-DTI produces chemically plausible hit distributions even when CDK2 is withheld from training. In enrichment studies against Glide docking and Boltz-2 co-folder, Tensor-DTI remains competitive on CDK2 and improves the screening budget required to recover moderate fractions of high-affinity ligands on out-of-family targets under strict family-holdout splits. Additionally, we explore its applicability to
protein-RNA and peptide-
protein interactions. Our findings highlight the benefits of integrating
multimodal information with contrastive objectives to enhance interaction-prediction accuracy and to provide more interpretable and reliability-aware models for virtual screening.
Read the full abstract ▾
All authors Manel Gil-Sorribes, Júlia Vilalta-Mor, Isaac Filella-Mercè, Robert Soliva, Álvaro Ciudad, Víctor Guallar, Alexis Molina
Machine Learning · Cross-listed from cs.HC
SAFE: Secure and Accurate Federated Learning for Privacy-Preserving Brain-Computer Interfaces
Tianwang Jia, Xiaoqing Chen and Dongrui Wu
Electroencephalogram (
EEG)-based
brain-computer interfaces (BCIs) are widely adopted due to their efficiency and portability; however, their decoding
algorithms still face multiple challenges, including inadequate generalization,
adversarial vulnerability, and
privacy leakage. This paper proposes Secure and Accurate
FEderated learning (SAFE), a
federated learning-based approach that protects user
privacy by keeping data local during model training. SAFE employs local batch-specific normalization to mitigate cross-subject feature distribution shifts and hence improves model generalization. It further enhances
adversarial robustness by introducing perturbations in both the input space and the parameter space through
federated adversarial training and
adversarial weight perturbation. Experiments on five
EEG datasets from motor imagery (MI) and event-related potential (ERP) BCI paradigms demonstrated that SAFE consistently outperformed 14 state-of-the-art approaches in both decoding accuracy and
adversarial robustness, while ensuring
privacy protection. Notably, it even outperformed centralized training approaches that do not consider
privacy protection at all. To our knowledge, SAFE is the first
algorithm to simultaneously achieve high decoding accuracy, strong
adversarial robustness, and reliable
privacy protection without using any calibration data from the target subject, making it highly desirable for real-world BCIs.
Read the full abstract ▾
Machine Learning
Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer
Yifan Zhang, Wei Bi, Kechi Zhang et al. (+3 more)
Algorithm extraction aims to synthesize executable programs directly from models trained on algorithmic tasks, enabling de novo recovery of executable mechanisms from weights without relying on human-written target programs. However, applying this paradigm to
Transformer is complicated by representation
entanglement (e.g., superposition), where features encoded in overlapping directions substantially hinder the recovery of symbolic expressions. We propose the Discrete
Transformer, an architecture explicitly designed to bridge the gap between continuous representations and discrete symbolic logic. By injecting discreteness through temperature-annealed sampling, our framework effectively leverages hypothesis testing and symbolic regression to extract human-readable programs. Empirically, the Discrete
Transformer achieves performance comparable to the RNN-based MIPS baseline on shared discrete tasks, while broadening extraction to tasks with continuous-valued intermediate computations. Finally, we show that architectural inductive biases provide fine-grained control over synthesized programs, establishing the Discrete
Transformer as a controllable testbed for
algorithm extraction and
Transformer interpretability.
Read the full abstract ▾
All authors Yifan Zhang, Wei Bi, Kechi Zhang, Dongming Jin, Jie Fu, Zhi Jin
Machine Learning
Variational Autoencoders for P-wave Detection on Strong Motion Earthquake Spectrograms
Turkan Simge Ispak, Salih Tileylioglu and Erdem Akagunduz
Accurate P-wave detection is critical for earthquake early warning, yet strong-motion records pose challenges due to high noise levels, limited labeled data, and complex waveform characteristics. This study reframes P-wave arrival detection as a self-supervised anomaly detection task to evaluate how architectural variations regulate the trade-off between reconstruction fidelity and anomaly discrimination. Through a comprehensive grid search of 492 Variational Autoencoder configurations, we show that while skip connections minimize reconstruction error (Mean Absolute Error approximately 0.0012), they induce "overgeneralization", allowing the model to reconstruct noise and masking the detection signal. In contrast,
attention mechanisms prioritize global context over local detail and yield the highest detection performance with an area-under-the-curve of 0.875. The
attention-based Variational Autoencoder achieves an area-under-the-curve of 0.91 in the 0 to 40-kilometer near-source range, demonstrating high suitability for immediate early warning applications. These findings establish that architectural constraints favoring global context over pixel-perfect reconstruction are essential for robust, self-supervised P-wave detection.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
ViTNT-FIQA: Training-Free Face Image Quality Assessment with Vision Transformers
Guray Ozgur, Eduarda Caldeira, Tahar Chettaoui et al. (+4 more)
Face Image Quality Assessment (FIQA) is essential for reliable face recognition systems. Current approaches primarily exploit only final-layer representations, while training-free methods require multiple forward passes or backpropagation. We propose ViTNT-FIQA, a training-free approach that measures the stability of patch
embedding evolution across intermediate
Vision Transformer (ViT) blocks. We demonstrate that high-quality face images exhibit stable feature refinement trajectories across blocks, while degraded images show erratic transformations. Our method computes Euclidean distances between L2-normalized patch
embeddings from consecutive
transformer blocks and aggregates them into image-level quality scores. We empirically validate this correlation on a quality-labeled synthetic
dataset with controlled degradation levels. Unlike existing training-free approaches, ViTNT-FIQA requires only a single forward pass without backpropagation or architectural modifications. Through extensive evaluation on eight
benchmarks (LFW, AgeDB-30, CFP-FP, CALFW, Adience, CPLFW, XQLFW, IJB-C), we show that ViTNT-FIQA achieves competitive performance with state-of-the-art methods while maintaining computational efficiency and immediate applicability to any pre-trained ViT-based face recognition model.
Read the full abstract ▾
All authors Guray Ozgur, Eduarda Caldeira, Tahar Chettaoui, Jan Niklas Kolf, Marco Huber, Naser Damer, Fadi Boutros
Machine Learning
mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
Yongyi Yang and Jianyang Gao
Hyper-Connections (HC) generalizes residual connections by introducing dynamic residual
matrices that mix information across multiple residual streams, accelerating convergence in deep neural networks. However, unconstrained residual
matrices can compromise training stability. To address this, DeepSeek's
Manifold-Constrained Hyper-Connections (mHC) approximately projects these
matrices onto the Birkhoff polytope via iterative Sinkhorn–Knopp (SK) normalization. We identify two limitations of this approach: (i) finite SK iterations do not guarantee exact doubly stochasticity, leaving an approximation gap that can accumulate through network depth and undermine stability; (ii) efficient SK implementation requires highly specialized CUDA kernels, raising engineering barriers and reducing portability. Motivated by the Birkhoff–von Neumann
theorem, we propose mHC-lite, a simple reparameterization that explicitly constructs doubly
stochastic matrices as convex combinations of permutation
matrices. This approach guarantees exact doubly stochasticity by construction and can be implemented using only native
matrix operations. Extensive experiments demonstrate that mHC-lite matches or exceeds mHC in performance while achieving higher training throughput with a naive implementation and eliminating the residual instabilities observed in both HC and mHC. The code is publicly available at https://github.com/FFTYYY/mhc-lite.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Visualising Information Flow in Word Embeddings with Diffusion Tensor Imaging
Thomas Fabian
Understanding how large
language models (LLMs) represent natural
language is a central challenge in natural
language processing (NLP) research. Many existing methods extract word
embeddings from an LLM, visualise the
embedding space via point-plots, and compare the relative positions of certain words. However, this approach only considers single words and not whole natural
language expressions, thus disregards the context in which a word is used. Here we present a novel tool for analysing and visualising information flow in natural
language expressions by applying
diffusion tensor imaging (DTI) to word
embeddings. We find that DTI reveals how
embedding space representations change between
tokens. Tracking these changes within the layers of an LLM allows for comparing different model structures and could potentially reveal opportunities for
pruning an LLM's under-utilised layers. Our results show that our visualisation method permits novel insights into how LLMs represent actual natural
language expressions, extending the comparison of isolated word
embeddings and improving the interpretability of NLP models.
Read the full abstract ▾
Machine Learning
FLRQ: Faster LLM Quantization with Flexible Low-Rank Matrix Sketching
Hongyaoxing Gul, Lijuan Hu, Shuzi Niu et al. (+1 more)
Traditional post-training quantization (PTQ) is considered an effective approach to reduce model size and accelerate inference of large-scale
language models (LLMs). However, existing low-rank PTQ methods require costly fine-tuning to determine a compromise rank for diverse data and layers in large models, failing to exploit their full potential. Additionally, the current SVD-based low-rank approximation compounds the computational overhead. In this work, we thoroughly analyze the varying effectiveness of low-rank approximation across different layers in representative models. Accordingly, we introduce \underline{F}lexible \underline{L}ow-\underline{R}ank \underline{Q}uantization (FLRQ), a novel solution designed to quickly identify the accuracy-optimal ranks and aggregate them to achieve minimal storage combinations. FLRQ comprises two powerful components, Rank1-Sketch-based Flexible Rank Selection (R1-FLR) and Best Low-rank Approximation under Clipping (BLC). R1-FLR applies the R1-Sketch with Gaussian projection for the fast low-rank approximation, enabling outlier-aware rank extraction for each layer. Meanwhile, BLC aims at minimizing the low-rank quantization error under the scaling and clipping strategy through an iterative method. FLRQ demonstrates strong effectiveness and robustness in comprehensive experiments, achieving state-of-the-art performance in both quantization quality and
algorithm efficiency.
Read the full abstract ▾
All authors Hongyaoxing Gul, Lijuan Hu, Shuzi Niu, Fangfang Liu
Machine Learning
Infinite-Precision Autoregressive Modeling for Vector Graphics and Layouts
Yeonsang Shin, Insoo Kim, Bongkeun Kim et al. (+2 more)
While
Transformer-based autoregressive models excel in data generation, their
token discretization strategy inherently limits their precision in continuous domains. We analyze the scalability limitations of existing discretization-based approaches for generating hybrid discrete-continuous sequences, particularly in high-precision domains such as logos, layouts, and
semiconductor circuit designs, where precision loss potentially leads to visual artifacts, aesthetic degradation, and even functional failure. To address the challenge, we propose a novel unified framework that jointly models discrete and continuous values for variable-length sequences. Our approach employs a hybrid approach that combines categorical prediction for discrete values with
diffusion-based modeling for continuous values, incorporating two key technical components: an end-of-sequence (EOS) logit adjustment mechanism that uses an MLP to dynamically adjust EOS
token logits based on sequence context, and a length regularization term integrated into the loss function. Additionally, we present ContLayNet, a large-scale
benchmark comprising 334K high-precision
semiconductor layout samples with specialized evaluation metrics that capture functional correctness, where precision errors significantly impact performance. Experiments on multiple domains show that our approach achieves higher-fidelity hybrid vector representations than discretization-based and fixed-schema baselines, while effectively scaling to high-precision generation.
Read the full abstract ▾
All authors Yeonsang Shin, Insoo Kim, Bongkeun Kim, Keonwoo Bae, Bohyung Han
Machine Learning
Do Sparse Autoencoders Identify Reasoning Features in Language Models?
George Ma, Zhongyuan Liang, Irene Y. Chen et al. (+1 more)
We study how reliably sparse autoencoders (SAEs) support claims about
reasoning-related internal features in large
language models. We first give a stylized analysis showing that sparsity-regularized decoding can preferentially retain stable low-dimensional correlates while suppressing high-dimensional within-behavior variation, motivating the possibility that contrastively selected "
reasoning" features may concentrate on cue-like structure when such cues are coupled with
reasoning traces. Building on this perspective, we propose a falsification-based evaluation framework that combines causal
token injection with LLM-guided counterexample construction. Across 22 configurations spanning multiple model families, layers, and
reasoning datasets, we find that many contrastively selected candidates are highly sensitive to
token-level interventions, with 45%-90% activating after injecting only a few associated
tokens into non-
reasoning text. For the remaining context-dependent candidates, LLM-guided falsification produces targeted non-
reasoning inputs that trigger activation and meaning-preserving paraphrases of top-activating
reasoning traces that suppress it. A small steering study yields minimal changes on the evaluated
benchmarks. Overall, our results suggest that, in the settings we study, sparse decompositions can favor low-dimensional correlates that co-occur with
reasoning, underscoring the need for falsification when attributing high-level behaviors to individual SAE features. Code is available at https://github.com/GeorgeMLP/
reasoning-probing.
Read the full abstract ▾
All authors George Ma, Zhongyuan Liang, Irene Y. Chen, Somayeh Sojoudi
Machine Learning · Cross-listed from cs.SE
Tracing Stereotypes in Pre-trained Transformers: From Biased Neurons to Fairer Models
Gianmario Voria, Moses Openja, Foutse Khomh et al. (+2 more)
The advent of
transformer-based
language models has reshaped how AI systems process and generate text. In software engineering (SE), these models now support diverse activities, accelerating automation and decision-making. Yet, evidence shows that these models can reproduce or amplify social biases, raising
fairness concerns. Recent work on
neuron editing has shown that internal activations in pre-trained
transformers can be traced and modified to alter model behavior. Building on the concept of knowledge
neurons,
neurons that encode factual information, we hypothesize the existence of biased
neurons that capture stereotypical associations within pre-trained
transformers. To test this hypothesis, we build a
dataset of biased relations, i.e., triplets encoding stereotypes across nine bias types, and adapt
neuron attribution strategies to trace and suppress biased
neurons in BERT models. We then assess the impact of suppression on SE tasks. Our findings show that biased knowledge is localized within small
neuron subsets, and suppressing them substantially reduces bias with minimal performance loss. This demonstrates that bias in
transformers can be traced and mitigated at the
neuron level, offering an interpretable approach to
fairness in SE.
Read the full abstract ▾
All authors Gianmario Voria, Moses Openja, Foutse Khomh, Gemma Catolino, Fabio Palomba
Machine Learning
From Global to Local: Cluster-Aware Learning for Wi-Fi Fingerprinting Indoor Localisation
Miguel Matey-Sanz, Joaquín Torres-Sospedra, Joaquín Huerta et al. (+1 more)
Wi-Fi fingerprinting remains one of the most practical solutions for indoor positioning, however, its performance is often limited by the size and heterogeneity of fingerprint
datasets, strong Received Signal Strength Indicator variability, and the ambiguity introduced in large and multi-floor environments. These factors significantly degrade localisation accuracy, particularly when global models are applied without considering structural constraints. This paper introduces a clustering-based method that structures the fingerprint
dataset prior to localisation. Fingerprints are grouped using either spatial or
radio features, and clustering can be applied at the building or floor level. In the localisation phase, a clustering estimation procedure based on the strongest access points assigns unseen fingerprints to the most relevant cluster. Localisation is then performed only within the selected clusters, allowing learning models to operate on reduced and more coherent subsets of data. The effectiveness of the method is evaluated on three public
datasets and several machine learning models. Results show a consistent reduction in localisation errors, particularly under building-level strategies, but at the cost of reducing the floor detection accuracy. These results demonstrate that explicitly structuring
datasets through clustering is an effective and flexible approach for scalable indoor positioning.
Read the full abstract ▾
All authors Miguel Matey-Sanz, Joaquín Torres-Sospedra, Joaquín Huerta, Sergio Trilles
Machine Learning · Cross-listed from q-bio.GN
Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training
Haoran Wang, Xuanyi Zhang, Shuangsang Fang et al. (+5 more)
Recent advancements in single-
cell multi-omics, particularly RNA-seq, have provided profound insights into cellular heterogeneity and gene regulation. While pre-trained
language model (PLM) paradigm based single-
cell foundation models have shown promise, they remain constrained by insufficient integration of in-depth individual profiles and neglecting the influence of noise within multi-modal data. To address both issues, we propose an Open-world
Language Knowledge-Aided Robust Single-
Cell Foundation Model (OKR-
CELL). It is built based on a cross-modal
Cell-
Language pre-training framework, which comprises two key innovations: (1) leveraging Large
Language Models (LLMs) based workflow with
retrieval-augmented generation (RAG) enriches
cell textual descriptions using open-world knowledge; (2) devising a Cross-modal Robust
Alignment (CRA) objective that incorporates sample reliability assessment, curriculum learning, and coupled momentum contrastive learning to strengthen the model's resistance to noisy data. After pretraining on 32M
cell-text pairs, OKR-
CELL obtains cutting-edge results across 6 evaluation tasks. Beyond standard
benchmarks such as
cell clustering,
cell-type annotation, batch-effect correction, and few-shot annotation, the model also demonstrates superior performance in broader multi-modal applications, including zero-shot
cell-type annotation and bidirectional
cell-text
retrieval.
Read the full abstract ▾
All authors Haoran Wang, Xuanyi Zhang, Shuangsang Fang, Longke Ran, Ziqing Deng, Yong Zhang, Yuxiang Li, Shaoshuai Li
Machine Learning
Transformer Is Inherently a Causal Learner
Xinyue Wang, Stephen Wang and Biwei Huang
We reveal that
transformers trained in an autoregressive manner naturally encode time-delayed causal structures in their learned representations. When predicting future values in multivariate time series, the gradient sensitivities of
transformer outputs with respect to past inputs directly recover the underlying causal graph, without any explicit causal objectives or structural constraints. We prove this connection theoretically under standard identifiability conditions and develop a practical extraction method using aggregated gradient attributions. On challenging cases such as nonlinear dynamics, long-term dependencies, and non-stationary systems, this approach greatly surpasses the performance of state-of-the-art discovery
algorithms, especially as data heterogeneity increases, exhibiting scaling potential where causal accuracy improves with data volume and heterogeneity, a property traditional methods lack. This unifying view lays the groundwork for a future paradigm where causal discovery operates through the lens of foundation models, and foundation models gain interpretability and enhancement through the lens of causality.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs
Alireza Dehghanpour Farashah, Aditi Khandelwal, Marylou Fauchard et al. (+3 more)
As
multilingual large
language models become more widely used, ensuring their safety and
fairness across diverse linguistic contexts presents unique challenges. While existing research on machine unlearning has primarily focused on monolingual settings, typically English,
multilingual environments introduce additional complexities due to cross-lingual knowledge transfer and biases embedded in both pretraining and fine-tuning data. In this work, we study
multilingual unlearning using the Aya-Expanse 8B model under two settings: (1) data unlearning and (2) concept unlearning. We extend
benchmarks for factual knowledge and stereotypes to ten
languages through translation: English, French, Arabic, Japanese, Russian, Farsi, Korean, Hindi, Hebrew, and Indonesian. These
languages span five
language families and a wide range of resource levels. Our experiments show that unlearning in high-resource
languages is generally more stable, with asymmetric transfer effects observed between typologically related
languages. Furthermore, our analysis of linguistic distances indicates that syntactic similarity is the strongest predictor of cross-lingual unlearning behavior.
Read the full abstract ▾
All authors Alireza Dehghanpour Farashah, Aditi Khandelwal, Marylou Fauchard, Zhuan Shi, Negar Rostamzadeh, Golnoosh Farnadi
Machine Learning · Cross-listed from cs.CV
Efficient training for compact compression models via sequential distillation
Caroline Mazini Rodrigues, Nicolas Keriven and Thomas Maugey
Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to significantly reduce autoencoder-based compression networks in a more stable Knowledge
Distillation process. The intuition is that highly reduced architectures benefit from simplified optimization objectives in early training, with complexity gradually introduced later. Therefore, our approach begins with a sequential encoder–decoder
distillation stage that provides a robust initialization for the lightweight model. This is followed by standard training that can be regularized with latent
distillation. We evaluate the resulting lightweight autoencoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity in early epochs better than training lightweight autoencoders with the original loss, making it practical for resource-limited environments.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
GenCtrl – A Formal Controllability Toolkit for Generative Models
Emily Cheng, Carmen Amo Alonso, Federico Danieli et al. (+4 more)
As generative models become ubiquitous, there is a critical need for fine-grained control over the generation process. Yet, while controlled generation methods from prompting to fine-tuning proliferate, a fundamental question remains unanswered: are these models truly controllable in the first place? In this work, we provide a theoretical framework to formally answer this question. Framing human-model interaction as a control process, we propose a novel
algorithm to estimate the controllable sets of models in a dialogue setting. Notably, we provide formal guarantees on the estimation error as a function of sample complexity: we derive probably-approximately correct bounds for controllable set estimates that are distribution-free, employ no assumptions except for output boundedness, and work for any black-box nonlinear control system (i.e., any generative model). We empirically demonstrate the theoretical framework on different tasks in controlling dialogue processes, for both
language models and text-to-image generation. Our results show that model controllability is surprisingly fragile and highly dependent on the experimental setting. This highlights the need for rigorous controllability analysis, shifting the focus from simply attempting control to first understanding its fundamental limits.
Read the full abstract ▾
All authors Emily Cheng, Carmen Amo Alonso, Federico Danieli, Arno Blaas, Luca Zappella, Pau Rodriguez, Xavier Suau
Machine Learning
Continual Learning of Achieving Forgetting-free and Positive Knowledge Transfer
Zhi Wang, Zhongbin Wu, Yanni Li et al. (+3 more)
Existing research on continual learning (CL) of a sequence of tasks focuses mainly on dealing with catastrophic forgetting (CF) to balance the learning plasticity of new tasks and the
memory stability of old tasks. However, an ideal CL
agent should not only be able to overcome CF, but also encourage positive forward and backward knowledge transfer (KT), i.e., using the learned knowledge from previous tasks for the new task learning (namely FKT), and improving the previous tasks' performance with the knowledge of the new task (namely BKT). To this end, this paper first models CL as an optimization problem in which each sequential learning task aims to achieve its optimal performance under the constraint that both FKT and BKT should be positive. It then proposes a novel Enhanced Task Continual Learning (ETCL) method, which achieves forgetting-free and positive KT. Furthermore, the bounds that can lead to negative FKT and BKT are estimated theoretically. Based on the bounds, a new strategy for online task similarity detection is also proposed to facilitate positive KT. To overcome CF, ETCL learns a set of task-specific binary masks to isolate a sparse sub-network for each task while preserving the performance of a dense network for the task. At the beginning of a new task learning, ETCL tries to align the new task's gradient with that of the sub-network of the previous most similar task to ensure positive FKT. By using a new bi-objective optimization strategy and an orthogonal gradient projection method, ETCL updates only the weights of previous similar tasks at the classification layer to achieve positive BKT. Extensive evaluations demonstrate that the proposed ETCL markedly outperforms strong baselines on dissimilar, similar, and mixed task sequences.
Read the full abstract ▾
All authors Zhi Wang, Zhongbin Wu, Yanni Li, Bing Liu, Guangxi Li, Yuping Wang
Machine Learning
Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks
ShaoZhen Liu, Xinting Huang, Houwen Peng et al. (+4 more)
In recent years, large
language models (LLMs) have demonstrated significant potential in complex
reasoning tasks like mathematical problem-solving. However, existing research predominantly relies on reinforcement learning (RL) frameworks while overlooking supervised fine-tuning (SFT) methods. This paper proposes a new two-stage training framework that enhances models' self-correction capabilities through self-generated long chain-of-thought (CoT) data. During the first stage, a multi-turn dialogue strategy guides the model to generate CoT data incorporating verification, backtracking, subgoal decomposition, and backward
reasoning, with predefined rules filtering high-quality samples for supervised fine-tuning. The second stage employs a difficulty-aware rejection sampling mechanism to dynamically optimize data distribution, strengthening the model's ability to handle complex problems. The approach generates
reasoning chains extended over 4 times longer while maintaining strong scalability, proving that SFT effectively activates models' intrinsic
reasoning capabilities and provides a resource-efficient pathway for complex task optimization. Experimental results demonstrate performance improvements on mathematical
benchmarks including GSM8K and MATH500, with the fine-tuned model achieving a substantial improvement on competition-level problems like AIME24. Code will be open-sourced.
Read the full abstract ▾
All authors ShaoZhen Liu, Xinting Huang, Houwen Peng, Xin Chen, Xinyang Song, Qi Li, Zhenan Sun
Machine Learning
PiXTime: A Model for Federated Time Series Forecasting with Heterogeneous Data across Nodes
Yiming Zhou, Jiahao Wang, Mingyue Cheng et al. (+3 more)
While collaborative
forecasting on distributed time series is highly desirable, directly pooling localized
datasets is often impractical due to data sharing constraints.
Federated learning offers a promising alternative, yet conventional
federated learning
algorithms require homogeneous model architectures, which are incompatible with the structural discrepancies, such as unaligned temporal resolutions and mismatched variable channels, commonly observed across decentralized nodes. To bridge this gap, we introduce PiXTime, a novel
Transformer-based framework designed to natively accommodate and leverage structurally heterogeneous temporal data. At its core, PiXTime adopts a parameter-decoupling architecture, strategically partitioning the model into localized personalized modules and a globally aggregated shared backbone. Specifically, node-specific local modules act as dimensional adapters, projecting raw sequences of diverse lengths into a unified representation space. Concurrently, a globally synchronized VE Table injects consistent categorical identities into the feature space, allowing the shared backbone to collaboratively learn and generalize representations across inconsistent variable distributions. Comprehensive evaluations on multiple
benchmarks demonstrate that PiXTime achieves state-of-the-art performance in heterogeneous
federated environments, while maintaining robust superiority in standard homogeneous and centralized
forecasting settings.
Read the full abstract ▾
All authors Yiming Zhou, Jiahao Wang, Mingyue Cheng, Hao Wang, Defu Lian, Enhong Chen
Machine Learning
Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
Zijun Min, Bingshuai Liu, Ante Wang et al. (+4 more)
Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising framework for optimizing large
language models in
reasoning tasks. However, existing RLVR
algorithms focus on different granularities, and each has complementary strengths and limitations. Group Relative Policy Optimization (GRPO) updates the policy with
token-level importance ratios, which preserves fine-grained credit assignment but often suffers from high variance and instability. In contrast, Group Sequence Policy Optimization (GSPO) applies single sequence-level importance ratios across all
tokens in a response that better matches sequence-level rewards, but sacrifices
token-wise credit assignment. In this paper, we propose Dynamic Hybrid Policy Optimization (DHPO) to bridge GRPO and GSPO within a single clipped surrogate objective. DHPO combines
token-level and sequence-level importance ratios using weighting mechanisms. We explore two variants of the mixing mechanism, including an averaged mixing and an
entropy-guided mixing. To further stabilize training, we employ a branch-specific clipping strategy that constrains
token-level and sequence-level ratios within separate trust regions before mixing, preventing outliers in either branch from dominating the update. Across seven challenging mathematical
reasoning benchmarks, experiments on both dense and MoE models from the Qwen3 series show that DHPO consistently outperforms GRPO and GSPO. We will release our code upon acceptance of this paper.
Read the full abstract ▾
All authors Zijun Min, Bingshuai Liu, Ante Wang, Long Zhang, Anxiang Zeng, Haibo Zhang, Jinsong Su
Machine Learning · Cross-listed from cs.CV
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes
Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang et al. (+5 more)
Multimodal large
language models often struggle with faithful
reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This
reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified
reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit
language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying
reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful
reasoning. Across seven visual
reasoning benchmarks, SceneAlign consistently improves answer accuracy and
reasoning faithfulness, highlighting the effectiveness of grounding-aware
alignment for
multimodal reasoning.
Read the full abstract ▾
All authors Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, Jingbo Shang