Machine Learning
The Urysohn Ladder: Recursive Metric Contraction for Scalable Continual Learning
Xin Li
Continual learning systems face a fundamental
geometric obstacle: as experience accumulates on a fixed-capacity
manifold, covering numbers grow linearly with time, eventually forcing representational overlap and catastrophic interference. Prevailing approaches attack this problem by expansion - projecting into higher-dimensional spaces via kernels, overparameterization, or replay. We argue the solution is the opposite: contraction. We formalize abstraction as the Urysohn Ladder, a hierarchy of quotient maps that recursively collapse validated metric neighborhoods into compact
tokens, converting unbounded ambient-space search into bounded
navigation on a low-dimensional intrinsic scaffold. Geometrically, each collapsed
token acts as a shortcut - a region of extreme metric contraction that bridges distant experiences, much like a wormhole in the representational
manifold. We establish four results that collectively guarantee separability (metric contraction renders nonlinearly entangled structure linearly separable at each quotient level, and this separability propagates faithfully through the entire hierarchy), bounded capacity (covering numbers remain $O(1)$ per quotient level, independent of stream length), stability (parity-partitioned flow/scaffold subspaces enable unbounded plasticity without catastrophic interference), and scalability (inference cost scales with quotient distance, not ambient distance). We validate each claim empirically with pretrained models and real-world
datasets. Moreover, we demonstrate the potential of Urysohn Ladder for scalable continual learning via scaffold amortization.
Read the full abstract ▾
Machine Learning
Self-organizing maps for water quality assessment in reservoirs and lakes: A systematic literature review
Oraib Almegdadi, João Marcelino, Sarah Fakhreddine et al. (+2 more)
Sustainable water quality underpins ecological balance and water security. Assessing and managing lakes and reservoirs is difficult due to data sparsity, heterogeneity, and nonlinear relationships among parameters. This review examines how Self-Organizing Map (SOM), an unsupervised AI technique, is applied to water quality assessment. It synthesizes research on parameter selection, spatial and temporal sampling strategies, and clustering approaches. Emphasis is placed on how SOM handles multidimensional data and uncovers hidden patterns to support effective water management. The growing availability of environmental data from in-situ sensors, remote sensing imagery, IoT technologies, and historical records has significantly expanded analytical opportunities in environmental monitoring. SOM has proven effective in analysing complex
datasets, particularly when labelled data are limited or unavailable. It enables high-dimensional data visualization, facilitates the detection of hidden ecological patterns, and identifies critical correlations among diverse water quality indicators. This review highlights SOMs versatility in ecological assessments, trophic state classification, algal bloom monitoring, and catchment area impact evaluations. The findings offer comprehensive insights into existing methodologies, supporting future research and practical applications aimed at improving the monitoring and sustainable management of lake and reservoir ecosystems.
Read the full abstract ▾
All authors Oraib Almegdadi, João Marcelino, Sarah Fakhreddine, João Manso, Nuno C. Marques
Machine Learning · Cross-listed from cs.CL
Mitigating Spurious Correlations in NLI via LLM-Synthesized Counterfactuals and Dynamic Balanced Sampling
Christopher Román Jaimes
Natural
Language Inference (NLI) models frequently rely on spurious correlations rather than semantic
reasoning. Existing mitigation strategies often incur high annotation costs or trigger catastrophic forgetting during fine-tuning. We propose an automated, scalable pipeline to address these limitations. First, we introduce Log-Frequency LMI (LF-LMI) to accurately detect semantic artifacts. Second, we generate a high-quality synthetic contrast set via an LLM-synthesis pipeline with multi-judge verification. Finally, we introduce Dynamic Balanced Sampling, a training strategy that rotates the original data distribution to prevent forgetting. Our method improves consistency on a challenging
benchmark from 63.5% to 81.0% while maintaining 88.4% in-domain accuracy, significantly outperforming naive fine-tuning.
Read the full abstract ▾
Machine Learning
Out-of-Distribution Detection in Molecular Complexes via Diffusion Models for Irregular Graphs
David Graber, Victor Armegioiu, Rebecca Buller et al. (+1 more)
Predictive machine learning models generally excel on in-distribution data, but their performance degrades on out-of-distribution (OOD) inputs. Reliable deployment therefore requires robust OOD detection, yet this is particularly challenging for irregular 3D graphs that combine continuous geometry with categorical identities and are unordered by construction. Here, we present a probabilistic OOD detection framework for complex 3D graph data built on a
diffusion model that learns a density of the training distribution in a fully unsupervised manner. A key ingredient we introduce is a unified continuous
diffusion over both 3D coordinates and discrete features: categorical identities are embedded in a continuous space and trained with cross-
entropy, while the corresponding
diffusion score is obtained analytically via posterior-mean interpolation from predicted class probabilities. This yields a single self-consistent probability-flow ODE (PF-ODE) that produces per-sample log-likelihoods, providing a principled typicality score for distribution shift. We validate the approach on
protein-ligand complexes and construct strict OOD
datasets by withholding entire
protein families from training. PF-ODE likelihoods identify held-out families as OOD and correlate strongly with prediction errors of an independent binding-affinity model (GEMS), enabling a priori reliability estimates on new complexes. Beyond scalar likelihoods, we show that multi-scale PF-ODE trajectory statistics - including path tortuosity, flow stiffness, and vector-field instability - provide complementary OOD information. Modeling the joint distribution of these trajectory features yields a practical, high-sensitivity detector that improves separation over likelihood-only baselines, offering a label-free OOD quantification workflow for
geometric deep learning.
Read the full abstract ▾
All authors David Graber, Victor Armegioiu, Rebecca Buller, Siddhartha Mishra
Machine Learning
NOVA: Discovering Well-Conditioned Winograd Transforms through Numerical Optimization of Vandermonde Arithmetic
Jayant Lohia
Winograd convolution is the standard
algorithm for efficient inference, reducing arithmetic complexity by 2.25x for 3x3 kernels. However, it faces a critical barrier in the modern era of low precision computing: numerical instability. As tiles scale to maximize efficiency (e.g., F(6,3), F(8,3)), the condition numbers of standard integer based transforms explode, reaching kappa = 2 x 10^5 for F(8,3), rendering them unusable in FP16 or Int8. We introduce NOVA (Numerical Optimization of Vandermonde Arithmetic), a discovery framework that breaks the decades old convention of integer interpolation. Treating Winograd point selection as a continuous optimization problem, NOVA searches the
manifold R^n-1 via
Evolution Strategy, snaps candidates to simple rationals, and guarantees correctness via symbolic verification. This process uncovers a hidden landscape of stable, fractional configurations such as {+-5/6, +-7/6, +-3/5} that defy traditional vocabulary constraints. The impact is transformative: NOVA improves the conditioning of F(8,3) by 415x in 1D, which squares to a 172,484x improvement for 2D convolution. In real world FP16 ImageNet inference, where standard transforms collapse to random chance (e.g., 4.7 percent accuracy on VGG16), NOVA's points restore full accuracy (75 to 78 percent), recovering over 70 percentage points without retraining, calibration, or learned parameters. These discovered transforms act as drop in replacements, effectively unlocking the efficiency of large tile Winograd convolution for next generation hardware.
Read the full abstract ▾
Machine Learning
Secret mixtures of experts inside your LLM
Enric Boix-Adsera
Despite being one of the earliest neural network layers, the Multilayer Perceptron (MLP) is arguably one of the least understood parts of the
transformer architecture due to its dense computation and lack of easy visualization. This paper seeks to understand the MLP layers in dense LLM models by hypothesizing that these layers secretly approximately perform a sparse computation – namely, that they can be well approximated by sparsely-activating Mixture of Experts (MoE) layers. Our hypothesis is based on a novel theoretical connection between MoE models and Sparse Autoencoder (SAE) structure in activation space. We empirically validate the hypothesis on pretrained LLMs, and demonstrate that the activation distribution matters – these results do not hold for Gaussian data, but rather rely crucially on structure in the distribution of neural network activations. Our results shine light on a general principle at play in MLP layers inside LLMs, and give an explanation for the effectiveness of modern MoE-based
transformers. Additionally, our experimental explorations suggest new directions for more efficient MoE architecture design based on low-rank routers.
Read the full abstract ▾
Machine Learning
On the Universality of Transformer Architectures; How Much Attention Is Enough?
Amirreza Abbasi and Mohsen Hooshmand
Transformers are crucial across many AI fields, such as large
language models, computer
vision, and reinforcement learning. This prominence stems from the architecture's perceived universality and scalability compared to alternatives. This work examines the problem of universality in
Transformers, reviews recent progress, including architectural refinements such as structural minimality and approximation rates, and surveys state-of-the-art advances that inform both theoretical and practical understanding. Our aim is to clarify what is currently known about
Transformers expressiveness, separate robust guarantees from fragile ones, and identify key directions for future theoretical research.
Read the full abstract ▾
Machine Learning
MoE Pathfinder: Trajectory-driven Expert Pruning
Xican Yang, Yuanhe Tian and Yan Song
Mixture-of-experts (MoE) architectures used in large
language models (LLMs) achieve state-of-the-art performance across diverse tasks yet face practical challenges such as deployment complexity and low activation efficiency. Expert
pruning has thus emerged as a promising solution to reduce computational overhead and simplify the deployment of MoE models. However, existing expert
pruning approaches conventionally rely on local importance metrics and often apply uniform layer-wise
pruning, leveraging only partial evaluation signals and overlooking the heterogeneous contributions of experts across layers. To address these limitations, we propose an expert
pruning approach based on the trajectory of activated experts across layers, which treats MoE as a weighted computation graph and casts expert selection as a global optimal path planning problem. Within this framework, we integrate complementary importance signals from reconstruction error, routing probabilities, and activation strength at the trajectory level, which naturally yields non-uniform expert retention across layers. Experiments show that our approach achieves superior
pruning performance on nearly all tasks compared with most existing approaches.
Read the full abstract ▾
Machine Learning
Why Most Optimism Bandit Algorithms Have the Same Regret Analysis: A Simple Unifying Theorem
Vikram Krishnamurthy
Several optimism-based
stochastic bandit
algorithms – including UCB, UCB-V, linear UCB, and finite-arm GP-UCB – achieve logarithmic regret using proofs that, despite superficial differences, follow essentially the same structure. This note isolates the minimal ingredients behind these analyses: a single high-probability concentration condition on the
estimators, after which logarithmic regret follows from two short deterministic lemmas describing radius collapse and optimism-forced deviations. The framework yields unified, near-minimal proofs for these classical
algorithms and extends naturally to many contemporary bandit variants.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Automated Mosaic Tesserae Segmentation via Deep Learning Techniques
Charilaos Kapelonis, Marios Antonakakis, Konstantinos Politof et al. (+2 more)
Art is widely recognized as a reflection of civilization and mosaics represent an important part of cultural heritage. Mosaics are an ancient art form created by arranging small pieces, called tesserae, on a surface using adhesive. Due to their age and fragility, they are prone to damage, highlighting the need for digital preservation. This paper addresses the problem of digitizing mosaics by segmenting the tesserae to separate them from the background within the broader field of Image
Segmentation in Computer
Vision. We propose a method leveraging Segment Anything Model 2 (SAM 2) by Meta AI, a foundation model that outperforms most conventional
segmentation models, to automatically segment mosaics. Due to the limited open
datasets in the field, we also create an annotated
dataset of mosaic images to fine-tune and evaluate the model. Quantitative evaluation on our testing
dataset shows notable improvements compared to the baseline SAM 2 model, with Intersection over Union increasing from 89.00% to 91.02% and Recall from 92.12% to 95.89%. Additionally, on a
benchmark proposed by a prior approach, our model achieves an F-measure 3% higher than previous methods and reduces the error in the absolute difference between predicted and actual tesserae from 0.20 to just 0.02. The notable performance of the fine-tuned SAM 2 model together with the newly annotated
dataset can pave the way for real-time
segmentation of mosaic images.
Read the full abstract ▾
All authors Charilaos Kapelonis, Marios Antonakakis, Konstantinos Politof, Aristomenis Antoniadis, Michalis Zervakis
Machine Learning
The Challenger: When Do New Data Sources Justify Switching Machine Learning Models?
Vassilis Digalakis Jr, Christophe Pérignon, Sébastien Saurin et al. (+1 more)
Organizations often have an incumbent predictive model in production when new data sources become available. Because historical training data lack the new features, a challenger model must be trained on a small but growing full-feature
dataset. We study whether, and when, the organization should switch to the challenger. The decision is statistical and economic: the challenger's predictive performance improves as full-feature data accumulate, but repeated retraining is costly and delays benefits from deployment. We develop a framework linking learning-curve dynamics to model-switching economics. Under a standard power-law learning curve and finite data-collection horizon $T$, the optimal time to train and evaluate the challenger scales as $T^{1/(1+\alpha)}$: learning-curve shape (through its learning speed $\alpha$) is the primary theoretical determinant of when to stop experimenting; costs determine switching profitability. Even without knowing the learning curve, the operational problem is tractable: we show that any
algorithm stopping on the $T^{2/3}$ scale and making reliable switch/discard decisions achieves $O(T^{2/3}\sqrt{\log T})$ regret relative to a full-foresight oracle. We propose a sequential evaluation
algorithm that uses local learning-curve trends to anticipate improvement, and test it in a real-world credit-scoring study. Even with this local approximation, the
algorithm theoretically and empirically achieves near-oracle performance. It is also more stable than greedy sequential evaluation
algorithms, where noisy early estimates trigger premature discarding, or simple one-shot evaluation
algorithms, which work only when their fixed evaluation time matches the (unknown in practice) theoretical timing scale. Our framework offers a step toward principled model governance when new data sources require costly collection, validation, and deployment.
Read the full abstract ▾
All authors Vassilis Digalakis Jr, Christophe Pérignon, Sébastien Saurin, Flore Sentenac
Machine Learning · Cross-listed from eess.SY
Neural Proofs for Sound Verification and Control of Complex Systems
Alessandro Abate
This informal contribution presents an ongoing line of research that is pursuing a new approach to the construction of sound proofs for the formal verification and control of complex
stochastic models of dynamical systems, of reactive programs and, more generally, of models of Cyber-Physical Systems. Neural proofs are made up of two key components: 1) proof rules encode requirements entailing the verification of general temporal specifications over the models of interest; and 2) certificates that discharge such rules, namely they are constructed from said proof rules with an inductive (that is, cyclic, repetitive) approach; this inductive approach involves: 2a) accessing samples from the model's dynamics and accordingly training neural networks, whilst 2b) generalising such networks via SAT-modulo-theory (SMT) queries that leverage the full knowledge of the models. In the context of sequential decision making problems over complex
stochastic models, it is possible to additionally generate provably-correct policies/strategies/controllers, namely state-feedback functions that, in conjunction with neural certificates, formally attain the given specifications for the models of interest.
Read the full abstract ▾
Machine Learning
Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale
Ansh Nagwekar
Neural network optimization remains one of the most consequential yet poorly understood challenges in modern AI research, where improvements in training
algorithms can lead to enhanced feature learning in foundation models, order-of-magnitude reductions in training time, and improved interpretability into how networks learn. While
stochastic gradient descent (SGD) and its variants have become the de facto standard for training deep networks, their success in these over-parameterized regimes often appears more empirical than principled. This thesis investigates this apparent paradox by tracing the
evolution of optimization
algorithms from classical first-order methods to modern higher-order techniques, revealing how principled algorithmic design can demystify the training process. Starting from first principles with SGD and adaptive gradient methods, the analysis progressively uncovers the limitations of these conventional approaches when confronted with anisotropy that is representative of real-world data. These breakdowns motivate the exploration of sophisticated alternatives rooted in
curvature information: second-order approximation techniques, layer-wise preconditioning, adaptive learning rates, and more. Next, the interplay between these optimization
algorithms and the broader neural network training toolkit, which includes prior and recent developments such as maximal update parametrization, learning rate schedules, and exponential moving averages, emerges as equally essential to empirical success. To bridge the gap between theoretical understanding and practical deployment, this paper offers practical prescriptions and implementation strategies for integrating these methods into modern deep learning workflows.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.IV
PSI3D: Plug-and-Play 3D Stochastic Inference with Slice-wise Latent Diffusion Prior
Wenhan Guo, Jinglun Yu, Yaning Wang et al. (+2 more)
Diffusion models are highly expressive image priors for
Bayesian inverse problems. However, most
diffusion models cannot operate on large-scale, high-dimensional data due to high training and inference costs. In this work, we introduce a Plug-and-play
algorithm for 3D
stochastic inference with latent
diffusion prior (PSI3D) to address massive ($1024\times 1024\times 128$) volumes. Specifically, we formulate a Markov chain Monte Carlo approach to reconstruct each two-dimensional (2D) slice by sampling from a 2D latent
diffusion model. To enhance inter-slice consistency, we also incorporate total variation (TV) regularization stochastically along the concatenation axis. We evaluate our performance on optical coherence
tomography (OCT) super-resolution. Our method significantly improves reconstruction quality for large-scale scientific imaging compared to traditional and learning-based baselines, while providing robust and credible reconstructions.
Read the full abstract ▾
All authors Wenhan Guo, Jinglun Yu, Yaning Wang, Jin U. Kang, Yu Sun
Machine Learning · Cross-listed from cs.CV
Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance
Badr Moufad, Navid Bagheri Shouraki, Alain Oliviero Durmus et al. (+4 more)
Diffusion models have emerged as powerful priors for image editing tasks such as inpainting and local modification, where the objective is to generate realistic content that remains consistent with observed regions. In particular, zero-shot approaches that leverage a pretrained
diffusion model, without any retraining, have been shown to achieve highly effective reconstructions. However, state-of-the-art zero-shot methods typically rely on a sequence of surrogate likelihood functions, whose scores are used as proxies for the ideal score. This procedure however requires vector-Jacobian products through the denoiser at every reverse step, introducing significant
memory and runtime overhead. To address this issue, we propose a new likelihood surrogate that yields simple and efficient to sample Gaussian posterior transitions, sidestepping the backpropagation through the denoiser network. Our extensive experiments show that our method achieves strong observation consistency compared with fine-tuned baselines and produces coherent, high-quality reconstructions, all while significantly reducing inference cost. Code is available at https://github.com/YazidJanati/ding.
Read the full abstract ▾
All authors Badr Moufad, Navid Bagheri Shouraki, Alain Oliviero Durmus, Thomas Hirtz, Eric Moulines, Jimmy Olsson, Yazid Janati
Machine Learning · Cross-listed from cs.RO
Dynamic Entropy Tuning in Reinforcement Learning Low-Level Quadcopter Control: Stochasticity vs Determinism
Youssef Mahran, Zeyad Gamal and Ayman El-Badawy
This paper explores the impact of dynamic
entropy tuning in Reinforcement Learning (RL)
algorithms that train a
stochastic policy. Its performance is compared against
algorithms that train a deterministic one.
Stochastic policies optimize a probability distribution over actions to maximize rewards, while deterministic policies select a single deterministic action per state. The effect of training a
stochastic policy with both static
entropy and dynamic
entropy and then executing deterministic actions to control the quadcopter is explored. It is then compared against training a deterministic policy and executing deterministic actions. For the purpose of this research, the Soft Actor-Critic (SAC)
algorithm was chosen for the
stochastic algorithm while the Twin Delayed Deep Deterministic Policy Gradient (TD3) was chosen for the deterministic
algorithm. The training and simulation results show the positive effect the dynamic
entropy tuning has on controlling the quadcopter by preventing catastrophic forgetting and improving exploration efficiency.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Reinforcement Learning Position Control of a Quadrotor Using Soft Actor-Critic (SAC)
Youssef Mahran, Zeyad Gamal and Ayman El-Badawy
This paper proposes a new Reinforcement Learning (RL) based control architecture for quadrotors. With the literature focusing on controlling the four rotors' RPMs directly, this paper aims to control the quadrotor's thrust vector. The RL
agent computes the percentage of overall thrust along the quadrotor's z-axis along with the desired Roll ($\phi$) and Pitch ($\theta$) angles. The
agent then sends the calculated control signals along with the current quadrotor's Yaw angle ($\psi$) to an attitude PID controller. The PID controller then maps the control signals to motor RPMs. The Soft Actor-Critic
algorithm, a model-free off-policy
stochastic RL
algorithm, was used to train the RL
agents. Training results show the faster training time of the proposed thrust vector controller in comparison to the conventional RPM controllers. Simulation results show smoother and more accurate path-following for the proposed thrust vector controller.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
A two-stream network with global-local feature fusion for bone age assessment
Qiong Lou, Han Yang and Fang Lu
Bone Age Assessment (BAA) is a widely used clinical technique that can accurately reflect an individual's growth and development level, as well as maturity. In recent years, although deep learning has advanced the field of bone age assessment, existing methods face challenges in efficiently balancing global features and local skeletal details. This study aims to develop an automated bone age assessment system based on a two-stream deep learning architecture to achieve higher accuracy in bone age assessment. We propose the BoNet+ model incorporating global and local feature extraction channels. A
Transformer module is introduced into the global feature extraction channel to enhance the ability in extracting global features through multi-head self-
attention mechanism. A RFAConv module is incorporated into the local feature extraction channel to generate adaptive
attention maps within multiscale receptive fields, enhancing local feature extraction capabilities. Global and local features are concatenated along the channel dimension and optimized by an Inception-V3 network. The proposed method has been validated on the Radiological Society of North America (RSNA) and Radiological Hand Pose Estimation (RHPE) test
datasets, achieving mean absolute errors (MAEs) of 3.81 and 5.65 months, respectively. These results are comparable to the state-of-the-art. The BoNet+ model reduces the clinical workload and achieves automatic, high-precision, and more objective bone age assessment.
Read the full abstract ▾
Machine Learning
Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systems
Vincent Bezold, Patrick Wagner, Jakob Hofmann et al. (+2 more)
This paper presents a trustworthy reinforcement learning approach for the control of industrial compressed air systems. We develop a framework that enables safe and energy-efficient operation under realistic boundary conditions and introduce a multi-level explainability pipeline combining input perturbation tests, gradient-based sensitivity analysis, and SHAP (SHapley Additive exPlanations) feature attribution. An empirical evaluation across multiple compressor configurations shows that the learned policy is physically plausible, anticipates future demand, and consistently respects system boundaries. Compared to the installed industrial controller, the proposed approach reduces unnecessary overpressure and achieves energy savings of approximately 4\,% without relying on explicit physics models. The results further indicate that system pressure and forecast information dominate policy decisions, while compressor-level inputs play a secondary role. Overall, the combination of efficiency gains, predictive behavior, and transparent validation supports the trustworthy deployment of reinforcement learning in industrial energy systems.
Read the full abstract ▾
All authors Vincent Bezold, Patrick Wagner, Jakob Hofmann, Marco Huber, Alexander Sauer
Machine Learning
Embedded Safety-Aligned Intelligence via Differentiable Internal Alignment Embeddings
Harsh Rathva, Ojas Srivastava and Pruthwik Mishra
We introduce Embedded Safety-Aligned Intelligence (ESAI), a theoretical framework for multi-
agent reinforcement learning that embeds
alignment constraints directly into
agents internal representations using differentiable internal
alignment embeddings. Unlike external reward shaping or post-hoc safety constraints, internal
alignment embeddings are learned latent variables that predict externalized harm through counterfactual
reasoning and modulate policy updates toward harm reduction through
attention and graph-based propagation. The ESAI framework integrates four mechanisms: differentiable counterfactual
alignment penalties computed from soft reference distributions,
alignment-weighted perceptual
attention, Hebbian associative
memory supporting temporal credit assignment, and similarity-weighted graph
diffusion with bias mitigation controls. We analyze stability conditions for bounded internal
embeddings under Lipschitz continuity and spectral constraints, discuss computational complexity, and examine theoretical properties including contraction behavior and
fairness-performance tradeoffs. This work positions ESAI as a conceptual contribution to differentiable
alignment mechanisms in multi-
agent systems. We identify open theoretical questions regarding convergence guarantees,
embedding dimensionality, and extension to high-dimensional environments. Empirical evaluation is left to future work.
Read the full abstract ▾
Machine Learning
AL-GNN: Privacy-Preserving and Replay-Free Continual Graph Learning via Analytic Learning
Xuling Zhang, Jindong Li, Yifei Zhang et al. (+2 more)
Continual graph learning (CGL) aims to enable graph neural networks to incrementally learn from a stream of graph structured data without forgetting previously acquired knowledge. Existing methods particularly those based on experience replay typically store and revisit past graph data to mitigate catastrophic forgetting. However, these approaches pose significant limitations, including
privacy concerns, inefficiency. In this work, we propose AL GNN, a novel framework for continual graph learning that eliminates the need for backpropagation and replay buffers. Instead, AL GNN leverages principles from analytic learning theory to formulate learning as a recursive least squares optimization process. It maintains and updates model knowledge analytically through closed form classifier updates and a regularized feature autocorrelation
matrix. This design enables efficient one pass training for each task, and inherently preserves data
privacy by avoiding historical sample storage. Extensive experiments on multiple dynamic graph classification
benchmarks demonstrate that AL GNN achieves competitive or superior performance compared to existing methods. For instance, it improves average performance by 10% on CoraFull and reduces forgetting by over 30% on Reddit, while also reducing training time by nearly 50% due to its backpropagation free design.
Read the full abstract ▾
All authors Xuling Zhang, Jindong Li, Yifei Zhang, Mingqi Yang, Menglin Yang
Machine Learning
FedSUM Family: Efficient Federated Learning Methods under Arbitrary Client Participation
Runze You and Shi Pu
Federated Learning (FL) methods are often designed for specific client participation patterns, limiting their applicability in practical deployments. We introduce the FedSUM family of
algorithms, which supports arbitrary client participation without additional assumptions on data heterogeneity. Our framework models participation variability with two delay metrics, the maximum delay $\tau_{\max}$ and the average delay $\tau_{\text{avg}}$. The FedSUM family comprises three variants: FedSUM-B (basic version), FedSUM (standard version), and FedSUM-CR (communication-reduced version). We provide unified convergence guarantees demonstrating the effectiveness of our approach across diverse participation patterns, thereby broadening the applicability of FL in real-world scenarios.
Read the full abstract ▾
Machine Learning
LeJOT: An Intelligent Job Cost Orchestration Solution for Databricks Platform
Lizhi Ma, Yi-Xiang Hu, Yuke Wang et al. (+5 more)
With the rapid advancements in big data technologies, the Databricks platform has become a cornerstone for enterprises and research institutions, offering high computational efficiency and a robust ecosystem. However, managing the escalating operational costs associated with job execution remains a critical challenge. Existing solutions rely on static configurations or reactive adjustments, which fail to adapt to the dynamic nature of workloads. To address this, we introduce LeJOT, an intelligent job cost orchestration framework that leverages machine learning for execution time prediction and a solver-based optimization model for real-time resource allocation. Unlike conventional scheduling techniques, LeJOT proactively predicts workload demands, dynamically allocates computing resources, and minimizes costs while ensuring performance requirements are met. Experimental results on real-world Databricks workloads demonstrate that LeJOT achieves an average 20% reduction in cloud computing costs within a minute-level scheduling timeframe, outperforming traditional static allocation strategies. Our approach provides a scalable and adaptive solution for cost-efficient job scheduling in Data Lakehouse environments.
Read the full abstract ▾
All authors Lizhi Ma, Yi-Xiang Hu, Yuke Wang, Yifang Zhao, Yihui Ren, Jian-Xiang Liao, Feng Wu, Xiang-Yang Li
Machine Learning · Cross-listed from eess.AS
TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognition
Haolong Zheng, Yekaterina Yegorova and Mark Hasegawa-Johnson
Children's
speech recognition remains challenging due to substantial acoustic and linguistic variability, limited labeled data, and significant differences from adult
speech.
Speech foundation models can address these challenges through
Speech In-Context Learning (SICL), allowing adaptation to new domains without fine-tuning. However, the effectiveness of SICL depends on how in-context examples are selected. We extend an existing
retrieval-based method, Text-
Embedding KNN for SICL (TICL), introducing an acoustic reranking step to create TICL+. This extension prioritizes examples that are both semantically and acoustically aligned with the test input. Experiments on four children's
speech corpora show that TICL+ achieves up to a 53.3% relative word error rate reduction over zero-shot performance and 37.6% over baseline TICL, highlighting the value of combining semantic and acoustic information for robust, scalable ASR in children's
speech.
Read the full abstract ▾
Machine Learning · Cross-listed from cond-mat.mtrl-sci
CrystalFormer-CSP: Thinking Fast and Slow for Crystal Structure Prediction
Zhendong Cao, Shigang Ou and Lei Wang
Crystal structure prediction is a fundamental problem in materials science. We present CrystalFormer-CSP, an efficient framework that unifies data-driven heuristic and physics-driven optimization approaches to predict stable
crystal structures for given chemical compositions. The approach combines pretrained generative models for space-group-informed structure generation and a universal machine learning force field for energy minimization. Reinforcement fine-tuning can be employed to further boost the accuracy of the framework. We demonstrate the effectiveness of CrystalFormer-CSP on
benchmark problems and showcase its usage via web interface and
language model integration.
Read the full abstract ▾
Machine Learning
On the Convergence Rate of LoRA Gradient Descent
Siqiao Mu and Diego Klabjan
The low-rank adaptation (LoRA)
algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two “adapter"
matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the original LoRA gradient descent
algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter
matrices, ii) a modified descent lemma for the “Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations. We conduct numerical experiments to validate our theoretical findings.
Read the full abstract ▾
Machine Learning
Offline Behavioral Data Selection
Shiye Lei, Zhihao Cheng and Dacheng Tao
Behavioral cloning is a widely adopted approach for offline policy learning from expert demonstrations. However, the large scale of offline behavioral
datasets often results in computationally intensive training when used in downstream tasks. In this paper, we uncover the striking data saturation in offline behavioral data: policy performance rapidly saturates when trained on a small fraction of the
dataset. We attribute this effect to the weak
alignment between policy performance and test loss, revealing substantial room for improvement through data selection. To this end, we propose a simple yet effective method, Stepwise Dual Ranking (SDR), which extracts a compact yet informative subset from large-scale offline behavioral
datasets. SDR is build on two key principles: (1) stepwise clip, which prioritizes early-stage data; and (2) dual ranking, which selects samples with both high action-value rank and low state-density rank. Extensive experiments and ablation studies on D4RL
benchmarks demonstrate that SDR significantly enhances data selection for offline behavioral data.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
AutoSchA: Automatic Hierarchical Music Representations via Multi-Relational Node Isolation
Stephen Ni-Hahn, Rico Zhu, Jerry Yin et al. (+3 more)
Hierarchical representations provide powerful and principled approaches for analyzing many musical genres. Such representations have been broadly studied in music theory, for instance via Schenkerian analysis (SchA). Hierarchical music analyses, however, are highly cost-intensive; the analysis of a single piece of music requires a great deal of time and effort from trained experts. The representation of hierarchical analyses in a computer-readable format is a further challenge. Given recent developments in hierarchical deep learning and increasing quantities of computer-readable data, there is great promise in extending such work for an automatic hierarchical representation framework. This paper thus introduces a novel approach, AutoSchA, which extends recent developments in graph neural networks (GNNs) for hierarchical music analysis. AutoSchA features three key contributions: 1) a new graph learning framework for hierarchical music representation, 2) a new graph pooling mechanism based on node isolation that directly optimizes learned pooling assignments, and 3) a state-of-the-art architecture that integrates such developments for automatic hierarchical music analysis. We show, in a suite of experiments, that AutoSchA performs comparably to human experts when analyzing Baroque fugue subjects.
Read the full abstract ▾
All authors Stephen Ni-Hahn, Rico Zhu, Jerry Yin, Yue Jiang, Cynthia Rudin, Simon Mak
Machine Learning · Cross-listed from cs.HC
Dimensionality Reduction Considered Harmful (Some of the Time)
Hyeon Jeon
Visual analytics now plays a central role in decision-making across diverse disciplines, but it can be unreliable: the knowledge or insights derived from the analysis may not accurately reflect the underlying data. In this dissertation, we improve the reliability of visual analytics with a focus on dimensionality reduction (DR). DR techniques enable visual analysis of high-dimensional data by reducing it to two or three dimensions, but they inherently introduce errors that can compromise the reliability of visual analytics. To this end, I investigate reliability challenges that practitioners face when using DR for visual analytics. Then, I propose technical solutions to address these challenges, including new evaluation metrics, optimization strategies, and interaction techniques. We conclude the thesis by discussing how our contributions lay the foundation for achieving more reliable visual analytics practices.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization
Lichen Yang, Qiang Wang, Zhonghao Yang et al. (+2 more)
Graph Neural Networks (GNNs) have demonstrated remarkable efficacy in handling graph-structured data; however, they exhibit failures after deployment, which can cause severe consequences. Hence, conducting thorough testing before deployment becomes imperative to ensure the reliability of GNNs. However, thorough testing requires numerous manually annotated test data. To mitigate the annotation cost, strategically prioritizing and labeling high-quality unlabeled inputs for testing becomes crucial, which facilitates uncovering more model failures with a limited labeling budget. Unfortunately, existing test input prioritization techniques either overlook the valuable information contained in graph structures or are overly reliant on attributes extracted from the target model, i.e., model-aware attributes, whose quality can vary significantly. To address these issues, we propose a novel test input prioritization framework, named GraphRank, for GNNs. GraphRank introduces model-agnostic attributes to compensate for the limitations of the model-aware ones. It also leverages the graph structure information to aggregate attributes from neighboring nodes, thereby enhancing the model-aware and model-agnostic attributes. Furthermore, GraphRank combines the above attributes with a binary classifier, using it as a ranking model to prioritize inputs. This classifier undergoes iterative training, which enables it to learn from each round's feedback and improve its performance accordingly. Extensive experiments demonstrate GraphRank's superiority over existing techniques.
Read the full abstract ▾
All authors Lichen Yang, Qiang Wang, Zhonghao Yang, Daojing He, Yu Li