Machine Learning · Cross-listed from stat.AP
Multi-output Extreme Spatial Model for Complex Aircraft Production Systems
Cheolhei Lee, Xing Wang, Xiaowei Yue et al. (+1 more)
Problem definition: Data-driven models in machine learning have enabled efficient management of production systems. However, a majority of machine learning models are devoted to modeling the mean response or average pattern, which is inappropriate for studying abnormal extreme events that are often of primary interest in aircraft manufacturing. Since extreme events from heavy-tailed distributions give rise to prohibitive expenditures in system management, sophisticated extreme models are urgently needed to analyze complex extreme risks. Engineering applications of extreme models usually focus on individual extreme events, which is insufficient for complex systems with correlations. Methodology/results: We introduce an extreme spatial model for multi-output response control systems that efficiently captures the dynamics using a bilinear function on two spatial domains for control variables and measurement locations. Marginal parameter modeling and extremal dependence have been investigated. In addition, an efficient graph-assisted composite likelihood estimation and corresponding computational
algorithms are developed to cope with high-dimensional outputs. The application to composite aircraft production shows that the proposed model enables comprehensive analyses with superior predictive performance on extreme events compared to canonical methods. Managerial implications: Our method shows how to use an extreme spatial model for predicting extreme events and managing extreme risks in complex production systems such as aircraft. This can help achieve better quality management and operation safety in aircraft production systems and beyond.
Read the full abstract ▾
All authors Cheolhei Lee, Xing Wang, Xiaowei Yue, Jianguo Wu
Machine Learning
On the Properties of Feature Attribution for Supervised Contrastive Learning
Leonardo Arrighi, Julia Eva Belloni, Aurélie Gallet et al. (+3 more)
Most Neural Networks (NNs) for classification are trained using Cross-
Entropy as a loss function. This approach requires the model to have an explicit classification layer. However, there exist alternative approaches, such as Contrastive Learning (CL). Instead of explicitly operating a classification, CL has the NN produce an
embedding space where projections of similar data are pulled together, while projections of dissimilar data are pushed apart. In the case of Supervised CL (SCL), labels are adopted as similarity criteria, thus creating an
embedding space where the projected data points are well-clustered. SCL provides crucial advantages over CE with regard to
adversarial robustness and out-of-distribution detection, thus making it a more natural choice in safety-critical scenarios. In the present paper, we empirically show that NNs for image classification trained with SCL present higher-quality feature attribution explanations than CL with regard to faithfulness, complexity, and continuity. These results reinforce previous findings about CL-based approaches when targeting more trustworthy and transparent NNs and can guide practitioners in the selection of training objectives targeting not only accuracy, but also transparency of the models.
Read the full abstract ▾
All authors Leonardo Arrighi, Julia Eva Belloni, Aurélie Gallet, Ivan Gentile, Matteo Lippi, Marco Zullich
Machine Learning
An Integrated Framework for Explainable, Fair, and Observable Hospital Readmission Prediction: Development and Validation on MIMIC-IV
Isaac Tosin Adisa
Objective: To propose and retrospectively validate an integrated framework addressing three barriers to clinical translation of readmission prediction: lack of explainability, absence of deployment reliability infrastructure, and inadequate demographic
fairness evaluation. Materials and Methods: We constructed a cohort of 415231 adult admissions from the MIMIC-IV database (30-day readmission prevalence 18.0%), split 70/15/15. Logistic regression, XGBoost, and LightGBM models were trained on 26 features. SHAP provided per-patient explanations.
Fairness was evaluated across 16 subgroups using AUC-ROC, false negative rate (FNR), and positive predictive value (PPV). Calibration was assessed using Brier scores and calibration curves. Results: XGBoost achieved AUC-ROC 0.696 (95% CI 0.691-0.701), outperforming or matching the LACE baseline (AUC 0.60-0.68). LightGBM achieved best calibration (Brier 0.146). Prior admissions were the dominant predictor. All subgroups met equity thresholds (delta AUC <= 0.05, delta FNR <= 0.10). Conclusion: This framework delivers competitive performance, clinically actionable explanations, and strong demographic equity. Code is publicly available at https://github.com/Tomisin92/readmission-prediction.
Read the full abstract ▾
Machine Learning
FeatEHR-LLM: Leveraging Large Language Models for Feature Engineering in Electronic Health Records
Hojjat Karami, David Atienza, Jean-Philippe Thiran et al. (+1 more)
Feature engineering for Electronic Health Records (EHR) is complicated by irregular observation intervals, variable measurement frequencies, and structural sparsity inherent to clinical time series. Existing automated methods either lack clinical domain awareness or assume clean, regularly sampled inputs, limiting their applicability to real-world EHR data. We present FeatEHR-LLM, a framework that leverages Large
Language Models (LLMs) to generate clinically meaningful tabular features from irregularly sampled EHR time series. To limit patient
privacy exposure, the LLM operates exclusively on
dataset schemas and task descriptions rather than raw patient records. A tool-augmented generation mechanism equips the LLM with specialized routines for querying irregular temporal data, enabling it to produce executable feature-extraction code that explicitly handles uneven observation patterns and informative sparsity. FeatEHR-LLM supports both univariate and multivariate feature generation through an iterative, validation-in-the-loop pipeline. Evaluated on eight clinical prediction tasks across four ICU
datasets, our framework achieves the highest mean AUROC on 7 out of 8 tasks, with improvements of up to 6 percentage points over strong baselines. Code is available at github.com/hojjatkarami/FeatEHR-LLM.
Read the full abstract ▾
All authors Hojjat Karami, David Atienza, Jean-Philippe Thiran, Anisoara Ionescu
Machine Learning · Cross-listed from cs.CV
Different Strokes for Different Folks: Writer Identification for Historical Arabic Manuscripts
Hamza A. Abushahla, Ariel Justine N. Panopio, Layth Al-Khairulla et al. (+1 more)
Handwritten Arabic manuscripts preserve the Arab world's intellectual and cultural heritage, and writer identification supports provenance, authenticity verification, and historical analysis. Using the Muharaf
dataset of historical Arabic manuscripts, we evaluate writer identification from individual line images and, to the best of our knowledge, provide the first baselines reported under both line-level and page-disjoint evaluation protocols. Since the
dataset is only partially labeled for writer identification, we manually verified and expanded writer labels in the public portion from 6,858 (28.00%) to 21,249 lines (86.75%) out of 24,495 line images, correcting inconsistencies and removing non-handwritten text. After further filtering, we retained 18,987 lines (77.51%). We propose a Convolutional Neural Network (CNN)-based model with
attention mechanisms for closed-set writer identification, including rare two-writer lines modeled as composite writer-pair classes. We
benchmark fourteen configurations and conduct ablations across different feature extractors and training regimes. To assess generalization to unseen pages, the page-disjoint protocol assigns all lines from each page to a single split. Under the line-level protocol, a fine-tuned DenseNet201 with
attention achieves 99.05% Top-1 accuracy, 99.73% Top-5 accuracy, and 97.44% F1-score. Under the more challenging page-disjoint protocol, the best observed results are 78.61% Top-1 accuracy, 87.79% Top-5 accuracy, and 66.55% F1-score, thus quantifying the impact of page-level cues. By expanding the Muharaf
dataset's labeled subset and reporting both protocols, we provide a clearer
benchmark and a practical resource for historians and linguists engaged with culturally and historically significant documents. The code and implementation details are available on GitHub.
Read the full abstract ▾
All authors Hamza A. Abushahla, Ariel Justine N. Panopio, Layth Al-Khairulla, Mohamed I. AlHajri
Machine Learning
Decoding High-Dimensional Finger Motion from EMG Using Riemannian Features and RNNs
Martin Colot, Cédric Simar, Guy Cheron et al. (+2 more)
Continuous estimation of high-dimensional finger kinematics from forearm surface electromyography (EMG) could enable natural control for hand prostheses, AR/XR interfaces, and teleoperation. However, the complexity of human hand gestures and the
entanglement of forearm muscles make accurate recognition intrinsically challenging. Existing approaches typically reduce task complexity by relying on classification-based machine learning, limiting the controllable degrees of freedom and compromising on natural interaction. We present an end-to-end framework for continuous EMG-to-kinematics regression using only consumer-grade hardware. The framework combines an 8-channel EMG armband, a single webcam, and an automatic synchronization procedure, enabling the collection of the EMG Finger-Kinematics
dataset (EMG-FK), a 10-h
dataset of synchronized EMG and 15 finger joint angles from 20 participants performing rich, unconstrained right-hand motions. We also introduce the Temporal Riemannian Regressor (TRR), a lightweight GRU-based model that uses sequences of multi-band Riemannian covariance features to decode finger motion. Across EMG-FK and the public emg2pose
benchmark, TRR outperforms state-of-the-art methods in both intra- and cross-subject evaluation. On EMG-FK, it reaches an average absolute error of $9.79 \deg \pm 1.48$ in intra-subject and $16.71 \deg \pm 3.97$ in cross-subject. Finally, we demonstrate real-time deployment on a Raspberry Pi 5 and intuitive control of a robotic hand; TRR runs at nearly 10 predictions/s and is roughly an order of magnitude faster than state-of-the-art approaches. Together, these contributions lower the barrier to reproducible, real-time EMG-based decoding of high-dimensional finger motion, and pave the way toward more natural and intuitive control of embedded EMG-based systems.
Read the full abstract ▾
All authors Martin Colot, Cédric Simar, Guy Cheron, Ana Maria Cebolla Alvarez, Gianluca Bontempi
Machine Learning
Deep Learning for Model Calibration in Simulation of Itaconic Acid Production
Daria Fokina, Marco Baldan, Constantin Romankiewicz et al. (+3 more)
In this study, deep learning is used to estimate kinetic parameters for modeling itaconic acid production based on real batch experiments conducted at different agitation speeds and reactor scales. Two deep learning strategies, namely direct deep learning (DDL) and generative conditional flow matching (CFM) are compared and benchmarked against nonlinear regression as a reference method. Compared with DDL, CFM consistently yields more accurate results. The concentration profiles predicted by CFM closely match those obtained from nonlinear regression, whereas DDL results in larger deviations. Similar behavior is observed in the scale-up experiments, where the CFM model again generalizes better and is more robust than the direct approach. These findings demonstrate that CFM can reliably predict system behavior across different operating conditions and scales, offering a flexible and data-efficient framework for parameter estimation in dynamic bioprocess models.
Read the full abstract ▾
All authors Daria Fokina, Marco Baldan, Constantin Romankiewicz, Wolfgang Laudensack, Roland Ulber, Michael Bortz
Machine Learning · Cross-listed from stat.ML
FedSPDnet: Geometry-Aware Federated Deep Learning with SPDnet
Thibault Pautrel, Florent Bouchard, Ammar Mian et al. (+1 more)
We introduce two
federated learning frameworks for the classical SPDnet model operating on symmetric positive definite (SPD)
matrices with Stiefel-constrained parameters. Unlike standard Euclidean averaging, which violates orthogonality, our approach preserves
geometric structure through two efficient aggregation strategies: ProjAvg, projecting arithmetic means onto the Stiefel
manifold, and RLAvg, approximating tangent-space averaging via retractions and liftings. Both methods are computationally efficient, independent of the optimizer, and enable scalable
federated learning for signal processing applications whose features are SPD
matrices. Simulations on
EEG motor imagery
benchmarks show that FedSPDnet outperforms
federated EEGnet in F1 score and robustness to federation and partial participation, while using fewer parameters per communication round.
Read the full abstract ▾
All authors Thibault Pautrel, Florent Bouchard, Ammar Mian, Guillaume Ginolhac
Machine Learning · Cross-listed from cs.CV
Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples
Oussama Bouanani, Jim Berend, Wojciech Samek et al. (+2 more)
Neuron labeling assigns textual descriptions to internal units of deep networks. Existing approaches typically rely on highly activating examples, often yielding broad or misleading labels by focusing on dominant but incidental visual factors. Prior work such as FALCON introduced contrastive examples – inputs that are semantically similar to activating examples but elicit low activations – to sharpen explanations, but it primarily addresses subspace-level interpretability rather than scalable
neuron-level labeling. We revisit contrastive explanations for
neuron-level labeling in two stages: (1) candidate label generation with
vision language models (VLMs) and (2) label assignment with CLIP-like encoders. First, we show that providing contrastive image sets to VLMs yields candidate labels that are more specific and more faithful. Second, we introduce Contrastive Semantic Projection (CSP), an extension of SemanticLens that incorporates contrastive examples directly into its CLIP-based scoring and selection pipeline. Across extensive experiments and a case study on melanoma detection, contrastive labeling improves both faithfulness and semantic granularity over state-of-the-art baselines. Our results demonstrate that contrastive examples are a simple yet powerful and currently underutilized component of
neuron labeling and analysis pipelines.
Read the full abstract ▾
All authors Oussama Bouanani, Jim Berend, Wojciech Samek, Sebastian Lapuschkin, Maximilian Dreyer
Machine Learning · Cross-listed from cs.CV
All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streams
Marco Pegoraro, Jonas Seng, Dustin Heller et al. (+2 more)
Disciplines such as business process management and process mining aid organizations by discovering insights about processes on the basis of recorded event data. However, an obstacle to process analysis is data multi-modality: for instance, data in
video form are not directly interpretable as events. Existing approaches rely on a dictionary of activity label as input, cannot provide frame-by-frame labeling explanations, or rely on superseded computer
vision techniques. In this work, we present SnapLog, an approach to extract event data from
videos by converting frames to feature vectors using image
embeddings and performing temporal
segmentation through frame-wise similarity
matrices. A generalized few-shot classification is then used to assign labels to the
video segments, yielding labeled, timestamped sub-sequences of frames that are interpretable as events. Conventional process mining techniques can be used to analyze the resulting data. We show that our approach produces logs that accurately reflect the process in the
videos.
Read the full abstract ▾
All authors Marco Pegoraro, Jonas Seng, Dustin Heller, Wil M.P. van der Aalst, Kristian Kersting
Machine Learning
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
Haiyun Qiu, Xingyu Wu and Kay Chen Tan
Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (
Manifold-Aware Dynamic Expert
Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in
manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert
evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace
alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly
pruning redundant experts, particularly within generic modules and early layers.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents
Xirui Li, Ming Li, Yunze Xiao et al. (+4 more)
Collective intelligence refers to the ability of a group to achieve outcomes beyond what any individual member can accomplish alone. As large
language model
agents scale to populations of millions, a key question arises: Does collective intelligence emerge spontaneously from scale? We present the first empirical evaluation of this question in a large-scale autonomous
agent society. Studying MoltBook, a platform hosting over two million
agents, we introduce Superminds Test, a hierarchical framework that probes society-level intelligence using controlled Probing
Agents across three tiers: joint
reasoning, information synthesis, and basic interaction. Our experiments reveal a stark absence of collective intelligence. The society fails to outperform individual frontier models on complex
reasoning tasks, rarely synthesizes distributed information, and often fails even trivial coordination tasks. Platform-wide analysis further shows that interactions remain shallow, with threads rarely extending beyond a single reply and most responses being generic or off-topic. These results suggest that collective intelligence does not emerge from scale alone. Instead, the dominant limitation of current
agent societies is extremely sparse and shallow interaction, which prevents
agents from exchanging information and building on each other's outputs.
Read the full abstract ▾
All authors Xirui Li, Ming Li, Yunze Xiao, Ryan Wong, Dianqi Li, Timothy Baldwin, Tianyi Zhou
Machine Learning
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
Abhinaba Basu
We introduce HubRouter, a pluggable module that replaces O(n^2)
attention layers with O(nM) hub-mediated routing, where M << n is a small number of learned hub
tokens. We demonstrate it in two from-scratch architectures: a Jamba-style hybrid and a 12-layer
Transformer; retrofit into pretrained models is a tested negative case. HubRouter implements an encode-decode-score-council pipeline: M learned hubs cross-attend to all
tokens,
tokens project against hubs for routing fingerprints, a score head selects top-k
tokens, and a sparse council attends only to the selected subset. We validate HubRouter in three settings. (1) Hub-Jamba yields a nominal 4.2% PPL improvement (200.2 vs 209.0, single seed; possibly within seed noise) and up to ~90x training throughput at sequence length 1024 in matched PyTorch-native baselines; an optimised baseline would narrow this to ~10-15x. (2) Graduated replacement of 25% of
Transformer attention layers gives the best perplexity in our matched-budget sweep (268.0 vs 282.4 pure
Transformer). (3) Hub-GPT provides strictly causal routing, achieving PPL 211.5 +/- 0.4 over 3 seeds (post council-causal fix); approximately 3 PPL worse than Jamba's 208.5 +/- 0.7, a measurable quality cost for avoiding O(n^2) computation. Post-fix, chunk size C has little effect; the pre-fix chunk-size benefit was an artifact of a bidirectional-council leak we found in
adversarial review. A multi-seed hub-count sweep (~105 runs across M=1-32) reveals M=8-14 as the reliably-converging sub-band (4-5/5 seeds); M=6 is rescued to 5/5 by orthogonal regularization, while M>=20 shows increasing seed sensitivity. Companion paper arXiv:2603.20997 (Basu, 2026) defines the routing diagnostic task. Code and scripts will be released.
Read the full abstract ▾
Machine Learning
From physical surfaces to human-centric heat stress: LST and UTCI heat mapping reveals nonlinear effects of urban morphology
Yuan Wang, Shengao Yi, Xiaojiang Li et al. (+4 more)
Heat exposure connects the built environment and public health, directly shaping the livability and sustainability of urban areas. Understanding the spatial heterogeneity of heat exposure and its drivers is vital for
climate-adaptive urban planning. However, most planning-oriented studies rely on land surface temperature (LST), and whether LST adequately represents human heat exposure and how it differs from physiologically relevant heat stress remains insufficiently examined. Here, using Landsat-retrieved 30-m LST and GPU-accelerated 1-m universal thermal
climate index (UTCI) in Singapore, this study establishes a comprehensive "Modeling-Comparing-Assessing" framework to systematically evaluate the spatial and mechanistic differences between these two metrics. We further investigate their pronounced non-stationary and threshold-based relationships with urban factors using a novel geographically weighted XGBoost (GW-XGBoost) and generalized additive model (GAM) workflow. Our results reveal substantial differences in the spatial patterns of LST and UTCI, along with marked spatial heterogeneity in how 2D and 3D urban factors impact these thermal metrics, as demonstrated by explainable GW-XGBoost models (test R2 = 0.855 for LST and 0.905 for UTCI). Crucially, spatially explicit SHAP shows that sky view factor plays a central role in explaining UTCI variability but exhibits a comparatively marginal independent contribution to LST, indicating that LST inadequately captures shading-driven and radiative processes governing actual human heat stress. Moreover, SHAP-GAM analysis indicates that higher albedo is associated with increased UTCI. These findings provide model-informed planning implications for integrating physiologically relevant thermal indices to support targeted heat risk management and human-centric urban planning.
Read the full abstract ▾
All authors Yuan Wang, Shengao Yi, Xiaojiang Li, Pengyuan Liu, Zhiwei Yang, Ronita Bardhan, Rudi Stouffs
Machine Learning
From Local to Cluster: A Unified Framework for Causal Discovery with Latent Variables
Zongyu Li
Latent variables pose a fundamental obstacle to both causal discovery and inference. Local approaches exploiting direct neighborhood relations provide little beyond immediate dependencies. Cluster-level methods, though capable of broader
reasoning, generally require cluster assignments or causal sufficiency in advance, conditions that are rarely satisfied in practice. Applying single-variable discovery at cluster resolution violates these conditions and thereby produces systematic bias. L2C (Local to Cluster Causal Abstraction) learns cluster structure from local causal patterns without manual variable-to-cluster mapping. The framework rests on a cluster reduction
theorem: each cluster can be represented by at most three micro-variables, with no loss of macro-level identifiability. In the presence of latent variables, local discovery on maximal ancestral graphs recovers direct causes, effects, and V-structures, followed by cluster-level inference through a corresponding calculus on the resulting graph. Macro causal effects are formulated as joint interventions on the micro-variables within each target cluster. The framework is sound, atomically complete, and polynomially bounded. L2C establishes a direct connection from local micro-level information to macro causal
reasoning in settings where clusters are unknown and latent variables are unavoidable.
Read the full abstract ▾
Machine Learning
Distance-Misaligned Training in Graph Transformers and Adaptive Graph-Aware Control
Qinhan Hou and Jing Tang
Graph
Transformers can mix information globally, but this flexibility also creates failure modes: some tasks require long-range communication while others are better served by local interaction. We study this through a synthetic node-classification
benchmark on contextual
stochastic block model graphs, where labels are generated by a controllable mixture of local and far-shell signals. We define distance-misaligned training as a mismatch between where label-relevant information lies and where the model allocates communication over graph distance. On this
benchmark, we find three points. First, the preferred graph-distance bias changes systematically with task locality. Second, an oracle adaptive controller, given offline access to the task-side distance target, nearly matches the best fixed bias across regimes and strongly improves over a neutral baseline on mixed and local tasks. Third, a task-agnostic zero-gap controller is weaker, indicating that adaptation alone is not enough and that the control target matters. These results suggest that distance-resolved diagnosis is useful for understanding Graph
Transformer failures and for designing graph-aware control.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models
Alberto Messina and Stefano Scotta
Even when decoding with temperature $T=0$, large
language models (LLMs) can produce divergent outputs for identical inputs. Recent work by Thinking Machines Lab highlights implementation-level sources of nondeterminism, including batch-size variation, kernel non-invariance, and floating-point non-associativity. In this short note we formalize this behavior by introducing the notion of background temperature $T_{\mathrm{bg}}$, the effective temperature induced by an implementation-dependent perturbation process observed even when nominal $T=0$. We provide clean definitions, show how $T_{\mathrm{bg}}$ relates to a
stochastic perturbation governed by the inference environment $I$, and propose an empirical protocol to estimate $T_{bg}$ via the equivalent temperature $T_n(I)$ of an ideal reference system. We conclude with a set of pilot experiments run on a representative pool from the major LLM providers that demonstrate the idea and outline implications for reproducibility, evaluation, and deployment.
Read the full abstract ▾
Machine Learning
Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair
Yuelin Hu, Zhenbo Yu, Zhengxue Cheng et al. (+2 more)
Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5–12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5–4.8 units. The failure is largely invisible on clean
benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.
Read the full abstract ▾
All authors Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu, Li Song
Machine Learning
Robust Fuzzy local k-plane clustering with mixture distance of hinge loss and L1 norm
Junjun Huang, Xiliang Lu, Xuelin Xie et al. (+1 more)
K-plane clustering (KPC), hyperplane clustering, and mixture regression all essentially fall within the same class of problems. This problem can be conceptualized as clustering in relatively high-dimensional K subspaces or K linear
manifolds. Traditional KPC or fuzzy KPC models demonstrate a pronounced susceptibility to outliers, as they presuppose that the projection distance between data points and the plane normal vector adheres to the L2 distance. Meanwhile, the assumption of infinitely extending clusters adversely affects clustering performance. To solve these problems, this paper proposed a new robust fuzzy local k-plane clustering (RFLkPC) method that combines the mixture distance of hinge loss and L1 norm. The RFLkPC model assumes that each plane cluster is bounded to a finite area, which can flexibly and robustly handle plane clustering tasks with outliers or not. The corresponding model and optimization
algorithms of RFLkPC were provided. Compared to other related models on this topic, a large number of experiments verify the efficiency of RFLkPC on simulated data and real data. The source code for the proposed RFLkPC method is publicly available at https://github.com/xuelin-xie/RFLkPC.
Read the full abstract ▾
All authors Junjun Huang, Xiliang Lu, Xuelin Xie, Jerry Zhijian Yang
Machine Learning · Cross-listed from stat.ML
Conformalized Super Learner
Zhanli Wu, Fabrizio Leisen, Miguel-Angel Luque-Fernandez et al. (+1 more)
The Super Learner (SL) is a widely used ensemble method that combines point predictions from a library of learners based on their predictive performance. Interval predictions are of considerable practical interest because they allow uncertainty in predictions produced by an individual learner or an ensemble to be quantified. Several methods have been proposed for constructing interval predictions based on the SL, however, these approaches are typically justified using asymptotic arguments or rely on computationally intensive procedures such as the bootstrap. Conformal prediction (CP) is a machine learning framework for constructing prediction intervals with finite-sample and asymptotic coverage guarantees under mild conditions. We propose coupling CP with the SL through a natural construction that mirrors the original SL framework, using individual learner weights and combining learner-specific conformal prediction sets via a weighted majority vote. We characterize the properties of the resulting SL-based prediction intervals for continuous outcomes. We cover settings under exchangeability, and data-generating mechanisms exhibiting heteroscedasticity, sparsity, and other forms of distributional heterogeneity. A comprehensive simulation study shows that the conformalized SL achieves valid finite-sample coverage with competitive performance relative to the true data-generating mechanism. We present an application of this methodology to predict creatinine levels using socio-demographic, biometric, and laboratory measurements. This example demonstrates the benefits of an ensemble with carefully selected learners designed to capture key aspects of complex regression functions, including non-linear effects, interactions, sparsity, heteroscedasticity, and robustness to outliers.
Read the full abstract ▾
All authors Zhanli Wu, Fabrizio Leisen, Miguel-Angel Luque-Fernandez, F. Javier Rubio
Machine Learning · Cross-listed from stat.ML
Pack only the essentials: Adaptive dictionary learning for kernel ridge regression
Daniele Calandriello, Alessandro Lazaric and Michal Valko
One of the major limits of kernel ridge regression (KRR) is that storing and manipulating the kernel
matrix K_n for n samples requires O(n^2) space, which rapidly becomes unfeasible for large n. Nystrom approximations reduce the space complexity to O(nm) by sampling m columns from K_n. Uniform sampling preserves KRR accuracy (up to epsilon) only when m is proportional to the maximum degree of freedom of K_n, which may require O(n) columns for
datasets with high coherence. Sampling columns according to their ridge leverage scores (RLS) gives accurate Nystrom approximations with m proportional to the effective dimension, but computing exact RLS also requires O(n^2) space. (Calandriello et al. 2016) propose INK-Estimate, an
algorithm that processes the
dataset incrementally and updates RLS, effective dimension, and Nystrom approximations on-the-fly. Its space complexity scales with the effective dimension but introduces a dependency on the largest
eigenvalue of K_n, which in the worst case is O(n). In this paper we introduce SQUEAK, a new
algorithm that builds on INK-Estimate but uses unnormalized RLS. As a consequence, the
algorithm is simpler, does not need to estimate the effective dimension for normalization, and achieves a space complexity that is only a constant factor worse than exact RLS sampling.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Pliable rejection sampling
Akram Erraqabi, Michal Valko, Alexandra Carpentier et al. (+1 more)
Rejection sampling is a technique for sampling from difficult distributions. However, its use is limited due to a high rejection rate. Common adaptive rejection sampling methods either work only for very specific distributions or without performance guarantees. In this paper, we present pliable rejection sampling (PRS), a new approach to rejection sampling, where we learn the sampling proposal using a kernel
estimator. Since our method builds on rejection sampling, the samples obtained are with high probability i.i.d. and distributed according to f. Moreover, PRS comes with a guarantee on the number of accepted samples.
Read the full abstract ▾
All authors Akram Erraqabi, Michal Valko, Alexandra Carpentier, Odalric-Ambrym Maillard
Machine Learning
Revisiting Neural Activation Coverage for Uncertainty Estimation
Benedikt Franke, Nils Förster, Frank Köster et al. (+3 more)
Neural activation coverage (NAC) is a recently-proposed technique for out-of-distribution detection and generalization. We build upon this promising foundation and extend the method to work as an uncertainty estimation technique for already-trained artificial neural networks in the domain of regression. Our experiments confirm NAC uncertainty scores to be more meaningful than other techniques, e.g. Monte-Carlo Dropout.
Read the full abstract ▾
All authors Benedikt Franke, Nils Förster, Frank Köster, Asja Fischer, Markus Lange, Arne Raulf
Machine Learning
SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions
Kang Liu, Jianchen Hu and Wei Peng
Classical ReLU-based Input Convex Neural Networks (ICNNs) are equivalent to the optimal value functions of Linear Programming (LP). This intrinsic structural equivalence restricts their representational capacity to piecewise-linear polyhedral functions. To overcome this representational bottleneck, we propose the SOC-ICNN, an architecture that generalizes the underlying optimization class from LP to Second-Order Cone Programming (SOCP). By explicitly injecting positive semi-definite
curvature and Euclidean norm-based conic primitives, our formulation introduces native smooth
curvature into the representation while preserving a rigorous optimization-theoretic interpretation. We formally prove that SOC-ICNNs strictly expand the representational space of ReLU-ICNNs without increasing the asymptotic order of forward-pass complexity. Extensive experiments demonstrate that SOC-ICNN substantially improves function approximation, while delivering competitive downstream decision quality. The code is available at https://anonymous.4open.science/r/SOC-ICNN-4B18/.
Read the full abstract ▾
Machine Learning
A Nationwide Japanese Medical Claims Foundation Model: Balancing Model Scaling and Task-Specific Computational Efficiency
Nanae Aratake, Taisei Tosaki, Yuji Okamoto et al. (+5 more)
Clinical risk prediction using longitudinal medical data supports individualized care. Self-supervised foundation models have emerged as a promising approach for leveraging large-scale unlabeled healthcare records. In natural
language processing, scaling laws suggest that larger models achieve predictably lower pretraining losses, supporting the foundation model paradigm. However, for structured medical data, characterized by a limited vocabulary and sparse observations, whether increasing model size consistently improves downstream predictions is unclear, as most studies evaluate only a single model scale. In this study, we evaluated the relationship between model scale and downstream task performance for structured medical foundation models. Using a random sample (2.3 million patients, 32 hospitals) from a nationwide 519-hospital Japanese claims database, we pretrained encoder-only
Transformers at five scales (2.2M-101M parameters) for disease incidence and medication prediction. Downstream performance saturated at task-dependent thresholds: disease prediction benefited from larger models (32M-101M), whereas medication prediction saturated at 11M, reducing pretraining time by 178 h. Across all tasks, the best-performing model consistently outperformed a Light Gradient Boosting Machine baseline in the area under the precision-recall curve. These findings indicate that, unlike the monotonically decreasing pretraining loss, the optimal model size varied depending on task characteristics. This task-dependent saturation provides practical guidance for balancing predictive performance and computational cost in structured medical foundation models.
Read the full abstract ▾
All authors Nanae Aratake, Taisei Tosaki, Yuji Okamoto, Eiichiro Uchino, Masaki Nakamura, Nobutomo Matsui, Akiko Hatakama, Yasushi Okuno
Machine Learning
TabSCM: A practical Framework for Generating Realistic Tabular Data
Sven Jacob, Bardh Prenkaj, Weijia Shao et al. (+1 more)
Most tabular-data generators match marginal statistics yet ignore causal structure, leading downstream models to learn spurious or unfair patterns. We present TabSCM, a mixed-type generator that preserves those causal dependencies. Starting from a Completed Partially Directed Acyclic Graph (CPDAG) found by any causal structure discovery
algorithm, TabSCM (i) orients edges to a DAG, (ii) fits root-node marginals with KDE or categorical frequencies, and (iii) learns topologically ordered structural assignments. Such assignments are achieved using conditional
diffusion models for continuous variables as child nodes and gradient-boosted trees for categorical ones. Ancestral sampling yields semantically valid records and enables exact counterfactual queries. On seven public
datasets, encompassing healthcare, finance, housing, environment, TabSCM matches or surpasses state-of-the-art GAN,
diffusion, and LLM baselines in statistical fidelity, downstream utility, and
privacy risk, while also cutting rule-violation rates and providing causally meaningful and robust conditional interventions. Because generation is decomposed into explicit equations, it runs up to 583$\times$ faster than
diffusion-only models and exposes interpretable knobs for
fairness auditing and policy simulation, making TabSCM a practical choice for realism, explainability, and causal soundness.
Read the full abstract ▾
All authors Sven Jacob, Bardh Prenkaj, Weijia Shao, Gjergji Kasneci
Machine Learning
FETS Benchmark: Foundation Models Enable Scalable and Generalizable Energy Time Series Forecasting
Marco Obermeier, Marco Pruckner, Florian Haselbeck et al. (+1 more)
Driven by the transition towards a
climate-neutral energy system, accurate energy time series
forecasting is critical for planning and operations. Yet, it remains a
dataset-specific task, requiring comprehensive training data, limiting scalability, and resulting in high model development and maintenance effort. Recently, foundation models aiming to learn generalizable patterns via extensive pretraining have shown strong performance in multiple prediction tasks. Despite their success and strong potential in energy
forecasting, a systematic, use-case-differentiated evaluation is still missing. We address this gap by presenting the Foundation Models in Energy Time Series
Forecasting (FETS)
benchmark. We (1) provide a structured overview of energy
forecasting use cases along three main dimensions, i.e., stakeholders, attributes, and data categories, (2) curate 54
datasets across 9 data categories, guided by typical stakeholder interests, and (3)
benchmark foundation models against task-specific machine learning across different
forecasting settings. In our
benchmark study,
covariate-informed zero-shot foundation models perform best in aggregate, with Chronos-2 attaining the lowest overall median NRMSE (0.472), closely followed by TiRex-2 (0.474). Both perform better than XGBoost (0.611) and random forest (0.696), although they were trained task-specifically on the full historic target data. Further analysis reveals a strong correlation between predictive performance and spectral
entropy. Performance saturates beyond a certain context length and improves with aggregation level, e.g., for national load, district heating, and power grid data. Overall, with the lowest median error, limited data requirements, and low inference and hardware demands, foundation models reduce development and maintenance effort, emerging as scalable and generalizable energy
forecasting solutions.
Read the full abstract ▾
All authors Marco Obermeier, Marco Pruckner, Florian Haselbeck, Andreas Zeiselmair
Machine Learning
A Brain-Inspired Deep Separation Network for Single Channel Raman Spectra Unmixing
Gaoruishu Long, Jinchao Liu, Bo Liu et al. (+2 more)
Raman spectra obtained in real world applications are often a noisy combination of several spectra of various substances in a tested sample. Unmixing such spectra into individual components corresponding to each of the substances is of great value and has been a longstanding challenge in Raman spectroscopy. Existing unmixing methods are predominantly designed to invert an overdetermined mixed model and therefore require multiple mixed spectra as input. However, open domain and/or non-cooperative detection applications in Raman spectroscopy such as controlled substance detection, call for single-channel solutions which can identify individual components from thousands of candidates by analyzing only a single noisy mixed spectrum. To our knowledge, sparse regression is the only existing solution which can cope with this scenario, yet it has very low tolerance to noises and can hardly be applicable in practice. To address these limitations, we introduce a novel neural approach for single-channel Raman spectrum unmixing inspired by
speech separation. It aims at solving underdetermined systems and can decompose a noisy mixed spectrum from a library of thousands of components (substances). The core of our method is a deep separation neural network (RSSNet) which takes a mixed spectrum as input and outputs spectra of pure components. We created two synthetic
datasets of single-channel Raman spectra unmixing and demonstrated feasibility and superiority of RSSNet on these
datasets (outperform competing methods by >4dB). Furthermore, we verified that RSSNet, trained solely on synthetic data, can successfully unmix real-world mixed spectra of mixtures of mineral powders, exhibiting strong generalization. Our approach represents a new paradigm for Raman unmixing and enables new possibilities for fast detection of Raman mixtures.
Read the full abstract ▾
All authors Gaoruishu Long, Jinchao Liu, Bo Liu, Jie Liu, Xiaolin Hu
Machine Learning · Cross-listed from cs.AR
HGQ-LUT: Fast LUT-Aware Training and Efficient Architectures for DNN Inference
Chang Sun, Zhiqiang Que, Bakhtiar Zadeh et al. (+4 more)
Lookup-table (LUT) based neural networks can deliver ultra-low latency and excellent hardware efficiency on FPGAs by mapping arithmetic operations directly onto the logic primitives. However, state-of-the-art LUT-aware training (LAT) approaches remain difficult to use in practice: they are often orders of magnitude slower to train than conventional networks, require non-trivial manual tuning for hardware efficiency, and lack an end-to-end workflow. This work presents HGQ-LUT, integrated in https://github.com/calad0i/HGQ2, a new LAT approach that achieves state-of-the-art hardware efficiency while accelerating training by over 100 times on modern GPUs. HGQ-LUT introduces LUT-Dense and LUT-Conv layers that are implemented with regular, accelerator-efficient tensor operations during training, which are then compiled into logic LUTs for hardware. By combining these layers with fine-grained, element-wise heterogeneous quantization (including zero-bit
pruning) and a LUT-aware resource surrogate, HGQ-LUT enables the automatic exploration of accuracy-resource trade-offs without manual bit-width tuning. We further integrate HGQ-LUT into open-source toolchains, enabling unified design, compilation, and bit-exact verification of hybrid architectures that mix LUT-based with conventional arithmetic blocks. These features make LAT-based DNNs practical for real-world deployment, such as at the CERN Large
Hadron Collider's experiments.
Read the full abstract ▾
All authors Chang Sun, Zhiqiang Que, Bakhtiar Zadeh, Qibin Liu, Kevin H. Alvarez, Wayne Luk, Maria Spiropulu
Machine Learning
How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
Dharshan Kumaran, Viorica Patraucean, Simon Osindero et al. (+2 more)
Large
language models can detect their own errors and sometimes correct them without external feedback, but the underlying mechanisms remain unknown. We investigate this through the lens of second-order models of confidence from decision neuroscience. In a first-order system, confidence derives from the generation signal itself and is therefore maximal for the chosen response, precluding error detection. Second-order models posit a partially independent evaluative signal that can disagree with the committed response, providing the basis for error detection. Kumaran et al. (2026) showed that LLMs cache a confidence representation at a
token immediately following the answer (i.e. post-answer newline: PANL) – that causally drives verbal confidence and dissociates from log-probabilities. Here we test whether this PANL signal extends beyond confidence to support error detection and self-correction. Here we test whether this signal supports error detection and self-correction, deriving predictions from the second-order framework. Using a verify-then-correct paradigm, we show that: (i) verbal confidence predicts error detection far beyond
token log-probabilities, ruling out a first-order account; (ii) PANL activations predict error detection beyond verbal confidence itself; and (iii) PANL predicts which errors the model can correct – where all behavioural signals fail. Causal interventions confirm that PANL signals rescue error detection behavior when answer information is corrupted. All findings replicate across models (Gemma 3 27B and Qwen 2.5 7B) and tasks (TriviaQA and MNLI). These results reveal that LLMs naturally implement a second-order confidence architecture whose internal evaluative signal encodes not only whether an answer is likely wrong but whether the model has the knowledge to fix it.
Read the full abstract ▾
All authors Dharshan Kumaran, Viorica Patraucean, Simon Osindero, Petar Veličković, Nathaniel Daw