Machine Learning
Forecasting Green Skill Demand in the Automotive Industry: Evidence from Online Job Postings
Sabur Butt, Joshua N. Arrazola E., Hector G. Ceballos et al. (+1 more)
The global transition toward sustainable economies is reshaping labor
markets, yet systematic methods for identifying and
forecasting green skills remain limited. This study presents a computational framework to measure and predict green skill demand using online job postings from Mexico's automotive industry, which contributes about 4% of national GDP. We compile a
dataset of job advertisements from Indeed Mexico, OCC Mundial, and LinkedIn (July 2024 to July 2025), yielding 204,373 skill records. A two-stage pipeline combining
multilingual embeddings and ESCO validation identifies 274 unique green skills across 8,576 occurrences (4.22% of all skills). We
benchmark 15 time series
forecasting models using a rolling origin evaluation.
Transformer-based models, especially FEDformer, Reformer, and Informer, achieve the best performance, with MAE around 2.5e-5 and relative RMSE below 15. We further propose a framework to classify skills by absolute and relative growth, identifying stable, emerging, and high-impact competencies. Results show current demand is concentrated in operational sustainability practices, while the fastest-growing skills relate to renewable energy, recycling, and hydrogen technologies. This pipeline supports data-driven workforce planning in the green transition.
Read the full abstract ▾
All authors Sabur Butt, Joshua N. Arrazola E., Hector G. Ceballos, Patricia Caratozzolo
Machine Learning
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
Mohammad Reza Deylam Salehi and Ali Khalesi
Resource-efficient machine learning increasingly uses sparse Mixture-of-Experts (MoE) architectures, where the gate acts as both a learning component and a routing interface controlling computation, communication, and accuracy. Motivated by finite-rate interpretations of MoE gating, we treat the gate as a
stochastic channel and use $I(X;T)$ to quantify the routing information available to the selected expert. To make the associated information quantities tractable beyond synthetic examples, we develop a finite-bank MNIST construction using pretrained CNN experts and a discrete, data-dependent selection rule. Since the selected model belongs to a finite candidate set, the algorithmic mutual information $I(S;W)$ admits a closed-form discrete-
entropy estimator from the empirical posterior $q(W|S)$. Sweeping a data-dependence parameter $\alpha$, we observe that $\widehat I(S;W)$ monotonically tracks the generalization gap, while the Xu-Raginsky bound exhibits the expected looseness. We also compare with a uniform union-bound baseline and introduce an empirical
estimator of $I(X;T)$ together with a Blahut-Arimoto procedure for tracing an accuracy-rate curve over the expert bank. The proposed framework provides a practical tool for analyzing resource-aware MoE inference systems and for interpreting $I(X;T)$ and $D(R_g)$ as design proxies for efficient expert routing.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Forecasting Oncology Demand Trends with Boosting-Based Bayesian Conjugate Models
Ademir Batista dos Santos Neto, Tiago Alessandro Espinola Ferreira and Paulo Renato Alves Firmino
Accurate trend
forecasting in healthcare time series is essential for planning and resource allocation. This paper proposes a
Bayesian framework for predicting oncology demand trends, modeling weekly appointments as a Poisson process with a Gamma prior to the demand rate. To enhance adaptability and capture persistent directional patterns, we incorporate a residual-based boosting mechanism grounded in a Gamma-Log-Normal conjugate structure. This boosting approach allows the model to track both short- and long-term trend shifts while maintaining the analytical tractability of conjugate
Bayesian updating. The methodology was evaluated on real oncology service data from Cariri, Ceara, Brazil, and compared against established baselines, including linear regression, ARIMA, naive
forecasting, LSTM neural networks, and XGBoost. Results showed that the proposed model outperforms competing methods in trend detection accuracy, with gains in terms of percentage of correct direction of 38.25% in relation to the second best approach in some cases.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
Differential Privacy in the Extensive-Form Bandit Problem
Stephen Pasteris, Rahul Savani and Theodore Turocy
We consider the extensive-form bandit problem, where on each trial the learner (a user coordinated by a server) plays an extensive-form game against an oblivious adversary, observing the information sets it finds itself in as well as the resulting payoff/loss. We give an
algorithm for this problem that satisfies $\epsilon$-local differential
privacy and attains a regret of $\tilde{O}(\sqrt{A\ln(S)T}/\epsilon)$, where $A$ is the total number of actions that the learner can possibly take, $S$ is the number of the learner's possible reduced strategies, and $T$ is the number of trials. On each trial, the time complexity of our
algorithm is, up to a factor logarithmic in the maximum number of actions at an infoset, equal to the time required for the server to transmit the reduced strategy to the user. We note that local differential
privacy is the strongest version of differential
privacy and, to the best of our knowledge, this is the first work to study differential
privacy of any form in the extensive-form bandit problem.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning
Yuelin Hu, Zhenbo Yu, Zhengxue Cheng et al. (+2 more)
We formalize Rollout Informativeness under a Fixed Budget (RIFB) as the expected non-vanishing policy-gradient mass that a tool-use rollout set injects into Group Relative Policy Optimization (GRPO). We prove that any budget-agnostic independent sampler suffers a collapse rate bounded away from zero for hard
prompts regardless of the budget. Motivated by this, we recast intermediate state selection as a monotone submodular maximization problem, where a greedy one-step selector enjoys a 1 minus 1/e approximation guarantee. Our Uncertainty-aware Upper Confidence Bound (UUCB) terms arise as closed-form marginal gains of this objective. This turns the
token-level
entropy bonus from an empirical trick into an analytic consequence of the formulation. We present InfoTree, a training-time tree-search framework coupling UUCB with a learned Adaptive Budget Allocator (ABA) and an asynchronous Speculative Expansion scheme. ABA rescues
prompts whose initial tree is wasted on uniform outcomes, lifting the mixed-outcome ratio from 58.1 percent to 76.3 percent with less than 5 percent budget overhead. Speculative Expansion reduces wall-clock overhead from 14.3 percent to 4.8 percent by tolerating bounded staleness in UUCB scores. Across nine
benchmarks spanning math
reasoning (AIME 2024 and 2025, MATH-500, OlympiadBench, USAMO), web-search
agents (
GAIA, HLE-100, BrowseComp-lite), and tool-rich coding and OS
agents (APPS-verified, AgentBench-OS), InfoTree outperforms flat GRPO, DeepSearch, Tree-GRPO, AT2PO, CW-GRPO, and RC-GRPO. Head-to-head compositions with Tree-GRPO prefix sharing and CW-GRPO contribution weights deliver further gains, confirming that our selector operates orthogonally to rollout reuse and trajectory re-weighting. A 5 by 5 by 5 robustness grid reveals that over three quarters of the hyperparameter space lies on a performance plateau, confirming UUCB robustness.
Read the full abstract ▾
All authors Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu, Li Song
Machine Learning · Cross-listed from cs.CV
Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu et al. (+3 more)
We study outlier
tokens in
Diffusion Transformers (DiTs) for image generation. Prior work has shown that
Vision Transformers (ViTs) can produce a small number of high-norm
tokens that attract disproportionate
attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier
tokens, especially in intermediate layers. Moreover, simply masking high-norm
tokens does not improve performance, indicating that the problem is not only caused by a few extreme values, but is more closely related to corrupted local patch semantics. To address this issue, we introduce Dual-Stage Registers (DSR), a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and
diffusion registers for the denoiser. Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality. Our results highlight outlier-
token control as an important ingredient in building stronger DiTs.
Read the full abstract ▾
All authors Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei
Machine Learning · Cross-listed from stat.ML
Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learning
Nicholas Barnfield, Juno Kim, Eshaan Nichani et al. (+2 more)
How many key-value associations can a $d\times d$ linear
memory store? The answer depends not only on the $d^2$ degrees of freedom in the
memory matrix, but also on the
retrieval criterion. Under isotropic Gaussian
embeddings, we prove a sharp threshold for top-1
retrieval, where every signal must beat its largest distractor: the critical value of $d^2/(n\log n)$ is $2$. Above the threshold, we explicitly construct a linear
memory that retrieves all $n$ associations with high probability; below it, no data-dependent linear
memory can do so. The $\log n$ factor is therefore the unavoidable extreme-value cost of winner-take-all decoding. Without the logarithmic factor—that is, when $n/d^2\to\alpha\in(0,\infty)$—simultaneous top-1
retrieval is impossible. The matched target can nevertheless remain near the top of the ranking. We capture this weaker
retrieval goal with the Tail-Average Margin (TAM), which, for list size $k$, compares each signal with the average of its $k$ strongest competitors; a positive TAM margin certifies that the target belongs to the top-$k$ candidate list. When $k/n\to r\in(0,1)$, we learn the
memory by empirical risk minimization with a smoothed TAM objective and derive an exact high-dimensional characterization through a two-parameter scalar variational problem. The result gives limiting laws for signal and competitor scores, margins, and percentile ranks. Sending the ridge parameter to zero after the high-dimensional limit yields a closed-form critical load $\alpha_c(r)$ separating vanishing from positive average loss. The analysis in this work also develops a coupled leave-one-out method for
matrix-valued empirical risk problems in which each sample enters many dependent comparisons, a tool that may be useful beyond associative
memory.
Read the full abstract ▾
All authors Nicholas Barnfield, Juno Kim, Eshaan Nichani, Jason D. Lee, Yue M. Lu
Machine Learning
Estimating the expected output of wide random MLPs more efficiently than sampling
Wilson Wu, Victor Lecomte, Michael Winer et al. (+3 more)
By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, we produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions. We show both theoretically and empirically that for sufficiently wide networks, our
estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling. We find moreover that our methods perform particularly well at estimating the probabilities of rare events, and additionally demonstrate how they can be used for model training. Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.
Read the full abstract ▾
All authors Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, Paul Christiano
Machine Learning
Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer
Alexander Hsu, Zhaiming Shen, Wenjing Liao et al. (+1 more)
Pre-trained
transformers are able to learn from examples provided as part of the
prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in
attention, we explicitly construct
transformer networks to realize nonlinear features, such as
polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.
Read the full abstract ▾
All authors Alexander Hsu, Zhaiming Shen, Wenjing Liao, Rongjie Lai
Machine Learning · Cross-listed from cs.CL
PSK at SemEval-2026 Task 9: Multilingual Polarization Detection Using Ensemble Gemma Models with Synthetic Data Augmentation
Srikar Kashyap Pulipaka
We present our system for SemEval-2026 Task 9:
Multilingual Polarization Detection, a binary classification task spanning 22
languages. Our approach fine-tunes separate Gemma~3 models (12B and 27B parameters) per
language using Low-Rank Adaptation (LoRA), augmented with synthetic data generated by a large
language model (LLM). We employ three synthetic data strategies (direct generation, paraphrasing, and contrastive pair creation) using GPT-4o-mini, with a multi-stage quality filtering pipeline including
embedding-based deduplication. We find that per-
language threshold tuning on the development set yields 2 to 4% F1 improvements without retraining. We also use weighted ensembles of 12B and 27B model predictions with per-
language strategy selection. Our final system achieves a mean macro-F1 of 0.811 across all 22
languages, ranking 2nd overall of the participating teams, with 1st place finishes in 3
languages and top-3 in 8
languages. We also find that alternative architectures (XLM-RoBERTa, Qwen3) that showed strong development set performance suffered 30 to 50% F1 drops on the test set, highlighting the importance of generalization.
Read the full abstract ▾
Machine Learning
Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
Alper Y{\i}ld{\i}r{\i}m
Transformer architectures have been widely adopted for time series
forecasting, yet whether the representational mechanisms that make them powerful in NLP actually engage on time series data remains unexplored. The persistent competitiveness of simple linear models such as DLinear has fueled ongoing debate, but no mechanistic explanation for this phenomenon has been offered. We address this gap by applying sparse autoencoders (SAEs), a tool from mechanistic interpretability, to probe the internal representations of PatchTST. We first establish that a single-layer, narrow-dimensional
transformer matches the
forecasting performance of deeper configurations across commonly used
benchmarks. We then train SAEs on the post-GELU intermediate FFN activations with dictionary sizes ranging from 0.5x to 4.0x the native dimensionality. Expanding the dictionary yields negligible downstream performance change (average 0.214%), with large portions of overcomplete dictionaries remaining inactive. Targeted causal interventions on dominant latent features produce minimal forecast perturbation. Across all evaluated settings, we observe no empirical evidence that the analyzed FFN representations rely on strong superposition. Instead, the representations remain sparse, stable under aggressive dictionary expansion, and largely insensitive to latent interventions. These results demonstrate that superposition is not necessary for competitive performance on standard
forecasting benchmarks, suggesting they may not demand the rich compositional representations that drive
transformer success in
language modeling, and helping explain the persistent competitiveness of simple linear models
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
What Matters in Practical Learned Image Compression
Kedar Tatwawadi, Parisa Rahimzadeh, Zhanghao Sun et al. (+5 more)
One of the major differentiators unlocked by learned codecs relative to their hard-coded traditional counterparts is their ability to be optimized directly to appeal to the human visual system. Despite this potential, a perceptual yet practical image codec is yet to be proposed. In this work, we aim to close this gap. We conduct a comprehensive study of the key modeling choices that govern the design of a practical learned image codec, jointly optimized for perceptual quality and runtime – including within the ablations several novel techniques. We then perform performance-aware neural architecture search over millions of backbone configurations to identify models that achieve the target on-device runtime while maximizing compression performance as captured by perceptual metrics. We combine the various optimizations to construct a new codec that achieves a significantly improved tradeoff between speed and perceptual quality. Based on rigorous subjective user studies, it provides 2.3-3x bitrate savings against AV1, AV2, VVC, ECM and JPEG-AI, and 20-40% bitrate savings against the best learned codec alternatives. At the same time, on an iPhone 17 Pro Max, it encodes 12MP images as fast as 230ms, and decodes them in 150ms – faster than most top ML-based codecs run on a V100 GPU.
Read the full abstract ▾
All authors Kedar Tatwawadi, Parisa Rahimzadeh, Zhanghao Sun, Zhiqi Chen, Ziyun Yang, Sanjay Nair, Divija Hasteer, Oren Rippel
Machine Learning
Human-AI Co-Mentorship in Project-Based Learning: A Case Study in Financial Forecasting
Freyaa Chawla, Ahan Chawla, Rishi Singh et al. (+2 more)
This paper reflects on a AI research project carried out by a team of high-school and early-undergraduate students under the mentorship of graduate researchers and ably assisted by AI tools. We share our experience in not only on the learning experience for the high school students, but also on how AI tools accelerated the process that enabled the high school students to focus on higher order problem formulation and solution. Although the participants entered the project with limited background in both AI and finance, they showed strong enthusiasm for technical
market analysis and ETF price prediction. Traditional learning settings would first teach the necessary methods in a classroom setting and only later let students apply them. In contrast, our project emphasized workflow design: students identified the sequence of steps needed to address the problem and then used AI-driven tools to execute each step. We note that the high school students developed the necessary code through iterating with the AI tools, and we used our daily stand-ups to debug and answer conceptual questions. Each of the student was able to dig deeper into their area of interest whether computer science or finance, while collaboratively making a significant advance over the summer of 2025. This project was an important pedagogical exercise on how AI tools can be used for mentoring high school students, allowing them to focus on their specific interests and using the daily stand-ups to focus on problem definition and conceptual understanding. Despite their limited technical qualifications, the students were able to leverage AI tools to build meaningful models with real-world application.
Read the full abstract ▾
All authors Freyaa Chawla, Ahan Chawla, Rishi Singh, Joe Germino, Grigorii Khvatskii
Machine Learning
Low-Cost Black-Box Detection of LLM Hallucinations via Dynamical System Prediction
Dan Wilson and Mohamed Akrout
Large
Language Models (LLMs) frequently generate plausible but non-factual content, a phenomenon known as
hallucination. While existing detection methods typically rely on computationally expensive sampling-based consistency checks or external knowledge
retrieval, we propose a new method that treats the LLM as a black-box dynamical system. By projecting LLM responses into a high-dimensional
manifold via an
embedding model, we characterize the resulting vector sequences as observable realizations of the model's latent state-space dynamics. Leveraging Koopman operator theory, we fit the transition operators for both factual and hallucinated regimes and define a differential residual score based on their respective prediction errors. This approach enables low-cost
hallucination detection in a single-sample pass, avoiding the need for secondary sampling or external grounding. Extensive testing across three data
benchmarks demonstrates that our method achieves state-of-the-art performance with reduced resource overhead.
Read the full abstract ▾
Machine Learning
Transformed Latent Variable Multi-Output Gaussian Processes
Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka et al. (+2 more)
Multi-Output Gaussian Processes (MOGPs) provide a principled probabilistic framework for modelling correlated outputs but face scalability bottlenecks when applied to
datasets with high-dimensional output spaces. To maintain tractability, existing methods typically resort to restrictive assumptions, such as employing low-rank or sum-of-separable kernels, which can limit expressiveness. We propose the Transformed Latent Variable MOGP (T-LVMOGP), a novel framework that scales MOGPs to a massive number of outputs while preserving the capacity to capture meaningful inter-output dependencies. T-LVMOGP constructs a flexible multi-output deep kernel by mapping inputs and output-specific latent variables into an
embedding space using a Lipschitz-regularised neural network. Combined with
stochastic variational inference, our model effectively scales to high-dimensional output settings. Across diverse
benchmarks, including
climate modelling with over 10,000 outputs and zero-inflated spatial transcriptomics data, T-LVMOGP outperforms baselines in both predictive accuracy and computational efficiency.
Read the full abstract ▾
All authors Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka, Magnus Rattray, Mauricio A Álvarez
Machine Learning
Joint Treatment Effect Estimation from Incomplete Healthcare Data: Temporal Causal Normalizing Flows with LLM-driven Evolutionary MNAR Imputation
Olivia Jullian Parra, Sara Zoccheddu, David Catalan Cerezo et al. (+7 more)
Target trial emulation (TTE) enables causal questions to be studied with observational data when randomized controlled trials (RCTs) are infeasible. Yet treatment-effect methods often address causal estimation, missingness, and temporal structure separately, limiting their robustness in electronic health records (EHRs), where time-varying confounding and missing-not-at-random (MNAR) biomarkers can reach 50%–80%. We propose a two-stage pipeline for treatment effect estimation from incomplete longitudinal EHRs. First, CausalFlow-T, a directed acyclic graph (DAG)-constrained normalizing flow with long short-term
memory (LSTM)-encoded patient history, performs exact invertible counterfactual inference, avoiding approximation errors from variational inference and separating confounding through explicit causal structure. Ablations on four synthetic and one semi-synthetic
benchmark with known counterfactuals show that DAG constraints and exact inference address distinct failure modes: neither compensates for the other. Second, because CausalFlow-T requires completed inputs, we introduce an LLM-driven evolutionary imputer that proposes executable imputation operators rather than individual entries, and evaluate it with three large
language model (LLM) backends, including two open-source models. Across 30%–80% MNAR missingness, this imputer achieves the best pooled rank over biomarker and causal metrics, leading in point-wise accuracy and temporal extrapolation while preserving average treatment effect (ATE) recovery as statistical baselines degrade. On Swiss primary-care EHRs from adults with type 2 diabetes initiating a GLP-1 receptor agonist or SGLT-2 inhibitor, the pipeline estimates a per-protocol weight-loss difference of -0.98 kg [95% CI -1.01, -0.96] favoring GLP-1 receptor agonists, consistent with randomized evidence and obtained from realistically incomplete real-world EHRs.
Read the full abstract ▾
All authors Olivia Jullian Parra, Sara Zoccheddu, David Catalan Cerezo, Tom Forzy, Franziska Ulrich, William Sutcliffe, Jakob Martin Burgstaller, Oliver Senn, Patrick Owen, Nicola Serra
Machine Learning
Conditional outlier detection for clinical alerting
Milos Hauskrecht, Michal Valko, Shyam Visweswaran et al. (+3 more)
We develop and evaluate a data-driven approach for detecting unusual (anomalous) patient-management actions using past patient cases stored in an electronic health record (EHR) system. Our hypothesis is that patient-management actions that are unusual with respect to past patients may be due to a potential error and that it is worthwhile to raise an alert if such a condition is encountered. We evaluate this hypothesis using data obtained from the electronic health records of 4,486 post-cardiac
surgical patients. We base the evaluation on the opinions of a panel of experts. The results support that anomaly-based alerting can have reasonably low false alert rates and that stronger anomalies are correlated with higher alert rates.
Read the full abstract ▾
All authors Milos Hauskrecht, Michal Valko, Shyam Visweswaran, Iyad Batal, Gilles Clermont, Gregory Cooper
Machine Learning
Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang et al. (+2 more)
In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected
datasets and then further fine-tuned for tasks via limited online interactions. In a typical O2O-RL pipeline, candidate policies trained with offline RL are evaluated via either off-policy evaluation (OPE) or online evaluation (OE). The policy with the highest estimated value is then deployed and continually fine-tuned. However, this setup has two main issues. First, OPE can be unreliable, making it risky to deploy a policy based solely on those estimates, whereas OE may identify a viable policy with substantial online interaction, which could have been used for fine-tuning. Second–and more importantly–it is also often not possible to determine a priori whether a pretrained policy will improve with post-deployment fine-tuning, especially in non-stationary environments. As a result, procedures committing to a single deployed policy are impractical in many real-world settings. Moreover, a naive remedy that exhaustively fine-tunes all candidates would violate interaction budget constraints and is likewise infeasible. In this paper, we propose a novel adaptive approach for policy selection and fine-tuning under online interaction budgets in O2O-RL. Following the standard pipeline, we first train a set of candidate policies with different offline RL
algorithms and hyperparameters; we then perform OPE to obtain initial performance estimates. We next adaptively select and fine-tune the policies based on their predicted performance via an upper-confidence-bound approach thereby making efficient use of online interactions. We demonstrate that our approach improves upon O2O-RL baselines with various
benchmarks.
Read the full abstract ▾
All authors Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang, Miroslav Pajic, Yuichi Motai
Machine Learning
Physiologically Grounded Driver Behavior Classification: SHAP-Driven Elite Feature Selection and Hybrid Gradient Boosting for Multimodal Physiological Signals
Sahar Askari, Mohammad Mahdi Mirza Ali Mohammadi, Fatemeh Ensafdoust et al. (+2 more)
An interpretable and scalable framework for decoding driving behaviors from
multimodal physiological signals is proposed in this study. We utilize
multimodal physiological driving behavior large-scale
dataset comprising synchronized electroencephalogram (
EEG), electromyography (EMG), and galvanic skin response (GSR) signals. Our approach involves rigorous preprocessing followed by a domain-specific feature extraction pipeline targeting time-domain, frequency-domain, and derived physiological indices. To address high dimensionality, we employ SHAP-based elite feature selection, retaining the top 250 features to reduce computational overhead while preserving predictive power. Hyperparameter optimization for extreme gradient boosting (XGBoost) and light gradient boosting machine (LightGBM) models is conducted using
Bayesian optimization via Optuna. Finally, a weighted soft-voting ensemble is constructed to leverage the complementary strengths of both gradient boosting frameworks. The results demonstrate that the proposed ensemble achieves a test accuracy of 80.91% and a macro-F1 score of 0.79, significantly outperforming single-modality baselines and traditional machine learning models. Ablation studies confirm an 8% performance gain over the best single modality (
EEG), validating the necessity of
multimodal fusion. SHAP analysis further validates the physiological plausibility of the model, revealing that the
EEG contributes the majority of predictive weight, GSR and EMG features provide critical discriminatory signals for high-arousal and motor-intensive maneuvers.
Read the full abstract ▾
All authors Sahar Askari, Mohammad Mahdi Mirza Ali Mohammadi, Fatemeh Ensafdoust, Amin Golnari, Saeid Sanei
Machine Learning
On the Wasserstein Gradient Flow Interpretation of Drifting Models
Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov et al. (+3 more)
Recently, Deng et al. (2026) proposed Generative Modeling via Drifting (GMD), a novel framework for generative tasks. This note presents an analysis of GMD through the lens of Wasserstein Gradient Flows (WGF), i.e., the path of steepest descent for a functional in the space of probability measures, equipped with the geometry of optimal transport. Unlike previous WGF-based contributions, GMD can be thought of as directly targeting a fixed point of a specific WGF flow. We demonstrate three main results: first, that one
algorithm proposed by Deng et al. (2026) corresponds to finding the limiting point of a WGF on the KL divergence, with Parzen smoothing on the densities. Second, that the
algorithm actually implemented by Deng et al. (2026) corresponds to a different procedure, which bears some resemblance to the fixed point of a WGF on the Sinkhorn divergence, but lacks certain desirable properties of the latter. Third, the same same idea can be extended to the limiting point of other WGFs, including the Maximum Mean Discrepancy (MMD), the sliced Wasserstein distance, and GAN critic functions.
Read the full abstract ▾
All authors Arthur Gretton, Li Kevin Wenliang, Alexandre Galashov, James Thornton, Valentin De Bortoli, Arnaud Doucet
Machine Learning
On the Hardness of Junking LLMs
Marco Rando and Samuel Vaiter
Large
language models (LLMs) are known to be vulnerable to jailbreak attacks, which typically rely on carefully designed
prompts containing explicit semantic structure. These attacks generally operate by fixing an
adversarial instruction and optimizing small
adversarial components (e.g., suffixes or prefixes). In this setting,
prompt structure is fundamental for performance, and recent results show that even simple random search can achieve strong performance when combined with sophisticated
prompt design. Recently, it has been observed that harmful behaviors can be elicited even without the
adversarial prompt, relying solely on optimized
token sequences. This suggests the existence of natural
backdoors, i.e.,
token sequences naturally emerged during LLMs training that trigger unsafe outputs without any meaningful instruction. However, despite these observations, this setting remains largely unexplored, and in particular the hardness of finding natural
backdoors has not been assessed yet. In this work, we provide a first proof-of-concept study investigating the hardness of this task, which we refer to as the junking problem. We formalize it as the problem of finding
token sequences that maximize the probability of generating a target prefix of harmful responses, propose a greedy random-search method to assess is such sequences can be discovered easily. Our results show that this problem is harder than standard jailbreak attacks, confirming the importance of semantic information in
prompt design. At the same time, we find that our simple strategy is sufficient to solve it with a high success rate, suggesting that natural
backdoors are present and easily recoverable. Finally, through perplexity analysis, we observe that the discovered
token sequences lie in low-probability regions of the model distribution, supporting the hypothesis that they emerged implicitly from the training process.
Read the full abstract ▾
Machine Learning
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Daniel Wurgaft, Can Rager, Matthew Kowal et al. (+13 more)
Neural representations carry rich
geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space defined by different geometries, and measure the behavioral trajectories they induce. In particular, we test whether interventions that respect the geometry of activation space will yield behaviors close to those the model exhibits naturally. Concretely, we first fit an activation
manifold $M_h$ to representations and a behavior
manifold $M_y$ to output probability distributions. We then test the link $M_h \leftrightarrow M_y$ via interventions: we find that steering along $M_h$, which we term
manifold steering, yields behavioral trajectories that follow $M_y$, while linear steering – which assumes a Euclidean geometry – cuts through off-
manifold regions and hence produces unnatural outputs. Moreover, optimizing interventions in activation space to produce paths along $M_y$ recovers activation trajectories that trace the
curvature of $M_h$. We demonstrate this bidirectional relationship between the geometry of representation and behavior across tasks and modalities. In
language models, we use
reasoning tasks with cyclic and sequential geometries as well as in-context learning tasks with more complex graph geometries. In a
video world model, we use a task with geometry corresponding to physical dynamics. Overall, our work shows that geometry in neural representation is not merely incidental, but is in fact the proper object for enabling principled control via intervention on internals. This recasts the core problem of steering from finding the right direction to finding the right geometry.
Read the full abstract ▾
All authors Daniel Wurgaft, Can Rager, Matthew Kowal, Vasudev Shyam, Sheridan Feucht, Usha Bhalla, Tal Haklay, Eric Bigelow, Raphael Sarfati, Thomas McGrath, Owen Lewis, Jack Merullo, Noah Goodman, Thomas Fel, Atticus Geiger, Ekdeep Singh Lubana
Machine Learning
How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences
Mariia Seleznova
We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth $t$ grows jointly with width $n$. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large $t$. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth–width scaling regimes that govern signal propagation: (i) a subcritical regime $t=o(\sqrt n)$, in which the infinite-width approximation remains valid; (ii) a critical regime $t\sim c\sqrt n$, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime $t\gg \sqrt n$, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results show that what constitutes "large width" depends crucially on the architecture: recurrent weight sharing makes finite-width effects visible at substantially shorter depths than in feedforward networks.
Read the full abstract ▾
Machine Learning
Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime
Tianshu Zhu, Wenyu Zhang, Xiaoying Zuo et al. (+8 more)
Agentic reinforcement learning (RL) for software engineering spends much of its compute on stateful trajectories whose grouped binary rewards are highly skewed and weakly contrastive. We frame this as pass-rate control and show that the binary reward-side signal is strongest near a 50% rollout pass rate under four criteria: reward
entropy, group-filtering survival, leave-one-out (RLOO) advantage energy under Group Relative Policy Optimization (GRPO), and success-failure pair count. We propose Prefix Sampling (PS), which replays self-generated trajectory prefixes to steer skewed groups toward this regime: successful prefixes give mostly failing groups a head start, while failing prefixes handicap mostly passing groups. Replayed states are reconstructed through the existing rollout path, and replayed
tokens are masked from the loss so optimization applies only to current-policy continuations. On SWE-bench Verified, PS reaches the baseline high-score regime within evaluation variability while delivering 2.01x and 1.55x end-to-end wall-clock speedups on Qwen3-14B and Qwen3-32B; the 14B peak improves from 0.274 to 0.295. AIME 2025 experiments on 4B and 8B show the same pass-rate-control pattern, and 4B ablations attribute gains to replay, bidirectional coverage, and adaptive control.
Read the full abstract ▾
All authors Tianshu Zhu, Wenyu Zhang, Xiaoying Zuo, Lun Tian, Haotian Zhao, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu, Dawei Yin, Dou Shen
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Diversity-Aware Active Learning Builds Materials Datasets for Current and Future Objectives
Rafael Espinosa Castañeda, Ashley Dale, Hongchen Wang et al. (+6 more)
Materials science data collection can be expensive, making
dataset reuse and long-term utility important for future discovery. In practice, data collection often prioritizes properties of current interest while other properties are excluded from the acquisition process. This can produce
datasets that are well suited for current objectives but lose information needed for properties of future interest. Here, we present how to construct
datasets that remain informative for both. We develop and evaluate acquisition frameworks that combine predictive disagreement with diversity in material features, predicted outcomes, or both. We test these approaches across four DFT
datasets and an experimental thermoelectric
dataset. Diversity inclusion in
dataset construction can reduce present and future objectives trade-off by preserving a broader range of materials and property behavior during acquisition. Across all
datasets, the Non-dominated Sorting Genetic
Algorithm II with disagreements and feature diversity (NSGA-II Dis. + diversity) framework developed in this work provides the most robust balance between current and future objectives. It maintains prediction performance above random sampling for current objectives while mitigating bias on excluded properties and, in many cases, improving their performance beyond random sampling. This diversity-aware acquisition framework provides a practical route to building materials
datasets with unbiased quality entries and mitigating cold-start limitations in subsequent modeling and discovery campaigns.
Read the full abstract ▾
All authors Rafael Espinosa Castañeda, Ashley Dale, Hongchen Wang, Yonatan Kurniawan, Hao Wan, Runze Zhang, Adji Bousso Dieng, Kangming Li, Jason Hattrick-Simpers
Machine Learning
Unified Framework of Distributional Regret in Multi-Armed Bandits and Reinforcement Learning
Harin Lee and Min-hwan Oh
We study the distribution of regret in
stochastic multi-armed bandits and episodic reinforcement learning through a unified framework. We formalize a distributional regret bound as a probabilistic guarantee that holds uniformly over all confidence levels $\delta \in (0,1]$, thereby characterizing the regret distribution across the full range of $\delta$. We present a simple UCBVI-style
algorithm with exploration bonus $\min\{c_{1,k}/N, c_{2,k}/\sqrt{N}\}$, where $N$ denotes the visit count and $(c_{1,k},c_{2,k})$ are user-specified parameters. For arbitrary parameter sequences, we derive general gap-independent and gap-dependent distributional regret bounds, yielding a principled characterization of how the parameters control the trade-off between expected performance, tail risk, and instance-dependent behavior. In particular, our bounds achieve optimal trade-offs between expected and distributional regret in both minimax and instance-dependent regimes. As a special case, for multi-armed bandits with $A$ arms and horizon $T$, we obtain a distributional regret bound of order $\mathcal{O}(\sqrt{AT}\log(1/\delta))$, confirming the
conjecture of Lattimore & Szepesvári (2020, Section 17.1) for the first time.
Read the full abstract ▾
Machine Learning
Continual Knowledge Updating in LLM Systems: Learning Through Multi-Timescale Memory Dynamics
Andreas Pattichis and Constantine Dovrolis
LLMs are trained once, then deployed into a world that never stops changing. External
memory compensates for this, but most systems manage it explicitly rather than letting it adapt on its own. Biological
memory works differently: coupled multi-timescale dynamics make new associations immediately usable, strengthen what repetition confirms, and let the rest fade. We argue that external
memory should follow a similar principle. In Memini, this view takes the form of an associative
memory that organizes knowledge as a directed graph. Each edge carries two coupled internal variables, one fast and one slow, following the Benna-Fusi model of synaptic consolidation. From this coupling, episodic sensitivity, gradual consolidation, and selective forgetting are expected to emerge as facets of a single mechanism, reframing external
memory as a learning substrate that reorganizes through its own dynamics. This workshop article describes an early-stage conceptual design without experimental evaluation.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.GR
A Bayesian Approach for Task-Specific Next-Best-View Selection with Uncertain Geometry
Jingsen Zhu, Silvia Sellán and Alexander Terenin
We develop a framework for task-specific active next-best-view selection in 3D reconstruction from point clouds, by casting the problem in the
language of
Bayesian decision theory. Our framework works by (a) placing a prior distribution over the space of implicit surfaces, (b) using recently-developed
stochastic surface reconstruction methods to calculate the resulting posterior distribution, then (c) using the posterior distribution to carefully reason about which view to scan next. This enables us to perform camera selection in a manner that is directly optimized for the intended use of the reconstructed data - meaning, we reduce uncertainty only in those regions that make a difference in the task at hand, as opposed to prior approaches that reduce it uniformly across space. We evaluate our method across three distinct downstream tasks: semantic classification,
segmentation, and PDE-guided physics simulation. Experimental results demonstrate that our framework achieves superior task performance with fewer views compared to commonly used baselines and prior general uncertainty-reduction techniques.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Proximal Projection for Doubly Sparse Regularized Models
Jia Wei He, R. Ayesha Ali and Gerarda Darlington
Regularization is often used in high-dimensional regression settings to generate a sparse model, which can save tremendous computing resources and identify predictors that are most strongly associated with the response. When the predictors can be represented by a Gaussian graphical model, the structure of the predictor graph can be exploited during regularization. Our proposed model exploits this underlying predictor graph structure by decomposing the estimated coefficient vector into a sum of latent variables that correspond to the sum of each node contribution to the coefficient vector. Regularization is then performed on the latent variables rather than on the coefficient vector directly. We use a penalty function that permits a clear user-defined trade-off between the L1 and L2 penalties and propose a novel proximal projection during optimization. Further, our implementation computes the projection operator for the intersection of selected groups, which conserves more computing resources compared to predictor duplication methods, especially for high-dimensional data. Through simulation, we evaluate the performance of our approach under different graph structures and node counts, and present results on real-world data. Results suggest that our method exhibits stable performance relative to other singly or doubly sparse graphical regression models.
Read the full abstract ▾
Machine Learning
Gated Multimodal Learning for Interpretable Property Energy Performance Prediction and Retrofit Scenario Analysis
Yunfei Bai, Aaron Tesfa Tsion, Raul Rosales et al. (+2 more)
Achieving resilient and sustainable cities requires scalable approaches to decarbonising residential buildings, which account for about 20% of UK greenhouse gas emissions and 25% of energy-related emissions in the European Union. Energy Performance Certificates (EPCs) support regulation and retrofit planning, but their reliance on on-site inspections limits timely city-scale assessment. This study introduces a gated
multimodal model to predict Standard Assessment Procedure (SAP) energy efficiency and Environmental Impact (EI) scores by integrating EPC tabular variables, assessor-written free text, and Geographic Information System (GIS)-derived spatial features describing footprint geometry, height, area, and orientation. Sample-wise gating learns property-specific modality weights, while an auxiliary band classification head stabilises training. In a Westminster, London case study, the model predicts SAP and EI scores with MAEs of 4.03 and 4.76 points and R2 values of 0.757 and 0.748, respectively, achieving a mean MAE of 4.39. Ablation results show that full
multimodal fusion outperforms unimodal and bimodal baselines for both score prediction and band-level classification. Interpretability analyses provide decision-relevant evidence: gating weights indicate strong reliance on assessor text; SHAP highlights main fuel, built form, and construction age band; text occlusion prioritises roof and wall fields; and spatial attribution is dominated by height and footprint area, with sensitivity to footprint shape. The validated framework is further applied to retrofit scenarios for wall insulation, roof insulation, and window glazing upgrades, indicating projected improvements in SAP, EI, annual energy cost, and equivalent CO2 emissions. Overall, the framework provides scalable property-level evidence for retrofit screening, intervention prioritisation, and net-zero housing transitions.
Read the full abstract ▾
All authors Yunfei Bai, Aaron Tesfa Tsion, Raul Rosales, Barbara Shollock, Wei He