Machine Learning · Cross-listed from quant-ph
Reducing the Complexity of Matrix Multiplication by Quantum Computing
Jiaqi Yao, Tianjian Huang, Tonghe Zhang et al. (+1 more)
Matrix multiplication is a fundamental operation in compute-intensive tasks and a key component of modern
quantum acceleration frameworks. Here we present a
quantum matrix multiplication
algorithm based on
quantum kernels (QKMM), achieving an elementary gate complexity of \(O(N^2\log_2N)\), with amplitude encoding overhead explicitly included and without assuming a QRAM oracle. This scaling is asymptotically lower than that of the best-known classical
matrix multiplication
algorithm \(O(N^{2.371339})\). Building upon QKMM, we establish a family of
quantum linear
algebra operators, including
Quantum Vector Inner Product (V${\scriptstyle 2}$V),
Quantum Vector-
Matrix Multiplication (V${\scriptstyle 2}$M), QKMM (M${\scriptstyle 2}$M),
Quantum One-to-Many
Matrix Multiplication(O${\scriptstyle 2}$M) and
Quantum Sequential
Matrix Multiplication (SMM), providing a unified framework from vector operations to parallel and sequential
matrix transformations. Through noiseless simulations, realistic noise modelling and experiments on a superconducting
quantum processor, we systematically characterize the numerical accuracy, resource requirements and hardware execution limits of this operator framework. Furthermore, we integrate SMM into deep neural-network inference, enabling intermediate features to propagate coherently across layers without repeated measurement and re-encoding. These results establish a pathway from
quantum circuit-level
algorithm design to end-to-end coherent computation, providing a
quantum computing framework for
matrix-centric compute-intensive applications.
Read the full abstract ▾
All authors Jiaqi Yao, Tianjian Huang, Tonghe Zhang, Ding Liu
Machine Learning
Steering Large Reasoning Models towards Concise Reasoning via Flow Matching
Yawei Li, Benjamin Bergner, Yinghan Zhao et al. (+3 more)
Large
Reasoning Models (LRMs) excel at complex
reasoning tasks, but their efficiency is often hampered by overly verbose outputs. Prior steering methods attempt to address this issue by applying a single, global vector to hidden representations – an approach grounded in the restrictive linear representation hypothesis. In this work, we introduce FlowSteer, a nonlinear steering method that goes beyond uniform linear shifts by learning a complete transformation between the distributions associated with verbose and concise
reasoning. This transformation is learned via Flow Matching as a velocity field, enabling precise, input-dependent control over the model's
reasoning process. By aligning steered representations with the distribution of concise-
reasoning activations, FlowSteer yields more compact
reasoning than the linear shifts. Across diverse
reasoning benchmarks, FlowSteer demonstrates strong task performance and
token efficiency compared to leading inference-time baselines. Our work demonstrates that modeling the full distributional transport with generative techniques offers a more effective and principled foundation for controlling LRMs.
Read the full abstract ▾
All authors Yawei Li, Benjamin Bergner, Yinghan Zhao, Vihang Prakash Patil, Bei Chen, Cheng Wang
Machine Learning
When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging
Yayuan Li, Ze Peng, Jian Zhang et al. (+3 more)
Model merging combines multiple fine-tuned models into a single model by adding their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across
vision and
language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0%. Code is available at https://github.com/lyymuwu/SVC.
Read the full abstract ▾
All authors Yayuan Li, Ze Peng, Jian Zhang, Jintao Guo, Yue Duan, Yinghuan Shi
Machine Learning
Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification
Tao Huang, Rui Wang, Xiaofei Liu et al. (+3 more)
%Large
vision-
language models (LVLMs) have shown substantial advances in
multimodal understanding and generation. However, when presented with incompetent or
adversarial inputs, they frequently produce unreliable or even harmful content, such as fact
hallucinations or dangerous instructions. This misalignment with human expectations, referred to as misbehaviors of LVLMs, raises serious concerns for deployment in critical applications. These misbehaviors are found to stem from epistemic uncertainty, specifically either conflicting internal knowledge or the absence of supporting information. However, existing uncertainty quantification methods, which typically capture only overall epistemic uncertainty, have shown limited effectiveness in identifying such issues. To address this gap, we propose Evidential Uncertainty Quantification (EUQ), a fine-grained method that captures both information conflict and ignorance for effective detection of LVLM misbehaviors. In particular, we interpret features from the model output head as either supporting (positive) or opposing (negative) evidence. Leveraging Evidence Theory, we model and aggregate this evidence to quantify internal conflict and knowledge gaps within a single forward pass. %We extensively evaluate our method across four categories of misbehavior, including
hallucinations, jailbreaks,
adversarial vulnerabilities, and out-of-distribution (OOD) failures, using state-of-the-art LVLMs, and find that EUQ consistently outperforms strong baselines, showing that
hallucinations correspond to high internal conflict and OOD failures to high ignorance. Furthermore, layer-wise evidential uncertainty dynamics analysis helps interpret the
evolution of internal representations from a new perspective. The source code is available at https://github.com/HT86159/EUQ.
Read the full abstract ▾
All authors Tao Huang, Rui Wang, Xiaofei Liu, Yi Qin, Li Duan, Liping Jing
Machine Learning · Cross-listed from cs.AI
Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities
Florian Dietz, William Wale, Oscar Gilg et al. (+5 more)
Detecting misalignment in large
language models is challenging because models may learn to conceal misbehavior during training. Standard
auditing techniques fall short: black-box methods often cannot distinguish misaligned outputs from benign ones, and mechanistic interpretability does not scale with model capabilities. We introduce Split Personality Training (SPT), which fine-tunes a second “honest persona” into LoRA parameters that remain inactive during normal operation. After the main model responds, we activate the LoRA adapter and insert a trigger string, enabling the honest persona to review the response while accessing the main model's latent states. We test our method on the Anthropic
Auditing Game Model Organism, a
benchmark where Llama-3.3-70B is trained to exploit reward hacks while concealing this behavior. SPT achieves 96% overall accuracy, whereas Anthropic reports near 0% accuracy. The honest persona reveals latent knowledge inaccessible to external observers, such as the fictional biases the compromised model was trained on.
Read the full abstract ▾
All authors Florian Dietz, William Wale, Oscar Gilg, Robert McCarthy, Felix Michalak, Gustavo Ewbank Rodrigues Danon, Miguelito de Guzman, Dietrich Klakow
Machine Learning · Cross-listed from math.OC
Solving Stochastic Variational Inequalities without the Bounded Variance Assumption
Ahmet Alacaoglu and Jun-Hyun Kim
We analyze
algorithms for solving
stochastic variational inequalities (VI) without the bounded variance or bounded domain assumptions, where our main focus is min-max optimization with possibly unbounded constraint sets. We focus on two classes of problems: monotone VIs; and structured nonmonotone VIs that admit a solution to the weak Minty VI. The latter assumption allows us to solve structured nonconvex-nonconcave min-max problems. For both classes of VIs, to make the expected residual norm less than $\varepsilon$, we show an oracle complexity of $\widetilde{O}(\varepsilon^{-4})$, which is the best-known for constrained VIs. In our setting, this complexity had been obtained with the bounded variance assumption in the literature, which is not even satisfied for bilinear min-max problems with an unbounded domain. We obtain this complexity for
stochastic oracles whose variance can grow as fast as the squared norm of the optimization variable.
Read the full abstract ▾
Machine Learning
A Unified Framework for Rethinking Policy Divergence Measures in GRPO
Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan et al. (+6 more)
Reinforcement Learning with Verified Reward (RLVR) has emerged as a critical paradigm for advancing the
reasoning capabilities of Large
Language Models (LLMs). Most existing RLVR methods, such as GRPO and its variants, ensure stable updates by constraining policy divergence through clipping likelihood ratios. This paper introduces a unified clipping framework that characterizes existing methods via a general notion of policy divergence, encompassing both likelihood ratios and Kullback-Leibler (KL) divergences and extending to alternative measures. The framework provides a principled foundation for systematically analyzing how different policy divergence measures affect exploration and performance. We further identify the KL3
estimator, a variance-reduced Monte Carlo
estimator of the KL divergence, as a key policy divergence constraint. We theoretically demonstrate that the KL3-based constraint is mathematically equivalent to an asymmetric ratio-based clipping that reallocates probability mass toward high-confidence actions, promoting stronger exploration while retaining the simplicity of GRPO-style methods. Empirical results on mathematical
reasoning benchmarks demonstrate that incorporating the KL3
estimator into GRPO improves both training stability and final performance, highlighting the importance of principled policy divergence constraints in policy optimization.
Read the full abstract ▾
All authors Qingyuan Wu, Yuhui Wang, Simon Sinong Zhan, Yanning Dai, Shilong Deng, Sarra Habchi, Qi Zhu, Matthias Gallé, Chao Huang
Machine Learning · Cross-listed from math.OC
Convergence Rate of the Last Iterate of Stochastic Proximal Algorithms
Kevin Kurian Thomas Vaidyan, Michael P. Friedlander and Ahmet Alacaoglu
We analyze two classical
algorithms for solving additively composite convex optimization problems where the objective is the sum of a smooth term and a nonsmooth regularizer: proximal
stochastic gradient method for a single regularizer; and the randomized incremental proximal method, which uses the proximal operator of a randomly selected function when the regularizer is given as the sum of many nonsmooth functions. We focus on relaxing the bounded variance assumption that is common, yet stringent, for getting last iterate convergence rates. We prove the $\widetilde{O}(1/\sqrt{T})$ rate of convergence for the last iterate of both
algorithms under componentwise convexity and smoothness, which is optimal up to log terms. Our results apply directly to graph-guided regularizers that arise in multi-task and
federated learning, where the regularizer decomposes as a sum over edges of a collaboration graph.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Semantic Purification for Conditional Representation Learning
Jiaquan Wang, Yan Lyu, Chen Li et al. (+1 more)
Conditional representation learning aims to extract criterion-specific features for customized tasks. Recent methods construct conditional subspaces spanned by criterion-specific text bases in the
embedding space of
vision-
language models (VLMs). Image
embeddings are then projected onto these subspaces to obtain conditional representations. However, since VLMs are not explicitly trained to disentangle semantics associated with different criteria, the corresponding conditional subspaces remain coupled. This coupling induces semantic leakage during projection, thereby degrading the semantic purity of conditional representations. To suppress semantic leakage, we propose Semantic Purification for Conditional Representation Learning (SP-CRL). Specifically, SP-CRL first decomposes the original text basis and performs
curvature-based adaptive truncation on the resulting basis vectors to construct a purer conditional subspace. It then identifies an appropriate noise subspace and projects image
embeddings onto its null space to remove irrelevant semantic components. Extensive experiments across customized clustering, customized few-shot classification, and customized
retrieval tasks demonstrate that SP-CRL achieves state-of-the-art performance with superior generalization.
Read the full abstract ▾
All authors Jiaquan Wang, Yan Lyu, Chen Li, Yuheng Jia
Machine Learning
Thermodynamic Limits of Physical Intelligence
Koichi Takahashi and Yusuke Hayashi
Modern AI systems achieve remarkable capabilities at the cost of substantial energy consumption. To connect intelligence to physical efficiency, we propose two complementary bits-per-joule metrics under explicit accounting conventions: (1) Thermodynamic Epiplexity per Joule, new bits of structure about a specified environment-instance variable encoded in an
agent's state per unit energy, and (2) Empowerment per Joule, sensorimotor channel capacity per expected energetic cost over a fixed horizon. These give two axes of physical intelligence, recognition versus control, but the resulting numbers are
benchmark-relative rather than universal. Drawing on
stochastic thermodynamics, we formulate a Landauer-scale closed-cycle
benchmark for epiplexity acquisition by combining a thermodynamic-learning inequality with data processing, and clarify why boundary closure is required; conversely, a decoupling construction shows that without such assumptions information gain and in-boundary dissipation need not be tightly linked. For empirical settings where the latent structure variable is unavailable, we recommend compute-bounded MDL epiplexity / compression-gain surrogates. Finally, we propose a unified efficiency framework with a minimal checklist of conventions for relative bits-per-joule comparisons, and give a compact
language-model reporting example.
Read the full abstract ▾
Machine Learning
Beyond Success Rates: Trainability and Extractability for Offline GCRL
Jan Malte Töpperwien, Aditya Mohan and Marius Lindauer
Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.IV
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
Budhaditya Mukhopadhyay, Chirag Mandal, Pavan Tummala et al. (+3 more)
Liver tumour ablation presents a significant clinical challenge: whilst tumours are clearly visible on pre-operative MRI, they are often effectively invisible on intra-operative CT due to minimal contrast between pathological and healthy tissue. This work investigates the feasibility of cross-modality weak supervision for scenarios where pathology is visible in one modality (MRI) but absent in another (CT). We present a hybrid registration-
segmentation framework that combines MSCGUNet for inter-modal image registration with a UNet-based
segmentation module, enabling registration-assisted pseudo-label generation for CT images. Our evaluation on the CHAOS
dataset demonstrates that the pipeline can successfully register and segment healthy liver anatomy, achieving a Dice score of 0.72. However, when applied to clinical data containing tumours, performance degrades substantially (Dice score of 0.16), revealing the fundamental limitations of current registration methods when the target pathology lacks corresponding visual features in the target modality. We analyse the "domain gap" and "feature absence" problems, demonstrating that whilst spatial propagation of labels via registration is feasible for visible structures, segmenting truly invisible pathology remains an open challenge. Our findings highlight that registration-based label transfer cannot compensate for the absence of discriminative features in the target modality, providing important insights for future research in cross-modality medical image analysis. Code an weights are available at: https://github.com/BudhaTronix/Weakly-Supervised-Tumour-Detection
Read the full abstract ▾
All authors Budhaditya Mukhopadhyay, Chirag Mandal, Pavan Tummala, Naghmeh Mahmoodian, Andreas Nürnberger, Soumick Chatterjee
Machine Learning
BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs
Sheshansh Agrawal, Thien Hang Nguyen and Douwe Kiela
Selecting the top $m$ from $n$ items via expensive $k$-wise comparisons is central to settings ranging from LLM-based document reranking to crowdsourced evaluation and tournament design. Existing methods either rely on heuristics that discard comparison information, or exploit it at prohibitive cost. We introduce a tournament graph framework that provides a principled foundation for $k$-wise ranking. Our key observation is that each $k$-item comparison reveals an induced tournament of $\binom{k}{2}$ pairwise preferences; aggregating these into a global preference graph and computing its transitive closure yields many additional orderings without further oracle calls. We formalize when the current top-$m$ output is certifiably determined and design a greedy query schedule that maximizes information gain towards identifying the top-$m$ items. The framework also gracefully handles non-transitive preferences – cycles induced by real-world oracles – by collapsing them into equivalence classes that yield principled tiered rankings. Applied to LLM reranking across 14
benchmarks and 5 models, BlitzRank achieves Pareto dominance over existing approaches: matching or exceeding accuracy while requiring 25–40% fewer
tokens than comparable methods; against pairwise reranking, it achieves near-identical quality with 7$\times$ fewer
tokens. Code available at https://github.com/ContextualAI/BlitzRank.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Day-Ahead Electricity Price Forecasting for Volatile Markets Using Foundation Models with Regularization Strategy
Kritchanat Ponyuenyong, Pengyu Tu, Jia Wei Tan et al. (+3 more)
Electricity price
forecasting (EPF) is essential for energy
markets stakeholders (e.g. grid operators, energy traders, policymakers) but remains challenging due to the inherent
volatility and nonlinearity of price signals. Traditional statistical and deep learning (DL) models often struggle to capture complex temporal dependencies and integrate heterogeneous data effectively. While time series foundation models (TSFMs) have shown strong performance in general time series
forecasting tasks, such as traffic
forecasting and weather
forecasting. However, their effectiveness in day-ahead EPF, particularly in volatile
markets, remains underexplored. This paper presents a spike regularization strategy and evaluates a wide range of TSFMs, including Tiny Time Mixers (TTMs), MOIRAI, MOMENT, and TimesFM, against traditional statistical and DL models such as Autoregressive Integrated Moving Average (ARIMA), Long-short Term
Memory (LSTM), and Convolutional Neural Network - LSTM (CNN-LSTM) using half-hourly wholesale
market data with volatile trends in Singapore. Exogenous factors (e.g. weather and calendar variables) are also incorporated into models where applicable. Results demonstrate that TSFMs consistently outperform traditional approaches, achieving up to 37.4% improvement in MAPE across various evaluation settings. The findings offer practical guidance for improving forecast accuracy and decision-making in volatile electricity
markets.
Read the full abstract ▾
All authors Kritchanat Ponyuenyong, Pengyu Tu, Jia Wei Tan, Wei Soon Cheong, Jamie Ng Suat Ling, Lianlian Jiang
Machine Learning · Cross-listed from cs.CE
Reduced-Order Surrogates for Forced Flexible Mesh Coastal-Ocean Models
Freja Høgholm Petersen, Jesper Sandvig Mariegaard, Rocco Palmitessa et al. (+1 more)
While proper orthogonal decomposition (POD)-based surrogates are widely explored for hydrodynamic applications, the use of Koopman autoencoders for real-world coastal-ocean modelling remains relatively limited. This paper introduces a flexible Koopman autoencoder formulation that incorporates meteorological forcings and boundary conditions, and systematically compares its performance against POD-based surrogates. The Koopman autoencoder employs a learned linear temporal operator in latent space, enabling
eigenvalue regularization to promote temporal stability. This strategy is evaluated alongside temporal unrolling techniques for achieving stable and accurate long-term predictions. The models are assessed on three test cases spanning distinct dynamical regimes, with prediction horizons up to one year at 30-minute temporal resolution. Across all cases, the reduced order surrogates with temporal unrolling achieve high accuracy with relative root-mean-squared-errors of 0.0068-0.14 and $R^2$-values of 0.61-0.995, where prediction errors are largest for current velocities, and smallest for water surface elevations. In two of the three cases, the Koopman Autoencoder have higher accuracy than the POD-based surrogates. Comparing to in-situ observations, the surrogate yields -0.64% to 12% increase in water surface elevation prediction error when compared to prediction errors of the physics-based model. These error levels, corresponding to a few centimeters, are acceptable for many practical applications, while inference speed-ups of 300-1400x enables workflows such as ensemble
forecasting and long
climate simulations for coastal-ocean modelling.
Read the full abstract ▾
All authors Freja Høgholm Petersen, Jesper Sandvig Mariegaard, Rocco Palmitessa, Allan P. Engsig-Karup
Machine Learning
Robust Federated Learning via Byzantine Filtering over Encrypted Updates
Adda Akram Bendoukha, Aymen Boudguiga, Nesrine Kaaniche et al. (+3 more)
Federated Learning (FL) aims to train a collaborative model while preserving data
privacy. However, the distributed nature of this approach still raises
privacy and security issues, such as the exposure of sensitive data due to inference attacks and the influence of Byzantine behaviors on the trained model. In particular, achieving both secure aggregation and Byzantine resilience remains challenging, as existing solutions often address these aspects independently. In this work, we propose to address these challenges through a novel approach that combines homomorphic encryption for
privacy-preserving aggregation with property-inference-inspired meta-classifiers for Byzantine filtering. First, following the property-inference attacks blueprint, we train a set of filtering meta-classifiers on labeled shadow updates, reproducing a diverse ensemble of Byzantine misbehaviors in FL, including
backdoor, gradient-inversion, label-flipping and shuffling attacks. The outputs of these meta-classifiers are then used to cancel the Byzantine encrypted updates by reweighting. Second, we propose an automated method for selecting the optimal kernel and the dimensionality hyperparameters with respect to homomorphic inference, aggregation constraints and efficiency over the CKKS cryptosystem. Finally, we demonstrate through extensive experiments the effectiveness of our approach against Byzantine participants on the FEMNIST, CIFAR10, GTSRB, and acsincome
benchmarks. More precisely, our SVM filtering achieves accuracies between $90$% and $94$% for identifying Byzantine updates at the cost of marginal losses in model utility and encrypted inference runtimes ranging from $6$ to $24$ seconds and from $9$ to $26$ seconds for an overall aggregation.
Read the full abstract ▾
All authors Adda Akram Bendoukha, Aymen Boudguiga, Nesrine Kaaniche, Renaud Sirdey, Didem Demirag, Sébastien Gambs
Machine Learning · Cross-listed from stat.ML
Optimal Bayesian Stopping for Efficient Inference of Consistent LLM Answers
Jingkai Huang, Will Ma and Zhengyuan Zhou
A simple strategy for improving LLM accuracy, especially in math and
reasoning problems, is to sample multiple responses and submit the answer most consistently reached. In this paper we leverage
Bayesian prior information to save on sampling costs, stopping once sufficient consistency is reached. Although the exact posterior is computationally intractable, we further introduce an efficient "L-aggregated" stopping policy that tracks only the L-1 most frequent answer counts. Theoretically, we prove that L=3 is all you need: this coarse approximation is sufficient to achieve asymptotic optimality, and strictly dominates prior-free baselines, while having a fast posterior computation. Empirically, this identifies the most consistent (i.e., mode) LLM answer using fewer samples, and can achieve similar answer accuracy while cutting the number of LLM calls (i.e., saving on LLM inference costs) by up to 50%.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better
Ji Zhao, Yufei Gu, Shitong Shao et al. (+3 more)
As Large
Language Models (LLMs) achieve remarkable empirical success through scaling model and data size, pretraining has become increasingly critical yet computationally prohibitive, hindering rapid development. Despite the availability of numerous pretrained LLMs developed at significant computational expense, a fundamental real-world question remains underexplored: Can we leverage existing small pretrained models to accelerate the training of larger models? In this paper, we propose a Late-to-Early Training (LET) paradigm that enables LLMs to explicitly learn later knowledge in earlier steps and earlier layers. The core idea is to guide the early layers of an LLM during early training using representations from the late layers of a pretrained (i.e. late training phase) model. We identify two key mechanisms that drive LET's effectiveness: late-to-early-step learning and late-to-early-layer learning. These mechanisms significantly accelerate training convergence while robustly enhancing both
language modeling capabilities and downstream task performance, enabling faster training with superior performance. Extensive experiments on 1.4B and 7B parameter models demonstrate LET's efficiency and effectiveness. Notably, when training a 1.4B LLM on the Pile
dataset, our method achieves up to 1.6$\times$ speedup with nearly 5% improvement in downstream task accuracy compared to standard training, even when using a pretrained model with 10$\times$ fewer parameters than the target model.
Read the full abstract ▾
All authors Ji Zhao, Yufei Gu, Shitong Shao, Xun Zhou, Liang Xiang, Zeke Xie
Machine Learning
Assessing Electricity Demand Forecasting with Exogenous Data in Time Series Foundation Models
Wei Soon Cheong, Lian Lian Jiang and Jamie Ng Suat Ling
Time-series foundation models have emerged as a new paradigm for
forecasting, yet their ability to effectively leverage exogenous features – critical for electricity demand
forecasting – remains unclear. This paper empirically evaluates foundation models capable of modeling cross-channel correlations against a baseline LSTM with reversible instance normalization across Singaporean and Australian electricity
markets at hourly and daily granularities. We systematically assess MOIRAI, MOMENT, TinyTimeMixers, ChronosX, and Chronos-2 under three feature configurations: all features, selected features, and target-only. Our findings reveal highly variable effectiveness: while Chronos-2 achieves the best performance among foundation models (in zero-shot settings), the simple baseline frequently outperforms all foundation models in Singapore's stable
climate, particularly for short-term horizons. Model architecture proves critical, with synergistic architectural implementations (TTM's channel-mixing, Chronos-2's grouped
attention) consistently leveraging exogenous features, while other approaches show inconsistent benefits. Geographic context emerges as equally important, with foundation models demonstrating advantages primarily in variable climates. These results challenge assumptions about universal foundation model superiority and highlight the need for domain-specific models, specifically in the energy domain.
Read the full abstract ▾
Machine Learning
A Decomposition-based State Space Model for Multivariate Time-Series Forecasting
Shunya Nagashima, Shuntaro Suzuki, Shuitsu Koyama et al. (+1 more)
Multivariate time series (MTS)
forecasting is crucial for decision-making in domains such as weather, energy, and finance. It remains challenging because real-world sequences intertwine slow trends, multi-rate seasonalities, and irregular residuals. Existing methods often rely on rigid, hand-crafted decompositions or generic end-to-end architectures that entangle components and underuse structure shared across variables. To address these limitations, we propose DecompSSM, an end-to-end decomposition framework using three parallel deep state space model branches to capture trend, seasonal, and residual components. The model features adaptive temporal scales via an input-dependent predictor, a refinement module for shared cross-variable context, and an auxiliary loss that enforces reconstruction and orthogonality. Across standard
benchmarks (ECL, Weather, ETTm2, and PEMS04), DecompSSM outperformed strong baselines, indicating the effectiveness of combining component-wise deep state space models and global context refinement.
Read the full abstract ▾
All authors Shunya Nagashima, Shuntaro Suzuki, Shuitsu Koyama, Shinnosuke Hirano
Machine Learning · Cross-listed from cs.CV
Parallel Swin Transformer-Enhanced 3D MRI-to-CT Synthesis for MRI-Only Radiotherapy Planning
Zolnamar Dorjsembe, Hung-Yi Chen, Furen Xiao et al. (+1 more)
MRI provides superior soft tissue contrast without ionizing radiation; however, the absence of electron density information limits its direct use for dose calculation. As a result, current radiotherapy workflows rely on combined MRI and CT acquisitions, increasing registration uncertainty and procedural complexity. Synthetic CT generation enables MRI only planning but remains challenging due to nonlinear MRI-CT relationships and anatomical variability. We propose Parallel Swin
Transformer-Enhanced Med2
Transformer, a 3D architecture that integrates convolutional encoding with dual Swin
Transformer branches to model both local anatomical detail and long-range contextual dependencies. Multi-scale shifted window
attention with hierarchical feature aggregation improves anatomical fidelity. Experiments on public and clinical
datasets demonstrate higher image similarity and improved
geometric accuracy compared with baseline methods. Dosimetric evaluation shows clinically acceptable performance, with a mean target dose error of 1.69%. Code is available at: https://github.com/mobaidoctor/med2
transformer.
Read the full abstract ▾
All authors Zolnamar Dorjsembe, Hung-Yi Chen, Furen Xiao, Hsing-Kuo Pao
Machine Learning · Cross-listed from cs.CV
VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models
Tina Khezresmaeilzadeh, Jike Zhong and Konstantinos Psounis
Vision-
language model (VLM) errors on visual
reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a
benchmark of 1,391 human-verified IQ-style visual
reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and
reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3–70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9–12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual
reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Variance Reduction Based Experience Replay for Policy Optimization
Hua Zheng, Wei Xie, M. Ben Feng et al. (+1 more)
Effective reinforcement learning (RL) for complex
stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is
algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy
algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization
algorithms
Read the full abstract ▾
All authors Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy
Machine Learning
Erase at the Core: Representation Unlearning for Machine Unlearning
Jaewon Lee, Yongwoo Kim and Donghyun Kim
Many approximate machine unlearning methods demonstrate strong logit-level forgetting – such as near-zero accuracy on the forget set – yet continue to preserve substantial information within their internal feature representations. We refer to this discrepancy as superficial forgetting. Recent studies indicate that most existing unlearning approaches primarily alter the final classifier, leaving intermediate representations largely unchanged and highly similar to those of the original model. To address this limitation, we introduce the Erase at the Core (EC), a framework designed to enforce forgetting throughout the entire network hierarchy. EC integrates multi-layer contrastive unlearning on the forget set with retain set preservation through deeply supervised learning. Concretely, EC attaches auxiliary modules to intermediate layers and applies both contrastive unlearning and cross-
entropy losses at each supervision point, with layer-wise weighted losses. Experimental results show that EC not only achieves effective logit-level forgetting, but also substantially reduces representational similarity to the original model across intermediate layers. Furthermore, EC is model-agnostic and can be incorporated as a plug-in module into existing unlearning methods, improving representation-level forgetting while maintaining performance on the retain set.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks
Chaimae Abouzahir, Congbo Ma, Nizar Habash et al. (+1 more)
In recent years, Large
Language Models (LLMs) have become widely used in medical applications, such as clinical decision support, medical education, and medical question answering. Yet, these models are often English-centric, limiting their robustness and reliability for linguistically diverse communities. Recent work has highlighted discrepancies in performance in low-resource
languages for various medical tasks, but the underlying causes remain poorly understood. In this study, we conduct a cross-lingual empirical analysis of LLM performance on Arabic and English medical question and answering. Our findings reveal a persistent
language-driven performance gap that intensifies with increasing task complexity. Tokenization analysis exposes structural fragmentation in Arabic medical text, while reliability analysis suggests that model-reported confidence and explanations exhibit limited correlation with correctness. Together, these findings underscore the need for
language-aware design and evaluation strategies in LLMs for medical tasks.
Read the full abstract ▾
All authors Chaimae Abouzahir, Congbo Ma, Nizar Habash, Farah E. Shamout
Machine Learning
Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting
Hongyi Li, Han Lin and Jun Xu
Oblique decision trees combine the transparency of trees with the power of multivariate decision boundaries, but learning high-quality oblique splits is NP-hard, and practical methods still rely on slow search or theory-free heuristics. We present the Hinge Regression Tree (HRT), which reframes each split as a non-linear least-squares problem over two linear predictors whose max/min envelope induces ReLU-like expressive power. The resulting alternating fitting procedure is exactly equivalent to a damped Newton (Gauss-Newton) method within fixed partitions. We analyze this node-level optimization and, for a backtracking line-search variant, prove that the local objective decreases monotonically and converges; in practice, both fixed and adaptive damping yield fast, stable convergence and can be combined with optional ridge regularization. We further prove that HRT's model class is a universal approximator with an explicit $O(\delta^2)$ approximation rate, and show on synthetic and real-world
benchmarks that it matches or outperforms single-tree baselines with more compact structures.
Read the full abstract ▾
Machine Learning
Bayesian Neighborhood Adaptation for Graph Neural Networks
Paribesh Regmi, Rui Li and Kishan KC
The neighborhood scope (i.e., number of hops) where graph neural networks (GNNs) aggregate information to characterize a node's statistical property is critical to GNNs' performance. Two-stage approaches, training and validating GNNs for every pre-specified neighborhood scope to search for the best setting, is a time-consuming task and tends to be biased due to the search space design. How to adaptively determine proper neighborhood scopes for the aggregation process for both homophilic and heterophilic graphs remains largely unexplored. We thus propose to model the GNNs' message-passing behavior on a graph as a
stochastic process by treating the number of hops as a beta process. This
Bayesian framework allows us to infer the most plausible neighborhood scope for message aggregation simultaneously with the optimization of GNN parameters. Our theoretical analysis shows that the scope inference improves the expressivity of a GNN. Experiments on
benchmark homophilic and heterophilic
datasets show that the proposed method is compatible with state-of-the-art GNN variants, achieving competitive or superior performance on the node classification task, and providing well-calibrated predictions. Implementation is available at : https://github.com/paribeshregmi/BNA-GNN
Read the full abstract ▾
Machine Learning
Smoothness Errors in Dynamics Models and How to Avoid Them
Edward Berman, Luisa Li, Jung Yeon Park et al. (+1 more)
Modern neural networks have shown promise for solving partial differential equations over surfaces, often by discretizing the surface as a mesh and learning with a mesh-aware graph neural network. However, graph neural networks suffer from oversmoothing, where a node's features become increasingly similar to those of its neighbors.
Unitary graph convolutions, which are mathematically constrained to preserve smoothness, have been proposed to address this issue. Despite this, in many physical systems, such as
diffusion processes, smoothness naturally increases and unitarity may be overconstraining. In this paper, we systematically study the smoothing effects of different GNNs for dynamics modeling and prove that
unitary convolutions hurt performance for such tasks. We propose relaxed
unitary convolutions that balance smoothness preservation with the natural smoothing required for physical systems. We also generalize
unitary and relaxed
unitary convolutions from graphs to meshes. In experiments on PDEs such as the heat and wave equations over complex meshes and on weather
forecasting, we find that our method outperforms several strong baselines, including mesh-aware
transformers and equivariant neural networks.
Read the full abstract ▾
All authors Edward Berman, Luisa Li, Jung Yeon Park, Robin Walters
Machine Learning · Cross-listed from stat.ML
Decision-Focused Sequential Experimental Design: A Directional Uncertainty-Guided Approach
Beichen Wan, Mo Liu, Paul Grigas et al. (+1 more)
We consider the sequential experimental design problem in the predict-then-optimize paradigm. In this paradigm, the outputs of the prediction model are used as coefficient vectors in a downstream linear optimization problem. Traditional sequential experimental design aims to control the input variables (features) so that the improvement in prediction accuracy from each experimental outcome (label) is maximized. However, in the predict-then-optimize setting, performance is ultimately evaluated based on the decision loss induced by the downstream optimization, rather than by prediction error. This mismatch between prediction accuracy and decision loss renders traditional decision-blind designs inefficient. To address this issue, we propose a directional-based metric to quantify predictive uncertainty. This metric does not require solving an optimization oracle and is therefore computationally tractable. We show that the resulting sequential design criterion enjoys strong consistency and convergence guarantees. Under a broad class of distributions, we demonstrate that our directional uncertainty-based design attains an earlier stopping time than decision-blind designs. This advantage is further supported by real-world experiments on an LLM job allocation problem.
Read the full abstract ▾
All authors Beichen Wan, Mo Liu, Paul Grigas, Zuo-Jun Max Shen
Machine Learning · Cross-listed from cs.CV
Consistency-Preserving Concept Erasure via Unsafe-Safe Pairing and Directional Fisher-Weighted Adaptation
Yongwoo Kim, Sungmin Cha, Hyunsoo Kim et al. (+2 more)
With the increasing versatility of text-to-image
diffusion models, the ability to selectively erase undesirable concepts (e.g., harmful content) has become indispensable. However, existing concept erasure approaches primarily focus on removing unsafe concepts without providing guidance toward corresponding safe alternatives, which often leads to failure in preserving the structural and semantic consistency between the original and erased generations. In this paper, we propose a novel framework, PAIRed Erasing (PAIR), which reframes concept erasure from simple removal to consistency-preserving semantic realignment using unsafe-safe pairs. We first generate safe counterparts from unsafe inputs while preserving structural and semantic fidelity, forming paired unsafe-safe
multimodal data. Leveraging these pairs, we introduce two key components: (1) Paired Semantic Realignment, a guided objective that uses unsafe-safe pairs to explicitly map target concepts to semantically aligned safe anchors; and (2) Fisher-Weighted Initialization for DoRA, which initializes parameter-efficient low-rank adaptation
matrices using unsafe-safe pairs, encouraging the generation of safe alternatives while selectively suppressing unsafe concepts. Together, these components enable fine-grained erasure that removes only the targeted concepts while maintaining overall semantic consistency. Extensive experiments demonstrate that our approach achieves the best overall trade-off among baselines, providing effective concept erasure while preserving structural integrity, semantic coherence, and generation quality.
Read the full abstract ▾
All authors Yongwoo Kim, Sungmin Cha, Hyunsoo Kim, Jaewon Lee, Donghyun Kim