Machine Learning
Towards Systematic Generalization for Power Grid Optimization Problems
Zeeshan Memon, Yijiang Li, Hongwei Jin et al. (+2 more)
AC Optimal Power Flow (ACOPF) and Security-Constrained Unit Commitment (SCUC) are fundamental optimization problems in power system operations. ACOPF serves as the physical backbone of grid simulation and real-time operation, enforcing nonlinear power flow feasibility and network limits, while SCUC represents a core
market-level decision process that schedules generation under operational and security constraints. Although these problems share the same underlying transmission network and physical laws, they differ in decision variables and temporal coupling, and prior learning-based approaches address them in isolation, resulting in disjoint models and representations.We propose a learning framework that jointly models ACOPF and SCUC through a shared graph-based backbone that captures grid
topology and physical interactions, coupled with task-specific decoders for static and temporal decision-making. Training includes solver supervision with physics-informed objectives to enforce AC feasibility and inter-temporal operational constraints. To evaluate generalization, we assess cross-case transfer on unseen grid
topologies for ACOPF and SCUC without retraining, and systematic generalization on the UC-ACOPF problem using unsupervised, physics-based objectives and a power-dispatch consensus mechanism. Experiments across multiple grid scales demonstrate improved performance and transferability relative to existing learning-based baselines, indicating that the model can support learning across heterogeneous power system optimization problems.
Read the full abstract ▾
All authors Zeeshan Memon, Yijiang Li, Hongwei Jin, Kibaek Kim, Liang Zhao
Machine Learning
Robust and Explainable Divide-and-Conquer Learning for Intrusion Detection
Yan Zhou, Kevin Hamlen, Michael De Lucia et al. (+5 more)
Machine learning-based intrusion detection requires complex models to capture patterns in high-dimensional, noisy, and class-imbalanced raw network traffic, yet deploying such models remains impractical on resource-constrained devices with limited processing power and
memory. In this paper, we present a correlation-aware divide-and-conquer learning technique that decomposes a complex learning problem into smaller, more manageable subproblems. This enables lightweight models as simple as decision trees to be trained on focused subtasks, yielding up to 43.3% higher local accuracy and up to 257 times reduction in model size on real-world network intrusion detection
datasets, while also improving
adversarial robustness and explainability.
Read the full abstract ▾
All authors Yan Zhou, Kevin Hamlen, Michael De Lucia, Murat Kantarcioglu, Latifur Khan, Sharad Mehrotra, Ananthram Swami, Bhavani Thuraisingham
Machine Learning · Cross-listed from stat.ML
MIRA: A Score for Conditional Distribution Accuracy and Model Comparison
Sammy Sharief, Justine Zeghal, Gabriel Missael Barco et al. (+3 more)
We introduce Mira, a sample-based score for assessing the accuracy of a candidate conditional distribution using only joint samples from the true data-generating process. Relying on the principle that distributions coincide if they assign equal probability mass to all regions, we derive an analytic expression for the Mira statistic, whose average defines the Mira score. This formulation further allows us to compute theoretical reference values and uncertainty estimates when the candidate distribution matches the true one. This framework enables model comparison by quantifying the
alignment between the conditional distribution of a candidate model and the true data generating process. Consequently, Mira enables
Bayesian model comparison through direct posterior validation, bypassing the challenging evidence computation. We demonstrate its effectiveness across several toy problems and
Bayesian inference tasks.
Read the full abstract ▾
All authors Sammy Sharief, Justine Zeghal, Gabriel Missael Barco, Pablo Lemos, Yashar Hezaveh, Laurence Perreault-Levasseur
Machine Learning · Cross-listed from eess.SP
Benchmarking Wireless Representations: High-Dimensional vs. Compressed Embeddings for Efficiency and Robustness
Murilo Batista, Shirin Salehi, Saeed Mashdour et al. (+3 more)
Building on recent advances in representation learning for wireless channels, this work investigates the cost-benefit trade-offs of high-dimensional channel
embeddings in practical systems. We
benchmark multiple wireless representations: high-dimensional learned
embeddings from a wireless foundation model, compact autoencoder-based representations with significantly lower dimensionality, and raw data baselines, evaluating their performance across diverse downstream tasks. We then systematically analyze data efficiency, noise robustness, and computational complexity, explicitly characterizing the resource overhead associated with high-dimensional
embeddings. Beyond standard tasks such as line-of-sight/non-line-of-sight (LoS/NLoS) classification and beam selection, we introduce power allocation as a new downstream task. Our results reveal clear trade-offs: while high-dimensional
embeddings can perform well in few-shot regimes for certain tasks, they incur substantial latency and parameter overhead. In contrast, compressed latent representations learned by autoencoders demonstrate improved noise robustness and more stable performance across tasks, while significantly reducing computational and transmission costs.
Read the full abstract ▾
All authors Murilo Batista, Shirin Salehi, Saeed Mashdour, Paul Zheng, Rodrigo C. de Lamare, Anke Schmeink
Machine Learning
How Can One Choose the Best CAM-Based Explainability Method for a CNN Model?
Daniel da Silva Costa, Pedro Nuno de Souza Moura and Adriana C. F. Alvim
In recent years, several advances have been observed in Deep Learning with surprising results. Models in this area have been increasingly used in numerous applications, including those sensitive to human life, which require clear explanations and justifications. Various explainability methods have been proposed, but not many metrics to evaluate these methods. The most commonly used metric is the Intersection over Union (IoU). However, due to the characteristics of the results of the explainability methods, called saliency maps, which do not have a known shape, we hypothesise that there must be a better metric that allows one to find an explainability method that produces results that best resemble the human perception. We propose using different metrics to assess the similarity between human perception and the explanation saliency maps to find a better metric. An investigation was conducted employing a subset of the Chihuahuas images from ImageNet
dataset. Several CAM-based explainability methods were used to generate saliency maps for each chihuahua image.
Alignment was measured by applying distance metrics between the bounding box of human annotations and the saliency maps produced by each explainability method. Rankings of the best saliency maps were created using the results of the distance metrics and compared to the ranking obtained using people's choice, collected through crowdsourcing, of the best explanation saliency maps for each selected image. Comparison between rankings was performed using the Rank-Biased Overlap (RBO) metric. The results indicate the feasibility of our method to find the explainability method that best resembles human perception. In our experiments, the two metrics that best resemble human perception corresponded to Manhattan and Correlation. Besides, the best explainability methods regarding human perception were LayerCAM, Score-CAM, and IS-CAM.
Read the full abstract ▾
Machine Learning
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
Mario Koddenbrock, Christoph Lange, Robin Legner et al. (+6 more)
Machine learning (ML) has transformed many scientific fields, yet key applications still lack standardized
benchmarks. Raman spectroscopy, a widely used technique for non-invasive molecular analysis, is one such field where progress is limited by fragmented
datasets, inconsistent evaluation, and models that fail to capture the structure of spectral data. We introduce RamanBench, the first large-scale, fully reproducible
benchmark for ML on Raman spectroscopy, consisting of streamlined data access (https://pypi.org/project/raman-data/), evaluation protocols and code (https://pypi.org/project/raman-bench/), as well as a live leaderboard (https://huggingface.co/spaces/HTW-KI-Werkstatt/RamanBench). It unifies 77
datasets (including 17 first released with this
benchmark) across four domains, comprising 325,862 spectra and spanning classification and regression tasks under diverse experimental conditions. We
benchmark 28 models under a standardized protocol, including classical methods (e.g., PLS), Raman-specific (e.g., RamanNet), tabular foundation models (TFMs; e.g., TabPFN), and time-series approaches (e.g., ROCKET). TFMs consistently outperform domain-specific and gradient boosting baselines, while time-series models remain competitive. However, no single method dominates across these heterogeneous
datasets, revealing that Raman spectroscopy remains an open challenge for machine learning. Therefore, we invite the community to contribute new approaches to our living
benchmark, with the potential to accelerate advances in critical applications such as medical diagnostics, biological research, and materials science.
Read the full abstract ▾
All authors Mario Koddenbrock, Christoph Lange, Robin Legner, Martin Jäger, Martin Kögler, Mariano N. Cruz Bournazou, Peter Neubauer, Felix Biessmann, Erik Rodner
Machine Learning · Cross-listed from cs.IT
Real-Time Text Transmission via LLM-Based Entropy Coding over Fixed-Rate Channels
Vishnu Teja Kunde, Jean-Francois Chamberland, Krishna R. Narayanan et al. (+1 more)
Learning, prediction, and compression are intimately connected: a model that accurately predicts the next symbol in a sequence can be coupled with a source coder to compress that sequence near its information-theoretic limit. When tokenized characters arriving at a fixed reading pace are encoded into variable-length codewords and streamed over a fixed-rate channel, a queue forms whose per-
token delay depends on the mean and variance of the bit lengths and on the coder's algorithmic latency. This paper investigates the compression–delay tradeoff that arises when a causal
language model serves as the sequential predictor within a predict-then-code architecture for real-time text transmission. Several coding schemes are compared: Shannon (ideal), Huffman, arithmetic coding, rANS at various block sizes, and gzip. The analysis separates algorithmic delay, inherent to the coder, from computational delay, which shrinks as hardware improves. Huffman is the practical choice for over-provisioned channels, with zero algorithmic delay and modest compression overhead. Arithmetic coding achieves near-optimal compression at the cost of decodability delay. Findings are validated across two scales: GPT-2 (124M) and Llama~3.2 (3B), a twenty-five-fold parameter range. This scaling yields an approximately 38% reduction in bits per character, effectively over-provisioning the channel and thereby changing which coder is optimal.
Read the full abstract ▾
All authors Vishnu Teja Kunde, Jean-Francois Chamberland, Krishna R. Narayanan, Jamison Ebert
Machine Learning
DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training
Zechen Ma, Zixi Qu, Jinyan Yi et al. (+2 more)
Distributed machine learning (ML) training has become a necessity with the prevalence of billion to trillion-parameter-scale models. While prior work has improved training efficiency from the ML perspective at the application layer, it often fails to address transient congestion events at the network layer that introduce severe tail latency and training-time variability, thereby undermining the quality of service of distributed ML training systems. Existing network optimizations treat all gradients equally and thus fail to integrate sufficient model-training insights into communication protocol design. In this paper, we present Dynamic Bounded-Loss Protocol (DBLP), a burst-resilient, training-phase-aware, and hardware-agnostic transport protocol that incorporates model-level tolerance properties into gradient communication. By dynamically adjusting gradient loss tolerance across training phases, DBLP reduces overall training time and mitigates tail-latency collapse during transient high-loss events (i.e., microbursts). Compared to the current state-of-the-art solution (baseline), DBLP tolerates significantly higher loss while achieving comparable test accuracy, and reduces end-to-end training time by an average of 24.4% and a maximum of 33.9%. At microburst events, DBLP achieves up to 5.88x single-round communication latency speedups over the baseline, preventing burst-induced tail-latency spikes and maintaining stable training performance.
Read the full abstract ▾
All authors Zechen Ma, Zixi Qu, Jinyan Yi, David Lin, Yashar Ganjali
Machine Learning
Misclassification Rate and Privacy-Utility Trade-offs in Graph Convolutional Networks via Subsampling Stability
Yexin Zhang, Zhongtian Ma, Qiaosheng Zhang et al. (+1 more)
We study differential
privacy (DP) in Graph Convolutional Networks (GCNs) through the framework of subsampling stability. We derive upper bounds on the misclassification rate that depend explicitly on the subsampling probability $p_s$. Furthermore, we characterize the
privacy–utility trade-off by identifying feasible ranges of $p_s$; if $p_s$ is too large, the stability-based
privacy condition becomes difficult to satisfy, yielding vacuous guarantees, whereas if it is too small, accuracy deteriorates. Our results provide the first rigorous theoretical framework for understanding subsampling stability in GCNs under DP.
Read the full abstract ▾
All authors Yexin Zhang, Zhongtian Ma, Qiaosheng Zhang, Zhen Wang
Machine Learning · Cross-listed from cs.CL
Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM
Luo Ji, Qi Qin, Ningyuan Xi et al. (+3 more)
Conventional LLMs may suffer from corpus heterogeneity and subtle condition changes. While finetuning can create the catastrophe forgetting issue, application of meta-learning on LLMs is also limited due to its complexity and scalability. In this paper, we activate the meta-signal of $\beta$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed which dynamically produces $\beta$ on textual conditions, providing meta-controllability on LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our code can be found in https://github.com/AaronJi/MeGan.
Read the full abstract ▾
All authors Luo Ji, Qi Qin, Ningyuan Xi, Teng Chen, Qingqing Gu, Hongyan Li
Machine Learning
AdamO: A Collapse-Suppressed Optimizer for Offline RL
Nan Qiao, Sheng Yue, Shuning Wang et al. (+1 more)
Offline reinforcement learning (RL) can fail spectacularly when bootstrapped temporal-difference (TD) updates amplify their own errors, driving the critic toward extreme and unusable Q-values. A key counterintuitive insight of this work is that collapse is not only a property of the backup rule or network architecture: optimizer dynamics themselves can directly trigger or suppress instability. From a control-theoretic viewpoint, we model offline TD learning as a feedback system and analyze Adam-based critic updates. This yields a necessary and sufficient condition for stability of the induced local update dynamics: within the regime we analyze, these dynamics are stable if and only if the spectral radius of the corresponding update operator is strictly below one. Further analysis suggests that standard Adam updates can inadvertently distort the parameter geometry, motivating explicit orthogonality constraints to prevent TD error amplification. To this end, we propose AdamO, an Adam-based optimizer with a decoupled orthogonality correction regulated by a strict task-
alignment budget. We prove that this design theoretically guarantees worst-case task safety and preserves Adam's continuous-time dissipative dynamics. Empirically, AdamO is broadly compatible with diverse offline RL baselines, improving stability and returns across a broad suite of
benchmarks.
Read the full abstract ▾
All authors Nan Qiao, Sheng Yue, Shuning Wang, Ju Ren
Machine Learning
MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
Yavuz Yarici and Ghassan AlRegib
Deploying
multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as
multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-
Entropy Regularization for Domain Generalization (MER-DG), which maximizes the
entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing
multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC
benchmarks demonstrate average improvements of approximately 5% over standard fusion and approximately 2% over state-of-the-art methods.
Read the full abstract ▾
Machine Learning
Retrieval with Multiple Query Vectors through Anomalous Pattern Detection
Allassan Tchangmena A Nken, Baimam Boukar Jean Jacques, Miriam Rateike et al. (+2 more)
A classical vector
retrieval problem typically considers a single query
embedding vector as input and retrieves the most similar
embedding vectors from a vector database. However, complex
reasoning and
retrieval tasks frequently require multiple query vectors, rather than a single one. In this work, we propose a
retrieval method that considers multiple query vectors simultaneously and retrieves the most relevant vectors from the database using concepts from anomalous pattern detection. Specifically, our approach leverages a set of query vectors $Q$ (with $|Q|\geq 1$), and identifies the subset of vector dimensions within $Q$ that standout (anomalous) from the rest of dimensions. Next, we scan the vector database to retrieve the set of vectors that are also anomalous across the previously identified vector dimensions and return them as our retrieved set of vectors. We validate our approach on two image
datasets, a text
dataset, and a tabular
dataset. Overall, we observe that, across most
datasets, larger query sets lead to improved
retrieval performance. The improvement is most pronounced when increasing the query sets from 1 to 8, while the gains become smaller beyond that.
Read the full abstract ▾
All authors Allassan Tchangmena A Nken, Baimam Boukar Jean Jacques, Miriam Rateike, Celia Cintas, Skyler Speakman
Machine Learning
Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare
Maheed H. Ahmed and Mahsa Ghasemi
Learning from human preference data is becoming a useful tool, from fine-tuning large
language models to training reinforcement learning
agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider
fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote
fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of $\Omega(T^{2/3}\min(K,D)^\frac{1}{3})$ for a time horizon $T$, $K$ arms, and $D$ users, which, to the best of our knowledge, is the first result quantifying the cost of
fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-$\epsilon$-Greedy
algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on $T$ up to logarithmic factors.
Read the full abstract ▾
Machine Learning
Flexi-LoRA with Input-Adaptive Ranks: Efficient Finetuning for Speech and Reasoning Tasks
Zongqian Li, Yixuan Su, Han Zhou et al. (+2 more)
Parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) have become essential for deploying large
language models, yet their static parameter allocation remains suboptimal for inputs of varying complexity. We present Flexi-LoRA, a novel framework that dynamically adjusts LoRA ranks based on input complexity during both training and inference. Through empirical analysis across question answering, mathematical
reasoning, and
speech tasks, we demonstrate that maintaining consistency between training and inference dynamics is important for effective adaptation, particularly for sequential
reasoning tasks. Our findings reveal that input-dependent parameter allocation achieves higher performance with fewer parameters by optimally matching rank configurations to question complexity. Furthermore, task-specific dependency on rank dynamics varies, with mathematical
reasoning tasks exhibiting higher dependency than QA tasks. Successful adaptation manifests not only in correctness but also in
reasoning quality and instruction adherence. Flexi-LoRA consistently outperforms static LoRA while using fewer parameters, with performance gains more pronounced on tasks requiring strict
reasoning chains. Our approach realizes key benefits of mixture-of-experts frameworks through a more streamlined implementation, reducing parameter redundancy while improving model capabilities. We provide comprehensive empirical studies across diverse tasks, establishing a basis for future work in input-adaptive and efficient fine-tuning approaches.
Read the full abstract ▾
All authors Zongqian Li, Yixuan Su, Han Zhou, Zihao Fu, Nigel Collier
Machine Learning
TRAP: Tail-aware Ranking Attack for World-Model Planning
Siyuan Duan, Ke Zhang and Xizhao Luo
World models enable long-horizon planning by internally generating and evaluating imagined trajectories, making them a promising foundation for generalist
agents. However, this imagination-driven decision process also introduces new security risks. Existing
backdoor attacks typically aim to manipulate local features, one-step predictions, or instantaneous policy outputs. While such objectives may suffice for weaker reactive models, they are often ineffective against world models, where the learned dynamics prior and planning process can absorb or wash out the effects of shallow perturbations. More importantly, we find that world models exhibit a distinct
backdoor vulnerability rooted in the long-tailed ranking structure of imagined trajectories, where disrupting the ordering of a few decision-critical trajectories can systematically hijack planning. To exploit this vulnerability, we propose TRAP, a
backdoor attack framework for world models that targets imagined trajectory ranking. TRAP combines a tail-aware ranking loss to focus optimization on decision-critical trajectories with dual gating mechanisms that stabilize optimization and regulate when and where the attack penalty is applied. Under trigger conditions, TRAP alters the relative ranking of imagined trajectories to redirect planning outcomes, while largely maintaining the normal ranking structure on clean inputs. Experiments on DreamerV3 and TD-MPC2 across diverse tasks show that TRAP consistently induces sustained behavioral deviations and significant performance degradation, highlighting the need for dedicated security evaluation of world-model-based
agents.
Read the full abstract ▾
Machine Learning
PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction
Zhiwen Yang, Pan Liu, Yifan Li et al. (+2 more)
Tandem mass spectrometry provides a high-throughput framework for identifying and quantifying
proteins in complex biological samples. In computational proteomics, predicting peptide MS/MS spectra is a critical task, enabling downstream applications such as large-scale peptide identification and quantification. While deep learning architectures have substantially improved prediction accuracy, three evaluation challenges obscure the true progress of the field. First, inconsistent data preprocessing and incompatible model output spaces hinder fair model comparison. Second, flawed data splitting strategies can permit hidden sequence leakage and inflate reported performance. Third, existing evaluations typically lack comprehensive cross-species benchmarking and systematic assessment of model robustness to influential experimental conditions. To address these challenges, we propose PepSpecBench, a unified
benchmark for peptide MS/MS spectrum prediction. PepSpecBench standardizes data preprocessing across complementary public
datasets, enforces a strict backbone-disjoint splitting strategy to eliminate sequence leakage, and evaluates diverse architectures within a shared fragment-ion representation space. It further introduces a comprehensive multi-species evaluation suite and physically grounded metadata perturbation probes to assess model robustness and instrument awareness. We uncover previously unrecognized performance discrepancies and robustness limitations across six representative models, providing actionable insights for future model design, evaluation and practical deployment.
Read the full abstract ▾
All authors Zhiwen Yang, Pan Liu, Yifan Li, Yunhua Zhong, Jun Xia
Machine Learning
Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search
Gerardo A. Flores, Yash Deshpande, Jannis R. Brea et al. (+1 more)
In sequential search, alternatives are tested until the true class is found. Standard proper scoring rules like log loss are local, ignoring the ranking of competitors and misaligning model evaluation with search utility. We show that sequential search induces a pairwise structure that overcomes this. By analyzing the expected cost of optimal search under varying testing costs, we derive Pandora's Regret: a closed-form, pairwise-additive, and strictly proper scoring rule. Pandora's Regret both elicits true probabilities and penalizes rank-reversing miscalibrations where distractors outrank the true class. Our construction yields a one-parameter Beta family that balances penalties for rank-swapping versus probability magnitude, while retaining a grounded interpretation as expected search cost. We prove that log loss, accuracy, and macro-F1 rely on implicit decision models misaligned with sequential search. Across 597 MedMNIST models, Pandora-based metrics better predict clinical diagnostic costs than standard alternatives, extending decision-theoretic scoring rule construction to the multiclass setting.
Read the full abstract ▾
All authors Gerardo A. Flores, Yash Deshpande, Jannis R. Brea, Ashia C. Wilson
Machine Learning · Cross-listed from cs.AR
ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA
Shengzhe Lyu, Yuhan She, Patrick S. Y. Hung et al. (+2 more)
Vision Mamba (ViM) models offer a compelling efficiency advantage over
Transformers by leveraging the linear complexity of State Space Models (SSMs), yet efficiently deploying them on FPGAs remains challenging. Linear layers struggle with dynamic activation outliers that render static quantization ineffective, while uniform quantization fails to capture the weight distribution at low bit-widths. Furthermore, while associative scan accelerates SSMs on GPUs, its
memory access patterns are misaligned with the streaming dataflow required by FPGAs. To address these challenges, we present ViM-Q, a scalable
algorithm-hardware co-design for end-to-end ViM inference on the edge. We introduce a hardware-aware quantization scheme combining dynamic per-
token activation quantization and per-channel smoothing to mitigate outliers, alongside a custom 4-bit per-block Additive Power-of-Two (APoT) weight quantization. The models are deployed on a runtime-parameterizable FPGA accelerator featuring a linear engine employing a Lookup-Table (LUT) unit to replace multiplications with shift-add operations, and a fine-grained pipelined SSM engine that parallelizes the state dimension while preserving sequential recurrence. Crucially, the hardware supports runtime configuration, adapting to diverse dimensions and input resolutions across the ViM family. Implemented on an AMD ZCU102 FPGA, ViM-Q achieves an average 4.96x speedup and 59.8x energy efficiency gain over a quantized NVIDIA RTX 3090 GPU baseline for low-batch inference on ViM-tiny. This co-design shows a viable path for deploying ViM models on resource-constrained edge devices.
Read the full abstract ▾
All authors Shengzhe Lyu, Yuhan She, Patrick S. Y. Hung, Ray C. C. Cheung, Weitao Xu
Machine Learning · Cross-listed from cs.IT
SwiftChannel: Algorithm-Hardware Co-Design for Deep Learning-Based 5G Channel Estimation
Shengzhe Lyu, Yuhan She, Di Duan et al. (+5 more)
Channel estimation is crucial in 5G communication networks for optimizing transmission parameters and ensuring reliable, high-speed communication. However, the use of multiple-input and multiple-output (
MIMO) and millimeter-wave (mmWave) in 5G networks presents challenges in achieving accurate estimation under strict latency requirements on resource-limited hardware platforms. To address these challenges, we propose SwiftChannel, an
algorithm-hardware co-design framework that integrates a hardware-friendly deep learning-based channel
estimator with a dedicated accelerator. Our approach employs a convolutional neural network enhanced with a parameter-free
attention mechanism, which effectively reconstructs full-resolution spatial-frequency domain channel
matrices from low-resolution least squares (LS) estimates. We further develop a multi-stage model compression pipeline combining knowledge
distillation, convolution re-parameterization, and quantization-aware training, resulting in substantial model size reduction with negligible accuracy loss. The hardware accelerator, implementing the compressed model and the LS
estimator on FPGA platforms using High-level Synthesis (HLS), features a fine-grained pipeline architecture and optimized dataflow strategies. Tested on a Zynq UltraScale+ RFSoC, the accelerator achieves sub-millisecond latency, providing up to 24x speed-up and over 33x improvement in energy efficiency compared to GPU-based solutions. Extensive evaluations demonstrate that the proposed design generalizes not only across various noise levels and user mobilities, but also to a variety of unseen channel profiles, outperforming state-of-the-art baselines. By unifying algorithmic innovation with hardware-aware design, our work presents a future-proof channel estimation solution for 5G
MIMO systems.
Read the full abstract ▾
All authors Shengzhe Lyu, Yuhan She, Di Duan, Tao Ni, Yu Hin Chan, Chengwen Luo, Ray C. C. Cheung, Weitao Xu
Machine Learning
Training Non-Differentiable Networks via Optimal Transport
An T. Le
Hard thresholds, quantization, and discrete routing can produce training losses with flat regions and jumps, where ordinary gradients vanish or are undefined. We introduce PolyStep, a forward-only optimizer that evaluates rotated polytope probes and moves parameter blocks along weighted averages of the probe directions. We derive the weights from one-sided entropic transport and use its uncoupled softmax solution in our primary experiments. Our analysis explains when variation among probe costs produces motion and when that motion decreases the loss. On a regular simplex, nonconstant costs always give a nonzero direction. For monotone ridge losses, the softmax update cannot increase the loss at any positive temperature; a perturbation bound gives sufficient conditions for descent near curved jumps. For bounded measurable losses, we randomize the probe radii and identify an exact smoothing whose gradient equals the expected linear cost-weighted direction up to scale. This identity yields a stationarity bound for an idealized fixed-temperature variant: under regularity and sampling assumptions stronger than those met by our trained configurations, the bound has an $O(T^{-1/2})$ term and a persistent bias floor. We evaluate the practical method on networks with hard operations, discrete optimization, and policy search. On MNIST with hard-threshold spiking
neurons, PolyStep reaches $93.0 \pm 0.2\%$, compared with $79.6 \pm 5.2\%$ for the best-tuned gradient-free baseline at matched evaluations. These gains come with a query cost proportional to the search dimension per fresh step, which limits the number of updates available at a fixed budget.
Read the full abstract ▾
Machine Learning
Deep learning-based pavement performance modeling using multiple distress indicators and road work history
Lu Gao, Zhe Han and Yunshen Chen
The deterioration of pavement is a complex and dynamic process determined by different factors including material, environment, design, and some other unobserved variables. Accurate predictions of pavement condition can help maximize the use of available resources for pavement management agencies through better coordinated preservation and maintenance activities. This paper uses deep neural networks such as the convolutional neural network (CNN) and the long short-term
memory (LSTM) to model the pavement deterioration process. In this paper, pavement condition data and maintenance and rehabilitation history collected by the Texas Department of Transportation over the past 18 years were used. Twenty-one flexible pavement condition indicators, including cracking, rutting, raveling, and roughness, collected from more than 100,000 pavement sections were included in the proposed models. Promising preliminary results were obtained. Case study results show that the proposed CNN model outperforms standard machine learning models in predicting pavement condition values.
Read the full abstract ▾
Machine Learning
RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
Sadia Asif and Mohammad Mohammadi Amiri
Fine-tuning safety-aligned
language models for downstream tasks often leads to substantial degradation of refusal behavior, making models vulnerable to
adversarial misuse. While prior work has shown that safety-relevant features are encoded in structured representations within the model's activation space, how these representations change during fine-tuning and why
alignment degrades remains poorly understood. In this work, we investigate the representation-level mechanisms underlying
alignment degradation. Our analysis shows that standard fine-tuning induces systematic drift in safety-relevant representations, distorts their
geometric structure, and introduces interference between task optimization and safety features. These effects collectively lead to increased harmful compliance. Motivated by these findings, we introduce REFUSALGUARD, a representation-level fine-tuning framework that preserves safety-relevant structure during model adaptation. Our approach constrains updates in hidden representation space, ensuring that safety-mediating components remain stable while allowing task-specific learning in complementary directions. We evaluate REFUSALGUARD across multiple model families, including LLaMA, Gemma, and Qwen, on
adversarial safety
benchmarks such as AdvBench, DirectHarm4, and JailbreakBench, as well as downstream utility tasks. Our approach achieves attack success rates comparable to base safety-aligned models while maintaining competitive task performance, significantly outperforming baselines.
Read the full abstract ▾
Machine Learning
Kyle Lee, Corentin Delacour, Kevin Callahan-Coray et al. (+5 more)
Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each
token requires reading all $n_k$ key and value vectors from KV cache. We present
Stochastic Additive No-mulT
Attention (SANTA), a method that sparsifies value-cache access by sampling $S \ll n_k$ indices from the post-softmax distribution and aggregates only those value rows. This yields an unbiased
estimator of the post-softmax value aggregation while replacing value-stage multiply-accumulates with gather-and-add. We introduce stratified and systematic sampling to design variance-reduced, GPU-friendly variants. Evaluated on Llama-3.1-8B-Instruct at 32k-
token contexts, S$^2$ANTA matches baseline accuracy while achieving up to $1.5\times$ decode-step
attention-kernel speedup over FlashInfer and FlashDecoding on an NVIDIA RTX 6000 Ada. In batched long-context generation, these kernel gains translate to up to $1.25\times$ end-to-end decode-latency speedup. Finally, we propose Bernoulli $qK^\mathsf{T}$ sampling as a complementary technique to sparsify the score stage, reducing key-feature access through
stochastic ternary queries. Both methods are complementary to upstream quantization, low-rank projection, KV-cache compression, and KV-cache selection methods. Together, they point toward sparse, multiplier-free, and energy-efficient inference. We open-source our kernels at: https://github.com/OPUSLab/SANTA.git
Read the full abstract ▾
All authors Kyle Lee, Corentin Delacour, Kevin Callahan-Coray, Kyle Jiang, Can Yaras, Samet Oymak, Tathagata Srimani, Kerem Y. Camsari
Machine Learning · Cross-listed from stat.ML
Extrapolation in Statistical Learning with Extreme Value Theory
Sebastian Engelke, Nicola Gnecco and Anne Sabourin
Extreme value theory provides rigorous theory and statistical tools for extrapolation in machine learning, particularly in settings where traditional methods struggle due to data scarcity in the tails. A broad range of tasks benefit from these advances, including regression and classification beyond the training data, extreme
quantile regression, supervised and unsupervised dimension reduction, generative artificial intelligence and anomaly detection. This review synthesizes recent developments in these fields at the intersection of statistical learning and extreme value theory, with a focus on principled methods based on asymptotically motivated representations of the tail of univariate and multivariate distributions. We consider different theoretical frameworks for both asymptotically dependent and independent data and discuss how they translate into efficient statistical methods for extrapolation to extreme regions. By addressing both theoretical and practical aspects, we offer a comprehensive overview of the state-of-the-art in this quickly evolving field, and identify promising directions for future research.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Adaptive Estimation and Inference in Semi-parametric Heterogeneous Clustered Multitask Learning via Neyman Orthogonality
Hanxiao Chen and Debarghya Mukherjee
We study clustered multitask learning in a semiparametric setting where tasks share a latent cluster structure in their target parameters but exhibit heterogeneous, potentially infinite-dimensional nuisance components. Such heterogeneity poses a major challenge for existing multitask learning methods, which typically rely on aligned feature spaces or homogeneous task structures. To address this challenge, we propose an adaptive fused orthogonal
estimator that integrates Neyman-orthogonal losses with data-driven pairwise fusion penalties. Our framework leverages task-specific pilot estimates to calibrate the fusion penalties and combines adaptive aggregation with orthogonalization to mitigate the impact of nuisance-parameter estimation error. Theoretically, we show that the proposed
estimator achieves exact recovery of the latent clustering with high probability and attains pooled parametric convergence rates proportional to cluster size. Moreover, we establish asymptotic normality and show that, asymptotically, our
estimator matches the performance of an oracle procedure that knows the true clustering in advance. Empirically, we show that the proposed method consistently outperforms strong baselines in various simulation setups. A real-world application to U.S. residential energy consumption demonstrates the effectiveness of our approach in uncovering meaningful regional clustering in electricity price elasticity, showcasing the efficacy of our method.
Read the full abstract ▾
Machine Learning
How Label Imbalance Shapes Geometry: A General Spectral Analysis of Multi-Label Neural Collapse
Xiaoxuan Ma, Yixuan Yang, Song Li et al. (+1 more)
This work investigates the phenomenon of Neural Collapse (NC) in multi-label classification, extending its conceptual framework from multi-class learning to general correlated and imbalanced multi-label settings. Although recent studies have identified a ”tag-wise averaging” structure for multi-label features, this view relies on implicit assumptions of label balance and combinatorial
symmetry. Consequently, it fails to account for the geometrical distortions caused by intrinsic label correlations and data imbalance, which are common in practice. We resolve the multiplicity-one imbalance
conjecture raised by Li et al. (2024), showing that higher-multiplicity prototypes obey a class-frequency-weighted synthesis rule rather than uniform averaging. To address this, we propose a rigorous spectral-control framework to analyze the terminal phase of multi-label learning under general imbalanced conditions. We introduce the label covariance spectrum $\kappa_m$, a scalar controlling the distribution-dependent lower-bound geometry, derived from the second-order moment
matrix of the label distribution. Contrary to the averaging perspective, our analysis reveals that the centered label covariance spectrum controls the stability of terminal geometry by quantifying the weakest centered inter-class contrast directions. We prove that the classical Tag-wise Averaging emerges only as a special case under perfect orthogonality. Numerical experiments on synthetic distributions validate our theoretical bounds. This work resolves the scaled-average aspect of the imbalance
conjecture and establishes a unifying theoretical framework that extends Neural Collapse to complex, imbalanced multi-label settings.
Read the full abstract ▾
All authors Xiaoxuan Ma, Yixuan Yang, Song Li, Xiangyun Hui
Machine Learning · Cross-listed from cs.CV
Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts
Hongkun Pan, Yuwei Wu, Wanyi Hong et al. (+8 more)
Multimodal large
language models (MLLMs) have shown considerable potential in chart understanding and
reasoning tasks. However, they still struggle with high information density (HID) charts characterized by multiple subplots, legends, and dense annotations due to three major challenges: (1) limited fine-grained perception results in the omission of critical visual cues; (2) redundant or noisy visual information undermines the performance of
multimodal reasoning; (3) lack of adaptive deep
reasoning relative to the amount of visual information. To tackle these challenges, we present a novel focus-driven fine-grained chart
reasoning model, Chart-FR1, to improve perception, focusing efficiency, and adaptive deep
reasoning on HID charts. Specifically, we propose Focus-CoT, a visual focusing chain-of-thought that enhances fine-grained perception by explicitly linking
reasoning steps to key visual cues, such as local image regions and OCR signals. Building on this, we introduce Focus-GRPO, a focus-driven reinforcement learning
algorithm with an information-efficiency reward that compresses redundant visual information for efficient focusing, and an adaptive KL penalty mechanism that enables flexible control over
reasoning depth as more visual cues are discovered. Furthermore, to fill the gap in
benchmarks for HID charts, we build HID-Chart, a challenging
benchmark with an information-density metric designed to evaluate fine-grained chart
reasoning capabilities. Extensive experiments on multiple chart
benchmarks demonstrate that Chart-FR1 outperforms state-of-the-art MLLMs in chart understanding and
reasoning. Code is available at https://github.com/phkhub/Chart-FR1.
Read the full abstract ▾
All authors Hongkun Pan, Yuwei Wu, Wanyi Hong, Shenghui Hu, Qitong Yan, Yi Yang, Rufei Han, Changju Zhou, Minfeng Zhu, Dongming Han, Wei Chen
Machine Learning
Leveraging Data Symmetries to Select an Optimal Subset of Training Data under Label Noise
Kumar Shubham, Pavan Karjol, Kiran M K et al. (+1 more)
The performance of machine learning models often relies on large labeled
datasets; however, data collected from diverse sources can contain label noise. Recent work has shown that, in noisy settings, there may exist a subset of the training data on which models can achieve performance comparable to training on a noise-free
dataset. A widely used method for identifying such subsets is cutstats, which employs k-nearest neighbors (k-NN) to detect low-noise samples. However, its performance on high-dimensional data remains largely unexplored. In this work, we formally establish that the performance of a classifier trained on a subset of a noisy
dataset selected via cutstats is influenced by the accuracy of k-NN. We further demonstrate that, in noisy environments, exploiting data invariance and knowledge of underlying
symmetries can significantly enhance the performance of k-NN, bringing it closer to the Bayes optimal classifier even in high-dimensional regimes. Finally, we show that for real-world scenarios, where information about the underlying invariance is only partially known, learnt invariant representations can still facilitate the identification of near-optimal subsets.
Read the full abstract ▾
All authors Kumar Shubham, Pavan Karjol, Kiran M K, Prathosh AP
Machine Learning
Robust Conditional Conformal Prediction via Branched Normalizing Flow
Rui Xu, Xingyuan Chen, Wenxing Huang et al. (+4 more)
Conformal prediction (CP) constructs prediction sets with marginal coverage guarantees under the assumption that the calibration and test distributions are identical. However, under distribution shift, existing approaches primarily align marginal conformal score distributions, which is sufficient to preserve marginal coverage but does not control the conditional coverage error at individual test inputs. As a consequence, CP can remain unreliable in regions where the conditional score distributions are mismatched. In this work, we bound the conditional invalidity of CP under distribution shift in terms of the Wasserstein distance between the calibration and test distributions. This result highlights the role of invertible transport in mitigating conditional coverage degradation. Motivated by this insight, we introduce Branched Normalizing Flow (BNF), a two-branch architecture that normalizes a test input to the calibration distribution and transforms the prediction set of the normalized input back to the test distribution while preserving conditional guarantees. Empirically, BNF consistently improves conditional coverage robustness on nine
datasets across a wide range of confidence levels.
Read the full abstract ▾
All authors Rui Xu, Xingyuan Chen, Wenxing Huang, Minxuan Huang, Weiyan Chen, Sihong Xie, Hui Xiong