Machine Learning
SODA: Semi On-Policy Black-Box Distillation for Large Language Models
Xiwen Chen, Jingjing Wang, Wenhui Zhu et al. (+7 more)
Black-box knowledge
distillation for large
language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge
distillation) struggle to correct the student's inherent errors. Fully on-policy methods (e.g., Generative
Adversarial Distillation) solve this via
adversarial training but introduce well-known training instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy
Distillation with
Alignment), a highly efficient alternative motivated by the inherent capability gap between frontier teachers and much smaller base models. Because a compact student model's natural, zero-shot responses are almost strictly inferior to the powerful teacher's targets, we can construct a highly effective contrastive signal simply by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This demonstrates that exposing the small student to its own static inferior behaviors is sufficient for high-quality distribution
alignment, eliminating the need for costly dynamic rollouts and fragile
adversarial balancing. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms the state-of-the-art methods on 15 out of 16
benchmark results. More importantly, it achieves this superior
distillation quality while training 10 times faster, consuming 27% less peak GPU
memory, and completely eliminating
adversarial instability.
Read the full abstract ▾
All authors Xiwen Chen, Jingjing Wang, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Yueyue Deng, Hejian Sang, Zhipeng Wang, Alborz Geramifard, Feng Luo
Machine Learning
Where to Steer: Input-Dependent Layer Selection for Steering Improves LLM Alignment
Soham Gadgil, Chris Lin and Su-In Lee
Steering vectors have emerged as a lightweight and effective approach for aligning large
language models (LLMs) at inference time, enabling modulation of model behaviors by shifting intermediate representations toward a target behavior. Many existing steering methods apply interventions at globally fixed locations in an LLM, effectively treating the optimal intervention location as input-invariant. However, representations relevant to a target behavior can be encoded at different LLM layers depending on the input. Theoretically, we show that different inputs may require steering at different layers to achieve
alignment with a given behavior. Empirically, we observe that the most effective layer for steering varies substantially between inputs. Motivated by these observations, we introduce Where to Steer (W2S), a general framework that adaptively selects intervention layers conditioned on the inputs by learning a mapping from input representations to steering layers. Across multiple LLMs and
alignment behaviors, W2S consistently outperforms fixed-layer baselines in both in-distribution and out-of-distribution settings. Our findings suggest that input-dependent layer selection is a key design dimension complementary to methods for steering vector extraction.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
SecureAFL: Secure Asynchronous Federated Learning
Anjun Gao, Feng Wang, Zhenglin Wan et al. (+3 more)
Federated learning (FL) enables multiple clients to collaboratively train a global machine learning model via a server without sharing their private training data. In traditional FL, the system follows a synchronous approach, where the server waits for model updates from numerous clients before aggregating them to update the global model. However, synchronous FL is hindered by the straggler problem. To address this, the asynchronous FL architecture allows the server to update the global model immediately upon receiving any client's local model update. Despite its advantages, the decentralized nature of asynchronous FL makes it vulnerable to poisoning attacks. Several defenses tailored for asynchronous FL have been proposed, but these mechanisms remain susceptible to advanced attacks or rely on unrealistic server assumptions. In this paper, we introduce SecureAFL, an innovative framework designed to secure asynchronous FL against poisoning attacks. SecureAFL improves the robustness of asynchronous FL by detecting and discarding anomalous updates while estimating the contributions of missing clients. Additionally, it utilizes Byzantine-robust aggregation techniques, such as coordinate-wise median, to integrate the received and estimated updates. Extensive experiments on various real-world
datasets demonstrate the effectiveness of SecureAFL.
Read the full abstract ▾
All authors Anjun Gao, Feng Wang, Zhenglin Wan, Yueyang Quan, Zhuqing Liu, Minghong Fang
Machine Learning
A Bayesian Information-Theoretic Approach to Data Attribution
Dharmesh Tailor, Nicolò Felicioni and Kamil Ciosek
Training Data Attribution (TDA) seeks to trace model predictions back to influential training examples, enhancing interpretability and safety. We formulate TDA as a
Bayesian information-theoretic problem: subsets are scored by the information loss they induce - the
entropy increase at a query when removed. This criterion credits examples for resolving predictive uncertainty rather than label noise. To scale to modern networks, we approximate information loss using a Gaussian Process surrogate built from tangent features. We show this aligns with classical influence scores for single-example attribution while promoting diversity for subsets. For even larger-scale
retrieval, we relax to an information-gain objective and add a variance correction for scalable attribution in vector databases. Experiments show competitive performance on counterfactual sensitivity, ground-truth
retrieval and coreset selection, showing that our method scales to modern architectures while bridging principled measures with practice.
Read the full abstract ▾
Machine Learning
Understanding When Poisson Log-Normal Models Outperform Penalized Poisson Regression for Microbiome Count Data
Daniel Agyapong, Julien Chiquet, Jane Marks et al. (+1 more)
Multivariate count models are often justified by their ability to capture latent dependence, but researchers receive little guidance on when this added structure improves on simpler penalized marginal Poisson regression. We study this question using real microbiome data under a unified held-out evaluation framework. For count prediction, we compare PLN and GLMNet(Poisson) on 20
datasets spanning 32 to 18,270 samples and 24 to 257 taxa, using held-out Poisson deviance under leave-one-taxon-out prediction with 3-fold sample cross-validation rather than synthetic or in-sample criteria. For network inference, we compare PLNNetwork and GLMNet(Poisson) neighborhood selection on five publicly available
datasets with experimentally validated microbial interaction truth. PLN outperforms GLMNet(Poisson) on most count-prediction
datasets, with gains up to 38 percent. The primary predictor of the winner is the sample-to-taxon ratio, with mean absolute correlation as the strongest secondary signal and overdispersion as an additional predictor. PLNNetwork performs best on broad undirected interaction
benchmarks, whereas GLMNet(Poisson) is better aligned with local or directional effects. Taken together, these results provide guidance for choosing between latent multivariate count models and penalized Poisson regression in biological count prediction and interaction recovery.
Read the full abstract ▾
All authors Daniel Agyapong, Julien Chiquet, Jane Marks, Toby Dylan Hocking
Machine Learning
Collapse-Free Prototype Readout Layer for Transformer Encoders
Giansalvo Cirrincione and Rahul Ranjeev Kumar
DDCL-
Attention is a prototype-based readout layer for
transformer encoders that replaces simple pooling methods, such as mean pooling or class
tokens, with a learned compression mechanism. It uses a small set of global prototype vectors and assigns
tokens to them through soft probabilistic matching, producing compact
token summaries at linear complexity in sequence length. The method offers three main advantages. First, it avoids prototype collapse through an exact decomposition of the training loss into a reconstruction term and a diversity term, ensuring that prototypes remain distinct. Second, its joint training with the encoder is shown to be stable under a practical timescale condition, using Tikhonov's singular perturbation theory and explicit learning-rate constraints. Third, the same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor. Experiments on four
datasets confirm the theoretical predictions: the loss decomposition holds exactly, prototype separation grows as expected when the stability condition is met, and the codebook reaches full utilization, outperforming standard hard vector quantization. An additional study on orbital debris classification shows that the method also applies beyond standard NLP and
vision tasks, including scientific tabular data.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CR
Explainability-Guided Adversarial Attacks on Transformer-Based Malware Detectors Using Control Flow Graphs
Andrew Wheeler, Kshitiz Aryal and Maanak Gupta
Transformer-based malware detection systems operating on graph modalities such as control flow graphs (CFGs) achieve strong performance by modeling structural relationships in program behavior. However, their robustness to
adversarial evasion attacks remains underexplored. This paper examines the vulnerability of a RoBERTa-based malware detector that linearizes CFGs into sequences of function calls, a design choice that enables
transformer modeling but may introduce
token-level sensitivities and ordering artifacts exploitable by adversaries. By evaluating evasion strategies within this graph-to-sequence framework, we provide insight into the practical robustness of
transformer-based malware detectors beyond aggregate detection accuracy. This paper proposes a white-box
adversarial evasion attack that leverages explainability mechanisms to identify and perturb most influential graph components. Using
token- and word-level attributions derived from integrated gradients, the attack iteratively replaces positively attributed function calls with synthetic external imports, producing
adversarial CFG representations without altering overall program structure. Experimental evaluation on small- and large-scale Windows Portable Executable (PE)
datasets demonstrates that the proposed method can reliably induce misclassification, even against models trained to high accuracy. Our results highlight that explainability tools, while valuable for interpretability, can also expose critical attack surfaces in
transformer-based malware detectors.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ME
New insights into Elo algorithm for practitioners and statisticians
Leszek Szczecinski
This work reconciles two perspectives on the Elo ranking that coexist in the literature: the practitioner's view as a heuristic feedback rule, and the statistician's view as online maximum likelihood estimation via
stochastic gradient ascent. Both perspectives coincide exactly in the binary case (iff the expected score is the logistic function). However, estimation noise forces a principled decoupling between the model used for ranking and the model used for prediction: the effective scale and home-field advantage parameter must be adjusted to account for the noise. We provide both closed-form corrections and a data-driven identification procedure. For multilevel outcomes, an exact relationship exists when outcome scores are uniformly spaced, but approximations are preferred in general: they account for estimation noise and better fit the data. The decoupled approach substantially outperforms the conventional one that reuses the ranking model for prediction, and serves as a diagnostic of convergence status. Applied to six years of FIFA men's ranking, we find that the ranking had not converged for the vast majority of national teams. The paper is written in a semi-tutorial style accessible to practitioners, with all key results accompanied by closed-form expressions and numerical examples.
Read the full abstract ▾
Machine Learning
k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS
Jonas De Schouwer, Haitz Sáez de Ocáriz Borde and Xiaowen Dong
Graph
transformers have shown promise in overcoming limitations of traditional graph neural networks, such as oversquashing and difficulties in modeling long-range dependencies. However, their application to large-scale graphs is hindered by the quadratic
memory and computational complexity of the all-to-all
attention mechanism. Although alternatives such as linearized
attention and restricted
attention patterns have been proposed, these often degrade performance or limit expressive power. To better balance efficiency and effectiveness, we introduce k-Maximum Inner Product (k-MIP)
attention for graph
transformers. k-MIP
attention selects the most relevant key nodes per query via a top-k operation, yielding a sparse yet flexible
attention pattern. Combined with an
attention score computation based on symbolic
matrices, this results in linear
memory complexity and practical speedups of up to an order of magnitude compared to all-to-all
attention, enabling the processing of graphs with over 500k nodes on a single A100 GPU. We provide a theoretical analysis of expressive power, showing that k-MIP
attention does not compromise the expressiveness of graph
transformers: specifically, we prove that k-MIP
transformers can approximate any full-
attention transformer to arbitrary precision. In addition, we analyze the expressive power of the GraphGPS framework, in which we integrate our
attention mechanism, and establish an upper bound on its graph distinguishing capability in terms of the S-SEG-WL test. Finally, we validate our approach on the Long Range Graph
Benchmark, the City-Networks
benchmark, and two custom large-scale inductive point cloud
datasets, consistently ranking among the top-performing scalable graph
transformers.
Read the full abstract ▾
Machine Learning
Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus
Dipkumar Patel
Multi-
agent LLM committees replicate the same model under different role
prompts and aggregate outputs by majority vote, implicitly assuming that
agents contribute complementary evidence. We embed each
agent's chain-of-thought rationale and measure pairwise similarity: across 100 GSM8K questions with three Qwen2.5-14B
agents, mean cosine similarity is 0.888 and effective rank is 2.17 out of 3.0, a failure mode we term representational collapse. DALC, a training-free consensus protocol that computes diversity weights from
embedding geometry, reaches 87% on GSM8K versus 84% for self-consistency at 26% lower
token cost. Ablation experiments reveal 1-3 point per-protocol run-to-run variance, confirm that hint sharing contributes more than diversity weighting alone, and show that encoder choice strongly modulates collapse severity (cosine 0.908 with mxbai versus 0.888 with nomic) and downstream accuracy. The more robust finding is that collapse is measurable, worsens on harder tasks, and that the choice of
embedding proxy is a first-order design decision for any latent communication protocol.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Hiroaki Aizawa and Yuki Igaue
Transformers are strong baselines in both
vision and
language because self-
attention captures long-range dependencies across
tokens. However, the cost of self-
attention grows quadratically with the number of
tokens. Patch
pruning mitigates this cost by estimating per-patch importance and removing redundant patches. To identify informative patches for
pruning, we introduce a criterion based on the Shannon
entropy of the
attention distribution. Low-
entropy patches, which receive selective and concentrated
attention, are kept as important, while high-
entropy patches with
attention spread across many locations are treated as redundant. We also extend the criterion from Shannon to Rényi
entropy, which emphasizes sharp
attention peaks and supports
pruning strategies that adapt to task needs and computational limits. In experiments on fine-grained image recognition, where patch selection is critical, our method reduced computation while preserving accuracy. Moreover, adjusting the
pruning policy through the Rényi
entropy measure yields further gains and improves the trade-off between accuracy and computation.
Read the full abstract ▾
Machine Learning
Automated Conjecture Resolution with Formal Verification
Haocheng Ju, Guoxiong Gao, Jiedong Jiang et al. (+13 more)
Recent advances in large
language models have significantly improved their ability to perform mathematical
reasoning, extending from elementary problem solving to increasingly capable performance on research-level problems. However, reliably solving and verifying such problems remains challenging due to the inherent ambiguity of natural
language reasoning. In this paper, we propose an automated framework that integrates natural
language reasoning with formal verification to tackle research-level mathematical problems. Our framework consists of two components: an informal
reasoning agent, Rethlas, and a formal verification
agent, Archon. Rethlas combines
reasoning primitives with our
theorem search engine, Matlas, to explore solution strategies and construct candidate proofs. Archon, equipped with LeanSearch, translates informal arguments into formalized Lean 4 projects through task decomposition, iterative refinement, and automated proof synthesis, ensuring machine-checkable correctness. Using this framework, we resolve an open problem in commutative
algebra and formally verify the resulting proof in Lean 4 with essentially no human involvement. Additional case studies illustrate the capabilities of Rethlas in informal mathematical
reasoning and discovery, as well as the ability of Archon to formalize research-level proofs in Lean 4. Our experiments demonstrate that strong
theorem retrieval tools enable the discovery and application of cross-domain mathematical techniques, while the formal
agent can autonomously fill nontrivial gaps in informal arguments. More broadly, our work illustrates a promising paradigm for mathematical research in which informal and formal
reasoning systems, equipped with
theorem retrieval tools, operate in tandem to produce verifiable results, reduce human effort, and support human-AI collaborative mathematical research.
Read the full abstract ▾
All authors Haocheng Ju, Guoxiong Gao, Jiedong Jiang, Bin Wu, Zeming Sun, Shurui Liu, Leheng Chen, Yutong Wang, Yuefeng Wang, Zichen Wang, Wanyi He, Peihao Wu, Liang Xiao, Ruochuan Liu, Bryan Dai, Bin Dong
Machine Learning · Cross-listed from cs.DS
Dimension-Free Sinkhorn for Optimal Transport: Bypassing Extremal Costs
Kun He
The Sinkhorn–Knopp (SK)
algorithm is a cornerstone method for
matrix scaling and entropically regularized optimal transport (EOT). Despite its empirical efficiency, existing bounds for achieving marginal accuracy $\varepsilon$ deteriorate severely in the presence of extreme entries, bottlenecked either by the global maximum regularized cost $\eta\|C\|_\infty$ (where $\eta$ is the regularization parameter and $C$ the cost
matrix) or the
matrix's minimum-to-maximum entry ratio $\nu$. These bounds may be overly sensitive to rare extreme entries that do not dominate the overall
matrix structure. We identify a broad structural regime in which the dependence on the magnitude of global extreme entries can be circumvented. For EOT, we introduce well-boundedness, a local mass property that captures the row- and column-wise bulk structure of the underlying
matrix while maintaining robustness against scattered entrywise outliers. We prove that, governed by this structural condition, SK converges to the target transport plan for a problem of dimension $n$ in $O(\log n + \log(1/\varepsilon))$ iterations. This complexity bound is independent of the maximum regularized cost $\eta\|C\|_\infty$. A computationally negligible pre-scaling step eliminates this dimensional dependence, reducing the iteration complexity to a strictly dimension-free $O(\log(1/\varepsilon))$. Beyond EOT, we establish a sharp phase transition for general $(\boldsymbol{u},\boldsymbol{v})$-scaling governed by a critical
matrix density threshold. We prove that when a
matrix's density exceeds this threshold, the iteration complexity is strictly independent of $\nu$. Conversely, when the density falls below this threshold, the dependence on $\nu$ becomes unavoidable; in this sub-critical regime, we construct instances where SK requires $\Omega(n/\varepsilon)$ iterations.
Read the full abstract ▾
Machine Learning
CountsDiff: A Diffusion Model on the Natural Numbers for Generation and Imputation of Count-Based Data
Renzo G. Soatto, Anders Hoel, Greycen Ren et al. (+5 more)
Diffusion models have excelled at generative tasks for both continuous and
token-based domains, but their application to discrete ordinal data remains underdeveloped. We present CountsDiff, a
diffusion framework designed to model distributions on the natural numbers. CountsDiff extends the Blackout
diffusion framework by simplifying its formulation through a direct parameterization in terms of a survival probability schedule and an explicit loss weighting. This introduces flexibility through design parameters with direct analogues in existing
diffusion modeling frameworks. Beyond this reparameterization, CountsDiff introduces features from modern
diffusion models, previously absent in counts-based domains, including continuous-time training, classifier-free guidance, and churn/remasking reverse dynamics that allow non-monotone reverse trajectories. We propose an initial instantiation of CountsDiff and validate it on natural image
datasets (CIFAR-10, CelebA), exploring the effects of the introduced design parameters in a complex, well-studied, and interpretable data domain. We then highlight biological count assays as a natural use case, evaluating CountsDiff on single-
cell RNA-seq imputation in fetal and heart
cell atlases. Remarkably, we find that even this simple instantiation matches or surpasses the performance of a state-of-the-art discrete generative model and leading scRNA-seq imputation methods, while leaving substantial headroom for further gains through optimized design choices in future work.
Read the full abstract ▾
All authors Renzo G. Soatto, Anders Hoel, Greycen Ren, Shorna Alam, Stephen Bates, Nikolaos P. Daskalakis, Caroline Uhler, Maria Skoularidou
Machine Learning · Cross-listed from stat.ML
Debiased Machine Learning for Conformal Prediction of Counterfactual Outcomes Under Runtime Confounding
Keith Barnatchez, Kevin P. Josey, Rachel C. Nethery et al. (+1 more)
Data-driven decision making frequently relies on predicting counterfactual outcomes. In practice, researchers commonly train counterfactual prediction models on a source
dataset to inform decisions on a possibly separate target population. Conformal prediction has arisen as a popular method for producing assumption-lean prediction intervals for counterfactual outcomes that would arise under different treatment decisions in the target population of interest. However, existing methods require that every confounding factor of the treatment-outcome relationship used for training on the source data is additionally measured in the target population, risking miscoverage if important confounders are unmeasured in the target population. In this paper, we introduce a computationally efficient debiased machine learning framework that allows for valid prediction intervals when only a subset of confounders is measured in the target population, a common challenge referred to as runtime confounding. Grounded in semiparametric efficiency theory, we show the resulting prediction intervals achieve desired coverage rates with faster convergence compared to standard methods. Through numerous synthetic and semi-synthetic experiments, we demonstrate the utility of our proposed method.
Read the full abstract ▾
All authors Keith Barnatchez, Kevin P. Josey, Rachel C. Nethery, Giovanni Parmigiani
Machine Learning · Cross-listed from cs.AI
RL-Driven Sustainable Land-Use Allocation for the Lake Malawi Basin
Ying Yao
Unsustainable land-use practices in ecologically sensitive regions threaten biodiversity, water resources, and the livelihoods of millions. This paper presents a deep reinforcement learning (RL) framework for optimizing land-use allocation in the Lake Malawi Basin to maximize total ecosystem service value (ESV). Drawing on the benefit transfer methodology of Costanza et al., we assign biome-specific ESV coefficients – locally anchored to a Malawi wetland valuation – to nine land-cover classes derived from Sentinel-2 imagery. The RL environment models a 50x50
cell grid at 500m resolution, where a Proximal Policy Optimization (PPO)
agent with action masking iteratively transfers land-use pixels between modifiable classes. The reward function combines per-
cell ecological value with spatial coherence objectives: contiguity bonuses for ecologically connected land-use patches (forest, cropland, built area etc.) and buffer zone penalties for high-impact development adjacent to water bodies. We evaluate the framework across three scenarios: (i) pure ESV maximization, (ii) ESV with spatial reward shaping, and (iii) a regenerative agriculture policy scenario. Results demonstrate that the
agent effectively learns to increase total ESV; that spatial reward shaping successfully steers allocations toward ecologically sound patterns, including homogeneous land-use clustering and slight forest consolidation near water bodies; and that the framework responds meaningfully to policy parameter changes, establishing its utility as a scenario-analysis tool for environmental planning.
Read the full abstract ▾
Machine Learning
Automated Attention Pattern Discovery at Scale in Large Language Models
Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes et al. (+2 more)
Large
language models have found success by scaling up capabilities to work in general settings. The same can unfortunately not be said for interpretability methods. The current trend in mechanistic interpretability is to provide precise explanations of specific behaviors in controlled settings. These often do not generalize, or are too resource intensive for larger studies. In this work we propose to study repeated behaviors in large
language models by mining completion scenarios in Java code
datasets, through exploiting the structured nature of code. We collect the
attention patterns generated in the
attention heads to demonstrate that they are scalable signals for global interpretability of model components. We show that
vision models offer a promising direction for analyzing
attention patterns at scale. To demonstrate this, we introduce the
Attention Pattern - Masked Autoencoder(AP-MAE), a
vision transformer-based model that efficiently reconstructs masked
attention patterns. Experiments on StarCoder2 show that AP-MAE (i) reconstructs masked
attention patterns with high accuracy, (ii) generalizes across unseen models with minimal degradation, (iii) reveals recurring patterns across inferences, (iv) predicts whether a generation will be correct without access to ground truth, with accuracies ranging from 55% to 70% depending on the task, and (v) enables targeted interventions that increase accuracy by 13.6% when applied selectively, but cause collapse when applied excessively. These results establish
attention patterns as a scalable signal for interpretability and demonstrate that AP-MAE provides a transferable foundation for both analysis and intervention in large
language models. Beyond its standalone value, AP-MAE also serves as a selection procedure to guide fine-grained mechanistic approaches. We release code and models to support future work in large-scale interpretability.
Read the full abstract ▾
All authors Jonathan Katzy, Razvan-Mihai Popescu, Erik Mekkes, Arie van Deursen, Maliheh Izadi
Machine Learning · Cross-listed from cs.CR
Spatiotemporal-Aware Bit-Flip Injection on DNN-based Advanced Driver Assistance Systems (extended version)
Taibiao Zhao, Xiang Zhang, Mingxuan Sun et al. (+2 more)
Modern advanced driver assistance systems (ADAS) rely on deep neural networks (DNNs) for perception and planning. Since DNNs' parameters reside in DRAM during inference, bit flips caused by
cosmic radiation or low-voltage operation may corrupt DNN computations, distort driving decisions, and lead to real-world incidents. This paper presents a SpatioTemporal-Aware Fault Injection (STAFI) framework to locate critical fault sites in DNNs for ADAS efficiently. Spatially, we propose a Progressive Metric-guided Bit Search (PMBS) that efficiently identifies critical network weight bits whose corruption causes the largest deviations in driving behavior (e.g., unintended acceleration or steering). Furthermore, we develop a Critical Fault Time Identification (CFTI) mechanism that determines when to trigger these faults, taking into account the context of real-time systems and environmental states, to maximize the safety impact. Experiments on DNNs for a production ADAS demonstrate that STAFI uncovers 29.56x more hazard-inducing critical faults than the strongest baseline.
Read the full abstract ▾
All authors Taibiao Zhao, Xiang Zhang, Mingxuan Sun, Ruyi Ding, Xugui Zhou
Machine Learning · Cross-listed from stat.ML
The Generalised Kernel Covariance Measure
Luca Bergen, Dino Sejdinovic and Vanessa Didelez
We consider the problem of conditional independence (CI) testing and adopt a kernel-based approach. Kernel-based CI tests embed variables in reproducing kernel
Hilbert spaces, regress their
embeddings on the conditioning variables, and test the resulting residuals for marginal independence. This approach yields tests that are sensitive to a broad range of conditional dependencies. Existing methods, however, rely heavily on kernel ridge regression, which is computationally expensive when properly tuned and yields poorly calibrated tests when left untuned, which limits their practical usefulness. We propose the Generalised Kernel Covariance Measure (GKCM), a regression-model-agnostic kernel-based CI test that accommodates a broad class of regression
estimators. Building on the Generalised Hilbertian Covariance Measure framework (Lundborg et al., 2022), we characterise conditions under which GKCM satisfies uniform asymptotic level guarantees. In simulations, GKCM paired with tree-based regression models frequently outperforms state-of-the-art CI tests across a diverse range of data-generating processes, achieving better type I error control and competitive or superior power.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Learning Superpixel Ensemble and Hierarchy Graphs for Melanoma Detection
Asmaa M. Elwer, Muhammad A. Rushdi and Mahmoud H. Annaby
Graph signal processing (GSP) is becoming a major tool in biomedical signal and image analysis. In most GSP techniques, graph structures and edge weights have been typically set via statistical and computational methods. More recently, graph structure learning methods offered more reliable and flexible data representations. In this work, we introduce a graph learning approach for melanoma detection in dermoscopic images based on two graph-theoretic representations: superpixel ensemble graphs (SEG) and superpixel hierarchy graphs (SHG). For these two types of graphs, superpixel maps of a skin lesion image are respectively generated at multiple levels without and with parentchild constraints among superpixels at adjacent levels, where each level corresponds to a subgraph with a different number of nodes (20, 40, 60, 80, or 100 nodes). Two edge weight assignment techniques are explored: handcrafted Gaussian weights and learned weights based on optimization methods. The graph nodal signals are assigned based on texture,
geometric, and color superpixel features. In addition, the effect of graph edge thresholding is investigated by applying different thresholds (25%, 50%, and 75%) to prune the weakest edges and analyze the impact of
pruning on the melanoma detection performance. Experimental evaluation of the proposed method is performed with different classifiers trained and tested on the publicly available ISIC2017
dataset. Data augmentation is applied to alleviate class imbalance by adding more melanoma images from the ISIC archive. The results show that learned superpixel ensemble graphs with textural nodal signals give the highest performance reaching an accuracy of 99.00% and an AUC of 99.59%.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
LightThinker++: From Reasoning Compression to Memory Management
Yuqi Zhu, Jintian Zhang, Zhenjie Wan et al. (+7 more)
Large
language models (LLMs) excel at complex
reasoning, yet their efficiency is limited by the surging cognitive overhead of long thought traces. In this paper, we propose LightThinker, a method that enables LLMs to dynamically compress intermediate thoughts into compact semantic representations. However, static compression often struggles with complex
reasoning where the irreversible loss of intermediate details can lead to logical bottlenecks. To address this, we evolve the framework into LightThinker++, introducing Explicit Adaptive
Memory Management. This paradigm shifts to behavioral-level management by incorporating explicit
memory primitives, supported by a specialized trajectory synthesis pipeline to train purposeful
memory scheduling. Extensive experiments demonstrate the framework's versatility across three dimensions. (1) LightThinker reduces peak
token usage by 70% and inference time by 26% with minimal accuracy loss. (2) In standard
reasoning, LightThinker++ slashes peak
token usage by 69.9% while yielding a +2.42% accuracy gain under the same context budget for maximum performance. (3) Most notably, in long-horizon agentic tasks, it maintains a stable footprint beyond 80 rounds (a 60%-70% reduction), achieving an average performance gain of 14.8% across different complex scenarios. Overall, our work provides a scalable direction for sustaining deep LLM
reasoning over extended horizons with minimal overhead.
Read the full abstract ▾
All authors Yuqi Zhu, Jintian Zhang, Zhenjie Wan, Yujie Luo, Shuofei Qiao, Zhengke Gui, Da Zheng, Lei Liang, Huajun Chen, Ningyu Zhang
Machine Learning
Delayed homomorphic reinforcement learning for environments with delayed feedback
Jongsoo Lee, Jangwon Kim and Soohee Han
Reinforcement learning in real-world systems often involves delayed feedback, which violates the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches address this issue by augmenting the state with action histories, but suffer from state-space explosion and the resulting sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based methods either primarily alleviate this burden for the critic or employ separate mechanisms for the actor and critic. To address this limitation, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that introduces a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states into a compact abstract state space. This framework yields exact abstraction under deterministic dynamics and approximate abstraction under
stochastic dynamics, providing a unified abstraction mechanism that benefits both the actor and critic. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, while approximate abstraction admits a bound on the value loss of the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (D$^2$HPG), a deep actor-critic instantiation of the DHRL framework. Experiments on MuJoCo continuous-control
benchmarks demonstrate that D$^2$HPG outperforms strong augmentation-based baselines, especially in long-delay regimes. Code is available at https://github.com/JongsooLee/DHRL.
Read the full abstract ▾
Machine Learning
Algebraic Diversity: Group-Theoretic Spectral Estimation from Single Observations
Mitchell A. Thornton
We establish that temporal averaging over multiple observations is the degenerate case of algebraic group action with the trivial group $G=\{e\}$. A General Replacement
Theorem proves that a group-averaged
estimator from one snapshot achieves equivalent subspace decomposition to multi-snapshot covariance estimation. The Trivial Group
Embedding Theorem proves that the sample covariance is the accumulation of trivial-group estimates, with variance governed by a $(G,L)$ continuum as $1/(|G|\cdot L)$. The processing gain $10\log_{10}(M)$ dB equals the classical beamforming gain, establishing that this gain is a property of group order, not sensor count. The DFT, DCT, and KLT are unified as group-matched special cases. We
conjecture a General Algebraic Averaging
Theorem extending these results to arbitrary statistics, with variance governed by the effective group order $d_{\mathrm{eff}}$. Monte Carlo experiments on the first four sample moments across five group types confirm the
conjecture to four-digit precision. The framework exploits the $structure$ of information (representation-theoretic
symmetry of the data object) rather than the content, complementing Shannon's theory. Five applications are demonstrated: single-snapshot MUSIC, massive
MIMO, single-pulse waveform classification, graph signal processing, and analysis of
transformer LLMs. Techniques for blind group matching are described.
Read the full abstract ▾
Machine Learning
One-Shot Localisation of the Global Minimum of a Noisy One-Dimensional Function: A Controlled Comparison of an Iterative Neural Minimizer, Set Transformers and Classical Estimators
Qusay Muzaffar, David Levin and Michael Werman
We study passive global optimisation: from twenty noisy samples of an unknown one-dimensional function, with no further queries, estimate where its global minimum lies. We build a
benchmark and a self-checking harness and use them for a controlled comparison. The Neural Function Minimizer (NFM), which we introduce, moves a position across the domain for fifteen steps and re-reads the samples from it at every step; a same-size Set
Transformer answers in one pass. In a 2x2 design each has one output head or an added mixture head, with the same data and inputs and three seeds each. Within the training families the two are tied on location error, and the mixture loss lowers it by about 0.45 points of the domain in both. The NFM's answers have lower regret, by 2.90 and 3.12 points of the range with one and two heads (significant after correction, and again on a block of fresh cases), mainly because its right-basin answers lie closer to the floor; a Set
Transformer looped to the NFM's computation does not reproduce this within the training families, though it has the lower error on unseen functions (exploratory). The NFM also differs from the Set
Transformers in encoder, local read, activation, losses and training time. At near-tied minima the form of the answer decides: point answers hedge in about a quarter of exact ties, mixture modes almost never, mixture means in more than 80%. Gaussian-process
estimators, with no training, have the lowest regret and are the most accurate on unseen functions. We will release the
benchmark, the models and the harness.
Read the full abstract ▾
Machine Learning
BlazeFL: Fast and Deterministic Federated Learning Simulation
Kitsuya Azuma and Takayuki Nishio
Federated learning (FL) research increasingly relies on single-node simulations with hundreds or thousands of virtual clients, making both efficiency and reproducibility essential. Yet parallel client training often introduces nondeterminism through shared random state and scheduling variability, forcing researchers to trade throughput for reproducibility or to implement custom control logic within complex frameworks. We present BlazeFL, a lightweight framework for single-node FL simulation that alleviates this trade-off through free-threaded shared-
memory execution and deterministic randomness management. BlazeFL uses thread-based parallelism with in-
memory parameter exchange between the server and clients, avoiding serialization and inter-process communication overhead. To support deterministic execution, BlazeFL assigns isolated random number generator (RNG) streams to clients. Under a fixed software/hardware stack, and when
stochastic operators consume BlazeFL-managed generators, this design yields bitwise-identical results across repeated high-concurrency runs in both thread-based and process-based modes. In CIFAR-10 image-classification experiments, BlazeFL substantially reduces execution time relative to a widely used open-source baseline, achieving up to 3.1$\times$ speedup on communication-dominated workloads while preserving a lightweight dependency footprint. Our open-source implementation is available at: https://github.com/kitsuyaazuma/blazefl.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Stochastic Generative Plug-and-Play Priors
Chicago Y. Park, Edward P. Chandler, Yuyang Hu et al. (+4 more)
Plug-and-play (PnP) methods are widely used for solving imaging inverse problems by incorporating a denoiser into an optimization
algorithm. Score-based
diffusion models (SBDMs) have recently demonstrated strong generative performance via a denoiser trained across a wide range of noise levels. Despite their shared reliance on denoisers, it remains unclear how to systematically use SBDMs as priors within the PnP framework without relying on reverse
diffusion sampling. In this paper, we establish a score-based interpretation of PnP that justifies using pretrained SBDMs directly within PnP
algorithms. Building on this connection, we introduce a
stochastic generative PnP (SGPnP) framework that uses
stochastic re-noising to better leverage the broad noise-level conditioning of pretrained SBDMs, significantly improving reconstruction quality in severely ill-posed inverse problems. We show that noise injection induces optimization on a Gaussian-smoothed objective and enables avoidance of strict saddle points under suitable conditions. Experiments on challenging inverse tasks, such as multi-coil MRI reconstruction and large-mask natural image inpainting, demonstrate consistent improvement over conventional PnP methods and achieve performance competitive with
diffusion-based solvers.
Read the full abstract ▾
All authors Chicago Y. Park, Edward P. Chandler, Yuyang Hu, Michael T. McCann, Cristina Garcia-Cardona, Brendt Wohlberg, Ulugbek S. Kamilov
Machine Learning
Evaluation of Bagging Predictors with Kernel Density Estimation and Bagging Score
Philipp Seitz, Jan Schmitt and Andreas Schiffler
For a larger set of predictions of several differently trained machine learning models, known as bagging predictors, the mean of all predictions is taken by default. Nevertheless, this proceeding can deviate from the actual ground truth in certain parameter regions. An approach is presented to determine a representative y_BS from such a set of predictions using Kernel Density Estimation (KDE) in nonlinear regression with Neural Networks (NN) which simultaneously provides an associated quality criterion beta_BS, called Bagging Score (BS), that reflects the confidence of the obtained ensemble prediction. It is shown that working with the new approach better predictions can be made than working with the common use of mean or median. In addition to this, the used method is contrasted to several approaches of nonlinear regression from the literatur, resulting in a top ranking in each of the calculated error values without using any optimization or feature selection technique.
Read the full abstract ▾
Machine Learning
Simple yet Effective: Low-Rank Spatial Attention for Neural Operators
Zherui Yang, Haiyang Xin, Tao Du et al. (+1 more)
Neural operators have emerged as data-driven surrogates for solving partial differential equations (PDEs), and their success hinges on efficiently modeling the long-range, global coupling among spatial points induced by the underlying physics. In many PDE regimes, the induced global interaction kernels are empirically compressible, exhibiting rapid spectral decay that admits low-rank approximations. We leverage this observation to unify representative global mixing modules in neural operators under a shared low-rank template: compressing high-dimensional pointwise features into a compact latent space, processing global interactions within it, and reconstructing the global context back to spatial points. Guided by this view, we introduce Low-Rank Spatial
Attention (LRSA) as a clean and direct instantiation of this template. Crucially, unlike prior approaches that often rely on non-standard aggregation or normalization modules, LRSA is built purely from standard
Transformer primitives, i.e.,
attention, normalization, and feed-forward networks, yielding a concise block that is straightforward to implement and directly compatible with hardware-optimized kernels. In our experiments, such a simple construction is sufficient to achieve high accuracy, yielding an average error reduction of over 17% relative to second-best methods, while remaining stable and efficient in mixed-precision training.
Read the full abstract ▾
All authors Zherui Yang, Haiyang Xin, Tao Du, Ligang Liu
Machine Learning · Cross-listed from cs.RO
Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme et al. (+1 more)
Bimanual
robot learning from demonstrations is fundamentally limited by the cost and narrow visual diversity of real-world data, which constrains policy robustness across viewpoints, object configurations, and embodiments. We present Canny-guided
Robot Data Generation using
Video Diffusion Transformers (CRAFT), a
video diffusion-based framework for scalable bimanual demonstration generation that synthesizes temporally coherent manipulation
videos while producing action labels. By conditioning
video diffusion on edge-based structural cues extracted from simulator-generated trajectories, CRAFT produces physically plausible trajectory variations and supports a unified augmentation pipeline spanning object pose changes, camera viewpoints, lighting and background variations, cross-embodiment transfer, and multi-view synthesis. We leverage a pre-trained
video diffusion model to convert simulated
videos, along with action labels from the simulation trajectories, into action-consistent demonstrations. Starting from only a few real-world demonstrations, CRAFT generates a large, visually diverse set of photorealistic training data, bypassing the need to replay demonstrations on the real
robot (Sim2Real). Across simulated and real-world bimanual tasks, CRAFT improves success rates over existing augmentation strategies and straightforward data scaling, demonstrating that
diffusion-based
video generation can substantially expand demonstration diversity and improve generalization for dual-arm manipulation tasks. Our project website is available at: https://craftaug.github.io/
Read the full abstract ▾
All authors Jason Chen, I-Chun Arthur Liu, Gaurav Sukhatme, Daniel Seita
Machine Learning
Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks
Benjamin S. Knight and Ahsaas Bajaj
This study surveys the historical development of regularization, tracing its
evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features,
Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks – Ridge, Lasso, ElasticNet, and Post-Lasso OLS – across 134,400 simulations spanning a 7-dimensional
manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.
Read the full abstract ▾