Machine Learning
The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence
Xiaoyu Yang, Jie Lu, Wei Duan et al. (+1 more)
Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient
attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance
Alignment), a t-distributed directional matching mechanism that reshapes the context
retrieval distribution, directing more
attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained
benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/
Read the full abstract ▾
All authors Xiaoyu Yang, Jie Lu, Wei Duan, En Yu
Machine Learning
Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Yuanteng Chen, Zhilei Liu, Peisong Wang et al. (+7 more)
Quantization-aware
distillation (QAD) restores much of the short-form question-answering performance lost to sub-3-bit quantization, yet leaves mathematical and code
reasoning substantially impaired. Long generations often degenerate into repetitive loops, exhausting the decoding budget without completing a solution. We trace this gap to quantization-amplified exposure bias: QAD trains on fixed corpus prefixes, while quantization-induced deviations compound along the model's own autoregressive trajectories. To address this mismatch, we introduce an on-policy
distillation (OPD) stage that places teacher supervision where the quantized model actually goes. Starting from a QAD checkpoint, the student generates through the quantized forward path used at deployment and receives feedback from a frozen full-precision teacher on its own prefixes, combining dense
token-level guidance with task-verifier rewards. Across four models at 2.79 and 1.88 effective bits, OPD raises average BF16 performance retention from 35% to 70% on MATH-500 and from 66% to 91% on HumanEval while preserving short-form performance, with
reasoning gains substantially exceeding those of continued teacher-forced QAD in matched-budget comparisons. By coupling QAD's stable low-bit initialization with OPD's on-policy
reasoning recovery, our framework provides a comprehensive sub-3-bit solution that preserves broad capabilities while restoring long-form
reasoning.
Read the full abstract ▾
All authors Yuanteng Chen, Zhilei Liu, Peisong Wang, Yuantian Shao, Chuangyi Li, Weining Wang, Shuang Qiu, Gang Li, Jing Liu, Jian Cheng
Machine Learning · Cross-listed from stat.ME
Optimal Sequential Annotations for Off-Policy Evaluation
Woojin Chae, Ezinne Nwankwo, Haitong Qin et al. (+1 more)
Offline reinforcement learning and off-policy evaluation evaluates dynamic treatment rules based on retrospectively collected data prior to deployment. In recent AI applications, state and reward information is recorded as complex text or image, which recent AI advancements such as LLM-as-a-judge can label with unknown bias. Expert annotation may be available but at a higher cost. For example, safety classification via cheap but imperfect classifiers vs. expensive expert review. We show how a limited budget for ground-truth data-annotation can be used via doubly-robust OPE with missing rewards, and we optimize variance-optimal annotation probabilities for sequential off-policy evaluation, where the target policy value is estimated from annotated data. We characterize the optimal annotation probabilities for sequential forward-monotone annotation protocols, and provide a feasible batch-adaptive implementation. Our work is motivated by a collaboration with a homelessness services nonprofit that writes casenotes for individuals over time. Our method can be used to unlock trustworthy inference from casenote data and answer new inferential questions such as: how does expanding outreach effort over time affect progress towards a housing application and improvement in housing placement? In simulations and on two real
datasets - casenotes from the nonprofit and human-preference votes from LMArena - we see reductions in RMSE of 34-65% for housing placement and 17-68% for progress towards a housing application at budgets of 40% of full annotation and above, and by 55-62% at every budget on LMArena.
Read the full abstract ▾
All authors Woojin Chae, Ezinne Nwankwo, Haitong Qin, Angela Zhou
Machine Learning · Cross-listed from quant-ph
When are bosonic Gaussian states classical to learn?
Senrui Chen, Antonio Anna Mele, Francesco Anna Mele et al. (+1 more)
A fundamental question in physics is: When does classical behavior emerge from
quantum systems? Bosonic Gaussian states provide a natural setting to explore this
quantum-classical boundary, as they capture both the classical field behavior and the intrinsic
quantum nature of light. Here, we address this problem from a learning-theoretic perspective by asking: When are bosonic Gaussian states classical to learn? That is, under what conditions (if any) can an n-mode bosonic Gaussian state be learned with as few samples, and with operations as simple, as are needed to learn a classical 2n-variate Gaussian distribution? We establish a smooth crossover in learnability governed by the state's thermal fluctuations: - Cold Gaussian states are non-classical to learn: When the covariance
matrix satisfies $\Sigma\le(\frac12+O(\frac1n))I$, i.e. close to the
vacuum covariance,
tomography under single-copy (i.e., non-entangled) measurements fundamentally requires $\Omega(n^3)$ copies, strictly exceeding the sample complexity $\Theta(n^2)$ of learning classical Gaussian distributions. We show that this hardness persists even when few-copy entangled measurements are allowed. - Warm Gaussian states are classical to learn: When thermal fluctuations exceed the
vacuum noise, parameterized by $\Sigma\ge(\frac12+\nu)I$ for any parameter $\nu>0$, we prove that single-copy
tomography requires $N=\Theta\left(n^2\min(n,1+\nu^{-1})\right)$ copies. This bound is tight and is achieved by simple, non-adaptive, unentangled heterodyne measurements. Crucially, for $\nu=\Omega(1)$, the sample complexity drops to $\Theta(n^2)$, matching the classical case. Our results tightly characterize a
quantum-to-classical crossover in the learnability of bosonic Gaussian states, reveal a novel connection between fundamental physics and statistical learning theory, and have implications for real-world sensing experiments.
Read the full abstract ▾
All authors Senrui Chen, Antonio Anna Mele, Francesco Anna Mele, John Preskill
Machine Learning · Cross-listed from astro-ph.SR
PROSWIN: Probabilistic Solar Wind Speed Forecasting Using Deep Distributional Regression From Solar Images
Daniel Collin, Yuri Shprits, Luca Chiarabini et al. (+3 more)
Accurately predicting fast
solar wind conditions is challenging, as uncertainties are large and unquantified by traditional single-value prediction models. In particular, the risks of high-speed
solar wind streams (HSSs), which can cause damage to technological infrastructure, cannot be reliably assessed without probabilistic forecasts. We present PROSWIN, a probabilistic machine learning model that forecasts the hourly
solar wind speed (SWS) at Earth with a four-day lead time. The approach combines
solar images and magnetograms using a deep neural network coupled to a distributional regression
algorithm. Because standard error metrics underweight the relevance of HSS peaks, we further introduce the prediction score, a model-selection metric that jointly rewards timeline and HSS peak accuracy. On 14 years of data, our forecast achieves very well-calibrated uncertainties (<1% average deviation). Using the continuous ranked probability score (CRPS), a metric that assesses distributional accuracy, we obtain a timeline CRPS of 41.0 km/s, an HSS peak CRPS of 45.3 km/s, and a prediction score of 42.3 km/s. We find that the 171 Å channel is an important complement to the typically used 193 Å and 211 Å channels and that the prediction score for model selection improves the applicability of the model. Compared to selected models from the literature, ours is the only one that is accurate for both timeline and HSS peak values, rather than trading one off against the other. These results support the advantages of probabilistic over single-value
solar wind models. The introduced methods are also transferable to other
forecasting problems.
Read the full abstract ▾
All authors Daniel Collin, Yuri Shprits, Luca Chiarabini, Stefan J. Hofmeister, Nadja Klein, Guillermo Gallego
Machine Learning
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
Lenz Pracher, Pascal de Jong, Oskar Lieshaus et al. (+2 more)
In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and $L_2$ weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK
eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is controlled by the product of learning rate and weight decay, that feature learning slows logarithmically near a critical decay above which task-aligned NTK structure can no longer support generalization, and that stronger decay can prevent fitting altogether. We test these predictions in modular addition. In a homogeneous MLP, task-aligned
Fourier structure continues to emerge in the NTK after training accuracy has saturated, and an 84$\times$90-grid of trained networks across varying learning rate and weight decay recovers the predicted phase geometry and inverse-product scaling of the generalization time with learning rate and weight decay. A one-block
Transformer shows similar macroscopic phase structure in a 42$\times$45-grid, as well as the same transition-time scaling despite violating exact homogeneity. Together, these results provide a mechanistic derivation connecting post-fit feature learning to both the onset of generalization and its phase structure in the learning rate and weight decay plane.
Read the full abstract ▾
All authors Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
Machine Learning
MAGIC: Mixed-Granularity Agent Graphs via Incremental Construction with Dense-Reward Reinforcement Learning
Kairui Yang, Ziheng Yi, Xunkai Li et al. (+4 more)
Collaboration
topology shapes both the performance and execution cost of LLM-based multi-
agent systems. Because tasks differ in complexity and required capabilities, recent approaches generate task-specific collaboration graphs that specify
agent participation and information flow. However, representative
topology generators use either individual
agents or predefined groups throughout an organization, overlooking differing collaboration needs across subtasks. Our key insight is to select granularity locally for each functional role, combining fine-grained control with reusable collaboration patterns within one organization. Learning such organizations requires exploring a combinatorial construction space with limited intermediate feedback from final-answer rewards. Therefore, we propose MAGIC, a dense-reward reinforcement learning framework for mixed-granularity graph generation. Specifically, MAGIC constructs a mixed-granularity
agent graph by sequentially selecting a functional role, instantiating it as a single
agent or reusable group, and connecting it to existing units. We directly optimize the construction policy using returns from trajectories sampled under the current policy and use potential-based reward shaping to provide intermediate feedback from probe-based utility and structural signals while preserving the cumulative task reward. MAGIC outperforms state-of-the-art baselines across eight
benchmarks and demonstrates strong inference efficiency in our efficiency study.
Read the full abstract ▾
All authors Kairui Yang, Ziheng Yi, Xunkai Li, Minghao An, Zhanke Liu, Zekai Chen, Rong-Hua Li
Machine Learning · Cross-listed from cs.IR
Discovery-Driven Integration of Disjoint Tables via Text
Md Ataur Rahman, Dimitris Sacharidis, Oscar Romero et al. (+1 more)
Integrating heterogeneous
datasets within data lakes is a critical challenge, particularly for semantically related tables that lack the explicit attributes needed to be joined. We study Discovery-Driven Integration, where the relevant sources and their missing relational structure must be discovered before integration. In this setting, unstructured text provides the evidence that connects otherwise disjoint tables. The fundamental challenge is to discover the relationships at a fine-grained level that connect individual rows from different tables through specific sentences. We formalize this task as Text-Mediated Join Path Discovery and propose a horizontal bidirectional cross-
attention architecture called LOKI Latent-space Optimization for Knowledge Integration) that learns contextualized representations of table rows and sentences. Through a global table-text contrastive objective, fine-grained row-sentence associations emerge without explicit local supervision. Existing multi-modal discovery methods largely retrieve coarse-grained column-text associations, whereas integration systems assume supplied row-text links, schemas, or queries. LOKI instead transforms these implicit associations into explicit, interpretable join paths, organizes them into relation-consistent groups, and materializes them as typed integrated tables with sentence-level provenance. Comprehensive evaluations on real-world
benchmarks demonstrate that LOKI consistently outperforms state-of-the-art multi-modal data discovery approaches, and materializes typed integrated tables with 0.982 macro typed-pair precision while being up to 40 times cheaper in LLM API cost than direct prompting.
Read the full abstract ▾
All authors Md Ataur Rahman, Dimitris Sacharidis, Oscar Romero, Sergi Nadal
Machine Learning · Cross-listed from math.ST
Statistical Rates for Entropic Optimal Transport in the Discrete to SubGaussian Regime
Tomas Gonzalez and Gonzalo Mena
We study statistical rates in entropic optimal transport in the semi-discrete regime where one measure has finite support and the other is subGaussian. Our main result establishes parametric convergence rates for the empirical dual potentials to their population counterparts, with no dimension dependence in the leading term. Our result relies on tailored strong concavity analysis of the semi-dual objective, coupled with specialized bounds for the semi-discrete potentials. As a consequence, we obtain fast rates for downstream quantities derived from the optimal coupling. Chiefly, the empirical barycentric projection achieves a squared-error rate $n^{-1}$, matching the fully compact case and improving over the less favorable $n^{-1/2}$ rate known for fully subGaussian settings. Altogether, these results may indicate a lower complexity adaptation phenomenon whereby the statistical complexity of the barycentric projection is governed by the discrete measure. As an application, we analyze Sinkhorn-EM, an EM-type
algorithm in which the E-step is replaced by an entropic optimal transport problem. In a well-specified and balanced two-component Gaussian mixture model, we prove $\sqrt{n}$-consistency of the empirical iterates to their population counterparts for any fixed number of iterations, matching classical EM rates up to a $\sqrt{\log n}$ factor. Simulations support the theory.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
The Delegation Blind Spot: Auditing Product Decisions from Agent Choices
Shivam Gupta
Successful
agent execution need not identify which future product improvement its user would value. We present a decision-specific audit that maps a declared observation channel and product-value contrast to compatible intervals and witness populations. Its foundations are established identification and decision theory; the contribution is an executable measurement workflow and a controlled study of its limits. A frozen experiment makes 4,800 requests to two pinned model snapshots on shared synthetic tasks. All 36 conservative primary intervals remain unresolved despite different execution accuracy. An exploratory 2,400-call follow-up records supplied preferences and resolves three of nine comparisons per model. A deterministic extractor resolves seven of nine without model calls or calibration observations, exposing unnecessary uncertainty introduced by model-generated reports. A further 14,400 controlled multinomial simulations distinguish structural ambiguity from weak identification and finite calibration precision. We propose a source-labeled decision receipt and provide an offline viewer for inspecting the audit. These results motivate preserving decision-relevant structured input and diagnosing why a decision is unresolved before collecting more telemetry. The study contains no human participants or real customer outcomes. Full proofs, raw model provenance, controlled experiments, and reproducible analyses accompany the report.
Read the full abstract ▾
Machine Learning
Label-Efficient Learning for Ground-Based Sky-Image Classification: A Benchmark of Transfer Learning, Active Learning, and Pseudo-Labeling on GCD
Esther Bou Dagher, Viktoriya Bu-Dager and Boguslaw Zegarlinski
Accurate ground-based cloud classification is important for atmospheric monitoring,
solar-energy
forecasting, aviation weather assessment, and
climate observation systems. However, reliable sky-image annotation is time-consuming, especially when cloud types are visually similar or mixed. We study the label efficiency of deep learning for ground-based cloud classification using the Ground-based Cloud
Dataset (GCD). Rather than proposing a new architecture, we
benchmark three practical strategies under limited annotation budgets: supervised transfer learning, uncertainty-based active learning, and high-confidence pseudo-labeling. An ImageNet-pretrained ResNet50 is used as a common frozen backbone, with experiments repeated over five random seeds for label budgets from $1\%$ to $100\%$ of the training labels. Supervised transfer learning is already highly label-efficient: test accuracy increases from $0.635 \pm 0.018$ with $1\%$ labels to $0.730 \pm 0.002$ with $40\%$ labels, approaching the full-label result of $0.735 \pm 0.003$. Active learning and pseudo-labeling are competitive with supervised sampling and provide small improvements for some metrics and budgets, but neither gives a large or consistent aggregate gain. Diagnostic analyses show that accepted pseudo-labels are reliable, with accuracy from $0.946$ to $0.977$, but biased toward easier high-confidence sky-type groups. In contrast, uncertainty sampling preferentially queries visually challenging groups, including Mixed and the confusable Stratocumulus and Cumulonimbus groups, but these targeted acquisitions yield only modest gains. Overall, transfer learning substantially reduces annotation requirements for GCD, while simple active and semi-supervised strategies provide limited additional benefit over a strong supervised baseline.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
On Basis Function Selection for Sparse Gaussian Process Regression
Marnix Van Soom and Ivan De Boi
Sparse Gaussian processes achieve $O(N)$ inference by replacing the kernel with an appropriate expansion in a fixed basis $\{\phi_j\}$ on the input space. Given a compute budget $M \ll N$, practitioners conventionally truncate the basis to its first $M$ entries. Nothing in the formalism, however, prevents one from selecting only those $M$ basis functions that matter for the data at hand. This would avoid spending budget on basis functions where there is no signal, but it requires a criterion for ranking the candidates. We propose three such criteria derived from an information-theoretic view of the basis-function selection problem. Each criterion matches a different state of knowledge at selection time: a no-data state, a no-prior state, and an in-between state. We then study the performance of truncation versus selection strategies on six UCI regression
benchmarks across three basis families:
Hilbert-space Gaussian processes (HSGP), variational
Fourier features (VFF), and variational inducing spherical harmonics (VISH). We observe that the no-data criterion is a safe default, matching or improving on truncation for HSGP, VFF and VISH, with substantial gains for VISH and improvements over a recently developed selection heuristic for that basis family. The data-aware no-prior and in-between criteria provide substantial gains over truncation specifically for HSGP, which is the most broadly used of the three families in practice.
Read the full abstract ▾
Machine Learning
Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Gaoyuan Du, Anam Nawaz Khan, Rex Zhou et al. (+4 more)
Greedy decoding from large
language models is commonly treated as deterministic. We show it is not precision-invariant: the same model,
prompt, and decoding
algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our evaluations of six models (1.1B-7B parameters, four families; divergence additionally characterised at 12B) and three
benchmarks, 49-100% of
prompts diverge; a single
token flip often cascades into trajectory-level divergence. We develop an empirical error-propagation analysis and find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates. The analysis makes five testable predictions about intervention outcomes, including that applying more FP32 compute (broader scope) makes agreement worse. The experiments match all five predictions. The best-performing low-overhead intervention we evaluate, selective FP32 LM head recomputation, triggered only when the margin falls below a threshold, delivers +22-36 pp exact agreement on A10G (+12-21 pp on L4 and A100) at less than 4% latency overhead in low-batch (batch size <=4) single-stream inference. We map the applicability boundary across six models and four batch sizes, and hypothesise that training-time precision stability is a determining factor. The method is a partial mitigation rather than a universal determinism guarantee: its benefit vanishes when body-originated error dominates, including at batch size >=8 and under end-to-end FP8 in our tests.
Read the full abstract ▾
All authors Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu, Deepayan Chakrabarti, Fnu Suya, Xueping Li
Machine Learning · Cross-listed from cs.CV
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Fiona Kekwick, Matthew Baugh, Bernhard Kainz et al. (+2 more)
Clinical decision making heavily relies on predicting the disease progression trajectory by seeking to understand patient's health status which is characterised by
multimodal medical data. AI holds great potential for learning useful representations from
multimodal medical data to predict disease progression and aid clinical decision making. However, development of predictive AI models is constrained by missing modalities and incomplete tabular data frequently occurring in medical
datasets. In addition, disease labels alone may only provide limited supervisory signals for learning representations from high-dimensional
multimodal data. Here, we present MMAP, a novel
Multimodal Missing-aware
Alignment Pretraining method for learning image-tabular representations from incomplete data. An image encoder is pretrained with efficient sigmoid contrastive learning combined with generative reconstruction. A tabular encoder is built upon a tabular foundation model. A missing
token generator enables the two encoders to take incomplete data as input, enabling the model to be robust against missing modalities, either with missing images or missing tabular data. We evaluate the clinical usefulness of the learnt
multimodal representations on two challenging longitudinal clinical tasks for Alzheimer's disease: predicting disease stage conversion and predicting amyloid status. The proposed method outperforms strong
multimodal and unimodal baselines.
Read the full abstract ▾
All authors Fiona Kekwick, Matthew Baugh, Bernhard Kainz, Paul M. Matthews, Wenjia Bai
Machine Learning · Cross-listed from cs.CV
Foundation model embeddings capture pre-diagnostic changes on screening mammograms
Kalina P. Slavkova, Eric Brattain, Aditya Gowd et al. (+6 more)
Foundation model
embeddings of screening mammograms may encode pre-diagnostic tissue change without task-specific adaptation. We tested whether
embeddings move faster along a data-derived "cancer direction" in women later biopsied for cancer than in matched screen-negative controls, and whether this depends on pretraining domain. We studied 1,773 biopsied women (785 malignant, 988 biopsy-negative) and 1,773 matched controls, each with at least two annual screening exams before their index exam. An identical pipeline was applied to four 2D models: Mammo-CLIP (MC, out-of-distribution mammography), HOPPR (in-distribution mammography), MedImageInsight (MII, general medical imaging), and BiomedCLIP (biomedical
vision-
language pretraining on literature figures). Breast-level
embeddings quantified longitudinal movement along the cancer direction. We compared cases and controls using a between-patient design with complementary mixed-effects analysis, and biopsied versus healthy contralateral breasts within patients. Under matched modality in MII
embedding space, malignant cases drifted significantly faster than controls in the first two screening intervals preceding the index exam; biopsy-negative cases showed significance only in the first. MC differences were significant in the first interval for both biopsy groups. Within-patient comparisons showed a broadly similar pattern, with MC significance extending to the second interval in both groups and HOPPR showing significance at interval 1. BiomedCLIP showed no significant differences in either design or biopsy group. Overall, directional
embedding velocity emerges as a property of clinically grounded rather than general biomedical pretraining, showing that foundation model
embeddings can encode pre-diagnostic mammographic change without task-specific adaptation.
Read the full abstract ▾
All authors Kalina P. Slavkova, Eric Brattain, Aditya Gowd, Akash Pattnaik, Jean-Benoit Delbrouck, Matthew Morgan, Julie Bauml, Javid Abderezaei, Khan Siddiqui
Machine Learning
Hierarchical GNNs for power flow: letting physics shape the hierarchy
Carmine Delle Femine, Leire Garin Atxaga, Asier Diaz-Iglesias et al. (+4 more)
Hierarchical latent communication improves the generalization of a power-flow model, shared across three grids, to new operating scenarios. The module exchanges information through two reduced graphs inside the corrective network of GENCO, replacing two of its local correction steps. We compare Kron-derived transports, a same-anchor Quotient construction and the flat GENCO Base architecture, all trained under one protocol of our own with about a hundred times fewer optimizer updates per grid than GENCO's reference training: 200 epochs on three grid
topologies, fewer than 1,900 training scenarios per grid and three initialization seeds per model. Evaluation uses 200 newly generated, preselected scenarios per grid. On the training
topologies, Kron reaches a macro family-balanced voltage error of $0.851\pm0.110$, 51.3% below a per-bus mean fitted on training solutions (1.747). Kron is below this reference on 98.5% of the 600 fresh scenarios, and both hierarchical models outperform it on every training
topology in all three seeds. The flat baseline reaches $5.660\pm0.899$ and does not outperform the reference on any training
topology, so Kron's 85.0% reduction relative to it compares architectures within our training regime. Kron is also 31.0% below Quotient ($1.235\pm0.225$). These results demonstrate generalization across operating scenarios within the studied
topologies, with one set of learned parameters shared across grids. On two
topologies unseen in training, the current models do not yet outperform the fitted reference in calibrated transfer; extrapolation to new
topologies is the next development objective.
Read the full abstract ▾
All authors Carmine Delle Femine, Leire Garin Atxaga, Asier Diaz-Iglesias, Juan Pablo Maroto Herrera, Ane Miren Florez-Tapia, Marco Quartulli, Izaro Goienetxea Urziku
Machine Learning · Cross-listed from eess.SP
Unlocking Cross-Scenario Physical Layer Security: A Mixture-of-Experts Framework with Generative Diffusion Models
Xiao Tang, Tong Hui, Chao Shen et al. (+4 more)
The future 6G networks are expected to incorporate a proliferation of wireless services in diverse environments, which presents a significant challenge for information security. Conventionally optimization always requires recalculation and learning strategy often suffers poor generalization, which are thus incapable for the security provisioning with wide scenario coverage. In this paper, we propose an adaptive and robust learning framework that leverages a mixture-of-experts (MoE) architecture to achieve cross-scenario physical layer security guarantee. Specifically, we first select a few representative scenarios and establish the scenario-specific generative
diffusion model (GDM)-based experts for secure transmission beamforming with artificial noise. The
diffusion nature of experts learns the overall probability distribution of security strategy solution landscape and the
Transformer-based denoising process enhances the ability to generalize across varying network configurations. Then, a lightweight gating network is constructed to identify the scenarios by engineering the channel features and select the most relevant experts. Finally, an
attention-based combiner is introduced to synthesize the security proposals from the top-rated experts to produce a high-fidelity security strategy to cover the unseen scenarios. Simulation results demonstrate that the proposed GDM-based MoE framework can accurately recognize the scenarios and properly select the experts, maintaining near-optimal secrecy rates across a continuum of wireless scenarios and outperforming traditional single-model paradigms.
Read the full abstract ▾
All authors Xiao Tang, Tong Hui, Chao Shen, Yichen Wang, Qinghe Du, Li Sun, Zhu Han
Machine Learning · Cross-listed from cs.CV
GTR: Gated Token Recurrence for Efficient Dense Prediction
Zhe Feng, Longfei Liu, Wei Liu et al. (+7 more)
Self-
attention-based
vision backbones perform well on dense prediction, but the quadratic computational cost of global softmax
attention limits their efficiency as image resolution increases. We introduce Gated
Token Recurrence (GTR), a softmax-free recurrent
vision backbone that combines gated linear
attention, alternating spatial scan directions, and spatially enhanced SwiGLU blocks. GTR is distilled from a detection-specialized DINOv3 teacher using only final-layer patch-
token alignment through a linear projection and squared $\ell_2$ loss, without masked-
token prediction or intermediate-layer supervision. With Objects365 detector pre-training, GTR-L achieves 58.9 box AP on COCO val2017 with 1.908\,ms median batch-one latency under compiled FP16 execution on an RTX~4090. The same backbone also transfers to instance
segmentation, pose estimation, oriented detection, semantic
segmentation, and
monocular depth estimation. In an isolated kernel
benchmark, our specialized chunkwise CUDA operator is $4.0\times$ faster than FLA v0.5.0 at 1.6K
tokens on RTX~4090. TensorRT deployment on DRIVE AGX Thor achieves 2.282–8.769\,ms median batch-one latency across the evaluated models. These results show that recurrent
token mixing can provide an efficient alternative to global softmax
attention for high-resolution dense prediction and edge deployment. Project page: https://intellindust-ai-lab.github.io/projects/GTR/
Read the full abstract ▾
All authors Zhe Feng, Longfei Liu, Wei Liu, Kai Chen, Jiangang Kong, Wei Zhou, Yifeng Qian, Dexiong Chen, Xuanlong Yu, Xi Shen
Machine Learning · Cross-listed from math.OC
Polyak-Type Extragradient Methods for Monotone Root-Finding Problems
TaeHo Yoon, Sayantan Choudhury, Ezra Greenberg et al. (+1 more)
We study Polyak-type step-size selection for extragradient methods for solving deterministic and
stochastic monotone root-finding problems. We show that the known projection-type correction for deterministic extragradient arises from minimizing an upper bound on the distance to a solution, paralleling the classical Polyak step-size construction. Using this viewpoint, we provide a unified deterministic analysis of the Polyak-type Extragradient Method (PolyakEG), based on a local critical condition controlling the variation of operator $F$ along the extrapolation direction. This analysis does not require global Lipschitz continuity, and covers sublinear convergence under broader conditions such as Hölder continuity or $(L_0, L_1)$-Lipschitzness and linear convergence under additional strong monotonicity, all through a single framework. We then study the
stochastic extensions of this approach. We first prove convergence of a direct
stochastic variant, PolyakSEG, when all
stochastic component operators share a common solution. We also show that, without this condition, PolyakSEG with nonvanishing step-sizes may fail to converge to a zero of the mean operator. To address this limitation, we propose DecPolyakSEG, which combines decreasing step-sizes with Polyak-type updates, and establish a sublinear residual convergence result without requiring a common solution across the component operators. These results parallel recent developments in
stochastic Polyak step-sizes from the convex minimization literature and establish an analogous research avenue in the broader root-finding regime.
Read the full abstract ▾
All authors TaeHo Yoon, Sayantan Choudhury, Ezra Greenberg, Nicolas Loizou
Machine Learning
Notes on Fourier-Bessel wavelets
Marcel Venturotti and Georgios Exarchakis
These notes develop the mathematical foundations and construction of a
Fourier-Bessel wavelet family inspired by the disk harmonics of Shaqfa et al.[9]. We begin with the relevant properties of Bessel and modified Bessel functions and introduce the wavelet properties required for the construction. We then derive the
Fourier-Bessel disk harmonics as solutions to the Helmholtz equation on the unit disk subject to a Neumann boundary condition. Building on this basis, we construct a wavelet family by applying a Gaussian spatial envelope and introducing a zero-mean correction for the zeroth angular order. We derive the corresponding normalisation constants for $L^2$-based applications and discuss $L^1$-based normalisation for frequency-domain peak consistency. Finally, we derive a closed-form
Fourier-domain representation of the resulting wavelets. The main motivation is the approximately linear spacing, which converges to $\pi$ between consecutive radial
eigenvalues. Rather than replacing the conventional dyadic organisation of wavelet families, this construction lays out the foundation to explore whether a more uniform radial frequency allocation can be useful for applications in which broad and balanced frequency coverage is desirable.
Read the full abstract ▾
Machine Learning
Gap-Free Streaming PCA Beyond Rank-One Updates: Near-Optimal Rates and Applications to Differential Privacy
Anming Gu, Syamantak Kumar, Kevin Tian et al. (+1 more)
Streaming principal component analysis (PCA) seeks to recover a leading spectral subspace in a single pass over a data stream. We give a new analysis of the ubiquitous Oja's
algorithm [Oja82] for the most general, gap-free variant of this problem, where no eigengap assumptions are made on the underlying mean
matrix, complemented by a nearly-matching lower bound. Prior works achieving near-optimal rates for streaming PCA either required gap assumptions [JJK+16, HNWW21], or were limited to rank-one updates [AZL17, Lia23]. Our proof only uses a second moment bound on the individual
stochastic updates, bypassing the almost sure bounds needed by prior near-optimal analyses, and the analogous offline
matrix Bernstein bound. We also extend our result to a Rayleigh quotient notion of approximate PCA, addressing an open question of [JJK+16]. As our main application, we give gap-free differentially private PCA guarantees for sub-Gaussian data, settling
Conjecture 1.1 of [Bro26] up to logarithmic factors.
Read the full abstract ▾
All authors Anming Gu, Syamantak Kumar, Kevin Tian, Chutong Yang
Machine Learning · Cross-listed from cond-mat.mtrl-sci
Deep Generative Crystal Structure Prediction: A Benchmark Study and a Controlled Test of Prototype Dependence
Lai Wei, Rongzhi Dong, Ying Feng et al. (+2 more)
Deep generative models are widely reported to enable de novo
crystal structure prediction (CSP), but their capability has not been measured consistently against template-based methods. We evaluate 12 representative generative CSP models, spanning latent-variable,
diffusion, flow-matching, autoregressive, and
manifold random-walk architectures, against TCSP 2.0 on 180 test structures and a leakage-controlled subset of 46. All methods use identical structure-matching,
symmetry, and consensus criteria. Template
retrieval is the strongest single method, reaching 68.3% top-1 success;
symmetry-aware EquiCSP (66.4%) and Uni-3DAR (62.9%) form the next tier. However, comparison with TCSP 2.0 shows that most structures correctly predicted by generative models are also correctly predicted by template substitution. Thus, the set of structures uniquely reachable by generation is small, limiting its practical advantage for discovering structures outside existing prototype libraries. To test the source of this performance, we removed entire stoichiometric prototype families from the training set and retrained the strongest generative model. Accuracy declined by 50-78% across four families, establishing that performance is substantially prototype-dependent. A small minority of structures survived removal of their prototype family, demonstrating a real but limited
retrieval-independent predictive capacity. Present generative CSP models therefore function largely as implicit, softer-edged prototype libraries rather than genuinely de novo predictors. Enlarging this residual capacity, rather than aggregate match rate alone, is the central open problem.
Read the full abstract ▾
All authors Lai Wei, Rongzhi Dong, Ying Feng, Madeline Miklos, Jianjun Hu
Machine Learning
When Recursive Models Finish Computing
Hare Krishna, Shubham Singh, Stephen Ebert et al. (+1 more)
Recursive models can continue updating their latent states beyond their nominal inference budget, so an incorrect output at that budget does not show whether computation is unfinished or has entered a persistently unsuccessful regime. We study the dynamics of completion in
attention- and MLP-based Tiny Recursive Models (TRMs) on 1,000 hard Sudoku puzzles. Extending recurrence from the nominal 16 steps to 512 steps increases cumulative exact-solve accuracy from 59.2% to 87.5% for the
attention model and from 74.4% to 91.9% for the MLP model, solving more than two-thirds of the puzzles unsolved in the nominal budget. Across both architectures, latent-state motion drops sharply after the first exact solution. Completed states are typically locally contractive along the trajectory direction, even though the same local Jacobian retains strongly expanding directions. We characterize this phenomenon as trajectory-conditioned anisotropic stability. Perturbation experiments confirm this directional stability across both models. The multi-step fate of the maximally expanding direction differs: it is absorbed within 16 steps in the
attention model but persists longer in the MLP model. The anisotropic-stability pattern also holds for a second
attention checkpoint. Together, these results distinguish nominal-budget failure from completed computation and identify a common dynamical signature of completion across two recurrent architectures.
Read the full abstract ▾
All authors Hare Krishna, Shubham Singh, Stephen Ebert, Hao-Yu Sun
Machine Learning · Cross-listed from cs.CV
PP-Net: A Hybrid Physical-Prior Neural Network for Scattered Light Removal in Biomedical Images on Embedded Devices
Yongfei Guo, Tingjin Chu, Mengzhuo Liu et al. (+2 more)
Scattered light is common in biomedical images, yet its removal remains challenging. The difficulty arises from three aspects: first, aligned scattered-light-free biomedical ground truth is often unavailable; second, scattering is coupled with weak illumination and sensor-induced noise; and third, many learning-based restoration models are computationally expensive for embedded devices in Internet of Medical Things (IoMT) scenarios. To address these issues, this paper proposes PP-Net, a hybrid physical-prior neural network for biomedical scattered light removal. The proposed method consists of three components: DFN-Net suppresses sensor-induced noise, ASAP estimates the scattering map and recovers a physics-based prior map, and GF-Net refines the prior map by fusing it with the denoised observation. To reduce the dependence on paired biomedical ground truth, a progressive synthetic training and cross-domain transfer strategy is developed. Experiments show that the physical-prior branch improves the peak signal-to-noise ratio (PSNR) by up to 1.26 dB on paired synthetic
benchmarks. Under joint noise-and-scattering degradation, PP-Net improves PSNR by more than 10.8 dB and the structural similarity index measure (SSIM) by more than 0.62 compared with representative baseline methods. On real W2S biomedical images, the proposed method reduces the average Natural Image Quality Evaluator (NIQE) score by 43.3%. Edge deployment with RKNN conversion and INT8 quantization achieves an average inference latency of approximately 200 ms per $512\times512$ image over 360 test images. These results demonstrate that PP-Net provides an effective and deployable solution for microscopic imaging, endoscopic inspection, and edge-assisted biomedical analysis in IoMT scenarios.
Read the full abstract ▾
All authors Yongfei Guo, Tingjin Chu, Mengzhuo Liu, Hongwei Lou, Yuanhao Gong
Machine Learning
Can We Predict Anomaly Detection Performance from Embedding-Space Geometry?
Kevin Wilkinghoff and Zheng-Hua Tan
Anomaly detection systems are often trained using normal data alone, while model selection and evaluation typically require labeled anomalies. We study whether anomaly detection performance can be predicted without access to anomalous data. For kNN-based detectors, we derive a lower bound on the area under the ROC curve (AUC) that relates detection performance to the separation between inlier and outlier scores and to their respective variances. Under a local scaling model, we use this bound to characterize how density variation, intrinsic-dimensional heterogeneity, and cross-domain mismatch contribute to score variability. We then investigate anomaly-free model selection and show that inlier score variance alone does not reliably predict performance across different representations. To address this limitation, we introduce simple pseudo-anomaly probes that provide a reference for estimating relative score separation. Experiments on the DCASE 2022-2025
benchmarks, spanning four
embedding models and 208 candidate systems, show that pseudo-anomaly-based
estimators substantially improve anomaly-free model selection. In particular, diverse pseudo-anomalies enable anomaly-free model selection to outperform conventional development-set selection under domain shift. These results show that
embedding-space geometry contains predictive information about anomaly detection performance while also highlighting the representation-dependent nature of inlier-only performance estimates.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Recursive self-improvement of AI research agents
Dhruv Srikanth, Bingchen Zhao, Dixing Xu et al. (+2 more)
AI
agents are beginning to automate research and development across the AI stack, from improving training efficiency to optimizing inference. A natural next step is to improve the research efficiency of the
agents themselves. When an AI research
agent's own code is the object of optimization, each accepted rewrite becomes the
agent that the next round edits. We refer to this loop as recursive self-improvement. Its significance lies in a long-standing trend, in which increased cumulative spending on R&D yields diminishing returns. Sustained self-improvement offers a way to counter this trend. We present AIDE^2, a system that implements this loop for a frontier AI research
agent. It proposes changes to its own code,
benchmarks modified versions of itself on a suite of AI R&D tasks, and keeps the changes that perform best on hidden evaluations. In an autonomous 8-day run, AIDE^2 discovered seven successive improvements, ranging from a new search policy to
memory mechanisms that compress and manage the
agent's growing context. These gains generalize to four held-out
benchmarks spanning machine learning engineering, heuristic
algorithm engineering, and physics-based weather
forecasting, the last of which is out of distribution from the selection tasks. On all four, the strongest discovered
agent matches or exceeds a human-engineered production research
agent that ranks among the strongest on FML-Bench. On a separate held-out task family, the discovered
agents also exhibit reduced reward hacking, a property the loop never explicitly optimized for: the rate falls from 55% to 32% during the run, 7 percentage points below the human-engineered
agent. Together, these results show that an AI research
agent can improve its own research efficiency through recursive self-improvement, and that these gains transfer to tasks and domains the loop never encountered.
Read the full abstract ▾
All authors Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu, Zhengyao Jiang
Machine Learning · Cross-listed from stat.ML
A Practical Guide on Graphical Model Validation
Mario V. Wüthrich
This manuscript formalizes the most popular model validation tools used in general insurance actuarial modeling. These include graphical tools like calibration plots, actual-vs-expected plots, lift charts, Murphy diagrams, as well as classical statistical tools such as Bregman losses, deviance losses, elementary losses, Murphy's decomposition and Gini scores. Particular emphasis is placed on whether calibration and discrimination are studied under a policy-weighted or an exposure-weighted population measure. This distinction is crucial in ensuring that premium schemes are calibrated on the correct scale.
Read the full abstract ▾
Machine Learning
One-Step Generative Surrogate Models via Block-Triangular Joint Drifting
Nicholas Geissler, Shreya Jha, Ricardo Baptista et al. (+1 more)
Drifting provides a direct route to one-step generative models, but applying it directly to
stochastic transition modeling requires multiple samples of the next state conditioned on the same current state. Standard trajectory data, however, typically provide only one realized next state for each observed current state and therefore do not provide an empirical approximation of the corresponding conditional distribution over possible next states. We introduce block-triangular joint drifting, which instead applies a projected drift field to the empirically accessible joint distribution of consecutive states. Importantly, the block-triangular architecture preserves the current-state marginal while making its second component a direct sampler of the conditional distribution of possible next states. The resulting surrogate generates
stochastic trajectories with one model evaluation per time step, without auxiliary generative steps between time steps. Numerical experiments demonstrate accurate marginal and trajectory-dependent statistics and favorable accuracy-cost tradeoffs compared with deterministic,
diffusion-, flow-, and
distillation-based generative surrogate models.
Read the full abstract ▾
All authors Nicholas Geissler, Shreya Jha, Ricardo Baptista, Benjamin Peherstorfer
Machine Learning
DeepFEAv2: Deep Learning for Transient Finite Element Analysis Beyond Structured Meshes
Georgios Triantafyllou, Panagiotis G. Kalozoumis and Dimitris K. Iakovidis
Finite Element Analysis (FEA) is widely used for transient mechanical simulations, but its high computational cost limits real-time and high-resolution applications. Deep learning surrogate models can reduce this cost; however, many existing approaches are restricted to steady-state prediction or cannot jointly predict Node- and Element-based Outputs (NEO) over time. The state-of-the-art DeepFEA framework has addressed these issues but remains limited to structured finite element (FE) meshes. To overcome this limitation, this study proposes DeepFEAv2, a deep learning surrogate framework that enables prediction of transient FEA simulations across different FE mesh
topologies and element types. The main contributions of DeepFEAv2 are: (a) a module that uses the FE connectivity
matrix to organize input features by element and arrange them into an input sequence guided by the mesh
topology; (b) a novel neural network architecture designed to process the input sequence and jointly predict NEO over time; and (c) a FEA-informed optimization strategy for regularizing these NEO predictions. DeepFEAv2 was evaluated on structured and unstructured 3D linear elastic
datasets, as well as on a pressure-driven aortic valve
dataset. DeepFEAv2 achieved R^2 values up to 0.99 and normalized errors as low as 0.38%. Compared with DeepFEA, it achieved up to 38.0% relative increase in R^2 and up to 87.1% reduction in normalized error. DeepFEAv2 also performed inference up to three orders of magnitude faster than traditional FEA. These results demonstrate that DeepFEAv2 can efficiently model transient FEA simulations across increasingly complex FE settings, providing a scalable surrogate framework for transient FEA.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
SuperPCA: subspace analysis and an efficient algorithm for high-dimensional PCA
Irina-Beatrice Haas, Maike Meier, Yuji Nakatsukasa et al. (+1 more)
Principal component analysis (PCA) is a fundamental tool to reduce the dimensionality of the data in many applications. PCA finds a few signal directions that contain most of the variability of the data by computing the eigenvectors of the sample covariance
matrix. In this work, we focus on the spiked covariance model, in which the data vectors are defined by a few orthogonal signals plus an isotropic Gaussian noise, and our goal is to estimate one or more of the leading signals. Our main theoretical finding is that the subspace spanned by several leading eigenvectors of the sample covariance
matrix contains significant information about the desired signals long before the individual eigenvectors converge to the population principal components. To prove this, we derive a posteriori bounds for the angle between the subspace spanned by the desired population signals and the subspace obtained from the sample using perturbation theory for singular vectors. This leads to a new
algorithm, SuperPCA (SUbsPace subsamplER PCA), which capitalizes on an approximate eigenspace of the sample covariance
matrix to find the leading signals far more efficiently and accurately than classical PCA in the high-dimensional, multi-signal setting. SuperPCA exploits only a small number of subsampled coordinates of the data, which can lead to tremendous savings in data acquisition cost, especially when the signals are approximately sparse. For the same number of measurements, SuperPCA can offer a factor $10$ improvement in accuracy compared to the classical PCA method.
Read the full abstract ▾
All authors Irina-Beatrice Haas, Maike Meier, Yuji Nakatsukasa, Taejun Park