Machine Learning
SVI-DAG: A Structured Variational Inference Approach to Bayesian Causal Discovery
Shrenik Zinage
Bayesian causal discovery seeks to determine the posterior distribution of causal theories, which are interpreted as directed acyclic graphs (DAGs) that explain the observed data. The resulting posterior allows systematic
reasoning regarding epistemic uncertainty within these theories. Nonetheless, finding such graphs is difficult due to identifiability problems and limited observational data. Furthermore, precisely approximating posterior over graphs is challenging given vast range of potential DAGs. Recent
Bayesian approaches have addressed some of these challenges, yet they remain limited as they fail to encode dependencies between edges, and lack principled ways to incorporate domain knowledge as inductive biases during the search process. To overcome these limitations, we propose SVI-DAG, a structured variational inference approach to
Bayesian causal discovery using observational data and prior beliefs that uses normalizing flows to model dependencies between edges, supporting expressive and
multimodal posterior learning over DAGs. To mitigate mode seeking behaviour in evidence lower bound optimization and promote mode coverage, we use stein variational gradient descent to update the node potentials using a kernel in acyclicity space. We evaluate SVI-DAG against 5 state-of-the-art
Bayesian DAG learning methods and demonstrate competitive performance in terms of both accuracy and uncertainty quantification.
Read the full abstract ▾
Machine Learning
Optimal Training-Time Scaling in Gradual Adaptation
Zonghuan Xu and Krishna Harish
In gradual adaptation, how should the training time on each task change as the number of intermediate tasks increases? We study this question for overparameterized linear regression tasks that change smoothly and share a zero-loss solution. With $N$ tasks and training time $s_N$ on each, the final learning progress converges to a continuum curve when $Ns_N\to\tau$. The limiting progress is $\Theta(\tau)$ for small $\tau$ and $\Theta(\tau^{-1})$ for large $\tau$, so both very short and very long training produce little progress. It follows that optimal per-task training times scale as $s_N^\star=\Theta(N^{-1})$, equivalently $Ns_N^\star=\Theta(1)$. Experiments on gradually rotated MNIST and a natural Yearbook time shift are consistent with less per-task training as the path is divided more finely.
Read the full abstract ▾
Machine Learning
Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning
Xuehang Guo, Pengyuan Li, Tom Hope et al. (+3 more)
As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fundamental challenges for AI systems: the relationships across representations are inherently one-to-many, supervision is ambiguous and costly, and model optimization lacks a principled signal that is both direction-adaptive and representation-generalizable beyond task-specific objectives. We introduce CoCoEvolve to improve consistency across chart, table, and code representations. Instead of treating cross-representation mapping as a one-to-many problem, we define explicit one-to-one correspondences and optimize models using agreement between representations, without additional annotations. During training, CoCoEvolve@Train performs co-
evolution across the chart-table-code cycle, while CoCoEvolve@Test applies the same consistency objective at inference time for test-time co-optimization. We also present CoCoEvolve@Eval, an evaluation suite covering all six cross-representation tasks. Across four
benchmarks, CoCoEvolve improves performance in both training-time and test-time settings. Our project page: https://xhguo7.github.io/CoCoEvolve/.
Read the full abstract ▾
All authors Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun Wang
Machine Learning · Cross-listed from cs.CV
Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
Zehua Chen, Junyou Wang, Yuxuan Jiang et al. (+5 more)
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional
diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or
reasoning from large
multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during
diffusion training and sampling process, respectively. Extensive experiments on
benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.
Read the full abstract ▾
All authors Zehua Chen, Junyou Wang, Yuxuan Jiang, Zhenying Fang, Yusheng Dai, Jianfei Chen, Ziwei Liu, Jun Zhu
Machine Learning · Cross-listed from cs.CR
When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs
Jiaming Cheng, Subhransu Das and Rajiv Ramnath
Multi-
agent LLM systems relay key-value caches instead of text and credit their gains to exchanged "latent thoughts". That credit is a claim about which example's cache is relayed, not merely that one is. We audit it causally in released systems. The cache is replaced with deranged (mismatched-example), zeroed, and moment-matched random counterparts, under two regimes defined by whether the receiver needs the sender's private information. Where it does, the battery reads ceiling: 100% against 23-25% for answer-irrelevant relays on the primary backbone, a contrast replicated across three families, five checkpoints, and a prose document-QA surface. Where it does not, a pre-registered five-seed protocol establishes equivalence within 2.8 points, a margin anchored to the audited system's reported gain, under Holm-corrected TOST on GSM8K and ARC-Challenge across three Qwen3 scales and on MedQA at 8B (one
cell shows a small detected advantage inside the margin); a second family shows no detected advantage. A large cache effect need not be a pairing effect. In one natural
cell, zeroing the relay costs 14.7 points; a mismatched cache, 0.4. Nor is need sufficient: under the same test, delivered channels span ceiling (LatentMAS's native relay), partial (KVComm's layer subset), and no detected example-specific transfer (C2C's released projector).
Benchmark deltas do not by themselves establish latent-thought transmission; establishing it takes a mismatched-cache audit, which we release.
Read the full abstract ▾
Machine Learning
Variational Bounds for Perceptron Learning from Structured Data
Francesco Camilli, Pierluigi Contucci, Federica Gerace et al. (+1 more)
We introduce a variational approach to a finite-temperature continuous-
spin perceptron trained on a Gaussian mixture. The model allows for a broad class of concave utilities and log-concave separable prior measures on the
spins. By combining the interpolation method with log-concavity and concentration estimates, we derive lower and upper minimax variational bounds for the limiting quenched pressure. Remarkably, the two bounds differ only in the order of optimization of two variational parameters, while all remaining extrema are controlled by the concave–convex structure of the variational potential. Whenever the two optimizations commute, the two bounds match and identify the solution of the model. The same potential yields the fixed-point equations as stationarity conditions and provides a unified route to the computation of the ground-state energy, training loss, and generalization error.
Read the full abstract ▾
All authors Francesco Camilli, Pierluigi Contucci, Federica Gerace, Emanuele Mingione
Machine Learning
Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision
Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra et al. (+2 more)
Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by repeatedly applying one shared block. Rather than proposing a new architecture, we fix a bViT and provide a controlled empirical characterization of three training and inference regimes under a common CIFAR-100 protocol, asking: (i)~when does recurrence beat independently parameterized depth—at matched FLOPs or at matched parameter
memory? (ii)~when a residual recurrent block is trained through an ODE solver, does solver order act as numerical refinement or as an architectural bias? and (iii)~what does robustness beyond the training horizon cost in nominal accuracy? We find that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy–parameter trade-off under
memory constraints. Consistent with the standard view of residual networks as Euler discretizations of ODEs, the continuous-time analogue of a residual recurrent block is the state-subtracted vector field $\dot{z}=F_\theta(z)-z$; although known in principle, this distinction is easy to violate when the block is wrapped as a black-box vector field, and we qualify the cost at few accuracy points. Because the vector field is learned jointly with the solver, higher-order solvers act as a solver-induced architectural bias rather than a numerical-accuracy improvement, and their gains are not uniform. Finally, stage-wise deep supervision traces an accuracy–robustness frontier: it does not improve nominal accuracy, but degrades gracefully far beyond the training horizon, where naive recurrence collapses to near-random performance.
Read the full abstract ▾
All authors Grzegorz Gruszczynski, Pawel Olszowiec, Michal Byra, Grzegorz Stefanski, Alberto Presta
Machine Learning · Cross-listed from cs.CL
A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination
Wenxiao Zhao, Dong Liu, Kaiyi Xu et al. (+9 more)
Symbolic regression aims to discover closed-form equations from data, but existing LLM-guided methods often rely on a unified proposal loop that compresses heterogeneous search failures into a scalar score and a single
prompt. We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views. A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process
memory. During search, evaluator feedback characterizes reliability and productivity, updates role-level utilities, and routes elite motifs, failure traces, and validity diagnostics to different
agents. The framework self-evolves at two timescales: within a run, it adapts the search process without updating LLM parameters; across runs, recorded trajectories can be distilled into open-source LLMs as role-conditioned proposal priors. Averaged over the four LSR-Synth scientific domains in LLM-SRBench, A-SR improves Acc@0.01 over baselines from 25.79% to 48.30% with Llama3.1-8B, while A-SR-LoRA improves the corresponding Qwen3-4B result from 24.58% to 38.29%. On four real-world scientific discovery tasks, A-SR obtains the best in-distribution or out-of-distribution normalized mean squared error on 7 of 8 reported metrics.
Read the full abstract ▾
All authors Wenxiao Zhao, Dong Liu, Kaiyi Xu, Feng Liu, Zhen Zhao, Fei Ben, Shu Wang, Wenhao Li, Ying Nian Wu, Fenghua Ling, Haobo Li, Lei Bai
Machine Learning · Cross-listed from eess.SP
The Neural Echo: A Signal Processing Perspective for Understanding Neural Networks
Chongbiao Wang, Daniel Gaa, Joachim Weickert et al. (+1 more)
We introduce the neural echo as a tool for understanding the behavior of neural networks. It generalizes the model-based concepts of impulse responses,
diffusion echoes, and filter echoes to learning-based methods. It provides local, space-adaptive impulse responses and filter kernels for a neural network, its so-called echoes. These echoes depend on the input image and can be visualized to understand the learned dynamics of the network via an affine mapping. Neural echoes build a bridge from classical signal processing to modern explainable AI. They are very general and can be applied to both image-to-image and classification networks, with convolutional or fully connected structure, of feedforward or recurrent type, including modern
transformer networks. Network differentiability is not required. In the differentiable case, neural echoes comprise concepts based on the network Jacobian, such as saliency maps and the analysis of
adversarial perturbations, as special instances. As a simple blueprint to explain our framework, we derive neural echoes for the denoising convolutional neural network (DnCNN). Our experiments suggest that this network weights pixels based on their spatial and gray value distances. This not only clarifies its behavior, but also shows that it can reproduce key concepts of classical model-based denoisers such as bilateral filtering.
Read the full abstract ▾
All authors Chongbiao Wang, Daniel Gaa, Joachim Weickert, Karl Schrader
Machine Learning · Cross-listed from stat.ME
Nonparametric Goodness-of-fit Testing under Covariate Shift
Zhen Hou and Dong Xia
This paper develops procedures for nonparametric goodness-of-fit testing under
covariate shift, where labelled data are drawn from a source population but goodness-of-fit is evaluated for a target population. The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio. Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function. The truncation stabilizes the importance- weighting kernel ridge regression as well as the bootstrap calibration, making our approach applicable even when the density ratio has heavy tails. We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions, and establish explicit error rates for coverage probability under specific conditions on the target- to-source density ratio and on the spectral decay of the kernel integral operator. Numerical experiments corroborate our theoretical findings.
Read the full abstract ▾
Machine Learning
Robust Control under Stationary Ambiguity
Konrad J. Mueller, Amira Akkari, Ben Wood et al. (+1 more)
Control policies optimized in simulation can perform poorly in the real system when the parameters $x$ of the simulator are estimated from limited data but the resulting parameter uncertainty is not represented inside the simulation. A common way to incorporate such ambiguity is to simulate each trajectory of the system under a randomly drawn value for $x$. Since the policy cannot observe the drawn value, it must initially choose controls that perform well across many possible parameter values. However, if the policy progressively observes the system, it can often gradually infer the value of $x$, so that ambiguity vanishes. Over time, the policy then specializes to its estimate of $x$ and loses its robustness. This is undesirable in many real systems, where latent factors are expected to shift. In financial
markets, for example, a policy hedging a derivative payoff should remain robust to changes in the
volatility regime. To induce such continual robustness, we propose training policies in simulators where ambiguity varies with the system's state but does not systematically decay over time. We formalize this requirement as stationary ambiguity: the simulator should induce a stationary filter process over the latent state. We show how to construct such simulators and demonstrate, on hedging problems, that policies trained under stationary ambiguity preserve robustness to latent factors over time, leading to strong performance on real
market data. As a modeling principle, stationary ambiguity informs many simulator design decisions: which models make realistic simulators, how their parameters should be randomized, and how simulator and policy should be initialized. While our experiments focus on hedging, stationary ambiguity may also be useful for other sequential control problems driven by exogenous
stochastic processes with shifting latent structure.
Read the full abstract ▾
All authors Konrad J. Mueller, Amira Akkari, Ben Wood, Lukas Gonon
Machine Learning · Cross-listed from stat.ML
Intrinsic-Hybrid Latent Diffusion Models for Generative Modeling on Unknown Manifolds
Yizhu Wang, Mu Niu and Xiaochen Yang
We introduce the Intrinsic Hybrid Latent
Diffusion Model (ILDM), a generative framework that integrates probabilistic dimensionality reduction with geometry-aware
diffusion on unknown
manifolds. While
diffusion models (DMs) have achieved state-of-the-art results in high-dimensional data synthesis, they rely on large training
datasets and ignore intrinsic
geometric structure. Latent
diffusion models (LDMs) address the high dimensionality by learning a latent space, but they typically impose a Euclidean structure, failing to capture the underlying
manifold geometry, especially problematic in data-sparse regimes. ILDM addresses these limitations by interpreting the latent space as a chart of an unknown Riemannian
manifold, with geometry and uncertainty quantified through a probabilistic decoder. The forward process is a hybrid
diffusion that switches between Riemannian and Euclidean dynamics based on local uncertainty, where the Riemannian component is governed by a probabilistic metric tensor derived from the decoder. To learn the generative dynamics, we introduce an approximate denoising score matching method tailored to the hybrid
diffusion setting, enabling a backward process defined by hybrid Langevin dynamics. Experiments on COIL-100, MNIST, and cardiac MRI
datasets demonstrate that ILDM significantly improves generation quality, achieving lower FID and LPIPS scores compared to standard
diffusion and latent
diffusion models.
Read the full abstract ▾
Machine Learning
MGSB: Manifold Gated Signature Branch Pressure-Domain Baseline Architecture for Two-Phase Pipeline Flows Under Distributional Shift
Issah Suleiman, Sormeh Serpoosh, Nadine Elkholy et al. (+3 more)
Leak detection models for multiphase pipelines often degrade when deployed under flow regimes that differ from training. Existing evaluations typically assess performance under in-distribution operating conditions, masking failures caused by regime transitions such as bubble-to-slug flow. We propose the
Manifold Gated Signature Bias (MGSB), a regime-aware architecture combining regime-conditioned feature fusion, a TT-RoughPath encoder, and Mean-Teacher consistency regularization to improve robustness under distribution shift. Under leave-one-group-out evaluation, MGSB achieves a detection F1 of 0.930 and an OOD F1 of 0.783, substantially outperforming CNN-LSTM and fully connected baselines under severe feature corruption. Ablations show the proposed architecture, not the training procedure, is the primary contributor to OOD robustness, while Mahalanobis-distance analysis confirms the held-out conditions are genuinely out-of-distribution. These results show that explicit regime-aware modelling is a practical path toward robust, sensor-agnostic leak detection in industrial multiphase pipelines.
Read the full abstract ▾
All authors Issah Suleiman, Sormeh Serpoosh, Nadine Elkholy, Hicham Ferroudji, Mohammad Azizur Rahman, Matthew Hamilton
Machine Learning · Cross-listed from cs.AI
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
Sarthak Harne, Chinmay Karkar, Yash Pandya et al. (+2 more)
Self-
distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-
token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across
reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-
token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-
token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information
tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory
tokens incur the highest divergence, so it penalizes the hesitation that
reasoning requires. The result is a flatter, less decisive student that is no better at
reasoning: as a lone objective, SD optimizes a signal decoupled from task success.
Read the full abstract ▾
All authors Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi
Machine Learning
Multi-Modal, Multi-Temporal Geospatial Foundation Models Improve Global Above-Ground Biomass Estimation
Ghjulia Sialelli, Linus Scheibenreif, Jan Dirk Wegner et al. (+1 more)
Accurate estimation of above-ground biomass (AGB) from satellite imagery is essential for the large-scale monitoring of carbon stocks, yet it remains a challenging regression task at global scale. Geospatial Foundation Models (GFMs) have recently emerged as a promising machine learning paradigm to derive general-purpose representations from Earth observation (EO) data, but their utility for quantitative regression tasks like biomass estimation remains largely unexplored, as most
benchmarks focus on classification and
segmentation. Here, we
benchmark 13 GFMs for above-ground biomass estimation, specifically GEDI L4A biomass values. We compare them against a fully supervised state-of-the-art (SOTA) baseline trained on A Global-scale Biomass
Dataset (AGBD), a machine learning-ready
benchmark spanning diverse biomes and geographies. Most of the GFMs evaluated substantially underperform with respect to the supervised baseline, whether frozen or fine-tuned; only TESSERA and AlphaEarth Foundations (AEF) features with certain prediction heads yield a better performance. Our SOTA model trained on AEF
embeddings and selected raw EO features achieves the best overall result (RMSE of 50.79 Mg/ha vs. 53.73 Mg/ha on AGBD features). This feature combination also generalizes better than AGBD features: it reduces RMSE by up to 25% in selected regions excluded from training, and, trained on single-year data alone, it outperforms the model trained on multi-year AGBD features. With labels from only 11 regions around the world, the AEF-based model also performs on par with the globally calibrated ESA CCI biomass product on independent reference plots. These findings indicate that GFM features can be highly informative for biomass estimation, when the underlying model is trained on rich multi-modal data.
Read the full abstract ▾
All authors Ghjulia Sialelli, Linus Scheibenreif, Jan Dirk Wegner, Konrad Schindler
Machine Learning
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
Yi Yang, Cong Qin, Xiaodan Liu et al. (+8 more)
Large
language model
agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual
tokens should be updated. On-Policy Self-
Distillation (OPSD) addresses this by re-scoring generated
tokens under a privileged replay view to obtain dense,
token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs
token scores. To resolve this confounding, we propose Observation-Calibrated Self-
Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate
token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.
Read the full abstract ▾
All authors Yi Yang, Cong Qin, Xiaodan Liu, Chishui Chen, Qing Dong, Yan Zhang, Cao Liu, Zhao Yang, Lu Pan, Jiaye Lin, Yi Feng
Machine Learning
Continual-Learning Physics-Informed Neural Networks for Parameterized Partial Differential Equations
Xujia Chen, Xinyue Hu, Letian Chen et al. (+2 more)
Physics-informed neural networks (PINNs) incorporate governing equations into neural-network training and can approximate PDE solutions without requiring large observational
datasets. Parameterized PINNs (ParamPINNs) further take physical parameters as inputs, allowing a single model to represent a family of PDE solutions over a parameter domain. Existing ParamPINNs, however, still face inefficient training, uneven accuracy across parameters, and overfitting to a limited set of sampled parameter tasks, which can impair generalization to unsampled parameters. To address these issues, we propose a continual-learning physics-informed neural network (CL-PINN), which treats PDE instances at different parameter values as related tasks and learns them sequentially. CL-PINN combines
Bayesian-optimization-based active parameter selection, task-wise dynamic loss weighting, sparse physics-constrained replay, and an optional parameter subnetwork to improve task allocation and knowledge retention under bounded active-task capacity. It requires no observational data and is designed to solve parameterized PDEs over relatively broad parameter domains under limited computational resources. Multi-seed evaluations on five
benchmarks, including one continuous function and four parameterized PDEs, show that
Bayesian selection substantially reduces objective-loss queries relative to grid-greedy search, while sparse replay mitigates forgetting of earlier tasks. Under the prescribed within-case resource protocols, CL-PINN generally provides higher and more balanced solution accuracy than fixed-sampling and grid-greedy baselines. CL-PINN offers a practical route toward learning PDE solutions that generalize across physical parameters and has the potential to support reusable physics-informed surrogates for large-scale engineering parameter studies.
Read the full abstract ▾
All authors Xujia Chen, Xinyue Hu, Letian Chen, Yi Liu, Wenhui Fan
Machine Learning
IMFACT: Counterfactual Explanations for Time Series via Intrinsic Mode Function Substitution
Udo Schlegel, Julian Rakuschek, Thomas Seidl et al. (+3 more)
Oscillatory signals, such as vibration, carry class-discriminative information in specific frequency bands; perturbing them in raw feature space for counterfactual analysis easily destroys their temporal structure and produces physically implausible results. In this work, we introduce IMFACT (IMF-based counterfACTuals), a model-agnostic framework for generating plausible counterfactual explanations for time series classifiers that operates in the decomposition space of Empirical Mode Decomposition. An input signal is split into Intrinsic Mode Functions (IMFs), and selected IMFs are progressively substituted with those of a Nearest Unlike Neighbour (NUN) until the classifier flips to the target class. We evaluate six IMF-selection strategies and a multi-NUN cycling extension on two UCR
benchmarks (FaultDetectionA, FruitFlies). The variance-based strategy with three NUNs outperforms two prominent baseline techniques on reliability and plausibility metrics, while cycling across three NUNs yields the best proximity across both
datasets.
Read the full abstract ▾
All authors Udo Schlegel, Julian Rakuschek, Thomas Seidl, Andreas Holzinger, Tobias Schreck, Javier Del Ser
Machine Learning
Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection
Mihailo Ilić, Miloš Savić, Vladimir Kurbalija et al. (+3 more)
Attention layers are the backbone of today's most powerful and impactful models. Models with multi-million and billion parameters rely on contextual knowledge provided by
attention layers. However, their use goes well beyond just being the core component of large
language models. One particularly interesting application is in
Memory Augmented Autoencoders (MemAE), specifically for unsupervised representation learning in outlier detection tasks. It was shown that
attention helps these models be more effective in centralized learning scenarios. Our work aims to address the lack of specialized aggregation techniques in
Federated Learning (FL) when it comes to MemAE models. In this paper we analyze the intricacies of the architecture behind
Memory Augmented Autoencoders, and propose novel, guided approaches to effectively aggregate these models in
federated scenarios. We demonstrate our approach on non-IID
datasets and show that these novel aggregation schemes are more robust when dealing with numerous edge nodes in environments with unbalanced
datasets, specifically for unsupervised anomaly detection scenarios. This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.
Read the full abstract ▾
All authors Mihailo Ilić, Miloš Savić, Vladimir Kurbalija, Mirjana Ivanović, Giancarlo Fortino, Dušan Jakovetić
Machine Learning · Cross-listed from cs.SE
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend
Shahed Masoudian, Passant Shafaei, Monorama Swain et al. (+1 more)
Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six
benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence
benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias
benchmarks. Overall,
benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.
Read the full abstract ▾
All authors Shahed Masoudian, Passant Shafaei, Monorama Swain, Markus Schedl
Machine Learning
A 6G Integrated Sensing and Communication Framework for Railway Intrusion Detection and Collision Prediction
Ajeet Kumar Yadav, Sankaran Balasubramaniam, Aritra Chatterjee et al. (+3 more)
Integrated Sensing and Communication (
ISAC) combines sensing and communication to efficiently utilize wireless resources and is emerging as a key paradigm for next-generation wireless networks. By leveraging the wide bandwidth, high frequencies, and massive
antenna arrays of 5G-Advanced and 6G systems,
ISAC enables physical-layer sensing using Channel State Information (CSI). The 3rd Generation Partnership Project (3GPP) Release 19 identifies 32 potential
ISAC use cases, with particular emphasis on detecting and tracking moving objects. In this work, we address the Sensing for Railway Intrusion Detection use case, where intruders, including wildlife, entering a railway track can pose serious collision risks. We generated 22,695 CSI
matrices with corresponding ground truth using a 3D-rendered railway environment and the Sionna
radio simulator. We developed a machine learning model combining a three-dimensional Convolutional Neural Network (3D CNN) and Bidirectional Long Short-Term
Memory (BiLSTM) network to detect intruders in the track danger zone and estimate their real-time position relative to the train, velocity, and time to collision. On synthetic CSI data, the model achieves 99.57% intruder-detection accuracy on a balanced test set and a combined Mean Absolute Error (MAE) of 0.4240 for position, velocity, and time-to-collision prediction. These results demonstrate the potential of CSI-based
ISAC sensing with machine learning for reliable railway intrusion detection. The complete codebase for CSI generation, preprocessing, and model development is publicly available at https://github.com/EdgeIntelligenceLab/6g-
isac-railway-intrusion-detection.
Read the full abstract ▾
All authors Ajeet Kumar Yadav, Sankaran Balasubramaniam, Aritra Chatterjee, Vinod Aduru, Yogesh Simmhan, Pandarasamy Arjunan
Machine Learning
Benchmarking Deep Learning Models for Dense Event Classification of Offshore Wind Infrastructure in Sentinel-1 Time Series
Thorsten Hoeser, Felix Bachofer and Claudia Kuenzer
Monitoring of offshore wind energy infrastructure life cycles, especially during the deployment phase, is an important contribution for stakeholders to make informed decisions in a phase of increasing deployment activities. ESA's Sentinel-1 Synthetic Aperture Radar (SAR) mission produces large data archives that enable the global monitoring of offshore wind infrastructure. Turning these high-volume archives into information requires
algorithms that automatically extract single event labels from dense time series at a global scale. In this study, we present a structured comparison of ten deep learning model-training variants for the dense classification of Sentinel-1 based offshore wind infrastructure time series, aiming to advance rule-based event classification of this task. We trained LSTM,
Transformer, and fully connected model variants with monotemporal, unidirectional, and bidirectional context awareness, each with and without self-supervised pretraining. Among these, the supervised BiLSTM performs best, raising the target AUC score from 0.7853 for the rule-based baseline to 0.8509, and the perfect match rate from 0.3508 to 0.5063. Combining the BiLSTM predictions with the existing baseline labels in a label-transition-minimising ensemble further improves agreement with the test data. Using these improved labels, we isolate the deployment phase of individual turbines at a global scale and conduct a regional and subregional analysis covering 2016-01-01 to 2025-03-31, reporting median deployment durations of 84 d (China), 242 d (EU), and 258 d (UK). Deployment-related drivers, including legal regulations such as subsidies, and environmental conditions, emerge clearly from the analysed results across multiple spatial scales.
Read the full abstract ▾
Machine Learning
Design Choices That Matter: A Functional ANOVA Analysis for Remote Sensing Multi-Label Classification
Maryam Gholami Shiri, Eva Tuba, Sašo Džeroski et al. (+2 more)
Benchmarking deep learning (DL) models for multi-label classification (MLC) of remote sensing images (RSI) typically yields rankings that do not generalize beyond the evaluated
datasets. In this work, we move beyond rankings by employing functional analysis of variance (fANOVA) to systematically quantify the contributions of individual design choices and their interactions to performance variability. We conduct two empirical analyses covering 48 and 20 DL models, respectively, spanning design choices such as network architecture, fine-tuning strategy, learning strategy, and initialization. By applying fANOVA across seven MLC RSI
datasets, we construct
dataset meta-representations that capture design-choice sensitivity profiles. Hierarchical clustering of these meta-representations reveals that
datasets naturally group according to how they respond to design decisions, with patterns strongly linked to intrinsic
dataset properties such as scale, spatial resolution, and label space complexity. Our findings show that for large-scale
datasets, fine-tuning strategy and architecture are dominant factors, while in data-limited regimes, initialization becomes decisive. For intermediate regimes, the interaction between architecture and learning strategy governs performance.
Read the full abstract ▾
All authors Maryam Gholami Shiri, Eva Tuba, Sašo Džeroski, Tome Eftimov, Ana Nikolikj
Machine Learning
Personalized Federated Sparse Adaptation of Time-Series Foundation Models
Priyanka Nihalchandani, Naman Srivastava, Varun Ojha et al. (+1 more)
Federated adaptation of time-series foundation models (TSFMs) is attractive for building energy
forecasting because meter data are private, distributed, and highly non-IID. However, a single parameter-sharing strategy is unlikely to serve all pretrained TSFMs or building clients: fully shared adapters can suppress building-specific temporal behavior, while fully local adaptation discards cross-building transfer. We propose a personalized
federated sparse adaptation framework with a heterogeneous temporal mixture-of-experts (MoE) adapter placed after the pretrained TSFM representation. A sequence-level router maps each 168-hour context window to a top-$k$ subset of experts specialized for periodicity, long-range interactions, local variation, trend-residual structure, and multi-resolution behavior. We compare global FL, local training, and personalized FL variants with globally shared or client-private expert banks. Across 50 buildings and three TSFM backbones, personalization consistently outperforms Global FL-MoE and Local MoE, while the best sparse-adaptation strategy varies by backbone and metric. Routing behavior further reveals client-level expert specialization, expert concentration, and near-uniform routing across backbones, showing that
federated TSFM adaptation should be both client-aware and backbone-aware.
Read the full abstract ▾
All authors Priyanka Nihalchandani, Naman Srivastava, Varun Ojha, Pandarasamy Arjunan
Machine Learning · Cross-listed from cs.RO
Suppression Sticks, Locality Is Fragile: A Closed-Loop Target-and-Control Audit of Task-Vector Negation in VLA Policies
Shaoguang Wang, Weiyu Guo, Rushi Dai et al. (+3 more)
Task-vector arithmetic offers a closed-form way to modify a model, yet its behavioral locality remains unclear in closed-loop
robot control. We present a target-and-control audit of per-skill task-vector subtraction from multitask
vision-
language-action (VLA) policies. Across all ten LIBERO-Goal skills, subtraction produces three qualitatively different regimes: target-control separation for five skills, resistance for three, and global collapse for two. On held-out initial states, the five suppressible targets remain at 0% success; however, mean baseline-normalized control retention is only 52%, and each target-suppressing edit materially harms at least one nominally unrelated control. Additional Goal panels show separation across tested policies with continuous-regression, discrete-
token, and flow-matching action heads, whereas we observe no clean separation on Spatial and control collapse on the tested Object and Long-horizon panels. Mean task-vector cosine does not account for this variation. A matched-norm control identifies a local sign asymmetry around one Goal anchor, while multi-vector outcomes vary with anchor and scale. Retain-aware gradient baselines provide data-dependent comparators but require removal-time data and optimization; subtraction is data- and gradient-free only at edit time, assuming precomputed expert deltas. Finally, a single-skill relearning probe is consistent with behavioral masking, not certified unlearning. These results characterize task-vector subtraction as a fast but brittle intervention and underscore the need for closed-loop target-and-control evaluation when assessing locality in
embodied model editing.
Read the full abstract ▾
All authors Shaoguang Wang, Weiyu Guo, Rushi Dai, Yiren Zhao, Yandong Guo, Hui Xiong
Machine Learning
The Sample Complexity of Distributionally Robust PAC Learning under Cressie–Read Divergences
Elad Aigner-Horev, Daniel Rosenberg and Roi Weiss
We study distributionally robust PAC learning for the $0$–$1$-loss, where
adversarial perturbations of the data distribution are constrained by a Cressie–Read divergence of order $k>1$ and radius $\rho\geq 0$. For hypothesis classes with VC dimension $d$, we establish realizable and agnostic sample-complexity bounds tight up to constant and logarithmic factors, respectively; ordinary empirical risk minimization attains both rates up to logarithmic factors. For target accuracy $\varepsilon\in(0,1)$ and confidence $\delta\in(0,1)$, their respective orders are \[ \max\!\left\{\frac{1}{\varepsilon}, \frac{\rho^{\frac 1{k-1}}}{\varepsilon^{k_\star}} \right\}\cdot(d+\log \delta^{-1}) \qquad\text{and}\qquad \max\!\left\{\frac{1}{\varepsilon^2}, \frac{\rho^{\frac1{k-1}}}{\varepsilon^{k_\star\vee 2}} \right\}\cdot(d+\log \delta^{-1}), \] where $k_\star={k}/{(k-1)}$. For every fixed $\rho>0$, robustness changes the realizable $\varepsilon$-dependence from $\varepsilon^{-1}$ to $\varepsilon^{-k_\star}$ as $\varepsilon\downarrow0$. In the agnostic case, for $1<k<2$, robustness changes the $\varepsilon$-dependence from $\varepsilon^{-2}$ to $\varepsilon^{-k_\star}$, whereas for $k\geq2$ the exponent remains the classical $2$, with nontrivial $\rho$-dependence. Building on the known scalar reduction of robust $0$–$1$ risk to ordinary classification error, our analysis reveals a scale-sensitive interaction between the statistical estimation of classification error and its amplification by robustness, sharply explaining the transition in the agnostic rate. We extend the previously studied $\chi^2$-divergence case to every Cressie–Read order $k>1$, close its upper–lower gaps, and recover standard PAC learning rates as $\rho\to0$, unlike previous bounds that fail to interpolate correctly in this limit.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
George Fountzoulas
Papers 1-2 of the Kathleen series showed that a byte-level,
attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining. We ask whether the same ingredients can generate. (1) Scaling: on byte-level
language modeling (WikiText-103, raw UTF-8, no tokenizer), the reverberant model beats a parameter-matched
transformer at every
dataset scale measured (2-512 MB), e.g. 1.84 vs 2.04 bits/byte at 512 MB with ~0.5M parameters; the
transformer needs more than 512 MB to match what the
attention-free model learns from 32 MB. (2) Measurement: we introduce FORM DISTANCE, a non-parametric, gaming-resistant instrument for "reads like text": nine statistical axes of human text define a reference cloud, and five constructed fakes are all rejected. (3) Generation: decoding policy dominates architecture – widening the sampler halves the same model's distance (3.17 to 1.52), and a
retrieval-augmented decoding scheme takes the frozen model further (1.52 to 1.14) with no training step involved; the ablation attributes the gain to the sparse phrase dose itself, not the selection gate. The gain has a sharp boundary condition: the phrases must come from the model's own training corpus – a 40x larger foreign library helps not at all, an effect the
attention twin shares, consistent with in-context integration being a capability of scale. We also report four architectural additions that did not help, and a computed lexicon reaching 94% of a learned table's top-1 accuracy at one fifth of the parameters. Everything runs offline; all experiments are reproducible on a free Kaggle T4.
Read the full abstract ▾
Machine Learning
Diverse and Plausible Algorithmic Recourse via Tractable Recourse Distributions
Anagha Sabu, Hrithik Suresh and Narayanan C. Krishnan
Algorithmic recourse seeks to help individuals reverse unfavorable automated decisions by recommending actionable changes that achieve a desired outcome. As an individual usually has several distinct routes to a favorable decision, and different people can act on different ones, a recourse system should offer multiple realistic alternatives rather than one. Existing approaches formulate recourse as an optimization problem that constructs one or a small set of counterfactuals rather than modeling the underlying space of feasible solutions, and in practice each sacrifices diversity, plausibility, or feasibility to secure the others. We propose Tractable Recourse Distributions, a probabilistic framework that represents the space of feasible alternatives for a given factual instance as a probability distribution over favorable outcomes. For commonly used cost functions based on proximity and the number of feature changes, we show that this distribution admits an exact representation as a probabilistic
circuit, obtained by exponentially tilting the
circuit; each individual's distribution is therefore available in closed form, without retraining the model. Sampling from these distributions naturally produces diverse and plausible recourses, while the tilting parameters provide explicit control over their proximity and sparsity. Experiments on standard algorithmic recourse
benchmark datasets demonstrate that the proposed framework attains diversity, plausibility, and feasibility simultaneously, while retaining sufficient probability mass over feasible counterfactuals for rejection sampling to be practical. A visual study on MNIST illustrates how the tilt strength trades proximity against validity.
Read the full abstract ▾
Machine Learning
Differentiating Through Dual Prices: End-to-End Policy Learning Under Capacity Constraints
Mohammadsaeed Haghi, Mahdi Salmani, Nima Kelidari et al. (+2 more)
Many social services assign scarce resources, such as housing assistance or hospital interventions, to people who arrive one at a time. Each arrival must receive a decision immediately, and the long-run usage of every resource must stay within its capacity. We study how to learn such a policy from logged observational data. Recent work (Tang et al., 2024) proposes a decision-blind pipeline that fits one outcome model per treatment by regression, prices each capacitated resource from those models, and assigns each arrival the treatment with the largest predicted outcome minus price. We instead train the outcome models end-to-end, differentiating an off-policy estimate of the deployed policy's value through the dual prices themselves. We study two formulations: an exact nonconvex one, and a convex relaxation whose optimum always satisfies the capacity constraints in expectation and whose suboptimality can be explicitly bounded. Every method is evaluated in a queueing simulation to capture the online decision structure. Across six
datasets, our two end-to-end variants lead the delay-adjusted policy value wherever capacity is scarce, once a waiting period carries even a small cost. In particular, decision-blind regression predicts ground truth more accurately, but when resource capacities are stringent these baselines frequently violate them and incur much longer queueing delays. Thus, end-to-end training is best suited to settings where resources are genuinely scarce and feasibility matters.
Read the full abstract ▾
All authors Mohammadsaeed Haghi, Mahdi Salmani, Nima Kelidari, Vishal Gupta, Phebe Vayanos
Machine Learning · Cross-listed from stat.ML
Automatic Statistical Test for Rationally Expressible Algorithms by Selective Inference, with Applications to Feature Selection
Teruyuki Katsuoka, Tomohiro Shiraishi, Shuichi Nishino et al. (+1 more)
Selective inference (SI) provides statistically valid $p$-values for hypotheses selected by applying an
algorithm to the data, correcting for the bias that arises when the same data are used both to select and to test a hypothesis. Developing an SI procedure for a new
algorithm, however, has required an expert to derive, and then implement, the selection event, i.e., the conditions under which the hypothesis is selected. Repeating this specialized effort for every new
algorithm is why exact SI has so far been available for only a narrow class. We propose AutoSI, a framework that removes this barrier in two ways. First, AutoSI constructs the selection event automatically from the
algorithm's individual operations, so the user only writes the
algorithm as ordinary NumPy-like code and derives nothing by hand. Second, AutoSI broadens the class of selection events SI can handle: existing exact methods are limited to selection events characterized by linear or quadratic inequalities in the data, whereas AutoSI covers any
algorithm expressible through rational functions of the data (ratios of
polynomials). We prove that the $p$-values computed by AutoSI are exactly valid in finite samples. We demonstrate AutoSI on three feature-selection methods, each written in a few dozen lines of code. One of these methods, the lasso with its tuning parameter selected by cross-validated $R^2$, cannot be handled within existing exact SI frameworks and is made possible by AutoSI. Experiments on synthetic and real
datasets show that the resulting $p$-values control the type I error rate (i.e., the false positive rate) at the nominal level while retaining high power.
Read the full abstract ▾
All authors Teruyuki Katsuoka, Tomohiro Shiraishi, Shuichi Nishino, Ichiro Takeuchi