Machine Learning · Cross-listed from cs.CV
A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT
Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy et al. (+5 more)
Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload. To address these challenges, we introduce a novel multi-center
benchmark for multi-organ abdominal disease diagnosis and automated radiology report generation, which learns to synthesize contrast-enhanced findings from single-phase non-contrast CT (NCCT). To support this, we curated a large-scale
dataset of paired NCCT-CECT studies and their corresponding contrast-enhanced radiology reports from two centers, partitioned into internal sets and an external validation cohort. Under a unified evaluation protocol, we benchmarked five contemporary deep learning architectures encompassing chest-specific, abdomen-specific, and general-purpose
multimodal domains. Extensive experiments demonstrate that NCCT retains diagnostic signals, achieving an average multi-organ AUC of 69.1% on the internal cohort and 63.1% on the external cohort, respectively. By releasing this
dataset and standardized
benchmark publicly, this study aims to catalyze future research into safer, resource-efficient, and globally accessible contrast-free abdominal imaging workflows. Code is available at: https://github.com/xmed-lab/TriALS-Report.
Read the full abstract ▾
All authors Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel
Machine Learning
Analytic Torsion and Spectral Gap Capture Persistent-Laplacian Performance
Jernej Grlj and Aaron D. Lauda
While persistent
Laplacians (PL) offer a richer
geometric representation of data than persistent homology, utilizing their full eigenspectrum for learning tasks is often hampered by high dimensionality and the “varying length” problem across different filtration scales. We propose a compact spectral representation that distills the persistent
Laplacian into three mathematically grounded invariants: Betti numbers, the spectral gap, and analytic torsion. Across
benchmark datasets including MNIST, QM-3D, and SKEMPI WT, we demonstrate that this reduced feature space captures the essential predictive signal of the full spectrum, and in some cases outperforms it, while significantly reducing computational overhead and preventing the noise introduced by higher-frequency
eigenvalues. Our results suggest that these invariants provide a principled, fixed-length interface between spectral geometry and topological learning.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SE
Agent trajectories as programs: fingerprinting and programming coding-agent behavior
Hamidah Oderinwale
Benchmark scores tell you what an
agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing
agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten
agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct
agent at 85.7% accuracy. We develop procedural representations for
agent problem-solving procedures with an emergent vocabulary induction technique which is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation
dataset SWE-Bench to study the structural distinctness of
agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for
auditing and evaluating
agents for how they approach tasks at a procedural level given their traces in a top-down fashion. ProcGrep is available at https://github.com/hamidahoderinwale/procgrep.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Dynestyx: A Probabilistic Programming Library for Dynamical Systems
Daniel Waxman, Dmitry Batenkov, John Feser et al. (+4 more)
State-space models (SSMs) are the standard formalism for
Bayesian treatment of dynamical systems, with natural applications in statistics, signal processing, and machine learning. Despite their importance in both theory and application, dynamical systems have proven difficult to incorporate in modern probabilistic programming
languages (PPLs), making state-of-the-art methods less accessible to practitioners and introducing friction in following the "
Bayesian workflow." We introduce dynestyx, a probabilistic programming library with first-class support for SSMs, including state-of-the-art methods in the estimation of both states and parameters. Through a single, unified interface, users may specify arbitrary priors for discrete-time or continuous-time dynamical systems, perform inference over mixed-effect data, and make state and parameter estimates with principled uncertainty quantification.
Read the full abstract ▾
All authors Daniel Waxman, Dmitry Batenkov, John Feser, Andy Zane, Eli Bingham, Youssef Marzouk, Matthew E. Levine
Machine Learning · Cross-listed from cs.DB
Decoupling Inference from State Updates in Low-Latency Feature Engines via Probabilistic Thinning
Augusto Peres, Iker Perez, Pedro Valdeira et al. (+4 more)
Streaming data systems increasingly underpin Machine Learning workflows that maintain large numbers of continuously updated aggregations. In production settings, each incoming event typically triggers read-modify-write operations to persistent storage, making high-frequency state updates a dominant source of latency, contention, and operational cost. In this work, we decouple inference from state persistence in streaming Machine Learning pipelines via probabilistic thinning: every event is scored, but durable state updates are selectively triggered by informative events. Unlike approaches that shed input or state, we show that persistence-path control is achievable without a high-frequency in-
memory control plane or cross-worker coordination, relying exclusively on approximate statistics retrieved from disk-backed key-value stores. We model the resulting
stochastic processes, derive bounds on filtering rates, and prove that common time-based aggregations remain unbiased under variance-aware formulations, preventing systemic error accumulation. We evaluate the approach in a controlled setting that isolates per-event costs, demonstrating substantial reductions in storage Input/Output and serialization overhead. Across experiments, up to 90% of events are excluded from the persistence path while preserving and in some cases improving downstream utility.
Read the full abstract ▾
All authors Augusto Peres, Iker Perez, Pedro Valdeira, Guilherme Jardim, Ana Sofia Gomes, Hugo Ferreira, Pedro Bizarro
Machine Learning
Scalable Pairwise Kernel Learning with Stochastic Vec Trick
Napsu Karmitsa, Tapio Pahikkala and Antti Airola
Pairwise learning is a specialized form of supervised learning that focuses on predicting outcomes for pairs of objects. In this work, we introduce SPaiK, a new scalable kernel learning method tailored for pairwise settings. Our approach preserves the expressive power of kernel methods while substantially reducing computational and
memory requirements. The key innovation is the
stochastic generalized vec trick (sGVT), a
stochastic extension of the sparse Kronecker product multiplication
algorithm, which enables efficient large-scale training with pairwise kernels. By incorporating sGVT, SPaiK makes it possible to apply kernel-based pairwise learning to
datasets of a size previously out of reach. We evaluate the performance of SPaiK on seven real-world drug-target affinity
datasets and compare the results with state-of-the-art methods in pairwise learning.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Task-Error Residual Learning for Real-Robot Five-Ball Juggling
Kai Ploeger, Alap Kshirsagar and Jan Peters
For residual learning that refines existing behavior, sample efficiency depends on two things: how much information each rollout returns, and how efficiently the learner uses that information. Reinforcement learning's standard scalar reward carries far less information than the directional task error that defines the task. Random exploration further discards whatever information each rollout returns. Through residual learning with directional task-error supervision and a task error model that drives sample selection, we achieve stable three-, four-, and five-ball juggling on anthropomorphic Barrett WAM arms. Despite planning and controlling through a simple, idealized stack, the system converges from the second attempt. The first attempt drops, after which task error decreases monotonically without further failures. In comparison, five-ball juggling typically takes humans years of practice. We compare residual learners across two ternary axes, the directional information in the learning feedback and the commitment of the analytic prior, spanning Newton-style Jacobian updates, Composite
Bayesian Optimization, and
stochastic search methods. Both axes prove necessary for fast convergence: neither directional feedback nor an informative prior alone matches their combination, and the simplest method that combines them, a fixed-Jacobian Newton update, is the fastest and fully reliable. The learned residual tolerates substantial prior misalignment and degraded joint tracking, affecting mainly convergence speed. The bottleneck for residual learning on real
robots is therefore the information content of the supervision signal and how the learner uses it, not the accuracy of the surrounding stack.
Video documentation of all experiments is available at https://kai-ploeger.com/residual-juggling.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy
Baicheng Li, Haizhao Yang and Shijun Zhang
In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in $W^{2,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy, measured in the $W^{1,\infty}$-norm, by a fixed-size neural network using the Elementary Universal Activation Function ($\mathrm{EUAF}$). To extend this result to $W^{s,\infty}((a,b)^d)$ for $s\in\mathbb{N}$, we introduce a smooth activation $\mathrm{DUAF}_{\infty}$ from the family of Differentiable Universal Activation Functions ($\mathrm{DUAF}_n$). We prove that any function in $W^{s,\infty}((a,b)^d)$ can be approximated with arbitrary accuracy in the $W^{s-1,\infty}$-norm by a fixed-size $\mathrm{DUAF}_{\infty}$-activated network. We further construct sigmoidal variants $\widetilde{\mathrm{DUAF}}_n$ and show that, for every $1\leq s\leq n$, fixed-size $\widetilde{\mathrm{DUAF}}_n$-activated networks still approximate any $f\in W^{s,\infty}((a,b)^d)$ with arbitrary accuracy in the $W^{s-1,\infty}$-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.
Read the full abstract ▾
Machine Learning
Beyond the Smile: A Hybrid Convolutional VAE for Crypto Volatility Surfaces
Sadanand Singh, Allam Reddy and Manan Chopra
We present a convolutional variational autoencoder for cryptocurrency implied-
volatility surfaces, together with a deployable predictor that combines it with a quadratic smile re-fit through a deterministic per-tenor routing rule. Trained on 6,034 fully-filled hourly Binance Options surfaces of BTC and ETH spanning May-October 2023 and parameterised on a common $6 \times 7$ tenor-delta grid, the model attains a hidden-
cell surface-completion RMSE in the 0.94-1.56 vol-point range across both
markets and mask rates 10-50%. The hybrid predictor attains 0.83 vol points at 50% masking against 7.00 for the smile re-fit alone, an eightfold reduction obtained at no additional inference cost. Under structurally-correlated hole patterns that emulate the withdrawal of an entire tenor of strikes, the smile re-fit incurs 9.6-13.1 vol points of error while the learned model remains at 1.5-1.9, isolating a regime in which the generative model is the only viable predictor. Joint training on BTC and ETH improves the in-distribution model on both
markets by 9-27% relative to the better-performing single-symbol counterpart, indicating a substantially shared vol-surface
manifold across the two largest cryptocurrencies over the observation window. The hybrid is calendar- and butterfly-arbitrage-free at the listed strikes, a property that the parametric smile re-fit alone fails at high mask rates. The per-snapshot reconstruction error of the trained model flags the late-October ETF-anticipation rally and the August $17$, $2023$ flash crash as elevated-error periods without supervision. All training and evaluation infrastructure is released to support reproducible follow-on work.
Read the full abstract ▾
Machine Learning
Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Data
Kareem Amin, Rudrajit Das, Alessandro Epasto et al. (+4 more)
The rapid adoption of generative AI and Large
Language Models (LLMs) has spurred interest in synthetic data as a
privacy-preserving alternative to sensitive real-world
datasets. However, generating high-utility synthetic data often carries the risk of memorizing and regurgitating private information from the training corpus. In this work, we present a customizable empirical
auditing framework designed to detect and explain such data disclosures. Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures”-where the system incidentally generates a user's data. By partitioning input data into training and holdout sets and applying rigorous statistical hypothesis testing, we determine if observed disclosures are consistent with strict
privacy baselines, such as zero-learning or specific Differential
Privacy (DP) bounds. Crucially, this approach requires no model access, no canary insertion, and no reference model training -only the synthetic output and a held-out control set. We demonstrate that this framework effectively functions as a membership inference attack, providing empirical lower bounds on
privacy leakage that are tighter than prior data-based
auditing methods. Our approach is model-agnostic, applies to any synthetic data generation mechanism, and requires orders of magnitude fewer computational resources than shadow-model or canary-based alternatives.
Read the full abstract ▾
All authors Kareem Amin, Rudrajit Das, Alessandro Epasto, Adel Javanmard, Dennis Kraft, Mónica Ribero, Sergei Vassilvitskii
Machine Learning · Cross-listed from physics.ins-det
Latent space mapping of interpretable structural coordinates from stochastic single-molecule signals
Matteo Cartiglia, Sandro Kuppel, Wouter Botermans Wannes Peeters et al. (+7 more)
Nanopores are versatile single-molecular sensors, but their utility is fundamentally constrained by
stochastic translocation dynamics warping any encoded information. We resolve it by shifting from time-domain analysis to a learned latent-space mapping via a contrastive encoder trained exclusively on simulated signals from a physics-informed model. This encoder maps solid-state nanopore signals of engineered DNA barcodes into an interpretable molecular coordinate system. The learned representation is responsive to structural barcode parameters while remaining invariant to acquisition conditions and translocation conformation, allowing data pooling across devices. Molecule identification requires a single pass through the encoder, reducing computational cost by three orders of magnitude relative to
alignment-based methods. We experimentally validate through mixture quantification, rare-variant detection, consensus barcode reconstruction, and real-time signal acquisition. This shift from temporal analysis to mapping structural coordinates into a latent space changes the paradigm behind analyzing
stochastic sensor signals by linking classification to interpretable encoded molecular information.
Read the full abstract ▾
All authors Matteo Cartiglia, Sandro Kuppel, Wouter Botermans Wannes Peeters, Natan Biesmans, Liam Vandekerckhove, Eric Beamish, Koen Ongena, Wouter Renckens, Pol Van Dorpe, Sanjin Marion
Machine Learning · Cross-listed from stat.ML
A nonparametric two-sample test using a parametric integral probability metric
Yuha Park and Yongdai Kim
Detecting distributional differences between two independent samples is a fundamental problem in statistics and machine learning. Nonparametric two-sample testing provides a principled framework for determining whether two samples are drawn from the same underlying distribution, without assuming any specific parametric form for the distribution. In this study, we propose a new two-sample test statistic based on a newly introduced integral probability metric (IPM), using a specially designed parametric discriminator class with a single node of a neural network. We show that the resulting test statistic, called PReLU-IPM, is nonparametric and establish theoretical guarantees for the associated two-sample testing procedure, PReLU-TST, including its consistency and asymptotical equivalence to nonparametric IPM-based tests under regularity conditions. By analyzing multiple simulated and real
benchmark datasets, we demonstrate that PReLU-TST achieves higher power across a range of alternatives or performs comparably to its competitors, for finite samples.
Read the full abstract ▾
Machine Learning
Scalable Circuit Learning for Interpreting Large Language Models
Naiyu Yin, Dennis Wei, Tian Gao et al. (+3 more)
A prominent research direction in mechanistic interpretability is learning sparse
circuits over LLM components to reveal how they jointly produce model behavior. However, raw
neurons are polysemantic, making learned
circuits hard to interpret. Sparse autoencoder (SAE) features alleviate this, but their high dimensionality makes existing intervention-based
circuit learning methods computationally prohibitive. We propose CircuitLasso, a scalable
circuit learning approach based on sparse linear regression. CircuitLasso recovers
circuits whose structural accuracy matches that of state-of-the-art intervention-based methods on the
benchmark data, at a fraction of the computational cost. For interpretability, CircuitLasso efficiently uncovers relationships among SAE features, showing how human-interpretable semantic features propagate through the model and influence its predictions. Finally, we validate the utility of our learned
circuits by leveraging their insights to achieve comparable performance at substantially lower cost on a domain-generalization task.
Read the full abstract ▾
All authors Naiyu Yin, Dennis Wei, Tian Gao, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, Yue Yu
Machine Learning · Cross-listed from cs.RO
CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation
Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano et al. (+1 more)
Rovers rely on perception to maintain spatial maps that encode both objects and sensor quality (e.g., range reliability, lighting artifacts, data density), guiding data fusion,
embedding updates, and
navigation under partial observability. To study these coupled perception-
navigation processes, we present CrossMaps, a real-time confidence-aware open-vocabulary semantic mapping pipeline that constructs
language-queryable maps from RGB-D data. Building on VLMaps-style approaches, CrossMaps integrates multi-scale CLIP
embeddings with confidence-aware fusion and a dual-
memory architecture consisting of Short-Term
Memory (STM) and Long-Term
Memory (LTM). The STM aggregates noisy visual observations using
geometric, semantic, and temporal confidence cues, while confident and coherent
cells are promoted to the LTM as persistent semantic landmarks. Designed for deployment with a Jetson Orin-powered UGV alongside SLAM, CrossMaps runs in real time and produces semantic heatmaps that can be queried with natural
language to guide rover
navigation.
Read the full abstract ▾
All authors Jan-Niklas Klein, Sona Ghahremani, Christian Medeiros Adriano, Holger Giese
Machine Learning · Cross-listed from cs.CL
Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter
Patomporn Payoungkhamdee, Napat Laosaengpha, Jenta Wonglertsakul et al. (+8 more)
Reasoning with a Code Interpreter (CI) has emerged as an effective paradigm for enhancing the
reasoning capabilities of large
language models (LLMs) through executable computation and iterative verification. Despite its growing adoption, the behavioral properties underlying effective code
reasoning remain largely underexplored. In this work, we investigate code
reasoning from two distinct perspectives inspired by prior studies of natural
language reasoning: extrinsic properties, represented by crucial
tokens, and intrinsic properties, represented by code-specific cognitive behaviors. Across multiple LLMs, we find that stronger CI
reasoning models consistently exhibit a higher prevalence of crucial
tokens and cognitive behaviors, particularly verification, backtracking, and backward chaining. Building on these observations, we examine how these properties can be leveraged during both inference and training. At inference time, appending code-specific crucial
tokens improves performance on several
reasoning capabilities, including mathematical, ordering, and optimization, while yielding limited benefits elsewhere. At training time, augmenting a state-of-the-art framework with code-specific cognitive behaviors improves supervised fine-tuning and reinforcement learning performance in two of three evaluated models. Further analysis shows that these behaviors reduce overthinking in incorrect responses and improve
token efficiency, while also revealing factors that limit gains in a certain model. Our findings provide the first systematic characterization of effective
reasoning with CI and demonstrate both the potential and limitations of leveraging key properties to improve CI-based
reasoning.
Read the full abstract ▾
All authors Patomporn Payoungkhamdee, Napat Laosaengpha, Jenta Wonglertsakul, Pittawat Taveekitworachai, Pume Tuchinda, Panjapong Poobanchuen, Ekapol Chuangsuwanich, Can Udomcharoenchaikit, Samuel Cahyawijaya, Peerat Limkonchotiwat, Sarana Nutanong
Machine Learning
A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning
Ardianto Wibowo, Paulo E Santos, Amer Baghdadi et al. (+3 more)
Reinforcement learning (RL) systems often degrade when operating conditions differ from those previously encountered, reflecting distributional shifts in the underlying data-generating process. Such shifts may occur between training and evaluation, as in In-Distribution (ID) and Out-of-Distribution (OOD) generalization, or within non-stationary settings where environment dynamics evolve over time. However, the formal relationship between these views remains unclear, and existing work mainly focuses on mitigation rather than the causal origin of shift within the
agent-environment interaction. This work develops a unified causal-origin taxonomy that characterizes sources of distributional shift in RL and relates ID/OOD generalization to non-stationary settings. We transfer the classical
dataset-shift principle from supervised learning to RL by reformulating distributional shift in terms of the generative interaction process. Using a Partially Observable Markov Decision Process (POMDP), we decompose the interaction into structural components, including the state distribution, observation process, policy, reward, and transition dynamics, together with the shifted-time boundary. The proposed taxonomy distinguishes internal (
agent-driven) and external (environment-driven) distributional shifts. The shifted-time boundary perspective further characterizes explicit, implicit, and hybrid shifts. This formulation unifies ID/OOD generalization and non-stationarity as structured changes in the underlying process. We also introduce an evaluation framework for measuring shift impact and adaptation through performance degradation and recovery metrics. By grounding distributional shift in the causal-origin structure of RL, this work supports systematic analysis of robustness under distributional shift.
Read the full abstract ▾
All authors Ardianto Wibowo, Paulo E Santos, Amer Baghdadi, Matthew Stephenson, Karl Sammut, Jean-Philippe Diguet
Machine Learning · Cross-listed from math.OC
Functional Gradient Descent with Adaptive Representations
Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia et al. (+3 more)
Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that is, gradient descent directly in function space, which benefits from strong convergence results and admits a clean theory. However, FGD is difficult to implement in practice because functional gradients are infinite-dimensional, and thus cannot be fully computed nor stored in
memory. Existing implementations therefore rely on fixed approximations, which introduce approximation error. We propose a new, theoretically-grounded FGD
algorithm that adapts the representation of the functional gradients over the course of optimization. By explicitly incorporating this approximation into the analysis, we establish convergence to a stationary point (for smooth losses) and to a global minimizer (under smoothness + a Polyak-Lojasiewicz-type condition) regardless of our approximations. To the best of our knowledge, this is the first implementable FGD method with such guarantees in a general setting. We demonstrate the effectiveness of our method on regression, numerical solution of PDEs, and modern computer
vision. Across settings, our method consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency and accuracy.
Read the full abstract ▾
All authors Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia, Leonidas Guibas, Luiz Velho, Tiago Novello
Machine Learning
Demystifying Variance in Circuit Discovery of LLMs
Frank Zhengqing Wu, Francesco Tonin and Volkan Cevher
Circuit discovery is a key technique in mechanistic interpretability to pinpoint the model components that are crucial for performing a given task. Although the current state-of-the-art method (EAP-IG) performs well on the metric of (un)faithfulness, it suffers from substantial variability. This includes resampling variance, where the
circuit changes when we probe with a new batch of data from the same distribution; rephrasing variance, where the discovered
circuit shifts when the
prompts are rephrased; and sample-wise variance, where a
circuit with low population unfaithfulness exhibits large fluctuations in unfaithfulness across individual samples. This paper studies the roots of these variances. We demonstrate that CEAP, our new
circuit discovery method that improves upon EAP-IG with a theoretical guarantee, can substantially lessen resampling variance. We further show that rephrasing variance arises because
prompts with different templates tend to activate different
circuits in the model. This leads us to argue that it may be challenging to find a comprehensive
circuit that explains and controls the model's behavior on a task, which can be expressed in countless templates, suggesting that LLMs may be inherently hard to steer. We show that sparsity, which has been claimed to form more compact and interpretable task
circuits, fails to solve this problem. Regarding sample-wise variance, we argue that it is largely benign: extremely poor unfaithfulness scores often stem from how unfaithfulness is defined, rather than from defects in the measured
circuits. We show that the magnitude of unfaithfulness is affected by selective contribution scaling, a neural mechanism that accounts for the extremely poor scores sometimes observed.
Read the full abstract ▾
Machine Learning
Factorized Neural Operators Decompose Dynamic and Persistent Responses
Hao Tang, Yuechen Duan, Jiongyu Zhu et al. (+3 more)
Physical systems often exhibit heterogeneous mechanisms, where rapidly evolving dynamics coexist with persistent structures. Capturing such multiscale physical behavior remains challenging for existing neural operators, which typically rely on single dominant inductive bias and therefore couple distinct physical responses into a shared representation. We introduce the Unified Green's Function Framework across domains and propose the Factorized Neural Operators (FaNO), which decompose spectral representations into equivariant-inspired dynamic responses and invariant-inspired persistent responses, leading to better interpretability and generalization. Mechanistically, we show that the two operator branches spontaneously specialize into distinct physical roles that remain consistent across scales and domains: the equivariant-inspired branch captures rapidly varying transient dynamics, whereas the invariant-inspired branch extracts coherent persistent structures. This factorized mechanism of FaNO consistently improves prediction accuracy, parameter efficiency and cross-scale generalization across physical systems and domains. In particular, it maintains consistent predictions under long-horizon autoregressive rollout, cross-resolution extrapolation and physical-regime shifts. These findings suggest that scalable physical modeling may benefit from moving beyond single-inductive-bias formulations toward factorized operator representations that better reflect the heterogeneous organization of physical systems, accelerating the reliable deployment of machine learning for scientific computing and discovery.
Read the full abstract ▾
All authors Hao Tang, Yuechen Duan, Jiongyu Zhu, Zimeng Feng, Hao Li, Chao Li
Machine Learning
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Kaiyue Wen, Xingyu Dang, Kaifeng Lyu et al. (+2 more)
Matrix based optimizers such as
Muon can substantially speed up
language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or
Muon, Hyperball sets the Frobenius norms of weight
matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters,
Muon Hyperball achieves 20–30%
token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an
equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight
matrix changes.
Read the full abstract ▾
All authors Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, Percy Liang
Machine Learning
Beyond Weights and Gradients: A Taxonomy of Federated Learning Messages
Alvaro Javier Vargas Guerrero, Xinguang Wang, Quang Manh Doan et al. (+1 more)
Federated Learning is rapidly evolving beyond the exchange of traditional model weights and gradients, yet existing definitions fail to capture the full scope of modern payloads like synthetic data and
federated analytics. This paper addresses the gap by proposing a formal mathematical definition of a
federated message that accounts for both utility and
privacy. We introduce a taxonomy that organizes these exchanges into three categories: model structures, statistical summaries, and data-conditioned representations. By evaluating these groups based on computational demands, communication costs, and
privacy risks, we provide a clearer understanding of the trade-offs involved in decentralized training. Our review of 202 recent publications highlights a significant shift since 2021 toward diverse messaging paradigms, signaling a move away from standard deep learning updates toward more specialized information sharing. This framework provides a structured path for future research to optimize
federated systems for varying hardware and security requirements.
Read the full abstract ▾
All authors Alvaro Javier Vargas Guerrero, Xinguang Wang, Quang Manh Doan, Guy Nagels
Machine Learning
Upper Bounds on the Generalization Error of Deep Learning Models via Local Robustness and Stability
Abdul-Rauf Nuhu, Parham M. Kebria, Vahid Hemmati et al. (+3 more)
Generalization is a critical property of data-driven models, particularly deep learning models deployed in safety-critical applications. Robustness-based generalization bounds have gained
attention as a principled way to link robustness properties to generalization performance, often in a data-dependent manner. However, most existing bounds suffer from vacuousness in practical settings, yielding loose upper bounds that greatly exceed the actual error rates and limiting their usefulness for real-world evaluation. While this issue is often attributed to the uncertainty term, a substantial part of the problem originates from the robustness term itself, particularly for the 0-1 loss. Existing approaches typically treat the robustness term as a global measure, ignoring its variation across different sub-regions of the input space. In this work, we propose a generalization bound that addresses this limitation by scaling the robustness term according to the number of stable and unstable samples within each sub-region. Our bounds incorporate both data- and model-dependent factors while maintaining practical relevance (yielding tighter upper bounds on true error). Experiments on models trained on the ImageNet
dataset show that our bounds remain consistently non-vacuous and achieve the tightest estimates among existing methods, closely aligning with empirical performance across a range of robust deep neural networks.
Read the full abstract ▾
All authors Abdul-Rauf Nuhu, Parham M. Kebria, Vahid Hemmati, Mahmoud N. Mahmoud, Edward Tunstel, Abdollah Homaifar
Machine Learning
Integrated Marketing Attribution: A Bayesian Framework for Privacy-Safe Granular Measurement Anchored in MMM
Meghana R. Bhat, Ankit Umare, Utsav Aggarwal et al. (+4 more)
Retail marketing measurement increasingly requires granular campaign-level insights without relying on user-level tracking. However, the two dominant approaches, Marketing Mix Modeling (MMM) and Multi-Touch Attribution (MTA), often produce fragmented insights. MMM is
privacy-safe and robust for channel-level planning but is too coarse for campaign optimization, while MTA provides granular attribution but has become less reliable under increasing
privacy restrictions. We propose Integrated Marketing Attribution (IMA), a unified framework that combines MMM with channel specific
Bayesian attribution models to derive campaign-level effects from aggregated data. By leveraging MMM-informed priors, IMA delivers granular,
privacy-safe attribution while preserving consistency with MMM.
Read the full abstract ▾
All authors Meghana R. Bhat, Ankit Umare, Utsav Aggarwal, Richard Vecsler, Arunkumar Mani, Karthik Nair, Chandhu Nair
Machine Learning
HawkesNest: A Multi-Axis Synthetic Benchmark for Spatiotemporal Pattern Complexity
Yahya Aalaila, Sumantrak Mukherjee, Gerrit Großmann et al. (+1 more)
Evaluation of spatiotemporal point process (STPP) models relies heavily on opaque real-world
datasets, where latent generative structure is unknown and model failures are difficult to attribute. We introduce HawkesNest, a generator-aligned
benchmark for controlled spatiotemporal pattern complexity built on a multivariate Hawkes backbone. HawkesNest defines four complexity axes: space–time
entanglement, background heterogeneity, cross-type interaction, and domain
topology. Each axis is associated with a deterministic index computed from the latent data-generating mechanism. By varying these axes while holding global rate, stability, and simulation budget fixed, HawkesNest enables diagnostic stress tests of STPP models under known structural difficulty. We verify that the indices are monotone and nearly orthogonal under controlled sweeps. We illustrate its use by showing that Hawkes-family baselines degrade under joint heterogeneity–
entanglement complexity, even though they are structurally aligned with the Hawkes data-generating backbone. We further show that HawkesNest exposes neural-model sensitivity: AutoSTPP remains vulnerable under isolated increases in space–time
entanglement. Code. Available at https://github.com/YahyaAalaila/HawkesNest
Read the full abstract ▾
All authors Yahya Aalaila, Sumantrak Mukherjee, Gerrit Großmann, Sebastian Vollmer
Machine Learning
Deep Q-Learning on Hölder Spaces
Qian Qi
We study the operator-theoretic core of Q-learning in continuous-time
stochastic control with continuous states and actions. In value-based reinforcement learning, each Q-learning or DQN update is built from a Bellman optimality target; our analysis isolates this target in a
diffusion setting and studies its regularity and approximation complexity. Under uniform ellipticity and Hölder-regular coefficients, we show that a Bellman update maps bounded inputs into an anisotropic regularity class, smoothing the state variable while leaving only Lipschitz dependence on the action variable. This yields a compact family of Bellman iterates and motivates a tensor-product DeepONet architecture adapted to the mixed regularity of the problem. We then derive explicit approximation and resource bounds, together with a stiffness–complexity trade-off as the time step $\delta \to 0$. The resulting theory makes a direct contribution to Q-learning theory at the level of Bellman target regularity and approximation in continuous
stochastic control. At the same time, we do not claim a full convergence
theorem for practical sampled Q-learning with exploration, replay, and
stochastic gradient updates.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
Tying the Loop – Tied Expert Layers in Mixture-of-Experts Language Models
Martin Jaggi
Mixture-of-Experts (MoE) architectures efficiently scale Large
Language Models (LLMs) by activating only a small fraction of their experts per
token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference
memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive
transformer layers while preserving independent, layer-wise routing and
attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce
memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-
memory trade-off, advancing efficient training and scaling of next-generation LLMs.
Read the full abstract ▾
Machine Learning · Cross-listed from eess.SP
A Perception vs. Distortion Perspective on Score-Based Generative Channel Estimation
Marco Skocaj, Lukas Eller and Mate Boban
Driven by their remarkable success in computer
vision and inverse problem solving, score-based models are increasingly applied to wireless communications, where they show promise across a range of physical-layer tasks. However, despite this growing interest, the current literature often lacks a rigorous analysis of when score-matching offers a tangible advantage over traditional discriminative learning. This paper aims to address this gap through the use-case of channel estimation, a fundamental inverse problem in wireless systems. We present a theoretically grounded interpretation of score-based channel estimation through the lens of the perception-distortion tradeoff, identifying the conditions where score matching excels as well as its key limitations. In particular, by modeling downstream wireless tasks (e.g., capacity maximization) as functionals of the channel estimation process, we quantify the excess risk incurred by standard distortion-minimization approaches. Extensive numerical results show that under high predictive uncertainty, the large excess risk gap can be offset by score-based estimation, enabling near
Bayesian-optimal precoding via the learned posterior, whereas in the low predictive uncertainty regime, discriminative distortion-minimization approaches are preferable due to lower complexity and more efficient use of model capacity.
Read the full abstract ▾
Machine Learning
Decision-Weighted Flow Matching for Contextual Stochastic Optimization
Jize Xie, Haomiao Wu, Qiang Chen et al. (+2 more)
Conditional generative models are increasingly used as scenario generators for
stochastic optimization, but standard training objectives emphasize uniform distributional fit rather than the downstream decisions induced by generated scenarios. This creates an objective mismatch: errors in statistically common regions may have little effect on decision regret, whereas errors in decision-sensitive regions can substantially change the optimal action. We propose Decision-Weighted Flow Matching (DW-FM), a regret-aligned training framework that preserves the simplicity of standard flow matching while reweighting its velocity-regression objective using decision-sensitive endpoint information. Theoretically, we connect downstream regret to pathwise velocity mismatch through a loss-induced decision discrepancy and an adjoint transport argument, yielding an ideal regret-aligned surrogate and practical endpoint-weighted objectives with regret guarantees. Empirically, we demonstrate the effectiveness of DW-FM on three CVaR-based contextual
stochastic optimization
benchmarks spanning synthetic portfolio, semi-real financial, and traffic-CVaR tasks, where DW-FM improves downstream regret over standard baselines.
Read the full abstract ▾
All authors Jize Xie, Haomiao Wu, Qiang Chen, Xiu Su, Yi Chen
Machine Learning
We Need Explanation Cards to Connect Explanation Algorithms to the Real World
Eric Günther, Balázs Szabados, Kristof Meding et al. (+3 more)
Algorithmic explanations are intended to help stakeholders understand opaque algorithmic decisions, but in practice, they often fall short. First, the meaning of algorithmic explanations is often not what one might intuitively expect, so expert knowledge is required to interpret them correctly. Second, recent work has shown that popular explanation
algorithms are uninformative about the behavior of complex decision functions. Together, these issues create a gap between what explanations appear to convey and what they actually provide. In this work, we propose Explanation Cards for Explanation
Algorithms, which augment standard explanations with complementary information about robustness and validity, as well as clear instructions for interpretation. The complementary information can render otherwise uninformative explanations practically useful, while also helping to detect cases where they are not. Importantly, the interpretation instructions in explanation cards shift responsibility from users to providers: Rather than expecting users to recognize what can and cannot be concluded from an explanation, providers must make this explicit upfront. Using counterfactual explanations and SHAP as examples, we demonstrate how providers can construct explanation cards and that these cards provide users with the guidance needed for sound interpretation. We further argue that explanation cards offer a practical means of operationalising the explainability provisions of the EU AI Act. Overall, explanation cards are a significant step toward making explanation
algorithms fit for real-world use cases.
Read the full abstract ▾
All authors Eric Günther, Balázs Szabados, Kristof Meding, Gunnar König, Sebastian Bordt, Ulrike von Luxburg
Machine Learning · Cross-listed from cs.CV
Gen-VCoT: Generative Visual Chain-of-Thought Reasoning via Diffusion-Based RGB Intermediate Representations
Zhiqiang Zhou, Junliang Dai and Xu ling
Multimodal large
language models (MLLMs) excel at visual
reasoning but rely on text-based chain-of-thought (CoT), lacking interpretable visual intermediates. Existing methods use opaque
tokens or external tools, missing key properties. We propose Gen-VCoT, a framework using expert
vision models to generate RGB images as
reasoning intermediates. It has three stages: visual grounding (SAM
segmentation),
geometric reasoning (Marigold depth maps), and semantic
reasoning (Qwen2-VL integration). An adaptive router selects
reasoning depth. Evaluations show Gen-VCoT improves spatial (25% better) and depth (50% better) questions, but may hurt simple factual queries. Text CoT outperforms visual intermediates on CLEVR (91.2% vs 62.5%), showing task-dependent optimal representations. Gen-VCoT establishes a new paradigm for interpretable
multimodal reasoning.
Read the full abstract ▾