Machine Learning
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh et al. (+1 more)
A growing body of work establishes that large
language models are not mere statistical memorizers, but are capable of in-context learning: performing inference at test time using only examples provided in the
prompt, without any parameter updates. Prior theoretical work has shown that this capability extends to supervised learning tasks such as linear regression. We prove that in-context learning extends further to data generation: frozen
transformers can simulate iterative generative samplers from in-context samples. We first show that
transformers can realize closed-form and smoothed closed-form
diffusion samplers. The construction identifies a concrete generative role for softmax
attention: it computes responsibility weights and weighted empirical averages, while feedforward layers implement Euler updates. To empirically relate these constructions to pretrained
language models, we study semantic-topic sampling:
prompts consisting of words drawn from a common semantic category, such as animals, foods, or cities. Across
transformer layers, the normalized hidden states exhibit a two-stage geometry: they move toward a uniform spherical reference in intermediate layers and then return to structured, topic-dependent representations near the output. We further measure an interacting-particle energy on these hidden-state clouds and observe the same U-shape pattern. We then prove that
transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
Read the full abstract ▾
All authors Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh, Hadi Daneshmand
Machine Learning · Cross-listed from eess.AS
Orantqing, Shengpeng Ji, Junlong Tong et al. (+19 more)
In this work, we present Gander, a native
multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtime interaction with an asynchronous
agent loop. In contrast to conventional turn based systems, Gander continuously processes streaming user inputs, enabling full-duplex interaction in both everyday conversations and complex workflow
agent scenarios. Users can interrupt an ongoing response, while the model can proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) a Cerebellum-
Brain collaborative framework, Cerebellum is responsible for realtime interaction while the
Brain handles complex
reasoning and higher level agentic tasks. The two components interact continuously through tool calling and the
agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, where user inputs and model outputs are flattened into an ordered
token stream at the chunk level. We evaluate Gander across conversational ability, interactive capability, understanding, and tool assisted task execution. Internal human evaluations show that Gander maintains natural and expressive spoken dialogue, while
benchmark results demonstrate effective turn taking capability and encouraging results on spoken question answering and related understanding tasks. Gander also supports a range of challenging interaction settings, including background noise interference, multi-party interactions, and backchannel communication. While our current evaluation focuses on tool assisted settings, broader long horizon
agent tasks and more diverse deployment conditions remain promising directions for further study. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Read the full abstract ▾
All authors Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddy Sun, Steve Yves, Zhou Zhao
Machine Learning
GraphFAS: A Distributed System for Automated Graph Feature Generation and Selection in Industrial Transaction Networks
Yice Luo, Yun Zhu, Xi Chen et al. (+7 more)
Industrial fraud detection often relies on costly expert-crafted features that overlook graph-structured relational signals, while GNNs often do not meet the interpretability and deployment requirements of financial risk control. We propose GraphFAS (Graph Feature Automated Selection), a distributed feature selection procedure based on Boruta that bridges this gap through: (1) a non-parametric graph feature generation module that constructs explicit, interpretable structural features via multi-hop subgraph extraction and multi-scale aggregation without learned parameters; and (2) an automated distributed feature selection
algorithm extending Boruta with median-based aggregation across partitions to robustly identify informative features at scale with minimal domain expertise. Compared with end-to-end GNN pipelines, GraphFAS decouples feature aggregation from model training, enabling direct integration with tabular models and direct compatibility with TreeSHAPbased explanations. Deployed in Alipay, GraphFAS delivers orderof-magnitude improvements in engineering efficiency while showing strong performance against expert-driven and graph-learning baselines on large-scale graphs.
Read the full abstract ▾
All authors Yice Luo, Yun Zhu, Xi Chen, Yongchao Liu, Xintan Zeng, Chengying Huan, Kai Zhang, Jinrui Zhang, Juelu Zhang, Jiajun Zheng
Machine Learning · Cross-listed from cs.LO
Fitting and Learning Basis-Restricted Propositional Formulas
Balder ten Cate
For a finite set $O$ of Boolean functions, we consider the class of propositional formulas built using the functions in $O$ as connectives. We determine, for each possible choice of $O$, the complexity of various fitting and learning problems. These include: finding a formula that fits a given labeled sample, finding a small one (an Occam
algorithm), minimizing the number of misclassified examples when the sample is not realizable (empirical risk minimization), and several forms of PAC learning. Our results apply both to formulas (represented as trees) and to
circuits. We also briefly discuss the status of the same questions for other kinds of propositional fragments.
Read the full abstract ▾
Machine Learning · Cross-listed from physics.flu-dyn
ONE CYLinder: A Benchmark for Graph-Based Surrogate Modeling of Unsteady Bluff-Body Flows
Théodore Michel, Antoine Campos, Alban Dujardin et al. (+3 more)
Graph-based surrogate models offer a promising route to accelerate computational fluid dynamics (CFD) simulations on unstructured meshes. However, their development is limited by the scarcity of
benchmark datasets spanning multiple flow regimes and standardized protocols for long-horizon autoregressive prediction. We introduce ONECYL (ONE CYLinder), a new
benchmark for unsteady flow past a circular cylinder across laminar, transitional, and high-Reynolds-number regimes. The
benchmark comprises 450 high-fidelity Variational Multiscale finite-element simulations (270,000 flow snapshots) with randomized cylinder geometries, providing time-resolved velocity and pressure fields together with mesh connectivity,
geometric descriptors, Reynolds numbers, and integrated aerodynamic quantities. Beyond the
dataset, ONECYL establishes a unified evaluation framework combining full-field rollout errors, virtual probes, and drag and lift predictions to assess numerical accuracy and physical fidelity. To accompany the
benchmark, we develop a Graph
Transformer as a reference baseline predicting velocity and pressure fields autoregressively on unstructured meshes. Using ONECYL, we investigate
geometric representations and physics-based regularization across the three Reynolds-number regimes. The results show that explicitly encoding the cylinder geometry through a level-set representation consistently improves long-horizon prediction accuracy and generalization to unseen geometries, while divergence-based regularization becomes increasingly beneficial as flow complexity increases. The ONECYL
benchmark and its Graph
Transformer baseline provide a reproducible framework for evaluating graph-based surrogate models and establish a foundation for future research on long-horizon prediction of unsteady bluff-body flows.
Read the full abstract ▾
All authors Théodore Michel, Antoine Campos, Alban Dujardin, Henry Areiza, Philippe Meliga, Elie Hachem
Machine Learning
The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation
Aaryaman Kalani, Murari Mandal, Dhruv Kumar et al. (+2 more)
Approximate machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. We identify a previously undocumented confound in how unlearning is evaluated on BatchNorm-based architectures: a single forward pass over retain data, an operation that modifies no weight, can deterministically rewrite the model's normalization state and reverse the apparent surface-metric forgetting. We formalize this operation as a weight-preserving fixed-point operator and prove that any pre-versus-post gap it induces is provably attributable to BN running statistics rather than to any modification the unlearning method made to the weights. This attribution claim cleanly separates measurement failure (BN artifact) from encoder failure (residual weight-encoded information, recently documented in concurrent work), and the same operator framework yields a unique decomposition of linear-probe elevation into BN-measurement-bias and encoder-geometry components. Empirically, the artifact reverses headline forget accuracy by up to 78 pp across nine evaluated methods on standard
benchmarks; an attacker with as few as 10 unlabeled images recovers most of the masked accuracy; and a strict GroupNorm control reduces the artifact to zero across all methods. The tested membership-inference attacks change little under recalibration, locating the observed evaluation failure in forget accuracy and linear probing.
Read the full abstract ▾
All authors Aaryaman Kalani, Murari Mandal, Dhruv Kumar, Mohan Kankanhalli, Yash Sinha
Machine Learning · Cross-listed from cs.DS
High-Magnetization Sampling at Low Temperatures: Ising Models and Bayesian Sparse Linear Regression
Syamantak Kumar, Purnamrita Sarkar, Kevin Tian et al. (+1 more)
Sparsity is a powerful structural resource in optimization and statistics. We develop frameworks for leveraging sparsity in sampling problems over the Hamming slice $\mathcal{X}_k^d:=\{\mathbf{x}\in\{\pm 1\}^d:|\{i:\mathbf{x}_i=1\}|=k\}$, in high-dimensional regimes where $k\ll d$ (i.e., where $\mathcal{X}_k^d$ is highly magnetized). We use our frameworks to design improved samplers for canonical problems in the study of Ising models and
Bayesian sparse linear regression. Our first main result considers the Sherrington–Kirkpatrick (SK) model restricted to fixed-magnetization slices $\mathcal{X}_k^d$. We give a
polynomial-time sampler for fixed-magnetization SK models at any inverse temperature $\beta>0$, under arbitrary external fields, provided that $k\le c_\beta d$ for an appropriate constant $c_\beta$. By combining this result with an annealing strategy for estimating normalizing constants, we obtain
polynomial-time samplers for the SK model at arbitrarily low temperatures under a sufficiently strong external field of strength $h$. In the large-$\beta$ limit, our framework permits sampling at field strengths within constant factors of the Almeida–Thouless line delineating the replica-symmetric and replica-
symmetry-breaking regions ([dAT78]), improving polynomially over the field strength $h(\beta)$ required by the recent work of [BAR26]. Our second main result concerns the measurement complexity of
polynomial-time
Bayesian sparse linear regression. Recent work by [KSTZ25] shows how to sample from the canonical Gaussian spike-and-slab posterior with expected sparsity $k$, at any signal-to-noise ratio, given $n\gtrsim k^3\log^3 d$ Gaussian measurements. We improve this requirement to $n\gtrsim k^{3/2}\log^2 d+k\log^3 d$, using a common sparsity-aware framework underlying both our results.
Read the full abstract ▾
All authors Syamantak Kumar, Purnamrita Sarkar, Kevin Tian, Yusong Zhu
Machine Learning
Earth System World Model for What-If Simulations: A Case Study for Terrestrial Ecosystems
Zhihao Wang, Ruichen Wang, Ruohan Li et al. (+6 more)
Machine learning emulators have become essential for accelerating expensive Earth-system simulations, but most existing approaches remain passive forecasters: they reproduce simulator trajectories under prescribed forcings without an explicit interaction mechanism for user-specified interventions. This limits their use in interactive scientific workflows and Earth-system digital twins, where users often need to explore how a system would respond if selected state components were changed. We propose an action-conditioned world-modeling framework for Earth-system emulation that reformulates simulator trajectories as supervision for controllable state-transition learning. The key idea is transition-action pretraining: naturally observed state changes are treated as label-free action supervision, allowing the model to learn both prescribed dynamics and action-conditioned responses without manually annotated interventions. We further introduce masked response learning to infer unobserved variables under partial state edits and learn coupled system dependencies. We test this framework on ecosystem dynamics across six global regions and multiple stand ages. Experiments show that the model preserves competitive long-horizon emulation accuracy while enabling controllable structural interventions and coherent responses in coupled ecosystem-cycle variables. These results suggest a practical route from passive Earth-system emulators toward interactive, intervention-aware scientific surrogates.
Read the full abstract ▾
All authors Zhihao Wang, Ruichen Wang, Ruohan Li, Lei Ma, George Hurtt, Xiaowei Jia, Gengchen Mai, Shaowen Wang, Yiqun Xie
Machine Learning
Length Generalization for Transformers via Compression
Georg Zetzsche, Hongjian Jiang, Andy Yang et al. (+4 more)
Recent advancements in
transformer length generalization theory enable us to reliably predict when a
transformer can learn to solve a task. In particular, the C-RASP hypothesis (a formalized version of the so-called RASP-l
conjecture) posits that
transformers length-generalize on a task if and only if a solution is expressible in the C-RASP
language. While this hypothesis has strong empirical validation, theoretical problems arise from the fact that no computable length generalization bounds exist for C-RASP, alongside the discovery of seemingly contradictory experiments. To address these problems, we refine the C-RASP hypothesis utilizing the recently-proposed fragments C-RASP+ and C-RASP1. These fragments have computable length generalization bounds, though in the worst case requiring an extremely large (double exponential) sample size. It is an open question whether these sample size bounds are tight. In this paper, we resolve this open question by providing an exponentially tighter bound. In doing so, we show a
polynomial length generalization bound for
transformers if we adopt compressed strings, via a novel connection to power words. As an application, we show how this yields a fine-grained analysis of the C-RASP
conjecture that resolves contradicting experimental evidence against it.
Read the full abstract ▾
All authors Georg Zetzsche, Hongjian Jiang, Andy Yang, Pascal Bergsträßer, Marco Sälzer, David Chiang, Anthony W. Lin
Machine Learning · Cross-listed from stat.ME
A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
Veerendra Kumar Sunkavalli
When an external reference set (an anchor) is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error, standard practice assumes the anchor is uncontaminated: its error uncorrelated with the judges' shared error. We study when that assumption can be dropped and replaced by an estimate. Under a single-common-factor model, >=2 judges and >=2 anchors point-identify the quality variance, the common-mode variance, and each anchor's contamination correlation rho_k in closed form, with an exact per-anchor-pair failure boundary; a designated clean-anchor
estimator, by contrast, reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated. Because the single-common-factor assumption is itself untestable, the
estimator ships gated behind a calibrated diagnostic battery (judge-covariance dispersion; over-identification; a family-block test from judge metadata, with a family-blocked
estimator that removes family-level shared-residual bias exactly), bootstrap confidence intervals with measured coverage, and a weak-identification screen. A proposition maps which violations bias rho_k, in which direction, and which evade detection. For ordinal scores we show an identification hierarchy: with all variables ordinal, rho_k is not identified at any number of anchors; with ordinal judges and >=3 continuous anchors it is, and we give an
estimator for that case. On real data the validation is asymmetric, and we say so plainly: the diagnostics are validated in the rejecting direction (both real panels we test are correctly rejected by the model-adequacy pre-test), while the
estimator is validated in simulation and stress-tested semi-synthetically under oracle calibration; no real panel has yet passed the pre-test, and the pre-test exists precisely to say so. All results replay offline from shipped, checksummed artifacts.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Ostrich: Taking Large Strides Through Stiff Contact in Differentiable Dynamics
Aleš Kučera and Karel Zimmermann
Three properties determine whether a differentiable simulator can drive gradient-based optimization through contact: simulation accuracy, gradient reliability, and per-iteration cost. Tape-based engines such as MJX and Newton Semi-Implicit require timesteps small enough to keep contacts numerically tractable, and their backpropagation
memory grows linearly with the number of timesteps T. Surrogate models bound
memory by approximating contact away, but the resulting gradients lose the geometry the optimization depends on. We present Ostrich, a GPU-accelerated rigid-body simulator that resolves hard contacts and friction with non-smooth Newton iteration at large timesteps (h ~ 0.1 s), and differentiates the converged residual via the implicit function
theorem, reusing the forward Schur complement to compute the adjoint at O(1)
memory per timestep. On real-
robot trajectories over a pallet obstacle, Ostrich holds MuJoCo's sim-to-real accuracy up to a 50x larger timestep. Its gradients converge from random initializations where MJX descends slowly and Newton Semi-Implicit stalls; a warm iteration runs 211x faster than MJX's and 4.7x faster than Semi-Implicit's. On the same scene Ostrich differentiates 8,192 parallel worlds on a single 24 GB GPU, sustaining 29x checkpointed MJX's optimization throughput; without checkpointing both baselines exhaust
memory at far fewer worlds. We close with a gradient-based trajectory optimization demonstration over triangle-mesh terrain across a 10 s horizon, a setting where prior engines either restrict to primitive geometry or face the convergence and
memory limits shown above.
Read the full abstract ▾
Machine Learning
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Youngrok Park, Sangmin Bae, Hojung Jung et al. (+6 more)
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This is particularly important for successive model generations and multi-domain consolidation, where frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional
distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse
Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. Our evaluations cover mathematical
reasoning, logical
reasoning, code generation, and visual tool use. In both successive model transfer and multi-teacher
distillation, OPRD achieves higher performance with fewer student updates than existing RL and
distillation approaches. Response-style analysis shows that OPRD students remain closer to verifier-only RL models than to their weak teachers, suggesting that guidance accelerates rather than redirects student optimization. Strong-to-weak results further show that OPRD combines verifier-driven optimization with teacher guidance regardless of capacity ordering.
Read the full abstract ▾
All authors Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
Machine Learning
Adaptive Anisotropic Attention for Axis-Structured Signals
Mahir Jain, Parshva Runwal, Aditya Ray Mishra et al. (+4 more)
Dense self-
attention treats all
token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals. For structured, low signal-to-noise ratio (SNR) signals such as
EEG, dependencies are organized along the electrode and time axes, and this uniform prior exposes each
token to many irrelevant interactions. We introduce Adaptive Anisotropic
Attention (AAA), which splits
attention into two paths: a temporal path, where each
token attends to the
tokens of its own electrode across time, and a spatial path, where it attends to the
tokens of the other electrodes at the same time step. A small gate predicts, for every
token, a convex combination of the two path outputs: two non-negative weights that sum to one. On six
EEG downstream tasks, the resulting model, AXON (AXis-factorized Operator Network), improves mean balanced accuracy over a dense baseline under both linear probing and full fine-tuning. We show that both paths (temporal and spatial) are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network. Controlled audio spectrogram experiments show that axis factorization transfers beyond
EEG. These results suggest that aligning
attention with the natural axes of structured signals provides a useful inductive bias.
Read the full abstract ▾
All authors Mahir Jain, Parshva Runwal, Aditya Ray Mishra, Arvasu Kulkarni, Jeet Bandhu Lahiri, Sandeep Singh, Siddharth Panwar
Machine Learning · Cross-listed from cs.CR
ZK-Trace: Certified Collusion Tracing with Zero-Knowledge Credentials for Federated GNSS Interference Monitoring
Redwanul Karim, Nisha L. Raichur, Lucas Heublein et al. (+3 more)
Federated global
navigation satellite system (GNSS) monitoring distributes a proprietary classifier to partly trusted stations, any of which may leak its copy. ZK-Trace combines public identity marks, recipient-specific Tardos fingerprints, and zero-knowledge credential verification. The registry supports offline tracing without the leaker's cooperation. We establish conditional false-accusation bounds for arbitrary recovered bit patterns, a finite completeness bound under a hidden-bias residual channel, and a deterministic tracing-score bound for correlated feature-
distillation errors. An interval-arithmetic checker makes the conditional bound executable and allocates a common budget across accusation and tamper decisions. Under innocent-row independence, the certificate-based evaluation uses a false-naming budget of 0.001 per investigation. It isolates all 160 single-owner copies and traces 712 of 720 two-owner mixtures without naming an innocent. Experiments use a simulated GNSS federation and CIFAR-10. Feature matching preserves the feature mark in 20/20 runs and cross-architecture transfer in 19/20, at copy-accuracy costs of 4.8 and 6.1 percentage points on GNSS and CIFAR-10. Function-only
distillation erases the feature mark, and
distillation also removes weight-space marks. These results support verifiable tracing under explicit statistical and cryptographic assumptions. Credential knowledge and recipient evidence serve distinct roles.
Read the full abstract ▾
All authors Redwanul Karim, Nisha L. Raichur, Lucas Heublein, Tobias Feigl, Christopher Mutschler, Felix Ott
Machine Learning · Cross-listed from cs.IT
A Note on Scaling in Randomly Rotated Quantization and Its Connection to the CDEF +1 Pythagorean Relation
Uri Erez
Quantization schemes based on randomized rotations have recently received renewed
attention, including the roles of MMSE and unbiased reconstruction scalings. In this note, we point out the connection to classical results in statistical signal processing and communication theory. Specifically, the two reconstruction scales used in the EDEN line of work admit a natural interpretation as finite-dimensional, realization-dependent counterparts of the Wiener and unbiased coefficients in the classical CDEF formulation. At finite blocklength, the CDEF +1 relation holds pointwise for each rotation realization as an exact
geometric (Pythagorean) identity, but does not hold after averaging the distortions over the rotation. The classical SNR relation $\sf{SNR}_{\rm MMSE}=\sf{SNR}_{\rm MMSE,U}+1$ is recovered as $d\to\infty$: once the overall scale is handled separately, the empirical coordinate statistics of a randomly rotated vector approach their i.i.d. Gaussian counterparts, and the rotation-dependent quantities concentrate. Importantly, EDEN goes beyond this classical correspondence: for every finite $d$, its Haar-rotation formulation guarantees exact conditional unbiasedness, a stronger property than the second-order notion of unbiasedness in CDEF. We further comment on two distinct roles random rotations play in quantization: one is approximate Gaussianization of the coordinates; the other is decorrelation of reconstruction errors across quantization branches.
Read the full abstract ▾
Machine Learning
PAC-Bayesian Bounds for Learning Partially Observed Stochastic Linear Time-Invariant State-Space Systems with Inputs and Sub-Gaussian Noise
Mihaly Petreczky, Mohamad Al Ahdab and John Leth
In this paper we derive a Probably Approximately Correct (PAC)-
Bayesian error bound for partially observed linear time-invariant (LTI)
stochastic dynamical systems in state-space form with inputs and sub-Gaussian noise. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. The bound derived in this paper relates the expectation of prediction errors with the prediction error generated by the model on the data used for learning. In addition, we show that it can also be used to derive bounds for the parameter estimation error. In turn, this allows us to provide finite-sample error bounds for the prediction error and parameter estimation error for a wide class of system identification
algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-
Bayesian bounds for RNNs.
Read the full abstract ▾
Machine Learning
BAFF: Bid-Aware Filter Family for Mitigating Training Data Interference in RTB A/B Tests
Jeonglyul Oh, Ikkyu Choi, Inseop Youn et al. (+1 more)
In online A/B tests for real-time bidding (RTB), control and treatment models are typically trained on a shared serving log that includes data generated by the counterpart model. This shared-log training biases each model's training data through two channels: the counterpart model may have selected a different ad from the ad-candidate pool (ad-ranking disagreement) and may have bid a different price (bid-pricing disagreement), potentially distorting the A/B test outcome. Log-splitting eliminates the bias but sacrifices training data; log-sharing retains all data but leaves the bias unaddressed. We formalize the Bid-Aware Filter Family (BAFF), a class of (k,l)-parameterized hard filters that controls tolerance to each channel independently, providing a structured search space between these two extremes. We further propose a three-stage online measurement protocol that enables evaluating data-sharing strategies by their deviation from an interference-free reference model in production. In offline simulation, a (k,l) sweep surfaces operating points with smaller deviation from the interference-free reference model than both log-sharing and log-splitting. In a live RTB deployment on a demand-side platform (DSP), filter-based variants preserve the reference model's business metrics (e.g., CPC, CTR) more closely than both baselines. The best operating point is setting-dependent, underscoring the practical value of the search space itself.
Read the full abstract ▾
All authors Jeonglyul Oh, Ikkyu Choi, Inseop Youn, Youngjae Kim
Machine Learning
Chimaera: A Mixture-of-Graph-Experts Architecture for Cross-Task and Cross-Dataset Graph Learning
Jonathan Frank, David Richerby and Ansgar Scherp
Designing foundation models for graphs is challenging due to the irregular structure of graphs and the different sizes and characteristics of
embeddings. Chimaera integrates mixture-of-experts with graph foundation models (GFM). It integrates different GFM architectures, such as graph
prompts and linear GNN models. Large
language models are used to generate
embeddings, and experts can be trained and combined following different strategies, GFMs,
embeddings, etc. Furthermore, Chimaera extends existing linear GNNs to support link-level and graph-level tasks in addition to node-level tasks. Empirical analyses are performed on same-task and cross-task experiments with node, link, and graph classification tasks using six
benchmark text-attributed graph
datasets. The experiments demonstrate the effectiveness of Chimaera and its capabilities for transfer across tasks and
datasets. Further insights include the need to use both large and small
language models to generate
embeddings for the experts, a strong cross-task transferability of simple but effective linear GNNs, and using few samples only to provide strong results.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.SD
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
Fritz Cremer and Jonathan Cremer
Text-to-
speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality usually comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU.
Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived
transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived
transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional
multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches about 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. In a blinded study, 40 listeners preferred TontaubeV1's prosody over Fish Audio S2 Pro in 70.25% of 400 comparisons (95% CI: 64.2-75.7%). On an LLM-as-a-judge
benchmark, it matches ElevenLabs Flash v2.5 and outperforms the April 2026 Gradium API and Cartesia Sonic 3 on prosody. The weights are released on Hugging Face under the Tontaube Community Model License 1.0.
Read the full abstract ▾
Machine Learning
Hyperparameter Scaling Laws Across MoE Sparsity
Changxin Tian, Kunlong Chen, Jia Liu et al. (+3 more)
Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-
embedding parameters, processing approximately 20 trillion
tokens at a cost of 200,000 equivalent H800 GPU-hours. Our results reconcile conflicting findings in prior work by revealing two scaling regimes. At fixed sparsity, the optimal batch size follows a power-law relationship with training
tokens $D$, whereas the optimal learning rate scales with training compute $C$ and remains robust to the allocation between model size and data. Across sparsity levels, the activation ratio $A$ enters both relationships as an additional multiplicative power-law factor. These observations lead to unified hyperparameter scaling laws that transfer across MoE sparsity levels. Large-scale evaluation shows that the scaling form outperforms alternative functional forms. On a held-out ultra-sparse MoE with 12B total parameters and only 1/64 of its experts activated, the predicted hyperparameters remain close to the observed optima, supporting joint extrapolation across model scale and sparsity. Further experiments demonstrate transfer across expert granularities and isolate the effect of activation ratio from that of total expert count.
Read the full abstract ▾
All authors Changxin Tian, Kunlong Chen, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou
Machine Learning
HOPE: Heterophily-Aware Open-Set Node Classification with Pseudo-Extrapolation
Yumeng Dai, Yue Tan, Yixin Liu et al. (+3 more)
Standard open-set node classification methods rely on the homophily assumption, where connected nodes share labels. However, real-world graphs are often heterophilic, exposing the limitations of current methods and posing new challenges to open-set node classification. On the one hand, cross-class connectivity causes representations from different known or unknown classes to become intertwined after aggregation, undermining their discriminative capacity. On the other hand, structural mixture invalidates threshold-based open-set methods and cross-class feature interpolation, leading to unreliable unknown-class rejection. To address these challenges, we propose HOPE, a Heterophily-aware Open-set node classification method with Pseudo-Extrapolation. To adapt open-set graph neural networks (GNNs) to heterophilic scenarios, HOPE uses a structure-augmented feature initialization layer to capture multi-hop structural patterns. Meanwhile, we design a trustworthy neighborhood aggregation mechanism for standard GNNs to dynamically filter noisy cross-class neighbors. To enhance unknown-class rejection, we introduce a heterophily-guided pseudo-extrapolation strategy. It dynamically maintains known-class centers and extrapolates along cross-class neighborhood displacement directions, synthesizing pseudo-unknown proxies near structurally ambiguous regions. Finally, we optimize the network with joint classification and logit margin regularization, routing synthetic proxies into a dedicated rejection slot without imposing
geometric margin constraints in the representation space. Extensive experiments on multiple
datasets show that HOPE consistently outperforms state-of-the-art models, validating its effectiveness, robustness, and efficiency.
Read the full abstract ▾
All authors Yumeng Dai, Yue Tan, Yixin Liu, Chenxu Wang, Pinghui Wang, Tao Qin
Machine Learning
Neither Adversarial Training Nor Purification: Emergent Adversarial Robustness from Oscillatory Predictive Learning
Mohammed-Yassine Habibi, Klea Ziu, Martin Takáč et al. (+1 more)
Adversarial robustness in computer
vision is still largely achieved through
adversarial training or test-time
adversarial purification, both of which introduce significant computational overhead by generating
adversarial examples during training or performing iterative denoising at test time. We study whether empirical robustness can instead emerge from architectural and representation-learning inductive biases. We introduce Oscillatory Predictive Learning (OPL), a two-stage framework that combines Artificial Kuramoto Oscillatory
Neurons (AKOrN) with predictive self-supervised pretraining using X-PhiNet. Because our default checkpoint uses randomized initial oscillator states, we compare it with other randomized
adversarial defense methods that provide precise, reproducible, and strong attack protocols. Experiments on CIFAR-10 and CIFAR-100, with additional corruption evaluation on CIFAR-10-C, demonstrate that our method achieves competitive results under the AutoAttack-rand evaluation protocol. On CIFAR-10 and CIFAR-100, OPL attains 76.63$\pm$0.76$\%$ and 50.44$\%$ robust accuracy, respectively, under $\ell_\infty$, $\epsilon=8/255$, AutoAttack-rand with EoT $K=20$.
Read the full abstract ▾
All authors Mohammed-Yassine Habibi, Klea Ziu, Martin Takáč, Makoto Yamada
Machine Learning · Cross-listed from stat.ML
Optimal estimation for Functional Linear Regression with Noisy Discretized Data
Sixtine Sphabmixay
In this paper, we consider the scalar-on-function linear regression model under a realistic sampling scheme in which the functional
covariates are observed on a regular grid and contaminated by additive noise. We propose a two-step estimation procedure: first, the underlying curves are reconstructed from the discrete noisy observations using a
Fourier-based projection method; second, the slope function is estimated by a penalized least-squares criterion over finite-dimensional trigonometric spaces, with data-driven selection of the model dimension. We establish oracle-type inequalities for the prediction error, both with respect to the reconstructed curves and to the true latent curves. Under regularity assumptions on the slope function and
polynomial decay of the
eigenvalues of the
covariate, we derive convergence rates for the prediction error and show that our
estimator attains the minimax rate when the number of grid points is sufficiently large. Finally, the proposed method is illustrated on simulated data and on a real meteorological
dataset.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
Learning to build covering structures with continuous adjustments
Gabriel Vallat, Maryam Kamgarpour and Stefana Parascho
Robotic construction offers the potential to use materials more efficiently and create complex geometries, but current methods rely on rigid, high-precision plans that cannot accommodate the tolerances, inaccuracies, and unexpected changes inherent in physical fabrication. In this work, we introduce a reinforcement learning approach that forgoes predefined plans entirely, instead generating construction sequences adaptively as the structure is built. Our method operates on graph-structured state representations and a mixed (parameterized) action space, requiring both discrete block selection and continuous placement parameters. Because the stability simulation of a structure is computationally heavy, we develop an efficient exploration strategy by incorporating unilateral edges into graph neural networks, extending soft actor-critic (SAC) to this hybrid setting. We evaluate our
algorithm, HSAC, against the prior method hybrid-PPO (HPPO), demonstrating significantly higher asymptotic performance and good sample efficiency. We also demonstrate HSAC's robustness to hyperparameter choices and its exploration capability, handling up to 10 discrete actions without performance degradation. Finally, we validate our approach on a physical two-
robot setup, successfully building a spanning arch with 3D-printed blocks in closed-loop execution, confirming that policies trained in simulation transfer to real hardware.
Read the full abstract ▾
Machine Learning
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models
Xuanming Cui, Shlok Kumar Mishra, Wentao Bao et al. (+6 more)
Universal
multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size,
retrieval effort, or scales the encoder into a heavy
multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via
reasoning tokens. However,
embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and
retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based
embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based
embedding models towards large-scale
retrieval and recommendation systems.
Read the full abstract ▾
All authors Xuanming Cui, Shlok Kumar Mishra, Wentao Bao, Aashu Singh, Zihao Wang, Xiangjun Fan, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng
Machine Learning · Cross-listed from cs.CV
Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing
Xiaochuan Zhong, Yifan Hou, Chenxi Pang et al. (+1 more)
Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart
benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a
benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.
Read the full abstract ▾
All authors Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui
Machine Learning
Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Youngjun Yu, Sanghwan Jang and Hwanjo Yu
Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large
Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic
reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-
entropy-guided forking overcomes the localization phenomenon of
token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-
entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand
reasoning coverage during training. Experiments on mathematical
reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.
Read the full abstract ▾
Machine Learning
SUN: Reaching for Novelty in Reinforcement Learning
Wenyan Yang, Arsenii Mustafin, Dominik Baumann et al. (+2 more)
Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL
algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN is the value of a count-scaled goal-reaching reward, bounds short-horizon hitting probabilities, and rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, along with an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough
benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces. Code at https://github.com/sparisi/rl-anvil
Read the full abstract ▾
All authors Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
Machine Learning
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Oleksandr Cherednichenko and Roman Klypa
Integrating robust safety guardrails into Large
Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization
algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and
benchmarks demonstrate that Suan achieves superior safety
alignment while fully preserving response utility.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
SynthRCT: Scalable Conditional Deformation Synthesis for Synthetic Repeat CT Generation
Tomas Guija-Valiente, Blanca Rodriguez-Gonzalez and Norberto Malpica
In
proton therapy, plans are typically optimized on a single planning CT, making robustness evaluation essential under anatomical changes. However, current scenarios often rely on simplified perturbations that poorly capture complex, patient-specific variability. We propose SynthRCT, a scalable conditional generative framework for 3D anatomical deformation synthesis. Based on a conditional variational autoencoder, SynthRCT learns a latent deformation space and decodes sampled latent codes into local stationary velocity fields conditioned on an input anatomy. Local fields are assembled into coherent full-volume transformations, enabling
memory-scalable generation for large field-of-view CT data. We validate the approach on respiratory 4DCT data with multiple breathing-phase anatomies per subject. SynthRCT enables patient-specific sampling of plausible anatomical transformations beyond predefined robustness scenarios. Code available at: https://github.com/TomasGuija/SynthRCT.
Read the full abstract ▾