Machine Learning
Conditional Transfer from Controlled Pretraining Mixtures to Code
Ohad Rubin
Synthetic tasks are increasingly used both as probes of
language-model capability and as pretraining data. Both uses are often justified by loss reduction: falling loss is treated as informative, and faster loss reduction with more sampling as evidence that a task is worth sampling. We separate three signals. A task is diagnostic when its loss tracks global pretraining progress; it is teachable when its loss responds to its own
token budget; and a data source transfers when including it improves a downstream target. We study controlled pretraining in which 70% of the corpus is fixed general Python and the remaining 30% is a simplex over three source families: OpenCodeInstruct, a curated suite of 12 code-adjacent synthetic tasks, and 15 literature-derived probe tasks. Across a task-budget sweep we detect teachability for 14 of 27 tasks, with a sharp asymmetry between the two synthetic families (10/12 curated versus 4/15 literature-derived). Teachability and downstream transfer give different rankings. On the mixture-simplex edge between the curated suite and OpenCodeInstruct, HumanEval pass@20 after a fixed fine-tuning stage rises from 15.9 at pure curated data to its highest observed value, 22.6, at a mixture that is 75% OpenCodeInstruct, then falls to 19.5 at pure OpenCodeInstruct. Curated synthetic data therefore has conditional value: it contributes as a limited share of a mixture that a target-aligned source still dominates. Finally, a loss-based adaptive scheduler exposes the mismatch between residual loss reducibility and downstream transfer. Across three 60k-step free-ratio runs, Ado drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2–1.4% by the end of training, and underperforms its matched fixed-mixture controls by 2.4–11.0 percentage points. Optimizing near-term task-loss reduction moves the mixture away from the region that transfers.
Read the full abstract ▾
Machine Learning · Cross-listed from stat.ML
LAIR-Net: Leaky Alignment-Impulse Residual Networks for Tabular Regression
Rahul Goswami, Aryan Bhambu and Bittu Karmakar
Deep randomized models fix hidden-layer parameters through random initialization and learn only closed-form readouts, typically adding depth by stacking random trans formations without target-aware control of hidden-state
evolution. We propose LAIR Net, the Leaky
Alignment-Impulse Residual Network, which mixes a shallow learned anchor into each hidden state through a leaky residual transition. We derive a depth uniform bound on input-perturbation sensitivity and use controlled simulations to attribute gains over a randomized baseline to the anchor rather than recursion or added capacity. Benefits emerge when a nonlinear target structure is learnable at the available noise level and diminish for nearly linear targets or dominant noise. Across 23
benchmark datasets, LAIR-Net achieves the best average rank among eight randomized networks and twelve conventional models, with relative performance associated with the same nonlinear-structure and noise quantities identified in simulation.
Read the full abstract ▾
Machine Learning · Cross-listed from quant-ph
An Efficient Quantum Circuit for Flow Model Execution Using Quantum Neural Networks
Rui Che and Ludvig af Klinteberg
Flow models generate trajectories from an initial distribution to a target distribution by solving an ordinary differential equation defined by a velocity field. Flow matching learns this velocity field by modeling the transport dynamics between the two distributions. Wavefunction flow establishes a formal connection between flow models and
quantum dynamics by introducing a continuity
Hamiltonian, which drives the Schrödinger
evolution of
quantum states. In this paper, we investigate accurate and efficient
quantum simulation of the wavefunction flow, thereby realizing the efficient implementation of flow models on
quantum computers. We first leverage a
quantum read-only
memory (QROM)-based phase kickback framework for the wavefunction flow simulation, generating probability densities that closely match those produced by the corresponding conventional flow model. To address the high
circuit-resource cost, we further incorporate a trained
quantum neural network (QNN) into the phase kickback framework, replacing QROM for data encoding. Numerical experiments demonstrate that our proposed method implements flow models on
quantum computers more efficiently, since it maintains the accuracy of wavefunction flow simulation compared with the QROM-based framework, and significantly reduces the
circuit resources.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification
Michael W. Spratling and Heiko H. Schütt
DNNs exhibit robustness and generalisation issues not seen in humans. They are also far less data-efficient learners, requiring considerably more training samples to accurately classify novel exemplars. Inductive bias could help with these issues by providing in-built mechanisms to improve generalisation, and hence, reduce reliance on learning from data. We incorporate a biologically-inspired inductive bias into a new activation function, HAND (Homeostasis, Accelerating Nonlinearity, and Divisive-nomalisation), and show its effectiveness with CNNs trained on image classification. Using HAND a ConvNeXt-tiny required 25 training epochs to reach the same accuracy on ImageNet1k as the unmodified model achieved after 200 epochs. Consistent with the effects of an inductive bias, the performance gap reduced with training time and increased data augmentation. When the volume of training data was reduced and unevenly distributed between classes (Long-tailed ImageNet) the improvements in accuracy were even larger and did not reduce with increased training time. Generalisation performance with the common-corruptions data, and the ability to reject samples from unknown classes, were unaffected or improved by HAND. Results generalised across CNN architectures and training data-sets. HAND can, therefore, reduce the required training time and/or the required volume and variety of training data, helping to improve sample efficiency.
Read the full abstract ▾
Machine Learning
When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning
Shutong Zheng and Sijia Chen
Federated reinforcement learning (FRL) enables distributed
agents to collaboratively train decision-making policies, but its decentralized training process also exposes global policy learning to Byzantine manipulation. Existing poisoning attacks primarily focus on how to construct malicious updates, while trajectory-level intervention timing remains largely implicit. In sequential decision making, however, where an intervention is applied can alter subsequent trajectories and learning signals. Through controlled experiments, we find that changing the selected trajectory states materially alters attack efficacy even when the malicious-update construction is fixed. We therefore identify when as a distinct attack dimension and introduce the Viability-constrained Behavioral Steering Attack (V-BSA), which uses local policy uncertainty to select sparse intervention states and applies envelope-constrained behavioral steering. Across discrete-action
benchmarks, V-BSA achieves substantial degradation against robust aggregators and ensemble defenses with only a fraction of the interventions used by dense poisoning, while revealing task- and aggregation-dependent boundaries. Overall, our results highlight intervention timing as a distinct dimension of sequential robustness in FRL. The code is available at https://github.com/Yodeesy/V-BSA
Read the full abstract ▾
Machine Learning
Compactness and Consistency: A Conjoint Framework for Deep Graph Clustering
Wei Ju, Siyu Yi, Kangjie Zheng et al. (+6 more)
Graph clustering is a fundamental task in data analysis, aiming at grouping nodes with similar characteristics in the graph into clusters. This problem has been widely explored using graph neural networks (GNNs) due to their ability to leverage node attributes and graph
topology for effective cluster assignments. However, representations learned through GNNs typically struggle to capture global relationships between nodes via local message-passing mechanisms. Moreover, the redundancy and noise inherently present in graph data may easily result in node representations lacking compactness and robustness. To address these issues, we propose a conjoint framework CoCo, which captures compactness and consistency in the learned node representations for deep graph clustering. Technically, our CoCo leverages graph convolutional filters to learn robust node representations from both local and global views, and then encodes them into low-rank compact
embeddings, thus effectively removing the redundancy and noise as well as uncovering the intrinsic underlying structure. To further enrich the node semantics, we develop a consistency learning strategy based on compact
embeddings to facilitate knowledge transfer from the two perspectives. Our experimental results indicate that our CoCo outperforms state-of-the-art counterparts on various
datasets.
Read the full abstract ▾
All authors Wei Ju, Siyu Yi, Kangjie Zheng, Yifan Wang, Ziyue Qiao, Li Shen, Yongdao Zhou, Xiaochun Cao, Jiancheng Lv
Machine Learning · Cross-listed from physics.optics
Learning qBIC Resonances across Metasurface Families in Dielectric Fourier Space
Shuangteng Lei, Li Yu, Tianxin Li et al. (+1 more)
Bound states in the continuum (BIC) metasurfaces are typically described by geometry-specific parameters, hindering cross-geometry comparison, while ultranarrow qBIC features are easily diluted in full-spectrum learning. Here, 2015 samples from seven dielectric metasurface families are mapped to a shared reciprocal-
lattice grid, where two frozen low-order
Fourier channels capture resonance shifts with mean within-branch $R^2$ values of 0.871-0.999. Field-level analysis of two representative branches further confirms that these shifts are consistent with the Maxwell-
Fourier perturbation picture. A five-channel K-space backbone models the broadband spectrum, while a local complex K-space expert parameterizes the qBIC resonance through a differentiable Fano layer. The expert reduces resonance-position mean absolute error (MAE) from 3.2 to 0.95 nm and the resonance-depth error by 14-fold on a geometry-blocked test set. The same coordinate supports spectrum-to-structure reconstruction.
Read the full abstract ▾
All authors Shuangteng Lei, Li Yu, Tianxin Li, Wei Lu
Machine Learning · Cross-listed from stat.ML
PSI-SINDy: Post-Selection Inference for Sparse Identification of Nonlinear Dynamics
Ashraful Islam, Shuichi Nishino, Tomohiro Shiraishi et al. (+1 more)
Sparse identification of nonlinear dynamics (SINDy) is a data-driven framework for discovering governing dynamics from time-series data by identifying a sparse subset of candidate dynamical terms from a prespecified library. In this work, we develop a statistical inference framework for quantifying the reliability of dynamical terms selected by SINDy through hypothesis tests and confidence intervals. A key difficulty is that using the same noisy trajectory for both selecting dynamical terms and assessing their statistical significance can introduce selection bias. Post-selection inference provides a principled framework for addressing such bias, and we propose PSI-SINDy, a post-selection inference method tailored to SINDy. Direct application of existing post-selection inference techniques is challenging because SINDy involves measurement error in the candidate terms and shared noise between the response and design. To address these challenges, PSI-SINDy uses data thinning to decompose a single observed trajectory into four mutually independent views with distinct roles in selection and inference. This construction enables inference for selected dynamical terms while accounting not only for selection bias but also for measurement-error and shared noise effects. We establish the theoretical validity of PSI-SINDy under stated conditions and evaluate its performance through numerical experiments on simulated and experimental dynamical-system data.
Read the full abstract ▾
All authors Ashraful Islam, Shuichi Nishino, Tomohiro Shiraishi, Ichiro Takeuchi
Machine Learning · Cross-listed from cs.SE
Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina et al. (+2 more)
Context: Large
language model (LLM)
agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight
agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts. Objectives: We evaluate whether local LLM
agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure. Methods: We introduce a
benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU. Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB
memory footprint. Gains are largest when intermediate artifacts expose errors the
agent can inspect and repair. Conclusion: Local open-weight
agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The
benchmark provides a reproducible method for evaluating complete
agent configurations before adoption in engineering workflows.
Read the full abstract ▾
All authors Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo
Machine Learning · Cross-listed from cs.CV
Conditional Residual Prediction: Improving Autoregressive Video Diffusion without a Bidirectional Teacher
Bowen Zheng, Zhiguang Liu, Jiarong Ou et al. (+2 more)
Causal
video diffusion models generate
video autoregressively, which suits streaming, interactive, and long-
video generation. Under standard training, however, they often yield lower generation quality than bidirectional models of the same size. Many existing approaches address this gap by initializing from or distilling a pretrained bidirectional teacher. We instead train a causal model from an image-model initialization, with no bidirectional
video model at any stage. Because this path requires neither a large bidirectional teacher nor a complex
distillation pipeline, it is simpler and more scalable. On this path, we find that a causal model trained on ground-truth history becomes strongly dependent on it, so that at inference errors in its own generated history propagate forward. We hypothesize that much of this dependence is unnecessary, because the current input already determines much of what the history provides. We propose Conditional Residual Prediction (CRP), a simple recipe for reducing a model's reliance on a condition: the model first predicts the target without the condition, and the condition may only add a residual on top of this prediction. Applied to history, CRP makes the model predict each chunk from the present as far as it can and use the past only for what the present cannot supply. In controlled experiments, CRP nearly closes the 6.14-point gap to a bidirectional model trained under the same setup. Scaling this recipe, we train Optica, a 2B-parameter causal
video model that autoregressively generates 5-second 480p
videos and reaches 82.78 on VBench with only about 15M training
videos.
Read the full abstract ▾
All authors Bowen Zheng, Zhiguang Liu, Jiarong Ou, Rui Chen, Tianyang Hu
Machine Learning
Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD
Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai et al. (+3 more)
Population gradient flow is a common tool for
reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch
stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the two
neurons toward positive proportionality and a target task rewards separating them. After source training for time $T$, gradient flow recovers on the target in time linear in $T$. Online SGD with batch size $b$ and step size $\eta$ in both phases instead fails with high probability throughout a horizon of order $e^{c/\eta}$ once $T \gtrsim \log(b/\eta)$, uniformly on an explicit set of initializations with Gaussian probability above one percent. For each fixed $T$, small-step SGD still recovers, so the failure requires the joint limit of small steps and long pretraining. At the target clone, the population instability is carried entirely by inputs on which the two ReLU gates disagree. For units at angle $\delta$ these inputs form a wedge of probability $\delta/\pi$, and weight decay shrinks the angle exponentially during pretraining. On every other input both units receive the same random linear update, which contracts their separation in conditional expectation. Bounding the cumulative probability of sampling the wedge along the exact online recursion, without a
diffusion approximation, shows that recovery with fixed probability from an identical source-gradient-flow checkpoint, within $e^{c/\eta}$ updates, requires $Nb \gtrsim e^{\lambda T}$ target samples and batch size $b \gtrsim \eta e^{\lambda T}$, where $N$ counts updates and $\lambda$ is the weight decay. In simulations, recovery is approximately a function of the disagreement budget $b\delta/\eta$ and saturates in the horizon.
Read the full abstract ▾
All authors Ruoyu Zhao, Mingxuan Zhang, Jianbo Dai, Jiaqi Wu, Chenyu Zhu, Tong Che
Machine Learning · Cross-listed from cs.AI
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Xing Zhang, Guanghui Wang, Yanwei Cui et al. (+4 more)
We changed the
agent: did it actually get better? Every self-improving
agent loop answers this hundreds of times, and every answer comes from a verifier. On open-ended tasks none exists, so the loop is handed a hand-written rubric or a bare LLM judge grading output from a model like itself, inviting reward hacking and shared blind spots. We make the verifier the evolving object: an inspectable expression over small, mostly deterministic drawback detectors, synthesized from clustered failures, gated at birth, and selected for agreement with a ten-item anchored reference set plus consensus over unlabeled outputs, never for the
agent's score. On MBPP+ it gains +0.21 held-out agreement over the hand-authored seed composition, on every seed, and ends ahead of the bare LLM judge it contains. One finding should change how co-evolved verifiers are validated: removing the anchor guards collapses the verifier into a vacuous always-pass grader, yet that collapsed verifier trains skills just as well. Downstream task score cannot certify a self-evolved verifier. Score does answer sufficiency, and there an evolved verifier can substitute: Double Ratchet, pairing the verifier with a lifecycle-managed skill loop, retains 88-110% of the lift that ground truth or a rubric buys the same loop, across code generation, enterprise text-to-SQL, and reference-free report generation. When evolved skills gamed the report rubric, an outer judge caught it and one added detector repaired it; the judge itself was wrong until given the task contract.
Read the full abstract ▾
All authors Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He
Machine Learning · Cross-listed from stat.ML
Feature Space Adaptation for Effortless Gaussian Process Flows
Thomas Cowperthwaite, Louis Sharrock, Lachlan Astfalck et al. (+1 more)
Outside the linear-Gaussian regime, conditional sampling from Gaussian processes (GPs) is challenging. Recent methods such as FlowGP (Moss et al., (2026)) can condition on arbitrary non-linear and non-Gaussian statements, but at considerable cost: an expensive iterative and high-dimensional
diffusion that requires hand-specified kernel hyperparameters. In this paper, we alleviate two significant drawbacks of FlowGP by (1) introducing kernel approximations that enable scaling to high-resolution domains and (2) proposing a way to obtain the marginal likelihood by measuring the work needed to steer the
diffusion towards conditioning statements. We enable, for the first time, hyperparameter optimisation within FlowGP and demonstrate our approach on probabilistic downscaling from areal summary statistics, PDE solution inference on irregular domains, and recovery of sea level anomaly fields from non-Gaussian satellite observations.
Read the full abstract ▾
All authors Thomas Cowperthwaite, Louis Sharrock, Lachlan Astfalck, Henry Moss
Machine Learning
Kislay Aditya Oj, Nidhi Jain, Sri Surya Varma Datla et al. (+3 more)
AI research progress can be viewed as the interaction between two processes:
benchmark creation and method discovery. Historically, both were driven by human intelligence. However, recent advances in AI have accelerated automated method discovery, while automated
benchmark creation has received comparatively less
attention. To enable self-advancing systems, we propose Generative
Adversarial Loop (GAL), a generator-discriminator framework alternating between two agentic searches: (1) a discriminator that generates
adversarial data to expose weaknesses in current systems, and (2) a generator that discovers
algorithms to overcome them. We apply this framework to approximation
algorithms for efficient inference. Unlike existing auto research systems, which primarily focus on
algorithm discovery, GAL introduces a discriminator
agent that automates goalpost setting by continually searching for weaknesses in the current
algorithm. We demonstrate
adversarial data generation across four tasks: KV compression, sparse
video generation, sparse
attention, and context extension, where the discriminator identifies weaknesses in state of the art techniques. We further show that GAL enables autonomous improvement, with newly discovered
algorithms improving not only on adversarially generated data, but also on established
benchmarks. Specifically, GAL improves CompactorPress on KV compression with Qwen3-4B at 4x, raising performance on the discriminator
dataset from 0.35 to 0.97, while also outperforming RULER-HARD (+0.77 pts). For context extension, GAL boosts Dual Chunk
Attention from 0.20 to 0.90 on the discriminator
dataset, while yielding gains on standard
benchmarks(ScienceFiction (+6 pts) and PG19 32K (-0.33 PPL)). GAL thus provides a path toward autonomous goalpost setting and algorithmic improvement, where AI systems continually discover their own weaknesses and develop methods to overcome them.
Read the full abstract ▾
All authors Kislay Aditya Oj, Nidhi Jain, Sri Surya Varma Datla, Priyanka Jayaswal, Kumar Krishna Agrawal, Aditya Desai
Machine Learning
Zatom-2: Multitask Pretraining on Atomistic Data for Generative Modeling across Domains
Miruna Cretu, Alex Abrudan, Antonia Panescu et al. (+11 more)
Unified atomistic modeling has the potential to accelerate discovery in chemistry, materials science, and biology by bridging data-rich chemical domains and data-scarce biological contexts. However, existing generative approaches to atomistic modeling remain highly specialized to scientific disciplines (chemistry vs. biology) or do not leverage both high-volume organic (molecule) and inorganic (material) data for general-purpose pretraining. To this end, we introduce Zatom-2, an atomistic generative model pretrained on approximately five million structures from the OMol25 and OMat24 electronic structure
datasets. Zatom-2 features a multiscale
Transformer architecture coupled with conditional flow matching that supports force conditioning and foundational pretraining tasks such as generation, structure prediction, and prediction of molecular and material energies and forces. Empirically, Zatom-2 achieves better molecular distribution fidelity than Zatom-1 and achieves strong performance on existing molecule and material generation
benchmarks. Zatom-2 demonstrates the ability to control sample generation across low- and high-force regimes, and enhances
protein generation in a low-data setting through joint generative-predictive pretraining and transfer learning, increasing
protein backbone designability in a length extrapolation setting from 67.8% without pretraining to 74.8% after finetuning on 2,000
protein domains.
Read the full abstract ▾
All authors Miruna Cretu, Alex Abrudan, Antonia Panescu, Tynan Perez, Rishabh Anand, N. Benjamin Erichson, Michael W. Mahoney, Samuel Blau, Joseph Jacobson, Rafael Gómez-Bombarelli, Rex Ying, Tuomas Knowles, Pietro Liò, Alex Morehead
Machine Learning · Cross-listed from cs.SE
Closed-loop evaluation of LLM agents for embedded software development
Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté et al. (+1 more)
Large
language models (LLMs) are increasingly deployed as coding
agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-
agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a
benchmark for closed-loop evaluation of embedded coding
agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The
agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the
benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding
agents are emerging.
Read the full abstract ▾
All authors Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo
Machine Learning
MotiveMob: Motivation as Semantic Action for Closed-Loop Human Mobility Generation
Mengkun Gao, Zengqing Wu, Renhe Jiang et al. (+6 more)
Human mobility generation, an important task in urban research, synthesizes trajectory data for urban planning and transportation management. Human mobility can be characterized as a "why-where-when" decision process: people form an intention to move and then determine where and when the corresponding activity will take place. Trajectory generation under user-level and temporal distribution shifts may benefit from explicitly modeling this decision structure. However, many existing human mobility generation methods either represent behavioral intent at a coarse granularity, such as a daily plan or a trajectory-level description, or directly predict future locations without explicitly
reasoning about a possible motivation for each movement step. We introduce MotiveMob, a motivation-driven autoregressive framework for human mobility generation that first forms a hypothesis about why the next movement may occur and then jointly generates where and when it may occur. At each step, a motivation predictor conditions on the current mobility state, a long-term behavioral report, and the mobility history to infer a plausible motivation or determine whether the trajectory should terminate. Given the hypothesized motivation, a state predictor grounds it in a candidate next location and arrival time. The candidate then undergoes speed-feasibility and repetition checks before being fed back for the next decision. We evaluate MotiveMob under distribution shifts involving unseen users and unseen temporal periods, including seasonal changes and the substantial behavioral disruption caused by the COVID-19 pandemic. Experiments show that MotiveMob consistently achieves better distributional fidelity than competitive pretraining-based and prompting-based methods under user-level and temporal distribution shifts, demonstrating robust generalization to out-of-distribution mobility patterns.
Read the full abstract ▾
All authors Mengkun Gao, Zengqing Wu, Renhe Jiang, Jiawei Wang, Yusong Wang, Chuang Yang, Shuyuan Zheng, Makoto Onizuka, Chuan Xiao
Machine Learning · Cross-listed from cs.CL
BioBigBird: A Sparse Attention Model for Long-Range Dependency Processing in Biomedical Text
Roshan Balaji, Pavan Kumar S, Vasudev Gupta et al. (+3 more)
While domain-specific Large
Language Models (LLMs) have encoded vast biomedical knowledge, their limited context windows often hinder a deep understanding of nuanced relationships within and across texts. To address this limitation, we introduce BioBigBird, a bidirectional
language model pre-trained on extensive biomedical literature and clinical data, specifically designed to handle long-range dependencies. BioBigBird leverages a sparse
attention mechanism to process sequences up to 4096
tokens, and its training incorporates a multi-stage process to mitigate noise from the large-scale pre-training corpus. We further enhance its performance by employing a multi-task learning (MTL) framework that jointly optimizes for Named Entity Recognition and Relation Extraction. Comprehensive evaluations on the BLURB
benchmark reveal that our MTL-enhanced BioBigBird achieves highly competitive results against state-of-the-art models. Our work contributes an effective methodology for developing powerful, long-context
language models for specialized domains, demonstrating the value of extended sequence processing for complex text analysis. Our models are publicly available at https://huggingface.co/collections/bisectgroup/biobigbird.
Read the full abstract ▾
All authors Roshan Balaji, Pavan Kumar S, Vasudev Gupta, Sreejith N, Keerthana Sridhar, Nirav Bhatt
Machine Learning
Policy Alignment: New Signals for Membership Auditing in On-Policy Distillation
Yilong Yang, Wenzhuo Shang, Yule Liu et al. (+2 more)
On-policy
distillation (OPD) trains a student model by aligning its policy with a teacher model on trajectories generated by the student model itself. Through this process, the student policy moves toward the teacher on the
prompts used for
distillation. However, these
prompts are often private and costly, creating a need for
prompt-level membership auditing. Existing methods mainly rely on likelihood-based confidence signals or student policy drift between checkpoints, but they do not capture the teacher-induced direction of the student update. In this paper, we propose Policy
Alignment Membership Auditing (PAMA), a new auditing framework tailored for OPD. Our key observation is that a member
prompt directly contributes to the teacher-guided policy update, while a non-member
prompt only experiences indirect effects through cross-
prompt generalization. Based on this directional trace, PAMA measures whether the student update moves toward reducing the teacher loss on a candidate
prompt. Specifically, we introduce Teacher
Alignment Gain (TAG) to estimate the teacher-aligned update direction from model outputs, and further combine it with student drift and uncertainty
alignment signals for reliable membership auditing. We evaluate PAMA on six
datasets and three teacher-student model families. On MATH, the primary evaluation
benchmark, PAMA achieves AUC values of 0.791–0.941, improving AUC by 14.6–20.6% over state-of-the-art baselines.
Read the full abstract ▾
All authors Yilong Yang, Wenzhuo Shang, Yule Liu, Jiale Teng, Zhuo Ma
Machine Learning · Cross-listed from eess.SY
Causal-fate dynamics of unrealized influence
Yiwei Liu, Luwei Yang and Shunbo Lei
Many dynamical systems generate influences whose consequences are not fully exhausted in the realized trajectory at the moment they arise. Such consequences are often treated as absent, delayed or statically stored, leaving unclear how unrealized influence retains future relevance as the system evolves. Here we formulate causal-fate dynamics, in which generated influence may be realized, remain latent, or be transformed by subsequent dynamics, and give an exact finite-transport representation when the relevant maps are specified. A connectome-constrained Caenorhabditis elegans model first motivates the biological hypothesis that unresolved inter-neuronal influence may persist and contribute to later propagation; it does not establish such a mechanism in living animals. We next examine operational Internet routing, where a dynamically updated cross-observer history retains predictive information beyond the current local route state. We then use the representation to construct a
Transformer architecture that explicitly transports and selectively realizes latent contextual influence while retaining
language-modeling function. The three studies distinguish a model-motivated scientific hypothesis, an observational phenomenon compatible with future-relevant history and an executable construction for carrying unrealized influence through subsequent computation.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.GT
Refinement as a Service: Algorithmic Predictor Refinement
Wei Tang and Hanrui Zhang
Prediction aggregation aims to combine information from multiple predictors into a more informative one. We study this question in the setting of calibrated predictors, where each prediction must equal the conditional expectation of the quantity being predicted given the predictor's signal. Given several calibrated input predictors and the feature distribution, but not the underlying Bayes probabilities, we ask when one can construct refined calibrated predictors that preserve the information in the original predictors and cannot be further refined using the available information. We formulate calibrated predictors as signaling schemes and define refinement through feature-independent garblings: a predictor refines another if its signal can simulate the other's signal. Constructibility is characterized through observable linear information: each signal corresponds to a vector over the feature space, and a new signal is constructible exactly when its vector lies in the linear span of the input signal vectors. Under this formulation, we establish a sharp algorithmic picture. For deterministic output predictors, bilateral refinement admits a
polynomial-time
algorithm based on a bipartite graph between the two input signal partitions, while refinement with an arbitrary number of input predictors is $\mathsf{NP}$-hard. In contrast, when randomized output predictors are allowed, we give a
polynomial-time
algorithm for any number of input predictors by decomposing constructible signal vectors into extreme rays of the associated polyhedral cone.
Read the full abstract ▾
Machine Learning
MC-TRCM: Observation-Aware Recursive Fusion for Incomplete Mobile and Wearable Mental-Health Feature Views
Wentao Wang, Lifeng Han, Zining Ren et al. (+2 more)
Public mobile and wearable mental-health
datasets often provide summarized feature tables rather than synchronized raw sensor streams. In these releases, each anchor corresponds to a survey or label time and may combine phone or wearable summaries, prior symptom scores, demographics, clinical variables, and source-availability indicators. We propose the Modality-Conditioned Temporal Recursive Context Model (MC-TRCM), which preserves each feature source as a separate
token and incorporates missingness as part of the input context. Observed sources are encoded with values and missingness summaries, absent sources use learned absence
tokens,
dataset and task
embeddings condition fusion, and a recursive prediction head refines each output over validation-selected steps. We evaluated MC-TRCM on six predefined endpoints from DepreST-CAT and Prediction of Severity Change-Depression (PSYCHE-D) using participant-level splits and validation-only model selection. MC-TRCM achieved the lowest mean absolute error on DepreST-CAT Patient Health Questionnaire-9 (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) severity, improving over the best tabular reference by 0.181 and 0.217 scale points. Classification endpoints showed task-dependent behavior: MC-TRCM matched the best rounded GAD-7 category balanced accuracy, was numerically highest by 0.002 balanced-accuracy points on PSYCHE-D multiclass prediction, and remained close to the strongest references on PHQ-9 category and PSYCHE-D binary prediction. Ablations support Feature-wise Linear Modulation, absence
tokens, missingness projections, and recursive refinement, while calibration and feature-source controls characterize endpoint behavior. Our code is available at https://github.com/Botwwt/MC-TRCM.
Read the full abstract ▾
All authors Wentao Wang, Lifeng Han, Zining Ren, Hengyu Zhong, Guangyu Zou
Machine Learning · Cross-listed from stat.ML
Beyond Distributional Fidelity: Causal-Penalized Diffusion for Synthetic Tabular Data
Lan Tao, Yongxian He, Shirong Xu et al. (+2 more)
Synthetic tabular generators are commonly optimized for distributional fidelity, but statistical similarity alone does not guarantee preservation of causal effects. In this paper, we study whether causal fidelity can be improved directly within a fully generative tabular model. Causal Fidelity is defined with respect to a target estimand as the discrepancy between inferential distributions obtained from real and synthetic data, and theoretical results show that high statistical fidelity does not generally imply high causal fidelity. We then propose a causal-fidelity-aware training framework which adds a causal discrepancy penalty to the generative objective. The framework is instantiated with a causal-penalized TabDDPM and optimized using an on-policy score-function
estimator. We further establish conditions under which causal regularization improves expected causal fidelity. Experiments across diverse treatment-effect simulations and two
benchmark datasets evaluate the ability of our method to improve causal fidelity while preserving competitive statistical fidelity.
Read the full abstract ▾
All authors Lan Tao, Yongxian He, Shirong Xu, Yidong Ouyang, Guang Cheng
Machine Learning
Learning from Hetero Density for Cryo-EM Protein Reconstruction
Xu Han, Chaozhuo Li, Xiaowei Yuan et al. (+3 more)
Reconstructing
protein structures from cryo-electron microscopy (cryo-EM) maps is essential for understanding macromolecular assemblies. Although learning-based methods have improved
protein reconstruction, information from hetero components remains underused. Our analysis finds both false predictions and reference
protein sites near hetero components; filtering nearby candidates can improve or impair chain construction. We introduce CryoCue, a framework that uses hetero information to guide
protein reconstruction. An anchor-supervised detector learns hetero representations across five component classes. Multiscale hetero features guide backbone localization, while predicted hetero candidates condition structure refinement through their class, confidence, and frame-relative geometry. Experiments show that CryoCue improves backbone localization near hetero components and achieves more accurate
protein structure reconstruction.
Read the full abstract ▾
All authors Xu Han, Chaozhuo Li, Xiaowei Yuan, Yuancheng Sun, Kang Liu, Qiwei Ye
Machine Learning · Cross-listed from cs.RO
WAM-Cache: Staleness-Bounded KV Reuse for Efficient World Action Models
Kai Ding, Yang He, Ruijie Quan et al. (+1 more)
World Action Models (WAMs) enable generalist
robot manipulation by conditioning an action expert on representations from a pretrained
video Diffusion Transformer (DiT). In closed-loop control, the
video DiT runs at every chunk to encode the current observation into layerwise key-value (KV) pairs that the action expert queries. This prefill dominates the per-chunk computational cost, yet existing training-free accelerations leave it fully dense. We present WAM-Cache, a training-free framework that retains layerwise key-value representations across chunks and recomputes only a sparse refresh set of
tokens. Crucially, we find that the intuitive heuristic of refreshing visually drifted
tokens plateaus far below the dense baseline, even with an oracle predicting ground-truth KV drift. Downstream action accuracy is instead governed by where the action expert attends, not by what moved. WAM-Cache therefore selects the refresh set by uniting the action expert's cross-
attention with visual latent surprise, complemented by a strict age bound that suppresses compounding error. On Fast-WAM, WAM-Cache cuts
video DiT prefill FLOPs by 32-42% across RoboTwin 2.0, LIBERO, and real-world experiments, while staying within 0.7-1.8 percentage points of the dense policy in simulation and 2.5 points on a real
robot.
Read the full abstract ▾
All authors Kai Ding, Yang He, Ruijie Quan, Yi Yang
Machine Learning · Cross-listed from cs.AI
Estimating Great Expectations Under Autoregressive Language Models with Potentials
Francesco I. Re, Shubhangi Ghosh, Tim Vieira et al. (+1 more)
Many applications of
language models hinge not on individual samples but on the expectation of a test functional under the model. Estimating such expectations reliably can be computationally expensive. In this paper, we show how to make estimation more efficient by exploiting the next-
token conditional probabilities which are available as a by-product of sampling. We do so through potentials: real-valued functions on prefixes that decompose the test functional additively. We construct an
estimator whose variance depends on the chosen potential, and derive conditions under which a potential reduces this variance. We then develop practical potentials for several estimands and applications, and demonstrate substantial variance reductions across several estimands at comparable computational cost.
Read the full abstract ▾
All authors Francesco I. Re, Shubhangi Ghosh, Tim Vieira, Ryan Cotterell
Machine Learning
CoPoE: Multimodal Fusion via Decomposable Disease-Coordinate Product-of-Experts for Missing-Modality Alzheimer's Diagnosis
Chihun An and Ikbeom Jang
Multimodal Alzheimer's disease (AD) diagnosis benefits from integrating heterogeneous clinical, imaging, genomic, and biomarker evidence, but clinical cohorts frequently suffer from irregular modality missingness. Existing fusion methods often synthesize absent inputs, risking the introduction of artificial surrogates, or pool available signals into uninterpretable latent spaces. We present CoPoE (Disease-Coordinate Product-of-Experts), a disease-coordinate framework that maps
multimodal evidence into a structured latent space partitioned into four distinct biological and clinical axes: genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage (R/P/N/S). Each observed modality parameterizes a diagonal Gaussian expert over the full RPNS vector, and a masked Product-of-Experts architecture fuses only the available modalities. Consequently, absent modalities add no factor to the fusion path, allowing the network to preserve a robust, decomposable posterior for any non-empty modality subset without synthetic imputation in the RPNS path. Through extensive missing-modality experiments on the ADNI
dataset, CoPoE achieves the best all-modality performance and the highest mean AUROC across all 15 observed-subset evaluations among standardized missing-modality fusion baselines under a shared non-PET ADNI
embedding benchmark, while substantially improving raw-probability ECE, Brier score, and NLL. Furthermore, PET-supervised probing shows evidence enrichment within the pathology (P) block under full modalities, with tau-related signal retained even when direct fluid biospecimen inputs are withheld. Our code is available at https://github.com/labhai/CoPoE.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.RO
PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving
Jinchang Xu, Hongda Yu, Fengwei Dong et al. (+9 more)
World models in end-to-end autonomous driving predict future scene
evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future
reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight
Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.
Read the full abstract ▾
All authors Jinchang Xu, Hongda Yu, Fengwei Dong, Wenhui Huang, Xi Wei, Yongzhi Liu, Sunan Zhang, Jirao Wang, Chen Lv, Bingbing Li, Guodong Yin, Weichao Zhuang
Machine Learning
From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning
Fengyuan Liu, Yue Wang, Hangxi Guo et al. (+4 more)
Continual learning remains challenging for large
language models, which must enable models to acquire new skills and knowledge without degrading existing capabilities. Existing approaches typically address this challenge by carefully designing how model parameters are updated. In contrast,
prompt optimization avoids costly parameter updates while achieving competitive or even superior performance to reinforcement learning methods such as GRPO on individual knowledge-intensive and
reasoning tasks. This raises a natural question: Can
prompt optimization, as an efficient adaptation approach, be directly applied to continual learning? Our analysis shows that, under sequential task adaptation, it suffers from catastrophic forgetting, while optimized
prompts accumulate rules that overfit to local task distributions. To address these limitations, we propose Evolving Functional REpertoires (EFRE), which replaces a single
prompt with a repertoire of functions that evolves as new tasks arrive: compatible updates refine existing functions, while conflicting updates trigger the emergence of new ones. On a three-task continual-learning stream, EFRE achieves a final average performance 7.50 percentage points higher than GRPO. Moreover, after adaptation to the Bio task, its performance on FinQA decreases by only 1.56 percentage points, compared with 25.10 percentage points for the base
prompt optimization method. We further instantiate EFRE in a minimal
agent system and observe consistent improvements across different backbone models. Overall, these results demonstrate EFRE's strong performance in continual learning for large
language models and highlight its substantial potential for continual learning in advanced
agent systems.
Read the full abstract ▾
All authors Fengyuan Liu, Yue Wang, Hangxi Guo, Fengyuan Liu, Chenxu Wu, Yanguang Liu, Mengnan Du
Machine Learning
SpatialOPSD: Self-Distilling Spatial Intelligence from Verified Coding Agent Traces
Rongxue Li, Meng Yang, Yiru Mao et al. (+6 more)
Spatial coding
agents significantly improve spatial
reasoning in
Multimodal Large
Language Models (MLLMs) by using external tools to generate verified execution traces. However, this paradigm inherently suffers from prohibitive inference-time overhead and external dependencies. In this paper, we explore whether an MLLM can internalize this agentic capability to operate entirely tool-free. We begin with a simple observation: prompting an MLLM with summarized execution traces of a spatial coding
agent naturally unlocks the model's internal spatial Chain-of-Thought (CoT). Motivated by this, we introduce SpatialOPSD, an on-policy self-
distillation framework that internalizes spatial
reasoning into a standalone MLLM by formulating verified
agent traces as privileged information. To mitigate privileged-information leakage during
distillation, we introduce Repetition-Aware
Distillation, which combines repetition masking with unlikelihood regularization. Experiments across multiple
benchmarks demonstrate that self-distilling SpatialOPSD achieves higher average accuracy than SFT and GRPO on both spatial and OOD
datasets, exhibiting superior performance and generalization.
Read the full abstract ▾
All authors Rongxue Li, Meng Yang, Yiru Mao, Yongliang Tao, Lulu Hu, Bin Yang, Zhao Xu, Weihua Luo, Bowen Xu