Machine Learning
Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics
Zubin Zheng, Jiahao Wu, Shaofeng Zhang et al. (+3 more)
On-policy
distillation (OPD) trains multi-turn
language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy
reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose ActFirst-OPD, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for
token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of $2.3\times$ on ALFWorld, $1.8\times$ on WebShop, and $4.9\times$ on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine
benchmark-model settings. These results demonstrate that
reasoning need not block acting during multi-turn
agent distillation.
Read the full abstract ▾
All authors Zubin Zheng, Jiahao Wu, Shaofeng Zhang, Zhirui Zhang, Yew-Soon Ong, Shengcai Liu
Machine Learning · Cross-listed from cs.CV
Scaling Video Generation for Reasoning: At What Cost?
Weihang Guo, Xiaoyu Wu, Yifei Wang et al. (+2 more)
We study whether scaling
video generation enables models to reason about hidden information from the past frames, and at what computational cost. Our controlled
benchmark requires predicting nine prescribed moves of an initially solved 2x2x2 Rubik's Cube from a fixed view of three faces. Correct predictions require inferring how actions change hidden states, and the simulator provides exact ground truth for evaluation. Models learn plausible cube geometry early, while correct sticker configurations require substantially more training. Although validation MSE follows approximate power-law scaling, lower MSE loss does not reliably indicate downstream
reasoning capabilities. Smaller autoregressive models achieve higher state accuracy with limited compute, while larger models reach higher accuracy after more training. At roughly 0.1 PF-days, the 70M-parameter model correctly predicts the visible sticker configuration in 44.6% of post-action frames, compared with 0.3% for the 1B model, which reaches 83.7% at 3.14 PF-days. Symbolic state supervision raises the 20M model's frame accuracy from 31.1% to 67.3% at the same training-data budget, suggesting that learning representations of state changes can complement scaling.
Read the full abstract ▾
All authors Weihang Guo, Xiaoyu Wu, Yifei Wang, Niloofar Mireshghallah, Lydia E. Kavraki
Machine Learning · Cross-listed from cs.CL
SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
Hankun Lin, Patrick Pynadath and Ruqi Zhang
Self-speculative decoding accelerates large
language model (LLM) inference by drafting
tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-
token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only
transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit
tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple
tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple
benchmarks show that SEED achieves up to 2.7$\times$ average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.
Read the full abstract ▾
Machine Learning
Learned Reporting Preferences in RLVR Can Conflict with the Current Request
Yupeng Chang, Wenxuan Zhang and Yuan Wu
Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving
language-model performance on
reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request that the initial policy already follows. To test this, we train matched policies under two reporting conventions and evaluate each policy under both current requests, using the same initial policy as a shared reference. We complement this crossed design with controlled interventions and independent human calibration. On GSM8K, boxed-format RLVR reduces the fraction of Qwen2.5-7B responses containing the requested hash-format payload by 35.33–74.37 percentage points relative to a 95.45% initial baseline in four of five training seeds; the fifth improves by 2.50 points. In the four deteriorating runs, almost every response that omits the requested payload instead retains the trained boxed convention, and the same four seeds deteriorate under two fixed paraphrases. Changing only the final-answer marker in supervised targets reverses which reporting convention the model prefers across three seeds, providing controlled evidence that this preference is learnable. Across three settings with independent human calibration, gains under a convention-sensitive scorer exceed the corresponding gains in committed-answer correctness, i.e., the correctness of the answer the model actually commits to. Together, these results separate three distinct post-training outcomes: learned reporting preference, current-request adherence, and committed-answer correctness. They show that convention-matched accuracy alone does not fully characterize post-training behavior and motivate evaluating current-request adherence alongside convention-matched task accuracy.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.AI
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Zehao Jin, Ruixuan Deng and Junran Wang
Pretrained
transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1.4-3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA at one early layer extends this computation with all model weights frozen. Qwen3-8B improves from 15.5% to 99% exact accuracy on 24-line chains; a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers. Frozen heads read progressively further up the chain, and removing parent-line
attention stops the relay. A frozen-model measurement locates the last useful intervention layer within tolerance in three of four held-out models. Task-specific LoRAs also improve MuSiQue. Default answers therefore understate the computation accessible through a tiny edit. Code and an interactive demo are available at https://lunamos.github.io/stop-thinking-too-early/
Read the full abstract ▾
Machine Learning
Making Analog Training Scale: Co-Designing Mapping, Optimizer, and Converters
Zhaoxian Wu, Tayfun Gokmen, Omobayode Fagbohungbe et al. (+2 more)
Analog in-
memory computing (AIMC) offers an alternative for model training by executing
matrix operations directly where weights are stored. However, scaling AIMC to train modern deep models remains an open challenge due to severe hardware non-idealities, including physical weights with finite dynamic range and write granularity, analog-digital converters with finite resolution, and noisy and asymmetric updates. Guided by the insight that gradient accumulation is sensitive to precision and rounding errors, we adopt a mixed-precision training paradigm: executing forward and backward
matrix multiplications in the analog domain while computing weight gradients in the digital domain. To enable scalable training, we present a holistic system-
algorithm co-design that co-optimizes weight mapping to ensure well-conditioned physical and logical weight profiles, couples a preconditioned optimizer with threshold-triggered open-loop pulsing to stabilize training trajectories, and aligns converter dynamic ranges to suppress quantization errors. Evaluated via hardware-calibrated architectural simulations calibrated with electrochemical RAM measurements, our framework scales
Transformer training up to $123\text{M}$ parameters with validation loss scaling as $L\propto N^{-0.231}$, where $N$ is the parameter count, comparable to $L\propto N^{-0.238}$ for digital training.
Read the full abstract ▾
All authors Zhaoxian Wu, Tayfun Gokmen, Omobayode Fagbohungbe, T. Patrick Xiao, Tianyi Chen
Machine Learning
Factorized Scheduling Principle: Learning Interpretable and Transferable Policies via Structured Additive Functions
Hong Je-Gal and Hyun-Suk Lee
Scheduling problems arise from repeatedly selecting one item from a set of candidates based on their states. These problems often reduce to assigning priority scores and choosing the highest-ranked item. In this work, we propose a factorized scheduling principle (FSP) framework to learn interpretable and transferable scheduling rules. The FSP framework represents system states as condition distributions and decomposes a global scheduling principle into additive univariate and pairwise components with identifiability constraints. The scheduling principle enables the framework to maintain a simple priority-based structure during deployment. This principle is learned by using a policy-based objective combined with a temporal-difference signal defined on the condition distribution. Experiments on synthetic and realistic scheduling tasks demonstrate the FSP framework's strong performance, interpretability, and zero-shot generalization across different system scales.
Read the full abstract ▾
Machine Learning
From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning
Yupeng Chang, Wenxuan Zhang and Yuan Wu
Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fixed-budget comparisons do not by themselves distinguish three empirical claims: whether more validation data improve checkpoint selection, whether a selection rule outperforms validation-loss selection, and whether it improves over simply retaining the final checkpoint. We therefore treat checkpoint selection as a finite-information decision problem. Holding completed training trajectories, candidate checkpoints, and independent test items fixed, we vary the validation budget and separately measure improvement from additional validation data, gain over negative log-likelihood (NLL) selection, and gain over the final checkpoint. Across 60 mathematical SFT trajectories and 19 configurations, increasing the validation budget from 32 to 305-313 examples raises independent-test accuracy by 0.32 percentage points (pp) for generated-accuracy selection and 0.29 pp for checkpoint agreement, with 95% configuration-bootstrap CIs of [0.10, 0.56] and [0.11, 0.50], respectively. At the full validation budget, the two generation-based rules outperform matched NLL selection by 0.71 and 0.85 pp, respectively, while their gains over the final checkpoint remain unresolved. A cross-domain replication on 12 newly trained Commonsense trajectories shows the same qualitative separation: increasing the validation budget from 32 to 1,024 questions improves generated-accuracy and checkpoint-agreement selection by 0.87 and 0.27 pp, while gains over the final checkpoint again remain unresolved. Together, these results show that benefiting from more validation data, outperforming NLL selection, and outperforming the final checkpoint are distinct empirical claims that require separate evidence.
Read the full abstract ▾
Machine Learning
Sharp Convergence and Sampling Trade-offs for Riemannian Diffusion under Nonnegative Ricci Curvature
Yuhao Liu and Longbo Huang
Diffusion models have emerged as state-of-the-art generative models, with recent extensions from Euclidean spaces to Riemannian
manifolds. However, existing convergence guarantees for Riemannian
diffusion models typically require $\tilde{O}(\mathrm{poly}(d,T)/\epsilon^2)$ score evaluations, with potentially unfavorable dependence on the dimension. In this work, we develop a general framework that separates score discretization from Brownian-motion simulation and allows multiple geodesic random-walk steps per score evaluation. Under nonnegative Ricci
curvature assumption and an exact Brownian-motion simulation oracle, we show that $\tilde{O}(d/\epsilon^2)$ score evaluations suffice to achieve an $\epsilon^2$ KL divergence from the target distribution, matching the existing convergence rate of Euclidean
diffusion models. We further show that $\tilde{O}(d^4T/\epsilon^2)$ geodesic random-walk steps suffice to approximate the required drifted Brownian motion to $\epsilon$ total variation error. Combining these results yields a sampling scheme with $\tilde{O}(d/\epsilon^2)$ score evaluations and $\tilde{O}(d^4T/\epsilon^2)$ geodesic random-walk steps, motivating multiple random-walk steps between consecutive score evaluations. Our results provide a sharper characterization of the convergence and sampling complexity of Riemannian
diffusion models.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models
Ruochen Zhang, Yao Huang, Yitong Sun et al. (+5 more)
While
Multimodal Large
Language Models (MLLMs) are increasingly deployed in safety-critical domains, their reliability is threatened by
multimodal implicit risks. Unlike explicit threats, these hazards emerge when individually benign text and neutral visual entities logically converge to induce unsafe outputs. Current detection methods fail to address this because they overlook the underlying risk activation mechanisms that govern cross-modal risk activation, leading to single-modality shortcut learning and hallucinated rationalizations. To bridge this gap, we first construct TriggerBench, the first
dataset explicitly modeling risk compositionality (5,600 instances). By formally isolating Key Elements and Trigger Elements to build counterfactual contrastive pairs, TriggerBench eliminates risk residues and forces models to perform genuine logical deduction rather than superficial pattern matching, which provides a rigorous foundation for both large-scale training and fine-grained evaluation. Building on this, we propose a Step-Supervised Structured
Reasoning training framework and employ it to train ThinkingGuard, a specialized guard model. Inspired by Situation Awareness theory, we decouple implicit risk identification into progressive cognitive stages, and utilize a step-reward Monte Carlo Tree Search
algorithm to explore optimal
reasoning trajectories, which are then distilled into the model through Dual-Constraint Preference
Alignment. Extensive experiments across both standard and implicit safety
benchmarks demonstrate that ThinkingGuard achieves strong performance. Project resources are available at https://github.com/FroggyChen/ThinkingGuard.
Read the full abstract ▾
All authors Ruochen Zhang, Yao Huang, Yitong Sun, Jiahe Xie, Jin Yan, Jifan Ma, Yuanfang Guo, Xingxing Wei
Machine Learning
HiTS-CL: A Continual Learning Framework for Long-Horizon Temporal Knowledge Graph Extrapolation
Yansong Liu, Rui Liu, Yuan Zuo et al. (+6 more)
Extrapolative temporal knowledge graph
reasoning (TKGR) predicts future facts from historical snapshots. Most existing methods train once on an early prefix of the timeline and then use a frozen model for all future timestamps. We argue that this fixed-prefix protocol is misaligned with extrapolation. It learns from a static prefix, whereas the target stream is non-stationary: new entities and facts emerge, temporal dependencies shift across regimes, and recurring historical signals must be refreshed online. As a result, models trained only on early snapshots become outdated and degrade over long horizons. We address this mismatch by formulating extrapolative TKGR as continual learning over streaming snapshots. Under this view, effective extrapolation must jointly handle current dynamics, stable knowledge, and recurring historical evidence. Based on these requirements, we propose History-enhanced Two-Step Continual Learning (HiTS-CL), a backbone-agnostic continual learning framework for extrapolative TKGR. HiTS-CL tracks current dynamics via continual fine-tuning, preserves stable knowledge via multi-teacher adaptive
distillation, and retains recurring historical evidence via a selective
memory of recent and frequent facts. We integrate HiTS-CL into five representative TKGR backbones and evaluate it on four
benchmark datasets. HiTS-CL consistently improves extrapolation accuracy, reduces long-horizon degradation, and outperforms strong continual-learning baselines, including a recent method for temporal knowledge graphs. Source code and data are available at https://github.com/liuyansong98/HiTS-CL.
Read the full abstract ▾
All authors Yansong Liu, Rui Liu, Yuan Zuo, Hongwei Zhao, Da Fu, Fuwei Zhang, Fuzhen Zhuang, Yong Chen, Zhe Li
Machine Learning
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
Zihao Chen, Fanxiang Xiong, Hongran Ren et al. (+6 more)
Maximum Likelihood Reinforcement Learning (MaxRL) targets
prompt-wise log-success and has shown strong performance on
reasoning tasks. Under finite rollout budgets, however, the
estimator used by MaxRL attenuates each
prompt's likelihood gradient by a factor that depends on its success probability and rollout count. Under uniform rollout allocation, the common rollout count fails to compensate for success-dependent attenuation, leaving low-success
prompts more strongly attenuated and distorting their relative contributions to the expected aggregate gradient. We introduce SERA (Scale-Equalized Rollout Allocation), which redistributes a fixed rollout budget to approximately equalize these finite-rollout scaling factors. Building on our theoretical analysis of how finite rollouts distort
prompt-wise likelihood gradients, we formulate the allocation as a fixed-budget max–min problem, derive a waterline solution to its continuous relaxation, and introduce a multiplicity correction to remove the additional
prompt weighting induced by heterogeneous rollout counts. Experiments show stronger
alignment with exact likelihood gradients in a controlled ImageNet setting and improved multi-sample solution coverage over MaxRL on maze navigation and mathematical
reasoning under matched training rollout budgets.
Read the full abstract ▾
All authors Zihao Chen, Fanxiang Xiong, Hongran Ren, Xuefeng Bai, Zhongxiang Dai, Kehai Chen, Zhiguo Zhang, Zhiyong Wang, Yu Cheng
Machine Learning
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Shutong Wu, Xiwen Chen, Brendan Rappazzo et al. (+4 more)
On-policy
distillation (OPD) trains a student model on its self-generated trajectories with dense
token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's
reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy
Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each
token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant
distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math
reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8
benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training
dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.
Read the full abstract ▾
All authors Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
Machine Learning · Cross-listed from cs.RO
Reactive Real-Time Flow Policies via Asynchronous Distribution Alignment
Moritz Zoellner, Reece O'Mahoney, Ioannis Havoutis et al. (+1 more)
Generalist
robot policies such as
vision-
language-action models (VLAs) have achieved remarkable generalization, but their inference delays can conflict with the demands of real-time control. Asynchronous execution avoids pauses between action chunks by predicting the next sequence of actions while the
robot carries out the previous one. In this paper, we study whether asynchronous execution produces the same action distribution as the original VLA. We find that, for non-Markovian demonstrations, asynchronous execution can produce a fundamentally different action distribution, which can limit the policy's reactivity. In our method, we seek to restore this reactivity by aligning the asynchronously produced action distribution with that of the original VLA through two complementary mechanisms. First, Recursive Flow-Field
Distillation trains the asynchronous policy using the VLA's action-generation flow. We characterize the learned distribution theoretically and show experimentally that our asynchronous policy can generate nearly the full range of actions the original VLA would produce, while existing asynchronous methods recover only a fraction of that range. Second, Propose-Resolve prepares multiple action sequences asynchronously and uses the latest observation to select among them based on a lightweight approximation of their likelihood under the VLA's action distribution. Our resulting method matches the original VLA's success on LIBERO and retains about 80% of its success on RoboMimic, about 30 percentage points more than existing asynchronous methods.
Read the full abstract ▾
All authors Moritz Zoellner, Reece O'Mahoney, Ioannis Havoutis, Rohan Paleja
Machine Learning · Cross-listed from stat.ML
Hierarchical Utility Calibration for Structured Multiclass Decisions
Futoshi Futami, Jerry Huang and Ichiro Takeuchi
In multiclass probabilistic prediction, Utility Calibration (UC), which focuses auditing on specified utilities, has recently received
attention as a way to guarantee downstream decisions while controlling computational and sample requirements. At the same time, some multiclass problems have meaningful label hierarchies that play important roles in medicine and image classification, yet how UC evaluates utility within a hierarchy remains insufficiently understood. We show that the difference between realized utility and predicted mean utility admits an exact decomposition into a sum of contributions from the internal nodes of the label tree. This decomposition shows that positive and negative contributions from different nodes can cancel, and that even when UC is small, the utility errors remaining in parts of the hierarchy need not be small. To address this problem, we propose Hierarchical Utility Calibration (HUC), which evaluates each node contribution before summation while retaining the same target utility, subgroup, and predicted-utility interval. We further provide finite-sample evaluation over all predicted-utility intervals and propose HUC-Boost, which updates only violated internal nodes, with theoretical guarantees for both.
Read the full abstract ▾
Machine Learning
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Oliver Sieberling, Bharat Runwal, David Jin et al. (+3 more)
Recurrent neural networks (RNNs) compress the historical context into a
memory state of fixed size, thus allowing for constant-time inference. The
memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear
attention, which extends the vector-valued hidden states of ordinary RNNs to
matrix-valued hidden states. Crucially, linear
attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the
matrix-valued hidden state. We generalize this construction and propose triadic linear
attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order (i.e., 3D) tensor state, and reads from it by contracting both key axes with two queries. An $E$-dimensional second key thus yields an $E$-fold increase in state size while adding only two projections. Triadic linear
attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear
attention, triadic linear
attention substantially improves long-context
language modeling and recall, outperforming alternatives that enlarge the state.
Read the full abstract ▾
All authors Oliver Sieberling, Bharat Runwal, David Jin, Ryan Chin, Rameswar Panda, Yoon Kim
Machine Learning
SCOPE: Observation-Conditioned Full-Target Prediction for Sparse PDE Inference
Ruichen Xu, Siyao Wang, Fang Wan et al. (+8 more)
Recovering complete physical fields from sparse observations is challenging because the measurements may not uniquely determine the underlying state.
Diffusion-based PDE solvers address this problem through iterative sampling whereas neural operators provide deterministic one-pass predictions. We propose SCOPE (Sparse-Context Observability-aware Predictive
Embeddings) to recover complete PDE fields from sparse observations by coupling full-field latent prediction with physical reconstruction. A shared decoder reconstructs fields from both predicted and complete-view representations so that representation learning is guided by both physical recovery and latent matching. We derive a quadratic risk decomposition at fixed teacher-decoder pairs showing why optimal latent prediction need not yield optimal field reconstruction. We also establish sufficient conditions for decoder improvements on complete inputs to transfer to recovery from partial observations. Experiments across five PDE settings show that SCOPE outperforms mask-aware neural operators on all ten forward and inverse tasks and achieves lower errors than those reported for
diffusion-based solvers including DiffusionPDE and FunDPS. Decoder-only adaptation further improves recovery without retraining the backbone while retaining deterministic single-pass inference.
Read the full abstract ▾
All authors Ruichen Xu, Siyao Wang, Fang Wan, Jiacheng Qiu, Wenhan Gao, Jiaxing Zhang, Linsey Pang, Ravid Shwartz-Ziv, Prakhar Mehrotra, Yann LeCun, Yuefan Deng
Machine Learning
Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations
Guanghui Min, Liang Wu, Mingjia Shi et al. (+4 more)
Long-horizon
agents require context compression to manage growing interaction histories. Compression quality, however, is ultimately determined by downstream execution. Existing
prompt-adaptation methods infer compression errors by comparing full-context and compressed trajectories. Such comparisons cannot isolate individual compressions and are confounded by
agent stochasticity. We first find that compression degrades reliability before solvability. Using matched counterfactual continuations that compare execution from the same
agent state with versus without compression, we further show that severe degradation concentrates at isolated compression events. Motivated by this finding, we propose PAIR (
Prompt Adaptation using Interventional Rollouts) for adapting structured compression
prompts. PAIR identifies individual compressions that degrade subsequent execution, diagnoses their effects, and revises the relevant sections of a fixed compression template. PAIR achieves the strongest cross-run reliability among compressed methods in every main
benchmark-scope combination, consistently exceeding the competing
prompt-adaptation baseline. Without modifying the downstream
agent, PAIR brings compressed execution close to the no-compression baseline and sometimes numerically exceeds it.
Read the full abstract ▾
All authors Guanghui Min, Liang Wu, Mingjia Shi, Yinhan He, Mayank Darbari, Liangjie Hong, Chen Chen
Machine Learning
PDE-OBS: Controlled Evaluation Across Observation Patterns
Ruichen Xu, Siyao Wang, Fang Wan et al. (+7 more)
Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern does not characterize performance when that pattern changes. We introduce PDE-OBS, an integrated benchmarking platform spanning numerical data generation, model training, and inference and evaluation under varying observation conditions. It combines 560,000 fields and trajectories from seven partial differential equation families with configurable observation operators and seven adapted baseline methods for stationary reconstruction and short-horizon forecasting. Separating observation construction from physical records allows users to specify parameterized patterns and deterministic mixtures for training and testing while preserving prediction targets and data splits. The evaluation protocol uses references trained for each test pattern to compare models on identical test observations and targets, alongside equal-count groups for spatial-layout comparisons. On a 14,000-record subset, we evaluate 441 trained models under nine test patterns, yielding 3,969 evaluations. Mean cross-pattern error exceeds mean matched-pattern error in all 49 PDE-method pairs, and this finding persists in a configuration-matched subset of 117 models. Denser test observations do not consistently reduce error for a fixed model. Mixed-pattern training on five completed pairs reduces large single-pattern transfer errors, although destination-trained references usually remain more accurate. Together, the
benchmark and findings support systematic evaluation of observation-pattern sensitivity and provide a reusable workflow for developing methods under changing measurement conditions. Code: https://github.com/ru1ch3n/PDE-OBS.
Read the full abstract ▾
All authors Ruichen Xu, Siyao Wang, Fang Wan, Jiacheng Qiu, Wenhan Gao, Jiaxing Zhang, Linsey Pang, Ravid Shwartz-Ziv, Yann LeCun, Yuefan Deng
Machine Learning · Cross-listed from cs.AI
BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning
Quan Xiao, Mingda Liu, Gaowen Liu et al. (+2 more)
Agentic reinforcement learning (ARL) with verifiable rewards improves the ability of large
language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and
reasoning. However, most existing ARL methods optimize only LLM-generated
tokens and treat retrieved evidence as environment observations. This creates an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than to the retriever, which motivates training the LLM and the retriever jointly. In this paper, we show that
retrieval and LLM policy learning are order-sensitive: adapting the retriever before optimizing the policy yields a larger reward gain than the reverse order. To preserve this hierarchy while allowing both components to co-adapt, we formulate
retrieval-augmented agentic RL as a bilevel optimization problem. To solve it efficiently, we introduce BRIDGE, a
memory-efficient first-order bilevel method motivated by a loss-landscape analysis of the RL and
retrieval objectives. Across seven open-domain QA
benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving the multi-hop average over the strongest baseline by 9.6 and 3.4 EM points, respectively. It also achieves the best averaged answer accuracy and
reasoning quality across medical QA
benchmarks.
Read the full abstract ▾
All authors Quan Xiao, Mingda Liu, Gaowen Liu, Katsuki Fujisawa, Tianyi Chen
Machine Learning
Efficient and Scalable Physics-Guided Fully Convolutional Spatiotemporal Learning for 3D Microstructure Evolution Prediction
Michael Trimboli, Wenxi Liu and Xianqi Li
Accurate prediction of three-dimensional (3D) microstructure
evolution remains computationally demanding because high-fidelity phase-field simulations require repeated numerical integration over large volumetric domains and long temporal horizons. This study develops an efficient and scalable physics-guided fully convolutional spatiotemporal framework for direct multi-frame prediction of complete 3D microstructure sequences. The model combines shared 3D spatial encoding and decoding with a factorized latent translator that integrates temporal, local 3D spatial, and channel interactions. A discrete Cahn–Hilliard (CH) residual is incorporated during training to regularize the learned
evolution toward the governing dynamics without altering the inference pathway. The framework is evaluated on high-resolution 3D spinodal-decomposition trajectories under nominal, long-horizon, and reduced-temporal-context forecasting. Under full temporal context, the model accurately reproduces volumetric
evolution, with average 3D structural similarity remaining above 0.97 over the nominal prediction horizon. Physics guidance becomes increasingly beneficial as temporal information is reduced, improving predictive robustness and preservation of interface-level morphology. The framework also achieves more than a 30-fold wall-clock speedup relative to the reference spectral phase-field solver, while physics guidance introduces no additional inference cost. These results establish direct multi-frame, physics-guided fully convolutional learning as a high-throughput surrogate strategy for dense 3D phase-field dynamics and repeated microstructure forecasting.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CV
Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics
Yicong Li, Junjie Wang, Leander Lauenburg et al. (+5 more)
We benchmarked
vision-
language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse detection (presence and polarity) and proofreading (split errors and merge errors). For synapse detection, we evaluated 19 open and 2 closed models across various architectures and sizes under zero-shot, four-shot in-context learning and LoRA settings, against specialist models, on
datasets constructed by us using public resources. For proofreading, we evaluated 3 open and 2 closed models on the ConnectomeBench2
dataset, with cross-species transfer from fly and mouse to human and zebrafish. Most models were at chance zero-shot; a few examples helped mainly the closed and largest open ones. LoRA on a few thousand labels brought open models level with specialist models. When evaluated on unseen species, the best adapted VLMs outperformed specialist models trained on the same data in identifying merge errors. The project will be publicly available upon acceptance.
Read the full abstract ▾
All authors Yicong Li, Junjie Wang, Leander Lauenburg, Ella Hugie, Alexandra Irger, Wanhua Li, Donglai Wei, Hanspeter Pfister
Machine Learning
LLMs Learn to Evade Latent Monitors from Prior Feedback Alone
Hugo Lyons Keenan, Christopher Leckie and Sarah Erfani
Latent space monitors aim to detect undesired behaviors in LLM
agents by inspecting an
agent's internal activations rather than its outputs. However, interactive monitoring creates a feedback channel where each verdict the monitor delivers leaks information to the model about how its internal states are being evaluated. We ask whether an
agent can infer the monitor's decision rule from this feedback and then selectively edit its activations to evade detection. Unlike prior evasion attacks, the model is never explicitly told what the monitor detects. Surprisingly, off-the-shelf models already produce activation edits aligned with the monitored direction, but at insufficient magnitude for evasion. Simply scaling up these edits by a factor of 8 reduces the monitor's TPR from 100% to 27%. A rank-1 LoRA amplifies this behavior into effective evasion within the forward pass, reducing TPR further to 4% on held-out concept monitors while leaving other concepts at their normal detection rates. Capabilities on standard
benchmarks are retained under this finetuning, and the evasion skill survives retraining the monitors on the new activations. Mechanistically, we find evidence that the model computes its activation edit from the prior in-context turns, and show that the edit becomes more aligned with the monitored direction as more examples are provided. These results demonstrate feedback-conditioned control over activations and suggest that latent monitoring should be treated as an interactive process in which
agents can observe and respond to oversight measures.
Read the full abstract ▾
Machine Learning
AdaptArena: Evaluating Test-Time Personalization of Web Agents
Dongchan Shin, Xing Han Lù, Jiaqi Deng et al. (+7 more)
Large
language model (LLM)
agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring
agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing
benchmarks. To address this gap, we introduce AdaptArena, a
benchmark for evaluating test-time personalization of web
agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a
retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle
agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM
agents using our framework reach at most 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when
agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web
agents. We release our code: https://github.com/McGill-NLP/web-
agents-test-time-adaptations
Read the full abstract ▾
All authors Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne, Jaewon Moon, Fengyuan Liu, Alexandre Drouin, Siva Reddy, Alexandre Lacoste
Machine Learning
Optimal Multi-Reward Reinforcement Learning
Zijun Chen and Zihan Zhang
We study an unknown-transition finite-horizon Markov decision process (MDP) with a finite collection of known reward functions $\{r^1, r^2, \ldots, r^M\}$. The goal is to output an $\epsilon$-optimal policy for every reward using online episodic interaction only. Performance is measured by the policy error $V_{0}^{*, m} - V_{0}^{\widehat\pi^{m}, m}$ where $m\in [M]$ represents the reward function and $V_{0}^{*, m}=\mathbb{E}_{s_1\sim \mu}[V_{1}^{*, m}(s_1)]$. Under this setting, we design a provably efficient
algorithm to establish a minimax sample complexity bound of $$ O\left(\frac{SAH^3}{\epsilon^2}\log M \mathrm{polylog}\left(\frac{SAH\log M}{\min\left\{\epsilon, 1\right\}\delta}\right)\right)$$ episodes, with no additional burn-in cost. This matches the information-theoretic lower bound up to a factor of $ \mathrm{polylog}(SAH\log M/(\min\left\{\epsilon, 1\right\}\delta))$. Our method combines three technical ingredients. First, we adapt MVP to reward-switching learning to construct optimistic value estimates. Second, we use fresh replay samples to conservatively evaluate the candidate policies. Third, gap-based multiplicative weights updates adjust the reward-sampling distribution using the differences between these estimates, converting weighted learning progress into simultaneous guarantees for all rewards.
Read the full abstract ▾
Machine Learning
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
Hao Li, MeiJia Chen, Weijie Ren et al. (+4 more)
On-policy
distillation (OPD) trains a student to match the teacher's next-
token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The
language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-
token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and
token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: https://github.com/xixixixixxxx/RIDE.
Read the full abstract ▾
All authors Hao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang
Machine Learning · Cross-listed from quant-ph
Quantum Computing for Network Security Classification: Near-Term Classification and Long-Term Memory Efficiency
Yuqing Li, Poonam Bala Nehru, Yunpeng Zhang et al. (+4 more)
Quantum computing has already been explored in several network-security applications. However, how
quantum computing may contribute to network-security classification in both the near term and the longer term has not been systematically discussed. This paper studies this question through two complementary experiments. First, we evaluate near-term
quantum-kernel support vector machines (SVMs) on practical network-security classification tasks and compare them with classical SVM baselines on KDD Cup 1999, CICIDS2017, and BoT-IoT. Across these runs,
quantum kernels are competitive. They can match or improve classical baselines in some settings, while classical RBF kernels remain stronger in others. This suggests that near-term
quantum-kernel methods should be evaluated as practical,
dataset-dependent alternatives to classical kernels rather than as uniformly superior replacements. Second, we use
quantum oracle sketching (QOS) to study a longer-term
memory advantage for classification with streaming classical samples. In QOS, samples are processed online and used to incrementally construct an approximate
quantum oracle, which provides coherent query access for downstream
quantum algorithms without retaining the entire
dataset. Under the QOS-inspired machine-size estimate, comparable accuracy corresponds to a substantially smaller effective
memory-size proxy than explicit sparse/QRAM-style storage. Compared with a simple streaming proxy, the result is more nuanced because aggressive feature filtering can make the streaming dimension small. This suggests that the long-term value of
quantum computing for network-security classification may lie in
memory-efficient data access rather than immediate runtime speedup. Together, these experiments show how
quantum computing may contribute to network-security classification from near-term classification performance and longer-term
memory efficiency.
Read the full abstract ▾
All authors Yuqing Li, Poonam Bala Nehru, Yunpeng Zhang, Danindu Gammanpilage, Xin Jin, Zeguan Wu, Junyu Liu
Machine Learning
Guard Models Are Overconfident Where Base Models Are Uncertain
Jonghyun Hong, MinJae Jung and Minwoo Kim
Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for
prompt classification and find that although several are nearly calibrated on clean inputs,
adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while
adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.
Read the full abstract ▾
Machine Learning · Cross-listed from cs.CL
FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance
Rikhiya Ghosh, Himanshu Kumar, Sriram Venkatapathy et al. (+3 more)
In regulated industries like consumer finance, seemingly harmless user queries can exploit large
language model vulnerabilities, triggering safety failures and pushing responses dangerously close to policy limits. Existing automated red-teaming methods trade off attack effectiveness against generation cost, while treating coverage, severity, and diversity as incidental rather than joint objectives. We introduce FinRT, a structured framework that builds reusable
adversarial prompt generators from adaptive red-teaming strategies. Across the six victim models in consumer finance, FinRT substantially outperforms adaptive search baselines while amortizing target-facing attack generation into a reusable generator. FinRT nearly doubles the attack success rate over the adaptive baseline Rainbow Teaming (32.9% vs. 17.2%), increases maximum
adversarial severity by 33%, and preserves comparable intra-policy-domain semantic diversity to iterative search methods. Our method achieves high cross-model transferability while exhibiting distinct victim-family specialization patterns.
Read the full abstract ▾
All authors Rikhiya Ghosh, Himanshu Kumar, Sriram Venkatapathy, Sahil Wadhwa, Alexandre G.R. Day, Pranab Mohanty
Machine Learning · Cross-listed from cs.AI
Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery
Akhil Bagaria, Anita De Mello Koch and George Konidaris
Temporal abstraction via options can improve exploration in large environments. However, existing option discovery
algorithms find subgoals that target all aspects of the state simultaneously. This state-reaching approach produces options that only apply in narrow regions of the state-space, eventually causing an explosion in the number of options that overwhelms the
agent, and impedes progress on its primary task of reward maximization. We introduce an
algorithm that instead identifies a small, relevant subset of features for each subgoal, yielding options that generalize broadly and accelerate exploration. Our approach learns abstract, transferrable options and achieves rapid exploration in three sparse-reward, image-based domains, including the Atari game MontezumasRevenge.
Read the full abstract ▾