Computer Vision
DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding
Boyuan Sun, Deshui Miao, Wenzhao Gao et al. (+8 more)
We propose DeltaStreamer, a novel agentic framework for streaming
video understanding inspired by how humans observe, remember, and investigate. Streaming
video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing
video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware
memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured
memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides
memory retrieval and targeted replay. By revisiting historical frames, the
agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming
video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/
Read the full abstract ▾
All authors Boyuan Sun, Deshui Miao, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Chi Liu, Yan Shu, Derek Li, Bryan Dai, Shengsheng Qian, Qibin Hou
Computation and Language
OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score
Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang et al. (+3 more)
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large
language models, built around two
benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system
prompt moved one model from last (58.2%) to first (83.9%) on the same
benchmark; doubling the thinking budget from 8k to 16k
tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all
datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai
Read the full abstract ▾
All authors Liang-Hsun Huang, Min-Yi Chen, Chao-Chun Chuang, Yu-Wei Liu, Che-Yu Wu, Po-Feng Lee
Computer Vision
VisibilitySplat: Ray-Traced Occlusion for Relighting with 3D Gaussian Splatting
Sven Kluge and Oliver Staadt
We address inverse rendering with 3D Gaussian Splatting (3DGS) by introducing explicit visibility modeling for relighting as well as material and light decomposition. Our approach augments the standard rasterized 3DGS pipeline with a sparse visibility oracle that estimates transmittance-weighted occlusion for Gaussian primitives from sampled lighting directions. This oracle is implemented using a point-based ray tracer accelerated by a bounding volume hierarchy (BVH) and queried sparsely within the inverse-rendering objectives, while the original 3DGS loop is retained for training efficiency, interactive visualization, and tooling compatibility. Shading follows a physically based rendering (PBR) formulation with a principled microfacet bidirectional reflectance distribution function (BRDF) and low-resolution high-dynamic-range (HDR) environment maps. We evaluate our method on the TensoIR synthetic
dataset against baselines based on neural radiance fields (NeRFs), tensor representations, and 3DGS. The results show competitive performance and improvements over several earlier baselines for novel-view synthesis (NVS), albedo reconstruction, and relighting. Ablations indicate that explicit visibility modeling is an important contributor to relighting and material-illumination disentanglement, while pure PBR supervision can favor appearance fitting in NVS. Overall, the findings suggest that sparse, point-based visibility queries effectively complement rasterized splatting for inverse rendering while preserving the practical advantages of the 3DGS ecosystem.
Read the full abstract ▾
Machine Learning
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Sifan Yang, Hongyi Tao, Dingzhi Yu et al. (+5 more)
Memory-efficient subspace optimization has emerged as a practical approach to reducing the
memory cost of LLM training while preserving optimization performance. Previous
algorithms typically rely on the leading singular subspace of
stochastic gradients, yet the gradient noise may dominate the subspace in later training stages and can prevent convergence. To overcome this limitation, we adopt a residual correction framework, which maintains the optimizer states in the subspace and adds the orthogonal residual gradient to the parameter update. We first integrate this scheme into momentum SGD to develop SMRC, which attains an $O(T^{-1/4})$ convergence rate for a fixed orthonormal basis, matching that of full-space momentum SGD. Our analysis introduces a subspace virtual iterate to resolve the coupling between the momentum and the residual correction. Furthermore, we extend SMRC to a multi-stage variant which periodically recomputes bases to track the evolving dominant subspace and retains the same rate. Since each basis refresh breaks the consistency of the virtual iterates, we devise a boundary correction approach that aligns them across stages. Beyond momentum SGD, we incorporate our framework into
Muon and propose PMRC, which applies the
matrix sign to the projected momentum. Through a weighted momentum tracking analysis that accounts for the residual gradient, we establish an $O(T^{-1/4})$ convergence rate for PMRC, which matches the rate of full-space
Muon. We also design a multi-stage PMRC with periodically refreshed bases, which maintains a convergence rate of the same order. Extensive experimental results confirm the effectiveness and efficiency of our
algorithms.
Read the full abstract ▾
All authors Sifan Yang, Hongyi Tao, Dingzhi Yu, Haomin Bai, Wei Jiang, Silin Gao, Pi Bu, Lijun Zhang
Artificial Intelligence
LAGO: Language Agents with Reusable Growing Options
Yuran Wu, Mengkang Hu, Qionglin Qiu et al. (+5 more)
Long-horizon tasks expose a weakness of monolithic tool-use loops: one
agent must decompose, execute, and verify a task within a single trajectory. We introduce LAGO, which treats specialized
language agents as reusable temporally extended options. A meta-
agent retrieves options from a persistent library or spawns new ones, dispatches sub-tasks, verifies results, and admits successful novel options. The library grows across tasks without updating model weights. On 500 SWE-bench Verified tasks, frozen Qwen3.5-35B-A3B in both roles resolves 58.6% under single-attempt evaluation with the official Docker evaluator. On a fixed 150-task subset, we re-run mini-SWE-
agent, Terminus 2, and LAGO with the same policy, temperature, task IDs, and evaluator; their resolve rates are 49.3%, 57.3%, and 60.3%, respectively. Cross-model experiments provide evidence consistent with a role for decomposition quality. A 50-task component ablation finds four fewer resolved tasks without decomposition, two fewer with shuffled reused content, and six fewer without the library. These results support reusable
agent-options as a practical abstraction for task streams, while leaving component-level causality task- and model-dependent.
Read the full abstract ▾
All authors Yuran Wu, Mengkang Hu, Qionglin Qiu, Fanqing Meng, Qiguang Chen, Yuhang Zhou, Qiudan Zhang, Xu Wang
Robotics
FOMI: A Force-Feedback Manipulation Interface for Teaching Contact-Rich Manipulation
Yishu Shen, Yiming Luo, Ruilin Wang et al. (+7 more)
Robot-free collection with handheld interfaces enables scalable acquisition of human demonstrations for policy learning. However, existing handheld interfaces often provide limited intuitive force feedback, making contact-rich demonstration collection challenging for human operators. To address this, we present FOMI, a force-feedback manipulation interface with a scissor-shaped mechanism, which transmits object interaction forces directly to the operator's hands through actuation links without force sensors. This intuitive force feedback supports real-time action adjustment during
robot-free collection of contact-rich demonstrations. Around FOMI, we build a demonstration collection system that integrates sensor calibration, data acquisition, and synchronization for
robot-free collection. By further mapping pose changes and gripper opening of FOMI to
robot end-effector poses and gripper commands, the system also supports teleoperation. Across four contact-rich tasks, FOMI improves collection efficiency by approximately 29% and 18% over the best baselines in
robot-free collection and teleoperation, respectively. Demonstrations collected with FOMI in both modes enable policies to perform challenging contact-rich manipulation tasks with high success rates.
Read the full abstract ▾
All authors Yishu Shen, Yiming Luo, Ruilin Wang, Bowen Yang, Shuai Yang, Fangyi Xu, Yujun Shen, Fu Zhang, Boyu Zhou, Yinghao Xu
Computation and Language
EGCForest: Structured Agent Memory Organization for Enterprise Group Chats
Lingxi Cui, Jinhao Ruan, Wenqing Wei et al. (+4 more)
Enterprise group chats record project progress, decisions, task ownership, and other organizational information valuable to enterprise
agents. Reliable
retrieval is challenging because evidence spans interleaved threads, channels, and time, with meaning and authority conditioned on the speaker. Neither RAG's independent chunk
retrieval nor
memory systems' summary-based consolidation can reliably recover such distributed evidence. We present EGCForest, a novel multi-view hierarchical
memory framework for enterprise group chats. EGCForest organizes an immutable message log into three complementary forests: a Channel forest for temporal workflows, an Actor forest for speaker-conditioned context, and an Artifact forest for deliverable lifecycles. Cross-view connections are enabled by a Placement Map and sparse structural bridges. Built on this
memory backbone, EGCForest retrieves and assembles evidence by jointly leveraging message-level relevance, hierarchical context from forest summaries, and cross-view links, recovering distributed evidence. Experiments on three enterprise-chat
benchmarks show that EGCForest consistently outperforms RAG and
memory-system baselines, improving answer accuracy by up to 20.85 points over MemOS while supporting efficient online
retrieval.
Read the full abstract ▾
All authors Lingxi Cui, Jinhao Ruan, Wenqing Wei, Guo Yi, Li Rao, Ke Chen, Lidan Shou
Artificial Intelligence
Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents
Md Farhan Ishmam and Kenneth Marino
Frontier web
agents using closed models have high inference and computational costs. Smaller open models can be an alternative, but to be performant, they require training supervision from a human or teacher model.
Token-level teacher
distillation is not an option for closed teachers as it needs access to the model's logits. The standard approach is to imitate teacher demonstrations offline, but it trains the student on states it has never visited, and in a style it does not write in. We introduce WebKD, where the teacher is the editor of the student's failures. Given a failed episode, the teacher finds the first point of mistake, and rewrites the student's
reasoning traces, harness states, and actions from that point onward. The editing reveals that the teacher's corrections can be more surprising to the student than the student's own text. WebKD instead aims to edit minimally. In the data space, the teacher writes each correction in the student's style. In the objective space, the
reasoning is weighted by its contribution to the action. A role-asymmetric KL anchor allows the corrected action to move freely while keeping the
reasoning near the student's prior. On the TimeWarp
benchmark, WebKD students gain about 5.5-10.7 pp over behavior cloning, outcome-reward RL, and teacher editing baselines. The performance transfers to unseen versions and to the live web in the OnlineMind2Web
benchmark. WebKD opens a new way of distilling black-box models to improve the capabilities of open-weight web
agents.
Read the full abstract ▾
Machine Learning
Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent
Zixiao Wang, Xingbo Du, Duzhen Zhang et al. (+1 more)
Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching
estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.
Read the full abstract ▾
All authors Zixiao Wang, Xingbo Du, Duzhen Zhang, Le Song
Machine Learning
CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting
Chengyi Pan, Huangqimei Zheng and Hao Xue
Although multivariate time series forecasting has broad application prospects, effectively incorporating exogenous variables remains challenging. Prevailing discrete-time architectures often impose rigid temporal
alignment across heterogeneous variables, which may limit their ability to capture continuously evolving external effects. To address this limitation, we introduce CTIA-ODE, a continuous-time forecasting framework centered on Continuous-Time Inter-Variable
Attention (CTIA). Specifically, CTIA is embedded within an ordinary differential equation (ODE) solver, enabling the continuously evolving endogenous trajectory to dynamically query a static exogenous context along the integration path without requiring explicit temporal
alignment. To improve the stability of the continuous latent dynamics, we further introduce a dual-stabilization strategy: architecturally, a Linear Residual Augmentation (LRA) module provides a macroscopic trend anchor, while an Outlier-Robust Hybrid Loss (ORHL) mitigates the influence of heavy-tailed anomalies during optimization. Extensive experiments across eight real-world
datasets demonstrate that CTIA-ODE achieves state-of-the-art forecasting performance across diverse
datasets and prediction horizons, while maintaining a favorable accuracy-efficiency trade-off with a compact
memory footprint and low inference latency, particularly under long-horizon forecasting settings. Anonymous code is available at https://anonymous.4open.science/r/CTIA-ODE-15A6/.
Read the full abstract ▾
Cryptography and Security
DarKnight: Retrieval-Guided Agentic Framework for Breaking Dark Web CAPTCHA Gates
Binghui Wu, Yeo Zi Yi, Darren Tan et al. (+3 more)
Cyber Threat Intelligence (CTI) investigators who monitor dark web marketplaces for leaked credentials and breach listings must pass the diverse CAPTCHAs that operators deploy to block automated collection. Existing solvers either need unscalable per-site fine-tuning or rely on commercial
Vision-
Language Model (VLM) APIs, which surrender operational sovereignty: provider policies, Tor-exit blocking, government regulation, or model updates outside the investigator's control can halt or alter the solver overnight, and per-call costs recur with every session and retry. We present DarKnight, a self-improving agentic framework designed to run on a single workstation GPU inside an investigator's Tor crawler and solve CAPTCHAs with small, locally hosted open-weight VLMs that no provider decision can withdraw, without per-
market training or third-party services. For each CAPTCHA, DarKnight retrieves successful and failed past attempts on similar images (dual-signal RAG), composes an image-specific preprocessing pipeline, and learns from the site's pass/fail verdict. In a case study on four real-world dark web marketplaces, agentic tool selection outperformed parameter scaling. A deployable open-weight VLM with DarKnight reached 83.2% on the most heavily defended
market, 37.2 percentage points above its base model, outperforming even the frontier Claude Opus~5 (44%). DarKnight recovered 48.2% of baseline failures at a 7.6% degeneration cost, indicating that selective tool use can avoid much of the degradation caused by uniform preprocessing.
Read the full abstract ▾
All authors Binghui Wu, Yeo Zi Yi, Darren Tan, Manit Baser, Levente Csikor, Dinil Mon Divakaran
Machine Learning
MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation
Minsuh Joo, Youna Kim, Choonghyun Park et al. (+2 more)
LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as activation steering, decoding adjustments, and prompting modify inference, potentially reinforcing existing predictions or surfacing less accessible answers. Yet their evaluations centered on aggregate accuracy, which obscures diverse effects by collapsing recovered failures and disrupted successes into a single score. We introduce MAPLE, a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics disentangle these effects at the surface, probability, and beyond-sampling levels by measuring answer recovery and disruption, gold-
token rank shifts on unrecovered failures, and recovery of answers missed by fixed-budget sampling. We evaluate six KEE techniques spanning decoding, activation steering, and prompting across three model families: OLMo2, Qwen2.5, and Llama3. Our analysis reveals distinct effects and trade-offs across techniques, highlighting the value of MAPLE for understanding and comparing KEE beyond aggregate accuracy.
Read the full abstract ▾
All authors Minsuh Joo, Youna Kim, Choonghyun Park, Soo Kyung Kim, Hyunsoo Cho
Robotics
What Matters for World Modeling in Vision-Language-Action Models?
Qisen Ma, Yixiang Chen, Yuan Xu et al. (+10 more)
Integrating world models into
Vision-
Language-Action models (VLAs) has emerged as a promising direction, enabling policies to anticipate future dynamics beyond reactive observation-to-action mapping. However, existing approaches vary simultaneously in their world representations, integration paradigms, and policy backbones, making it difficult to determine what truly matters for effective world model integration. To disentangle these factors, we categorize existing efforts into four representative paradigms, namely Implicit Future
Alignment, Explicit Future Query, MoT Future Expert, and External World Predictor. We instantiate each within a unified framework and systematically evaluate these paradigms with diverse world representations and policy backbones across in-domain, out-of-domain, long-horizon, dynamic, few-shot, and real-world settings. Our study yields four main findings: 1) world representation effectiveness is paradigm-dependent, whereas 2D semantic representations offer robust, cross-paradigm gains; 2) no single paradigm is universally optimal, yet the decoupled MoT Future Expert offers competitive overall performance, particularly in OOD robustness; 3) world modeling consistently benefits policies built on standard VLMs, while decoupled designs exhibit greater compatibility with pretrained VLAs; and 4) world modeling improves performance over action-only baselines across multiple settings, with notable gains on long-horizon and dynamic tasks. Project page and
video demonstrations are available at https://dylanmqs.github.io/WM-in-VLAs/.
Read the full abstract ▾
All authors Qisen Ma, Yixiang Chen, Yuan Xu, Peiyan Li, Jiabing Yang, Kai Wang, Jianhua Yang, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
Systems and Control
Resilient Angle-Constrained Formation Control and Localization Under DoS Attacks (Extended Version)
Nhat-Minh Le-Phan, Ziliang Lyu and Lihua Xie
This paper studies resilient angle-constrained leader–follower formation control and localization of multi-
agent systems under asynchronous denial-of-service (DoS) attacks, which block individual sensing and communication links. To preserve the well-posedness of angle-based localization, an all-or-nothing triangle integrity protocol (ANTIP) is proposed to convert edge-wise attacks into triangle-wise ones, and DoS attacks are classified into localizability-maintained and localizability-broken attacks. First, with relative position measurements over attack-free links, resilient formation tracking laws are designed for leaders with constant velocities and with time-varying velocities comprising translational, rotational, and scaling motions. These laws guarantee asymptotic tracking and finite-time recovery of tracking, respectively, when the duration of localizability-broken attacks is suitably bounded. Second, with only angle measurements and communications, a resilient simultaneous localization and formation control (RSLAF) scheme is developed. It combines a finite-time distributed observer, which estimates the exosystem
matrices and states under asynchronous DoS attacks, with a perturbation-based localization law that prevents localization from getting stuck at degenerate configurations. The theoretical results are rigorously proved and illustrated by numerical simulations.
Read the full abstract ▾
Computer Vision
PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA
Siwei Meng, Xuerou Cai, Rui Hu et al. (+1 more)
Text-to-
video (T2V) generation models can produce visually realistic
videos that nevertheless violate the physical consequences implied by the
prompt and scene. Existing
benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic
benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a
vision-
language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human
alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a
benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection. Our project page is available at: https://mengsiwei.github.io/PhyMetric/.
Read the full abstract ▾
All authors Siwei Meng, Xuerou Cai, Rui Hu, Ping Liu
Distributed Computing
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
Daijin Hu
LLM
agents are evolving from a single model calling tools to planners that orchestrate multiple expert models whose capabilities live in distinct weights. This shift breaks a hidden assumption of today's serving systems: that the working set of models fits in GPU
memory. On the
memory-constrained GPUs typical of private and edge deployments, naive whole-model swapping serializes weight transfer with computation and, without host-backed KV restoration, discards session state at a switch. We present JointServe, a serving system that treats weight shards and KV blocks as residency objects competing for one unified
memory budget. JointServe exploits three structural properties of
agent workloads (directed calls, gapped execution, and dual-state residency) to prepare models before they are called: an intent signal parsed from the caller's own output stream triggers prefetch during generation, partial k* residency reduces switching to incremental layers, and a preemptive transfer scheduler protects live-session KV with a guaranteed bandwidth share. A carve-out control step adjusts the weight/KV allocation as workload demands change. Evaluations show that JointServe reduces session completion time by 21.7% over the strongest whole-model swap baseline and stays usable in a crash regime where every whole-model swap baseline fails to finish (a 46-fold gap). On replayed SWE-bench, tau-bench, and OSWorld traces, it cuts session completion time by up to 49% and lifts SLO attainment from 2.8-24.5% to 76.8-100%.
Read the full abstract ▾
Artificial Intelligence
ActionOPD: Distilling Decisions, Not Tokens
Huining Yuan, Xiangmin Yi, Zelai Xu et al. (+5 more)
Distilling capable
language-model
agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this
alignment be performed? Offline imitation provides dense but
token-specific supervision and can suffer from state-distribution mismatch.
Token-level on-policy
distillation (OPD) addresses this mismatch but typically requires teacher
token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for
agent distillation. We introduce ActionOPD, an action-level on-policy
distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written
reasoning or access to teacher
token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution
benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On $\tau^2$-bench, ActionOPD achieves competitive performance against offline
distillation and
token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to
agent distillation.
Read the full abstract ▾
All authors Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen, Xiao-Ping Zhang, Yu Wang, Chao Yu
Distributed Computing
The Cost of Context: Profiling the Energy Footprint of Input Tokens in Large Language Models
Boris Ruf and Marcin Detyniecki
The rapid adoption of
Retrieval-Augmented Generation (RAG) and long-context LLMs has shifted the computational burden toward the prefill phase—the initial stage where the model processes all input
tokens in parallel before generating a response—yet its energy impact remains poorly characterised. We investigate the contradiction between massive hardware-level power spikes and the sub-linear marginal cost of input
tokens reported in system-level studies. Through high-resolution telemetry of Llama-3.1 and Qwen-2 models on NVIDIA A100 hardware, we identify the efficiency threshold where $O(n^2)$
attention complexity overcomes GPU parallelisation. Our results quantify how optimised
memory orchestration (PagedAttention) delays this scaling bottleneck across three orders of context scaling, yielding up to an 8.9$\times$ energy efficiency advantage at 112,000
tokens compared to standard baselines. Furthermore, we characterise the energy disparity between input and output
tokens, demonstrating that optimised serving (vLLM) causes this ratio to narrow (from $\approx$850$\times$ to 750$\times$) as context scales due to high decode efficiency, whereas unoptimised baselines show a much smaller, roughly constant gap ($\approx$150–230$\times$) because their prefill is itself expensive. This work establishes context-scaling laws that differentiate engine-level optimisations from inherent model constraints, providing an empirical framework for sustainable LLM deployment.
Read the full abstract ▾
Artificial Intelligence
One Monitor Does Not Fit All: Formalism-Agnostic Runtime Verification for Agentic AI
Angelo Ferrando
Agentic AI systems combine model decisions, mutable state, tool calls, and delegation to external
agents. Monitoring them requires observations from different execution boundaries, while prevention requires control before an operation produces its effect. We present AgenticRV, a runtime-verification architecture that separates these boundaries from the monitoring formalism. Adapters emit canonical events; each monitor receives its own projection and abstraction; and a decision policy uses normalised verdicts to select a reaction supported by the responsible adapter. We formalise when observations suffice, monitor replacement preserves decisions, and blocking prevents unsafe effects. We also distinguish rejected attempts from completed effects, avoiding the assumption that a blocked request must leave every monitor in a permanent violation state. The Python implementation supports LangGraph, MCP, and A2A. A controlled LangGraph experiment combines an external Reelay temporal monitor with a programmatic report monitor; ablations show why both are needed for the selected requirements. Differential checks and cross-adapter tests provide further integration evidence. An LLM-driven incident-response example illustrates the same enforcement path, while existing microbenchmarks isolate framework costs. The results support reuse of monitoring methods at explicit control boundaries, subject to the stated observation and deployment assumptions.
Read the full abstract ▾
Computation and Language
Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations
Enrico Liscio, Anne Arzberger, Aditya Shankar et al. (+1 more)
Human values shape how we expect
language models to behave. Dominant
alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free
alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four
prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track
alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.
Read the full abstract ▾
All authors Enrico Liscio, Anne Arzberger, Aditya Shankar, Pradeep K. Murukannaiah
Robotics
Learning from Demonstrations meets Symbolic Control - A Supervisory Control Architecture for Formal Certification of Learned Motion
Ashok Kumar, Mahathi Anand, Angela Schoellig et al. (+1 more)
Dynamical system (DS)-based learning from demonstration generates flexible
robot motions, but existing guarantees rely on certificate generation to obtain formal guarantees of safety and goal convergence. We propose a symbolic supervisory control architecture that automatically generates a fail-safe backup plan, certifying safety, reachability, and actuator feasibility directly from the
robot's dynamics, independent of the learned DS. A nominal controller tracks the learned DS, while the supervisory controller monitors safety and target convergence, providing non-conservative intervention when the nominal controller violates necessary constraints. Experimental results include simulations and real
robot deployment with various 2D and 3D
datasets, including handwriting
datasets, validating the effectiveness of our approach in providing formal guarantees with learned motion even in complex, dynamic environments.
Read the full abstract ▾
All authors Ashok Kumar, Mahathi Anand, Angela Schoellig, Abdalla Swikir
Machine Learning
When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models
Wenwen Gong, Haotian Gu, Zhen Yang et al. (+6 more)
Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while processing diverse inputs through a shared stack of layers. Recent layerwise analyses identify strong predictive performance in early TFM representations and substantial overlap in computation across layers. However, early predictive accessibility alone does not establish whether later layers are dispensable for every input, and aggregate accuracy can obscure which inputs benefit from additional computation. We study all 24 in-context-learning layers of TabPFN-3 across 272 classification
datasets, combining representation geometry, prediction trajectories, truncation, and structural interventions, while also examining related representation dynamics in four additional TFMs. We uncover a pronounced expansion-compression trajectory, with stronger late dimensional contraction associated with larger increases in class-prototype separation. A targeted analysis of early-stabilizing samples shows that substantial representational changes persist after predicted classes stabilize, while locally ambiguous queries obtain larger accuracy gains from shallow to late layers than well-separated queries. Repeating a single pretrained block or perturbing layer order reduces predictive performance on average, constraining simple post-hoc substitutions in the frozen model. These findings distinguish predictive accessibility, class stability, and the benefits of continuing computation, supporting an input-dependent assessment of depthwise redundancy in TFMs.
Read the full abstract ▾
All authors Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma, Hanqing Zhang, Zhongshen Wang, Jingjun Wang, Shouyi Xu, Wei Wang
Machine Learning
VaryST: Generalizable Spatio-Temporal Forecasting Beyond the Closed-World Setting
Shengnan Guo, Jinjun Cheng, Xiaowei Mao et al. (+7 more)
Spatio-temporal forecasting is fundamental to many real-world systems, yet deployment environments are inherently varying: temporal distributions may evolve, spatial domains may change, and observation availability may shift from training conditions. Existing models are often tightly coupled to their training environments through node-specific representations, predefined spatial support sets, and reliance on consistently available observations, limiting generalization to changing environments. To this end, we propose VaryST, a compact spatio-temporal forecasting model trained from scratch that achieves strong generalization by dynamically constructing predictive context from available long-term observations. To accommodate temporal changes, VaryST derives node representations from long-term histories, capturing current behavior in the context of longer-term patterns. To adapt to spatial changes, it dynamically selects functionally relevant support nodes beyond predefined neighborhoods and integrates their information with recent and long-term target states and global context. To handle varying observation availability, VaryST selectively masks historical patches on which the model relies most during training, encouraging it to exploit alternative temporal and spatial dependencies for accurate forecasting when key observations become unavailable. Extensive experiments on six real-world
datasets against seventeen baselines show that VaryST achieves strong in-domain forecasting accuracy, substantially improves zero- and few-shot generalization under spatial and temporal changes, and remains robust to varying observation availability.
Read the full abstract ▾
All authors Shengnan Guo, Jinjun Cheng, Xiaowei Mao, Jianhao Zhang, Haochen Lv, Yang Jiang, YiHeng Huang, Yan Lin, Youfang Lin, Huaiyu Wan
Machine Learning · Cross-listed from stat.ML
Calibration and Risk-Budgeted Decisions with System One Models
Soundouss Messoudi and Youness Mansar
LLM workflows and
agents rely on a set of binary decisions like allow/block an input, or discard/retain a retrieved document. When human deferral is unavailable, uncertain cases must still receive an action. We study System One decision scores in this setting by defaulting to a conservative fallback when uncertain, and controlling the false omission risk within the subset of inputs assigned the utility-maximizing action. Across six
datasets covering moderation, safety, and reranking, we compare general System One models with task baselines and a local open-weights fine-tuned System One model. We show that naive baselines for threshold selection exceed the target risk in roughly half of settings with defined test risk, compared with 7.8% for Learn Then Test (LTT), which leverages finite-sample statistical guarantees. Our results show why relying on a probability-like interface is insufficient for deployment decisions and how risk-budgeted selection trades useful action against observed error.
Read the full abstract ▾
Artificial Intelligence
ReproBAIT: Can coding agents build their own biological AI tools?
Shiv Muthupandiyan, Tolga H. Dur, Mac Walker et al. (+8 more)
Large
language models are increasingly competent at leveraging scientific knowledge and existing computational tools to execute complex workflows. At present, little information exists on their ability to reconstruct tools when given limited descriptions of such tools. We introduce an evaluation titled ReproBAIT (Reproducing Biological AI Tools), which measures the ability of coding
agents to design, implement, and train biological AI tools taken from published research across a range of information access conditions. The coding
agent is securely sandboxed without internet access and receives training data, a fixed time and compute budget, and limited information about the tool to replicate. We then score the reproduced biological AI tool based on its performance (tool-specific metrics) on held-out data. We evaluated a set of biological AI tools of varying complexities: ProteinMPNN, an inverse
protein folding tool, RFdiffusion3, an all-atom generative
diffusion tool, and a closed-source molecular design tool. We institute a methodology to check for memorisation, as the open source tools are likely to be included in models' pretraining corpus. Across a range of leading closed- and open-source coding
agents, we find substantial reproduction capability for some selected tools, though limited capability for a highly complex tool. We find moderate evidence consistent with memorisation, varying across coding
agents. These results suggest that tool developers need to be increasingly cautious when releasing information about the tools publicly; re-creation of the tool based on such information could undermine a tool's commercial value or invalidate dual-use managed access strategies.
Read the full abstract ▾
All authors Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, Timo Flesch, Mahmoud Ghanem, Steph Suddell, Giles Harper-Donnelly, Sophie Rose, Jasper Götting, Coleman Breen, Jake Pencharz
Cryptography and Security
Adaptive and Proximal Federated Optimization for Transferable Network Intrusion Detection
Rezwan Islam Salvi, MD Rafin Jawad Hafiz, Abeer Mahmood Kalam et al. (+1 more)
Network Intrusion Detection Systems (NIDS) based on deep learning often fail to detect attack classes that are absent from training data, limiting their robustness against zero-day threats. This property called transferability across attack classes, remains underexplored especially in
federated learning (FL) settings where data heterogeneity and client drift pose additional challenges. Prior work (TabFIDS) showed that FedAvg combined with bootstrapping and temporal averaging improves transferability for cross-device NIDS. In this paper, we explore whether other
federated optimizers can do better. We compare two optimizers inspired by FedAdam and FedProx against the FedAvg baseline on the CIC-IDS 2017
dataset using an 11-client
federated setup, sweep 10 hyperparameter configurations, and contrast two data-balancing strategies: bootstrapping and SMOTE. Using the count of client pairs whose transferred-model accuracy exceeds 70% as our primary metric (following TabFIDS), the proximal optimizer with a strong regularization term yields 48 transferable pairs versus 42 for FedAvg, while the adaptive optimizer with an aggressive server learning rate yields 46 pairs. SMOTE improves per-pair accuracy over bootstrapping but reduces breadth, revealing a depth-vs-breadth trade-off: high accuracy on a few client pairs versus moderate accuracy across most. These results indicate that the choice of
federated optimizer, the rule by which client updates are regularized and aggregated into the global model, is very important for
federated NIDS.
Read the full abstract ▾
All authors Rezwan Islam Salvi, MD Rafin Jawad Hafiz, Abeer Mahmood Kalam, Md Moniruzzaman
Numerical Analysis
Stability of the P_2 semi-Lagrangian scheme
Roberto Ferretti and Michel Mehrenberger
Among the various space interpolation operators used in high-order semi-Lagrangian schemes, the case of finite element interpolation presents a somewhat incomplete stability theory, partly due to the impossibility of applying in a standard form the framework of Von Neumann analysis, even in the constant coefficient case. In this work, the stability of the P_2 semi-Lagrangian scheme is stated in the case of the constant coefficient equation on a uniform mesh, using a suitable extension of the Von Neumann technique to treat block circulant
matrices.
Read the full abstract ▾
Machine Learning
Efficient contextual partial monitoring with general function approximation
Kapilan Balagopalan, Hao Qin, Kwang-Sung Jun et al. (+1 more)
We study finite
stochastic partial monitoring (PM) games with side information, a paradigm of sequential learning with partial feedback. On each round $t \in [T]$, both player and environment observe a context and player chooses an action and environment simultaneously chooses an outcome. The player neither observes the outcome nor the loss; instead, they only observe a feedback symbol which is partially informative about the outcome. We utilize the Estimation-to-Decisions (E2D) meta-
algorithm of Foster et al.(2021) to design E2D.PMSide, a computationally efficient framework for contextual partial monitoring, which uses an online regression oracle to efficiently estimate the outcome distributions based on historical (context, action, feedback symbol) tuples. Since the exploration rule given by the original E2D
algorithm is intractable to compute, we propose two computationally efficient exploration rules that achieve optimal regret orders in $T$: in the globally observable setting, our exploration rule is based on $\epsilon$-greedy, while in the locally observable setting, our exploration rule uses inverse gap weighting (Foster & Rakhlin, 2020), which was previously only known to be effective in the bandit setting. We show that a variant of E2D.PMSide named OE2D.PMSide can be combined with offline regression oracles to give an efficient
algorithm with optimal regret when the contexts are drawn independently from a fixed distribution. We evaluate our
algorithm on a synthetic label-efficient prediction game and an LLM routing task, in which a learner decides from
prompt, uncertainty, and internal state features whether to escalate a query from a weak model to a strong one. We show that our method has superior performance to prior approaches.
Read the full abstract ▾
All authors Kapilan Balagopalan, Hao Qin, Kwang-Sung Jun, Chicheng Zhang
Computer Vision
Improving Category-Aware Active Domain Adaptation via Uncertainty-Guided Region Coverage
My Tra Nguyen Huynh and Viet Cuong Ta
Active domain adaptation researches focus on alleviating the annotation cost, which comprises source data and target data labelling, as well as enhancing the performance of unsupervised domain adaptation. However, optimizing the adaptation model based on the average accuracy can lead to the drop in performance of some difficult categories. To overcome this issue, a point-wise selection strategy built upon the influence function can be used to improve domain adaptation performance. In our work, we first address the sensitive and redundant aspects of point-wise selection, which can lead to overfitting of the acquisition function. We thus introduce a region-based approach that ensures the improvement of the adaptation model by maximizing a positive
manifold coverage. An uncertainty score is further introduced to the selection mechanism to learn a better decision boundary for the target domain. In comprehensive experiments, our proposed method surpasses the first category-aware active domain adaptation baseline as well as existing active domain adaptation methods across multiple domain adaptation
benchmarks. Our implementation is available at: https://github.com/nhtm2702/WHO IS NEXT.
Read the full abstract ▾
Machine Learning
Asymptotically Exact Negative Guidance of Diffusion Models via Positive-Unlabeled Learning
Hirohane Takagi and Masashi Sugiyama
Negative guidance of
diffusion models generates samples that avoid an undesired condition, specified either by reference samples or by a conditional score alongside an unconditional model. Existing methods inject repulsive terms derived from local information at each sample, so the resulting sampler does not target an explicit density, and its bias cannot be systematically reduced by increasing computation. We instead present a negative-guidance method that samples from an explicit target density, which uniformly handles both reference-sample and score-based specifications and suppresses probability mass near the undesired condition with an avoidance strength that parametrizes the target density itself. Realizing this target requires estimating, at each
diffusion timestep, how likely each sample is to belong to the undesired condition. We cast this estimation as a Positive-Unlabeled learning problem that admits lightweight online training along the
diffusion trajectory, using condition samples as positives and
diffusion particles as unlabeled data. Because the guided particles drift from the unconditional marginal, we further correct the residual mismatch via Sequential Monte Carlo, yielding a sampler that is asymptotically exact in the particle limit and that turns any existing negative-guidance method into its proposal kernel. Empirically, our method approaches the ground-truth target on Gaussian mixtures and Pareto-dominates existing methods on trade-offs between avoidance and generation quality, distributional bias, and diversity, including in an image-generation setting with a pretrained score network.
Read the full abstract ▾