Robotics
HorizonNav: Geometry-Guided Video Diffusion for Anticipatory 3D Robot Navigation
Hajira Amjad, Valerii Serpiva, Viacheslav Liubimov et al. (+2 more)
We present HorizonNav, a
video diffusion-based framework for anticipatory
robot navigation that transforms a single RGB image and a text
prompt into a geometrically aligned representation of plausible surrounding 3D space. Given the initial observation and a textual description of the reconstruction objective, a
vision-
language model, guided by designed instructions, interprets the scene and constructs structured
prompts for a
video diffusion model. The resulting
video sequence provides plausible future observations of the environment, which are subsequently processed using monocular ORB-SLAM3 for camera pose estimation and
Video Depth Anything for dense depth prediction. Our central contribution is a geometry-guided depth
alignment stage that combines the estimated camera poses with triangulated visual landmarks to constrain dense depth predictions across multiple views. This
alignment improves the
geometric consistency of the reconstructed 3D environment, addressing inconsistencies that temporal coherence of the generated
video alone cannot resolve. The experimental evaluation of navigation in imagined 3D environment shows that across a 19-scene horizon study, planning-feasible scenes increase from 13/19 at P20 to 15/19 at P81, while successful RRT trials increase from 62/90 to 75/90. In a separate corridor study, 6 of 25 evaluable generated-camera trajectories violate the reconstructed collision constraints; RRT planner recovers feasible alternatives in 5 of those 6 cases. A larger archive contains 58 usable reconstructions from 67 attempted cases, of which 36 support a downstream plan and 22 do not, exposing reconstruction and free-space failure modes rather than hiding them without
lidar. Across 15 corridor reconstructions, the mean reconstruction error is approximately 0.30 m at matched structural landmarks.
Read the full abstract ▾
All authors Hajira Amjad, Valerii Serpiva, Viacheslav Liubimov, Didar Seyidov, Dzmitry Tsetserukou
Robotics
Mapless, Stabilizing, and Anti-Stuck Flipper Control for Articulated Tracked Robots
Valentýn Číhala, Pavel Sarga, Martin Pecka et al. (+2 more)
Nearly every autonomous or semi-autonomous flipper controller assumes access to an elevation map of the terrain under and in front of the
robot. The terrain beneath the chassis is occluded in the sensors, so the map is remembered through odometry integration, whose drift corrupts it. We present Stabilizing Anti-Stuck (SAS), a mapless controller that combines a stabilization term for body roll and pitch with an anti-stuck term that frees the
robot when it gets stuck. As input, SAS uses the IMU, the flipper joint angles, and a motion cue that only tells whether the
robot is moving. The cue is computed over a short window and never accumulated, so SAS does not inherit the odometry-integration drift that corrupts the map. We deploy SAS on a tracked
robot with four flippers and evaluate it against manual teleoperation and prior flipper controllers in a large comparison, both in simulation and on a real obstacle. In simulation and on the real
robot, SAS achieves traversal quality comparable to state-of-the-art map-based controllers. On a log hidden in tall grass, invisible to the
LiDAR, the best map-based controller gets stuck while SAS traverses the hidden obstacle without a problem.
Read the full abstract ▾
All authors Valentýn Číhala, Pavel Sarga, Martin Pecka, Tomáš Svoboda, Karel Zimmermann
Robotics
MC-EvoBench: Auditing Experience-Driven Self-Improvement in Minecraft Agents
Jingwei Song, Zhisheng Chen, Bingfan Zeng et al. (+5 more)
A central challenge in embodied recursive self-improvement is enabling
agents to convert interaction history into capabilities that persist beyond one world and transfer to later tasks. A saved
memory or successful final action does not by itself establish that a reusable capability was acquired; evaluation must separate capability change from carried-over resources, unequal source exposure, and early execution failures. We construct MC-EvoBench as a controlled Minecraft
benchmark with a fixed catalog of 19 registered
evolution chains, including 12 primary evaluation chains and seven calibration or probe chains, together with three graph-aligned paired probe families for transfer. The
benchmark represents each task as a task-state graph, defines stage-level environment verifiers, assigns matched source and target roles, resets the target world, and permits only a declared
agent artifact to cross the source-target boundary. For each probe, no-prior, related, unrelated, and shuffled source conditions are evaluated under matched environment-step budgets and five seeds; we report TSR, WGP, and VMP and retain provenance records. The
benchmark reveals a consistent direct-capability gap between controller and planner-backed variants, while related experience yields no stable transfer: it does not improve either tool probe, its small wood-species gain hinges on one seed, and all intervals include zero. Stage audits further show that early wood-collection and crafting bottlenecks often leave later capabilities unobserved. MC-EvoBench thus offers a controlled testbed for durable, transferable embodied capabilities. Code and data are available at https://github.com/xzw-ustc/mc-bench-open-source
Read the full abstract ▾
All authors Jingwei Song, Zhisheng Chen, Bingfan Zeng, Jing Tang, Yubo Hou, Jinhao Jing, Zhengwei Xie, Linfeng Zhang
Robotics
Dream to Remember: Continual Steering of Generalist Robot Policies with World Models
Bahey Tharwat, Iman Nematollahi, Marvin Alles et al. (+3 more)
Generalist
robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. When tasks arrive sequentially, both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, $\pi_0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $\pi_0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka
robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.
Read the full abstract ▾
All authors Bahey Tharwat, Iman Nematollahi, Marvin Alles, Xingyuan Zhang, Akshay L Chandra, Abhinav Valada
Robotics
When LiDAR Fades, Radar Takes Over: LiDAR Reliability-Aware Adaptive Localization Under Airborne Obscurants
Moumita Mukherjee, Avijit Banerjee, Anton Koval et al. (+1 more)
LiDAR-inertial odometry (LIO) is widely used for real-time localization and mapping, but its reliability can degrade substantially in subterranean environments affected by smoke, dust, and other airborne obscurants, whereas radar remains comparatively robust. This article presents a one-class temporal learning framework for
LiDAR degradation detection and reliability-aware localization under such conditions. A broad set of degradation-sensitive cues, including spatial, cross-modal,
geometric/motion, intensity, and measurement-recency features, is evaluated to characterize changes in
LiDAR sensing quality. Their temporal
evolution is modelled over sliding windows using an LSTM autoencoder trained exclusively on nominal operation, enabling deviations from healthy sensor behaviour to be detected without requiring labelled degradation data. The resulting
LiDAR reliability estimate governs adaptive selection between LIO and radar inertial odometry (RIO), retaining LIO during reliable sensing and transferring localization authority to RIO when degradation is detected. The framework is evaluated using real-world ground
robot data collected in GNSS-denied subterranean tunnels under nominal and dense-smoke conditions. Experimental results demonstrate degradation detection, adaptive modality switching, and continued localization output under severe
LiDAR degradation. A feature family ablation study further shows that a reduced feature representation can improve degradation-aware localization performance in the evaluated dense-smoke scenarios, highlighting the importance of feature parsimony in one-class sensor degradation detection. The code will be released as open source to support reproducibility and further research.
Read the full abstract ▾
All authors Moumita Mukherjee, Avijit Banerjee, Anton Koval, George Nikolakopoulo
Robotics · Cross-listed from cs.SE
MARGoS: A Configuration-Driven Platform for Reproducible MARL Experiments in ARGoS
Tom Philipp Backert, Mohammad D. Soorati and Javad Ghofrani
Multi-
agent reinforcement learning (MARL) has become a prominent research direction, particularly within collective robotics, where the ARGoS simulator is widely used to study emergent swarm behavior. Yet despite this growth, systematic support for reuse and reproducibility in MARL-based swarm robotics remains limited and insufficiently standardized: a single experiment is assembled from manually aligned simulator files, training scripts, configuration values, paths, seeds, and environment details, so reuse and reproduction stay fragile even within one research group. To address this gap, we introduce MARGoS, a lightweight, configuration-driven platform developed in this project that elevates these experiment-defining inputs to first-class, platform-managed artifacts. MARGoS enables reproducible experimentation through configuration hashing, end-to-end seed propagation, frozen (immutable) configurations, structured logging, environment fingerprinting, and exportable experiment bundles that facilitate sharing, replay, and verification of results across machines and over time. A five-dimension evaluation (architecture, efficiency, reproducibility, usability, and collaboration) supports our claim that managing experiment variation as a platform responsibility makes ARGoS+MARL experiments easier to reuse, reproduce, and hand off. Our code is available at https://github.com/tombackert/margos.
Read the full abstract ▾
Robotics
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Timi Owolabi, Hikmah Olawore, Frances Adelakun et al. (+1 more)
World-action models (WAMs) predict a
robot's actions together with the observations those actions will produce. However, the strongest models rely on
video diffusion transformers that are far too computationally expensive for resource-constrained edge devices. We present SmolWAM, a latent WAM that we derive from the 2B-parameter Cosmos Policy by keeping its pretrained trunk and rebuilding its inference interface: single-frame visual latents, cached text features that preserve the trunk's
attention sinks, 12 of 28
transformer blocks, and 1,008
tokens from which actions and future visual latents are read out in one rectified-flow step. Trained in 3.6 GPU-hours, SmolWAM succeeds in 95.5% of 400 LIBERO episodes at 57 ms and 2.3 TFLOP per 16-action chunk with 2.4 GiB of GPU
memory, 11x faster than its teacher with 18.5x fewer FLOPs and 3.2x less
memory, and its world head still forecasts scene motion.
Read the full abstract ▾
All authors Timi Owolabi, Hikmah Olawore, Frances Adelakun, Ayotomiwa Oyewumi
Robotics · Cross-listed from cs.CV
LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos
Zhiyuan Xiang, Peidong Liu, Mingyang Li et al. (+4 more)
Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human
videos offer a rich alternative, but they lack
robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human
videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-
video latent-action pre-training, target-
robot embodiment
alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow
alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric
video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-
robot success from 63.3% to 85.0% as annotated human
video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.
Read the full abstract ▾
All authors Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng
Robotics
Towards a Unified Taxonomy of Agricultural Harvesting Technologies: Linking Physical Interaction, Automation, and Selectivity
Renato Cristiano Torres, Fabiano Mariath D Oliveira and Claudio Gottschalg Duque
Harvesting technologies have evolved from conventional mechanized systems based on bulk mechanical interaction toward increasingly automated and robotic systems capable of perception, decision-making, and localized actuation. This
evolution has also created a classification problem, as harvesting systems are commonly described according to crop type, mechanical removal principle, sensing technology, robotic architecture, or degree of automation, making direct comparison difficult. This paper proposes a unified taxonomy for agricultural harvesting technologies based on two primary classification axes: the physical principle governing plant-machine interaction and the level of automation incorporated into the harvesting process. Selectivity is treated as an emergent functional attribute that expresses the biological and spatial scale at which perception and decision-making influence physical actuation. Conventional mechanized methods, including cutting, pulling, vibration, impact, brushing, pneumatic action, and combined systems, are analyzed together with semi-automated and robotic harvesting approaches. The analysis shows that increasing automation does not necessarily result in greater selectivity. A qualitative transition occurs when perception and decision-making directly influence product-level actuation, shifting harvesting from distributed or bulk interaction toward localized and selective manipulation. The proposed framework provides a common conceptual basis for positioning and comparing mechanized, automated, and robotic harvesting technologies. By separating the physical mechanism of product removal from the automation level and resulting selectivity, the taxonomy clarifies technological boundaries, identifies intermediate configurations, and supports the analysis and development of future harvesting systems.
Read the full abstract ▾
Robotics
Cooperative Aerial Transportation of a Payload with a Sliding Cable Attachment
Sebastian Tilaguy Lezama and Diego S D'Antonio
This paper presents a dual-quadrotor hook- suspended transport system for cooperative aerial manipulation of a payload that is not rigidly attached to a fixed point along the cable. Two quadrotors are connected by a single inextensible cable that passes through a frictionless hook mounted on the payload, allowing the cable to slide between its two segments. This introduces a cable-distribution variable that couples the quadrotor motion, cable geometry, and payload dynamics. A feasible reference-generation framework is formulated using the desired payload pose, cable distribution, and system shape while satisfying the cable-length constraint and elliptical workspace. The quadrotors track the geometrically generated trajectories while maintaining positive cable tension and respecting actuator limits. The payload pose is estimated from the system geometry, and its measured position is used only to validate the desired trajectory. The approach is evaluated through experimental demonstrations.
Read the full abstract ▾
Robotics
DexBioLab: Benchmarking Dexterous Manipulation with Dynamic Experimental Phenomena in Realistic Laboratories
Xuanye Wu, Yue Shi, Yongcan Yang et al. (+8 more)
Automating biological laboratories requires
robots to perform dexterous tool interactions under realistic visual conditions while handling dynamic experimental phenomena, presenting significant challenges for robotic manipulation. To enable systematic evaluation under these conditions, we introduce DexBioLab, a simulation
benchmark and framework for dexterous manipulation in realistic biological laboratory environments. DexBioLab comprises 14 tasks across 4
robot embodiments, spanning functional tool use, precision manipulation,
language-conditioned visual grounding, and instrument operation. We introduce controllable dynamic experimental phenomena, including flames, vapor, and bubbles, to capture the evolving visual conditions of laboratory procedures. To further enhance environmental realism, we develop a hybrid rendering pipeline that integrates interactive simulated foreground objects with Gaussian-splatting backgrounds reconstructed from 6 real laboratory scenes. For efficient large-scale data generation and evaluation, we further build a parallel simulation framework on top of mjlab, proposing Parallel Finite-State Machine (PFSM) and integrating batched rendering with decoupled trajectory replay for demonstration generation and model-independent closed-loop policy evaluation. Our evaluation reveals substantial variation in policy performance across tasks, exposing distinct strengths and weaknesses of current manipulation policies. Real-world experiments further demonstrate a reduced sim-to-real gap, supporting the validity of AutoBio+ for realistic manipulation evaluation. Supplementary materials are available at https://cumulusalpha.github.io/DexBioLabWebPage.
Read the full abstract ▾
All authors Xuanye Wu, Yue Shi, Yongcan Yang, Yiqun Wang, Zhaoqi Wan, Dayu Xia, Yanming Shao, Jiezhang Cao, Xiangfei Li, Yao Mu, Guangtao Zhai
Robotics
JitterVLA: A Replay Benchmark for Motion Quality in VLA-Driven Human-Robot Manipulation
Faith Ogunwoye, Didar Seyidov, Konstantin Gubernatorov et al. (+1 more)
Vision-
language-action (VLA) policies issue sparse targets whose arrival at the
robot controller is delayed and irregular, so a reasonable predicted action can still produce abrupt motion near people. Remedies are evaluated on different streams, delays and metrics. JitterVLA replays identical commands and arrival traces through alternative controllers, isolating the controller's effect. We apply it to synthetic reaches, hand trajectories from 20 people, three open VLA pipelines (838 episodes) and a physical UR3. Interpolation lowered modeled p99 jerk for every source, by 83% (615 to 106 m/s^3) on human motion with 200 ms delay, at a 2.0 cm tracking cost, and the effect replicated on 598 new episodes; jerk-limited generation lowered jerk further at a larger cost.
Entropy-discontinuity associations had opposite, replicated signs in two pipelines, and
entropy gating helped in one but not the other. On the UR3, interpolation lowered accelerometer-derived jerk by 3.0-14.0% in all six exploratory pairs, while a position-derived metric did not resolve the contrast. JitterVLA supports screening before human-centred studies; code and data will be released.
Read the full abstract ▾
All authors Faith Ogunwoye, Didar Seyidov, Konstantin Gubernatorov, Dzmitry Tsetserukou
Robotics
RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation
Xuhui Kang, Sung-Wook Lee, Yan Zhuang et al. (+2 more)
Pre-trained
robot policies, such as
Vision-
Language-Action (VLA) models, provide a strong foundation for generalist robotic manipulation, but adapting them to new tasks with only a few demonstrations remains challenging. Directly fine-tuning a pre-trained policy often achieves strong in-distribution performance, yet it can distort the representations learned during pre-training and weaken the model's generalization ability under out-of-distribution shifts. We propose
Retrieval-Anchored Representation
Alignment (RARA), a two-stage adaptation method, to address this issue. First, RARA retrieves representation features from pretraining with similar task intent and motion, and aligns the policy's representations of the target demonstrations with the pre-trained representations of the retrieved demonstrations. Second, RARA fine-tunes the policy on the target demonstrations while limiting how much its representations of the retrieved demonstrations change during fine-tuning. By reducing the representation mismatch before action learning and preserving pre-trained knowledge during adaptation, RARA limits unnecessary representation drift during fine-tuning. Experiments with
Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa, and on a real
robot show that RARA improves the average success of fine-tuning by 9.5 points in-distribution and 13.6 points out-of-distribution, and outperforms all compared baselines. Project page: https://shua-kang.github.io/RARA_
robot/
Read the full abstract ▾
All authors Xuhui Kang, Sung-Wook Lee, Yan Zhuang, Tianle Zhong, Yen-Ling Kuo
Robotics
ATHLETE: Learning Reactive Humanoid Tennis via Trajectory-Guided Motion Matching
Zhihao Tian, Zifan Wang, Enzheng Zhu et al. (+4 more)
Humanoid tennis requires reactive whole-body coordination under rapidly changing ball dynamics, yet collecting task-specific motion-capture demonstrations that cover diverse approach states, contact locations, and timing conditions is costly. Inspired by how humans acquire new skills by combining reusable motor experience with task-specific demonstrations, we present ATHLETE, a framework for learning reactive humanoid tennis from generic locomotion data and ordinary tennis
videos. ATHLETE combines LAFAN1 locomotion with
video-derived striking motions and uses Trajectory-Guided Motion Matching (TGMM) to construct diverse task-conditioned whole-body references. These references are transformed into a reactive policy through physics-based teacher–student learning with online DAgger
distillation, followed by PPO fine-tuning for the tennis task. The resulting policy responds directly to
robot and ball observations without requiring reference-motion inputs at deployment. On a real humanoid
robot, ATHLETE achieves forehand and backhand hit rates of $90.00\%$ and $86.67\%$, respectively, and enables multi-shot rallies with human players. These results demonstrate that ATHLETE can learn dynamic humanoid tennis skills from reusable locomotion data and
video demonstrations. Result
videos and code can be found at https://anonymous.4open.science/w/ATHLETE2027/.
Read the full abstract ▾
All authors Zhihao Tian, Zifan Wang, Enzheng Zhu, Xun Yang, Jialong Qin, Mengbo Wang, Junwei Liang
Robotics · Cross-listed from cs.LG
Probabilistic Ship Trajectory Forecasting using Transformer Models on AIS Data
Frederik E. T. Schöller
Anticipation of vessel trajectories is an essential task when navigating at sea. This applies to both human navigators and autonomous surface vessels. Trajectory forecasting enables sub-routines such as collision avoidance to assess the situation and react accordingly in time. This paper investigates
Transformer and Performer models for trajectory prediction of surface vessels in confined waters. Using widely available position data from the Automatic Identification System (AIS), candidate models are trained and evaluated. Historical information on how other vessels have moved in the local area surrounding each
agent is conveyed using positional heat maps. The heat maps are generated using Kernel Density Estimation (KDE) and encoded with a Convolutional Neural Network (CNN) before entering the
Transformer models. The models are trained with
quantile loss to efficiently obtain a model-aware prediction interval, as this is crucial for timely collision avoidance. The proposed model architectures are shown to greatly outperform previous work while the inclusion of local heat maps enables the prediction of vessel movement that is not otherwise indicated from the previous positions.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
MAGMA-Bench: Evaluating Long-Horizon Embodied Agents under Dynamic Tasks and Execution Failures
Loan Bernat, Matthieu Grard, Ariane Herbulot et al. (+1 more)
Long-horizon embodied
agents must maintain task-relevant information, coordinate heterogeneous
robots, and adapt their decisions as execution unfolds. Existing
benchmarks investigate dynamic task execution and embodied multi-
agent coordination, but provide limited evaluation of their intersection. We introduce MAGMA-Bench, an architecture-agnostic
benchmark for persistent task
reasoning in dynamic multi-
robot environments. MAGMA-Bench evaluates
agents across increasing task horizons and controlled online perturbations, including execution failures, task interruptions, and mission updates, with scenarios requiring coordination across
robots with heterogeneous capabilities. It uses a structured skill-level interface to separate task-level
reasoning from perception and low-level control. We evaluate locally deployable open-weight planners, multi-
agent state-tracking strategies, and frontier models. Even the strongest evaluated models fail on a substantial fraction of clean episodes, and their success rates decrease further under online perturbations. Supervised adaptation greatly improves a small model, but performance remains substantially lower on longer and multi-
robot missions. We release the
benchmark, 261,646 task-level training examples, and an architecture-agnostic expert-trace generation pipeline.
Read the full abstract ▾
All authors Loan Bernat, Matthieu Grard, Ariane Herbulot, Florent Lamiraux
Robotics
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Peiyan Tu, Cong Wang, Hanxin Zhu et al. (+2 more)
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic
datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human
videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples
video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and
robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action
alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth
videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned
video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.
Read the full abstract ▾
All authors Peiyan Tu, Cong Wang, Hanxin Zhu, Xin Jin, Zhibo Chen
Robotics
RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors
David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin et al. (+1 more)
Runtime failure monitors for
robot policies are typically evaluated by how accurately they distinguish failed from successful executions. Yet an accurate alarm may arrive too late for recovery, while an unnecessary intervention may disrupt an execution that would otherwise succeed. We introduce RESCUE, a counterfactual framework for evaluating whether a monitor's alarms enable useful interventions. By restoring simulator and policy states at execution checkpoints, RESCUE compares alternative actions, identifies the latest checkpoint satisfying a specified recovery criterion, and evaluates whether acting on alarms reduces expected task-completion cost. We evaluate RESCUE with OpenVLA-OFT and $\pi_{0.5}$ over LIBERO-Spatial and LIBERO-Plus using 128,340 counterfactual branches. Recoverability depends on the state–intervention pair rather than on
robot state alone: at $\tau=0.5$, 14 of 66 OpenVLA-OFT/LIBERO-Plus failures contain a recoverable checkpoint under continuation, compared with 34 of 66 under continuation or scripted recovery, while recovery reduces recoverability in other conditions. On the primary strong-policy conditions, favourable detection scores do not establish lower intervention cost. Under the evaluated stop-and-reset model, even an idealised detector can afford only a 2.0–3.3% false-alarm rate before unnecessary resets outweigh the savings from stopping failures early. These results show that monitor value depends jointly on the intervention available and the workload in which it is deployed.
Read the full abstract ▾
All authors David Ahmedt-Aristizabal, Abdelwahed Khamis, Mohammad Ali Armin, Lars Petersson
Robotics · Cross-listed from cs.AI
A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance
Victoria J. Hodge, Colin Paterson and Rob Alexander
Generating edge case scenarios is crucial for training, testing and safety assuring autonomous systems. This identifies hazardous failures to increase system reliability and robustness, ensures the system can still function safely on the boundaries of its operational design domain and learns the expected system behaviour for these edge cases. The rapid advancement in the generation and analysis capabilities of
language models (LMs) has expanded their applicability and recent work has investigated LM scenario generation. However, many recent approaches rely on human-in-the-loop or rule-based strategies. This restricts the LM's ability to generate diverse and edge case scenarios; the long tail of the distribution. Our work uses local LMs in a laptop-based agentic AI framework for adaptive scenario generation. This is cheap, fully-controlled, ensures data
privacy and security. It uses think, act, observe, update, and continue loops with lightweight JSON encoding to generate scenarios. We use the system requirements, system description and environment description as knowledge bases in contextual
prompts for an LM to ensure scenario realism. A second, independent LM critiques each scenario using domain specific criteria so it can be refined. The refined scenarios are run in simulation and evaluation metrics collected. These metric scores are analysed and similar scenarios are created for edge cases (exploitation) or novel scenarios are generated to explore the scenario space. Post-hoc clustering using metric scores and thematic-type analysis of the rich-text scenario descriptions produced by the LM pinpoints functional gaps and their triggering conditions. This creates traceable evidence for the safety assurance case. Our work highlights the potential of a flexible and adaptive local agentic AI framework for automating edge case generation and producing traceable safety assurance evidence.
Read the full abstract ▾
Robotics
Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving
Yuechen Luo, He Chong, Fang Li et al. (+7 more)
Reinforcement Learning (RL) substantially improves planning performance in autonomous-driving
Vision-
Language-Action (VLA) models. However, existing RL methods rely on scalar trajectory rewards that measure overall quality but fail to identify specific planning errors or provide actionable correction signals. In long-tail scenarios, errors in scene understanding, behavior selection, and trajectory execution are compressed into a single score. We observe that Reflective Guidance turns trajectory evaluation into explicit diagnoses and corrective instructions, substantially improving planning accuracy. Yet such guidance requires an initial prediction and its evaluation results, making it unavailable for direct single-pass inference. We propose Reflect-VLA, which treats Reflective Guidance as training-time privileged information and internalizes its planning benefits through Dual-Path On-Policy Self-
Distillation (Dual-OPSD). Unlike scalar-reward optimization, Dual-OPSD uses the driving score only to route student-generated trajectories. It distills dense
token-level corrective distributions from the teacher for risky trajectories while applying self-imitation to preserve reliable ones. To provide reliable corrective distributions on student rollouts, the teacher is progressively constructed through Driving-Knowledge Pretraining, Planning-and-Refinement SFT, and Self-Refinement Rejection Sampling. At inference time, Reflect-VLA generates trajectories in a single pass without explicit guidance or iterative refinement. Experiments on NAVSIM v1/v2 and Bench2Drive demonstrate competitive planning performance while reducing training time by over 90% compared with RL.
Read the full abstract ▾
All authors Yuechen Luo, He Chong, Fang Li, Shaoqing Xu, Hexin Zhang, Ruiqi Zhang, Ziying Song, Lei Yang, Zhi-Xin Yang, Fuxi Wen
Robotics
ManeuverFlow: Maneuver Hypothesis Generation with Future Evidence for Trajectory Evaluation and Refinement in End-to-End Autonomous Driving
Yuhao Wei, Yong Chen, Chen Yang et al. (+7 more)
End-to-end planning in complex scenes must model diverse driving intentions and select the most appropriate trajectory using scene context and future risks. However, multi-query regression may collapse to one mode, while anchor-based and generative planners share a scaling bottleneck: small candidate sets limit maneuver coverage or discovery of high-quality trajectories, whereas large sets increase generation and scoring costs. We propose ManeuverFlow to generate trajectories for distinct intentions in one step and evaluate and refine them using current and future evidence. The model selects relevant maneuver modes, draws corresponding noise from the Gaussian mixture prior, and maps it to intention-specific trajectories through a maneuver-routed mixture-of-experts MeanFlow field. Current features and future features predicted by the world model are aggregated along each candidate path into trajectory-specific evidence for PDM-factor prediction. Using the same evidence, the refiner predicts local Frenet residuals for selected candidates; refined and retained candidates are then jointly re-scored for selection. With a lightweight TransFuser-style ResNet-34 backbone, ManeuverFlow achieves 90.6 PDMS and 90.1 EPDMS on NAVSIM v1/v2 navtest, respectively. On WorldEngine's recent 3DGS closed-loop
benchmark, it attains the highest listed PDMS$^\ast$ scores of 75.19 and 74.58 under logged and IDM-controlled traffic, respectively. Code: https://github.com/wyh-autonomy/maneuverflow.
Read the full abstract ▾
All authors Yuhao Wei, Yong Chen, Chen Yang, Yingchen Wang, Delin Ouyang, Lingfeng Qi, Qi Lan, Zhanhua Zhang, Jie Li, Guofa Li
Robotics
Robot-WATCHDOG : failure detection through object-centric graph representation
Quentin Rolland, Fabrice Mayran de Chamisso and Jean-Baptiste Mouret
Reliable real-time failure detection is critical for deploying learned robotic policies, yet defining failures explicitly is infeasible in open-world settings. This motivates failure detection from nominal demonstrations alone. Existing methods, however, struggle to balance robustness to benign variations with the ability to capture semantic and interaction-level errors. We introduce
Robot-WATCHDOG, a self-supervised framework that models manipulation scenes as object-centric spatio-temporal graphs. Building on this representation, we investigate two complementary failure scores: Trajectory Projection and Correlation (TPC), a non-parametric measure of
geometric deviation and relational consistency, and GnnT, a Graph
Attention Network with a
Transformer that predicts future object states. We evaluate both on a public
benchmark and on a new
dataset of semantically complex manipulation tasks, across six tasks and ten random seeds. Both scores outperform image-based failure detectors, particularly on semantic and interaction-level failures. TPC provides the strongest performance in AUPR and MCC on both
dataset averages, with low variance across seeds. Qualitative results are made available at https://cea-list.github.io/
Robot-WATCHDOG/.
Read the full abstract ▾
Robotics
EndoRSI: Physical-Evidence-Driven Recursive Improvement of Endoscopic Robot Policies
Chi Kit Ng, Yidong Zhang, Tianchun Wu et al. (+4 more)
Artificial intelligence is beginning to automate scientific discovery, yet physical
robot-policy research still relies on scientists and engineers to interpret experimental failures and decide how controllers should be revised. We present EndoRSI (Endoscopic Recursive Self-Improvement), a human-governed, self-improving robotic research system in which the embedded EndoForge coding
agent iteratively develops and refines endoscopic
robot policies using evidence from physical experiments. Physical execution remains isolated within a safety-constrained runtime under human authorisation. In motorised endoscopy, validation evidence first caused the
agent to reject an initial action representation, while subsequent physical experiments falsified an offline-supported insertion strategy by revealing hazardous closed-loop behaviour absent from the source data. Failed physical rollouts were then incorporated into policy revision; insertion success subsequently increased from 2/21 in the first generation to 17/21 in the fourth. The frozen final policy retained competence across visual and hardware shifts and was the best performing autonomous policy across six independent ex-vivo porcine specimens. These results show that physical
robot interaction can serve as experimental evidence for
agent-driven
robot policy research.
Read the full abstract ▾
All authors Chi Kit Ng, Yidong Zhang, Tianchun Wu, Siu Hing Lui, Hanlin Zhao, Huxin Gao, Hongliang Ren
Robotics
PartiGen: Part-Aware Representations for Language-Conditioned Humanoid Motion Generation
Chuxuan Huang, Zhanxiang Cao, Tianyue Zhan et al. (+14 more)
Translating
language into humanoid motion requires coordinating body parts, determining motion duration, and producing references suitable for
robot execution. We introduce PartiGen, a part-aware latent
diffusion framework that incorporates anatomical structure into a shared whole-body motion representation. Part-specific
attention processes anatomically masked frame features, and its outputs are fused to combine local limb information with whole-body context. Conditioned on text and motion history, the model generates motion primitives autoregressively. Forward-kinematics supervision encourages kinematic consistency, while a learned end-of-sequence predictor selects a frame-level endpoint after generation. On HumanML3D retargeted to the Unitree G1, PartiGen achieves a segment-level FID of 3.165, a 16.8% reduction over TextOp, and improves R@1 by 1.4 percentage points. In the duration study with 16-frame primitives, endpoint prediction reduces mean absolute duration error from 12.471 to 3.080 frames compared with a fixed-length cutoff. We further evaluate
robot execution using a pretrained, frozen GRIT tracker: 409 of 500 generated references (81.8%) pass the posture-based stability screen in MuJoCo. Following simulation screening and manual selection, 26 of 27 attempted motions are judged successful on the real G1.
Read the full abstract ▾
All authors Chuxuan Huang, Zhanxiang Cao, Tianyue Zhan, Haoyang Li, Yang Zhang, Liyun Yan, Chenlin Ming, Keqi Zhu, Yizhi Chen, Shengcheng Fu, Zihao Zhou, Xinyi Peng, Kaixuan Zhang, Boyang Xing, Jiawei Bao, Yang Yang, Yue Gao
Robotics · Cross-listed from cs.AI
AlphaNIST: Emergent Assembly Skills from Binary Task Rewards
Octi Zhang, Iretiayo Akinola, Rosario Scalise et al. (+12 more)
The NIST Assembly Task Board #1 remains a challenging
benchmark for learning-based methods in robotic manipulation. Over short horizons, it demands precise, contact-rich control of small, tight-fitting parts. Over long horizons, it requires recovery and coordination across extended action sequences. Demonstrations, task decomposition, and dense progress rewards have enabled research advances by providing guidance on how to assemble. But how much of the strategy must be specified? The progression from AlphaGo to AlphaGo Zero showed that RL at scale can discover superhuman strategies without prescribing how to win. AlphaNIST asks whether a sparse binary task reward and task diversity admitted by this task-agnostic reward can similarly let a
robot discover how to assemble. In simulation, we train a single policy from scratch across all 20 assembly tasks on the NIST Assembly Task Board #1, reaching 98.2% overall training success without demonstrations, progress shaping, scripted primitives, or per-task tuning. The framework also trains policies to complete ten assembly pairs in a single episode without a prescribed order, with task success rewarded only when all ten pairs are simultaneously complete. The learned policies discover dexterous and planning-like behaviors: choosing grasps for later assembly, using surrounding geometry to reorient parts, and preparing one part, assembling another, then returning to finish the first without an explicit planner. For
videos, see our website: https://alpha-nist.github.io/
Read the full abstract ▾
All authors Octi Zhang, Iretiayo Akinola, Rosario Scalise, Mateo Guaman Castro, Patrick Yin, Tobias Widmer, Miles Macklin, Chris Amevor, Dylan Turpin, Dirk Van Gelder, Jan Carius, Ankur Handa, Gavriel State, Yunrong Guo, Yashraj Narang
Robotics
Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience
Kai Ye, Yifan Jin, Qianhang Luo et al. (+7 more)
Existing real-world reinforcement learning methods improve
robot policies by combining human corrections with autonomous rollouts. However, accumulated experience varies in its value for policy improvement, and even successful trajectories can contain inefficient or redundant behavior, motivating more selective experience reuse. To this end, we propose Evo-RL, an iterative reinforcement learning method that combines human interventions with advantage-guided experience selection. Evo-RL uses human corrections and experience with positive estimated advantages from successful trajectories to iteratively refine the policy. To support learning from these different experience sources, we introduce RW-RL-
Dataset, which spans over 1,000 hours across four
robot embodiments. On a real-world high-precision manipulation task, Evo-RL improves attempt success rate over RECAP by 30.46%. Our code, model checkpoints, and
dataset are publicly available.
Read the full abstract ▾
All authors Kai Ye, Yifan Jin, Qianhang Luo, Hongru Zhang, Jiedong Zhang, Shuyuan Hu, Song-Yuan Wu, Jihai Xu, Jie Zhao, Bo Zhao
Robotics · Cross-listed from cs.CV
NovaSplat: Model-Free 6D Object Pose Estimation via 3D Gaussian Splatting Onboarding and Refinement
Georgios Floros, Evangelos Sartinas, Orestis Vangelis et al. (+1 more)
6D pose estimation of novel objects without CAD models remains an open challenge: model-free methods must build the object representation from scratch during onboarding and have long trailed their model-based counterparts. We present NovaSplat, a model-free pipeline that closes this gap. From a short smartphone
video captured at onboarding, we train a 3D Gaussian Splatting (3DGS) model that serves both as a template source for detection and coarse pose estimation and as a differentiable backbone for pose refinement. Our refiner optimizes the object-to-camera pose with combined photometric and
geometric losses under a continuous rotation parametrization free of gradient singularities. On a household-object
benchmark, NovaSplat surpasses all model-based methods. On a challenging industrial
benchmark with textureless and specular surfaces, it outperforms all model-free competitors while substantially narrowing the gap to model-based ones. We further introduce NovaSplat-Lite, an inference variant that is $3.1\times$ faster at negligible accuracy cost. Code will be made available at \href{https://github.com/POSE-Lab/novasplat}{https://github.com/POSE-Lab/novasplat}.
Read the full abstract ▾
All authors Georgios Floros, Evangelos Sartinas, Orestis Vangelis, Maria Pateraki
Robotics · Cross-listed from cs.CV
Sonic-AMG: Streaming Human Motion Generation from Spatial Audio
Jingcheng Ni, Zekun Li, Zhiyang Dou et al. (+6 more)
Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate
tokens. Conditioned on these
tokens, a causal
transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
Read the full abstract ▾
All authors Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura, Srinath Sridhar
Robotics
Yang Wang, Ping Liu, Ruiyu Wang et al. (+3 more)
Memory-dependent manipulation requires a policy to track execution state and recover visual evidence that is no longer observable. We propose dual-stream
memory for
vision-
language-action policies, assigning these requirements to complementary representations. A task-agnostic Gated DeltaNet compresses history into one temporal
token, supervised by progress prediction and action-conditioned feature reconstruction. A task-conditioned retriever retains frames above a fixed relevance threshold; linear multi-scale fusion encodes each frame into sixteen spatial
tokens. We then freeze the temporal compressor and jointly optimize
retrieval and policy learning. Across sixteen RoboMME tasks, the complete model achieves 49.22% mean success versus 44.51% for the published FrameSamp+Modul reference, improving fourteen tasks and all four suite means. On four real-
robot tasks, mean success reaches 62.5% versus 37.5%, with the largest gains on repeated actions requiring execution-stage discrimination. These results support combining execution context with selective visual access, while failures under cup swaps expose persistent identity tracking as an unresolved requirement.
Read the full abstract ▾
All authors Yang Wang, Ping Liu, Ruiyu Wang, Xuhua Chen, Lingfeng Zhang, Tao Zhang
Robotics
Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models
Min Lin, Bokai Zhou, Jianqi Lin et al. (+8 more)
Actions specify what a
robot should execute, but do not describe what those commands will achieve.
Vision-
language-action (VLA) policies nevertheless learn primarily by predicting action chunks from images,
language, and state. We introduce Action–Future Coupled Effect Learning (AFCE), which makes a future-grounded physical Effect the policy's primary prediction target. An offline tokenizer constructs temporally ordered Effect
tokens from demonstrated actions, realized
robot responses, and sparse visual transitions. Reconstruction objectives encourage the representation to retain executable detail and observed physical change. A flow-matching VLA predicts Effects from current images, instruction, and
robot state, while a state-conditioned decoder converts them into action chunks. Embodiment-specific interfaces allow single- and dual-arm tasks to share the Effect model. On DexJoCo, AFCE achieves $50.97\%$ overall success, improving over $\pi_{0.5}$ and UVT by 11.82 and 15.21 percentage points, respectively. It leads on all five bimanual tasks, highlighting the value of Effect supervision for coordinated dexterous manipulation. The project page is available at https://afceproject.github.io/AFCE-Web/.
Read the full abstract ▾
All authors Min Lin, Bokai Zhou, Jianqi Lin, Kaiyuan Yang, Yicheng Chang, Junjie Yu, Xiaofan Liu, Jianhua Han, Rui Li, Shen Zhao, Xiaodan Liang