Robotics
Trajectory Planning and Control near the Limits: an Open Experimental Benchmark on the RoboRacer Platform
Mattia Piccinini, Patrick Zambiasi, Aniello Mungiello et al. (+3 more)
We present a modular framework to
benchmark new and existing methods for trajectory planning and control in high-acceleration maneuvers that push autonomous driving to the limits. Our framework includes time-optimal raceline generation, online time-optimal velocity replanning,
geometric path tracking controllers, and a new model-structured neural network (MS-NN) to learn the inverse dynamics for steering control. We deploy our framework on a 1:10-scale RoboRacer platform, using two
circuits. Through several ablations with cautious and aggressive racelines, we study the performance of single modules and their combinations. We show that our MS-NN significantly improves tracking accuracy, decreases steering oscillations, and is physically interpretable. Moreover, online velocity replanning improves lap times by compensating for execution errors, and enables the vehicle to safely reach higher speeds and accelerations. To support future research, our code,
datasets,
videos and results are publicly available at https://roboracer-
benchmark.github.io/planning_control_
benchmark/.
Read the full abstract ▾
All authors Mattia Piccinini, Patrick Zambiasi, Aniello Mungiello, Mattia Piazza, Felix Jahncke, Johannnes Betz
Robotics
Justifying bio-inspired robotics research: A taxonomy of strategies
Margaret J. Zhang, Justin Ting and Talia Y. Moore
For most of human history, we have not thought systematically about how and why we incorporate aspects of the natural world into our designs. The lack of a systematic approach has resulted in inconsistencies in motivations and methods that make it difficult to predict or evaluate the success of bio-inspired design. This mismatch between expectations and results can lead to disappointment when a reader considers a bio-inspired design to be superficial, weak, or incomplete. This is especially true in the field of Robotics, in which similarity to a biological system might be the driving motivation for construction. In an effort to assist robotics researchers justify their specific bio-inspired approach and to assist funding program managers with discerning the value of different bio-inspired approaches, here we propose a taxonomy of motivations for bio-inspired design and describe the potential significant contributions that are likely to result from different approaches.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving
Sherif Khairy and Catherine M. Elias
Adverse weather (rain, fog, sand, and snow) degrades camera-based object detection in autonomous vehicles. Existing enhancement-then-detect approaches stall the safety-critical perception loop, violating hard real-time requirements. Progress on this problem is also constrained by an under-recognized evaluation ceiling: ground truth annotated on degraded images cannot credit a detector that recovers objects the annotators themselves could not see, so a genuinely useful enhancement can register as a near-flat F1 gain. This paper presents CADENet (Condition-Adaptive Asynchronous Dual-stream Enhancement Network), a training-free three-thread system: Thread S (YOLOv11n) delivers detections at full frame rate with zero added latency; Thread Q applies condition-adaptive enhancement (CAPE) and fuses results via
entropy-guided NMS (EG-NMS) without blocking Thread S; Thread E provides CLIP zero-shot weather classification, so new weather categories require only a new text
prompt, with no labeled data and no retraining. Evaluated on 1327 DAWN images (YOLOv11m, IoU = 0.5, confidence = 0.25), CADENet achieves Recall = 0.0103 (micro), F1 = 0.0230 on snow, and F1 = 0.0038 on rain. We formalize the annotation completeness bias on DAWN-class data, so the reported F1 values are lower bounds on the true gain; recall is the annotation-gap-immune headline metric. Thread S sustains approximately 44 FPS regardless of enhancement load. No model retraining or additional sensor hardware is required.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning
Ahmed Y. Gado, Omar Y. Goba, Alaa Hassanein et al. (+2 more)
Recent attempts to support high-level scene interpretation and planning in Autonomous Vehicles (AVs) using ensembles of Large
Language Models (LLMs) and Large
Multimodal Models (LMMs) continue to treat time as a secondary property. This lack of temporal grounding leads to inconsistencies in
reasoning about continuous actions, undermining both safety and interpretability. This work explores whether temporal conditioning within inter-
agent communication can preserve or enhance coherence without introducing degradation in semantic or logical consistency. To investigate this, we introduce three planner architectures with progressively increasing temporal integration and evaluate them on curated subsets of the BDD-X
dataset using semantic, syntactic, and logical metrics. Results show that while temporal conditioning reshapes
reasoning style, it yields no statistically significant improvements in standard NLP-based correctness metrics. However, qualitative analysis reveals predictive hazard
reasoning, stable corrective behavior, and strategic divergence in the Sentinel. These findings clarify the limits of
prompt-based temporal grounding and establish the first empirical
benchmark for temporal scene-to-plan
reasoning.
Read the full abstract ▾
All authors Ahmed Y. Gado, Omar Y. Goba, Alaa Hassanein, Catherine M. Elias, Ahmed Hussein
Robotics
Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives
Junli Wang, Zhihua Hua, Xueyi Liu et al. (+7 more)
Existing imitation learning methods for end-to-end autonomous driving predominantly learn from successful demonstrations by minimizing
geometric deviations from expert trajectories. This paradigm implicitly assumes that spatial proximity implies behavioral safety, leading to a critical objective mismatch: trajectories with nearly identical imitation losses may exhibit drastically different safety outcomes, where one remains recoverable while the other results in collision. To address this limitation, we propose BeyondDrive, a failure-aware imitation learning framework that jointly learns from successful and failed driving behaviors. First, we introduce a flow matching-based negative trajectory generator that synthesizes safety-critical yet expert-proximate trajectories, enabling explicit modeling of safety asymmetry. Second, we develop a diversity-aware sampling strategy that mitigates mode collapse and improves coverage of diverse failure modes during negative trajectory generation. Third, we propose a Repulsive Distance Loss that simultaneously attracts predictions toward expert demonstrations while repelling them from hard negative trajectories, thereby establishing discriminative safety boundaries in trajectory space. Applied to the uni-modal baseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on the NAVSIMv1 closed-loop
benchmark, outperforming prior state-of-the-art methods. Moreover, BeyondDrive generalizes effectively across different autonomous driving architectures, including multi-modal planners, and further demonstrates strong zero-shot transferability on the HUGSIM
benchmark.
Read the full abstract ▾
All authors Junli Wang, Zhihua Hua, Xueyi Liu, Zebin Xing, Haochen Tian, Kun Ma, Hangjun Ye, Guang Chen, Long Chen, Qichao Zhang
Robotics
KIO-planner: Attention-Guided Single-Stage Motion Planning with Dual Mapping for UAV Navigation
Dexing Yao, Haochen Li, Junhao Wei et al. (+11 more)
Autonomous
UAV flight in confined, wall-dense environments requires low-latency and reliable motion planning under strict safety constraints. Traditional optimization-based planners suffer from mapping latency and easily fall into local minima when navigating through dense structural obstacles. Meanwhile, existing end-to-end learning methods struggle to extract fine-grained
geometric features from raw depth images and lack hard kinodynamic constraints, leading to unpredictable collisions near walls. To address these issues, we propose KIO-planner, an
attention-guided single-stage trajectory planning framework. First, we integrate a Convolutional Block
Attention Module (CBAM) into the perception backbone to adaptively focus on critical structural edges and traversable space. Second, we introduce a novel Dual Mapping mechanism–comprising physical bounds activation and a deterministic
Geometric Safety Shield in the depth-pixel space–to enforce kinodynamic feasibility and collision-free flight without global map fusion. Extensive high-fidelity simulated experiments demonstrate that KIO-planner enables highly agile
navigation at speeds up to 3.0 m/s. Compared to the state-of-the-art baseline, KIO-planner achieves lower inference latency (approximately 24 ms) and generates significantly smoother trajectories, reducing control cost by 28.4%. Most notably, our Dual Mapping substantially increases the worst-case safety margin, measured by minimum distance to obstacles, from 0.48 m to 0.76 m, ensuring fast, smooth, and safer
navigation in highly constrained environments.
Read the full abstract ▾
All authors Dexing Yao, Haochen Li, Junhao Wei, Yifu Zhao, Yanxiao Li, Jiahui Xu, Jinxuan Hu, Lele Tian, Baili Lu, Zikun Li, Xu Yang, Sio-Kei Im, Dingcheng Yang, Yapeng Wang
Robotics
Multi-Session Ground Texture SLAM in Low-Dynamic Environments
Kyle M. Hart and Brendan Englot
The simultaneous localization and mapping community has introduced a growing number of systems adapted for multi-session operations where the operational environment features low-dynamic changes that impact mapping, such as surface wear, weather phenomena, or seasonal change. These systems allow for lifelong operations by a
robot within these environments. There is also growing interest in operations in environments where the unique ground texture is the only mapping feature available for use. These ground texture systems are not yet targeted for multi-session low-dynamic-change environments though. This work explores the impact of three different techniques on trajectory estimation accuracy in these multi-session low-dynamic ground texture environments. Of the three, the use of Kullback-Leibler Divergence, as a similarity score and a bias influencing loop closure confidence, is found to have the most success. We show an analysis of all three methods and a deeper exploration of the impact of Kullback-Leibler Divergence. We also introduce a
dataset for use by the robotics community that contains multi-session images where the ground changes between sessions and also high-accuracy pose information for use in evaluation.
Read the full abstract ▾
Robotics
D-CLING: Prior-Preserving Depth-Conditioned Fine-Tuning for Navigation Foundation Models
Shintaro Nakaoka, Takayuki Kanai and Kazuhito Tanaka
Navigation Foundation Models (NFMs) trained on large cross-
embodied datasets have demonstrated powerful generalizability in various scenarios. Adopting in-domain fine-tuning for an NFM efficiently calibrates the visuomotor policy, promising further improvement even in a novel scenario. However, the fine-tuned models still suffer from poor obstacle avoidance or fail to properly reach the provided goals. Furthermore, model updates using a small subset of data typically erode the pre-trained prior, compromising the pre-training generalization. Consequently, fine-tuning deteriorates the capability of the model for robust and accurate
navigation. In this work, we present a novel fine-tuning method that leverages large-scale pre-training while efficiently learning in novel setups, such as environments or camera configurations. In particular, inspired by ControlNet, we fine-tune an NFM by attaching a trainable copy of the pre-trained backbone using zero-initialized residual pathways, thereby learning
geometric cues. This design enables the model to efficiently acquire in-domain geometry while preserving pre-trained knowledge across various behaviors. Despite its simplicity, our comprehensive evaluation of real-world
navigation suggests that our proposal effectively enables robust long-horizon
navigation with minimal collisions and human intervention. Additionally, our offline analysis shows that the proposed method maintains or further improves action prediction capabilities beyond the fine-tuned
dataset, providing a key insight into continual learning for general
navigation. The project page: https://toyotafrc.github.io/DCLING-Proj/
Read the full abstract ▾
Robotics
RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models
Jingzhou Luo, Yifan Wen, Yongjie Bai et al. (+3 more)
Vision-
Language-Action (VLA) models have shown strong performance on
embodied manipulation, yet they remain brittle under visual observation changes, paraphrased
language instructions, and compounded perturbations. This limitation suggests that existing methods still rely heavily on shallow correlations in the training distribution, rather than learning stable couplings among task semantics, environment states, and action generation. Although recent efforts improve robustness through larger-scale training, post-training adaptation, or enhanced predictive modeling, they rarely enforce invariance-oriented consistency within the end-to-end policy itself. To address this issue, we propose RoVLA, a robust
vision-
language-action framework with multi-consistency constraints. RoVLA enforces consistency under three complementary transformations: instruction semantics, trajectory
evolution, and observation perturbation. Specifically, Instructional Consistency (IC) promotes stable grounding under semantically equivalent instruction rewrites, Evolutionary Consistency (EC) preserves coherent action intent throughout the generation process, and Observational Consistency (OC) improves robustness to visual and proprioceptive perturbations by enforcing consistent predictions before and after targeted disturbances. By explicitly modeling these invariances during training, RoVLA reduces reliance on superficial correlations and improves robustness and generalization. Experiments on LIBERO-Plus, RoboTwin 2.0, and real-world manipulation tasks show that RoVLA consistently outperforms strong baseline methods and exhibits superior robustness under diverse task and observation shifts. These results demonstrate the effectiveness of multi-consistency learning for robust
embodied control. Codes will be available at https://github.com/HCPLab-SYSU/RoVLA.
Read the full abstract ▾
All authors Jingzhou Luo, Yifan Wen, Yongjie Bai, Xinshuai Song, Yang Liu, Liang Lin
Robotics
HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models
Hoonhee Cho, Giwon Lee, Jae-Young Kang et al. (+3 more)
End-to-end autonomous driving has emerged as a compelling alternative to traditional modular pipelines by directly mapping raw sensor data to driving actions. While recent approaches achieve strong performance on single-domain
datasets, their performance degrades significantly when trained jointly across multiple heterogeneous domains. In practice, however, autonomous systems must operate across diverse environments with heterogeneous distributions, including different cities, sensor configurations, and traffic patterns, without domain-specific retraining. This gap highlights a key challenge in multi-domain learning: domain-specific variations across heterogeneous domains introduce conflicting learning signals, driving models toward compromised solutions that are suboptimal across domains. To address this, we propose a trajectory-driven learning paradigm that organizes training around planning trajectories, enabling the model to capture domain-invariant representations of driving intent. Furthermore, we incorporate a world model that predicts future latent features conditioned on ego actions, improving feature consistency and mitigating domain-induced biases. We evaluate our approach on three
benchmarks, nuScenes, NAVSIM, and the Waymo end-to-end
dataset, and show substantial improvements over existing methods across all domains. Our results demonstrate that a single unified model can be trained on heterogeneous
datasets while maintaining strong performance within each domain, highlighting a step toward scalable real-world deployment. We will make our code publicly available.
Read the full abstract ▾
All authors Hoonhee Cho, Giwon Lee, Jae-Young Kang, Hyemin Yang, Heejun Park, Kuk-Jin Yoon
Robotics
FlyMirage: A Fully Automated Generation Pipeline for Diverse and Scalable UAV Flight Data via Generative World Model
Jinhan Li, Xijie Huang, Weiqi Gai et al. (+9 more)
In the field of
Vision-
Language Navigation (VLN), aerial
datasets remain limited in their ability to combine scale, diversity, and realism, often relying on either costly real-world scenes or visually limited simulations. To address these challenges, we introduce FlyMirage, a highly scalable and fully automated data generation pipeline for aerial VLN. Our approach leverages large
language models (LLM) as an environment designer to promote scene diversity, paired with a generative world model that instantiates these designs into high-fidelity 3D Gaussian Splatting (3DGS) scenes. To substantially reduce human labor and ensure the feasibility of flight data, FlyMirage automates scene exploration and semantic information acquisition, and further integrates a dynamically feasible planner for uncrewed aerial vehicle (
UAV) trajectory generation. Utilizing this toolchain, we generate a large-scale, diverse, and photorealistic aerial VLN
dataset. We further fine-tune the widely used pi_0 model on this
dataset and demonstrate the improvement our data brings to the model's ability to navigate in completely unseen environments, both in simulation and in the real world. Project website: https://lij1500.github.io/flymirage_website/ .
Read the full abstract ▾
All authors Jinhan Li, Xijie Huang, Weiqi Gai, Zhaoqi Wang, Qiyi He, Yijin Wang, Mo Zhu, Zhiyang Liu, Fei Gao, Xin Zhou, Yuze Wu, Ziming Ding
Robotics
MCNav: Memory-Aware Dynamic Cognitive Map for Zero-shot Goal-oriented Navigation
Jingyu Li, Zhe Liu, Wenxiao Wu et al. (+1 more)
Navigating to instance-level targets in complex environments is a challenging problem. Many existing zero-shot methods achieve strong performance by modeling the entire environment and leveraging large
language models for scene understanding. However, such strategies primarily focus on exploring new regions while lacking a deeper exploitation of information from previously explored areas. Consequently, when targets are missed or misidentified within previously visited regions,
navigation failures occur frequently. To address these limitations, we propose MCNav, a
memory-aware
navigation framework with a dynamic cognitive map. This map stores efficiently queryable information about relevant objects in explored areas. Building on this
memory structure, MCNav introduces two
memory-aware exploration strategies: goal re-validation, which re-assesses previously seen objects to correct matching failures, and missed goal re-exploration, which estimates the likelihood that a target is present in an explored region from contextual cues. These strategies are further stabilized by a blacklist mechanism to prevent repeated errors and a double-check mechanism for high-confidence confirmation. We evaluate MCNav on the HM3Dv1 and HM3Dv2
datasets across three different tasks, where it achieves state-of-the-art performance, particularly on the instance-level goal
navigation task.
Read the full abstract ▾
All authors Jingyu Li, Zhe Liu, Wenxiao Wu, Li Zhang
Robotics
Implicit Action Chunking for Smooth Continuous Control
Bosun Liang, Shuo Pei, Zirui Chen et al. (+5 more)
Reinforcement learning often produces high-frequency oscillatory control signals that undermine the safety and stability required for physical deployment. Explicit action chunking addresses this by predicting fixed-horizon trajectories but scales the policy output dimension proportionally with the horizon length, leading to optimization difficulties and incompatibility with standard step-wise interaction. To overcome these challenges, this paper proposes Dual-Window Smoothing (DWS), an implicit action chunking framework for smooth continuous control. Unlike explicit methods, DWS enforces temporal coherence without expanding the action space. It uses a dual-window design: an execution window that ensures physical smoothness through deterministic modulation, and a value window that aligns temporal-difference targets over the horizon to correct critic bias caused by open-loop execution. DWS also includes a lightweight actor-side temporal regularizer based on first-order action differences to promote global continuity. This design effectively bridges the gap between temporal abstraction and reactive step-wise control. Experiments on
benchmarks including the DeepMind Control Suite and industrial energy management tasks show that DWS outperforms state-of-the-art (SOTA) baselines. In complex
vision-based autonomous driving tasks, DWS achieves smoother control, safer behavior with reduced jitter, and attains a 100% success rate.
Read the full abstract ▾
All authors Bosun Liang, Shuo Pei, Zirui Chen, Chuanzhi Fan, Chen Sun, Yuankai Wu, Huachun Tan, Yong Wang
Robotics
PAPO-VLA: Planning-Aware Policy Optimization for Vision-Language-Action Models
Peizheng Guo, Jingyao Wang, Changwen Zheng et al. (+1 more)
Vision-
Language-Action (VLA) models show promising ability in
language-guided robotic tasks. However, making VLA policies reliable remains challenging, because a manipulation task is completed through closed-loop interaction, where each action affects subsequent execution. To analyze this problem, we revisit VLA policy during execution and argue that a VLA policy acts both as a planner, which makes task-oriented decisions that change the direction of execution, and as an executor, which realizes these decisions through dense continuous actions. This view suggests that improving VLA reliability requires particular
attention to planning actions. Existing optimization methods can imitate actions or improve complete trajectories, but they usually do not explicitly identify planning actions or measure their importance for task success. To address this issue, we propose Planning-Aware Policy Optimization for VLA models (PAPO-VLA). PAPO-VLA first identifies planning actions by jointly considering action variation and trajectory outcome, then estimates their importance through causal sufficiency and causal necessity, and finally incorporates this importance into GRPO advantage estimation. In this way, more important planning actions receive stronger optimization emphasis, while the whole trajectory is still optimized by trajectory-level feedback. Experiments on multiple
benchmarks demonstrate the effectiveness of PAPO-VLA.
Read the full abstract ▾
All authors Peizheng Guo, Jingyao Wang, Changwen Zheng, Wenwen Qiang
Robotics
Learning-Accelerated Optimization-based Trajectory Planning for Cooperative Aerial-Ground Handover Missions
Jingshan Chen, Bochen Yu, Henrik Ebel et al. (+1 more)
This paper presents a learning-augmented trajectory planning framework for cooperative unmanned aerial vehicle (
UAV) and unmanned ground vehicle (UGV) handover missions. While centralized trajectory optimization ensures dynamic feasibility and task optimality, its high computational cost limits real-time applicability. We propose a neural surrogate planner utilizing decoupled encoder-decoder long short-term
memory (LSTM) networks to generate coordinated handover trajectory predictions from the task specifications. These predictions serve as informed warm starts for the downstream centralized optimizer, thereby accelerating convergence to dynamically feasible solutions.
Benchmark evaluations demonstrate that the learning-augmented planning framework achieves more than a threefold speedup and 100% optimization success rate compared to cold start optimization. The results indicate that combining data-driven inference with model-based refinement enables fast and reliable trajectory generation for heterogeneous multi-
robot systems.
Read the full abstract ▾
All authors Jingshan Chen, Bochen Yu, Henrik Ebel, Peter Eberhard
Robotics
SafeAlign-VLA: A Negative-Enhanced Safe Alignment Framework for Risk-Aware Autonomous Driving
Kefei Tian, Yuansheng Lian, Kai Yang et al. (+2 more)
End-to-end autonomous driving systems excel in common scenarios but struggle with safety-critical long-tail cases.
Vision-
Language-Action (VLA) models are promising due to their strong
reasoning capabilities. However, most VLA-based approaches rely on positive expert demonstrations, rarely exploiting negative samples, leading to insufficient understanding of risky behaviors and safety boundaries. To address this limitation, we propose SafeAlign-VLA, a unified negative-enhanced safe
alignment framework that incorporates negative data into supervised learning and reinforcement learning. First, we develop a counterfactual safety pairing paradigm to generate structured safety labels and counterfactual positive trajectories from risky scenarios via counterfactual
reasoning. Then, a two-stage training strategy is adopted: negative-enhanced supervised fine-tuning for failure feedback and trajectory correction, followed by anchor-based group relative policy optimization that uses positive and negative trajectories as contrastive anchors to steer sampling and penalize high-risk behaviors via group-relative advantages. Experiments on NAVSIM and DeepAccident validate the proposed framework. SafeAlign-VLA achieves 89.1 PDMS on the NAVSIM v1 testset, improving over the baseline without negative data by 1.3%. On DeepAccident, it reduces the collision rate to 3.36%, while achieving 84.2%
language accuracy and 85.8% risk prediction accuracy. These results demonstrate the effectiveness of the proposed negative-enhanced safe
alignment framework for safe and robust autonomous driving.
Read the full abstract ▾
All authors Kefei Tian, Yuansheng Lian, Kai Yang, Xiangdong Chen, Shen Li
Robotics
ARC-RL: A Reinforcement Learning Playground Inspired by ARC Raiders
Carlo Romeo and Andrew D. Bagdanov
Reinforcement learning for legged locomotion has matured into a stack of multi-component reward functions and physics-engine
benchmarks whose morphologies are uniformly derived from real commercial hardware. Game NPCs, however, are bound by stylistic constraints absent from sim-to-real robotics and routinely take the form of creatures with no real-
robot counterpart. We introduce ARC-RL, a suite of four MuJoCo continuous-control environments featuring robotic morphologies inspired by the bestiary of ARC Raiders: the 18-DoF tall hexapod Queen, the 12-DoF armoured hexapod Bastion, the 18-DoF compact hexapod Tick, and the 12-DoF quadruped Leaper. All four
robots share a unified observation template, action convention, simulation cadence, and a single closed-form multi-component reward function whose only per-morphology variation lives in a small set of weights and parameters. The reward fuses a velocity-tracking tent, a healthy survive bonus, a phase-locked gait-compliance bonus/cost pair, action regularisers, three safety penalties, and a posture anchor; no motion-capture data enters the reward at any point. We additionally provide hand-crafted Central Pattern Generator demonstrators per morphology, which serve both as fixed expert references and as sources of prior data for offline-to-online training. On this playground, we conduct a controlled empirical study comparing standard online
algorithms (SAC, SPEQ, SOPE-EO) and methods augmented with prior data (SACfD, SPEQ-O2O, SOPE), and characterise how each paradigm copes with the playground's morphological diversity and animation-style stylistic constraints. Source code is available at https://github.com/CarloRomeo427/ARC_RL.git.
Read the full abstract ▾
Robotics
CANINE: Coaching Visually Impaired Users for Interactive Navigation with a Robot Guide Dog
Cunjun Yu, Zishuo Wang, Anxing Xiao et al. (+2 more)
Robot guide dogs offer
navigation assistance that greatly expands the independent mobility of the visually impaired, but their effective use requires subtle human-
robot coordination that is difficult for users to learn from generic verbal instructions. To tackle this challenge, we present CANINE, an automated coaching system that trains users for interactive
navigation with a
robot guide dog, through personalized, adaptive verbal feedback. CANINE decomposes a complex coordination task into sub-skills and operates at two levels. At the high level, it decides what to train by tracking the learner's proficiency across sub-skills using knowledge tracing and prioritizing training on the weakest areas. At the low level, CANINE decides how to train each sub-skill by observing each human practice episode, using foundation models to infer the underlying causes of errors, and generating targeted verbal corrections adaptively. A controlled study with blindfolded participants, treated as a proxy population for quantitative evaluation, demonstrates that CANINE significantly improves both learning efficiency and final
navigation performance compared to generic verbal instructions. We further validate CANINE through a retention study and an exploratory case study. The retention study shows lasting skill improvement after two weeks. The case study confirms CANINE's effectiveness in training a visually impaired user, while revealing additional design considerations for real-world deployment. Both are well aligned with the findings of the controlled study. Project page: https://cunjunyu.github.io/project/canine/
Read the full abstract ▾
All authors Cunjun Yu, Zishuo Wang, Anxing Xiao, Linfeng Li, David Hsu
Robotics
Closed-Loop Hybrid Digital Twin Platform for Connected and Automated Vehicle Validation
Kanglong Quan, Zhebing Xia, Linfeng Jiang et al. (+4 more)
Comprehensive and efficient validation of connected and automated vehicles (CAVs) is critical prior to real-world deployment. While simulation-based testing offers scalability, existing approaches often lack seamless integration with real vehicles and field data, limiting their fidelity in capturing dynamic, real-world interactions. To bridge this gap, this paper proposes a novel real-time hybrid digital twin platform. Its core innovation lies in the tight coupling of a high-fidelity CARLA-SUMO co-simulation with a physical test site and vehicle via a low-latency Vehicle-to-Everything (V2X) communication link. A custom-developed middleware serves as the critical bridge, synchronizing a real CAV's kinematic state as a shadow vehicle in the simulation and translating virtual control commands into chassis-actuating Controller Area Network (CAN) messages for closed-loop control. Detailed implementation includes using photogrammetry for full-scale asset reconstruction and a cloud-edge collaborative architecture for scalable, multi-user operation. Experimental results demonstrate stable synchronization and effective closed-loop control with low latency, confirming the platform's practicality for multi-scenario CAV verification.
Read the full abstract ▾
All authors Kanglong Quan, Zhebing Xia, Linfeng Jiang, Hao Yu, Ziheng Qiao, Dapeng Dong, Dongyao Jia
Robotics · Cross-listed from cs.LG
Sampling-Based Safe Reinforcement Learning
Luca Vignola, Bruce D. Lee, Manish Prajapat et al. (+4 more)
Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL
agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL
algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.
Read the full abstract ▾
All authors Luca Vignola, Bruce D. Lee, Manish Prajapat, Manuel Wendl, Melanie Zeilinger, Andreas Krause, Yarden As
Robotics
Self-assembling Modular Aerial Robot for Versatile Aerial Tasks
Junichiro Sugihara, Masaki Kitagawa, Jinjie Li et al. (+4 more)
Multirotor aerial
robots excel at maneuvering in three-dimensional space, and recent advances enable nimble
navigation in cluttered and confined environments, especially for small airframes. By contrast, platforms built for high-altitude work tend to be larger to deliver high thrust for stable physical interaction with the environment. However, these conflicting design requirements create a long-standing trade-off between nimble
navigation and robust aerial manipulation. Here, we present LEGION units, which are reconfigurable modular aerial
robots capable of in-flight self-assembly for cooperative manipulation, drawing inspiration from the self-organized collectives formed by ants. Each unit retains nimble maneuverability while joint-equipped docking interfaces at both ends enable end-to-end self-assembly into a flying manipulator. We show that multiple units autonomously dock in flight; once latched, they maintain a zero-clearance interlock by controlling the contact force and torque, enabling reliable aggregation and articulated motion even outdoors. We further show that self-reconfigurability enables morphological switching between nimble individual flight and collective articulated manipulation, while realizing core in-flight manipulation primitives including pushing, pulling, rotating, grasping, and carrying. LEGION's self-organization enables aerial
robots, especially in swarms, to shift from passive observers to active participants in their environment, broadening the scope of aerial physical interaction.
Read the full abstract ▾
All authors Junichiro Sugihara, Masaki Kitagawa, Jinjie Li, Yunong Li, Takuzumi Nishio, Kei Okada, Moju Zhao
Robotics
Spiking Neural Network Control of a Flapping-Wing Robot on Resource-Constrained Hardware
Rim El Filali, Chenrui Feng, Chao Gao et al. (+1 more)
Flapping-Wing Micro Aerial Vehicles (FWMAVs) provide exceptional maneuverability and aerodynamic efficiency but pose significant challenges for onboard control due to nonlinear dynamics and stringent Size, Weight, and Power (SWaP) constraints, as exemplified by a butterfly-inspired
robot less than 30 gram. To this end, we present a hierarchical neuromorphic control framework that enables fully onboard, closed-loop flight on a widely available, resource-constrained ESP32 microcontroller with a unit cost of approximately $5. Specifically, our method deploys two lightweight Spiking Neural Networks (SNNs) onboard: one for state estimation from raw sensory feedback and another for control via modulation of a Central Pattern Generator (CPG) for wing actuation. Trained by imitation learning, the system achieves stable pitch and heading angle tracking during untethered real-world flight. Experimental results further reveal that the SNN-based controller reduces latency by 36% (1059us to 680us) and power by 18% (0.033W to 0.027W) for inference compared to the conventional Artificial Neural Network (ANN) baseline, demonstrating the viability of spike-based computation without specialized hardware. To the best of our knowledge, this work constitutes the first demonstration of fully onboard neuromorphic control for autonomous flight of a FWMAV, highlighting the potential of SNNs to enable energy-efficient autonomy under stringent SWaP constraints.
Read the full abstract ▾
All authors Rim El Filali, Chenrui Feng, Chao Gao, Weibin Gu
Robotics
Beyond Waypoints: Dual-Heatmap Grounding for Cross-Embodiment Semantic Navigation
Kaijie Yun and Yue Chen
Grounding open-ended semantic instructions into physically executable local goals is a fundamental challenge in human-
robot interaction. While existing
navigation frameworks often regress deterministic waypoints, this rigid formulation collapses spatial uncertainty and frequently targets non-traversable object centers, leading to severe execution failures. In this work, we focus on the practical setting of in-FOV semantic
navigation, where a
robot receives concise, interleaved
multimodal (text and image)
prompts. To bridge the gap between abstract semantic intent and physical reachability, we propose a unified
Vision-
Language framework that abandons single-point regression in favor of a Dual-Heatmap representation. Our framework predicts a
navigation affordance heatmap that captures continuous reachable regions, coupled with a facing heatmap for orientation constraints. These dense outputs inherently function as a differentiable semantic potential field, integrating seamlessly with downstream local planners. To support this paradigm, we build a fully automated, foundation-model-assisted synthetic data pipeline and establish a comprehensive simulation
benchmark. Extensive experiments demonstrate that our framework achieves state-of-the-art performance among comparable 8B baselines. Crucially, a feature-fusion study and simulation studies across diverse
robot embodiments (Jetbot, H1, Aliengo) reveal that explicit heatmap prediction drastically improves the Affordance Rate (AR). By placing targets reliably in executable free space, our framework effectively mitigates the brittleness of point regression, offering a transferable path toward safe cross-embodiment semantic
navigation.
Read the full abstract ▾
Robotics · Cross-listed from cs.CR
RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents
Doguhuan Yeke, Yanming Zhou, Leo Y. Lin et al. (+3 more)
Recent advances in
Vision-
Language Models (VLMs) facilitate a new class of
embodied AI systems, where these models are integrated into physical platforms, e.g.
robots and autonomous vehicles, to interpret visual scenes and execute natural
language commands in diverse environments. Previous research has introduced jailbreak attacks and defenses for
embodied AI. Their evaluations, however, rely on ad-hoc
datasets, limited metrics, and emphasize attack success while neglecting the trade-off between security and the ability to follow benign commands. Existing
benchmarks and evaluation frameworks either target traditional chat-based models or focus on non-
adversarial safety evaluation for
embodied AI; neither captures the
adversarial risks, inputs, consequences, and evaluation criteria necessary for jailbreak attacks in
embodied AI systems. In this paper, we address this gap with RoboJailBench, which consists of three core components. We establish a security taxonomy derived from ISO standards, regulatory rules, and documented incidents. This effort yields 18 categories of security violation consequences for
embodied AI. We introduce an intent contrast
dataset pipeline that augments existing
datasets with paired
adversarial and benign goals to measure both security and utility. Lastly, we provide an evolving repository with standardized metrics and a unified process for assessing and integrating new attacks and defenses. With this
benchmark, we construct a new taxonomy-balanced
dataset and augment five existing
datasets. We integrate four attacks and two defenses to evaluate their performance on leading
embodied VLMs. This
benchmark provides the first standardized evaluation framework for jailbreak attacks in
embodied AI and supports future research. We release our code,
datasets, and artifacts, and maintain a leaderboard at https://purseclab.github.io/
benchmark-for-robotics-security.
Read the full abstract ▾
All authors Doguhuan Yeke, Yanming Zhou, Leo Y. Lin, Hongyu Cai, Antonio Bianchi, Z. Berkay Celik
Robotics
ContextFlow: Hierarchical Task-State Alignment for Long-Horizon Embodied Agents
Shuhan Guo, Kun Zhang, Haifei Liu et al. (+4 more)
Long-horizon
embodied agents increasingly delegate
navigation, search, approach, and manipulation to specialist executors. As these executors become stronger, the main bottleneck shifts from local skill execution to maintaining a coherent task frontier across planning, monitoring,
memory, and execution. We study task-state misalignment, a task-level consistency failure in which the planner's active stage, runtime evidence, remembered context, and delegated executor no longer justify the same next-step decision. This failure can lead to unsupported handoffs, stage lock, executor-context mismatch, and unnecessary replanning. We propose ContextFlow, an inspectable
alignment framework that represents stages as explicit contracts, converts runtime observations into evidence packets, and applies scoped updates including continue, refine, transfer, promote, and repair. ContextFlow keeps specialist executors responsible for local closed-loop control while making task-frontier
alignment explicit and auditable. Experiments and demonstration traces on long-horizon
embodied tasks illustrate how evidence-grounded scoped updates diagnose and mitigate recurring task-state failures.
Read the full abstract ▾
All authors Shuhan Guo, Kun Zhang, Haifei Liu, Xingyu Gao, Yongqi Zhang, Yaqing Wang, Quanming Yao
Robotics
DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs
Yixiang Zhu, Yonghao Chen, Jiaxiang Zou et al. (+2 more)
Vision-
Language-Action (VLA) policies increasingly rely on asynchronous inference to hide model latency behind ongoing
robot motion. During inference, the
robot continues executing its current action chunk while the next chunk is generated from the observation available at inference start. By the time the new chunk is executed, the
robot or scene may have changed, so an action that fits the prediction-time state can be misaligned with the execution-time state. Training-time methods modify the policy input or supervised target based on the expected execution delay, but do not directly supervise how actions should change as the scene evolves between prediction and execution. Runtime repair methods improve chunk continuity or modify actions during deployment, but can still leave actions aligned with the prediction-time state rather than the execution-time state. Our key insight is that execution-time observations are unavailable when the online prediction is made but are available in recorded trajectories during offline training, making the prediction-execution mismatch observable during training. We propose DEFLECT, an offline post-training framework that turns this temporal contrast into preference supervision for a policy acting from stale inputs. Across Kinetix, LIBERO, and three real-
robot tasks, DEFLECT improves delay robustness across the evaluated VLA policies and settings. At the longest supported delay, it improves success by 4.2 percentage points over VLASH on Kinetix and by 22.7 points over naive asynchronous execution with GR00T N1.7 on LIBERO.
Read the full abstract ▾
All authors Yixiang Zhu, Yonghao Chen, Jiaxiang Zou, Yusong Hu, Xinyu Chen
Robotics · Cross-listed from cs.IT
Domain-Adaptive Communication-Rate Optimization for Sim-to-Real Humanoid-Robot Wireless XR Teleoperation
Caolu Xu, Zhiyong Chen, Meixia Tao et al. (+3 more)
Wireless extended reality (XR) teleoperation provides
embodied interaction capability for collecting
humanoid robot demonstrations, but the large-scale adoption is restricted by the overhead of high-frequency motion transmission. This paper develops a system framework that integrates sampling, transmission, interpolation, and reconstruction and formulates a communication-rate optimization that aims to minimize the communication energy while maintaining the reconstruction accuracy of
robot motion trajectories through dimension-wise sampling-rate control. Since acquiring real-time feedback from physical
robots is limited by hardware costs, it is necessary to solve the problem through simulator interaction with offline real-domain data correction. To guide sim-to-real adaptation, we provide a PAC-Bayes generalization characterization that reveals the effects of latent density-ratio estimation, finite-sample deviation, and encoder bias. Building on this analysis, we propose a proximal policy optimization (PPO) method with density-ratio weighting and trust-region regularization. Experiments on public
humanoid teleoperation
dataset show that the proposed method improves the tradeoff between reconstruction error and communication energy consumption under sim-to-real distribution shift. We further analyze the effectiveness of the proposed
algorithm across various wireless channels and dynamic motion trajectories.
Read the full abstract ▾
All authors Caolu Xu, Zhiyong Chen, Meixia Tao, Li Song, Feng Yang, Wenjun Zhang
Robotics
PRISM-SLAM: Probabilistic Ray-Grounded Inference for Scale-aware Metric SLAM
Eunsoo Im, Gyeonggwan Lee, Seunghwan Hong et al. (+1 more)
Monocular SLAM historically suffers from scale ambiguity and tracking failure in dynamic environments. While recent
vision foundation models (VFMs) provide remarkable zero-shot depth priors, naively integrating these deterministic predictions ignores predictive uncertainty and frame-to-frame scale inconsistencies. We propose PRISM-SLAM, a real-time framework that rigorously integrates VFM priors into a structured
Bayesian factor graph to achieve scale-aware, metric-consistent localization and mapping. Specifically, we introduce a Plücker Ray-Distance Factor to anchor
monocular observations in absolute space within a globally consistent metric coordinate system, mathematically resolving scale drift by making the metric scale Fisher-identifiable. To handle environmental dynamics, we derive an epistemic uncertainty proxy from temporal depth consistency and formulate a Dynamic Scene Uncertainty Gating (DSUG) mechanism. This soft-gating approach probabilistically down-weights dynamic distractors without incurring the heavy computational overhead associated with traditional semantic
segmentation masks. By employing a multi-process architecture that asynchronously processes VFM inference and
geometric tracking, PRISM-SLAM provides verified metric output at 30 FPS using solely RGB input, bridging the gap between foundation models and real-world robotic applications. Evaluated on the TUM RGB-D and 7-Scenes
benchmarks, PRISM-SLAM achieves a metric $SE(3)$ Absolute Trajectory Error (ATE) nearly identical to its oracle-aligned $Sim(3)$ error. This demonstrates that our system can produce deployment-ready metric trajectories by delivering robust metric SLAM solutions without any post-hoc scale correction. Project page: https://prismslam-cmd.github.io/prismslam_pr/
Read the full abstract ▾
All authors Eunsoo Im, Gyeonggwan Lee, Seunghwan Hong, Junghun Suh
Robotics
Bilateral Teleoperation with Compliant 6-DOF Pose-and-Force Sensing
Yue Feng, Weicheng Huang and I-Ming Chen
Existing bilateral teleoperation platforms still rely on costly rigid six-axis force/torque sensors, tightly coupled leader-follower hardware, and kilohertz control loops. We present a Cartesian bilateral framework built on the hardware-agnostic WinGs Operating Studio (WOS) middleware, in which a low-cost compliant 6-DOF pose-and-force sensing end-effector, Delta6, is mounted on both sides so that each manipulator behaves as an end-effector 6-DOF series elastic actuator (SEA). The leader runs a damping-only admittance loop with a 6-D biquad notch filter; the follower realizes a stiffness-damping impedance through a position-based outer loop with a PID wrench-to-pose mapping. Three time scales (hardware I/O, mid-rate impedance/admittance, low-rate teleoperation messages) are explicitly decoupled, enabling the same application to drive heterogeneous arms. On a Lite6/FR3 testbed at 150 Hz, the system tracks stably under delays up to $120\pm40$ ms and 1% packet loss, matches the prescribed virtual stiffness in contact, and shows a favorable cumulative energy signature in passivity-style tests.
Read the full abstract ▾
Robotics
Graph Neural Planning and Predictive Control for Multi-Robot Communication-Constrained Unlabeled Motion Planning
Manohari Goarin, Yang Zhou and Giuseppe Loianno
The multi-
robot unlabeled motion planning problem of concurrently assigning
robots to goals and generating safe trajectories is central in many collaborative tasks. Recent Graph Neural Network methods offer scalable decentralized solutions but rely on simplified dynamics and simulation environments, overlooking key challenges of real-world deployment such as dynamic feasibility and communication constraints. To address these gaps, we propose a hierarchical framework that combines a Graph
ATtention Planner (GATP) with a decentralized Nonlinear Model Predictive Controller (NMPC). GATP provides intermediate subgoals through multi-
robot cooperation, and the NMPC enforces safety under nonlinear dynamics and actuation constraints. We evaluate our framework in both simulation and real-world quadrotor experiments. Thanks to
attention mechanisms and minimal communication requirements, we demonstrate improved generalization to larger teams, robustness to communication delays up to 200 ms and practical feasibility with decentralized on-board inference.
Read the full abstract ▾