Robotics
Generalizable Robustness Testing of DNN-Based Robotic Navigation Systems via XAI-Guided Search
Khizra Sohail, Miren Illarramendi and Aitor Arrieta
**Context:** Deep Neural Networks (DNNs) increasingly control Cyber-Physical Systems (CPSs), yet small input perturbations can cause unsafe system-level behavior. Existing approaches often optimize perturbations for individual images and evaluate them only in simulation, limiting their generalizability and practical validity. **Objectives:** This work aims to generate robustness tests that remain effective across operational observations and to evaluate whether the resulting failures transfer from simulation to a physical
robot. **Methods:** We propose an explainability-guided multi-objective evolutionary approach that generates sparse perturbations over representative images selected through visual and behavioral clustering. Aggregated Integrated Gradients guide mutations toward influential image regions. We evaluate the approach on a DNN-controlled LeoRover in Gazebo, conduct an ablation study, and validate a stratified subset of perturbations on the physical
robot. **Results:** The approach achieved a median success rate of 70.0%, compared with 53.85% for unguided search, and increased median hypervolume from 0.65 to 0.73. Multi-image optimization improved the success rate from 50.0% to 57.5%, while XAI guidance further increased it to 70.0%. In the sim-to-real evaluation, simulation achieved 0.95 precision and 0.67 recall, and simulated and physical failure times showed a significant positive correlation of 0.617. **Conclusion:** Combining multi-image optimization with explainability-guided search improves robustness testing for DNN-controlled robotic systems. Simulation effectively identifies and prioritizes transferable failures, but physical validation remains necessary because some real-world failures are not reproduced in simulation.
Read the full abstract ▾
Robotics
Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving
Jörg Gamerdinger, Victor Schwarzenberger, Philipp Schmid et al. (+2 more)
Increasing safety is the primary objective of automated vehicles. Achieving this goal requires reliable safety metrics that incorporate safety-relevant factors such as object type, velocity, and criticality. A key capability of such metrics is the distinction between critical and non-critical objects, which is addressed through criticality or relevance estimation. Existing criticality metrics are typically designed for specific scenarios and primarily focus on vehicle-to-vehicle interactions. In this paper, we therefore propose a novel criticality metric tailored to vulnerable road users (VRUs), which require special consideration due to their less predictable motion behavior. Furthermore, to avoid the complexity introduced by scenario-specific metrics, we introduce a scenario-independent criticality prediction framework applicable to all traffic participant classes. The effectiveness of both the proposed VRU-centric criticality metric and the criticality prediction framework is evaluated using the DeepAccident
dataset, which contains a diverse set of safety-critical traffic scenarios. The proposed VRU-centric criticality metric improves pedestrian criticality classification performance by up to 50 %. In addition, the proposed criticality prediction framework outperforms state-of-the-art metrics by 275 %, achieving an F1-score of 0.96 and enabling scenario-independent criticality assessment across all object classes. These results demonstrate the strong potential of the proposed approaches to enhance criticality assessment for safety evaluation in automated driving systems.
Read the full abstract ▾
All authors Jörg Gamerdinger, Victor Schwarzenberger, Philipp Schmid, Sven Teufel, Oliver Bringmann
Robotics · Cross-listed from cs.AI
Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cycle
Yang Wang, Hongxuan Liu, Xinghui Xu et al. (+40 more)
Physical intelligence refers to intelligence systems that understand, reason about, and act in accordance with the physical world and its underlying laws, dynamics, and constraints. Unlike conventional AI systems, physical intelligence interacts continuously with uncertain physical environments, and its actions produce consequences that are physically irreversible. As existing trustworthy AI frameworks have been developed primarily for digital AI systems, they do not fully capture the distinctive challenges of Physical Intelligence, such as physical safety, cyber-physical security, and physical manufacturing process. To address this gap, we present a survey of trustworthy physical intelligence principles. First, we characterize the core capabilities and challenges of physical intelligence. Second, we examine the role of physics in AI. Third, we trace the end-to-end physical intelligence life cycle across five core stages and introduce Trustworthy Physical Intelligence Operationalization (T-PAIO). Fourth, we develop the Trustworthy Physical Intelligence (T-PAI) framework, a theoretical framework that organizes key trustworthiness principles and provides a foundation for governing trustworthy physical intelligence systems.
Read the full abstract ▾
All authors Yang Wang, Hongxuan Liu, Xinghui Xu, Arjun Menon, Xiaoran Cai, Yunyu He, Alex Tarvo, Jingzong Zhou, Mengzhong Ma, Xinpeng Wei, Yi Yu, Shaobo Wang, Cheng Peng, Aoran Jiao, Alexei Korolev, Yanyan Zhang, Kai Ye, Xinpeng Li, Chengquan Guo, Jingjing Fu, Nicholas Bai, Yongjun He, Junru Ren, Silei Ren, Mohamad Louai Shehab, Keshu Cai, Nathaniel Dennler, Traian Tus, Gaoyue Zhou, Abdullah Garra, Mason Nakamura, George Ortiz, Marius Urbanas, Lars Johannsmeier, Rohit Sharma, Suraj Deshmukh, Michael Spranger, Vinayak Gupta, Xu Chen, Ashis G. Banerjee, Yuxiang Feng, Yoshua Bengio, Peng Qi
Robotics
A Replay-Constrained Simulation Framework for Personalization of Powered Knee–Ankle Prosthesis Controllers
Duong Le, Ryan Posh, Shihao Cheng et al. (+2 more)
Personalization of impedance controllers for powered prosthetic legs is critical to accommodating individual gait biomechanics but remains challenging. Existing methods rely on time-intensive human-in-the-loop exploration and/or constrain optimization to low-dimensional, single-joint parameter subspaces. Sim-to-real transfer has enabled high-dimensional locomotion control for legged
robots, but in assistive device control the human partner remains un-modelable. We present a replay-constrained simulation framework: a MuJoCo-based simulator reproduces prosthetic knee-ankle dynamics while replaying recorded hip kinematics and feedback-based ground reaction forces from individual walking data, bypassing the need to model complex human neuromuscular control mechanisms. We demonstrate the framework with a deep reinforcement learning policy that personalizes phase-dependent stiffness, damping, and
equilibrium angle at both joints simultaneously, maximizing a biomimicry-based reward computed solely from onboard prosthesis measurements. Experiments with three participants with transfemoral amputation during level-ground walking at 0.8~m/s demonstrate strong simulation-to-hardware predictive validity (Pearson $r=0.96$–$0.997$). The best-performing policy on hardware was consistently predicted within the top five simulation policies for all participants. The learned controllers improved overall biomimicry rewards by 42–59% relative to the unpersonalized baseline. The framework supports scalable high-dimensional personalization of powered prosthetic legs and is amenable to extension to higher-dimensional controller parameterizations such as neural-network controllers.
Read the full abstract ▾
All authors Duong Le, Ryan Posh, Shihao Cheng, Maani Ghaffari, Robert D. Gregg
Robotics · Cross-listed from eess.IV
Learning-based Hierarchical Tracheal Anatomy Understanding from Sparse Surgical Demonstration Annotations for Ultrasound Robots
Hiu Ching Cheung, Wenchao Yue, Zhengran Han et al. (+4 more)
Tracheostomy requires precise localization of the tracheal incision site; however, conventional manual palpation is subjective and often unreliable, while ultrasound utility remains operator-dependent. This work presents a learning-based framework for hierarchical tracheal anatomy understanding, designed specifically for ultrasound-guided robotic systems. We propose a two-stage perception pipeline integrating a YOLOv8n localization backbone with a sparse,
prompt-optimized SAM2 decoder to achieve high-fidelity
segmentation from sparse
surgical annotations. Our hybrid training strategy, bridging curated laboratory data with unconstrained sequences, ensures clinical robustness. Experimental
benchmarks demonstrate that this decoupled architecture effectively balances generalization, precision, and efficiency. The YOLOv8n and SAM2 framework achieves a consistent Mean Dice Similarity Coefficient (DSC) of 0.777 across both controlled and generalized domains. This significantly outperforms U-Net baselines, which often suffer from anatomical fragmentation and performance degradation (Generalization DSC $\le$ 0.494). By constraining mask decoding to targeted, sparse regions of interest, our model achieves a throughput of 6.92 FPS, which is vital for closed-loop robotic teleoperation. This study confirms that a robust hierarchical understanding of tracheal anatomy can be derived by coupling lightweight localization with foundation-scale visual models. Our framework establishes a scalable foundation for standardized, autonomous
surgical assistance, effectively navigating the variability of real-world ultrasound to enhance the safety and precision of robotic-assisted tracheostomy.
Read the full abstract ▾
All authors Hiu Ching Cheung, Wenchao Yue, Zhengran Han, Mingcong Chen, Guanglin Cao, Hongbin Liu, Hongliang Ren
Robotics · Cross-listed from cs.CE
Multi-Domain Physics-Based MDO of Multirotor UAVs: A Deterministic Framework for Discrete COTS Sizing
Akshay Gupta Burela and P. B. Sujit
Multirotor Unmanned Aerial Vehicle (
UAV) design is governed by a tightly coupled system of non-linear equations spanning structural mechanics, electrochemistry, aerodynamics, and kinematics. Solved sequentially, a miscalibrated sub-model coefficient triggers a mass-compounding cascade failure–the Mass Snowball effect. This paper presents AeroEval, a physics-based, Multidisciplinary Design Optimization (MDO) engine that simultaneously resolves all subsystem couplings and maps continuous sizing optima to physically purchasable, commercial off-the-shelf (COTS) components. The MDO engine is validated under a strict calibrate/test protocol that eliminates circularity: three structural/packaging coefficients are fitted on a held-out cohort of 20 do-it-yourself (DIY) and legacy platforms, then frozen and evaluated blind on 19 modern commercial drone platforms spanning 377 g to 76 kg. On the commercial test cohort the engine predicts Maximum Takeoff Weight (MTOW) within 7.2% Mean Absolute Percentage Error (MAPE), with a mean complexity factor $k \approx 1.05 \pm 0.08$ and $\text{RMSE}_{\text{MTOW}} = 1.59$ kg; the DIY calibration cohort, characterized by high build variability, yields 26.1% MAPE. Battery mass is predicted within 7.9% on single-pack platforms, with a disclosed systematic underprediction on redundant multi-battery enterprise platforms. A 14-parameter sensitivity suite of >300 simulations quantifies partial derivatives of takeoff mass and flight time; forward velocity diverges beyond 25.5 m/s due to cubic parasite power growth. The path-dependent mass-shedding model for agricultural and delivery roles reduces structural frame mass by up to 40.6% and energy capacity by 33.7% relative to static baselines. Typical solver convergence requires 25-50 iterations, completing in under 50 ms on a standard desktop CPU.
Read the full abstract ▾
Robotics
Robot-Factored World Models via Robot Rendering
Byungjun Kim, Taeksoo Kim, Hyunsoo Cha et al. (+1 more)
Action-conditioned
video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into
robot motion by the
robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose
robot-factored world models, which move two
robot-specific factors outside the world model. First, action realization: each command is rolled through the
robot's own controller and kinematics into a deployment-available nominal trajectory, a middle signal that avoids both action-realization learning and future-state leakage. Second,
robot rendering: this nominal trajectory is rendered through the
robot URDF, factoring the
robot's geometry, kinematics, and appearance out of the model and into explicit rendered
robot geometry. To resolve depth ambiguity, we pair end-effector depth with scene depth, giving
geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered
robot geometry form a shared visual world-model interface that stays consistent across viewpoints and
robot embodiments, so the model sees the action only as visible
robot geometry and learns how objects respond to it. Our experiments show that the rendered interface outperforms vector-conditioned baselines and generalizes to unseen
robot embodiments at inference. We further demonstrate that our model generates
robot manipulation
videos from human demonstrations by retargeting and rendering the hand motion as
robot geometry.
Read the full abstract ▾
All authors Byungjun Kim, Taeksoo Kim, Hyunsoo Cha, Hanbyul Joo
Robotics · Cross-listed from cs.CV
SM4RT: Learning Structured Motion Geometry for 4D Reconstruction
Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo et al. (+3 more)
Geometry Foundation Models (GFMs) have substantially advanced
monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses
geometric structure: physical objects undergo a set of rigid-body transformations governed by SE(3), rather than unstructured point-wise displacements. Building on this insight, we propose SM4RT, a Structured Motion 4D Reconstruction
Transformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces Structure-of-Motion to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in SE(3). Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from
monocular RGB
video. SM4RT achieves strong motion reconstruction performance while preserving the
geometric structure of scene motion.
Read the full abstract ▾
All authors Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, Jiwen Lu
Robotics
ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
Yunao Huang, Shiyu Sang, Suting Ni et al. (+6 more)
Contact-rich
robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile
robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich
robot manipulation. ViTacWorld leverages public real tactile
datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given
robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for
robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
Read the full abstract ▾
All authors Yunao Huang, Shiyu Sang, Suting Ni, Haotao Lu, Shijie Wu, Rui Lv, Ziyang Guo, Ye Shi, Jingya Wang
Robotics
Plug, Play, and Comply: A Modular Framework for Online Variable Impedance with Arbitrarily Oriented Compliance Axes
Mihael Simonič and Xiaocong Li
The paper proposes a
robot-agnostic compliant-control framework that extends the ROS control ecosystem with standardized joint and Cartesian command interfaces. It addresses a key limitation of existing control software: no reusable infrastructure for implementing compliant-control
algorithms across different manipulators while preserving a common interface to higher-level applications. A plugin-based architecture separates controller infrastructure from control-law implementation. Generic wrappers use existing hardware abstractions to interface with different manipulators, while runtime-loaded plugins implement only the control law. Command interfaces support joint- and Cartesian-space references, stiffness and damping gains, nullspace targets, and feedforward terms, enabling variable impedance and diverse compliant-control formulations.
Robot kinematics and dynamics are computed from URDF models using Pinocchio. The architecture facilitates the development of compliant-control strategies and enables the same implementation to be deployed across platforms unchanged. The complete framework, including reference controllers, high-level task interfaces, and example configurations for various manipulators, is open-sourced. The reference Cartesian impedance controller supports task-dependent compliance by rotating translational and rotational stiffness and damping, allowing the principal compliance directions to be updated online according to local task geometry rather than remaining fixed in the
robot base or TCP frame. This is particularly important in contact-rich manipulation, where the desired directions of motion, constraints, and compliance directions may vary throughout task execution. Real-
robot experiments demonstrate task-dependent compliance in contact-rich manipulation, while simulations show portability across manipulators with distinct kinematic and dynamic characteristics.
Read the full abstract ▾
Robotics
Robot Learning to Communicate through Projected Visual Abstractions
Danyang Yan, Boyuan Wang, Jiaxun Liu et al. (+1 more)
Humans routinely communicate through abstractions of their bodies, including shadows, silhouettes, and reflections. Yet
robots remain largely confined to expressing themselves through their physical morphology. Enabling
robots to communicate through such projected visual abstractions requires
reasoning not only about bodily motion but also about how that motion is transformed into an external representation perceived by an observer. Among these abstractions, shadows provide a particularly compelling example because they emerge directly from the
robot's embodiment while remaining visually distinct from the body itself. Here, we present a robotic system capable of dynamic shadow expression using a 21-degree-of-freedom dexterous hand with compliant soft skin and a learned shadow self-model. The soft-skinned embodiment reduces light leakage to produce visually continuous silhouettes, while the differentiable self-model learns the mapping between hand configurations and projected shadow appearance through task-agnostic self-exploration. Given a target shadow image or
video, the
robot optimizes its hand configurations through gradient-based search over 1 the learned self-model and refines the solution through collision-aware simulation to obtain physically feasible motions. For dynamic shadow performance, we further introduce expressive-region objectives, temporal smoothness regularization, and keyframe-based optimization to preserve visually important motion cues while reducing optimization complexity. We demonstrate robotic shadow expression across sign-
language gestures, hand-shadow puppetry, and animal motion imitation in both simulation and physical experiments. These results establish a framework for enabling
robots to manipulate projected visual abstractions of themselves for communication and visual storytelling.
Read the full abstract ▾
All authors Danyang Yan, Boyuan Wang, Jiaxun Liu, Boyuan Chen
Robotics
Conformal Constraint Tightening for Chance-Constrained Motion Planning with Unknown Dynamics
Shubham Natraj, Bruno Sinopoli and Yiannis Kantaros
Motion planning
algorithms compute control sequences that drive autonomous
robots to goal regions while avoiding unsafe states. Existing methods, from sampling-based planning to deep reinforcement learning, typically provide task-completion guarantees only with respect to a nominal model or simulator, which may be invalidated when the true dynamics are unknown or difficult to model accurately. This letter addresses this limitation for systems with unknown dynamics and an available approximate nominal model, contributing a planner-agnostic constraint-tightening procedure that equips existing planners with a probabilistic task-completion guarantee on the true system. We leverage conformal prediction to provide a probabilistic bound on the nominal-to-true trajectory deviation over a distribution of planning problems. We tighten the planning constraints using that bound, and show that solving the tightened problem under the nominal model is a sufficient condition for solving the original problem on the true system with a prescribed probability. We validate the theoretical guarantees empirically and demonstrate substantially improved task completion relative to nominal-model planning.
Read the full abstract ▾
Robotics
Teachy Mini: Development and Preliminary Evaluation of a Knowledge-Based Generative Social Robot for Higher Education
Stephan Vonschallen, Karim Kaufmann, Dominique Oberle et al. (+2 more)
Generative social
robots (GSRs) powered by large
language models offer new possibilities for personalized tutoring in higher education, but also introduce risks related to misinformation, missing transparency, or reinforcing incorrect student responses. Prior work identified knowledge-based design (KBD) requirements that define the informational prerequisites for GSRs to manifest responsible and effective tutoring behavior in higher education. In this paper, we operationalized selected KBD requirements in the Reachy Mini
robot platform through system prompting,
retrieval-augmented generation, and stateful
prompt orchestration. As a result, we present Teachy Mini, a GSR tutoring system that was developed using KBD. To test the system, we conducted a preliminary evaluation study. Participants (N = 24) completed a
robot-guided learning session about research methodologies. They learned either with Teachy Mini or with a control version that did not follow KBD principles. Teachy Mini was perceived as significantly more aligned with responsible tutoring behavior than the control
robot. Moreover, a manipulation check illustrated that Teachy Mini used personalization, slide-grounded explanations, Socratic questioning, affective support, and learner-anchored feedback more consistently than the control
robot. No significant between-condition differences were found in system acceptance, intrinsic motivation, or learning effectiveness, although exploratory analyses suggested a positive effect of KBD on objective learning gains when accounting for learner preferences. Overall, the study offered an initial implementation and preliminary evaluation of KBD for GSR tutoring, indicating that KBD can shape responsible
robot behavior and potentially increase learning effectiveness in
robot-supported learning.
Read the full abstract ▾
All authors Stephan Vonschallen, Karim Kaufmann, Dominique Oberle, Friederike Eyssel, Theresa Schmiedel
Robotics · Cross-listed from cs.CV
Geometric 2D Scene Graph Generation
Christoph Jahn, Urs Waldmann and Bastian Goldluecke
In production processes for consumer products, assembly instructions are essential not only for planning but also for executing the production process. Likewise in robotics, it is crucial for an assembly
robot to understand how components fit together and can be assembled. To facilitate these tasks, we contribute a method for constructing scene graphs to represent and characterize assembly relationships between components. Our approach does not rely on semantic data and is capable of handling a very small
dataset. To realize this, the output of a Faster R-CNN model is used to create
geometric representations, which are then processed by a
transformer architecture to generate an adjacency
matrix. This
matrix serves as input to a Siamese network that uses message passing based on an attentional graph convolutional network (aGCN) architecture to characterize the connections between the components. We validate our method on a study
dataset of toy model components which can be assembled into transportation vehicles.
Read the full abstract ▾
Robotics
A Monolithic Hand with Asymmetric Origami Bending and Dual-chamber Actuators
Nan Huang, Yuming Zhu, Zicong Zhang et al. (+5 more)
The passive adaptability inherent in soft robotic hands affords them advantages in applications that require safe and compliant interaction. However, existing soft robotic hands often struggle to simultaneously achieve adequate output performance and easy manufacturing due to their complicated structures. In this paper, we introduce the asymmetric origami bending (AOB) pattern for generating bending motion and the asymmetric dual-chamber (ADC) design for obtaining multifunction capability. The AOB single (AOB-S) chamber and AOB dual-chamber (AOB-D) units are designed and constitute the finger and palm actuators of the proposed Origami-inspired SOft Robotic (OSOR) hand. The OSOR hand achieves bio-inspired fingers-palm motions and adequate output performance within a monolithic structure that significantly simplifies the manufacturing process. By defining the asymmetric ratio to characterize the
geometric asymmetry of the unit, the analytical models of the AOB and ADC structures are proposed. The Finite Element Analysis tool for the design of AOB actuators is obtained by
geometric analysis. The asymmetric origami design grants the integrated manufacturing of the OSOR hand through a Selective
Laser Sintering printing process with a single thermoplastic polyurethane material. The model and simulations are validated by experimental results. Experiments show the finger and palm maximum bending motion range of 203{\deg} and 40{\deg}, respectively, with output forces of 6.3 N and 16 N. The OSOR hand is capable of pinching a piece of tissue, stably grasping water bottles with two fingers, palm-only grasping, and completing the power grasps in the taxonomy of manufacturing grasps. The compactness, performance, and easy manufacturing of the proposed hand benefit the development of the soft robotic hand with new possibilities.
Read the full abstract ▾
All authors Nan Huang, Yuming Zhu, Zicong Zhang, Jianhui Liu, Xiaohuang Liu, Dihan Liu, Jiansheng Dai, Sicong Liu
Robotics
Design and Human Evaluation of Tactile Withdrawal Reflexes for a Skin-Covered Robot Arm
Laura Babayeva, Lukas Rustler and Matej Hoffmann
Nociception is a protective biological mechanism that links harmful stimulation to a reaction. This paper investigates artificial nociception for a robotic arm with whole-body tactile sensing. We present a complete pipeline that maps pressure changes from sensitive skin on a
robot manipulator to bio-inspired withdrawal motions. The system first converts skin pressure into a scalar pain gain using a nonlinear continuous model. We compare three reflexes: (i) uniform reflex moves four
robot joints by a fixed amount, whereby the withdrawal is approximated by a movement of the arm "toward the base", independent of where the
robot was touched; (ii) biologically motivated location-dependent joint-space withdrawal derived from human withdrawal reflex characteristics; (iii) Cartesian space withdrawal along the surface normal of the contacted skin pad. All behaviors are integrated in a reflex controller that interrupts the task, executes the withdrawal, and returns to a pre-contact pose. A user study with 15 participants compared the strategies using Godspeed questionnaire subscales, custom perceived-naturalness and safety items, forced-choice comparisons, and qualitative feedback. Interestingly, participants rated more highly the uniform reflex behavior over one or both competitors on the anthropomorphism, animacy, and likeability Godspeed subscales and on the Naturalness and Realism custom scale. When asked to compare the conditions, the uniform reflex was scored best in "felt safest", "most human-like", and "most natural". This suggests that predictability of the
robot behavior is key for user acceptance. The Cartesian reflex was judged the most appropriate reaction to touch. The bio-inspired reflex did not lead any evaluated measure. This may be partly attributed to the embodiment gap between the
robot arm and human arm and participants having different expectations from a
robot manipulator.
Read the full abstract ▾
Robotics
Ali Salmasi, Xianjia Yu and Tomi Westerlund
Foundation-model-based
vision-
language navigation (VLN) has advanced autonomous
robot navigation by enabling
robots to interpret natural-
language instructions, identify semantic goals, and follow user-specified behavioral rules. However, existing VLN systems rely heavily on cloud-hosted foundation models for
language understanding and semantic grounding, limiting their applicability where network connectivity is unavailable and reliable metric goal localization is required. Although recent small
language models (SLMs) enable fully onboard inference, their suitability for
navigation instruction decomposition has not been systematically evaluated. This paper makes three contributions toward fully onboard VLN for outdoor environments. First, we present the first systematic
benchmark of 17 edge-deployable SLMs against 4 online APIs for robotic
navigation instruction decomposition, evaluating accuracy and latency on human-annotated instructions across three computing platforms and providing practical guidance for selecting onboard
language models. Second, we propose a lightweight hybrid semantic-
geometric goal localization framework that combines open-vocabulary object detection, prompted
segmentation, and
LiDAR geometry to estimate metric goals, while maintaining visual bearing guidance when reliable
geometric observations are unavailable. Third, we integrate these advances into Edge-BehAV, a fully onboard extension of the BehAV architecture that enables cloud-independent behavior-guided
navigation. Experimental results show that the best offline SLM matches the instruction decomposition performance of the strongest cloud API while running approximately 9x faster and without network connectivity. The proposed goal localization framework reduces mean goal-distance error from 2.05 m to 0.20 m at lower computational cost, and the complete system succeeds in 31 of 32 closed-loop outdoor trials.
Read the full abstract ▾
Robotics
Safe Learning Predictive Control for Ego-World Robotic Systems
Davide Valenti and Giuseppe Notarstefano
Safe autonomous
navigation in shared environments requires the ability to anticipate and react to the latent behaviors of surrounding
robots. In this paper, we propose SOWL-MPC, a safe learning-based predictive control strategy for a novel scenario, which we name ego-world robotic framework. In this setting, the control policy of the world
robot is unknown and the ego exploits data to learn it and perform safe maneuvers. The proposed architecture combines an online learning mechanism based on Sparse Variational Gaussian Processes (SVGPs) with a receding-horizon control scheme. Relying solely on noisy state measurements, our approach infers a posterior distribution over the latent world policy, which is updated on streaming data via Online Variational Conditioning (OVC). The learned policy is propagated through the nonlinear world dynamics using an approximate moment propagation scheme, and fed to an uncertainty-aware Model Predictive Control (MPC), thus enabling safe maneuvering of the ego
robot. The real-time feasibility and safety guarantees of SOWL-MPC are demonstrated through extensive Monte Carlo virtual experiments in ROS 2, and validated on real-world robotic hardware in an indoor arena.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
JustDepth: Real-Time Radar-Camera Depth Estimation with Single-Scan LiDAR Supervision
Wooyung Yun, Dongwook Kim and Soomok Lee
Accurate yet low-latency depth is essential for radar-camera perception in autonomous systems. Cameras provide rich appearance but lack metric scale, whereas automotive radar offers metric range but is sparse and noisy. Many pipelines are multi-stage or depend on auxiliary annotations, increasing latency and limiting portability. We introduce JustDepth, a single-stage radar-camera depth
estimator trained only with radar, camera, and single-scan
LiDAR. All radar returns are aggregated into a fixed-width 1D representation, decoupling runtime from point count. A Height Fusion Block fuses modalities, a lightweight GNN propagates depth globally, and a training-only confidence decoder stabilizes learning with zero test-time cost. We mitigate stripe artifacts via simple augmentations and quantify them using the Vertical-Horizontal Gradient Ratio (VHGR). On nuScenes, compared to recent state-of-the-art methods, JustDepth maintains accuracy while reducing inference time by 39.7x and stripe artifacts by 66% as measured by VHGR.
Read the full abstract ▾
Robotics
Learning Spatiotemporal Decision Priors for Efficient Path Planning under Partial Observability
Yi Liu, Hongda Zhang, Leyao Zou et al. (+7 more)
Path planning under partial observability remains challenging because an
agent must make long-horizon
navigation decisions from only locally bounded observations. Nevertheless, historical trajectories contain reusable experience-guided directional preferences. Classical planners, however, typically solve each instance from scratch and lack an explicit mechanism to exploit such transferable decision knowledge, often leading to redundant node expansions and locally myopic search behaviors. Motivated by this limitation, this paper proposes ImiPath, a prior-guided learning framework that distills reusable spatiotemporal decision priors from demonstration trajectories and uses them as experience-informed directional guidance to bias planners toward reliable and promising search directions under partial observability. Specifically, ImiPath first constructs a local spatiotemporal observation representation, which encodes the spatial information of the local environment and the temporal information of historical trajectories. The SpatioTemporal-
Attention Policy Network (STAPNet) then transforms this representation into dicision priors. These priors are further incorporated into heterogeneous planners as directional guidance, biasing the search toward locally promising regions. Extensive experiments demonstrate that ImiPath achieves competitive path quality and improves search efficiency by reducing redundant node expansions under local observability. Additional physical experiments on a
magnetic microrobot platform further validate the adaptability and practical deployment potential of the proposed framework.
Read the full abstract ▾
All authors Yi Liu, Hongda Zhang, Leyao Zou, Chunlei Meng, Ziqing Zhou, Yuning Chen, Zhuo Zou, Lida Xu, Zhongxue Gan, Chun Ouyang
Robotics
Flight-Ready LiDAR-Inertial Odometry for Embedded Drone Platforms
Alvaro J. Gaona, David Perez-Saura, Francisco J. Anguita et al. (+1 more)
Open-source
LiDAR-inertial odometry (LIO) systems have achieved remarkable
benchmark accuracy, yet current state-of-the-art implementations are primarily optimized for evaluation performance rather than the requirements of real-time closed-loop aerial control. When deployed onboard
UAVs, this can introduce limitations that degrade flight performance. In this work, we identify five architectural deficiencies in a representative tightly coupled IESKF-based LIO implementation: odometry publishing tied to the
LiDAR rate (10 Hz instead of the IMU's 200 Hz), missing velocity outputs, execution bottlenecks that block IMU processing, mutex contention, and synchronization race conditions. We introduce corresponding modifications including IMU-rate forward propagation, direct body-frame velocity publishing, SLERP-based smoothing, dual-executor isolation, and explicit synchronization protection. The resulting system increases odometry output from ~10 Hz to a stable 200 Hz, provides a complete Twist state at every IMU sample, and preserves continuity during transient
LiDAR loss. Experiments on a Livox Mid-360 / Pixhawk 4 Mini autonomous
UAV with motion-capture ground truth validate the approach. Since the underlying
estimator (IESKF + ikd-Tree) remains unchanged, the proposed improvements can be directly applied to FAST-LIO2-derived implementations.
Read the full abstract ▾
All authors Alvaro J. Gaona, David Perez-Saura, Francisco J. Anguita, Pascual Campoy
Robotics
DB-VIO: Dual-Branch Visual Inertial Odometry with Enhanced Visual-Inertial Representation
Ziyu Wan and Lin Zhao
Visual inertial odometry (VIO) is essential for accurate 6-DoF motion estimation in mobile robotic systems. Recent learning-based VIO methods have shown promising progress, but they often rely on unified visual–inertial representations and a single temporal model for full-pose estimation, limiting their ability to capture the heterogeneous dynamics of rotation and translation. Moreover,
monocular visual features often lack explicit
geometric structure, while raw inertial encoding leaves the underlying rotational kinematics implicit, weakening the rotation-related cues in IMU features. To address these issues, we propose DB-VIO, a dual-branch visual inertial odometry framework with enhanced visual–inertial representation. DB-VIO incorporates depth cues to improve
monocular visual perception, injects an explicit integrated-attitude prior to strengthen rotation-aware inertial representation, and decouples pose estimation into dedicated rotational and translational branches for motion-specific temporal modeling. Experiments on autonomous driving and aerial
robot benchmarks show that DB-VIO achieves state-of-the-art performance, improving the corresponding baselines by 20% on KITTI and 33% on EuRoC. Notably, under the more agile motion patterns of EuRoC, DB-VIO improves the rotational metric by 65.7% over prior methods. These results demonstrate the effectiveness and generalization of DB-VIO across different platforms and motion scenarios.
Read the full abstract ▾
Robotics
One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments
Jan Ole von Hartz, Abhinav Valada and Joschka Boedecker
Multi-stream
robot manipulation policies achieve unparalleled sample efficiency and generalization by modeling actions relative to environmental reference frames. However, existing approaches typically assume these frames to be strictly exogenous. This causal assumption collapses in dynamic settings, such as when a single
robot arm manipulates a moving object or when two arms coordinate, where each arm effectively becomes part of the dynamic environment of the other. We propose DynaMAC, a lightweight, policy-agnostic framework that resolves this causal limitation while preserving the sample efficiency, computational speed, and flexibility of multi-stream policies, DynaMAC treats the opposite arm as a dynamic task parameter, thereby providing a unified formulation for dynamic manipulation and bimanual coordination without requiring an explicit leader-follower relationship. To rigorously evaluate these capabilities, we introduce DynaBench, a novel
benchmark for
robot manipulation in dynamic environments. Across both dynamic environments and bimanual manipulation tasks, DynaMAC outperforms leading probabilistic and generative baselines by over 35 percentage points while requiring 20 times fewer samples. Crucially, DynaMAC generalizes zero-shot from static demonstrations to dynamic environments, substantially simplifying data collection and establishing an elegant bridge toward human-
robot collaboration.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Constraint-Driven Synthesis of Hyper Petri Nets
Maksym Figat and Alessandro Pinto
This paper addresses the modeling and synthesis of constrained robotic system behaviors using Petri nets (PNs). It investigates how to construct models in which all observable system states satisfy given logical constraints while remaining consistent with executable transition semantics. To answer this, we introduce the Hyper Petri Net (HyPN) approach, which synthesizes Petri nets from Boolean specifications while explicitly distinguishing between observable markings and underlying Petri net execution. The proposed method introduces an explicit execution semantics over observable states, induced by admissible (atomic) firing sequences, ensuring by construction that all observable markings satisfy the constraints and revealing a fundamental mismatch between logical feasibility and executable behavior. This is demonstrated in two scenarios inspired by a lunar rover system. These results are particularly relevant for the design of robotic and autonomous systems, as they provide a structured way to ensure correct system configurations while explicitly accounting for execution constraints. The proposed framework further suggests new research directions in execution abstraction, admissible transition systems, and policy selection for navigating between constraint-satisfying states.
Read the full abstract ▾
Robotics
Impedance Control of Ship-Borne Manipulators via Optimization-based Task-Space Inverse Dynamics
Lingxiao Meng, Bi-Ke Zhu, Xuheng Gao et al. (+5 more)
Ship-borne manipulators operating in maritime environments are subject to
stochastic wave-induced base motions that introduce kinematic disturbances and dynamic coupling, degrading trajectory tracking accuracy and complicating safe, contact-rich manipulation. This paper proposes a torque-level optimization-based control framework that integrates high-precision trajectory tracking with task-space impedance for ship-borne manipulators. The controller is formulated using task-space inverse dynamics (TSID) and solved via quadratic programming to explicitly compensate for the dynamic coupling introduced by base motion. To enable accurate feedforward compensation, an error-state Kalman filter (ESKF) is developed to estimate the base state by fusing inertial measurements with end-effector pose feedback. The framework is validated in simulation and real-world experiments using a 7-DOF manipulator mounted on a 6-DOF Stewart platform. The proposed method reduces real-world end-effector position tracking error by over 25.7% compared with the best baseline. Furthermore, the controller enables dynamic peg-in-hole insertion with 1~mm clearance under base motion, increasing the success rate while reducing average contact forces by 45%, demonstrating precise and compliant manipulation in contact-rich environments.
Read the full abstract ▾
All authors Lingxiao Meng, Bi-Ke Zhu, Xuheng Gao, Zhe Zhang, Jiankun Yang, Jiankun Wang, Haibo Lu, Max Q.-H. Meng
Robotics
Embodying Multi-Hand Manipulation Policies by Searching the Assignment and Null Spaces
Yorai Shaoul, Jiaoyang Li and Maxim Likhachev
Learned manipulation policies increasingly predict motions for abstract "hands" and are attractive in practice because they rely on easily collected demonstrations and transfer across
robot platforms. Executing these trajectories on multi-arm
robots, however, is not trivial. Multi-hand policy outputs must be assigned to physical arms, each arm must realize a configuration-space motion that tracks its prescribed end-effector trajectory, and all arms must respect kinematic limits and avoid collisions. In the absence of
algorithms that directly address this problem, practitioners typically extend single-arm inverse-kinematics (IK) pipelines in an ad hoc way, with no guarantees of feasibility or safety. In this work, we close this execution gap with a search-based framework that is theoretically complete for grounding policy-generated multi-hand trajectories onto physical multi-arm systems. Building on Conflict-Based Search, our method explicitly searches over both the discrete assignment of trajectories to arms and the continuous Jacobian null spaces of redundant manipulators, using redundancy to avoid inter-arm collisions while tracking the prescribed motions. This unified treatment of assignment and null-space motion yields a practically efficient planner that safely realizes coordinated manipulation-policy outputs on multi-arm
robots. See omcbsa.github.io for more.
Read the full abstract ▾
Robotics · Cross-listed from cs.AI
Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents
Suman Navaratnarajah, Taehyoung Kim, Jona Ruthardt et al. (+5 more)
Multimodal Large
Language Models (MLLMs) are emerging as core
reasoning modules for
embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon
embodied tasks from a single high-level instruction. We introduce MissionBench, a
benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families.
Agents must autonomously plan, navigate, and report outcomes using only
egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step
embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot
embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive
reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for
embodied AI.
Read the full abstract ▾
All authors Suman Navaratnarajah, Taehyoung Kim, Jona Ruthardt, Ishaan Bhimwal, Ryousuke Yamada, Yannik Blei, Wolfram Burgard, Yuki M Asano
Robotics
Mag4D-SLAM Dataset: A Repeated-Traversal Multi-Modal 4D Geomagnetic Dataset for Localization and Mapping
Bibhutibhusan Nayak, Hyoseok Ju and Giseop Kim
Geomagnetic sensing offers an infrastructure-free, absolute orientation reference that is robust to GNSS denial and visual degradation, yet no large-scale outdoor robotics
dataset supports its systematic study in SLAM. Existing
magnetic datasets are confined to small-scale indoor environments and lack the synchronized multi-modal sensing, repeated-traversal structure, and high-precision 6-DoF ground truth required for geomagnetic SLAM research. We present Mag4D-SLAM, the first large-scale outdoor geomagnetic SLAM
dataset. It comprises 14 sequences totaling over 18 km of synchronized
LiDAR, camera, IMU, tri-axis magnetometer, and GNSS measurements with SE(3) ground-truth poses, collected along structured campus trajectories under paired day/night conditions in both forward and reverse directions. Through repeated-traversal experiments, we analyze three core properties:
magnetic field repeatability across different recording sessions (daytime and nighttime), drift-free global heading estimation, and location-discriminative
magnetic signatures for cross-session place recognition. Mag4D-SLAM is designed to support research on yaw drift mitigation,
magnetic loop closure, and long-term localization and to open new research questions on how geomagnetic sensing can complement visual and
LiDAR modalities or provide a fallback cue under illumination changes, structural repetition, and GNSS-denied long-term operation.
Read the full abstract ▾