Robotics · Cross-listed from cs.CV
PIXIE: A Zero-Shot texture-invariant 6D pose estimation framework for unseen objects with assembly defects
Leon Jungemeyer, Alejandro Magaña, Gautham Mohan et al. (+2 more)
6D pose estimation remains a key challenge in robotics and computer
vision, particularly in industrial environments. The deployment of currently available data-driven methods is often limited by resource-intensive data pipelines, reliance on textured 3D models, and sensitivity to
geometric deviations caused by damages or assembly defects. We present PIXIE, a zero-shot framework that estimates the 6D pose of an object from an RGB image using only an untextured 3D model. Synthetic depth and normal maps are rendered from sampled reference viewpoints and matched to the query image via a pretrained cross-modality feature matcher. Matched keypoints are back-projected to obtain 2D–3D correspondences for PnP-based pose estimation. Relying exclusively on geometry makes the method inherently robust to lighting and texture variation, while correspondence filtering handles
geometric deviations between the model and physical object. We evaluate on widely-used public
benchmarks, reporting state-of-the-art results on texture-less objects without object-specific training, and introduce a novel
dataset with assembly defects, texture variations, and occlusion to demonstrate real-world applicability.
Read the full abstract ▾
All authors Leon Jungemeyer, Alejandro Magaña, Gautham Mohan, Matthias Karl, Daniel Werdehausen
Robotics · Cross-listed from cs.CV
DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Prediction
Jehun Kang, Jungha Wang, Youngjun Hwang et al. (+1 more)
Multi-Task Learning (MTL) in robotics perception systems supports comprehensive 3D spatial scene understanding by integrating semantic
segmentation and depth estimation. While
Vision Foundation Models (VFMs) are increasingly adopted as robust feature encoders, existing decoding strategies present a critical bottleneck. To address this, we propose DPNeXt, a streamlined multi-scale feature fusion decoder and efficient alternative to the standard Dense Prediction
Transformer (DPT). DPNeXt uses dual depthwise separable inverted bottlenecks to improve frozen VFM utilization through fusion-centric decoding and independent task modularization. To further mitigate negative inductive transfer between tasks, we introduce the Multi-Task Boundary Guidance (MTBG) strategy. Unlike prior boundary-aware methods that add fusion modules or gating, MTBG applies symmetric boundary-focused supervision to encourage
geometric consistency without extra annotation or inference cost. Experiments on Cityscapes show that DPNeXt-S outperforms prior state-of-the-art (SOTA) MTL models, while DPNeXt-B further improves the overall performance and achieves the best results among the compared methods. On NYUv2, DPNeXt-B also achieves the best semantic
segmentation and depth estimation results among the compared methods while requiring substantially fewer trainable parameters than prior large-scale MTL models. Compared with the standard DPT, DPNeXt-S reduces trainable parameters by 78.6% and achieves the fastest inference speed among the compared models on resource-constrained laptop hardware. The source code, model checkpoints, and a demo
video will be made available at https://github.com/kangjehun/DPNeXt.
Read the full abstract ▾
All authors Jehun Kang, Jungha Wang, Youngjun Hwang, David Hyunchul Shim
Robotics
A Morphing-Designed Hexarotor Prototype combining Practical Resilience and Efficiency
Murat Bronz, Mahmoud Hamandi, Elgiz Baskaya et al. (+2 more)
This work demonstrates experimentally the existence of a hexarotor prototype, termed Opti-Hexa, that simultaneously achieves practical resilience to single-propeller failures and energy efficiency comparable to a standard
Star-shaped prototype with the same size, weight, hardware and software. Leveraging a novel open-source morphing platform, we investigate the trade-offs across a continuous range of geometries by varying the angles between adjacent propellers. We study practical efficiency through a data-fitted empirical power model and evaluate practical resilience by comparing the position accuracy and rotational kinetic energy during failure to those observed under nominal hovering conditions. Our experiments confirm the existence of a
geometric viability region for this specific morphing platform, where resilience is ensured without the aerodynamic efficiency losses typically associated with practically resilient designs found in the state of the art. The complete hardware and software of the morphing platform are released to support further research.
Read the full abstract ▾
All authors Murat Bronz, Mahmoud Hamandi, Elgiz Baskaya, Chiara Gabellieri, Antonio Franchi
Robotics
Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation
Oliver Hausdörfer, Linus Schwarz, Gabor Marko et al. (+7 more)
Learned policies trained end-to-end on large
datasets often remain brittle in high-precision contact-rich tasks. Further, they typically require large amounts of manually collected data and still struggle to generalize to scenes outside of the training scenarios. Classical motion planning, in turn, generalizes well but lacks the precision and contact awareness for tight-tolerance tasks. To reliably solve challenging contact-rich tasks, we propose to leverage
robot data and learning where it matters in a modular approach. First, we introduce an autonomous data collection scheme for the real world that collects dense data locally in the contact-rich segment of the task, eliminating reliance on a teleoperator's skill. Second, we show that using offline RL, the collected data can be translated into policies with high success rates. We then integrate our local policies with traditional motion planning to solve long-horizon assembly tasks. Across four real-world tasks, using only 2-2.5 h of autonomous data collection, we achieve an average success rate of 96% for long-horizon tasks, compared to 55% for a planning-only baseline and 37% for the strongest end-to-end baseline. Our local policies also reduce applied forces by 49%. Notably, performance remains at 90% for scene-level out-of-distribution scenarios, while end-to-end approaches struggle. In total, we run over 1500 real-world evaluation rollouts. Our results pave the way for targeted data collection in contact-rich tasks and for high success rates in precision applications.
Videos and qualitative results are available at https://learnsyslab.github.io/data_and_learning_where_it_matters/
Read the full abstract ▾
All authors Oliver Hausdörfer, Linus Schwarz, Gabor Marko, Christian Dietz, Timo Class, Luka Hofer, Jim Yun-Jin Li, Johannes Hechtl, Ralf Römer, Angela P. Schoellig
Robotics
Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection
Hadrien Crassous, Mohamed Yassine Kabouri, Minahil Raza et al. (+2 more)
This paper studies how to adapt a computer
vision object detector to an unknown environment under both a
robot navigation time and annotation budget constraint. Our approach selects informative
robot trajectories and image samples to retrain the detector, explicitly targeting its failure cases. Formally, the approach is an
embodied variant of batch active learning, where at each round an
agent has a limited
navigation budget to collect candidate samples and a limited annotation budget for the most relevant images. We leverage spatial consistency to identify images with inconsistent labels, which are likely to provide the greatest improvement to the
vision model. We evaluate the approach using different active learning objectives on large scenes from the AI2-THOR simulator and on a real-world setup using a Boston Dynamics Spot
robot with the real-time object detector YOLOv5. Through comparison against several baselines, our experimental results show that spatial inconsistency helps guide the
agent and select relevant images without external supervision, achieving the highest detection accuracy at the end of the adaptation process under the same budget. The open-source project can be found at https://mkabouri.github.io/
embodied-active-learning-od
Read the full abstract ▾
All authors Hadrien Crassous, Mohamed Yassine Kabouri, Minahil Raza, Joni Pajarinen, Riad Akrour
Robotics · Cross-listed from eess.SY
Vessel Trajectory Prediction using COLREGs-aware Optimal Planning
David Kaikkonen, Fredrik Ljungberg and Erik Frisk
This paper presents a trajectory prediction method for marine vessels based on optimal planning. Crude initial trajectories respecting static obstacles are first generated using A*-search to provide a feasible warm start. In the second step, a numerical optimizer is used to ensure COLREG compliance. The prediction problem is posed as sequential trajectory planning from the perspective of each surrounding vessel, requiring only their current positions, velocities, and intended destinations as input. As the latter is included in AIS messages, this enables faster predictions than learning-based methods that typically require longer data histories. The proposed method is validated using real-world scenarios constructed from AIS data.
Read the full abstract ▾
Robotics
Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark
Jonas Weihing and Shahram Eivazi
Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms. While this task serves as a baseline environment within the research community, the ability of DRL to effectively learn the each-avoid task in complex and realistic scenarios beyond simplified and restricted tabletop settings remains uncertain. In this paper, we present, for the first time, a comprehensive
benchmark for the reachavoid task that accurately captures real-world complexities without simplifications. We demonstrate a diverse range of settings for robotic arm reach-avoid task, which can be used for evaluating DRL research. We achieved this by utilizing the MuJoCo MJX physics engine and parallelizing both the simulation environment and DRL
algorithms using the Brax library. We achieved state-of-the-art results with success rates of 96.1% (UR5e) and 98.8% (Franka Emika
Robot) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the static reachavoid task. Our results indicate that while in previous works DRL
agents could solve, for example, a reach task in a simplified setting perfectly, their
agents performance collapses when evaluated in realistic scenarios. Overall, this work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL. The environment and benchmarking code is available as open source at the following link
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
Orbis 2: A Hierarchical World Model for Driving
Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso et al. (+4 more)
Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial
reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a
diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing – predicting only the next frame from clean context – produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with
diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established
benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/
Read the full abstract ▾
All authors Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso, Karim Farid, Jonannes Dienert, Rajat Sahay, Thomas Brox
Robotics
Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation
Zhenduo Shang, Xiyao Liu, Bohan Li et al. (+4 more)
Imitation Learning aims to learn skills from extensive observations and demonstrations for
robots, so it suffers from data scarcity and environment generalization. The existing methods predominantly focus on imitation from in-domain tasks and consequently struggle with generalization to unseen tasks. To bridge this generalization gap, we propose the Dynamics-Aware Meta-Imitation (DAMI) framework. By integrating meta-learning to construct a shared skill space, DAMI equips
agents for rapid adaptation to novel tasks. We introduce the Visual-Motor Trajectory (VMT) module to capture complex spatio-temporal dynamics within the task latent space. Furthermore, we propose the Unpaired Unified Task (U2T) block to fuse unstructured
multimodal observations. To coordinate these representations, we integrate a Task-Conditioned Feature Modulation (TCFM) mechanism customized for modulating low-level 3D features. By capturing intrinsic dynamics from a random complete reference demonstration, our framework learns the underlying task logic rather than memorizing static cues, ensuring effective generalization. Extensive experiments in both simulation and real-world settings demonstrate that our approach outperforms state-of-the-art baselines regarding direct inference on seen tasks and adaptation to unseen tasks via few-shot fine-tuning.
Read the full abstract ▾
All authors Zhenduo Shang, Xiyao Liu, Bohan Li, Xudong Wang, Teng Ren, Lianqing Liu, Zhi Han
Robotics · Cross-listed from cs.CV
EgoExoMoCap: Distributed Ego-Exo Human Motion Capture
Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang et al. (+6 more)
Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in
embodied AI and VR/AR. Existing approaches focus on either
egocentric body tracking, estimating the motion of the subject wearing the device, or exocentric tracking, capturing the movements of people in the wearer's surroundings. So far, these two paradigms have largely been explored in isolation. In this paper, we propose a novel distributed framework that jointly leverages ego- and exocentric multi-modal signals for human motion estimation from HMDs. Unlike traditional motion capture systems requiring bulky multi-camera setups or obtrusive mocap suits, our approach, EgoExoMoCap, is as simple as two (or more) people, each wearing a pair of smart glasses. The method leverages head (plus potentially wrist) tracking signals for accurate estimation of global motion in the 3D world and combines context-aware image features based on DINOv3 to achieve robustness in the presence of noise and occlusions. Extensive experiments on two in-the-wild
datasets show that our approach can robustly reconstruct motion even in challenging scenarios.
Read the full abstract ▾
All authors Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang, Lingni Ma, Sebastian Starke, Robin Kips, Nadine Bertsch, Christian Holz, Federica Bogo
Robotics
Minimum Time Dubins Airplane Paths with Asymmetric Climb Rates
Jaeyoung Lim and Giuseppe Loianno
Dubins airplane paths approximate the limited maneuverability of fixed-wing vehicles with minimum
curvature and climb rate constraints. However, the symmetric climb rate constraints result in sub-optimal paths and conservative vehicle performance. In this work, we propose asymmetric Dubins airplane paths, which consider asymmetric climb rates for climbing and descending. We revisit the time optimality conditions and show that the asymmetric flight path angle constraints preserve optimality. We show that by considering asymmetric climb rates, we can take advantage of full performance of the vehicle, reducing the minimum time by 71% for connecting randomly generated states. We also demonstrate that the added climb rate results in 2.8 times faster to find the median solution time when integrated into a sampling-based planning task on rugged terrain, due to the added feasibility. We further demonstrate the practicality of the approach with a real-world flight.
Read the full abstract ▾
Robotics
Beyond Frontiers: Scene-Anomaly Guided Autonomous Exploration
Akash Kumbar, Abhinav Raundhal and Madhava Krishna
Autonomous exploration of unknown 3D environments is traditionally driven by coverage-maximizing
geometric heuristics. However, these methods typically determine exploration targets without considering the underlying structural context. This leads to inefficient trajectories often limiting the fidelity of the final 3D reconstruction. To bridge the gap between spatial coverage and reconstruction quality, we introduce a novel paradigm: reframing exploration as a
geometric anomaly minimization problem. We present SCAGE: SCene Anomaly Guided Exploration, a novel autonomous exploration framework that operates directly on unstructured 3D point clouds. Instead of blindly chasing volumetric boundaries, we equip the
robot with a foundational understanding of standard indoor architecture. As the
robot navigates, it continuously evaluates its live 3D observations against these learned expectations. When the incoming geometry contradicts the learned priors of a typical indoor environment, such as a fragmented wall or a partial table, the system flags these regions as scene anomalies. These
geometric inconsistencies act as a guiding signal, naturally drawing the
robot to investigate and resolve these structural anomalies from optimal vantage points. By actively targeting poorly reconstructed regions rather than just empty space, our approach seamlessly couples spatial discovery with high-fidelity mapping. Extensive evaluations demonstrate that SCAGE achieves superior volumetric coverage (~90% in all scenes) and higher 3D reconstruction quality compared to state-of-the-art baselines.
Read the full abstract ▾
Robotics
Deployment-Ready UWB Localization for Industrial Ground Robots with Automatic Anchor Calibration and Terrain-Aware Fusion
Alexander Raab, Giulio Delama, Roland Jung et al. (+1 more)
Ultra-Wideband (UWB) ranging has become a viable option for industrial Autonomous Mobile
Robot (AMR) localization due to improved accuracy and low cost. However, real-world deployments remain limited by two recurring challenges: calibrating static anchors can be time-consuming and error-prone, and integrating UWB with existing onboard sensors requires careful design to ensure robust and consistent pose estimation. Addressing these challenges, this paper presents an end-to-end pipeline that combines automatic anchor calibration with a generic multi-sensor
estimator tailored to surface-bound vehicle motion. It targets existing AMR stacks in scenarios where
robot pose priors are available for initialization. The calibration stage estimates anchor positions and range biases, while the localization stage fuses UWB with proprioceptive sensing in a bias-aware Extended Kalman Filter to improve consistency without extensive parameter tuning. Experiments on a commercial logistics AMR in a warehouse setting demonstrate accurate positioning indoors and across outdoor transitions, with improved consistency compared to an earlier
estimator formulation. Evaluation on an independent forklift
dataset further indicates transferability to other platforms. The method remains effective in test cases with limited line-of-sight and sparse anchor coverage. These results show that UWB localization can be deployed with substantially reduced manual effort while preserving the accuracy required for industrial AMRs. The collected warehouse
dataset is made publicly available.
Read the full abstract ▾
All authors Alexander Raab, Giulio Delama, Roland Jung, Stephan Weiss
Robotics
SkillNav: Score-Level Skill Intervention for Zero-Shot Object Goal Navigation
Ruijie Sang, Yiqun Duan, Pinhan Fu et al. (+3 more)
Vision-
Language Model (VLM)
agents have advanced zero-shot object-goal
navigation, yet single-frame
reasoning leaves them without the cross-step behavioral awareness an
embodied navigator requires, producing recurring failures such as dead-end stalls, in-room loops, and circuitous approaches to detected targets.
Prompt-based remedies inflate
token budgets across multi-submodule episodes and still struggle to encode inherently spatial signals such as angles, map
cells, and viewpoint coordinates. In this paper, we propose SkillNav, an extensible behavioral skill framework for VLM-based
navigation that treats the curiosity value map already maintained by modern VLM navigators as a writable substrate on which composable skills inscribe behavioral
memory at zero
token cost. Skills are stratified into three tiers by their level of behavioral authority, namely soft scaling for proportional reweighting, lower-bound boost for region-level guarantees, and hard override for threshold-triggered forced actions, and cooperate across tiers under a fixed composition order that establishes a predictable, declared priority among skills. This design turns capability improvement into skill registration: new behaviors plug in without retraining the VLM or disturbing existing skills, opening a path for continual refinement. A minimal
prompt channel complements the score-level skills with category-level semantic hints, yielding a dual-representation design in which spatial
memory lives on the map and semantic
memory in short
prompts. Training-free, SkillNav establishes new state-of-the-art SPL across MP3D (25.5), HM3D v0.1 (39.3), and HM3D v0.2 (43.2), improving SPL by up to 6.0 absolute over the strongest prior method, and achieves the highest Success Rate on HM3D v0.1 (69.7) and v0.2 (75.9).
Read the full abstract ▾
All authors Ruijie Sang, Yiqun Duan, Pinhan Fu, Ruilin Wang, Wei Sui, Xianda Guo
Robotics
Towards Artificial Nerves: Biomimetic Optical-Fiber Tactile Sensing for Robots
Laura E. Butcher, Chris J. Ford, Nathan F. Lepora et al. (+1 more)
Robotic systems increasingly demand tactile sensing that approaches the adaptability and resolution of human skin to enable dexterous manipulation and safe interaction. OptiTac is a biomimetic tactile sensor that emulates the mechanoreceptor-to-nerve architecture of human touch by pairing each mechanical pin on a soft skin with an optical fiber acting as an artificial nerve. This design demonstrates an architectural principle for routing tactile information away from the sensing surface while preserving high spatial resolution, establishing a practical route toward distributed tactile sensing in future robotic systems. By treating tactile signals as images, simple analytical methods, rather than opaque deep-learning models, are used to infer contact location, size, and shape, providing interpretable and scalable tactile intelligence. This work demonstrates how evolutionary principles from biology can guide the development of artificial nerve systems for
robots, offering a pathway toward human-like tactile perception in next-generation robotic platforms. More broadly, OptiTac establishes an artificial nerve-inspired sensing framework for interpretable robotic touch and a scalable route toward future distributed tactile systems.
Read the full abstract ▾
All authors Laura E. Butcher, Chris J. Ford, Nathan F. Lepora, Efi Psomopoulou
Robotics
A Task-Space Receding Horizon Controller for Fast Collision Avoidance
Mattia Penzotti and Marco Controzzi
Real-time collision avoidance for robotic manipulators requires fast reactions to unexpected obstacle motion and lookahead to avoid becoming trapped by near-future constraints. Full model predictive control can provide this foresight, but its online cost may grow quickly with horizon length, model fidelity, and the number of active
geometric constraints. Conversely, horizon-free reactive methods are computationally efficient but can be short-sighted in dynamic clutter. We present a task-space receding-horizon controller that uses a short contact-consistent rollout to generate a terminal kinematic reference satisfying internal non-penetration constraints, then computes only the first input of a smooth minimum-acceleration transition toward that reference. Starting from a closed-loop inverse-kinematics regulation law, the rollout is performed with an iterative dynamics solver operating on inflated convex
robot and obstacle geometries, so that
robot-obstacle contacts, dynamic obstacle motion, and self-collisions can shape the terminal reference without requiring full constrained trajectory optimization. We analyze the contact-inactive closed loop and show local exponential task-space regulation under standard regularity assumptions. For contacts activated inside the rollout, we characterize the corresponding discrete updates and bound the effect of moving obstacles on regular operating sets. Simulations on a 40-DOF multi-chain system show that intermediate horizons balance anticipation, responsiveness, and computational cost. Hardware experiments on a 6-DOF platform demonstrate consistent sim-to-real behavior without accurate inertial parameter estimation, and comparisons against dynamic optimization fabrics and model predictive control (MPC) baselines show improved success rates in dynamic clutter while preserving solve times compatible with real-time execution in the tested regimes.
Read the full abstract ▾
Robotics
AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning
Xiaojiang Peng, Kai Peng, Jie Lu et al. (+3 more)
Vision-
Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization when familiar sub-tasks are recombined in unseen configurations. We identify two mutually reinforcing failure modes: trajectory overfitting, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and perceptual shortcut, where action
tokens over-rely on wrist-view textures at the expense of global spatial grounding. To address both, we introduce AC-VLA, a plug-and-play Action Compositional learning framework comprising two architecture-agnostic components: (i) a compositional learning module that uses an LLM-driven instruction decomposer and a proprioceptive trajectory aligner to generate dense sub-task supervision, followed by mixed training on complete demonstrations and decomposed data to endow the model with compositional generalization; and (ii) a state-conditioned asymmetric masking strategy that suppresses wrist-view inputs during closed-gripper phases, enforcing global semantic grounding. All components are architectural modification-free and directly integrable into any VLA backbone. Instantiated on $\pi_{0.5}$ and evaluated on LIBERO and LIBERO-OOD
benchmarks, AC-VLA achieves a ~28% absolute improvement on compositional OOD tasks while maintaining near-perfect in-distribution performance.
Read the full abstract ▾
All authors Xiaojiang Peng, Kai Peng, Jie Lu, Zheng Lian, Zitong YU, Xiaobo Wang
Robotics
Implicit Virtual Leader: Decentralized Vision-Only Relative Pose Estimation for Multi-Robot Formations
Shiyuan Yang, Zelin Wang, Zhijia Tao et al. (+8 more)
Classical leader-follower formation control suffers from single points of failure and error propagation, and relies on absolute localization sensors that are ill-suited for GPS-denied environments. We present a learned,
vision-only
estimator that maps each
robot's
monocular image, together with messages exchanged over a communication graph, directly to its 6-DoF relative pose. Its key ingredient is the implicit virtual leader (IVL): a non-physical reference frame at the team centroid, implicitly learned inside a
Transformer-based graph neural network, so that estimation has no privileged node and needs no absolute localization. The
estimator additionally reports well-calibrated aleatoric (heteroscedastic GNLL) uncertainty alongside epistemic (MC~Dropout) uncertainty, compared systematically across simulation and real-world test sets. Trained only in simulation, the
estimator generalizes to unseen scenes, to larger unseen team sizes, and to an external real-world
benchmark. It exhibits no single point of failure: removing any one
robot costs at most $1.24\times$ the median removal, and removing $71\%$ of the communication links costs $1.77\times$ in position error without retraining. Trained on real-
robot data from a single platform, it transfers without modification to a heterogeneous team, estimating relative pose to $0.22$\,m and $1.6^\circ$ on physical
robots, where it drives closed-loop formation control.
Read the full abstract ▾
All authors Shiyuan Yang, Zelin Wang, Zhijia Tao, Yilin Wang, Zhengyu Hou, Xiaosong Kong, Borong Zhang, Yip Fun Yeung, Yuankai Luo, Sharon Lee, Qingbiao Li
Robotics
RAVEN: Reinforcement-Adaptive Visibility-Graph Planning for Robust Humanoid Navigation with Collision-Free MPC
Ruochen Hou, Shiqi Wang, Beom Jun Kim et al. (+3 more)
Humanoid navigation in dynamic environments requires long-horizon planning while respecting short-horizon dynamic and safety constraints. Classical visibility-graph planners combined with model predictive control (MPC) can efficiently generate collision-free trajectories, but their performance depends on manually tuned parameters and accurate system modeling. In real robotic systems, control delays, state-estimation noise, and locomotion uncertainties can cause overshoot and constraint violations even when the nominal path is geometrically optimal. We propose RAVEN, a hierarchical reinforcement learning (RL)-MPC framework for robust
humanoid navigation. Unlike prior approaches that use learning to tune cost weights or replace planning entirely, RAVEN employs RL to adapt the
geometric construction of a visibility-graph planner by modifying obstacle inflation and related graph parameters. By directly reshaping the free-space geometry, the learned planner alters the
topology of the global path to compensate for delay and tracking imperfections. A collision-free MPC layer then tracks the planned trajectory while explicitly enforcing velocity bounds and obstacle-avoidance constraints. By training under realistic delays and observation noise, RAVEN learns planning adaptations that improve robustness while retaining explicit long-horizon
geometric planning and constrained optimization, in contrast to end-to-end learning approaches. We evaluate RAVEN against a manually tuned visibility-graph MPC baseline and a pure RL
navigation policy. Results demonstrate reduced overshoot near obstacles, improved robustness in narrow passages, and more reliable
navigation under delay and noise. These findings indicate that reinforcement-adaptive graph construction combined with constrained MPC provides an effective and interpretable alternative to end-to-end learning for robust
humanoid navigation.
Read the full abstract ▾
All authors Ruochen Hou, Shiqi Wang, Beom Jun Kim, Hanzhang Fang, Mehak Singal, Dennis W. Hong
Robotics
A Generative Partially Specified Finite State Machine Approach to Complex Behaviour Planning
Kalana Ratnayake, Michael Pritchard, David Hinwood et al. (+2 more)
Autonomous
robots operating in dynamic environments require behaviour planning systems that combine reactivity, interpretability, and adaptability. While Large
Language Models have been successfully integrated with Behaviour Trees for dynamic replanning, Finite State Machines, despite their widespread adoption and computational efficiency, remain unexplored for generative approaches. We propose a Generative Partially Specified Finite State Machine (GPSFSM) neurosymbolic architecture that utilises the symbolic and semantic structure of FSMs, including states and event-triggered transitions, to implement Behaviour Planning. This paper introduces the first GPSFSM framework for robotics, featuring Fabric, an FSM engine that parses, validates, and executes behaviour plans that contain Sequential, Recovery, Parallel-Any, and Parallel-All control structures. We extend the Capabilities2 package in ROS2 with an asynchronous event system for behaviour chaining and runtime parameter injection for configurable execution, addressing the ad-hoc function representations that limit current generative systems. PromptTools provides a unified ROS 2 interface to local and cloud LLMs, with
prompt buffering, enabling dynamic asynchronous composition of task and context information. Together, these components enable standardised semantic capability descriptions for
robot-agnostic development. Experimental evaluation on
navigation tasks demonstrates that our GPSFSM approach achieves consistently higher plan-generation success rates than the state-of-the-art BTGenBot system, particularly excelling in zero-shot scenarios where BTs typically struggle, while maintaining comparable or lower planning latency to frontier LLMs. We also demonstrate that our system can generate complex behaviours. We release an open-source ROS2 stack that makes generative FSM planning practical and reproducible for robotic systems.
Read the full abstract ▾
All authors Kalana Ratnayake, Michael Pritchard, David Hinwood, Maleen Jayasuriya, Damith Herath
Robotics
Continuously Stable Structure through Plastic Deformation
Junlong Xiao, Yaoqiang Pan, Xuan Zhang et al. (+2 more)
Soft
robots have seen widespread adoption in interactive tasks due to their inherent compliance and adaptability. However, these advantages often come at the cost of stability, posing challenges in a dynamic environment. This limitation is especially critical in soft grippers, where instability under acceleration or external disturbances can result in grasp failure. In this study, we present a continuously stable structure through plastic deformation (CSSPD), integrated into a soft gripper. By leveraging the mechanism of plastic deformation, the gripper maintains continuous configurations without energy input, while the added stiffness ensures both static and dynamic stability. We introduce a bioinspired paw pad that significantly enhances stability and enables sensing-based rapid object grasping. Then we develop the mathematical model and optimize the kirigami structure of the metal layer. Experimental results show that the gripper can sustain a passive holding force of up to 16 N without energy input, achieving performance comparable to pneumatic actuation at 0.3 MPa. When combined with pneumatic actuation, it remains stable under pulsed accelerations of up to 400 m/s^2. It can also passively perch on tree branches for extended periods without power, demonstrating promise for mobile robotic applications.
Read the full abstract ▾
All authors Junlong Xiao, Yaoqiang Pan, Xuan Zhang, Michael Yu Wang, Chao Chen
Robotics
Learning a System-Level Surrogate for Hydraulic Excavators: A Simulation-to-Real LSTM Approach
Shuai Wang, Shen Wang, Qiang Wang et al. (+4 more)
Developing autonomous hydraulic excavators is constrained by limited access to physical machines and the high cost of real-world experimentation. This paper proposes a simulation-to-real framework for learning a system-level digital surrogate using Long Short-Term
Memory (LSTM) networks. Instead of modeling internal dynamics, the excavator is treated as an input-output operator, and the surrogate is trained to reproduce its closed-loop behavior under identical control inputs. The approach is first validated in a MuJoCo simulation environment and then transferred to a real excavator. To address measurement inconsistencies in real-world data, a consistency-aware state estimation method based on adaptive Kalman filtering is introduced. Experimental results demonstrate that the learned surrogate achieves high fidelity in both angular velocity and long-horizon trajectory reproduction under closed-loop autoregressive evaluation. These results confirm that the proposed model can serve as a drop-in surrogate for both simulation and physical systems, enabling scalable and efficient development of excavation automation
algorithms.
Read the full abstract ▾
All authors Shuai Wang, Shen Wang, Qiang Wang, Muguo Du, Donghai Shi, Chenyu Wang, Xiaofeng Tao
Robotics
Difference-Based Relational Learning for Zero-Shot Object-Goal Visual Navigation With Direct Sim-to-Real Transfer
Guolei Qi and Feitian Zhang
End-to-end deep reinforcement learning (DRL) for zero-shot object-goal visual
navigation remains challenged by the sim-to-real gap, particularly variations in object appearance and restricted camera field-of-view (FoV). This letter proposes a Temporal Difference-Relational Network (T-DRN) for robust zero-shot sim-to-real transfer. T-DRN combines a Siamese difference-based feature extractor, which computes relational difference between the target and observed objects to produce domain-independent representations, with a dual-frame temporal buffer that preserves short-term object continuity under narrow FoV. Extensive experiments in AI2-THOR demonstrate that T-DRN improves zero-shot generalization in terms of success rates over strong baselines. Furthermore, T-DRN is systematically validated on a physical wheeled
robot, demonstrating robust performance under real sensing and actuation constraints and supporting the feasibility of direct sim-to-real transfer.
Read the full abstract ▾
Robotics
IMBench: A Benchmark for Intuitive Robotic Manipulation
Anurag Maurya, Sukhvansh Jain, Prajwal Avhad et al. (+9 more)
Humans combine
reasoning and motor control to solve complex manipulation tasks under diverse constraints. They build an understanding of the physical world that helps them convert
reasoning into actions and quickly adapt to new scenes, tasks, and rules. We refer to this capability as intuitive manipulation. Existing
benchmarks fail to capture this integration: they evaluate physical
reasoning in isolation from execution, or measure policy performance without requiring explicit
reasoning. We introduce IMBENCH, a
benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical
reasoning, action generation, and iterative execution. Our tasks require models to infer task-relevant physical structure and generate feasible action sequences under explicit constraints, including contact-rich manipulation, tool use, and multi-stage dependencies. We introduce a
benchmark of 35 tasks, 14K filtered trajectories, and scalable tools for generating diverse scenarios. Experiments reveal a consistent gap:
vision language models show partial physical
reasoning ability but fail to produce executable plans, while state-of-the-art
vision-
language-action models struggle to satisfy task constraints and generalize across scenarios. These results identify intuitive manipulation as a missing axis in current foundation models and generalist
robot policies, and position IMBENCH as a step toward evaluating and enabling more integrated, adaptive physical intelligence.
Read the full abstract ▾
All authors Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha
Robotics
Scalable Open-Source Visuotactile Sensor for 6-Axis Contact Wrench Estimation in Tensegrity Robots
Wenzhe Tong, Jonathan Mi, Xili Yi et al. (+2 more)
This paper presents a scalable, open-source visuotactile sensing system for tensegrity
robots that enables six-axis wrench estimation and contact detection. The proposed endcap sensor integrates an elastomeric shell, a 3D-printed thermoplastic polyurethane (TPU) interface, and a rigid base housing an embedded camera and LED illumination ring. A novel gyroid-infill bonding technique is introduced to form a durable elastomer-TPU interface without adhesives, yielding a lightweight and modular design compatible with large-scale tensegrity structures. A tactile-to-wrench neural network maps shear vector fields to six-dimensional force and torque measurements. Experimental results demonstrate accurate and stable wrench estimation with a mean squared error (MSE) of 0.1531 on static validation data and out-of-domain generalization under dynamic motion. Furthermore, full-system integration on a 12 kg tensegrity
robot confirms the sensor's ability to reliably identify ground contacts. The system substantially improves the practicality of tactile feedback for tensegrity
robots, offering a low-cost, reproducible, and physically interpretable pathway toward contact-aware proprioception and state estimation. Open source files are available at \href{https://github.com/Jonathan-Twz/tensegrity-gelfoot}{github.com/Jonathan-Twz/tensegrity-gelfoot}
Read the full abstract ▾
All authors Wenzhe Tong, Jonathan Mi, Xili Yi, Nima Fazeli, Xiaonan Huang
Robotics
Think at 5 Hz, Act at 20 Hz: Asynchronous Fast-Slow Vision-Language-Action Inference for Closed-Loop Driving
Yun Li, Jiachen Gong, Simon Thompson et al. (+7 more)
Large
language models bring instruction following and scene
reasoning to end-to-end driving, but their inference latency collides with the control rate a vehicle requires. Existing closed-loop
agents hide this gap by invoking the model on alternate simulation ticks and replaying the previous command in between, so half of all control outputs ignore the newest observations. We present a fast-slow architecture that removes this compromise. A frozen 7B
vision-
language backbone acts as the slow system, digesting
navigation instructions and visual history at low frequency while exposing its per-layer key-value cache as a standing representation of the scene. A lightweight action expert acts as the fast system, attending to this cache and to the current camera frame at every simulation tick to regress waypoints in a single forward pass. Since the cache lags behind the world at deployment, we train the expert under randomized staleness, aligning training with asynchronous execution. On LangAuto-Short routes in CARLA, our system produces fresh control at every 50 ms simulation tick and lifts route completion from 37.0 to 94.0 over the frame-skipping baseline. A frame-skip ablation with the same expert separates the two factors at work: the expert raises the driving score on its own, while per-tick freshness raises completion from 82.1 to 94.0 and cuts red-light violations by a third. Trained on a single town, the expert transfers zero-shot to two unseen towns, holding 84-94% route completion where the baseline reaches 31-41%. It reduces open-loop waypoint error by nearly a factor of four compared to the backbone's own action head, at a per-tick model cost of 32 ms that is independent of history length on a single consumer GPU.
Read the full abstract ▾
All authors Yun Li, Jiachen Gong, Simon Thompson, Ehsan Javanmardi, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada
Robotics
AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots
Priyanka V.Setty, Arvind Ramanathan, Ian Foster et al. (+1 more)
Self-driving laboratories increasingly rely on low-cost
liquid handlers such as the Opentrons OT-2, which ship without the pressure-based aspiration monitoring of Hamilton or Tecan systems and are typically run open-loop. Two failure modes go undetected: protocols that are syntactically valid but violate assay-specific invariants (e.g., tip reuse between a PCR template and a no-template control), and physical execution failures (partial dispense, air bubbles, missing tips) at runtime. We present AEGIS, a two-layer guardian for both. Layer 1 pairs a curated machine-readable assay rule database with an LLM that reasons over OT-2 Python code, reaching an adjusted F1 of 0.97 on a 24-protocol
benchmark across five assay families and beating rules-only and LLM-only ablations across five backends; a free open-weight model ties the best proprietary one, so no paid API is required. Layer 2 fits a PCA world model to YOLO-cropped four-frame pipette trajectories; under a leakage-free leave-one-plate-out evaluation it reaches average precision 0.89 and operating-point F1 0.71 (AUROC 0.80), a deployment-faithful number that matches the live demonstration, and we characterize the small-pipette (p20) resolution limit (F1 0.47). A live demonstration on a physical OT-2 (five replicates per condition) catches planted no-tip failures deterministically and partial dispense on coloured dyes, with an always-VLM self-vote gate lifting partial-dispense recall to 5/5; transparent water is a principled limit of any front-view-only monitor, which AEGIS surfaces as low-confidence VLM
reasoning rather than a wrong verdict. Cascade triage holds VLM cost near $1.63 per plate versus $10.33 for an always-VLM baseline. AEGIS is open source and, to our knowledge, the first system to unify pre-flight assay-aware validation with runtime visual monitoring for an open-source
liquid handler.
Read the full abstract ▾
All authors Priyanka V.Setty, Arvind Ramanathan, Ian Foster, Rick Stevens
Robotics
MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigation
Rajat Bhattacharjya, Hyeonjong Ju, Sing-Yao Wu et al. (+2 more)
Communication-limited
robots in mission-critical scenarios such as disaster inspection and search-and-rescue must make reliable onboard decisions without access to remote operators or high-capacity
reasoning services. Episodic
memory reuse is an attractive low-cost fallback, but
retrieval similarity does not guarantee execution validity, i.e., a retrieved action may match the current context yet be unsafe due to changed
topology, insufficient battery margin, or unreliable prior outcomes. We call such high-similarity but execution-invalid episodes
memory traps. This creates a safety-efficiency design space where similarity only reuse minimizes fallback cost but can be unsafe, while always invoking local
reasoning improves safety at high computational and energy cost. This paper presents MemoGuard, a lightweight adaptive runtime that validates episodic
memories against
topology, resource, and outcome contracts before reuse, invoking fallback only when validation fails. In a graph-based corridor-inspection simulator, MemoGuard reduces battery safety violations by 76.6% over similarity-only top-1 reuse while reducing fallback calls by 21.4% over always
reasoning. On an NVIDIA Jetson AGX Xavier with local llama3.2:3b fallback
reasoning, this corresponds to 3.67 s and 36.97 J of avoided fallback-
reasoning overhead per trial. We open-source MemoGuard at https://github.com/hetheiin/memoguard.
Read the full abstract ▾
All authors Rajat Bhattacharjya, Hyeonjong Ju, Sing-Yao Wu, Eli Bozorgzadeh, Nikil Dutt
Robotics
A Model-Based Decoupling Strategy for Proprioception and Contact Sensing in an Architected Soft Manipulator
Francesco Stella, Annan Zhang, Cosimo Della Santina et al. (+2 more)
Soft continuum
robots require embedded sensing for proprioception and contact detection, yet integrating sensors into sparse, highly deformable architected structures remains challenging. We present a model-based strategy that decouples proprioceptive and contact signals from a common set of fluidic pressure sensors embedded in a soft architected segment. Each segment of the Innervated Trimmed Helicoid (ITH) contains six air channels routed in a localized zigzag pattern along the circumference. With only three principal kinematic degrees of freedom (axial compression, bending in x, bending in y), the six pressure readings form an overdetermined system. A piecewise constant
curvature model maps pressures to shape, and Huber regression identifies outlier channels whose residuals indicate external contact. On a single ITH segment, this approach achieves proprioceptive shape estimation with a relative bending error of 0.11 +/- 0.02 and a contact detection rate of 97% across 178 trials. We integrate eight ITH segments into Air-Helix, a tendon-driven soft continuum manipulator, and present exploratory whole-arm demonstrations that include tactile teaching by demonstration, admittance-controlled force regulation, and tactile object reconstruction. The results suggest that localized fluidic innervation combined with model-based redundancy resolution is a practical path toward concurrent proprioception and contact sensing in architected soft
robots.
Read the full abstract ▾
All authors Francesco Stella, Annan Zhang, Cosimo Della Santina, Josie Hughes, Daniela Rus
Robotics
PACE: Persona Adaptation through Conversational Elicitation in Human-Robot Interaction
Peizhen Li, Longbing Cao, Megani Rajendran et al. (+3 more)
Equipping
humanoid robots with coherent and adaptable personas is crucial for fostering natural, engaging, and trustworthy human-
robot interaction (HRI). However, existing approaches often rely on static, hard-coded identities that lack the flexibility to adapt to individual user contexts. In this paper, we present PACE (Persona Adaptation through Conversational Elicitation), a novel framework for the interactive generation and deployment of structured personas on the Ameca
humanoid robot. Our system introduces an Interactive Persona Elicitation Pipeline, enabling the
robot to dynamically synthesize a tailored, psychologically grounded identity through user Q&A. This elicitation process feeds into a persona
prompt compilation phase, generating a structured persona
prompt built upon multi-perspective dimensions. We detail the
Embodied System Integration required to translate this structured specification into expressive,
multimodal humanoid behaviors. Through a comprehensive empirical HRI evaluation, we assess the impact of dynamically generated personas on user trust, perceived anthropomorphism, persona consistency, personal relevance, and interaction quality compared to a generic baseline. These contributions establish a scalable pathway for deploying personalized, interactive, and reliable identities in
embodied humanoid assistants.
Video demo is available at: https://lipzh5.github.io/PACE/
Read the full abstract ▾
All authors Peizhen Li, Longbing Cao, Megani Rajendran, Timothy Liu, Aik Beng Ng, Simon See