Robotics
Improving the performance of AI-powered Affordable Robotics for Assistive Tasks
Dharunish Yugeswardeenoo
By 2050, the global demand for assistive care is expected to reach 3.5 billion people, far outpacing the availability of human caregivers. Existing robotic solutions remain expensive and require technical expertise, limiting accessibility. This work introduces a low-cost robotic arm for assistive tasks such as feeding, cleaning spills, and fetching medicine. The system uses imitation learning from demonstration
videos, requiring no task-specific programming or manual labeling. The
robot consists of six servo motors, dual cameras, and 3D-printed grippers. Data collection via teleoperation with a leader arm yielded 50,000
video frames across the three tasks. A novel Phased Action Chunking
Transformer (PACT) captures temporal dependencies and segments motion dynamics, while a Temporal Ensemble (TE) method refines trajectories to improve accuracy and smoothness. Evaluated across five model sizes and four architectures, with ten hours of real-world testing, the system achieved over 90% task accuracy, up to 40% higher than baselines. PACT enabled a 5x model size reduction while maintaining 75% accuracy. Saliency analysis showed reliance on key visual cues, and phase
token gradients peaked at critical trajectory moments, indicating effective temporal
reasoning. Future work will explore bimanual manipulation and mobility for expanded assistive capabilities.
Read the full abstract ▾
Robotics
NEBULA: Do We Evaluate Vision-Language-Action Agents Correctly?
Jierui Peng, Yanyan Zhang, Yicheng Duan et al. (+3 more)
The evaluation of
Vision-
Language-Action (VLA)
agents is hindered by the coarse, end-task success metric that fails to provide precise skill diagnosis or measure robustness to real-world perturbations. This challenge is exacerbated by a fragmented data landscape that impedes reproducible research and the development of generalist models. To address these limitations, we introduce NEBULA, a unified ecosystem for single-arm manipulation that enables diagnostic and reproducible evaluation. NEBULA features a novel dual-axis evaluation protocol that combines fine-grained capability tests for precise skill diagnosis with systematic stress tests that measure robustness. A standardized API and a large-scale, aggregated
dataset are provided to reduce fragmentation and support cross-
dataset training and fair comparison. Using NEBULA, we demonstrate that top-performing VLAs struggle with key capabilities such as spatial
reasoning and dynamic adaptation, which are consistently obscured by conventional end-task success metrics. By measuring both what an
agent can do and when it does so reliably, NEBULA provides a practical foundation for robust, general-purpose
embodied agents.
Read the full abstract ▾
All authors Jierui Peng, Yanyan Zhang, Yicheng Duan, Tuo Liang, Vipin Chaudhary, Yu Yin
Robotics
Cosmos-Surg-dVRK: World Foundation Model-based Automated Online Evaluation of Surgical Robot Policy Learning
Lukas Zbinden, Nigel Nelson, Juo-Tung Chen et al. (+5 more)
The rise of
surgical robots and
vision-
language-action models has accelerated the development of autonomous
surgical policies and efficient assessment strategies. However, evaluating these policies directly on physical robotic platforms such as the da Vinci Research Kit (dVRK) remains hindered by high costs, time demands, reproducibility challenges, and variability in execution. World foundation models (WFM) for physical AI offer a transformative approach to simulate complex real-world
surgical tasks, such as soft tissue deformation, with high fidelity. This work introduces Cosmos-Surg-dVRK, a
surgical finetune of the Cosmos WFM, which, together with a trained
video classifier, enables fully automated online evaluation and benchmarking of
surgical policies. We evaluate Cosmos-Surg-dVRK using two distinct
surgical datasets. On tabletop suture pad tasks, the automated pipeline achieves strong correlation between online rollouts in Cosmos-Surg-dVRK and policy outcomes on the real dVRK Si platform, as well as good agreement between human labelers and the V-JEPA 2-derived
video classifier. Additionally, preliminary experiments with ex-vivo porcine cholecystectomy tasks in Cosmos-Surg-dVRK demonstrate promising
alignment with real-world evaluations, highlighting the platform's potential for more complex
surgical procedures.
Read the full abstract ▾
All authors Lukas Zbinden, Nigel Nelson, Juo-Tung Chen, Xinhao Chen, Ji Woong Kim, Mahdi Azizian, Axel Krieger, Sean Huver
Robotics
DeGrip: A Compact Cable-driven Robotic Gripper for Desktop Disassembly
Bihao Zhang, Davood Soleymanzadeh, Xiao Liang et al. (+1 more)
Intelligent robotic disassembly of end-of-life (EOL) products has been a long-standing challenge in robotics. While machine learning techniques have shown promise, the lack of specialized hardware limits their application in real-world scenarios. We introduce DeGrip, a customized gripper designed for the disassembly of EOL computer desktops. DeGrip provides three degrees of freedom (DOF), enabling arbitrary configurations within the disassembly environment when mounted on a robotic manipulator. It employs a cable-driven transmission mechanism that reduces its overall size and enables operation in confined spaces. The wrist is designed to decouple the actuation of wrist and jaw joints. We also developed an EOL desktop disassembly environment in Isaac Sim to evaluate the effectiveness of DeGrip. The tasks were designed to demonstrate its ability to operate in confined spaces and disassemble components in arbitrary configurations. The evaluation results confirm the capability of DeGrip for EOL desktop disassembly.
Read the full abstract ▾
All authors Bihao Zhang, Davood Soleymanzadeh, Xiao Liang, Minghui Zheng
Robotics
VAR-SLAM: Visual Adaptive and Robust SLAM for Dynamic Environments
João Carlos Virgolino Soares, Gabriel Fischer Abati and Claudio Semini
Visual SLAM in dynamic environments remains challenging, as several existing methods rely on semantic filtering that only handles known object classes, or use fixed robust kernels that cannot adapt to unknown moving objects, leading to degraded accuracy when they appear in the scene. We present VAR-SLAM (Visual Adaptive and Robust SLAM), an ORB-SLAM3-based system that combines a lightweight semantic keypoint filter to deal with known moving objects, with Barron's adaptive robust loss to handle unknown ones. The shape parameter of the robust kernel is estimated online from residuals, allowing the system to automatically adjust between Gaussian and heavy-tailed behavior. We evaluate VAR-SLAM on the TUM RGB-D, Bonn RGB-D Dynamic, and OpenLORIS
datasets, which include both known and unknown moving objects. Results show improved trajectory accuracy and robustness over state-of-the-art baselines, achieving up to 25% lower ATE RMSE than NGD-SLAM on challenging sequences, while maintaining performance at 27 FPS on average.
Read the full abstract ▾
Robotics · Cross-listed from cs.MA
Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards
Rupal Nigam, Niket Parikh, Hamid Osooli et al. (+3 more)
Real-world multi-
agent systems may require ad hoc teaming, where an
agent must coordinate with other previously unseen teammates to solve a task in a zero-shot manner. Prior work often either selects a pretrained policy based on an inferred model of the new teammates or pretrains a single policy that is robust to potential teammates. Instead, we propose to leverage all pretrained policies in a zero-shot transfer setting. We formalize this problem as an ad hoc multi-
agent Markov decision process and present a solution that uses two key ideas, generalized policy improvement and difference rewards, for efficient and effective knowledge transfer between different teams. We empirically demonstrate that our
algorithm, Generalized Policy improvement for Ad hoc Teaming (GPAT), successfully enables zero-shot transfer to new teams in three simulated environments: cooperative foraging, predator-prey, and Overcooked. We also demonstrate our
algorithm in a real-world multi-
robot setting.
Read the full abstract ▾
All authors Rupal Nigam, Niket Parikh, Hamid Osooli, Mikihisa Yuasa, Jacob Heglund, Huy T. Tran
Robotics · Cross-listed from cs.CV
Aria Gen 2 Pilot Dataset
Chen Kong, James Fort, Aria Kang et al. (+20 more)
The Aria Gen 2 Pilot
Dataset (A2PD) is an
egocentric multimodal open
dataset captured using the state-of-the-art Aria Gen 2 glasses. To facilitate timely access, A2PD is released incrementally with ongoing
dataset enhancements. The initial release features Dia'ane, our primary subject, who records her daily activities alongside friends, each equipped with Aria Gen 2 glasses. It encompasses five primary scenarios: cleaning, cooking, eating, playing, and outdoor walking. In each of the scenarios, we provide comprehensive raw sensor data and output data from various machine perception
algorithms. These data illustrate the device's ability to perceive the wearer, the surrounding environment, and interactions between the wearer and the environment, while maintaining robust performance across diverse users and conditions. The A2PD is publicly available at projectaria.com, with open-source tools and usage examples provided in Project Aria Tools.
Read the full abstract ▾
All authors Chen Kong, James Fort, Aria Kang, Jonathan Wittmer, Simon Green, Tianwei Shen, Yipu Zhao, Cheng Peng, Gustavo Solaira, Andrew Berkovich, Nikhil Raina, Vijay Baiyya, Evgeniy Oleinik, Eric Huang, Fan Zhang, Julian Straub, Mark Schwesinger, Luis Pesqueira, Xiaqing Pan, Jakob Julian Engel, Carl Ren, Mingfei Yan, Richard Newcombe
Robotics
Dynamic Recalibration in LiDAR SLAM: Integrating AI and Geometric Methods with Real-Time Feedback Using INAF Fusion
Zahra Arjmandi and Gunho Sohn
This paper presents a novel fusion technique for
LiDAR Simultaneous Localization and Mapping (SLAM), aimed at improving localization and 3D mapping using
LiDAR sensor. Our approach centers on the Inferred
Attention Fusion (INAF) module, which integrates AI with
geometric odometry. Utilizing the KITTI
dataset's
LiDAR data, INAF dynamically adjusts
attention weights based on environmental feedback, enhancing the system's adaptability and measurement accuracy. This method advances the precision of both localization and 3D mapping, demonstrating the potential of our fusion technique to enhance autonomous
navigation systems in complex scenarios.
Read the full abstract ▾
Robotics
DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation
Xinyue Xu, Jieqiang Sun, Jing Dai et al. (+8 more)
A dexterous hand succeeds or fails through contact, yet every scalable record of human manipulation (
video,
vision-based hand tracking, motion capture) observes how the hand moves and misses how it pushes. We present DexCanvas, a hybrid real-synthetic
dataset of 7,000 hours of dexterous hand-object interaction with per-frame contact points and contact forces, grown from 70 hours of optical motion capture that spans 21 manipulation types, from power and precision grasps to in-hand manipulation. The forces come from a real-to-sim pipeline: one reinforcement learning (RL) policy, conditioned on object geometry and manipulation type, drives an actuated MANO hand in physics simulation to reproduce each captured object trajectory, and the simulator measures the contacts that produce the reproduced motion. The policy reproduces 87.5% of the demonstrations and still succeeds on 80.9% of rollouts from perturbed initial object poses; these perturbed rollouts expand the 70 captured hours a hundredfold. On six YCB objects absent from the capture set it succeeds 72.5% of the time. The contact channels make better policies: a
Diffusion Policy that observes DexCanvas contact features succeeds 82.2% of the time, against 67.5% without them, over 18,522 trials. Contact-driven retargeting carries the same demonstrations to two dexterous
robot hands, where the RL stage again produces stable rollouts. A first release of 111,857 force-annotated trajectories is publicly available on Hugging Face.
Read the full abstract ▾
All authors Xinyue Xu, Jieqiang Sun, Jing Dai, Siyuan Chen, Lanjie Ma, Ke Sun, Bin Zhao, Jianbo Yuan, Sheng Yi, Haohua Zhu, Irvine Lu
Robotics
Few-Shot Demonstration-Driven Task Coordination and Trajectory Execution for Multi-Robot Systems
Taehyeon Kim, Vishnunandan L.N. Venkatesh and Byung-Cheol Min
Learning coordinated behaviors for multi-
robot systems from only a few demonstrations is difficult because temporal task dependencies and spatial trajectory generation are tightly coupled, which increases the hypothesis space and often yields unstable generalization in data-scarce regimes. We present DDACE, a structured few-shot learning framework that introduces a structural inductive bias by explicitly decoupling temporal coordination from spatial trajectory synthesis. Demonstrations are first processed via spectral clustering to extract coordination structure and form interaction graphs. A Temporal Graph Network predicts action dependencies and sequences, while Gaussian Process models generate progress-parameterized
geometric trajectories that adapt to new start/goal configurations. This factorized design reduces hypothesis coupling and improves data efficiency for few-shot multi-
robot coordination. Extensive simulation studies and real-
robot experiments show that DDACE produces stable coordinated executions from a small number of demonstrations and improves trajectory consistency compared to end-to-end imitation baselines under limited data. Additional materials are available at https://sites.google.com/view/ddace.
Read the full abstract ▾
Robotics
HEADER: Hierarchical Robot Exploration via Attention-Based Deep Reinforcement Learning with Expert-Guided Reward
Yuhong Cao, Yizhuo Wang, Jingsong Liang et al. (+4 more)
This work pushes the boundaries of learning-based methods in autonomous
robot exploration in terms of environmental scale and exploration efficiency. We present HEADER, an
attention-based reinforcement learning approach with hierarchical graphs for efficient exploration in large-scale environments. HEADER follows existing conventional methods to construct hierarchical representations for the
robot belief/map, but further designs a novel community-based
algorithm to construct and update a global graph, which remains fully incremental, shape-adaptive, and operates with linear complexity. Building upon
attention-based networks, our planner finely reasons about the nearby belief within the local range while coarsely leveraging distant information at the global scale, enabling next-best-viewpoint decisions that consider multi-scale spatial dependencies. Beyond novel map representation, we introduce a parameter-free privileged reward that significantly improves model performance and produces near-optimal exploration behaviors, by avoiding training objective bias caused by handcrafted reward shaping. In simulated challenging, large-scale exploration scenarios, HEADER demonstrates better scalability than most existing learning and non-learning methods, while achieving a significant improvement in exploration efficiency (up to 20%) over state-of-the-art baselines. We also deploy HEADER on hardware and validate it in complex, large-scale real-life scenarios, including a 300m*230m campus environment.
Read the full abstract ▾
All authors Yuhong Cao, Yizhuo Wang, Jingsong Liang, Shuhao Liao, Yifeng Zhang, Peizhuo Li, Guillaume Sartoretti
Robotics
Freehand 3D Ultrasound Imaging: Sim-in-the-Loop Probe Pose Optimization via Visual Servoing
Yameng Zhang, Dianye Huang, Max Q.-H. Meng et al. (+2 more)
Freehand 3D ultrasound (US) imaging using conventional 2D probes offers flexibility and accessibility for diverse clinical applications but faces challenges in accurate probe pose estimation. Traditional methods depend on costly tracking systems, while neural network-based methods struggle with image noise and error accumulation, compromising reconstruction precision. We propose a cost-effective and versatile solution that leverages lightweight cameras and visual servoing in simulated environments for precise 3D US imaging. These cameras capture visual feedback from a textured planar workspace. To counter occlusions and lighting issues, we introduce an image restoration method that reconstructs occluded regions by matching surrounding texture patterns. For pose estimation, we develop a simulation-in-the-loop approach, which replicates the system setup in simulation and iteratively minimizes pose errors between simulated and real-world observations. A visual servoing controller refines the
alignment of camera views, improving translational estimation by optimizing image
alignment. Validations on a soft vascular phantom, a 3D-printed conical model, and a human arm demonstrate the robustness and accuracy of our approach, with Hausdorff distances to the reference reconstructions of 0.359 mm, 1.171 mm, and 0.858 mm, respectively. These results confirm the method's potential for reliable freehand 3D US reconstruction.
Read the full abstract ▾
All authors Yameng Zhang, Dianye Huang, Max Q.-H. Meng, Nassir Navab, Zhongliang Jiang
Robotics
Integration of a Variable Stiffness Link for Long-Reach Aerial Manipulation
Manuel J. Fernandez, Alejandro Suarez, Anibal Ollero et al. (+1 more)
This paper presents the integration of a Variable Stiffness Link (VSL) for long-reach aerial manipulation, enabling adaptable mechanical coupling between an aerial multirotor platform and a dual-arm manipulator. Conventional long-reach manipulation systems rely on rigid or cable connections, which limit precision or transmit disturbances to the aerial vehicle. The proposed VSL introduces an adjustable stiffness mechanism that allows the link to behave either as a flexible rope or as a rigid rod, depending on task requirements. The system is mounted on a quadrotor equipped with the LiCAS dual-arm manipulator and evaluated through teleoperated experiments, involving external disturbances and parcel transportation tasks. Results demonstrate that varying the link stiffness significantly modifies the dynamic interaction between the
UAV and the payload. The flexible configuration attenuates external impacts and aerodynamic perturbations, while the rigid configuration improves positional accuracy during manipulation phases. These results confirm that VSL enhances versatility and safety, providing a controllable trade-off between compliance and precision. Future work will focus on autonomous stiffness regulation, multi-rope configurations, cooperative aerial manipulation and user studies to further assess its impact on teleoperated and semi-autonomous aerial tasks.
Read the full abstract ▾
All authors Manuel J. Fernandez, Alejandro Suarez, Anibal Ollero, Matteo Fumagalli
Robotics
Educational SoftHand-A: Building an Anthropomorphic Hand with Soft Synergies using LEGO MINDSTORMS
Jared K. Lepora, Haoran Li, Efi Psomopoulou et al. (+1 more)
This paper introduces an anthropomorphic
robot hand built entirely using LEGO MINDSTORMS: the Educational SoftHand-A, a tendon-driven, highly-underactuated
robot hand based on the Pisa/IIT SoftHand and related hands. To be suitable for an educational context, the design is constrained to use only standard LEGO pieces with tests using common equipment available at home. The hand features dual motors driving an agonist/antagonist opposing pair of tendons on each finger, which are shown to result in reactive fine control. The finger motions are synchonized through soft synergies, implemented with a differential mechanism using clutch gears. Altogether, this design results in an anthropomorphic hand that can adaptively grasp a broad range of objects using a simple actuation and control mechanism. Since the hand can be constructed from LEGO pieces and uses state-of-the-art design concepts for robotic hands, it has the potential to educate and inspire children to learn about the frontiers of modern robotics.
Read the full abstract ▾
All authors Jared K. Lepora, Haoran Li, Efi Psomopoulou, Nathan F. Lepora
Robotics
Adaptive Legged Locomotion via Online Learning for Model Predictive Control
Hongyu Zhou, Xiaoyu Zhang and Vasileios Tzoumas
We provide an
algorithm for adaptive legged locomotion via online learning and model predictive control. The
algorithm is composed of two interacting modules: model predictive control (MPC) and online learning of residual dynamics. The residual dynamics can represent modeling errors and external disturbances. We are motivated by the future of autonomy where quadrupeds will autonomously perform complex tasks despite real-world unknown uncertainty, such as unknown payload and uneven terrains. The
algorithm uses random
Fourier features to approximate the residual dynamics in reproducing kernel
Hilbert spaces. Then, it employs MPC based on the current learned model of the residual dynamics. The model is updated online in a self-supervised manner using least squares based on the data collected while controlling the quadruped. The
algorithm enjoys sublinear dynamic regret, defined as the suboptimality against an optimal clairvoyant controller that knows how the residual dynamics. We validate our
algorithm in Gazebo and MuJoCo simulations, where the quadruped aims to track reference trajectories. The Gazebo simulations include constant unknown external forces up to $12\boldsymbol{g}$, where $\boldsymbol{g}$ is the
gravity vector, in flat terrain, slope terrain with $20\degree$ inclination, and rough terrain with $0.25m$ height variation. The MuJoCo simulations include time-varying unknown disturbances with payload up to $8~kg$ and time-varying ground friction coefficients in flat terrain.
Read the full abstract ▾
Robotics
Improved Extended Kalman Filter-Based Disturbance Observers for Exoskeletons
Shilei Li, Dawei Shi, Makoto Iwasaki et al. (+3 more)
The nominal performance of mechanical systems is often degraded by unknown disturbances. A two-degree-of-freedom control structure can decouple nominal performance from disturbance rejection. However, perfect disturbance rejection is unattainable when the disturbance dynamic is unknown. In this work, we reveal an inherent trade-off in disturbance estimation subject to tracking speed and tracking uncertainty. Then, we propose two novel methods to enhance disturbance estimation: an interacting multiple model extended Kalman filter-based disturbance observer and a multi-kernel correntropy extended Kalman filter-based disturbance observer. Experiments on an exoskeleton verify that the proposed two methods improve the tracking accuracy $36.3\%$ and $16.2\%$ in hip joint error, and $46.3\%$ and $24.4\%$ in knee joint error, respectively, compared to the extended Kalman filter-based disturbance observer, in a time-varying interaction force scenario, demonstrating the superiority of the proposed method.
Read the full abstract ▾
All authors Shilei Li, Dawei Shi, Makoto Iwasaki, Yan Ning, Hongpeng Zhou, Ling Shi
Robotics
VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation
Zehao Ni, Yonghao He, Lingfeng Qian et al. (+7 more)
In the context of imitation learning, visuomotor-based
diffusion policy learning is one of the main directions in robotic manipulation. Most of these approaches rely on point clouds as observation inputs and construct scene representations through point clouds feature learning, which enables them to achieve remarkable accuracy. However, the existing literature lacks an in-depth exploration of
vision-only solutions that have significant potential. In this paper, we propose a
Vision-Only and single-view
Diffusion Policy learning method (VO-DP) that leverages pretrained visual foundation models to achieve effective fusion of semantic and
geometric features. We utilize intermediate features from VGGT incorporating semantic features from DINOv2 and
geometric features from Alternating
Attention blocks. Features are fused via cross-
attention and spatially compressed with a CNN to form the input to the policy head. Extensive experiments demonstrate that VO-DP not only outperforms the
vision-only baseline DP significantly but also exhibits distinct performance trends against the point cloud-based method DP3: in simulation tasks, VO-DP achieves an average success rate of 64.6% on par with DP3 64.0% and far higher than DP 34.8%, while in real-world tasks, it reaches 87.9%, outperforming both DP3 67.5% and DP 11.2% by a notable margin. Further robustness evaluations confirm that VO-DP remains highly stable under varying conditions including color, size, background, and lighting. Lastly, we open-source a training library for robotic manipulation. Built on Accelerate, this library supports multi-machine and multi-GPU parallel training, as well as mixed precision training. It is compatible with visuomotor policies such as DP, DP3 and VO-DP, and also supports the RoboTwin simulator.
Read the full abstract ▾
All authors Zehao Ni, Yonghao He, Lingfeng Qian, Jilei Mao, Fa Fu, Wei Sui, Hu Su, Junran Peng, Zhipeng Wang, Bin He
Robotics · Cross-listed from cs.CV
Exploring Conditions for Diffusion models in Robotic Control
Heeseong Shin, Byeongho Heo, Dongyoon Han et al. (+2 more)
While pre-trained visual representations have significantly advanced imitation learning, they are often task-agnostic as they remain frozen during policy learning. In this work, we explore leveraging pre-trained text-to-image
diffusion models to obtain task-adaptive visual representations for robotic control, without fine-tuning the model itself. However, we find that naively applying textual conditions - a successful strategy in other
vision domains - yields minimal or even negative gains in control tasks. We attribute this to the domain gap between the
diffusion model's training data and robotic control environments, leading us to argue for conditions that consider the specific, dynamic visual information required for control. To this end, we propose ORCA, which introduces learnable task
prompts that adapt to the control environment and visual
prompts that capture fine-grained, frame-specific details. Through facilitating task-adaptive representations with our newly devised conditions, our approach achieves state-of-the-art performance on various robotic control
benchmarks, significantly surpassing prior methods.
Read the full abstract ▾
All authors Heeseong Shin, Byeongho Heo, Dongyoon Han, Seungryong Kim, Taekyung Kim
Robotics
Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving
Aron Distelzweig, Faris Janjoš, Oliver Scheel et al. (+3 more)
Traditionally, prediction and planning in autonomous driving (AD) have been treated as separate, sequential modules. Recently, there has been a growing shift towards tighter integration of these components, known as Integrated Prediction and Planning (IPP), with the aim of enabling more informed and adaptive decision-making. However, it remains unclear to what extent this integration actually improves planning performance. In this work, we investigate the role of prediction in IPP approaches, drawing on the widely adopted Val14
benchmark, which encompasses more common driving scenarios with relatively low interaction complexity, and the interPlan
benchmark, which includes highly interactive and out-of-distribution driving situations. Our analysis reveals that even access to perfect future predictions does not lead to better planning outcomes, indicating that current IPP methods often fail to fully exploit future behavior information. Instead, we focus on high-quality proposal generation, while using predictions primarily for collision checks. We find that many imitation learning-based planners struggle to generate realistic and plausible proposals, performing worse than PDM - a simple lane-following approach. Motivated by this observation, we build on PDM with an enhanced proposal generation method, shifting the emphasis towards producing diverse but realistic and high-quality proposals. This proposal-centric approach significantly outperforms existing methods, especially in out-of-distribution and highly interactive settings, where it sets new state-of-the-art results.
Read the full abstract ▾
All authors Aron Distelzweig, Faris Janjoš, Oliver Scheel, Sirish Reddy Varra, Raghu Rajan, Joschka Boedecker
Robotics
VDRive: Leveraging Reinforced VLA and Diffusion Policy for End-to-end Autonomous Driving
Ziang Guo and Zufeng Zhang
In autonomous driving, dynamic environment and corner cases pose significant challenges to the robustness of ego vehicle's state understanding and decision making. We introduce VDRive, a novel pipeline for end-to-end autonomous driving that explicitly models state-action mapping to address these challenges, enabling interpretable and robust decision making. By leveraging the advancement of the state understanding of the
Vision Language Action Model (VLA) with generative
diffusion policy-based action head, our VDRive guides the driving contextually and geometrically. Contextually, VLA predicts future observations through
token generation pre-training, where the observations are represented as discrete codes by a Conditional Vector Quantized Variational Autoencoder (CVQ-VAE). Geometrically, we perform reinforcement learning fine-tuning of the VLA to predict future trajectories and actions based on current driving conditions. VLA supplies the current state
tokens and predicted state
tokens for the action policy head to generate hierarchical actions and trajectories. During policy training, a learned critic evaluates the actions generated by the policy and provides gradient-based feedback, forming an actor-critic framework that enables a reinforcement-based policy learning pipeline. Experiments show that our VDRive achieves state-of-the-art performance in the Bench2Drive closed-loop
benchmark and nuScenes open-loop planning.
Read the full abstract ▾
Robotics · Cross-listed from cs.LG
Towards Robust Zero-Shot Reinforcement Learning
Kexin Zheng, Lauriane Teyssier, Yinan Zheng et al. (+2 more)
The recent development of zero-shot reinforcement learning (RL) has opened a new avenue for learning pre-trained generalist policies that can adapt to arbitrary new tasks in a zero-shot manner. While the popular Forward-Backward representations (FB) and related methods have shown promise in zero-shot RL, we empirically found that their modeling lacks expressivity and that extrapolation errors caused by out-of-distribution (OOD) actions during offline learning sometimes lead to biased representations, ultimately resulting in suboptimal performance. To address these issues, we propose Behavior-REgularizEd Zero-shot RL with Expressivity enhancement (BREEZE), an upgraded FB-based framework that simultaneously enhances learning stability, policy extraction capability, and representation learning quality. BREEZE introduces behavioral regularization in zero-shot RL policy learning, transforming policy optimization into a stable in-sample learning paradigm. Additionally, BREEZE extracts the policy using a task-conditioned
diffusion model, enabling the generation of high-quality and
multimodal action distributions in zero-shot RL settings. Moreover, BREEZE employs expressive
attention-based architectures for representation modeling to capture the complex relationships between environmental dynamics. Extensive experiments on ExORL and D4RL Kitchen demonstrate that BREEZE achieves the best or near-the-best performance while exhibiting superior robustness compared to prior offline zero-shot RL methods. The official implementation is available at: https://github.com/Whiterrrrr/BREEZE.
Read the full abstract ▾
All authors Kexin Zheng, Lauriane Teyssier, Yinan Zheng, Yu Luo, Xianyuan Zhan
Robotics
Towards Automated Chicken Deboning via Learning-based Dynamically-Adaptive 6-DoF Multi-Material Cutting
Zhaodong Yang, Ai-Ping Hu and Harish Ravichandar
Automating chicken shoulder deboning requires precise 6-DoF cutting through a partially occluded, deformable, multi-material joint, since contact with the bones presents serious health and safety risks. Our work makes both systems-level and algorithmic contributions to train and deploy a reactive force-feedback cutting policy that dynamically adapts a nominal trajectory and enables full 6-DoF knife control to traverse the narrow joint gap while avoiding contact with the bones. First, we introduce an open-source custom-built simulator for multi-material cutting that models coupling, fracture, and cutting forces, and supports reinforcement learning, enabling efficient training and rapid prototyping. Second, we design a reusable physical testbed to emulate the chicken shoulder: two rigid "bone" spheres with controllable pose embedded in a softer block, enabling rigorous and repeatable evaluation while preserving essential multi-material characteristics of the target problem. Third, we train and deploy a residual RL policy, with discretized force observations and domain randomization, enabling robust zero-shot sim-to-real transfer and the first demonstration of a learned policy that debones a real chicken shoulder. Our experiments in our simulator, on our physical testbed, and on real chicken shoulders show that our learned policy reliably navigates the joint gap and reduces undesired bone/cartilage contact, resulting in up to a 4x improvement over existing open-loop cutting baselines in terms of success rate and bone avoidance. Our results also illustrate the necessity of force feedback for safe and effective multi-material cutting. The project website is at https://hal-zhaodong-yang.github.io/MultiMaterialWebsite/.
Read the full abstract ▾
Robotics
GaussGym: An open-source real-to-sim framework for learning locomotion from pixels
Alejandro Escontrela, Justin Kerr, Arthur Allshire et al. (+4 more)
We present a novel approach for photorealistic
robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed – exceeding 100,000 steps per second on consumer GPUs – while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve
navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene
datasets (e.g., GrandTour, ARKit), and outputs from generative
video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable
robot learning. All code and data will be open-sourced for the community to build upon.
Videos, code, and data available at https://escontrela.me/gauss_gym/.
Read the full abstract ▾
All authors Alejandro Escontrela, Justin Kerr, Arthur Allshire, Jonas Frey, Rocky Duan, Carmelo Sferrazza, Pieter Abbeel
Robotics
Nauplius Optimisation for Autonomous Hydrodynamics
Shyalan Ramesh, Scott Mann and Alex Stumpf
Autonomous
Underwater vehicles must operate in strong currents, limited acoustic bandwidth, and persistent sensing requirements where conventional swarm optimisation methods are unreliable. This paper formulates an irreversible hydrodynamic deployment problem for Autonomous
Underwater Vehicle (AUV) swarms and presents Nauplius Optimisation for Autonomous Hydrodynamics (NOAH), a novel nature-inspired swarm optimisation
algorithm that combines current-aware drift, irreversible settlement in persistent sensing nodes, and colony-based communication. Drawing inspiration from the behaviour of barnacle nauplii, NOAH addresses the critical limitations of existing swarm
algorithms by providing hydrodynamic awareness, irreversible anchoring mechanisms, and colony-based communication capabilities essential for
underwater exploration missions. The
algorithm establishes a comprehensive foundation for scalable and energy-efficient
underwater swarm robotics with validated performance analysis. Validation studies demonstrate an 86% success rate for permanent anchoring scenarios, providing a unified formulation for hydrodynamic constraints and irreversible settlement behaviours with an empirical study under flow.
Read the full abstract ▾
Robotics
Adaptive Cost-Map-based Path Planning in Partially Unknown Environments with Movable Obstacles
Liviu-Mihai Stan, Ranulfo Bezerra, Shotaro Kojima et al. (+4 more)
Reliable
navigation in disaster-response and other unstructured indoor settings requires
robots not only to avoid obstacles but also to recognise when those obstacles can be pushed aside. We present an adaptive,
LiDAR and odometry-based path-planning framework that embeds this capability into the ROS2 Nav2 stack. A new Movable Obstacles Layer labels all
LiDAR returns missing from a prior static map as tentatively movable and assigns a reduced traversal cost. A companion Slow-Pose Progress Checker monitors the ratio of commanded to actual velocity; when the
robot slows appreciably, the local cost is raised from light to heavy, and on a stall to lethal, prompting the global planner to back out and re-route. Gazebo evaluations on a Scout Mini, spanning isolated objects and cluttered corridors, show higher goal-reach rates and fewer deadlocks than a no-layer baseline, with traversal times broadly comparable. Because the method relies only on planar scans and CPU-level computation, it suits resource-constrained search and rescue
robots and integrates into heterogeneous platforms with minimal engineering. Overall, the results indicate that interaction-aware cost maps are a lightweight, ROS2-native extension for navigating among potentially movable obstacles in unstructured settings. The full implementation will be released as open source athttps://costmap-namo.github.io.
Read the full abstract ▾
All authors Liviu-Mihai Stan, Ranulfo Bezerra, Shotaro Kojima, Tsige Tadesse Alemayoh, Satoshi Tadokoro, Masashi Konyo, Kazunori Ohno
Robotics
ASBI: Leveraging Informative Real-World Data for Active Black-Box Simulator Tuning
Gahee Kim and Takamitsu Matsubara
Black-box simulators are widely used in robotics, but optimizing their parameters remains challenging due to inaccessible likelihoods. Simulation-Based Inference (SBI) tackles this issue using simulation-driven approaches, estimating the posterior from offline real observations and forward simulations. However, in black-box scenarios, preparing observations that contain sufficient information for parameter estimation is difficult due to the unknown relationship between parameters and observations. In this work, we present Active Simulation-Based Inference (ASBI), a parameter estimation framework that uses
robots to actively collect real-world online data to achieve accurate black-box simulator tuning. Our framework optimizes
robot actions to collect informative observations by maximizing information gain, which is defined as the expected reduction in Shannon
entropy between the posterior and the prior. While calculating information gain requires the likelihood, which is inaccessible in black-box simulators, our method solves this problem by leveraging Neural Posterior Estimation (NPE), which leverages a neural network to learn the posterior
estimator. Three simulation experiments quantitatively verify that our method achieves accurate parameter estimation, with posteriors sharply concentrated around the true parameters. Moreover, we show a practical application using a real
robot to estimate the simulation parameters of cubic particles corresponding to two real objects, beads and gravel, with a bucket pouring action.
Read the full abstract ▾
Robotics
Traversability-aware Consistent Situational Graphs for Indoor Localization and Mapping
Jeewon Kim, Minho Oh and Hyun Myung
Scene graphs enhance 3D mapping capabilities in robotics by understanding the relationships between different spatial elements, such as rooms and objects. Recent research extends scene graphs to hierarchical layers, adding and leveraging constraints across these levels. This approach is tightly integrated with pose-graph optimization, improving both localization and mapping accuracy simultaneously. However, when segmenting spatial characteristics, consistently recognizing rooms becomes challenging due to variations in viewpoints and limited field of view (FOV) of sensors. For example, existing real-time approaches often over-segment large rooms into smaller, non-functional spaces that are not useful for localization and mapping due to the time-dependent method. Conversely, their voxel-based room
segmentation method often under-segment in complex cases like not fully enclosed 3D space that are non-traversable for ground
robots or humans, leading to false constraints in pose-graph optimization. We propose a traversability-aware room
segmentation method that considers the interaction between
robots and surroundings, with consistent feasibility of traversability information. This enhances both the semantic coherence and computational efficiency of pose-graph optimization. Improved performance is demonstrated through the re-detection frequency of the same rooms in a
dataset involving repeated traversals of the same space along the same path, as well as the optimization time consumption.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
CuSfM: CUDA-Accelerated Structure-from-Motion
Jingrui Yu, Jun Liu, Kefei Ren et al. (+5 more)
Efficient and accurate camera pose estimation forms the foundational requirement for dense reconstruction in autonomous
navigation, robotic perception, and virtual simulation systems. This paper addresses the challenge via cuSfM, a CUDA-accelerated offline Structure-from-Motion system that leverages GPU parallelization to efficiently employ computationally intensive yet highly accurate feature extractors, generating comprehensive and non-redundant data associations for precise camera pose estimation and globally consistent mapping. The system supports pose optimization, mapping, prior-map localization, and extrinsic refinement. It is designed for offline processing, where computational resources can be fully utilized to maximize accuracy. Experimental results demonstrate that cuSfM achieves significantly improved accuracy and processing speed compared to the widely used COLMAP method across various testing scenarios, while maintaining the high precision and global consistency essential for offline SfM applications. The system is released as an open-source Python wrapper implementation, PyCuSfM, available at https://github.com/nvidia-isaac/pyCuSFM, to facilitate research and applications in computer
vision and robotics.
Read the full abstract ▾
All authors Jingrui Yu, Jun Liu, Kefei Ren, Joydeep Biswas, Rurui Ye, Keqiang Wu, Chirag Majithia, Di Zeng
Robotics
New Location Science Models with Applications to UAV-Based Disaster Relief
Sina Kazemdehbashi, Yanchao Liu and Boris S. Mordukhovich
Natural and human-made disasters can cause severe devastation and claim thousands of lives worldwide. Therefore, developing efficient methods for disaster response and management is a critical task for relief teams. One of the most essential components of effective response is the rapid collection of information about affected areas, damages, and victims. More data translates into better coordination, faster rescue operations, and ultimately, more lives saved. However, in some disasters, such as earthquakes, the communication infrastructure is often partially or completely destroyed, making it extremely difficult for victims to send distress signals and for rescue teams to locate and assist them in time. Unmanned Aerial Vehicles (
UAVs) have emerged as valuable tools in such scenarios. In particular, a fleet of
UAVs can be dispatched from a mobile station to the affected area to facilitate data collection and establish temporary communication networks. Nevertheless, real-world deployment of
UAVs faces several challenges, with adverse weather conditions–especially wind–being among the most significant. To address this, we develop a novel mathematical framework to determine the optimal location of a mobile
UAV station while explicitly accounting for the heterogeneity of the
UAVs and the effect of wind. In particular, we generalize the Sylvester problem to introduce the Sylvester-Fermat-Torricelli (SFT) problem, which captures complex factors such as wind influence,
UAV heterogeneity, and back-and-forth motion within a unified framework. The proposed framework enhances the practicality of
UAV-based disaster response planning by accounting for real-world factors such as wind and
UAV heterogeneity. Experimental results demonstrate that it can reduce wasted operational time by up to 84%, making post-disaster missions significantly more efficient and effective.
Read the full abstract ▾
Robotics
PolyFly: Polytopic Optimal Planning for Collision-Free Cable-Suspended Aerial Payload Transportation
Mrunal Sarvaiya, Guanrui Li and Giuseppe Loianno
Aerial transportation
robots using suspended cables have emerged as versatile platforms for disaster response and rescue operations. To maximize the capabilities of these systems,
robots need to aggressively fly through tightly constrained environments, such as dense forests and structurally unsafe buildings, while minimizing flight time and avoiding obstacles. Existing methods geometrically over-approximate the vehicle and obstacles, leading to conservative maneuvers and increased flight times. We eliminate these restrictions by proposing PolyFly, an optimal global planner which considers a non-conservative representation for aerial transportation by modeling each physical component of the environment, and the
robot (quadrotor, cable and payload), as independent polytopes. We further increase the model accuracy by incorporating the attitude of the physical components by constructing orientation-aware polytopes. The resulting optimal control problem is efficiently solved by converting the polytope constraints into smooth differentiable constraints via duality theory. We compare our method against the existing state-of-the-art approach in eight maze-like environments and show that PolyFly produces faster trajectories in each scenario. We also experimentally validate our proposed approach on a real quadrotor with a suspended payload, demonstrating the practical reliability and accuracy of our method.
Read the full abstract ▾