Robotics
FIRE-VLM: A Vision-Language-Driven Reinforcement Learning Framework for UAV Wildfire Tracking in a Physics-Grounded Fire Digital Twin
Chris Webb, Mobin Habibpour, Mayamin Hamid Raha et al. (+3 more)
Wildfire monitoring demands autonomous systems capable of
reasoning under extreme visual degradation, rapidly evolving physical dynamics, and scarce real-world training data. Existing
UAV navigation approaches rely on simplified simulators and supervised perception pipelines, and lack
embodied agents interacting with physically realistic fire environments. We introduce FIRE-VLM, the first end-to-end
vision-
language model (VLM) guided reinforcement learning (RL) framework trained entirely within a high-fidelity, physics-grounded wildfire digital twin. Built from USGS Digital Elevation Model (DEM) terrain, LANDFIRE fuel inventories, and semi-physical fire-spread solvers, this twin captures terrain-induced runs, wind-driven acceleration, smoke plume occlusion, and dynamic fuel consumption. Within this environment, a PPO
agent with dual-view
UAV sensing is guided by a CLIP-style VLM. Wildfire-specific semantic
alignment scores, derived from a single
prompt describing active fire and smoke plumes, are integrated as potential-based reward shaping signals. Our contributions are: (1) a GIS-to-simulation pipeline for constructing wildfire digital twins; (2) a VLM-guided RL
agent for
UAV firefront tracking; and (3) a wildfire-aware reward design that combines physical terms with VLM semantics. Across five digital-twin evaluation tasks, our VLM-guided policy reduces time-to-detection by up to 6 times, increases time-in-FOV, and is, to our knowledge, the first RL-based
UAV wildfire monitoring system demonstrated in kilometer-scale, physics-grounded digital-twin fires.
Read the full abstract ▾
All authors Chris Webb, Mobin Habibpour, Mayamin Hamid Raha, Ali Reza Tavakkoli, Janice Coen, Fatemeh Afghah
Robotics
Cost-Effective Radar Sensors for Field-Based Water Level Monitoring with Sub-Centimeter Accuracy
Anna Zavei-Boroda, J. Toby Minear, Kyle Harlow et al. (+2 more)
Water level monitoring is critical for flood management, water resource allocation, and ecological assessment, yet traditional methods remain costly and limited in coverage. This work explores radar-based sensing as a low-cost alternative for water level estimation, leveraging its non-contact nature and robustness to environmental conditions. Commercial radar sensors are evaluated in real-world field tests, applying statistical filtering techniques to improve accuracy. Results show that a single radar sensor can achieve centimeter-scale precision with minimal calibration, making it a practical solution for autonomous water monitoring using drones and robotic platforms.
Read the full abstract ▾
All authors Anna Zavei-Boroda, J. Toby Minear, Kyle Harlow, Dusty Woods, Christoffer Heckman
Robotics
Towards Zero-Knowledge Task Planning via a Language-based Approach
Liam Merz Hoffmeister, Brian Scassellati and Daniel Rakita
In this work, we introduce and formalize the Zero-Knowledge Task Planning (ZKTP) problem, i.e., formulating a sequence of actions to achieve some goal without task-specific knowledge. Additionally, we present a first investigation and approach for ZKTP that leverages a large
language model (LLM) to decompose natural
language instructions into subtasks and generate behavior trees (BTs) for execution. If errors arise during task execution, the approach also uses an LLM to adjust the BTs on-the-fly in a refinement loop. Experimental validation in the AI2-THOR simulator demonstrate our approach's effectiveness in improving overall task performance compared to alternative approaches that leverage task-specific knowledge. Our work demonstrates the potential of LLMs to effectively address several aspects of the ZKTP problem, providing a robust framework for automated behavior generation with no task-specific setup.
Read the full abstract ▾
Robotics · Cross-listed from eess.SY
Modeling and Control for UAV with Off-center Slung Load
Zongyang Lv, Yanmei Jia, Yongqing Liu et al. (+3 more)
Unmanned aerial vehicle (
UAV) with slung load system is a classic air transportation system. In practical applications, the suspension point of the slung load does not always align with the center of mass (CoM) of the
UAV due to mission requirements or mechanical interference. This offset creates coupling in the system's nonlinear dynamics which leads to a complicated motion control problem. In existing research, modeling of the system are performed about the
UAV's CoM. In this work we use the point of suspension instead. Based on the new model, a cascade control strategy is developed. In the middle-loop controller, the acceleration of the suspension point is used to regulate the swing angle of the slung load without the need for considering the coupling between the slung load and the
UAV. An inner-loop controller is designed to track the
UAV's attitude without the need of simplification on the coupling effects. We prove local exponential stability of the closed-loop using Lyapunov approach. Finally, simulations and experiments are conducted to validate the proposed control system.
Read the full abstract ▾
All authors Zongyang Lv, Yanmei Jia, Yongqing Liu, Alan F. Lynch, Qing Zhao, Yuhu Wu
Robotics
Lunar Rover Cargo Transport: Mission Concept and Field Test
Alexander Krawciw, Nicolas Olmedo, Faizan Rehmatullah et al. (+3 more)
In future operations on the lunar surface, automated vehicles will be required to transport cargo between known locations. Such vehicles must be able to navigate precisely in safe regions to avoid natural hazards, human-constructed infrastructure, and dangerous dark shadows. Rovers must be able to park their cargo autonomously within a small tolerance to achieve a successful pickup and delivery. In this field test,
Lidar Teach and Repeat provides an ideal autonomy solution for transporting cargo in this way. A one-tonne path-to-flight rover was driven in a semi-autonomous remote-control mode to create a network of safe paths. Once the route was taught, the rover immediately repeated the entire network of paths autonomously while carrying cargo. The closed-loop performance is accurate enough to align the vehicle to the cargo and pick it up. This field report describes a two-week deployment at the Canadian Space Agency's Analogue Terrain, culminating in a simulated lunar operation to evaluate the system's capabilities. Successful cargo collection and delivery were demonstrated in harsh environmental conditions.
Read the full abstract ▾
All authors Alexander Krawciw, Nicolas Olmedo, Faizan Rehmatullah, Maxime Desjardins-Goulet, Pascal Toupin, Timothy D. Barfoot
Robotics
Revisiting Continuous-Time Trajectory Estimation via Gaussian Processes and the Magnus Expansion
Timothy Barfoot, Cedric Le Gentil and Sven Lilge
Continuous-time state estimation has been shown to be an effective means of (i) handling asynchronous and high-rate measurements, (ii) introducing smoothness to the estimate, (iii) post hoc querying the estimate at times other than those of the measurements, and (iv) addressing certain observability issues related to scanning-while-moving sensors. A popular means of representing the trajectory in continuous time is via a Gaussian process (GP) prior, with the prior's mean and covariance functions generated by a linear time-varying (LTV)
stochastic differential equation (SDE) driven by white noise. When the state comprises elements of Lie groups, previous works have resorted to a patchwork of local GPs each with a linear time-invariant SDE kernel, which while effective in practice, lacks theoretical elegance. Here we revisit the full LTV GP approach to continuous-time trajectory estimation, deriving a global GP prior on Lie groups via the Magnus expansion, which offers a more elegant and general solution. We provide a numerical comparison between the two approaches and discuss their relative merits.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
CageDroneRF: A Large-Scale RF Benchmark and Toolkit for Drone Perception
Mohammad Rostami, Atik Faysal, Hongtao Xia et al. (+3 more)
We present CageDroneRF (CDRF), a large-scale
benchmark for
Radio-Frequency (RF) drone detection and identification built from real-world captures and systematically generated synthetic variants. CDRF addresses the scarcity and limited diversity of existing RF
datasets by coupling extensive raw recordings with a principled augmentation pipeline that (i)~precisely controls Signal-to-Noise Ratio (SNR), (ii)~injects interfering emitters, and (iii)~applies frequency shifts with label-consistent bounding-box recomputation for detection. The
dataset spans a wide range of contemporary drone models, many of which are unavailable in current public
datasets, and diverse acquisition conditions, derived from data collected at the Rowan University campus and within a controlled RF-cage facility. CDRF is released with interoperable open-source tools for data generation, preprocessing, augmentation, and evaluation that also operate on existing public
benchmarks. It enables standardized benchmarking for classification, open-set recognition, and object detection, supporting rigorous comparisons and reproducible pipelines. By releasing this comprehensive
benchmark and tooling, we aim to accelerate progress toward robust, generalizable RF perception models.
Read the full abstract ▾
All authors Mohammad Rostami, Atik Faysal, Hongtao Xia, Hadi Kasasbeh, Ziang Gao, Huaxia Wang
Robotics · Cross-listed from math.DS
Nonlinear Spectral Modeling and Control of Soft-Robotic Muscles from Data
Leonardo Bettini, Amirhossein Kazemipour, Robert K. Katzschmann et al. (+1 more)
Artificial muscles are essential for compliant musculoskeletal robotics but complicate control due to nonlinear multiphysics dynamics. Hydraulically amplified electrostatic (HASEL) actuators, a class of soft artificial muscles, offer high performance but exhibit
memory effects and hysteresis. Here we present a data-driven reduction and control strategy grounded in spectral submanifold (SSM) theory. In the adiabatic regime, where inputs vary slowly relative to intrinsic transients, trajectories rapidly converge to a low-dimensional slow
manifold. We learn an explicit input-to-output map on this
manifold from forced-response trajectories alone, avoiding decay experiments that can trigger hysteresis. We deploy the SSM-based model for real-time control of an antagonistic HASEL-clutch joint. This approach yields a substantial reduction in tracking error compared to feedback-only and feedforward-only baselines under identical settings. This record-and-control workflow enables rapid characterization and high-performance control of soft muscles and muscle-driven joints without detailed physics-based modeling.
Read the full abstract ▾
All authors Leonardo Bettini, Amirhossein Kazemipour, Robert K. Katzschmann, George Haller
Robotics
A High-Fidelity Digital Twin for Robotic Manipulation Based on 3D Gaussian Splatting
Ziyang Sun, Lingfan Bao, Tianhu Peng et al. (+2 more)
Developing high-fidelity, interactive digital twins is crucial for enabling closed-loop motion planning and reliable real-world
robot execution, which are essential to advancing sim-to-real transfer. However, existing approaches often suffer from slow reconstruction, limited visual fidelity, and difficulties in converting photorealistic models into planning-ready collision geometry. We present a practical framework that constructs high-quality digital twins within minutes from sparse RGB inputs. Our system employs 3D Gaussian Splatting (3DGS) for fast, photorealistic reconstruction as a unified scene representation. We enhance 3DGS with visibility-aware semantic fusion for accurate 3D labelling and introduce an efficient, filter-based geometry conversion method to produce collision-ready models seamlessly integrated with a Unity-ROS2-MoveIt physics engine. In experiments with a Franka Emika Panda
robot performing pick-and-place tasks, we demonstrate that this enhanced
geometric accuracy effectively supports robust manipulation in real-world trials. These results demonstrate that 3DGS-based digital twins, enriched with semantic and
geometric consistency, offer a fast, reliable, and scalable path from perception to manipulation in unstructured environments.
Read the full abstract ▾
All authors Ziyang Sun, Lingfan Bao, Tianhu Peng, Jingcheng Sun, Chengxu Zhou
Robotics · Cross-listed from cs.CL
Limited Linguistic Diversity in Embodied AI Datasets
Selma Wanna, Agnes Luhtaru, Jonathan Salfity et al. (+4 more)
Language plays a critical role in
Vision-
Language-Action (VLA) models, yet the linguistic characteristics of the
datasets used to train and evaluate these systems remain poorly documented. In this work, we present a systematic
dataset audit of several widely used VLA corpora, aiming to characterize what kinds of instructions these
datasets actually contain and how much linguistic variety they provide. We quantify instruction
language along complementary dimensions–including lexical variety, duplication and overlap, semantic similarity, and syntactic complexity. Our analysis shows that many
datasets rely on highly repetitive, template-like commands with limited structural variation, yielding a narrow distribution of instruction forms. We position these findings as descriptive documentation of the
language signal available in current VLA training and evaluation data, intended to support more detailed
dataset reporting, more principled
dataset selection, and targeted curation or augmentation strategies that broaden
language coverage.
Read the full abstract ▾
All authors Selma Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek, Mitch Pryor
Robotics
Dual-quaternion learning control for autonomous vehicle trajectory tracking with safety guarantees
Omayra Yago Nieto, Alexandre Anahory Simoes, Juan I. Giribet et al. (+1 more)
We propose a learning-based trajectory tracking controller for autonomous robotic platforms whose motion can be described kinematically on $\mathrm{SE}(3)$. The controller is formulated in the dual quaternion framework and operates at the velocity level, assuming direct command of angular and linear velocities, as is standard in many aerial vehicles and omnidirectional mobile
robots. Gaussian Process (GP) regression is integrated into a
geometric feedback law to learn and compensate online for unknown, state-dependent disturbances and modeling imperfections affecting both attitude and position, while preserving the algebraic structure and coupling properties inherent to rigid-body motion. The proposed approach does not rely on explicit parametric models of the unknown effects, making it well-suited for robotic systems subject to sensor-induced disturbances, unmodeled actuation couplings, and environmental uncertainties. A Lyapunov-based analysis establishes probabilistic ultimate boundedness of the pose tracking error under bounded GP uncertainty, providing formal stability guarantees for the learning-based controller. Simulation results demonstrate accurate and smooth trajectory tracking in the presence of realistic, localized disturbances, including correlated rotational and translational effects arising from magnetometer perturbations. These results illustrate the potential of combining
geometric modeling and probabilistic learning to achieve robust, data-efficient pose control for autonomous robotic systems.
Read the full abstract ▾
All authors Omayra Yago Nieto, Alexandre Anahory Simoes, Juan I. Giribet, Leonardo Colombo
Robotics
HEXAR: a Hierarchical Explainability Architecture for Robots
Tamlin Love, Ferran Gebellí, Pradip Pramanick et al. (+5 more)
As robotic systems become increasingly complex, the need for explainable decision-making becomes critical. Existing explainability approaches in robotics typically either focus on individual modules, which can be difficult to query from the perspective of high-level behaviour, or employ monolithic approaches, which do not exploit the modularity of robotic architectures. We present HEXAR (Hierarchical EXplainability Architecture for
Robots), a novel framework that provides a plug-in, hierarchical approach to generate explanations about robotic systems. HEXAR consists of specialised component explainers using diverse explanation techniques (e.g., LLM-based
reasoning, causal models, feature importance, etc) tailored to specific
robot modules, orchestrated by an explainer selector that chooses the most appropriate one for a given query. We implement and evaluate HEXAR on a TIAGo
robot performing assistive tasks in a home environment, comparing it against end-to-end and aggregated baseline approaches across 180 scenario-query variations. We observe that HEXAR significantly outperforms baselines in root cause identification, incorrect information exclusion, and runtime, offering a promising direction for transparent autonomous systems.
Read the full abstract ▾
All authors Tamlin Love, Ferran Gebellí, Pradip Pramanick, Antonio Andriella, Guillem Alenyà, Anais Garrell, Raquel Ros, Silvia Rossi
Robotics
A Fast Semidefinite Convex Relaxation for Optimal Control Problems With Spatio-Temporal Constraints
Shiying Dong, Zhipeng Shen, Rudolf Reiter et al. (+4 more)
Solving optimal control problems (OCPs) of autonomous
agents operating under spatial and temporal constraints fast and accurately is essential in applications ranging from eco-driving of autonomous vehicles to quadrotor
navigation. However, the nonlinear programs approximating the OCPs are inherently nonconvex due to the coupling between the dynamics and the event timing, and therefore, they are challenging to solve. Most approaches address this challenge by predefining waypoint times or just using nonconvex trajectory optimization, which simplifies the problem but often yields suboptimal solutions. To significantly improve the numerical properties, we propose a formulation with a time-scaling direct multiple shooting scheme that partitions the prediction horizon into segments aligned with characteristic time constraints. Moreover, we develop a fast semidefinite-programming-based convex relaxation that exploits the sparsity pattern of the lifted formulation. Comprehensive simulation studies demonstrate the solution optimality and computational efficiency. Furthermore, real-world experiments on a quadrotor waypoint flight task with constrained open time windows validate the practical applicability of the approach in complex environments.
Read the full abstract ▾
All authors Shiying Dong, Zhipeng Shen, Rudolf Reiter, Hailong Huang, Bingzhao Gao, Hong Chen, Wen-Hua Chen
Robotics
SOP: A Scalable Online Post-Training System for Vision-Language-Action Models
Mingjie Pan, Siyuan Feng, Qinglin Zhang et al. (+9 more)
Vision-
language-action (VLA) models achieve strong generalization through large-scale pre-training, but real-world deployment requires expert-level task proficiency in addition to broad generality. Existing post-training approaches for VLA models are typically offline, single-
robot, or task-specific, limiting effective on-policy adaptation and scalable learning from real-world interaction. We introduce a Scalable Online Post-training (SOP) system that enables online, distributed, multi-task post-training of generalist VLA models directly in the physical world. SOP tightly couples execution and learning through a closed-loop architecture in which a fleet of
robots continuously streams on-policy experience and human intervention signals to a centralized cloud learner, and asynchronously receives updated policies. This design supports
prompt on-policy correction, scales experience collection through parallel deployment, and preserves generality during adaptation. SOP is agnostic to the choice of post-training
algorithm; we instantiate it with both interactive imitation learning (HG-DAgger) and reinforcement learning (RECAP). Across a range of real-world manipulation tasks including cloth folding, box assembly, and grocery restocking, we show that SOP substantially improves the performance of large pretrained VLA models while maintaining a single shared policy across tasks. Effective post-training can be achieved within hours of real-world interaction, and performance scales near-linearly with the number of
robots in the fleet. These results suggest that tightly coupling online learning with fleet-scale deployment is instrumental to enabling efficient, reliable, and scalable post-training of generalist
robot policies in the physical world.
Read the full abstract ▾
All authors Mingjie Pan, Siyuan Feng, Qinglin Zhang, Xinchen Li, Jianheng Song, Chendi Qu, Yi Wang, Chuankang Li, Ziyu Xiong, Zhi Chen, Yi Liu, Jianlan Luo
Robotics
PiDR: Physics-Informed Inertial Dead Reckoning for Autonomous Platforms
Arup Kumar Sahoo and Itzik Klein
A fundamental requirement for full autonomy is the ability to sustain accurate
navigation in the absence of external data, such as GNSS signals or visual information. In these challenging environments, the platform must rely exclusively on inertial sensors, leading to pure inertial
navigation. However, the inherent noise and other error terms of the inertial sensors in such real-world scenarios will cause the
navigation solution to drift over time. Although conventional deep-learning models have emerged as a possible approach to inertial
navigation, they are inherently black-box in nature. Furthermore, they struggle to learn effectively with limited supervised sensor data and often fail to preserve physical principles. To address these limitations, we propose PiDR, a physics-informed inertial dead-reckoning framework for autonomous platforms in situations of pure inertial
navigation. PiDR offers transparency by explicitly integrating inertial
navigation principles into the network training process through the physics-informed residual component. PiDR plays a crucial role in mitigating abrupt trajectory deviations even under limited or sparse supervision. We evaluated PiDR on real-world
datasets collected by a mobile
robot and an autonomous
underwater vehicle. We obtained more than 29% positioning improvement in both
datasets, demonstrating the ability of PiDR to generalize different platforms operating in various environments and dynamics. Thus, PiDR offers a robust, lightweight, yet effective architecture and can be deployed on resource-constrained platforms, enabling real-time pure inertial
navigation in adverse scenarios.
Read the full abstract ▾
Robotics
Validating Generalist Robots with Situation Calculus and STL Falsification
Changwen Li, Rongjie Yan, Chih-Hong Cheng et al. (+1 more)
Generalist
robots are becoming a reality, capable of interpreting natural
language instructions and executing diverse operations. However, their validation remains challenging because each task induces its own operational context and correctness specification, exceeding the assumptions of traditional validation methods. We propose a two-layer validation framework that combines abstract
reasoning with concrete system falsification. At the abstract layer, situation calculus models the world and derives weakest preconditions, enabling constraint-aware combinatorial testing to systematically generate diverse, semantically valid world-task configurations with controllable coverage strength. At the concrete layer, these configurations are instantiated for simulation-based falsification with STL monitoring. Experiments on tabletop manipulation tasks show that our framework effectively uncovers failure cases in the NVIDIA GR00T controller, demonstrating its promise for validating general-purpose
robot autonomy.
Read the full abstract ▾
All authors Changwen Li, Rongjie Yan, Chih-Hong Cheng, Jian Zhang
Robotics
A Bi-directional Adaptive Framework for Agile UAV Landing
Chunhui Zhao, Xirui Kao, Yilin Lu et al. (+1 more)
Autonomous landing on mobile platforms is crucial for extending quadcopter operational flexibility, yet conventional methods are often too inefficient for highly dynamic scenarios. The core limitation lies in the prevalent “track-then-descend” paradigm, which treats the platform as a passive target and forces the quadcopter to perform complex, sequential maneuvers. This paper challenges that paradigm by introducing a bi-directional cooperative landing framework that redefines the roles of the vehicle and the platform. The essential innovation is transforming the problem from a single-
agent tracking challenge into a coupled system optimization. Our key insight is that the mobile platform is not merely a target, but an active
agent in the landing process. It proactively tilts its surface to create an optimal, stable terminal attitude for the approaching quadcopter. This active cooperation fundamentally breaks the sequential model by parallelizing the
alignment and descent phases. Concurrently, the quadcopter's planning pipeline focuses on generating a time-optimal and dynamically feasible trajectory that minimizes energy consumption. This bi-directional coordination allows the system to execute the recovery in an agile manner, characterized by aggressive trajectory tracking and rapid state synchronization within transient windows. The framework's effectiveness, validated in dynamic scenarios, significantly improves the efficiency, precision, and robustness of autonomous quadrotor recovery in complex and time-constrained missions.
Read the full abstract ▾
All authors Chunhui Zhao, Xirui Kao, Yilin Lu, Yang Lyu
Robotics
Learning to Act Robustly with View-Invariant Latent Actions
Youngjoon Jeong, Junha Chun and Taesup Kim
Vision-based robotic policies often struggle with even minor viewpoint changes, underscoring the need for view-invariant visual representations. This challenge becomes more pronounced in real-world settings, where viewpoint variability is unavoidable and can significantly disrupt policy performance. Existing methods typically learn invariance from multi-view observations at the scene level, but such approaches rely on visual appearance and fail to incorporate the physical dynamics essential for robust generalization. We propose View-Invariant Latent Action (VILA), which models a latent action capturing transition patterns across trajectories to learn view-invariant representations grounded in physical dynamics. VILA aligns these latent actions across viewpoints using an action-guided objective based on ground-truth action sequences. Experiments in both simulation and the real world show that VILA-based policies generalize effectively to unseen viewpoints and transfer well to new tasks, establishing VILA as a strong pretraining framework that improves robustness and downstream learning performance.
Read the full abstract ▾
Robotics
Parameter-Robust MPPI for Safe Online Learning of Unknown Parameters
Matti Vahs, Jaeyoun Choi, Niklas Schmid et al. (+2 more)
Robots deployed in dynamic environments must remain safe even when key physical parameters are uncertain or change over time. We propose Parameter-Robust Model Predictive Path Integral (PRMPPI) control, a framework that integrates online parameter learning with probabilistic safety constraints. PRMPPI maintains a particle-based belief over parameters via Stein Variational Gradient Descent, evaluates safety constraints using Conformal Prediction, and optimizes both a nominal performance-driven and a safety-focused backup trajectory in parallel. This yields a controller that is cautious at first, improves performance as parameters are learned, and ensures safety throughout. Simulation and hardware experiments demonstrate higher success rates, lower tracking error, and more accurate parameter estimates than baselines.
Read the full abstract ▾
All authors Matti Vahs, Jaeyoun Choi, Niklas Schmid, Jana Tumova, Chuchu Fan
Robotics
LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environments
Sara Micol Ferraina, Michele Brienza, Francesco Argenziano et al. (+4 more)
Tracking objects that move within dynamic environments is a core challenge in robotics. Recent research has advanced this topic significantly; however, many existing approaches remain inefficient due to their reliance on heavy foundation models. To address this limitation, we propose LOST-3DSG, a lightweight open-vocabulary 3D scene graph designed to track dynamic objects in real-world environments. Our method adopts a semantic approach to entity tracking based on word2vec and sentence
embeddings, enabling an open-vocabulary representation while avoiding the necessity of storing dense CLIP visual features. As a result, LOST-3DSG achieves superior performance compared to approaches that rely on high-dimensional visual
embeddings. We evaluate our method through qualitative and quantitative experiments conducted in a real 3D environment using a TIAGo
robot. The results demonstrate the effectiveness and efficiency of LOST-3DSG in dynamic object tracking. Code and supplementary material are publicly available on the project website at https://lab-rococo-sapienza.github.io/lost-3dsg/.
Read the full abstract ▾
All authors Sara Micol Ferraina, Michele Brienza, Francesco Argenziano, Emanuele Musumeci, Vincenzo Suriani, Domenico D. Bloisi, Daniele Nardi
Robotics
Warm-Starting Collision-Free Model Predictive Control With Object-Centric Diffusion
Arthur Haffemayer, Alexandre Chapin, Armand Jordana et al. (+4 more)
Acting in cluttered environments requires predicting and avoiding collisions while still achieving precise control. Conventional optimization-based controllers can enforce physical constraints, but they struggle to produce feasible solutions quickly when many obstacles are present.
Diffusion models can generate diverse trajectories around obstacles, yet prior approaches lacked a general and efficient way to condition them on scene structure. In this paper, we show that combining
diffusion-based warm-starting conditioned with a latent object-centric representation of the scene and with a collision-aware model predictive controller (MPC) yields reliable and efficient motion generation under strict time limits. Our approach conditions a
diffusion transformer on the system state, task, and surroundings, using an object-centric slot
attention mechanism to provide a compact obstacle representation suitable for control. The sampled trajectories are refined by an optimal control problem that enforces rigid-body dynamics and signed-distance collision constraints, producing feasible motions in real time. On
benchmark tasks, this hybrid method achieved markedly higher success rates and lower latency than sampling-based planners or either component alone. Real-
robot experiments with a torque-controlled Panda confirm reliable and safe execution with MPC.
Read the full abstract ▾
All authors Arthur Haffemayer, Alexandre Chapin, Armand Jordana, Krzysztof Wojciechowski, Florent Lamiraux, Nicolas Mansard, Vladimir Petrik
Robotics
Soft Responsive Materials Enhance Humanoid Safety
Chunzheng Wang, Yiyuan Zhang, Annan Tang et al. (+12 more)
Humanoid robots are envisioned as general-purpose platforms in human-centered environments, yet their deployment is limited by vulnerability to falls and the risks posed by rigid metal-plastic structures to people and surroundings. We introduce a soft-rigid co-design framework that leverages non-Newtonian fluid-based soft responsive materials to enhance
humanoid safety. The material remains compliant during normal interaction but rapidly stiffens under impact, absorbing and dissipating fall-induced forces. Physics-based simulations guide protector placement and thickness and enable learning of active fall policies. Applied to a 42 kg life-size
humanoid, the protector markedly reduces peak impact and allows repeated falls without hardware damage, including drops from 3 m and tumbles down long staircases. Across diverse scenarios, the approach improves
robot robustness and environmental safety. By uniting responsive materials, structural co-design, and learning-based control, this work advances interact-safe, industry-ready
humanoid robots.
Read the full abstract ▾
All authors Chunzheng Wang, Yiyuan Zhang, Annan Tang, Ziqiu Zeng, Haoran Chen, Quan Gao, Zixuan Zhuang, Boyu Li, Zhilin Xiong, Aoqian Zhang, Ce Hao, Siyuan Luo, Tongyang Zhao, Cecilia Laschi, Fan Shi
Robotics
Reinforcement Learning for Follow-the-Leader Robotic Endoscopic Navigation via Synthetic Data
Sicong Gao, Chen Qian, Laurence Xian et al. (+3 more)
Autonomous
navigation is crucial for both medical and industrial endoscopic
robots, enabling safe and efficient exploration of narrow tubular environments without continuous human intervention, where avoiding contact with the inner walls has been a longstanding challenge for prior approaches. We present a follow-the-leader endoscopic
robot based on a flexible continuum structure designed to minimize contact between the endoscope body and intestinal walls, thereby reducing patient discomfort. To achieve this objective, we propose a
vision-based deep reinforcement learning framework guided by
monocular depth estimation. A realistic intestinal simulation environment was constructed in NVIDIA Omniverse to train and evaluate autonomous
navigation strategies. Furthermore, thousands of synthetic intraluminal images were generated using NVIDIA Replicator to fine-tune the Depth Anything model, enabling dense three-dimensional perception of the intestinal environment with a single
monocular camera. Subsequently, we introduce a geometry-aware reward and penalty mechanism to enable accurate lumen tracking. Compared with the original Depth Anything model, our method improves $\delta_{1}$ depth accuracy by 39.2% and reduces the
navigation J-index by 0.67 relative to the second-best method, demonstrating the robustness and effectiveness of the proposed approach.
Read the full abstract ▾
All authors Sicong Gao, Chen Qian, Laurence Xian, Liao Wu, Maurice Pagnucco, Yang Song
Robotics
Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation
Zhe Zhao, Haoyu Dong, Zhengmao He et al. (+3 more)
Human-like dexterous hands with multiple fingers offer human-level manipulation capabilities, but training control policies that can directly deploy on real hardware remains difficult due to contact-rich physics and imperfect actuation. We close this gap with a practical sim-to-real reinforcement learning (RL) framework that utilizes dense tactile feedback combined with joint torque sensing to explicitly regulate physical interactions. To enable effective sim-to-real transfer, we introduce (i) a computationally fast tactile simulation that computes distances between dense virtual tactile units and the object via parallel forward kinematics, providing high-rate, high-resolution touch signals needed by RL; (ii) a current-to-torque calibration that eliminates the need for torque sensors on dexterous hands by mapping motor current to joint torque; and (iii) actuator dynamics modeling to bridge the actuation gaps with randomization of non-ideal effects such as backlash, torque-speed saturation. Using an asymmetric actor-critic PPO pipeline trained entirely in simulation, our policies deploy directly to a five-finger hand. The resulting policies demonstrated two essential skills: (1) command-based, controllable grasp force tracking, and (2) reorientation of objects in the hand, both of which were robustly executed without fine-tuning on the
robot. By combining tactile and torque in the observation space with effective sensing/actuation modeling, our system provides a practical solution to achieve reliable dexterous manipulation. To our knowledge, this is the first demonstration of controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred zero-shot on real hardware.
Read the full abstract ▾
All authors Zhe Zhao, Haoyu Dong, Zhengmao He, Yang Li, Xinyu Yi, Zhibin Li