Robotics
RubbleSim: A Photorealistic Structural Collapse Simulator for Confined Space Mapping
Constantine Frost, Chad Council, Margaret McGuinness et al. (+1 more)
Despite well-reported instances of
robots being used in disaster response, there is scant published data on the internal composition of the void spaces within structural collapse incidents. Data collected during these incidents is mired in legal constraints, as ownership is often tied to the responding agencies, with little hope of public release for research. While engineered rubble piles are used for training, these sites are also reluctant to release information about their proprietary training grounds. To overcome this access challenge, we present RubbleSim – an open-source, reconfigurable simulator for photorealistic void space exploration. The design of the simulation assets is directly informed by visits to numerous training rubble sites at differing levels of complexity. The simulator is implemented in Unity with multi-operating system support. The simulation uses a physics-based approach to build
stochastic rubble piles, allowing for rapid iteration between simulation worlds while retaining absolute knowledge of the ground truth. Using RubbleSim, we apply a state-of-the-art structure-from-motion
algorithm to illustrate how perception performance degrades under challenging visual conditions inside the emulated void spaces. Pre-built binaries and source code to implement are available online: https://github.com/mit-ll/rubble_pile_simulator.
Read the full abstract ▾
All authors Constantine Frost, Chad Council, Margaret McGuinness, Nathaniel Hanson
Robotics
A Parameter-Linear Formulation of the Optimal Path Following Problem for Robotic Manipulator
Tobias Marauli, Hubert Gattringer and Andreas Mueller
In this paper the computational challenges of time-optimal path following are addressed. The standard approach is to minimize the travel time, which inevitably leads to singularities at zero path speed, when reformulating the optimization problem in terms of a path parameter. Thus, smooth trajectory generation while maintaining a low computational effort is quite challenging, since the singularities have to be taken into account. To this end, a different approach is presented in this paper. This approach is based on maximizing the path speed along a prescribed path. Furthermore, the approach is capable of planning smooth trajectories numerically efficient. Moreover, the discrete reformulation of the underlying problem is linear in optimization variables.
Read the full abstract ▾
Robotics
Simultaneous Stiffness and Trajectory Optimization for Energy Minimization of Pick-and-Place Tasks of SEA-Actuated Parallel Kinematic Manipulators
Thomas Kordik, Hubert Gattringer and Andreas Mueller
A major field of industrial
robot applications deals with repetitive tasks that alternate between operating points. For these so-called pick-and-place operations, parallel kinematic manipulators (PKM) are frequently employed. These tasks tend to automatically run for a long period of time and therefore minimizing energy consumption is always of interest. Recent research addresses this topic by the use of elastic elements and particularly series elastic actuators (SEA). This paper explores the possibilities of minimizing energy consumption of SEA actuated PKM performing pick-and-place tasks. The basic idea is to excite eigenmotions that result from the actuator springs and exploit their oscillating characteristics. To this end, a prescribed cyclic pick-and-place operation is analyzed and a dynamic model of SEA driven PKM is derived. Subsequently, an energy minimizing optimal control problem is formulated where operating trajectories as well as SEA stiffnesses are optimized simultaneously. Here, optimizing the actuator stiffness does not account for variable stiffness actuators. It serves as a tool for the design and dimensioning process. The hypothesis on energy reduction is tested on two (parallel)
robot applications where redundant actuation is also addressed. The results confirm the validity of this approach.
Read the full abstract ▾
Robotics
Learning On The Job: Zero-Shot Task Execution under Parametric Uncertainty via Trajectory-Parametrized Dual Control
Victor Vantilborgh, Hrishikesh Sathyanarayan, Guillaume Crevecoeur et al. (+2 more)
Model-based control can achieve reliable task performance, but its effectiveness depends on the accuracy of the underlying model.
Robots operating under model uncertainty must often adapt to previously unseen payloads, objects, and interaction dynamics to complete a task successfully. Conventional approaches typically rely on a dedicated task- and control-agnostic excitation phase for estimating physics parameters, delaying task execution and collecting potentially irrelevant data. In this paper, we instead consider zero-shot task execution under parametric uncertainty, where online model learning and control proceed concurrently during task execution. Our approach formulates reference generation within a dual control framework to produce task-relevant, informative trajectories that reduce parameter uncertainty in directions critical to task success. We predefine a feedback policy with an explicit parameter adaptation law and optimize the reference through the resulting adaptive closed-loop dynamics. We propose two formulations: 1) minimizing expected task cost under parameter uncertainty, and 2) minimizing optimality loss, which quantifies the degradation in task performance caused by planning with an incorrect parameter estimate. By actively optimizing for informative trajectories via a natural Fisher information measure, we tightly approach a lower bound on task-relevant parameter uncertainty while simultaneously achieving reliable task execution. Across diverse tasks, controllers, and adaptation laws, targeted exploration enables the identification required for successful task execution, allowing
robots to learn relevant physical parameters while completing the task in a single attempt.
Read the full abstract ▾
All authors Victor Vantilborgh, Hrishikesh Sathyanarayan, Guillaume Crevecoeur, Ian Abraham, Tom Lefebvre
Robotics
Degradation-Aware Cooperative Multi-Modal GNSS-Denied Localization Leveraging LiDAR-Based Robot Detections
Václav Pritzl, Xianjia Yu, Tomi Westerlund et al. (+2 more)
Accurate long-term localization using onboard sensors is crucial for
robots operating in Global
Navigation Satellite System (GNSS)-denied environments. While complementary sensors mitigate individual degradations, carrying all the available sensor types on a single
robot significantly increases the size, weight, and power demands. Distributing sensors across multiple
robots enhances the deployability but introduces challenges in fusing asynchronous, multi-modal data from independently moving platforms. We propose a novel adaptive multi-modal multi-
robot cooperative localization approach using a factor-graph formulation to fuse asynchronous Visual-Inertial Odometry (VIO),
LiDAR-Inertial Odometry (LIO), and 3D inter-
robot detections from distinct
robots in a loosely-coupled fashion. The approach adapts to changing conditions, leveraging reliable data to assist
robots affected by sensory degradations. A novel interpolation-based factor enables fusion of the unsynchronized measurements. LIO degradations are evaluated based on the approximate scan-matching Hessian. A novel approach of weighting odometry data proportionally to the Wasserstein distance between the consecutive VIO outputs is proposed. A theoretical analysis is provided, investigating the cooperative localization problem under various conditions, mainly in the presence of sensory degradations. The proposed method has been extensively evaluated on real-world data gathered with heterogeneous teams of an Unmanned Ground Vehicle (UGV) and Unmanned Aerial Vehicles (
UAVs), showing that the approach provides significant improvements in localization accuracy in the presence of various sensory degradations.
Read the full abstract ▾
All authors Václav Pritzl, Xianjia Yu, Tomi Westerlund, Petr Štěpán, Martin Saska
Robotics
Robot Path and Trajectory Planning Considering a Spatially Fixed TCP
Bernhard Rameder, Hubert Gattringer, Andreas Mueller et al. (+1 more)
This paper presents a method for planning a trajectory in workspace coordinates using a spatially fixed tool center point (TCP), while taking into account the processing path on a part. This approach is beneficial if it is easier to move the part rather than moving the tool. Whether a mathematical description that defines the shape to be processed or single points from a design program are used, the
robot path is finally represented using B-splines. The use of splines enables the path to be continuous with a desired degree, which finally leads to a smooth
robot trajectory. While calculating the
robot trajectory through prescribed orientation, additionally a given velocity at the TCP has to be considered. The procedure was validated on a real system using an industrial
robot moving an arbitrary defined part.
Read the full abstract ▾
All authors Bernhard Rameder, Hubert Gattringer, Andreas Mueller, Ronald Naderer
Robotics · Cross-listed from eess.SY
Behavior-Aware Online Prediction of Obstacle Occupancy using Zonotopes
Alvaro Carrizosa-Rendon, Jian Zhou, Erik Frisk et al. (+2 more)
Predicting the motion of surrounding vehicles is key to safe autonomous driving, especially in unstructured environments without prior information. This paper proposes a novel online method to accurately predict the occupancy sets of surrounding vehicles based solely on motion observations. The approach is divided into two stages: first, an Extended Kalman Filter and a Linear Programming (LP) problem are used to estimate a compact zonotopic set of control actions; then, a reachability analysis propagates this set to predict future occupancy. The effectiveness of the method has been validated through simulations in an urban environment, showing accurate and compact predictions without relying on prior assumptions or prior training data.
Read the full abstract ▾
All authors Alvaro Carrizosa-Rendon, Jian Zhou, Erik Frisk, Vicenc Puig, Fatiha Nejjari
Robotics
MR-UBi: Mixed Reality-Based Underwater Robot Arm Teleoperation System with Reaction Torque Indicator via Bilateral Control
Kohei Nishi, Masato Kobayashi and Yuki Uranishi
We present a mixed reality-based
underwater robot arm teleoperation system with a reaction torque indicator via bilateral control (MR-UBi). The reaction torque indicator (RTI) overlays a color and length-coded torque bar in the MR-HMD, enabling seamless integration of visual and haptic feedback during
underwater robot arm teleoperation. User studies with sixteen participants compared MR-UBi against a bilateral-control baseline. MR-UBi significantly improved grasping-torque control accuracy, increasing the time within the optimal torque range and reducing both low and high grasping torque range during lift and pick-and-place tasks with objects of different stiffness. Subjective evaluations further showed higher usability (SUS) and lower workload (NASA–TLX). Overall, the results confirm that MR-UBi enables more stable, accurate, and user-friendly
underwater robot-arm teleoperation through the integration of visual and haptic feedback. For additional material, please check: https://mertcookimg.github.io/mr-ubi
Read the full abstract ▾
Robotics
PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
Xiaogang Jia, Qian Wang, Anrui Wang et al. (+12 more)
Robotic manipulation systems benefit from complementary sensing modalities, where each provides unique environmental information. Point clouds capture detailed
geometric structure, while RGB images provide rich semantic context. Current point cloud methods struggle to capture fine-grained detail, especially for complex tasks, which RGB methods lack
geometric awareness, which hinders their precision and generalization. We introduce PointMapPolicy, a novel approach that conditions
diffusion policies on structured grids of points without downsampling. The resulting data type makes it easier to extract shape and spatial relationships from observations, and can be transformed between reference frames. Yet due to their structure in a regular grid, we enable the use of established computer
vision techniques directly to 3D data. Using xLSTM as a backbone, our model efficiently fuses the point maps with RGB data for enhanced multi-modal perception. Through extensive experiments on the RoboCasa and CALVIN
benchmarks and real
robot evaluations, we demonstrate that our method achieves state-of-the-art performance across diverse manipulation tasks. The overview and demos are available on our project page: https://point-map.github.io/Point-Map/
Read the full abstract ▾
All authors Xiaogang Jia, Qian Wang, Anrui Wang, Han A. Wang, Balázs Gyenes, Emiliyan Gospodinov, Xinkai Jiang, Ge Li, Hongyi Zhou, Weiran Liao, Xi Huang, Maximilian Beck, Moritz Reuss, Rudolf Lioutikov, Gerhard Neumann
Robotics
NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control
Yijiong Lin, Bowen Deng, Keju Pu et al. (+4 more)
Grasping accuracy is a critical prerequisite for precise object manipulation, often requiring careful
alignment between the
robot hand and object. Neural Descriptor Fields (NDF) offer a promising
vision-based method to generate grasping poses that generalize across object categories. However, NDF alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Meanwhile, tactile sensing enables more precise contact, but existing approaches typically learn policies limited to simple, predefined contact geometries. In this work, we introduce NeuralTouch, a
multimodal framework that integrates NDF and tactile sensing to enable accurate, generalizable grasping through gentle physical interaction. Our approach leverages NDF to implicitly represent the target contact geometry, from which a deep reinforcement learning (RL) policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and zero-shot transfer to real-world manipulation tasks–such as peg-out-in-hole and bottle lid opening–without additional fine-tuning. Results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic manipulation.
Read the full abstract ▾
All authors Yijiong Lin, Bowen Deng, Keju Pu, Chenghua Lu, Max Yang, Efi Psomopoulou, Nathan F. Lepora
Robotics
Multi-Modal Decentralized Reinforcement Learning for Modular Reconfigurable Lunar Robots
Ashutosh Mishra, Shreya Santra, Elian Neppel et al. (+4 more)
Modular reconfigurable
robots suit task-specific space operations, but the combinatorial growth of morphologies hinders unified control. We propose a decentralized reinforcement learning (Dec-RL) scheme where each module learns its own policy: wheel modules use Soft Actor-Critic (SAC) for locomotion and 7-DoF limbs use Proximal Policy Optimization (PPO) for steering and manipulation, enabling zero-shot generalization to unseen configurations. In simulation, the steering policy achieved a mean absolute error of 3.63{\deg} between desired and induced angles; the manipulation policy plateaued at 84.6 % success on a target-offset criterion; and the wheel policy cut average motor torque by 95.4 % relative to baseline while maintaining 99.6 % success. Lunar-analogue field tests validated zero-shot integration for autonomous locomotion, steering, and preliminary
alignment for reconfiguration. The system transitioned smoothly among synchronous, parallel, and sequential modes for Policy Execution, without idle states or control conflicts, indicating a scalable, reusable, and robust approach for modular lunar
robots.
Read the full abstract ▾
All authors Ashutosh Mishra, Shreya Santra, Elian Neppel, Edoardo M. Rossi Lombardi, Shamistan Karimov, Kentaro Uno, Kazuya Yoshida
Robotics
Dino-Diffusion Modular Designs Bridge the Cross-Domain Gap in Autonomous Parking
Zixuan Wu, Hengyuan Zhang, Ting-Hsuan Chen et al. (+4 more)
Parking is a critical pillar of driving safety. While recent end-to-end (E2E) approaches have achieved promising in-domain results, robustness under domain shifts (e.g., weather and lighting changes) remains a key challenge. Rather than relying on additional data, in this paper, we propose Dino-
Diffusion Parking (DDP), a domain-agnostic autonomous parking pipeline that integrates visual foundation models with
diffusion-based planning to enable generalized perception and robust motion planning under distribution shifts. We train our pipeline in CARLA at regular setting and transfer it to more
adversarial settings in a zero-shot fashion. Our model consistently achieves a parking success rate above 90% across all tested out-of-distribution (OOD) scenarios, with ablation studies confirming that both the network architecture and algorithmic design significantly enhance cross-domain performance over existing baselines. Furthermore, testing in a 3D Gaussian splatting (3DGS) environment reconstructed from a real-world parking lot demonstrates promising sim-to-real transfer.
Read the full abstract ▾
All authors Zixuan Wu, Hengyuan Zhang, Ting-Hsuan Chen, Yuliang Guo, David Paz, Xinyu Huang, Liu Ren
Robotics
MemER: Scaling Up Memory for Robot Control via Experience Retrieval
Ajay Sridhar, Jennifer Pan, Satvik Sharma et al. (+1 more)
Humans routinely rely on
memory to perform tasks, yet most
robot policies lack this capability; our goal is to endow
robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under
covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing
vision-
language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and $\pi_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal
language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of
memory.
Videos and code can be found at https://jen-pan.github.io/memer/.
Read the full abstract ▾
All authors Ajay Sridhar, Jennifer Pan, Satvik Sharma, Chelsea Finn
Robotics
Kinaema: a recurrent sequence model for memory and pose in motion
Mert Bulent Sariyildiz, Philippe Weinzaepfel, Guillaume Bono et al. (+2 more)
One key aspect of spatially aware
robots is the ability to "find their bearings", ie. to correctly situate themselves in previously seen spaces. In this work, we focus on this particular scenario of continuous robotics operations, where information observed before an actual episode start is exploited to optimize efficiency. We introduce a new model, Kinaema, and
agent, capable of integrating a stream of visual observations while moving in a potentially large scene, and upon request, processing a query image and predicting the relative position of the shown space with respect to its current position. Our model does not explicitly store an observation history, therefore does not have hard constraints on context length. It maintains an implicit latent
memory, which is updated by a
transformer in a recurrent way, compressing the history of sensor readings into a compact representation. We evaluate the impact of this model in a new downstream task we call "Mem-Nav". We show that our large-capacity recurrent model maintains a useful representation of the scene, navigates to goals observed before the actual episode start, and is computationally efficient, in particular compared to classical
transformers with
attention over an observation history.
Read the full abstract ▾
All authors Mert Bulent Sariyildiz, Philippe Weinzaepfel, Guillaume Bono, Gianluca Monaci, Christian Wolf
Robotics
NODA-MMH: Certified Learning-Aided Nonlinear Control for Magnetically-Actuated Swarm Experiment Toward On-Orbit Proof
Yuta Takahashi, Atsuki Ochi, Yoichi Tomioka et al. (+1 more)
This study experimentally validates the principle of large-scale satellite swarm control through learning-aided
magnetic field interactions generated by satellite-mounted magnetorquers. This actuation presents a promising solution for the long-term formation maintenance of multiple satellites and has primarily been demonstrated in ground-based testbeds for two-satellite position control. However, as the number of satellites increases beyond three, fundamental challenges coupled with the high nonlinearity arise: 1) nonholonomic constraints, 2) underactuation, 3) scalability, and 4) computational cost. Previous studies have shown that time-integrated current control theoretically solves these problems, where the average actuator outputs align with the desired command, and a learning-based technique further enhances their performance. Through multiple experiments, we validate critical aspects of learning-aided time-integrated current control: (1) enhanced controllability of the averaged system dynamics, with a theoretically guaranteed error bound, and (2) decentralized current management. We design two-axis coils and a ground-based experimental setup utilizing an air-bearing platform, enabling a mathematical replication of orbital dynamics. Based on the effectiveness of the learned interaction model, we introduce NODA-MMH (Neural power-Optimal Dipole Allocation for certified learned Model-based Magnetically swarm control Harness) for model-based power-optimal swarm control. This study complements our tutorial paper on magnetically actuated swarms for the long-term formation maintenance problem.
Read the full abstract ▾
All authors Yuta Takahashi, Atsuki Ochi, Yoichi Tomioka, Shin-Ichiro Sakai
Robotics
A Contact-Driven Framework for Manipulating in the Blind
Muhammad Suhail Saleem, Lai Yuan and Maxim Likhachev
Robots often face manipulation tasks in environments where
vision is inadequate due to clutter, occlusions, or poor lighting–for example, reaching a shutoff valve at the back of a sink cabinet or locating a light switch above a crowded shelf. In such settings,
robots, much like humans, must rely on contact feedback to distinguish free from occupied space and navigate around obstacles. Many of these environments often exhibit strong structural priors–for instance, pipes often span across sink cabinets–that can be exploited to anticipate unseen structure and avoid unnecessary collisions. We present a theoretically complete and empirically efficient framework for manipulation in the blind that integrates contact feedback with structural priors to enable robust operation in unknown environments. The framework comprises three tightly coupled components: (i) a contact detection and localization module that utilizes joint torque sensing with a contact particle filter to detect and localize contacts, (ii) an occupancy estimation module that uses the history of contact observations to build a partial occupancy map of the workspace and extrapolate it into unexplored regions with learned predictors, and (iii) a planning module that accounts for the fact that contact localization estimates and occupancy predictions can be noisy, computing paths that avoid collisions and complete tasks efficiently without eliminating feasible solutions. We evaluate the system in simulation and in the real world on a UR10e manipulator across two domestic tasks–(i) manipulating a valve under a kitchen sink surrounded by pipes and (ii) retrieving a target object from a cluttered shelf. Results show that the framework reliably solves these tasks, achieving up to a 2x reduction in task completion time compared to baselines, with ablations confirming the contribution of each module.
Read the full abstract ▾
Robotics
Reinforcement Learning-based Robust Wall Climbing Locomotion Controller in Ferromagnetic Environment
Yong Um, Young-Ha Shin, Joon-Ha Kim et al. (+2 more)
We present a reinforcement learning framework for quadrupedal wall-climbing locomotion that explicitly addresses uncertainty in
magnetic foot adhesion. A physics-based adhesion model of a quadrupedal
magnetic climbing
robot is incorporated into simulation to capture partial contact, air-gap sensitivity, and probabilistic attachment failures. To stabilize learning and enable reliable transfer, we design a three-phase curriculum: (1) acquire a crawl gait on flat ground without adhesion, (2) gradually rotate the
gravity vector to vertical while activating the adhesion model, and (3) inject
stochastic adhesion failures to encourage slip recovery. The learned policy achieves a high success rate, strong adhesion retention, and rapid recovery from detachment in simulation under degraded adhesion. Compared with a model predictive control (MPC) baseline that assumes perfect adhesion, our controller maintains locomotion when attachment is intermittently lost. Hardware experiments with the untethered
robot further confirm robust vertical crawling on steel surfaces, maintaining stability despite transient misalignment and incomplete attachment. These results show that combining curriculum learning with realistic adhesion modeling provides a resilient sim-to-real framework for
magnetic climbing
robots in complex environments.
Read the full abstract ▾
All authors Yong Um, Young-Ha Shin, Joon-Ha Kim, Soonpyo Kwon, Hae-Won Park
Robotics
PathFormer: A Transformer with 3D Grid Constraints for Digital Twin Robot-Arm Trajectory Generation
Ahmed Alanazi, Duy Ho and Yugyung Lee
Robotic arms require precise, task-aware trajectory planning, yet sequence models that ignore motion structure often yield invalid or inefficient executions. We present a Path-based
Transformer that encodes
robot motion with a 3-grid (where/what/when) representation and constraint-masked decoding, enforcing
lattice-adjacent moves and workspace bounds while
reasoning over task graphs and action order. Trained on 53,755 trajectories (80% train / 20% validation), the model aligns closely with ground truth – 89.44% stepwise accuracy, 93.32% precision, 89.44% recall, and 90.40% F1 – with 99.99% of paths legal by construction. Compiled to motor primitives on an xArm Lite 6 with a depth-camera digital twin, it attains up to 97.5% reach and 92.5% pick success in controlled tests, and 86.7% end-to-end success across 60
language-specified tasks in cluttered scenes, absorbing slips and occlusions via local re-grounding without global re-planning. These results show that path-structured representations enable
Transformers to generate accurate, reliable, and interpretable
robot trajectories, bridging graph-based planning and sequence-based learning and providing a practical foundation for general-purpose manipulation and sim-to-real transfer.
Read the full abstract ▾