Robotics
Mind to Hand: Purposeful Robotic Control via Embodied Reasoning
Peijun Tang, Shangjin Xie, Binyan Sun et al. (+5 more)
Humans act with context and intention, with
reasoning playing a central role. While internet-scale data has enabled broad
reasoning capabilities in AI systems, grounding these abilities in physical action remains a major challenge. We introduce Lumo-1, a generalist
vision-
language-action (VLA) model that unifies
robot reasoning ("mind") with
robot action ("hand"). Our approach builds upon the general multi-modal
reasoning capabilities of pre-trained
vision-
language models (VLMs), progressively extending them to
embodied reasoning and action prediction, and ultimately towards structured
reasoning and
reasoning-action
alignment. This results in a three-stage pre-training pipeline: (1) Continued VLM pre-training on curated
vision-
language data to enhance
embodied reasoning skills such as planning, spatial understanding, and trajectory prediction; (2) Co-training on cross-embodiment
robot data alongside
vision-
language data; and (3) Action training with
reasoning process on trajectories collected on Astribot S1, a bimanual mobile manipulator with human-like dexterity and agility. Finally, we integrate reinforcement learning to further refine
reasoning-action consistency and close the loop between semantic inference and motor control. Extensive experiments demonstrate that Lumo-1 achieves significant performance improvements in
embodied vision-
language reasoning, a critical component for generalist robotic control. Real-world evaluations further show that Lumo-1 surpasses strong baselines across a wide range of challenging robotic tasks, with strong generalization to novel objects and environments, excelling particularly in long-horizon tasks and responding to human-natural instructions that require
reasoning over strategy, concepts and space.
Read the full abstract ▾
All authors Peijun Tang, Shangjin Xie, Binyan Sun, Baifu Huang, Kuncheng Luo, Haotian Yang, Weiqi Jin, Jianan Wang
Robotics · Cross-listed from cs.CV
Disturbance-Free Surgical Video Generation from Multi-Camera Shadowless Lamps for Open Surgery
Yuna Kato, Shohei Mori, Hideo Saito et al. (+3 more)
Video recordings of open surgeries are greatly required for education and research purposes. However, capturing unobstructed
videos is challenging since surgeons frequently block the camera field of view. To avoid occlusion, the positions and angles of the camera must be frequently adjusted, which is highly labor-intensive. Prior work has addressed this issue by installing multiple cameras on a shadowless lamp and arranging them to fully surround the
surgical area. This setup increases the chances of some cameras capturing an unobstructed view. However, manual image
alignment is needed in post-processing since camera configurations change every time surgeons move the lamp for optimal lighting. This paper aims to fully automate this
alignment task. The proposed method identifies frames in which the lighting system moves, realigns them, and selects the camera with the least occlusion to generate a
video that consistently presents the
surgical field from a fixed perspective. A user study involving surgeons demonstrated that
videos generated by our method were superior to those produced by conventional methods in terms of the ease of confirming the
surgical area and the comfort during
video viewing. Additionally, our approach showed improvements in
video quality over existing techniques. Furthermore, we implemented several synthesis options for the proposed view-synthesis method and conducted a user study to assess surgeons' preferences for each option.
Read the full abstract ▾
All authors Yuna Kato, Shohei Mori, Hideo Saito, Yoshifumi Takatsume, Hiroki Kajita, Mariko Isogawa
Robotics
RVC-NMPC: Nonlinear Model Predictive Control with Reciprocal Velocity Constraints for Mutual Collision Avoidance in Agile UAV Flight
Vit Kratky, Robert Penicka, Parakh M. Gupta et al. (+2 more)
This paper presents an approach to mutual collision avoidance based on Nonlinear Model Predictive Control (NMPC) with time-dependent Reciprocal Velocity Constraints (RVCs). Unlike most existing methods, the proposed approach relies solely on observable information about other
robots, eliminating the need for excessive communication. The computationally efficient
algorithm for computing RVCs, together with the direct integration of these constraints into the NMPC problem formulation at the controller level, allows the whole pipeline to run at 100 Hz. This high processing rate, combined with modeled nonlinear dynamics of the controlled Uncrewed Aerial Vehicles (
UAVs), is a key feature that facilitates the use of the proposed approach for agile
UAV flight. The proposed approach was evaluated through extensive simulations emulating real-world conditions in scenarios involving up to 10
UAVs and velocities of up to 25 m/s, and in real-world experiments with accelerations up to 30 m/s$^2$. Comparison with the state of the art shows 31% improvement in terms of flight time reduction in challenging scenarios, while maintaining a collision-free
navigation in all trials.
Read the full abstract ▾
All authors Vit Kratky, Robert Penicka, Parakh M. Gupta, Ondrej Prochazka, Martin Saska
Robotics
Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representations
Yuchi Zhang, Churui Sun, Shiqi Liang et al. (+4 more)
Recent end-to-end robotic manipulation research increasingly adopts architectures inspired by large
language models to enable robust manipulation. However, a critical challenge arises from severe distribution shifts between robotic action data, primarily due to substantial numerical variations in action commands across diverse robotic platforms and tasks, hindering the effective transfer of pretrained knowledge. To address this limitation, we propose a semantically grounded linguistic representation to normalize actions for efficient pretraining. Unlike conventional discretized action representations that are sensitive to numerical scales, the motion representation specifically disregards numeric scale effects, emphasizing directionality instead. This abstraction mitigates distribution shifts, yielding a more generalizable pretraining representation. Moreover, using the motion representation narrows the feature distance between action
tokens and standard vocabulary
tokens, mitigating modality gaps. Multi-task experiments on two
benchmarks demonstrate that the proposed method significantly improves generalization performance and transferability in robotic manipulation tasks.
Read the full abstract ▾
All authors Yuchi Zhang, Churui Sun, Shiqi Liang, Diyuan Liu, Chao Ji, Wei-Nan Zhang, Ting Liu
Robotics
vEDGAR – Can CARLA Do HiL?
Nils Gehrke, David Brecht, Dominik Kulmer et al. (+2 more)
Simulation offers advantages throughout the development process of automated driving functions, both in research and product development. Common open-source simulators like CARLA are extensively used in training, evaluation, and software-in-the-loop testing of new automated driving
algorithms. However, the CARLA simulator lacks an evaluation where research and automated driving vehicles are simulated with their entire sensor and actuation stack in real time. The goal of this work is therefore to create a simulation framework for testing the automation software on its dedicated hardware and identifying its limits. Achieving this goal would greatly benefit the open-source development workflow of automated driving functions, designating CARLA as a consistent evaluation tool along the entire development process. To achieve this goal, in a first step, requirements are derived, and a simulation architecture is specified and implemented. Based on the formulated requirements, the proposed vEDGAR software is evaluated, resulting in a final conclusion on the applicability of CARLA for HiL testing of automated vehicles. The tool is available open source: Modified CARLA fork: https://github.com/TUMFTM/carla, vEDGAR Framework: https://github.com/TUMFTM/vEDGAR
Read the full abstract ▾
All authors Nils Gehrke, David Brecht, Dominik Kulmer, Dheer Patel, Frank Diermeyer
Robotics
SensHRPS: Sensing Comfortable Human-Robot Proxemics and Personal Space With Eye-Tracking
Nadezhda Kushina, Ko Watanabe, Aarthi Kannan et al. (+3 more)
Social
robots must adjust to human proxemic norms to ensure user comfort and engagement. While prior research demonstrates that eye-tracking features reliably estimate comfort in human-human interactions, their applicability to interactions with
humanoid robots remains unexplored. In this study, we investigate user comfort with the
robot "Ameca" across four experimentally controlled distances (0.5 m to 2.0 m) using mobile eye-tracking and subjective reporting (N=19). We evaluate multiple machine learning and deep learning models to estimate comfort based on gaze features. Contrary to previous human-human studies where
Transformer models excelled, a Decision Tree classifier achieved the highest performance (F1-score = 0.73), with minimum pupil diameter identified as the most critical predictor. These findings suggest that physiological comfort thresholds in human-
robot interaction differ from human-human dynamics and can be effectively modeled using interpretable logic.
Read the full abstract ▾
All authors Nadezhda Kushina, Ko Watanabe, Aarthi Kannan, Ashita Ashok, Andreas Dengel, Karsten Berns
Robotics
Prospect Theory in Physical Human-Robot Interaction: A Pilot Study of Probability Perception
Yixiang Lin, Tiancheng Yang, Jonathan Eden et al. (+1 more)
Understanding how humans respond to uncertainty is critical for designing safe and effective physical human-
robot interaction (pHRI), as physically working with
robots introduces multiple sources of uncertainty, including trust, comfort, and perceived safety. Conventional pHRI control frameworks typically build on optimal control theory, which assumes that human actions minimize a cost function; however, human behavior under uncertainty often departs from such optimal patterns. To address this gap, additional understanding of human behavior under uncertainty is needed. This pilot study implemented a physically coupled target-reaching task in which the
robot delivered assistance or disturbances with systematically varied probabilities (10% to 90%). Analysis of participants' force inputs and decision-making strategies revealed two distinct behavioral clusters: a "trade-off" group that modulated their physical responses according to disturbance likelihood, and an "always-compensate" group characterized by strong risk aversion irrespective of probability. These findings provide empirical evidence that human decision-making in pHRI is highly individualized and that the perception of probability can differ to its true value. Accordingly, the study highlights the need for more interpretable behavioral models, such as cumulative prospect theory (CPT), to more accurately capture these behaviors and inform the design of future adaptive
robot controllers.
Read the full abstract ▾
All authors Yixiang Lin, Tiancheng Yang, Jonathan Eden, Ying Tan
Robotics
A Multi-Agent LLM Framework for Design Space Exploration in Autonomous Driving Systems
Po-An Shih, Shao-Hua Wang, Yung-Che Li et al. (+2 more)
Designing autonomous driving systems requires efficient exploration of large hardware/software configuration spaces under diverse environmental conditions, e.g., with varying traffic, weather, and road layouts. Traditional design space exploration (DSE) approaches struggle with multi-modal execution outputs and complex performance trade-offs, and often require human involvement to assess correctness based on execution outputs. This paper presents a multi-
agent, large
language model (LLM)-based DSE framework, which integrates multi-modal
reasoning with 3D simulation and profiling tools to automate the interpretation of execution outputs and guide the exploration of system designs. Specialized LLM
agents are leveraged to handle user input interpretation, design point generation, execution orchestration, and analysis of both visual and textual execution outputs, which enables identification of potential bottlenecks without human intervention. A prototype implementation is developed and evaluated on a robotaxi case study (an SAE Level 4 autonomous driving application). Compared with a genetic
algorithm baseline, the proposed framework identifies more Pareto-optimal, cost-efficient solutions with reduced
navigation time under the same exploration budget. Experimental results also demonstrate the efficiency of the adoption of the LLM-based approach for DSE. We believe that this framework paves the way to the design automation of autonomous driving systems.
Read the full abstract ▾
All authors Po-An Shih, Shao-Hua Wang, Yung-Che Li, Chia-Heng Tu, Chih-Han Chang
Robotics · Cross-listed from cs.AI
Privileged observations enable rapid and reliable policy discovery directly in the physical world
Antonio Terpin and Raffaello D'Andrea
We study how privileged information about a physical system affects the discovery of high-performing policies when training a reinforcement learning
agent directly in the physical world. We let the
agent control a cylinder in a tabletop water channel to maximize or minimize drag. The flow is chaotic and difficult to model or simulate accurately and good strategies are not obvious beforehand. Decades-old experimental studies provide recipes for simple, high-performance, periodic open-loop policies that increase drag by 26.6% +/- 0.7% and reduce it by 29.7% +/- 1.3%. With dense observations of the cylinder wake, the
agent learns within tens of minutes policies that increase drag by 25.5% +/- 0.9% and reduce it by 32.4% +/- 1.6%. We record action trajectories during online policy execution and replay them open loop as fixed sequences; these replays increase drag by 23.2% +/- 2.2% and reduce it by 32.1% +/- 3.0%. However, when we withhold flow observations during training, the
agent still learns to decrease drag by 31.4% +/- 2.1%, but no run learns to increase it (1.8% +/- 4.3% drag increase). Our physical experiments demonstrate an extreme case: privileged observations can be decisive for policy discovery even when the resulting actions can be replayed in open loop.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
SDT-6D: Fully Sparse Depth-Transformer for Staged End-to-End 6D Pose Estimation in Industrial Multi-View Bin Picking
Nico Leuze, Maximilian Hoh, Samed Doğan et al. (+2 more)
Accurately recovering 6D poses in densely packed industrial bin-picking environments remain a serious challenge, owing to occlusions, reflections, and textureless parts. We introduce a holistic depth-only 6D pose estimation approach that fuses multi-view depth maps into either a fine-grained 3D point cloud in its vanilla version, or a sparse Truncated Signed Distance Field (TSDF). At the core of our framework lies a staged heatmap mechanism that yields scene-adaptive
attention priors across different resolutions, steering computation toward foreground regions, thus keeping
memory requirements at high resolutions feasible. Along, we propose a density-aware sparse
transformer block that dynamically attends to (self-) occlusions and the non-uniform distribution of 3D data. While sparse 3D approaches has proven effective for long-range perception, its potential in close-range robotic applications remains underexplored. Our framework operates fully sparse, enabling high-resolution volumetric representations to capture fine
geometric details crucial for accurate pose estimation in clutter. Our method processes the entire scene integrally, predicting the 6D pose via a novel per-voxel voting strategy, allowing simultaneous pose predictions for an arbitrary number of target objects. We validate our method on the recently published IPD and MV-YCB multi-view
datasets, demonstrating competitive performance in heavily cluttered industrial and household bin picking scenarios.
Read the full abstract ▾
All authors Nico Leuze, Maximilian Hoh, Samed Doğan, Nicolas R.-Peña, Alfred Schoettl
Robotics · Cross-listed from cs.AI
Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems
Mingwei Li, Xiaoyuan Zhang, Chengwei Yang et al. (+2 more)
Model-based planning in robotic domains is challenged by the hybrid nature of physical dynamics, where continuous motion is punctuated by discrete events such as contacts and impacts. Conventional latent world models typically employ monolithic neural networks that enforce global continuity, which over-smooths distinct dynamic modes (e.g., sticking vs. sliding, flight vs. stance). For a planner, this smoothing results in compounding errors during long-horizon lookaheads, rendering the search process unreliable at physical boundaries. To address this, we introduce the Prismatic World Model (PRISM-WM), a structured architecture designed to decompose complex hybrid dynamics into composable primitives. PRISM-WM uses a context-aware Mixture-of-Experts (MoE) framework where a gating mechanism implicitly identifies the current physical mode, and specialized experts predict the associated transition dynamics. We further introduce a latent orthogonalization objective to ensure expert diversity, preventing mode collapse. By modeling the mode transitions in system dynamics, PRISM-WM reduces rollout drift. Experiments on continuous control
benchmarks, including high-dimensional
humanoids and multi-task settings, demonstrate that PRISM-WM provides a high-fidelity substrate for trajectory optimization
algorithms (e.g., TD-MPC), indicating its potential as a foundational model for model-based
agents.
Read the full abstract ▾
All authors Mingwei Li, Xiaoyuan Zhang, Chengwei Yang, Zilong Zheng, Yaodong Yang
Robotics
Learning Robot Manipulation from Audio World Models
Fan Zhang and Michael Gienger
World models have demonstrated impressive performance on robotic learning tasks. Many such tasks inherently demand
multimodal reasoning; for example, filling a bottle with water will lead to visual information alone being ambiguous or incomplete, thereby requiring
reasoning over the temporal
evolution of audio, accounting for its underlying physical properties and pitch patterns. In this paper, we propose a generative latent flow matching model to anticipate future audio observations, enabling the system to reason about long-term consequences when integrated into a
robot policy. We demonstrate the superior capabilities of our system through two manipulation tasks that require perceiving in-the-wild audio or music signals, compared to methods without future lookahead. We further emphasize that successful
robot action learning for these tasks relies not merely on multi-modal input, but critically on the accurate prediction of future audio states that embody intrinsic rhythmic patterns.
Read the full abstract ▾
Robotics
Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging
Yajat Yadav, Zhiyuan Zhou, Andrew Wagenmaker et al. (+2 more)
Generalist
robot policies, trained on large and diverse
datasets, have demonstrated the ability to generalize across a wide spectrum of behaviors, enabling a single policy to act in varied real-world environments. However, they still fall short on new tasks not covered in the training data. When finetuned on limited demonstrations of a new task, these policies often overfit to the specific demonstrations–not only losing their prior abilities to solve a wide variety of generalist tasks but also failing to generalize within the new task itself. In this work, we aim to develop a method that preserves the generalization capabilities of the generalist policy during finetuning, allowing a single policy to robustly incorporate a new skill into its repertoire. Our goal is a single policy that both learns to generalize to variations of the new task and retains the broad competencies gained from pretraining. We show that this can be achieved through a simple yet effective strategy: interpolating the weights of a finetuned model with that of the pretrained model. We show, across extensive simulated and real-world experiments, that such model merging produces a single model that inherits the generalist abilities of the base model and learns to solve the new task robustly, outperforming both the pretrained and finetuned model on out-of-distribution variations of the new task. Moreover, we show that model merging performance scales with the amount of pretraining data, and enables continual acquisition of new skills in a lifelong learning setting, without sacrificing previously learned generalist abilities.
Read the full abstract ▾
All authors Yajat Yadav, Zhiyuan Zhou, Andrew Wagenmaker, Karl Pertsch, Sergey Levine
Robotics
Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
Haldun Balim, Na Li and Yilun Du
Offline decision-making via
diffusion models often produces trajectories that are misaligned with system dynamics, limiting their reliability for control. We propose Model Predictive Diffuser (MPDiffuser), a compositional
diffusion framework that combines a
diffusion planner with a dynamics
diffusion model to generate task-aligned and dynamically plausible trajectories. MPDiffuser interleaves planner and dynamics updates during sampling, progressively correcting feasibility while preserving task intent. A lightweight ranking module then selects trajectories that best satisfy task objectives. The compositional design improves sample efficiency and adaptability by enabling the dynamics model to leverage diverse and previously unseen data independently of the planner. Empirically, we demonstrate consistent improvements over prior
diffusion-based methods on unconstrained (D4RL) and constrained (DSRL)
benchmarks, and validate practicality through deployment on a real quadrupedal
robot.
Read the full abstract ▾
Robotics
Zero-Splat TeleAssist: A Zero-Shot Pose Estimation Framework for Semantic Teleoperation
Srijan Dokania and Dharini Raghavan
We introduce Zero-Splat TeleAssist, a zero-shot sensor-fusion pipeline that transforms commodity CCTV streams into a shared, 6-DoF world model for multilateral teleoperation. By integrating
vision-
language segmentation,
monocular depth, weighted-PCA pose extraction, and 3D Gaussian Splatting (3DGS), TeleAssist provides every operator with real-time global positions and orientations of multiple
robots without fiducials or depth sensors in an interaction-centric teleoperation setup.
Read the full abstract ▾
Robotics · Cross-listed from cs.CV
RLCNet: An end-to-end deep learning framework for simultaneous online calibration of LiDAR, RADAR, and Camera
Hafeez Husain Cholakkal, Stefano Arrigoni and Francesco Braghin
Accurate extrinsic calibration of
LiDAR, RADAR, and camera sensors is essential for reliable perception in autonomous vehicles. Still, it remains challenging due to factors such as mechanical vibrations and cumulative sensor drift in dynamic environments. This paper presents RLCNet, a novel end-to-end trainable deep learning framework for the simultaneous online calibration of these
multimodal sensors. Validated on real-world
datasets, RLCNet is designed for practical deployment and demonstrates robust performance under diverse conditions. To support real-time operation, an online calibration framework is introduced that incorporates a weighted moving average and outlier rejection, enabling dynamic adjustment of calibration parameters with reduced prediction noise and improved resilience to drift. An ablation study highlights the significance of architectural choices, while comparisons with existing methods demonstrate the superior accuracy and robustness of the proposed approach.
Read the full abstract ▾
Robotics
Learning Spatiotemporal Tubes for Temporal Reach-Avoid-Stay Tasks using Physics-Informed Neural Networks
Ahan Basu, Ratnangshu Das and Pushpak Jagtap
This paper presents a Spatiotemporal Tube (STT)-based control framework for general control-affine
MIMO nonlinear pure-feedback systems with unknown dynamics to satisfy prescribed time reach-avoid-stay tasks under external disturbances. The STT is defined as a time-varying ball, whose center and radius are jointly approximated by a Physics-Informed Neural Network (PINN). The constraints governing the STT are first formulated as loss functions of the PINN, and a training
algorithm is proposed to minimize the overall violation. The PINN being trained on certain collocation points, we propose a Lipschitz-based validity condition to formally verify that the learned PINN satisfies the conditions over the continuous time horizon. Building on the learned STT representation, an approximation-free closed-form controller is defined to guarantee satisfaction of the T-RAS specification. Finally, the effectiveness and scalability of the framework are validated through two case studies involving a mobile
robot and an aerial vehicle navigating through cluttered environments.
Read the full abstract ▾
Robotics
Semantic-Metric Bayesian Risk Fields: Learning Robot Safety from Human Videos with a VLM Prior
Timothy Chen, Marcus Dominguez-Kuhne, Aiden Swann et al. (+2 more)
Humans interpret safety not as a binary signal but as a continuous, context- and spatially-dependent notion of risk. While risk is subjective, humans form rational mental models that guide action selection in dynamic environments. This work proposes a framework for extracting implicit human risk models by introducing a novel, semantically-conditioned and spatially-varying parametrization of risk, supervised directly from safe human demonstration
videos and VLM common sense. Notably, we define risk through a
Bayesian formulation. The prior is furnished by a pretrained
vision-
language model. In order to encourage the risk estimate to be more human aligned, a likelihood function modulates the prior to produce a relative metric of risk. Specifically, the likelihood is a learned ViT that maps pretrained features, to pixel-aligned risk values. Our pipeline ingests RGB images and a query object string, producing pixel-dense risk images. These images that can then be used as value-predictors in
robot planning tasks or be projected into 3D for use in conventional trajectory optimization to produce human-like motion. This learned mapping enables generalization to novel objects and contexts, and has the potential to scale to much larger training
datasets. In particular, the
Bayesian framework that is introduced enables fast adaptation of our model to additional observations or common sense rules. We demonstrate that our proposed framework produces contextual risk that aligns with human preferences. Additionally, we illustrate several downstream applications of the model; as a value learner for visuomotor planners or in conjunction with a classical trajectory optimization
algorithm. Our results suggest that our framework is a significant step toward enabling autonomous systems to internalize human-like risk. Code and results can be found at https://riskbayesian.github.io/
bayesian_risk/.
Read the full abstract ▾
All authors Timothy Chen, Marcus Dominguez-Kuhne, Aiden Swann, Xu Liu, Mac Schwager
Robotics · Cross-listed from cs.CV
Geometry-Aware Sparse Depth Sampling for High-Fidelity RGB-D Depth Completion in Robotic Systems
Tony Salloom, Dandi Zhou and Xinhai Sun
Accurate three-dimensional perception is essential for modern industrial robotic systems that perform manipulation, inspection, and
navigation tasks. RGB-D and stereo
vision sensors are widely used for this purpose, but the depth maps they produce are often noisy, incomplete, or biased due to sensor limitations and environmental conditions. Depth completion methods aim to generate dense, reliable depth maps from RGB images and sparse depth input. However, a key limitation in current depth completion pipelines is the unrealistic generation of sparse depth: sparse pixels are typically selected uniformly at random from dense ground-truth depth, ignoring the fact that real sensors exhibit geometry-dependent and spatially nonuniform reliability. In this work, we propose a normal-guided sparse depth sampling strategy that leverages PCA-based surface normal estimation on the RGB-D point cloud to compute a per-pixel depth reliability measure. The sparse depth samples are then drawn according to this reliability distribution. We integrate this sampling method with the Marigold-DC
diffusion-based depth completion model and evaluate it on NYU Depth v2 using the standard metrics. Experiments show that our geometry-aware sparse depth improves accuracy, reduces artifacts near edges and discontinuities, and produces more realistic training conditions that better reflect real sensor behavior.
Read the full abstract ▾
Robotics
High-Performance Dual-Arm Task and Motion Planning for Tabletop Rearrangement
Duo Zhang, Junshan Huang and Jingjin Yu
We propose Synchronous Dual-Arm Rearrangement Planner (SDAR), a task and motion planning (TAMP) framework for tabletop rearrangement, where two
robot arms equipped with 2-finger grippers must work together in close proximity to rearrange objects whose start and goal configurations are strongly entangled. To tackle such challenges, SDAR tightly knit together its dependency-driven task planner (SDAR-T) and synchronous dual-arm motion planner (SDAR-M), to intelligently sift through a large number of possible task and motion plans. Specifically, SDAR-T applies a simple yet effective strategy to decompose the global object dependency graph induced by the rearrangement task, to produce more optimal dual-arm task plans than solutions derived from optimal task plans for a single arm. Leveraging state-of-the-art GPU SIMD-based motion planning tools, SDAR-M employs a layered motion planning strategy to sift through many task plans for the best synchronous dual-arm motion plan while ensuring high levels of success rate. Comprehensive evaluation demonstrates that SDAR delivers a 100% success rate in solving complex, non-monotone, long-horizon tabletop rearrangement tasks with solution quality far exceeding the previous state-of-the-art. Experiments on two UR-5e arms further confirm SDAR directly and reliably transfers to
robot hardware. Source code and supplementary materials are available at https://github.com/arc-l/dual-arm.
Read the full abstract ▾
Robotics
Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model
Wenjiang Xu, Cindy Wang, Rui Fang et al. (+6 more)
World models have emerged as a pivotal component in
robot manipulation planning, enabling
agents to predict future environmental states and reason about the consequences of actions before execution. While
video-generation models are increasingly adopted, they often lack rigorous physical grounding, leading to
hallucinations and a failure to maintain consistency in long-horizon physical constraints. To address these limitations, we propose
Embodied Tree of Thoughts (EToT), a novel Real2Sim2Real planning framework that leverages a physics-based interactive digital twin as an
embodied world model. EToT formulates manipulation planning as a tree search expanded through two synergistic mechanisms: (1) Priori Branching, which generates diverse candidate execution paths based on semantic and spatial analysis; and (2) Reflective Branching, which utilizes VLMs to diagnose execution failures within the simulator and iteratively refine the planning tree with corrective actions. By grounding high-level
reasoning in a physics simulator, our framework ensures that generated plans adhere to rigid-body dynamics and collision constraints. We validate EToT on a suite of short- and long-horizon manipulation tasks, where it consistently outperforms baselines by effectively predicting physical dynamics and adapting to potential failures. Website at https://
embodied-tree-of-thoughts.github.io .
Read the full abstract ▾
All authors Wenjiang Xu, Cindy Wang, Rui Fang, Mingkang Zhang, Lusong Li, Jing Xu, Jiayuan Gu, Zecui Zeng, Rui Chen
Robotics
Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation
Meng Wei, Chenyang Wan, Jiaqi Peng et al. (+8 more)
While recent large
vision-
language models (VLMs) have improved generalization in
vision-
language navigation (VLN), existing methods typically rely on end-to-end pipelines that map
vision-
language inputs directly to short-horizon discrete actions. Such designs often produce fragmented motions, incur high latency, and struggle with real-world challenges like dynamic obstacle avoidance. We propose DualVLN, the first dual-system VLN foundation model that synergistically integrates high-level
reasoning with low-level action execution. System 2, a VLM-based global planner, "grounds slowly" by predicting mid-term waypoint goals via image-grounded
reasoning. System 1, a lightweight, multi-modal conditioning
Diffusion Transformer policy, "moves fast" by leveraging both explicit pixel goals and latent features from System 2 to generate smooth and accurate trajectories. The dual-system design enables robust real-time control and adaptive local decision-making in complex, dynamic environments. By decoupling training, the VLM retains its generalization, while System 1 achieves interpretable and effective local
navigation. DualVLN outperforms prior methods across all VLN
benchmarks and real-world experiments demonstrate robust long-horizon planning and real-time adaptability in dynamic environments.
Read the full abstract ▾
All authors Meng Wei, Chenyang Wan, Jiaqi Peng, Xiqian Yu, Yuqiang Yang, Delin Feng, Wenzhe Cai, Chenming Zhu, Tai Wang, Jiangmiao Pang, Xihui Liu
Robotics
RAVES-Calib: Robust, Accurate and Versatile Extrinsic Self Calibration Using Optimal Geometric Features
Haoxin Zhang, Shuaixin Li, Xiaozhou Zhu et al. (+2 more)
In this paper, we present a user-friendly
LiDAR-camera calibration toolkit that is compatible with various
LiDAR and camera sensors and requires only a single pair of
laser points and a camera image in targetless environments. Our approach eliminates the need for an initial transform and remains robust even with large positional and rotational
LiDAR-camera extrinsic parameters. We employ the Gluestick pipeline to establish 2D-3D point and line feature correspondences for a robust and automatic initial guess. To enhance accuracy, we quantitatively analyze the impact of feature distribution on calibration results and adaptively weight the cost of each feature based on these metrics. As a result, extrinsic parameters are optimized by filtering out the adverse effects of inferior features. We validated our method through extensive experiments across various
LiDAR-camera sensors in both indoor and outdoor settings. The results demonstrate that our method provides superior robustness and accuracy compared to SOTA techniques. Our code is open-sourced on GitHub to benefit the community.
Read the full abstract ▾
All authors Haoxin Zhang, Shuaixin Li, Xiaozhou Zhu, Hongbo Chen, Wen Yao
Robotics
Chat with UAV – Human-UAV Interaction Based on Large Language Models
Haoran Wang, Zhuohang Chen, Guang Li et al. (+2 more)
The future of
UAV interaction systems is evolving from engineer-driven to user-driven, aiming to replace traditional predefined Human-
UAV Interaction designs. This shift focuses on enabling more personalized task planning and design, thereby achieving a higher quality of interaction experience and greater flexibility, which can be used in many fileds, such as agriculture, aerial photography, logistics, and environmental monitoring. However, due to the lack of a common
language between users and the
UAVs, such interactions are often difficult to be achieved. The developments of Large
Language Models possess the ability to understand nature
languages and
Robots' (
UAVs') behaviors, marking the possibility of personalized Human-
UAV Interaction. Recently, some HUI frameworks based on LLMs have been proposed, but they commonly suffer from difficulties in mixed task planning and execution, leading to low adaptability in complex scenarios. In this paper, we propose a novel dual-
agent HUI framework. This framework constructs two independent LLM
agents (a task planning
agent, and an execution
agent) and applies different
Prompt Engineering to separately handle the understanding, planning, and execution of tasks. To verify the effectiveness and performance of the framework, we have built a task database covering four typical application scenarios of
UAVs and quantified the performance of the HUI framework using three independent metrics. Meanwhile different LLM models are selected to control the
UAVs with compared performance. Our user study experimental results demonstrate that the framework improves the smoothness of HUI and the flexibility of task execution in the tasks scenario we set up, effectively meeting users' personalized needs.
Read the full abstract ▾
All authors Haoran Wang, Zhuohang Chen, Guang Li, Bo Ma, Chuanghuang Li