This thesis work was done as a part of research at Unmanned System Research Group (USRG) led by Professor David Hyunchul Shim, KAIST, South Korea.
The development of the scale-aware monocular depth estimation aided monocular visual SLAM system for real-time robot navigation
Human pilots could fly any drone by relying only on the monocular visual signal in the goggle. In this work, we want to use high performance computer to replace the human brain to navigate the drone by relying only on the monocular camera sensor.
To enable the mobile robot to autonomously navigate in an unknown environment using only a monocular camera as its sensor
RGB-Deep D visual SLAM
: Augmented themonocular depth estimation
(Packnet-sfm) to provide the pseudo depth information to the monocular visual SLAM (ORB_SLAM2) to solves its inherent problems: scale-ambiguity, tracking robustness, and map initialization delay (See presentation file page 5).
Navigation stack
: Following thesee-think-act
scheme for autonomous mobile robot, we integrated visual SLAM (see,RGB-Deep D SLAM
, localization/ mapping), local path planner (think,Fast-Planner
, planning/ obstacle avoidance), and waypoint tracking controller (act,PD controller
, control).
Module | Employed system |
---|---|
Visual SLAM | ORB_SLAM2 |
Monocular Depth Estimation | packnet-sfm |
Local path planner | Fast-Planner |
Controller | PD controller |
We make 3
seperated workspaces. The installation instruction for each package is in the correspoding link.
- catkin_ws
catkin_ws/src
├── [drone_controller](https://github.com/surfii3z/drone_controller/tree/thesis)
├── [Fast-Planner](https://github.com/surfii3z/Fast-Planner/tree/thesis)
├── [image_proc](https://github.com/surfii3z/image_proc/tree/thesis)
├── [image_undistort](https://github.com/surfii3z/image_undistort/tree/thesis)
└── [tello_driver](https://github.com/surfii3z/tello_driver/tree/thesis)
- packnet_ros_ws
packnet_ros_ws/src/
├── [packnet_sfm_ros](https://github.com/surfii3z/packnet_sfm_ros)
└── vision_opencv
- ORB_SLAM2
[ORB_SLAM2](https://github.com/surfii3z/ORB_SLAM2/tree/thesis)
I experimented this system on DGX-Station (V100)
, tested with Ubuntu 18.04 LTS
, python 3.6.9
, TensorRT-7.1.3.4
, PyTorch 1.4.0
, Cuda 11.0
and CuDNN 8.0.5
.
We need to use 8 terminals to run
# -------------------------
# 00_roscore
roscore
# -------------------------
# 01_depth_estimation_module
# pwd: /path/to/packnet_ros_ws
source install/setup.bash --extend
rosrun packnet_sfm_ros trt_packnet_node
# -------------------------
# 02_tello_driver
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch tello_driver thesis_tello_node.launch
# -------------------------
# 03_visual_SLAM
# pwd: /path/to/ORB_SLAM2/ros
source devel/setup.bash
roslaunch orb_slam2 rgbd_tello_crop.launch
# -------------------------
# 04_path_planner_visualization
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch plan_manage rviz.launch
# -------------------------
# 05_path_planner
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch plan_manage tello_kino_replan.launch
# -------------------------
# 06_drone_controller
# pwd: /path/to/catkin_ws
source devel/setup.bash
roslaunch drone_controller waypoint_controller.launch
# -------------------------
# 07_mission
# pwd: /path/to/catkin_ws
source devel/setup.bash
rosrun drone_controller thesis_mission.py
NOTE: The monocular depth estimation network must be trained based on the application.
Please see this playlist on Youtube