PiTeR and OpenCV
PiTeR uses OpenCV for his vision system. OpenCV is a really powerful library of computer vision utilities which can run on Linux, Windows and Mac OSX. You can use OpenCV to make your robot do all kinds of extra-ordinary things. Find out much more about OpenCV here. Many of the examples found here were based on tutorials found on this site under the OpenCV Tutorials link. Much kudos and thanks to the wonderful people at OpenCV who put these tutorials together.
Video for Linux (v4l) enables easy access to the Raspberry Pi camera. If you do a websearch, you will find other ways to access the camera from OpenCV, including some which have been highlighted on the Raspberry Pi Foundation blog. However in my opinion, using v4l is by far the easiest way to get going. v4l2 is installed by default in the latest raspbian release, but you must start the driver like this:
sudo modprobe bcm2385-v4l2
To use OpenCV on your Raspberry Pi, you need to install the package python-opencv. This can be installed using apt:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python-opencv
The python programs which test PiTeRs visual system are found here in the repository. You don’t need a robot to try these programs for yourself, you just need a Pi camera or webcam.
What follows here are some notes that relate the programs themselves to an OpenCV article published in the MagPi magazine issue 26, November 2014. They all require X to be running. All these programs can be exited by pressing escape while one of the output windows is focused.
imageCap.py - This program covers the first three steps covered in the article:
import cv2
cap = cv2.VideoCapture(-1)
success, frame = cap.read()
cv2.imshow(frame)
colour.py - This program displays six sliders to control the range of the colour values; Hue, Saturation and Value. We use these values rather than the more familiar Red, Green and Blue because they are more suited to setting up a colour range. Each colour value has a high and low which mark the ends of the range that will be accepted as the part of the image we’re looking for.
This program opens other windows, one shows the original view, another shows the result of applying the OpenCV inRange function and a third shows the final mask after noise reduction. The image capture and filtering happen inside a while loop, so that you can adjust the sliders and see the effect of your adjustments.
Note that the objective here is to locate a patch in the image, the mask does not have to be perfect. We're going to use this patch to crop the image further on down, and the symbol recognition will be done on the cropped part of the original image, so no detail will be lost.
patch.py - This program incorporates more of the concepts discussed in the article. It takes a set of fixed colour range values and outputs the image captured by the camera. If it finds a patch of colour which passes the filter (as illustrated by colour.py above), it uses the findContours and boundingRect functions discussed in the article to draw a green rectangle around the relevant area in the output image.
tracking.py - Finally, this program builds on the previous three examples to perform full symbol recognition. It uses detectAndCompute to find the key points in the image and the knnMatch function to compute a match and locate the symbol. If the match is good enough, the matched points are used to compute a transformation of a rectangle within which the symbol is located.
If everything worked perfectly, the transformed rectangle shown in blue should look as if it lies right on top of the symbol in the camera image, even if the symbol appears at an angle to the camera. It doesn’t always get it perfectly right, but it does so often enough to be impressive in my opinion. Try it! If you have trouble, try stronger lighting. Don't forget though that if you change the lighting, you will likely need to retune the HSV parameters.
Important - Unfortunately, the version of openCV installed by default by apt-get does not include the SURF feature detection that tracking.py relies on. This will be fixed in upcoming revisions of OpenCV, but for now you'll need to manually upgrade to OpenCV 2.4.9 if you wish to run the tracking.py. See the section below called Upgrade to OpenCV 2.4.9 for the required steps.
The main robot control program PiTeR.py, will integrate these experimental programs and use their output to control PiTeR to autonomously find and visit each symbol in a trail. This will appear in GitHub as I complete the integration work ahead of the cub scout meeting in November.
Once that is complete, I hope to post some pictures here, so watch this space!
Wish me luck! :-)
Upgrade to OpenCV 2.4.9
You'll need to do these steps if you want to run the most advanced program that does the symbol recognition.
- Clone the prebuilt library files found at:
git clone https://github.com/Nolaan/libopencv_24
- Create a new folder 'opencv' at /usr/local/lib
sudo mkdir /usr/local/lib/opencv
- Copy the cloned files libopencv_. to this new folder
sudo cp libopencv_24/libopencv_* /usr/local/lib/opencv
- Add the new libraries to the dynamic linker path:
export LD_LIBRARY_PATH=/usr/local/lib/opencv (or add the path to this variable if it is already set)
- Install new dependency libavformat
sudo apt-get install libavformat53
- Copy prebuilt python library into python configuration
sudo cp MagPi/cv2.so /usr/lib/pyshared/python2.7
Note that this last step changes the version of OpenCV that python uses. You may wish to copy aside the previous cv2.so library file so that you can reinstate it later if you wish.
Upgrade to OpenCV 2.4.10
After upgrading to OpenCV 2.4.9, you may notice that some of the example programs are a little slow and 'laggy'. This seems to be resolved in release 2.4.10. However, to upgrade to that, you'll need to compile it yourself. You can find instructions for doing this on the OpenCV website. Be warned, this takes about 10 hours on the Raspberry Pi. Don't forget to enable build and install of the Python driver. Have fun experimenting.