Skip to content

GSoC 2018: Porting ViSP to Android devices

Jump to bottom
s-trinh edited this page Aug 13, 2018 · 1 revision

Whats ViSP?

ViSP standing for Visual Servoing Platform is a modular cross platform library that allows prototyping and developing applications using visual tracking and visual servoing technics at the heart of the researches done by Inria Rainbow team. ViSP is able to compute control laws that can be applied to robotic systems. It provides a set of visual features that can be tracked using real time image processing or computer vision algorithms. ViSP provides also simulation capabilities. ViSP can be useful in robotics, computer vision, augmented reality and computer animation. Our YouTube channel gives an overview of the applications that could be tackled.

About Me

Hi, I'm Akshay Sharma, junior undergraduate at Dept. of Mathematical Sciences, Indian Institute of Technology (B.H.U.) Varanasi. I'm primarily interested in developing Apps (mostly Android) with AR features, for which I used OpenCV. But the hefty app sizes and erroneous NDK code made me switch to ViSP - its open sourced, lightweight and has many inbuilt tracking and detection features. You can find my project proposal here.

About the GSoC Project

ViSP is already packaged for iOS. Community is asking a version for Android. The objective of this project is to propose to the community a SDK for Android and a set of tutorials for beginners. Expected results:

  • Create the set of tools and scripts to build ViSP for Android from source code (CMake toolchain file, python script to build the SDK...)
  • Adapt existing ViSP source code to be as much as possible compatible with Android
  • Basic set of tutorials: How to build ViSP Android SDK from source code? How to generate the visp_java on desktops? Simple vpApriltag detection demo on Android phones

Work Submission Summary

Click here for a complete list of commits.

Summary of the final work product

I created a whole new repository named GSoC-Prep for fiddling with ViSP project and to learn Java and basics of Android NDK during the first part of GSoC.

The majority of the work are contained in these pull-requests:

  • #307 Porting ViSP to android - This is the initial PR wherein we began working on SDK and discussed features to be incorporated. It's where Fabien pointed out to create a java module and later switch to Android SDK
  • #323 Building visp_java module - visp_java module comes to existence - its the set of scripts that parse C++ header files and generate NDK and Java code for it(find more of it here).
  • #392 Creating a GUI module - For testing visp_java module, we needed some graphic interface(to mark tags, additional image manipulations, etc). Java AWT was the preferred choice - its lightweight, platform independent and Java developers are well versed and will find it easy to access GUI class objects(JFrame and all).
  • #394 Wrapping imgproc module - Wrapping visp_imgproc module, for image transformations.
  • #396 Wrap detection module - This module addresses the central claim of my proposal - providing real time tag detection on Android smartphones.
  • #404 Build Android SDK - Finally, when the java module was working, we began creating the Android SDK. Its nothing but a few python scripts to handle architecture specific changes for SDK and engine apks, a special cmake file(android.toolchain.cmake) to build a cmake project for Android and lastly using ant to build Java class files.

There are many other smaller PR's. You can get the complete list here.

Alongside the project, I wrote a few blogs. They can be found here, here and here.

For fellow developers

  • Why wrap anyway? - Instead of creating Java code for the equivalent C++ code, its better to create a class (having native methods) that calls the original C++ methods via JNI. Just parse the C++ header files via some python scripts, handle some edge cases and voila - you're done. Refer here for a detailed explanation.
  • Manual Vs Automatic Wrapping - There are limitations to parsing files - some differences that are inherently present in C++ and missing in Java (or vice versa). For instance a template class in C++ can be turned to a generic class in Java, but that can't be done automatically; so such classes need to be handled manually (take this and this file as reference).
  • How to add Java support for a module - Head to the module's CMakeLists.txt and change
vp_add_module(<module-name> <flags>)

to

vp_add_module(<module-name> <flags> WRAP java)
  • Whats filelist - Its a plain text file (like this) specifying which classes to wrap in a module. If you want to wrap a module but forget to specify such a file, then scripts will wrap all classes for that module - something you might not want. Remember to use C++ only when its necessary and there's no alternative in Java. For instance, you won't to use C++ I/O on Android phones
  • Whats gen_dict.json - Just like filelist, this file must be defined for every module to be Java wrapped. It specifies a few things like which classes to ignore, which classes should have toString support in Java (equivalent to ostream&<< operator in C++), functions that are to be defined manually in a given file (note that manually wrapping a file and manually defining a method in automatically wrapped file are different), and custom type dictionary (C++ custom class/data type to Java)

Tutorials

  • Building Java Module - This tutorial will guide you to build visp_java module and allied libraries that may be used on standalone desktops(running Linux, Mac or Windows).
  • First Java Application - This tutorial will guide you to create a sample ViSP Java application, as an Eclipse project, demonstrating basic ViSP methods and simple Java AWT GUI.
  • Building ViSP Android SDK - This tutorial will guide you to create the Android SDK(and engine apks) which is used to create Android apps supporting ViSP functionalities. The SDK is compatible with all major Android architectures(x86, x86_64, armeabi-v7a, and arm64-v8a)
  • First Android App with ViSP - This tutorial will guide you to create a sample Android app for realtime AprilTag Detection.

Room for further improvement

  • Unit Testing - visp_java module and android platform are missing CI tests. We need to include pure tests(Java only) and android specific tests for different modules, and host platforms.
  • Implementing more samples - I see room for more android apps and Eclipse projects, demonstrating groups of ViSP modules, to cater a versatile audience.
  • Speed Optimization - Although its not included in my proposal, yet I believe that the project won't be worth a shot if tag detection and image manipulation isn't fast enough. Given the limited resources on Android devices, developing and improving the algorithms is worth the effort.

Acknowledgements

I would like to thank my mentors

  • Fabien Spindler (@fspindle)
  • Souriya Trinh (@s-trinh)

Despite the timezone difference, they always have been there to clear my doubts and provide suggestions. They often found and rectified lots of the initial bugs, and frequently provided the necessary information to fix these errors.
I would like thank whole GSoC team, who arranged the program so that students can undertake projects throughout the summer months and have a wonderful experience.

Clone this wiki locally