Nanodet C API. Onnx and Libtorch (JIT) modules implementations and Li…

…btorch Installation. (#352) * Onnx and Jit tracing-scripting implementation with python and c api inference. Docs and tests have changed acordingly * Implementation of libtorch and torch vision installation as part of tool installation. * add licence and file name corection * style, inconclusive cppcheck fixes. * Fixes for C API initilization errors * Deleting no used code * macos cppcheck -> C style fixes * macos cppcheck -c style fixes * c-style pointer delceration fixes * c-style pointer delceration fixes * cpp casting casting fixes * bug fixes * Update projects/python/perception/object_detection_2d/nanodet/export_torchscript.py Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update install_torch_c_api.sh * Update dependencies/install_torch_c_api.sh Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/nanodet_c.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/nanodet_c.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/nanodet_c.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/nanodet_c.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/opendr_utils.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update include/target.h Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update nanodet.md * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update nanodet.md * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update Makefile * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update docs/reference/nanodet.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update nanodet.md * Update projects/c_api/Makefile Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update projects/python/perception/object_detection_2d/nanodet/README.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update eval_demo.py * Update projects/python/perception/object_detection_2d/nanodet/export_onnx.py Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update projects/python/perception/object_detection_2d/nanodet/README.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * Update projects/python/perception/object_detection_2d/nanodet/README.md Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> * installing libtorch and vision with respect to CUDA of the user * better explanations of variables in optimization runtimes * small typing fixes * Fix dependency * Update dependencies.ini deleting torchmetrics * update docs for new data structures and utilities of c api * new source, samples, utilities and test for onnx C API. The new scripts are implemente the feed forward of almost all the onnx models that python api is provided. * bug fixes in onnx exporting. * Revert "update docs for new data structures and utilities of c api" This reverts commit 467c1f1. * Revert "bug fixes in onnx exporting." This reverts commit fcd4f9c. * Revert "Revert "bug fixes in onnx exporting."" This reverts commit 953a193. * Revert "new source, samples, utilities and test for onnx C API. The new scripts are implemente the feed forward of almost all the onnx models that python api is provided." This reverts commit 617572d. * Revert "bug fixes in onnx exporting." This reverts commit fcd4f9c. * revert last commits, change files for cpp styles as wiki says and better directory construction for easier navigation and new tools implementation. * change function name to be uniform with the others * doc for c nanodet * Added small Json parser and installation script for easier navigation and universal use for future c api * Update projects/c_api/README.md Co-authored-by: Nikolaos Passalis <passalis@users.noreply.github.com> * Update dependencies/install_torch_c_api.sh Co-authored-by: Nikolaos Passalis <passalis@users.noreply.github.com> * Update dependencies/install_torch_c_api.sh Co-authored-by: Nikolaos Passalis <passalis@users.noreply.github.com> * Update install_torch_c_api.sh not need for specified vision version in sh * Update download_torch.py not need for environment variables to be passed * Update utilities.py delete additional print from debugging * Fixes * Fixes of suggestions Combine optimizations and inference in one script * Fixes of suggestions Add more JSON parser capabilities Fix face recognition threshold not readed from JSON * Fixes of suggestions Delete not used loggers. * Fixes of suggestions * added json parser in utilities * unnecessary use of filesystem, easier implementation in embeded * default directory for saves change to `temp` * change to warnings errors and exceptions * Transfer warnings for jit and ort simultaneously loaded from _load to infer. * Change naming style. All .cpp files have only CPP style naming and .c files have CPP style naming in CPP functions and C style in everything else. * license test update * Apply suggestions from code review Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Apply suggestions from code review * fixe bug to find cuda * add automatic random colors * better implementation of optimize * add XMLBaseDataset option * train bug fix * train bug fix * tipo fix * add changes and some better namings Co-authored-by: ad-daniel <44834743+ad-daniel@users.noreply.github.com> Co-authored-by: ad-daniel <daniel.dias@epfl.ch> Co-authored-by: Nikolaos Passalis <passalis@users.noreply.github.com> Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
opendr-eu · Jan 24, 2023 · 2a5e5f1 · 2a5e5f1
1 parent b842bab
commit 2a5e5f1
Show file tree

Hide file tree

Showing 88 changed files with 2,787 additions and 1,850 deletions.
diff --git a/Makefile b/Makefile
@@ -37,6 +37,8 @@ install_compilation_dependencies:
 	@+echo "#"; echo "# * Install Compilation Dependencies *"; echo "#"
 	@+cd dependencies; ./install.sh compilation
 	@+cd dependencies; ./install_onnx.sh
+	@+cd dependencies; ./install_rapidjson.sh
+	@+cd dependencies; ./install_torch_c_api.sh
 	@+make --silent -C src/opendr/control/mobile_manipulation $(TARGET) OPENDR_HOME="$(OPENDR_HOME)";
 	@+make --silent -C src/opendr/control/single_demo_grasp $(TARGET) OPENDR_HOME="$(OPENDR_HOME)";
 

diff --git a/dependencies/download_torch.py b/dependencies/download_torch.py
@@ -0,0 +1,125 @@
+# Copyright 2020-2022 OpenDR European Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import argparse
+import glob
+from urllib.request import urlretrieve
+import os
+import warnings
+
+
+def search_on_path(filenames):
+    for p in os.environ.get('PATH', '').split(os.pathsep):
+        for filename in filenames:
+            full = os.path.join(p, filename)
+            if os.path.exists(full):
+                return os.path.abspath(full)
+    return None
+
+
+def get_cuda_path():
+    nvcc_path = search_on_path(('nvcc', 'nvcc.exe'))
+    if nvcc_path is not None:
+        cuda_path_default = os.path.normpath(os.path.join(os.path.dirname(nvcc_path), '..'))
+        _cuda_path = cuda_path_default
+    elif os.path.exists('/usr/local/cuda'):
+        _cuda_path = '/usr/local/cuda'
+    else:
+        _cuda_path = None
+
+    return _cuda_path
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--cuda_path", help="Path to installed cuda", type=str, default=None)
+    parser.add_argument("--opendr_device", help="Target device for installation",
+                        type=str, choices=["gpu", "cpu"], default="gpu")
+    parser.add_argument("--torch_version", help="Specifies LibTorch version to be installed", type=str, default="1.9.0")
+    args = parser.parse_args()
+
+    COMPATIBILITY_VERSIONS = {
+        "1.13.1": "0.14.1",
+        "1.13.0": "0.14.0",
+        "1.12.0": "0.13.0",
+        "1.11.0": "0.12.0",
+        "1.10.2": "0.11.3",
+        "1.10.1": "0.11.2",
+        "1.10.0": "0.11.1",
+        "1.9.1": "0.10.1",
+        "1.9.0": "0.10.0",
+    }
+
+    warnings.simplefilter("error")
+
+    TORCH_VERSION = args.torch_version
+    VISION_VERSION = COMPATIBILITY_VERSIONS[TORCH_VERSION]
+
+    CUDA_VERSION = None
+    DEVICE = None
+    # Find Device
+    if args.opendr_device == "gpu":
+        try:
+            if args.cuda_path is None:
+                CUDA_PATH = get_cuda_path()
+            else:
+                CUDA_PATH = args.cuda_path
+            version_file_type = glob.glob(f"{CUDA_PATH}/version*")
+            if version_file_type[0].endswith('.txt'):
+                version_file = open(f"{CUDA_PATH}/version.txt", mode='r')
+                version_line = version_file.readlines()
+                version_line = version_line[0].replace(".", "")
+                CUDA_VERSION = version_line[13:16]
+            elif version_file_type[0].endswith('.json'):
+                version_file = open(f"{CUDA_PATH}/version.json", mode='r')
+                version_dict = json.load(version_file)
+                CUDA_VERSION = version_dict["cuda"]["version"]
+                CUDA_VERSION = CUDA_VERSION.replace(".", "")
+                CUDA_VERSION = CUDA_VERSION[:3]
+            else:
+                warnings.warn("\033[93m No CUDA version file found.")
+            DEVICE = f"cu{CUDA_VERSION}"
+        except:
+            warnings.warn("\033[93m No CUDA installation found.\n"
+                          "Please install CUDA or specify CUDA path with export CUDA_PATH=/path/to/your/cuda.")
+            exit()
+    else:
+        DEVICE = "cpu"
+
+    # Download Libtorch
+    try:
+        file_url_libtorch = f"https://download.pytorch.org/libtorch/{DEVICE}/" \
+                   f"libtorch-cxx11-abi-shared-with-deps-{TORCH_VERSION}%2B{DEVICE}.zip"
+
+        DOWNLOAD_DIRECTORY = "libtorch.zip"
+
+        urlretrieve(file_url_libtorch, DOWNLOAD_DIRECTORY)
+
+    except:
+        warnings.warn("\033[93m No LibTorch found for your specific device and torch version.\n"
+                      "Please choose another version of torch or install a different version of CUDA.\n"
+                      "Please refer to https://download.pytorch.org/whl/torch_stable.html")
+        exit()
+    # Download Vision
+    try:
+        file_url_vision = f"https://github.com/pytorch/vision/archive/refs/tags/" \
+                          f"v{VISION_VERSION}.tar.gz"
+        DOWNLOAD_DIRECTORY = "vision.tar.gz"
+        urlretrieve(file_url_vision, DOWNLOAD_DIRECTORY)
+    except:
+        warnings.warn("\033[93m No torchvision found for your specific torch version.\n"
+                      "Please refer to https://github.com/pytorch/vision for more information.")
+        exit()
+
diff --git a/dependencies/install_rapidjson.sh b/dependencies/install_rapidjson.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+if [ ! -d /usr/local/include/rapidjson ]; then
+
+  VERSION="1.1.0"
+
+  wget https://github.com/Tencent/rapidjson/archive/refs/tags/v${VERSION}.tar.gz --quiet
+  tar zxf v${VERSION}.tar.gz
+  cd rapidjson-${VERSION}
+  sudo mkdir -p /usr/local/include/rapidjson
+  sudo mv include/rapidjson/* /usr/local/include/rapidjson
+  cd ..
+  rm -rf rapidjson-${VERSION}
+  rm -rf v${VERSION}.tar.gz
+
+
+fi
diff --git a/dependencies/install_torch_c_api.sh b/dependencies/install_torch_c_api.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+
+if [[ -z "$TORCH_VERSION" ]];
+then
+  echo "Torch version not defined, version 1.9.0 will be installed."
+  echo "For a specific Torch version please define TORCH_VERSION with 'export TORCH_VERSION=x.x.x'"
+  TORCH_VERSION="1.9.0"
+fi
+
+if [ ! -f /usr/local/lib/libtorchvision.so ]; then
+  TORCH_DIRECTORY="/usr/local/libtorch"
+
+  if [[ "$OPENDR_DEVICE" == "gpu" ]]
+  then
+    echo "Downloading and installing LibTorch and torchvision (gpu support) ..."
+    GPU="on"
+    DEVICE="cu"${CUDA_VERSION}
+    CUDA_COMPILER="/usr/local/cuda/bin/nvcc"
+  else
+    echo "Downloading and installing LibTorch and torchvision (cpu-only) ..."
+    GPU="off"
+    DEVICE="cpu"
+  fi
+
+  # Find CUDA version and download torch and vision
+  echo "Downloading LibTorch and torchvision ..."
+  # Make sure that we can download files
+  if [[ -z "$CUDA_PATH" ]];
+  then
+      python3 ./download_torch.py --opendr_device "$OPENDR_DEVICE" --torch_version "$TORCH_VERSION"
+  else
+      python3 ./download_torch.py --opendr_device "$OPENDR_DEVICE" --torch_version "$TORCH_VERSION" --cuda_path "$CUDA_PATH"
+  fi
+  echo "Downloading Libtorch and torchvision done."
+
+  # TORCH INSTALLATION
+  unzip -qq libtorch.zip
+  cd libtorch
+
+  sudo mkdir -p ${TORCH_DIRECTORY}
+  sudo cp -r ./* ${TORCH_DIRECTORY}
+  cd ..
+
+  # TORCH VISION INSTALLATION
+  tar zxf vision.tar.gz
+  mv vision-* vision
+  cd vision
+  sudo mkdir -p build
+  cd build
+  sudo cmake .. -DCMAKE_CUDA_COMPILER=${CUDA_COMPILER} -DCMAKE_PREFIX_PATH=${TORCH_DIRECTORY} -DWITH_CUDA=${GPU}
+  sudo make
+  sudo make install
+  cd ../..
+
+  # CLEAN
+  sudo rm -rf libtorch
+  sudo rm -rf libtorch.zip
+
+  sudo rm -rf vision
+  sudo rm -rf vision.tar.gz
+
+  sudo ldconfig
+
+fi
diff --git a/docs/reference/c-data-h.md b/docs/reference/c-data-h.md
@@ -3,24 +3,24 @@
 
 The *data.h* header provides definitions of OpenDR data types that can be used in the C API of OpenDR.
 
-### struct *opendr_image_t*
+### struct *OpendrImageT*
 ```C
-struct opendr_image {
+struct OpendrImage {
     void *data;
 };
-typedef struct opendr_image opendr_image_t;
+typedef struct OpendrImage OpendrImageT;
 ```
 
 
-The *opendr_image_t* structure provides a data structure for storing OpenDR images. 
+The *OpendrImageT* structure provides a data structure for storing OpenDR images. 
 Every function in the C API receiving images is expected to use this structure.
 Helper functions that directly convert images into this format are provided in *opendr_utils.h*.
 
-The *opendr_image_t* structure has the following field:
+The *OpendrImageT* structure has the following field:
 
 #### `void *data` field
 
 A pointer where image data are stored. 
-*opendr_image_t* is using internally OpenCV images (*cv::Mat*) for storing images. 
+*OpendrImageT* is using internally OpenCV images (*cv::Mat*) for storing images. 
 Therefore, only a pointer to the memory location of the corresponding *cv::Mat* is stored.
 Please note that the user is not expected to directly manipulate these data without first converting them into OpenCV data type or using the corresponding functions provided in *opendr_utils.h*.
diff --git a/docs/reference/c-face-recognition-h.md b/docs/reference/c-face-recognition-h.md
@@ -3,62 +3,62 @@
 
 The *face_recognition.h* header provides function definitions that can be used for accessing the OpenDR face recognition tool.
 
-### Struct *face_recognition_model_t*
+### Struct *FaceRecognitionModelT*
 ```C
 
-struct face_recognition_model {
+struct FaceRecognitionModel {
 ...
 };
-typedef struct face_recognition_model face_recognition_model_t;
+typedef struct FaceRecognitionModel FaceRecognitionModelT;
 ```
-The *face_recognition_model_t* structure keeps all the neccesary information that are required by the OpenDR face recognition tool (e.g., model weights, normalization information, database for person recognition, ONNX session information, etc.).
+The *FaceRecognitionModelT* structure keeps all the necessary information that are required by the OpenDR face recognition tool (e.g., model weights, normalization information, database for person recognition, ONNX session information, etc.).
 
 
-### Function *load_face_recognition_model()*
+### Function *loadFaceRecognitionModel()*
 ```C
-void load_face_recognition_model(const char *model_path, face_recognition_model_t *model);
+void loadFaceRecognitionModel(const char *modelPath, FaceRecognitionModelT *model);
 ```
- Loads a face recognition model saved in the local filesystem (*model path*) in OpenDR format.
+ Loads a face recognition model saved in the local filesystem (*modelPath*) in OpenDR format.
  This function also initializes a CPU-based ONNX session for performing inference using this model.
  The pre-trained models should follow the OpenDR conventions.
  The Python API can be used to train and export an optimized OpenDR model that can be used for inference using the C API.
  
-### Function *free_face_recognition_model()*
+### Function *freeFaceRecognitionModel()*
 ```C
-void free_face_recognition_model(face_recognition_model_t *model);
+void freeFaceRecognitionModel(FaceRecognitionModelT *model);
 ```
 Releases the memory allocated for a face recognition model (*model*).
 
 
-### Function *infer_face_recognition()*
+### Function *inferFaceRecognition()*
 ```C
-opendr_category_target_t infer_face_recognition(face_recognition_model_t *model, opendr_image_t *image);
+OpendrCategoryTargetT inferFaceRecognition(FaceRecognitionModelT *model, OpendrImageT *image);
 ```
 This function perform inference using a face recognition model (*model*) and an input image (*image*).
 The function returns an OpenDR category structure with the inference results.
 
 
-### Function *decode_category_face_recognition()*
+### Function *decodeCategoryFaceRecognition()*
 ```C
-void decode_category_face_recognition(face_recognition_model_t *model, opendr_category_target_t category, char *person_name);
+void decodeCategoryFaceRecognition(FaceRecognitionModelT *model, OpendrCategoryTargetT category, char *personName);
 ```
 Returns the name of a recognized person by decoding the category id into a string (this function uses the information from the built person database).
 
 
-### Function *build_database_face_recognition()*
+### Function *buildDatabaseFaceRecognition()*
 ```C
-void build_database_face_recognition(const char *database_folder, const char *output_path, face_recognition_model_t *model);
+void buildDatabaseFaceRecognition(const char *databaseFolder, const char *outputPath, faceRecognitionModelT *model);
 ```
 Build a face recognition database (containing images for persons to be recognized). 
-This function expects the *database_folder* to have the same format as the main Python toolkit.
-The function calculates the features of the person that are contained in the database and it stores it into a binary file that can be then loaded to perform inference (*output_path*).
-A loaded face recongition model should be provided (*model*), since this model will be used for the feature extraction process.
+This function expects the (*databaseFolder*) to have the same format as the main Python toolkit.
+The function calculates the features of the person that are contained in the database and it stores it into a binary file that can be then loaded to perform inference (*outputPath*).
+A loaded face recognition model should be provided (*model*), since this model will be used for the feature extraction process.
 
-### Function *load_database_face_recognition()*
+### Function *loadDatabaseFaceRecognition()*
 ```C
-void load_database_face_recognition(const char *database_path, face_recognition_model_t *model);
+void loadDatabaseFaceRecognition(const char *databasePath, FaceRecognitionModelT *model);
 
 ```
-Loads an already built database (*database_path) into a face recognition model (*model*).
+Loads an already built database (*databasePath*) into a face recognition model (*model*).
 After this step, the model can be used for performing inference. 
 
diff --git a/docs/reference/c-object-detection-2d-nanodet-jit-h.md b/docs/reference/c-object-detection-2d-nanodet-jit-h.md
@@ -0,0 +1,47 @@
+## C_API: object_detection_2d_nanodet_jit.h
+
+
+The *object_detection_2d_nanodet_jit.h* header provides function definitions that can be used for accessing the OpenDR object detection 2D Nanodet tool.
+
+### Struct *NanodetModelT*
+```C
+
+struct NanodetModel {
+  ...
+};
+typedef struct NanodetModel NanodetModelT;
+```
+The *NanodetModelT* structure keeps all the necessary information that are required by the OpenDR object detection 2D Nanodet tool (e.g., model weights, normalization information, etc.).
+
+
+### Function *loadNanodetModel()*
+```C
+void loadNanodetModel(char *modelPath, char *device, int height, int width, float scoreThreshold, NanodetModelT *model);
+```
+Loads a Nanodet object detection model saved in the local filesystem (*modelPath*) in OpenDR format.
+This function also initializes a (*device*) JIT network for performing inference using this model.
+The pre-trained models should follow the OpenDR conventions.
+The Python API can be used to train and export an optimized OpenDR model that can be used for inference using the C API.
+
+### Function *freeNanodetModel()*
+```C
+void freeNanodetModel(NanodetModelT *model);
+```
+Releases the memory allocated for an object detection 2D Nanodet model (*model*).
+
+
+### Function *inferNanodet()*
+```C
+OpendrDetectionVectorTargetT inferNanodet(NanodetModelT *model, OpendrImageT *image);
+```
+This function performs inference using an object detection 2D Nanodet model (*model*) and an input image (*image*).
+The function returns an OpenDR detection vector structure with the inference results.
+
+
+### Function *drawBboxes()*
+```C
+void drawBboxes(OpendrImageT *image, NanodetModelT *model, OpendrDetectionVectorTargetT *detectionsVector);
+```
+This function draws the given detections (*detectionsVector*) onto the input image (*image*) and then shows the image on screen.
+The (*model*) keeps all the necessary information.
+