add-mmpose-sdk (#259)

* Torchscript support (#159) * support torchscript * add nms * add torchscript configs and update deploy process and dump-info * typescript -> torchscript * add torchscript custom extension support * add ts custom ops again * support mmseg unet * [WIP] add optimizer for torchscript (#119) * add passes * add python api * Torchscript optimizer python api (#121) * add passes * add python api * use python api instead of executable * Merge Master, update optimizer (#151) * [Feature] add yolox ncnn (#29) * add yolox ncnn * add ncnn android performance of yolox * add ut * fix lint * fix None bugs for ncnn * test codecov * test codecov * add device * fix yapf * remove if-else for img shape * use channelshuffle optimize * change benchmark after channelshuffle * fix yapf * fix yapf * fuse continuous reshape * fix static shape deploy * fix code * drop pad * only static shape * fix static * fix docstring * Added mask overlay to output image, changed fprintf info messages to … (#55) * Added mask overlay to output image, changed fprintf info messages to stdout * Improved box filtering (filter area/score), make sure roi coordinates stay within bounds * clang-format * Support UNet in mmseg (#77) * Repeatdataset in train has no CLASSES & PALETTE * update result for unet * update docstring for mmdet * remove ppl for unet in docs * fix ort wrap about input type (#81) * Fix memleak (#86) * delete [] * fix build error when enble MMDEPLOY_ACTIVE_LEVEL * fix lint * [Doc] Nano benchmark and tutorial (#71) * add cls benchmark * add nano zh-cn benchmark and en tutorial * add device row * add doc path to index.rst * fix typo * [Fix] fix missing deploy_core (#80) * fix missing deploy_core * mv flag to demo * target link * [Docs] Fix links in Chinese doc (#84) * Fix docs in Chinese link * Fix links * Delete symbolic link and add links to html * delete files * Fix link * [Feature] Add docker files (#67) * add gpu and cpu dockerfile * fix lint * fix cpu docker and remove redundant * use pip instead * add build arg and readme * fix grammar * update readme * add chinese doc for dockerfile and add docker build to build.md * grammar * refine dockerfiles * add FAQs * update Dpplcv_DIR for SDK building * remove mmcls * add sdk demos * fix typo and lint * update FAQs * [Fix]fix check_env (#101) * fix check_env * update * Replace convert_syncbatchnorm in mmseg (#93) * replace convert_syncbatchnorm with revert_sync_batchnorm from mmcv * change logger * [Doc] Update FAQ for TensorRT (#96) * update FAQ * comment * [Docs]: Update doc for openvino installation (#102) * fix docs * fix docs * fix docs * fix mmcv version * fix docs * rm blank line * simplify non batch nms (#99) * [Enhacement] Allow test.py to save evaluation results (#108) * Add log file * Delete debug code * Rename logger * resolve comments * [Enhancement] Support mmocr v0.4+ (#115) * support mmocr v0.4+ * 0.4.0 -> 0.4.1 * fix onnxruntime wrapper for gpu inference (#123) * fix ncnn wrapper for ort-gpu * resolve comment * fix lint * Fix typo (#132) * lock mmcls version (#131) * [Enhancement] upgrade isort in pre-commit config (#141) * [Enhancement] upgrade isort in pre-commit config by refering to mmflow pr #87 * fix lint * remove .isort.cfg and put its known_third_party to setup.cfg * Fix ci for mmocr (#144) * fix mmocr unittests * remove useless * lock mmdet maximum version to 2.20 * pip install -U numpy * Fix capture_output (#125) Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com> * configs for all tasks * use torchvision roi align * remote unnecessary code * fix ut * fix ut * export * det dynamic * det dynamic * add ut * fix ut * add ut and docs * fix ut * skip torchscript ut if no ops available * add torchscript option to build.md * update benchmark and resolve comments * resolve conflicts * rename configs * fix mrcnn cuda test * remove useless * add version requirements to docs and comments to codes * enable empty image exporting for torchscript and accelerate ORT inference for MRCNN * rebase * update example for torchscript.md * update FAQs for torchscript.md * resolve comments * only use torchvision roi_align for torchscript * fix ut * use torchvision roi align when pool model is avg * resolve comments Co-authored-by: grimoire <streetyao@live.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com> * Update supported mmseg models (#181) * fix ocrnet cascade decoder * update mmseg support models * update mmseg configs * support emanet and icnet * set max K of TopK for tensorrt * update supported models for mmseg in docs * add test for emamodule * add configs and update docs * Update docs * update benchmark * [Features]Support mmdet3d (#103) * add mmdet3d code * add code * update code * [log]This commit finish pointpillar export and evaluate on onnxruntime.The model is sample with nvidia repo model * add tensorrt config * fix config * update * support for tensorrt * add config * fix config` * fix apis about torch2onnx * update * mmdet3d deploy version1.0 * map is ok * fix code * version1.0 * fix code * fix visual * fix bug * tensorrt support success * add docstring * add docs * fix docs * fix comments * fix comment * fix comment * fix openvino wrapper * add unit test * fix device about cpu * fix comment * fix show_result * fix lint * fix requirments * remove ci about det3d * fix ut * add ut data * support for new version pointpillars * fix comment * fix support_list * fix comments * fix config name * [Enhancement] Additional arguments support for OpenVINO Model Optimizer (#178) * Add mo args. * [Docs]: update docs and argument descriptions (#196) * bump version to v0.4.0 * update docs and argument descriptions * revert version change * fix unnecessary change of config for dynamic exportation (#199) * fix mmcls get classes (#215) * fix mmcls get classes * resolve comment * resolve comment * Add ModelOptimizerOptions. * Fix merge bugs. * Update mmpose.md (#224) * [Dostring]add example in apis docstring (#214) * add example in apis docstring * add backend example in docstring * rm blank line * Fixed get_mo_options_from_cfg args * fix l2norm test Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: Haofan Wang <frankmiracle@outlook.com> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: grimoire <yaoqian@sensetime.com> * add-mmpose-codebase * fix ci * fix img_shape after TopDownAffine * rename TopDown module -> XheadDecode & implement regression decode * align keypoints_from_heatmap * remove hardcode keypoint_head, need refactor, current only support topdown config * add mmpose python api * update mmpose-python code * can't clip fake box * fix rebase error * fix rebase error * link mspn decoder to base decoder * fix ci * compile with gcc7.5 * remove no use code * fix * fix prompt * remove unnecessary cv::parallel_for_ * rewrite TopdownHeatmapMultiStageHead.inference_model * add comment * add more detail docstring why use _cs2xyxy in sdk backend * fix Registry name * remove no use param & add comment of output result Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: grimoire <streetyao@live.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com> Co-authored-by: Haofan Wang <frankmiracle@outlook.com>
open-mmlab · Mar 31, 2022 · 43e1787 · 43e1787
1 parent c404a0b
commit 43e1787
Show file tree

Hide file tree

Showing 21 changed files with 1,422 additions and 20 deletions.
diff --git a/configs/mmpose/pose-detection_sdk_static.py b/configs/mmpose/pose-detection_sdk_static.py
@@ -0,0 +1,14 @@
+_base_ = ['./pose-detection_static.py', '../_base_/backends/sdk.py']
+
+codebase_config = dict(model_type='sdk')
+
+backend_config = dict(pipeline=[
+    dict(type='LoadImageFromFile', channel_order='bgr'),
+    dict(
+        type='Collect',
+        keys=['img'],
+        meta_keys=[
+            'image_file', 'center', 'scale', 'rotation', 'bbox_score',
+            'flip_pairs'
+        ])
+])
diff --git a/csrc/apis/c/CMakeLists.txt b/csrc/apis/c/CMakeLists.txt
@@ -5,7 +5,7 @@ project(capis)
 include(${CMAKE_SOURCE_DIR}/cmake/MMDeploy.cmake)
 
 if ("all" IN_LIST MMDEPLOY_CODEBASES)
-    set(TASK_LIST "classifier;detector;segmentor;text_detector;text_recognizer;restorer;model")
+    set(TASK_LIST "classifier;detector;segmentor;text_detector;text_recognizer;pose_detector;restorer;model")
 else ()
     set(TASK_LIST "model")
     if ("mmcls" IN_LIST MMDEPLOY_CODEBASES)
@@ -24,6 +24,9 @@ else ()
         list(APPEND TASK_LIST "text_detector")
         list(APPEND TASK_LIST "text_recognizer")
     endif ()
+    if ("mmpose" IN_LIST MMDEPLOY_CODEBASES)
+        list(APPEND TASK_LIST "pose_detector")
+    endif ()
 endif ()
 
 foreach (TASK ${TASK_LIST})

diff --git a/csrc/apis/c/pose_detector.cpp b/csrc/apis/c/pose_detector.cpp
@@ -0,0 +1,190 @@
+// Copyright (c) OpenMMLab. All rights reserved.
+
+#include "pose_detector.h"
+
+#include <numeric>
+
+#include "codebase/mmpose/mmpose.h"
+#include "core/device.h"
+#include "core/graph.h"
+#include "core/mat.h"
+#include "core/tensor.h"
+#include "core/utils/formatter.h"
+#include "handle.h"
+
+using namespace std;
+using namespace mmdeploy;
+
+namespace {
+
+const Value& config_template() {
+  // clang-format off
+  static Value v {
+    {
+      "pipeline", {
+        {"input", {"img_with_boxes"}},
+        {"output", {"key_points_unflat"}},
+        {
+          "tasks", {
+            {
+              {"name", "flatten"},
+              {"type", "Flatten"},
+              {"input", {"img_with_boxes"}},
+              {"output", {"patch_flat", "patch_index"}},
+            },
+            {
+              {"name", "pose-detector"},
+              {"type", "Inference"},
+              {"params", {{"model", "TBD"},{"batch_size", 1}}},
+              {"input", {"patch_flat"}},
+              {"output", {"key_points"}}
+            },
+            {
+              {"name", "unflatten"},
+              {"type", "Unflatten"},
+              {"input", {"key_points", "patch_index"}},
+              {"output", {"key_points_unflat"}},
+            }
+          }
+        }
+      }
+    }
+  };
+  // clang-format on
+  return v;
+}
+
+template <class ModelType>
+int mmdeploy_pose_detector_create_impl(ModelType&& m, const char* device_name, int device_id,
+                                       mm_handle_t* handle) {
+  try {
+    auto value = config_template();
+    value["pipeline"]["tasks"][1]["params"]["model"] = std::forward<ModelType>(m);
+
+    auto pose_estimator = std::make_unique<Handle>(device_name, device_id, std::move(value));
+
+    *handle = pose_estimator.release();
+    return MM_SUCCESS;
+
+  } catch (const std::exception& e) {
+    MMDEPLOY_ERROR("exception caught: {}", e.what());
+  } catch (...) {
+    MMDEPLOY_ERROR("unknown exception caught");
+  }
+  return MM_E_FAIL;
+}
+
+}  // namespace
+
+int mmdeploy_pose_detector_create(mm_model_t model, const char* device_name, int device_id,
+                                  mm_handle_t* handle) {
+  return mmdeploy_pose_detector_create_impl(*static_cast<Model*>(model), device_name, device_id,
+                                            handle);
+}
+
+int mmdeploy_pose_detector_create_by_path(const char* model_path, const char* device_name,
+                                          int device_id, mm_handle_t* handle) {
+  return mmdeploy_pose_detector_create_impl(model_path, device_name, device_id, handle);
+}
+
+int mmdeploy_pose_detector_apply(mm_handle_t handle, const mm_mat_t* mats, int mat_count,
+                                 mm_pose_detect_t** results) {
+  return mmdeploy_pose_detector_apply_bbox(handle, mats, mat_count, nullptr, nullptr, results);
+}
+
+int mmdeploy_pose_detector_apply_bbox(mm_handle_t handle, const mm_mat_t* mats, int mat_count,
+                                      const mm_rect_t* bboxes, const int* bbox_count,
+                                      mm_pose_detect_t** results) {
+  if (handle == nullptr || mats == nullptr || mat_count == 0 || results == nullptr) {
+    return MM_E_INVALID_ARG;
+  }
+
+  try {
+    auto pose_detector = static_cast<Handle*>(handle);
+    Value input{Value::kArray};
+    auto result_count = 0;
+    for (int i = 0; i < mat_count; ++i) {
+      mmdeploy::Mat _mat{mats[i].height,       mats[i].width, PixelFormat(mats[i].format),
+                         DataType(mats->type), mats[i].data,  Device{"cpu"}};
+
+      Value img_with_boxes;
+      if (bboxes && bbox_count) {
+        for (int j = 0; j < bbox_count[i]; ++j) {
+          Value obj;
+          obj["ori_img"] = _mat;
+          float width = bboxes[j].right - bboxes[j].left + 1;
+          float height = bboxes[j].bottom - bboxes[j].top + 1;
+          obj["box"] = {bboxes[j].left, bboxes[j].top, width, height, 1.0};
+          obj["rotation"] = 0.f;
+          img_with_boxes.push_back(obj);
+        }
+        bboxes += bbox_count[i];
+        result_count += bbox_count[i];
+      } else {
+        // inference whole image
+        Value obj;
+        obj["ori_img"] = _mat;
+        obj["box"] = {0, 0, _mat.width(), _mat.height(), 1.0};
+        obj["rotation"] = 0.f;
+        img_with_boxes.push_back(obj);
+        result_count += 1;
+      }
+      input.front().push_back(img_with_boxes);
+    }
+
+    auto output = pose_detector->Run(std::move(input)).value().front();
+
+    auto pose_outputs = from_value<vector<vector<mmpose::PoseDetectorOutput>>>(output);
+
+    std::vector<int> counts;
+    if (bboxes && bbox_count) {
+      counts = std::vector<int>(bbox_count, bbox_count + mat_count);
+    } else {
+      counts.resize(mat_count, 1);
+    }
+    std::vector<int> offsets{0};
+    std::partial_sum(begin(counts), end(counts), back_inserter(offsets));
+
+    auto deleter = [&](mm_pose_detect_t* p) {
+      mmdeploy_pose_detector_release_result(p, offsets.back());
+    };
+
+    std::unique_ptr<mm_pose_detect_t[], decltype(deleter)> _results(
+        new mm_pose_detect_t[result_count]{}, deleter);
+
+    for (int i = 0; i < mat_count; ++i) {
+      auto& pose_output = pose_outputs[i];
+      for (int j = 0; j < pose_output.size(); ++j) {
+        auto& res = _results[offsets[i] + j];
+        auto& box_result = pose_output[j];
+        int sz = box_result.key_points.size();
+
+        res.point = new mm_pointf_t[sz];
+        res.score = new float[sz];
+        res.length = sz;
+        for (int k = 0; k < sz; k++) {
+          res.point[k].x = box_result.key_points[k].bbox[0];
+          res.point[k].y = box_result.key_points[k].bbox[1];
+          res.score[k] = box_result.key_points[k].score;
+        }
+      }
+    }
+    *results = _results.release();
+    return MM_SUCCESS;
+
+  } catch (const std::exception& e) {
+    MMDEPLOY_ERROR("exception caught: {}", e.what());
+  } catch (...) {
+    MMDEPLOY_ERROR("unknown exception caught");
+  }
+  return MM_E_FAIL;
+}
+
+void mmdeploy_pose_detector_release_result(mm_pose_detect_t* results, int count) {
+  for (int i = 0; i < count; ++i) {
+    delete[] results[i].point;
+    delete[] results[i].score;
+  }
+  delete[] results;
+}
+void mmdeploy_pose_detector_destroy(mm_handle_t handle) { delete static_cast<Handle*>(handle); }
diff --git a/csrc/apis/c/pose_detector.h b/csrc/apis/c/pose_detector.h
@@ -0,0 +1,97 @@
+// Copyright (c) OpenMMLab. All rights reserved.
+
+/**
+ * @file pose_detector.h
+ * @brief Interface to MMPose task
+ */
+
+#ifndef MMDEPLOY_SRC_APIS_C_POSE_DETECTOR_H_
+#define MMDEPLOY_SRC_APIS_C_POSE_DETECTOR_H_
+
+#include "common.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+typedef struct mm_pose_detect_t {
+  mm_pointf_t* point;  ///< keypoint
+  float* score;        ///< keypoint score
+  int length;          ///< number of keypoint
+} mm_pose_detect_t;
+
+/**
+ * @brief Create a pose detector instance
+ * @param[in] model an instance of mmpose model created by
+ * \ref mmdeploy_model_create_by_path or \ref mmdeploy_model_create in \ref model.h
+ * @param[in] device_name name of device, such as "cpu", "cuda", etc.
+ * @param[in] device_id id of device.
+ * @param[out] handle handle of the created pose detector, which must be destroyed
+ * by \ref mmdeploy_pose_detector_destroy
+ * @return status code of the operation
+ */
+MMDEPLOY_API int mmdeploy_pose_detector_create(mm_model_t model, const char* device_name,
+                                               int device_id, mm_handle_t* handle);
+
+/**
+ * @brief Create a pose detector instance
+ * @param[in] model_path path to pose detection model
+ * @param[in] device_name name of device, such as "cpu", "cuda", etc.
+ * @param[in] device_id id of device.
+ * @param[out] handle handle of the created pose detector, which must be destroyed
+ * by \ref mmdeploy_pose_detector_destroy
+ * @return status code of the operation
+ */
+MMDEPLOY_API int mmdeploy_pose_detector_create_by_path(const char* model_path,
+                                                       const char* device_name, int device_id,
+                                                       mm_handle_t* handle);
+
+/**
+ * @brief Apply pose detector to a batch of images with full image roi
+ * @param[in] handle pose detector's handle created by \ref
+ * mmdeploy_pose_detector_create_by_path
+ * @param[in] images a batch of images
+ * @param[in] count number of images in the batch
+ * @param[out] results a linear buffer contains the pose result, must be release
+ * by \ref mmdeploy_pose_detector_release_result
+ * @return status code of the operation
+ */
+MMDEPLOY_API int mmdeploy_pose_detector_apply(mm_handle_t handle, const mm_mat_t* mats,
+                                              int mat_count, mm_pose_detect_t** results);
+
+/**
+ * @brief Apply pose detector to a batch of images supplied with bboxes(roi)
+ * @param[in] handle pose detector's handle created by \ref
+ * mmdeploy_pose_detector_create_by_path
+ * @param[in] images a batch of images
+ * @param[in] image_count number of images in the batch
+ * @param[in] bboxes bounding boxes(roi) detected by mmdet
+ * @param[in] bbox_count number of bboxes of each \p images, must be same length as \p images
+ * @param[out] results a linear buffer contains the pose result, which has the same length as \p
+ * bboxes, must be release by \ref mmdeploy_pose_detector_release_result
+ * @return status code of the operation
+ */
+MMDEPLOY_API int mmdeploy_pose_detector_apply_bbox(mm_handle_t handle, const mm_mat_t* mats,
+                                                   int mat_count, const mm_rect_t* bboxes,
+                                                   const int* bbox_count,
+                                                   mm_pose_detect_t** results);
+
+/** @brief Release result buffer returned by \ref mmdeploy_pose_detector_apply or \ref
+ * mmdeploy_pose_detector_apply_bbox
+ * @param[in] results result buffer by pose detector
+ * @param[in] count length of \p result
+ */
+MMDEPLOY_API void mmdeploy_pose_detector_release_result(mm_pose_detect_t* results, int count);
+
+/**
+ * @brief destroy pose_detector
+ * @param[in] handle handle of pose_detector created by \ref
+ * mmdeploy_pose_detector_create_by_path or \ref mmdeploy_pose_detector_create
+ */
+MMDEPLOY_API void mmdeploy_pose_detector_destroy(mm_handle_t handle);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif  // MMDEPLOY_SRC_APIS_C_POSE_DETECTOR_H_
diff --git a/csrc/apis/python/CMakeLists.txt b/csrc/apis/python/CMakeLists.txt
@@ -24,6 +24,7 @@ mmdeploy_python_add_module(segmentor)
 mmdeploy_python_add_module(text_detector)
 mmdeploy_python_add_module(text_recognizer)
 mmdeploy_python_add_module(restorer)
+mmdeploy_python_add_module(pose_detector)
 
 pybind11_add_module(${PROJECT_NAME} ${MMDEPLOY_PYTHON_SRCS})