example/notebook/image_classification_model_deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Registering, Converting and deploying Image Classification Model by ModelCI\n",
    "\n",
    "This is a getting started tutorial for those who are new to ML-ModelCI, by the end of this tutorial, you will be able to: \n",
    "\n",
    "- Install ML-ModelCI.\n",
    "- Setup environment.\n",
    "- Start and stop ModelCI service.\n",
    "- Master basic usages of ML-ModelCI, such as model loading, registering,retrieving, converting and deploying.\n",
    "- Have a basic understanding of machine learning model lifecycle.\n",
    "\n",
    "## 1. Installation\n",
    "\n",
    "Here are some prequisities before installation\n",
    "\n",
    "- Python version: 3.7\n",
    "- Docker service installed and started\n",
    "- NVIDIA Container Toolkit( For GPU-enabled environment )\n",
    "- Manually install [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html) if your linux distribution is not Ubuntu\n",
    "\n",
    "We can install the ModelCI python package based on <https://github.com/cap-ntu/ML-Model-CI/#installation-using--pip>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/cap-ntu/ML-Model-CI.git\n",
    "cd ML-Model-CI\n",
    "pip install -q ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Start the ModelCI Service\n",
    "\n",
    "For the first time you start the ModelCI service, you need to setup all the environment variables in a single `.env` file by the following script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read env-backend.env ...\n",
      "Read env-mongodb.env ...\n",
      "Read env-frontend.env ...\n",
      "Write .env for backend with setup:\n",
      " {\n",
      "  \"PROJECT_NAME\": \"modelci\",\n",
      "  \"SERVER_HOST\": \"localhost\",\n",
      "  \"SERVER_PORT\": \"8000\",\n",
      "  \"SECRET_KEY\": \"2a6c03b9ca06cd8fc3cf506f0ba924cb735f15918d54758426fd7282366a5e19\",\n",
      "  \"MONGO_HOST\": \"localhost\",\n",
      "  \"MONGO_PORT\": \"27017\",\n",
      "  \"MONGO_USERNAME\": \"modelci\",\n",
      "  \"MONGO_PASSWORD\": \"modelci@2020\",\n",
      "  \"MONGO_DB\": \"modelci\",\n",
      "  \"MONGO_AUTH_SOURCE\": \"modelci\",\n",
      "  \"BACKEND_CORS_ORIGINS\": \"localhost:3333\"\n",
      "}\n",
      "Write .env for frontend with setup:\n",
      " {\n",
      "  \"HOST\": \"localhost\",\n",
      "  \"PORT\": \"3333\",\n",
      "  \"REACT_APP_BACKEND_URL\": \"localhost:8000\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# set environment variables \n",
    "!python scripts/generate_env.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then start the modelci service by following command, it's recommended to execute this command in your own terminal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:04:51.579933: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2021-05-27 15:05:19,828 - ml-modelci Docker Container Manager - INFO - Container name=mongo-75712 stared\n",
      "2021-05-27 15:05:28,807 - ml-modelci Docker Container Manager - INFO - Container name=cadvisor-21447 started.\n",
      "2021-05-27 15:05:37,027 - ml-modelci Docker Container Manager - INFO - Container name=dcgm-exporter-49902 started.\n",
      "2021-05-27 15:05:47,032 - ml-modelci Docker Container Manager - INFO - gpu-metrics-exporter-66880 stared\n",
      "/home/shanshan/miniconda3/envs/test/lib/python3.7/site-packages/modelci/hub/client/trt_client.py:22: UserWarning: Module `tensorrtserver` not installed. You are not able to use TRT Client.\n",
      "  warnings.warn('Module `tensorrtserver` not installed. You are not able to use TRT Client.')\n",
      "2021-05-27 15:05:47,992 - modelci backend - INFO - Uvicorn server listening on http://localhost:8000, check full log at /home/shanshan/tmp/modelci.log\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "modelci service init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Register ResNet50 Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we load pre-trained resnet50 model from torchvision, and save the whole model file, you can refer to <https://pytorch.org/docs/stable/torchvision/models.html> for more examples of pretrained models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import models\n",
    "from pathlib import Path\n",
    "import torch\n",
    "import os\n",
    "\n",
    "model = models.resnet50(pretrained=True)\n",
    "torch_model_path = Path.home()/'.modelci/ResNet50/PyTorch-PYTORCH/Image_Classification/1.pth'\n",
    "if not Path.is_dir(torch_model_path.parent):\n",
    "    os.makedirs(torch_model_path.parent, exist_ok=True)\n",
    "torch.save(model, torch_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can specify a MLModel instance by a YAML file, here is an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "weight: \"~/.modelci/ResNet50/PyTorch-PYTORCH/Image_Classification/1.pth\"\n",
    "architecture: ResNet50\n",
    "framework: PyTorch\n",
    "engine: PYTORCH\n",
    "version: 1\n",
    "dataset: ImageNet\n",
    "task: Image_Classification\n",
    "metric:\n",
    "  acc: 0.76\n",
    "inputs:\n",
    "  - name: \"input\"\n",
    "    shape: [ -1, 3, 224, 224 ]\n",
    "    dtype: TYPE_FP32\n",
    "outputs:\n",
    "  - name: \"output\"\n",
    "    shape: [ -1, 1000 ]\n",
    "    dtype: TYPE_FP32\n",
    "convert: true\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Then we can use modelci `publish` api to register this model into modelhub, a list of generated IDs will be returned. In this case, we have three IDs, because modelci will automatically convert registered models into optimized formats,PyTorch model will be converted to TorchScipt and ONNX formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:08:22.003806: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "{'data': {'id': ['60af456c681d3e02a0099500', '60af456f681d3e02a009968b', '60af456f681d3e02a0099816']}, 'status': True}\n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub publish -f ../resnet50_explicit_path.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Display Models\n",
    "We can list all models published in MLModelCI by the following command: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-29 09:37:34.023488: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "┏━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━┳━━━━━┳━━━━━━┳━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━┓\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mID  \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mARCH\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mFRA…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mENG…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mVER…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mDA…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMET…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mSC…\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mTASK           \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mSTA…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃      ┃\u001b[1;35m \u001b[0m\u001b[1;35mNAME\u001b[0m\u001b[1;35m \u001b[0m┃      ┃      ┃      ┃     ┃      ┃     ┃                 ┃      ┃\n",
      "┡━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━╇━━━━━╇━━━━━━╇━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━┩\n",
      "│ 60b… │ Res… │ PyT… │ PYT… │ 1    │ Im… │ acc  │ 0.… │ Image_Classifi… │ 💔   │\n",
      "│      │      │      │      │      │     │      │     │                 │ Unk… │\n",
      "│      │      │      │      │      │     │      │     │                 │      │\n",
      "│ 60b… │ \u001b[1mRes…\u001b[0m │ PyT… │ TOR… │ 1    │ Im… │ acc  │ 0.… │ Image_Classifi… │ 💔   │\n",
      "│      │      │      │      │      │     │      │     │                 │ Unk… │\n",
      "│      │      │      │      │      │     │      │     │                 │      │\n",
      "│ 60b… │ \u001b[1mRes…\u001b[0m │ PyT… │ ONNX │ 1    │ Im… │ acc  │ 0.… │ Image_Classifi… │ 💔   │\n",
      "│      │      │      │      │      │     │      │     │                 │ Unk… │\n",
      "└──────┴──────┴──────┴──────┴──────┴─────┴──────┴─────┴─────────────────┴──────┘\n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view detatiled information of a single model by its ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:08:50.873389: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "\u001b[1mID       \u001b[0m  \u001b[1mArchitec…\u001b[0m  \u001b[1mFramework\u001b[0m  \u001b[1mVersion\u001b[0m  \u001b[1mPretrained\u001b[0m  \u001b[1mMetric\u001b[0m  \u001b[1mScore\u001b[0m  \u001b[1mTask       \u001b[0m\n",
      "                                          \u001b[1mDataset   \u001b[0m                            \n",
      "60af456c…  ResNet50   PyTorch    1        ImageNet    acc     0.76   Image      \n",
      "                                                                     Classifica…\n",
      "\u001b[1;38;5;43mConve…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mModel \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[1;38;5;43mInfo  \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247mServi…\u001b[0m\u001b[38;5;247m  \u001b[0mPYTOR…  \u001b[38;5;247mStatus\u001b[0m\u001b[38;5;247m  \u001b[0m💔      \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0mshans…  \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0m25      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[38;5;247mEngine\u001b[0m\u001b[38;5;247m  \u001b[0m                Unkno…                          secon…                  \n",
      "                                                        ago                     \n",
      "\u001b[1;38;5;247mInputs\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0minput   \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        3,      \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "                                        224,                                    \n",
      "                                        224]                                    \n",
      "\u001b[1;38;5;247mOutpu…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0moutput  \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        1000]   \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "\u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mProfi…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mResul…\u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247m  N.A.\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub detail 60af456c681d3e02a0099500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:09:03.627985: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "\u001b[1mID       \u001b[0m  \u001b[1mArchitec…\u001b[0m  \u001b[1mFramework\u001b[0m  \u001b[1mVersion\u001b[0m  \u001b[1mPretrained\u001b[0m  \u001b[1mMetric\u001b[0m  \u001b[1mScore\u001b[0m  \u001b[1mTask       \u001b[0m\n",
      "                                          \u001b[1mDataset   \u001b[0m                            \n",
      "60af456f…  ResNet50   PyTorch    1        ImageNet    acc     0.76   Image      \n",
      "                                                                     Classifica…\n",
      "\u001b[1;38;5;43mConve…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mModel \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[1;38;5;43mInfo  \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247mServi…\u001b[0m\u001b[38;5;247m  \u001b[0mTORCH…  \u001b[38;5;247mStatus\u001b[0m\u001b[38;5;247m  \u001b[0m💔      \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0mshans…  \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0m37      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[38;5;247mEngine\u001b[0m\u001b[38;5;247m  \u001b[0m                Unkno…                          secon…                  \n",
      "                                                        ago                     \n",
      "\u001b[1;38;5;247mInputs\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0minput   \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        3,      \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "                                        224,                                    \n",
      "                                        224]                                    \n",
      "\u001b[1;38;5;247mOutpu…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0moutput  \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        1000]   \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "\u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mProfi…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mResul…\u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247m  N.A.\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub detail 60af456f681d3e02a009968b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:09:18.770745: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "\u001b[1mID       \u001b[0m  \u001b[1mArchitec…\u001b[0m  \u001b[1mFramework\u001b[0m  \u001b[1mVersion\u001b[0m  \u001b[1mPretrained\u001b[0m  \u001b[1mMetric\u001b[0m  \u001b[1mScore\u001b[0m  \u001b[1mTask       \u001b[0m\n",
      "                                          \u001b[1mDataset   \u001b[0m                            \n",
      "60af456f…  ResNet50   PyTorch    1        ImageNet    acc     0.76   Image      \n",
      "                                                                     Classifica…\n",
      "\u001b[1;38;5;43mConve…\u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mModel \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[1;38;5;43mInfo  \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247mServi…\u001b[0m\u001b[38;5;247m  \u001b[0mONNX  \u001b[38;5;247mStatus\u001b[0m\u001b[38;5;247m  \u001b[0m💔      \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0mshansh…  \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0m52       \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[38;5;247mEngine\u001b[0m\u001b[38;5;247m  \u001b[0m              Unkno…                           seconds                  \n",
      "                                                       ago                      \n",
      "\u001b[1;38;5;247mInputs\u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0minput   \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1, 3,  \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32     \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                      224,     \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                         \n",
      "                                      224]                                      \n",
      "\u001b[1;38;5;247mOutpu…\u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0moutput  \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,     \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32     \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                      1000]    \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                         \n",
      "\u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mProfi…\u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mResul…\u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247m  N.A.\u001b[0m\u001b[38;5;247m  \u001b[0m      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m         \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub detail 60af456f681d3e02a0099816"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Convert Models\n",
    "We can convert models mannually. \n",
    "\n",
    "You can refer to <https://github.com/cap-ntu/ML-Model-CI/blob/master/docs/tutorial/convert.md> for more details.\n",
    "\n",
    "In the following example, we will convert ONNX model into TensorRT format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.types.bo import IOShape\n",
    "from modelci.types.trtis_objects import ModelInputFormat\n",
    "# get ONNX model saved path\n",
    "onnx_model_path = Path.home()/'.modelci/ResNet50/PyTorch-ONNX/Image_Classification/1.onnx'\n",
    "trt_model_path = Path.home()/'.modelci/ResNet50/PyTorch-TRT/Image_Classification/1'\n",
    "# specify inputs and outputs shape\n",
    "inputs = [IOShape([-1, 3, 224, 224], dtype=float, name='INPUT__0', format=ModelInputFormat.FORMAT_NCHW)]\n",
    "outputs = [IOShape([-1, 1000], dtype=float, name='probs')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading ONNX file from path /home/shanshan/.modelci/ResNet50/PyTorch-ONNX/Image_Classification/1.onnx...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from modelci.hub.converter import convert\n",
    "\n",
    "convert(\n",
    "    model=onnx_model_path,\n",
    "    src_framework='onnx',\n",
    "    dst_framework='trt',\n",
    "    save_path=trt_model_path,\n",
    "    inputs=inputs,\n",
    "    outputs=outputs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Deploy Models As a Service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ModelCI also supports model deployment, in this case, we will deploy a ONNX model, the following script will start a docker container of  onnx-serving. The port 8001 is the port of HTTP endpoint while 8002 is for gRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Container: 651b81eb03>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from modelci.hub.deployer.dispatcher import serve\n",
    "\n",
    "batch_size = 8\n",
    "server_name = 'onnx'\n",
    "\n",
    "serve(save_path=onnx_model_path, device='cuda:0', name=server_name, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "651b81eb034f   mlmodelci/\u001b[01;31m\u001b[Konnx\u001b[m\u001b[K-serving:latest-gpu   \"/bin/sh -c 'python …\"   19 seconds ago   Up 12 seconds   0.0.0.0:8001->8000/tcp, 0.0.0.0:8002->8001/tcp   \u001b[01;31m\u001b[Konnx\u001b[m\u001b[K\r\n"
     ]
    }
   ],
   "source": [
    "!docker ps | grep onnx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Profile Models\n",
    "\n",
    "Once a model is  registered and deployed as a service, we can profile our model to get information such as  memory usage and response latency.\n",
    "\n",
    "At first, we need to retrieve the ONNX model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.persistence.service import ModelService\n",
    "model = ModelService.get_model_by_id(\"60af456f681d3e02a0099816\")\n",
    "model.inputs[0].dtype = 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we build a client for ONNX serving platform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.hub.client.onnx_client import CVONNXClient\n",
    "test_img_bytes = torch.rand(3, 224, 224)\n",
    "onnx_client = CVONNXClient(test_img_bytes, model, batch_num=20, batch_size=8, asynchronous=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can init a profiler, one thing to note is the server name must be the same as the serving container's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " latency: 0.3299 sec throughput: 24.2529 req/sec\n",
      " latency: 0.3296 sec throughput: 24.2684 req/sec\n",
      " latency: 0.3288 sec throughput: 24.3295 req/sec\n",
      " latency: 0.3320 sec throughput: 24.0994 req/sec\n",
      " latency: 0.3295 sec throughput: 24.2803 req/sec\n",
      " latency: 0.3291 sec throughput: 24.3056 req/sec\n",
      " latency: 0.3339 sec throughput: 23.9628 req/sec\n",
      " latency: 0.3312 sec throughput: 24.1563 req/sec\n",
      " latency: 0.3297 sec throughput: 24.2679 req/sec\n",
      " latency: 0.3295 sec throughput: 24.2761 req/sec\n",
      " latency: 0.3298 sec throughput: 24.2566 req/sec\n",
      " latency: 0.3299 sec throughput: 24.2480 req/sec\n",
      " latency: 0.3340 sec throughput: 23.9506 req/sec\n",
      " latency: 0.3302 sec throughput: 24.2280 req/sec\n",
      " latency: 0.3297 sec throughput: 24.2668 req/sec\n",
      " latency: 0.3293 sec throughput: 24.2967 req/sec\n",
      " latency: 0.3304 sec throughput: 24.2139 req/sec\n",
      " latency: 0.3289 sec throughput: 24.3206 req/sec\n",
      " latency: 0.3293 sec throughput: 24.2927 req/sec\n",
      " latency: 0.3286 sec throughput: 24.3448 req/sec\n",
      "\n",
      "\n",
      "testing device: GeForce MX110\n",
      "total batches: 20, batch_size: 8\n",
      "total latency: 6.623807668685913 s\n",
      "total throughput: 24.15529073351584 req/sec\n",
      "50th-percentile latency: 0.32966148853302 s\n",
      "95th-percentile latency: 0.3338588953018189 s\n",
      "99th-percentile latency: 0.3339882707595825 s\n",
      "total GPU memory: 2101870592.0 bytes\n",
      "average GPU memory usage percentage: 0.2840\n",
      "average GPU memory used: 596901888.0 bytes\n",
      "average GPU utilization: 47.8333%\n",
      "completed at 2021-05-29 09:39:27.955909\n"
     ]
    }
   ],
   "source": [
    "from modelci.hub.profiler import Profiler\n",
    "profiler = Profiler(model_info=model, server_name='onnx', inspector=onnx_client)\n",
    "dps = profiler.diagnose(device='cuda:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<modelci.types.bo.dynamic_profile_result_bo.DynamicProfileResultBO at 0x7f9244126e10>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.Stop the ModelCI Service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-04-28 00:59:10.251530: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2021-04-28 00:59:12,892 - ml-modelci Docker Container Manager - INFO - Container name=gpu-metrics-exporter-31125 stopped.\n",
      "2021-04-28 00:59:13,535 - ml-modelci Docker Container Manager - INFO - Container name=cadvisor-12065 stopped.\n",
      "2021-04-28 00:59:14,859 - ml-modelci Docker Container Manager - INFO - Container name=mongo-49205 stopped.\n",
      "2021-04-28 00:59:14,994 - modelci backend - WARNING - No process is listening on port 8000\n"
     ]
    }
   ],
   "source": [
    "!conda activate test && modelci service stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or  you can remove all the stoppped docker containers by the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-27 15:04:21.636005: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2021-05-27 15:04:23,508 - ml-modelci Docker Container Manager - INFO - Container name=gpu-metrics-exporter-78925 stopped.\n",
      "2021-05-27 15:04:25,534 - ml-modelci Docker Container Manager - INFO - Container name=cadvisor-4409 stopped.\n",
      "2021-05-27 15:04:33,859 - ml-modelci Docker Container Manager - INFO - Container name=mongo-97086 stopped.\n",
      "2021-05-27 15:04:33,971 - modelci backend - WARNING - No process is listening on port 8000\n",
      "2021-05-27 15:04:34,541 - ml-modelci Docker Container Manager - INFO - Container 8cd47d49c1f594684ba9d61cb399c55fc66190c6f714b74014e377c9cee9e9e3 is removed.\n",
      "2021-05-27 15:04:35,250 - ml-modelci Docker Container Manager - INFO - Container ef64e035053f4b9d04bf2c2ae465e1ddce30a44bc97623ea145a8919e6900020 is removed.\n",
      "2021-05-27 15:04:35,995 - ml-modelci Docker Container Manager - INFO - Container 4f1b2203ec42d2f83ccaa0b970bd3de9e4b55c3229e79536edbad50497e4da27 is removed.\n",
      "2021-05-27 15:04:36,750 - ml-modelci Docker Container Manager - INFO - Container 4e6fb506ce97619521459209b690658ba2fac1319b7460d4a1ddfa5f079d6897 is removed.\n"
     ]
    }
   ],
   "source": [
    "!conda activate test && modelci service clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "```raw\n",
    "Copyright 2020 Nanyang Technological University, Singapore\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "    http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}