[Colaboratory]Tensorflow2-ObjectDetectionAPI-Colab-Hands-On.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VUCNWeZ2wBFp"
      },
      "source": [
        "# <b>[Colaboratory]Tensorflow2-ObjectDetectionAPI-Colab-Hands-On.ipynb</b><br>\n",
        "このノートブックはTensorflow2 Object Detection APIの学習/推論のハンズオン用スクリプトです(This notebook is a hands-on script for learning/inference of the Tensorflow2 Object Detection API)<br><br>\n",
        "Colaboratoryのハードウェア アクセラレータ設定をGPUにして実行してください(Set the hardware accelerator setting of Colaboratory to GPU and execute it.)<br><br>\n",
        "Github URL : [Tensorflow2-ObjectDetectionAPI-Colab-Hands-On](https://github.com/Kazuhito00/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gr36a9P3whVQ"
      },
      "source": [
        "# <b>Google Driveマウント(Google Drive mount)</b>\n",
        "※checkpoint、saved model格納先 (checkpoint, saved model storage location)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RdjNWM8HwitO"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('./gdrive')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FATF4iQ4ANdF"
      },
      "source": [
        "# <b>Tensorflow Object Detection API設定(Set Tensorflow Object Detection API)</b>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpgBfSdZAT82"
      },
      "source": [
        "### Protocol Buffers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tvMZTXVbARNZ"
      },
      "outputs": [],
      "source": [
        "!curl -OL https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip\n",
        "!unzip protoc-3.2.0-linux-x86_64.zip -d protoc3\n",
        "!sudo mv protoc3/bin/* /usr/local/bin/\n",
        "!sudo mv protoc3/include/* /usr/local/include/\n",
        "!rm -rf protoc3 protoc-3.2.0-linux-x86_64.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "80tsIvjRAV6m"
      },
      "outputs": [],
      "source": [
        "!git clone --depth 1 https://github.com/tensorflow/models\n",
        "%cd /content/models/research\n",
        "\n",
        "!/usr/local/bin/protoc object_detection/protos/*.proto --python_out=."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OFkjFNTGAcaT"
      },
      "source": [
        "### 必要ライブラリインストール(Installation of required libraries)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nem-gkjuAV_U"
      },
      "outputs": [],
      "source": [
        "!cp /content/models/research/object_detection/packages/tf2/setup.py .\n",
        "# ToDo：https://github.com/tensorflow/models/issues/10375\n",
        "!python -m pip install . --use-deprecated=legacy-resolver"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h7LjWeQFAV3b"
      },
      "outputs": [],
      "source": [
        "# インストール成否確認(Confirmation of successful installation)\n",
        "!python /content/models/research/object_detection/builders/model_builder_tf2_test.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A9_s3CAtzKFI"
      },
      "source": [
        "# <b>Tensorflow2-ObjectDetectionAPI-Colab-Hands-Onリポジトリクローン(Clone repository)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yg73roLzzVyW"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/Kazuhito00/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kfwGjF4kols"
      },
      "source": [
        "# <b>TFRecordをアップロード(Upload TF Record)</b>\n",
        "「Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/02_tfrecord」にVoTTからエクスポートしたTFRecordとtf_label_map.pbtxtを格納してください。<br>(Please store the TFRecord and tf_label_map.pbtxt exported from VoTT  in \"Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/02_tfrecord\".)\n",
        "<br><br>\n",
        "![](https://user-images.githubusercontent.com/37477845/94039064-31f8cc80-fe02-11ea-81f2-28427d759099.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08ZcM9-Bw8Xy"
      },
      "source": [
        "# <b>学習データ/検証データ 分割(Split Training data/validation data)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5g-2VP8j6bgB"
      },
      "outputs": [],
      "source": [
        "original_data_dir = '/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/02_tfrecord'\n",
        "train_data_dir = '/content/models/research/train_data'\n",
        "val_data_dir = '/content/models/research/val_data'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u7NoOeHkwk_V"
      },
      "outputs": [],
      "source": [
        "# ディレクトリ作成(Create a directory)\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "shutil.rmtree(train_data_dir, ignore_errors=True)\n",
        "shutil.rmtree(val_data_dir, ignore_errors=True)\n",
        "os.mkdir(train_data_dir)\n",
        "os.mkdir(val_data_dir)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_XAfNhTVjUes"
      },
      "outputs": [],
      "source": [
        "import glob\n",
        "\n",
        "# ファイル数カウント(Count the number of files)\n",
        "file_count = len(glob.glob(original_data_dir + '/*.tfrecord'))\n",
        "print('File count : ' + str(file_count))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KxJk4E1P11pT"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "\n",
        "# 学習データ/検証データ 分割(Split Training data/validation data.)\n",
        "train_ratio = 0.75\n",
        "\n",
        "file_list = glob.glob(original_data_dir + '/*.tfrecord')\n",
        "random_sample_list = random.sample(file_list, file_count)\n",
        "\n",
        "# ディレクトリへコピー(Copy to directory)\n",
        "for index, filepath in enumerate(random_sample_list):\n",
        "    if index < int(file_count * train_ratio):\n",
        "        # 学習データ(Training data)\n",
        "        shutil.copy2(filepath, train_data_dir)\n",
        "    else:\n",
        "        # 検証データ(Validation data)\n",
        "        shutil.copy2(filepath, val_data_dir)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_A-UvwHC7hLW"
      },
      "source": [
        "# <b>学習済モデル(Pre-Trained model)</b>\n",
        "\n",
        "このハンズオンでは「Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model」に学習済モデル(EfficientDet-D0)が格納してあります。<br>(In this hands-on, the pre-trained model(EfficientDet-D0) is stored in \"Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model\")<br><br>学習済モデル取得元(Pre-Trained model acquisition source)：http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1_bm5VtPk4SO"
      },
      "source": [
        "# <b>パイプラインコンフィグアップロード(Pipeline config upload)</b>\n",
        "\n",
        "パイプラインコンフィグを修正し「Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model」にアップロードしてください(Please modify the pipeline config and upload it to \"Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model\")<br><br>\n",
        "パイプラインコンフィグは以下の行を修正します(Pipeline Config modifies the following line)<br>\n",
        "\n",
        "* 3行目(Line 3)：クラス数(num_classes)<br>変更前(Before) : 90<br>変更後(After) : 1<br>\n",
        "* 134行目(Line 134)：バッチサイズ(batch_size)<br>変更前(Before) : 128<br>変更後(After) : 16<br>\n",
        "* 161行目(Line 161)：ファインチューニング用のチェックポイント格納先(fine_tune_checkpoint)<br>変更前(Before) : \"PATH_TO_BE_CONFIGURED\"<br>変更後(After) : \"/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model/efficientdet_d0_coco17_tpu-32/checkpoint/ckpt-0\"\n",
        "* 167行目(Line 167)：ファインチューニング方法(fine_tune_checkpoint_type)<br>変更前(Before) : \"classification\"<br>変更後(After) : \"detection\"<br>\n",
        "* 168行目(Line 168)：Googleカスタム 16ビットbrain浮動小数点の使用有無(use_bfloat16)<br>変更前(Before) : true<br>変更後(After) : false<br>\n",
        "* 172行目(Line 172)：ラベルマップファイルの格納先(label_map_path)<br>変更前(Before) : \"PATH_TO_BE_CONFIGURED/label_map.txt\"<br>変更後(After) : \"/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/02_tfrecord/tf_label_map.pbtxt\"<br>\n",
        "* 174行目(Line 174)：学習データの格納先(input_path)<br>変更前(Before) : \"PATH_TO_BE_CONFIGURED/train2017-?????-of-00256.tfrecord\"<br>変更後(After) : \"/content/models/research/train_data/??????.tfrecord\"<br>\n",
        "* 185行目(Line 185)：ラベルマップファイルの格納先(label_map_path)<br>変更前(Before) : \"PATH_TO_BE_CONFIGURED/label_map.txt\"<br>変更後(After) : \"/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/02_tfrecord/tf_label_map.pbtxt\"<br>\n",
        "* 189行目(Line 189)：バリデーションデータの格納先(input_path)<br>変更前(Before) : \"PATH_TO_BE_CONFIGURED/val2017-?????-of-00032.tfrecord\"<br>変更後(After) : \"/content/models/research/val_data/??????.tfrecord\"\n",
        "<br><br>\n",
        "![](https://user-images.githubusercontent.com/37477845/94040113-83558b80-fe03-11ea-8d8a-a5304efcca1d.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORYqE_A0Bm0y"
      },
      "source": [
        "# <b>モデル訓練(Model training)</b>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wO6EZSbnIY-s"
      },
      "source": [
        "### Googleドライブに保存先ディレクトリを作成(Create directory in Google Drive)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R-ElsZLHxHvK"
      },
      "outputs": [],
      "source": [
        "!mkdir '/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-_7GvE2KIbMk"
      },
      "source": [
        "### TensorBoard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ctvO4SJD11u3"
      },
      "outputs": [],
      "source": [
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a-M__9iNHK80"
      },
      "outputs": [],
      "source": [
        "tensorboard --logdir '/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EpEgC85TIeVR"
      },
      "source": [
        "# <b>学習(Training)</b>\n",
        "\n",
        "学習はColaboratory上で1000ステップにつき、約25分かかります(Learning takes about 25 minutes per 1000 steps on the Colaboratory)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WLOMjRA28IHg"
      },
      "outputs": [],
      "source": [
        "!python object_detection/model_main_tf2.py \\\n",
        "    --pipeline_config_path=\"/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model/pipeline.config\" \\\n",
        "    --model_dir=\"/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On\" \\\n",
        "    --num_train_steps=1000 \\\n",
        "    --alsologtostderr \\\n",
        "    --sample_1_of_n_eval_examples=1 \\\n",
        "    --num_eval_steps=100"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0ZTEuHAnMxp"
      },
      "source": [
        "# <b>saved model形式へエクスポート(Export to saved-model format)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VuL08MMmIBzQ"
      },
      "outputs": [],
      "source": [
        "!python object_detection/exporter_main_v2.py \\\n",
        "    --input_type=image_tensor \\\n",
        "    --pipeline_config_path=\"/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/03_pretrained_model/pipeline.config\" \\\n",
        "    --trained_checkpoint_dir=\"/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On\" \\\n",
        "    --output_directory=\"/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/output\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mB6Z34Wcs6NW"
      },
      "source": [
        "# <b>モデルロード(Load model)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XUaigXpntNVz"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "model_path = '/content/gdrive/My Drive/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/output/saved_model'\n",
        "\n",
        "DEFAULT_FUNCTION_KEY = 'serving_default'\n",
        "loaded_model = tf.saved_model.load(model_path)\n",
        "inference_func = loaded_model.signatures[DEFAULT_FUNCTION_KEY]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtKvTGJXLBZE"
      },
      "source": [
        "# <b>推論(Inference)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1VHkShurKtVC"
      },
      "outputs": [],
      "source": [
        "# 推論対象のテスト画像一覧(List of test images to be inferred)\n",
        "import glob\n",
        "import copy\n",
        "import cv2\n",
        "from google.colab.patches import cv2_imshow\n",
        "\n",
        "test_data_dir = '/content/models/research/Tensorflow2-ObjectDetectionAPI-Colab-Hands-On/04_test_data'\n",
        "testfile_list = sorted(glob.glob(test_data_dir + '/*.jpg'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EEIBDIeRpswu"
      },
      "outputs": [],
      "source": [
        "# 推論用関数(Function for inference)\n",
        "def run_inference_single_image(image, inference_func):\n",
        "    tensor = tf.convert_to_tensor(image)\n",
        "    output = inference_func(tensor)\n",
        "\n",
        "    output['num_detections'] = int(output['num_detections'][0])\n",
        "    output['detection_classes'] = output['detection_classes'][0].numpy()\n",
        "    output['detection_boxes'] = output['detection_boxes'][0].numpy()\n",
        "    output['detection_scores'] = output['detection_scores'][0].numpy()\n",
        "    return output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RPdZ-PLMLZeY"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "\n",
        "filenames = []\n",
        "\n",
        "# 動画書き出し用設定(VideoWriter setting)\n",
        "temp_image = cv2.imread(testfile_list[0], cv2.IMREAD_UNCHANGED)\n",
        "image_width, image_height = temp_image.shape[1], temp_image.shape[0]\n",
        "fourcc = 'mp4v'\n",
        "writer_fourcc = cv2.VideoWriter_fourcc(*fourcc)\n",
        "videowriter = cv2.VideoWriter('result.mp4', writer_fourcc, 10, (image_width, image_height))\n",
        "\n",
        "# 推論(Inference)\n",
        "for filecount, testfile in enumerate(testfile_list):\n",
        "    image = cv2.imread(testfile, cv2.IMREAD_UNCHANGED)\n",
        "    debug_image = copy.deepcopy(image)\n",
        "\n",
        "    image_width, image_height = image.shape[1], image.shape[0]\n",
        "    image = image[:, :, [2, 1, 0]]  # BGR2RGB\n",
        "    image_np_expanded = np.expand_dims(image, axis=0)\n",
        "\n",
        "    output = run_inference_single_image(image_np_expanded, inference_func)\n",
        "\n",
        "    num_detections = output['num_detections']\n",
        "    for i in range(num_detections):\n",
        "        score = output['detection_scores'][i]\n",
        "        bbox = output['detection_boxes'][i]\n",
        "        # class_id = output['detection_classes'][i].astype(np.int)\n",
        "\n",
        "        if score < 0.85:\n",
        "            continue\n",
        "\n",
        "        x1, y1 = int(bbox[1] * image_width), int(bbox[0] * image_height)\n",
        "        x2, y2 = int(bbox[3] * image_width), int(bbox[2] * image_height)\n",
        "\n",
        "        # 推論結果描画(Inference result drawing)\n",
        "        cv2.rectangle(debug_image, (x1, y1), (x2, y2), (255, 255, 255), 2)\n",
        "        cv2.putText(debug_image, str('{:.2f}'.format(score)), (x1, y1-10), cv2.FONT_HERSHEY_PLAIN, 1.5, (255, 255, 255), 2, cv2.LINE_AA)\n",
        "        cv2.rectangle(debug_image, (x1, y1), (x2, y2), (255, 255, 255), 2)\n",
        "    videowriter.write(debug_image)\n",
        "videowriter.release()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gXJBbDNYxnp0"
      },
      "source": [
        "# <b>推論結果確認(Inference result confirmation)</b>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W9kmLlk0-TPa"
      },
      "outputs": [],
      "source": [
        "import imageio\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.animation as animation\n",
        "from IPython.display import HTML\n",
        "\n",
        "def play_video(video, interval=100):\n",
        "    video = imageio.mimread(video)\n",
        "    fig = plt.figure(figsize=(9, 6))\n",
        "\n",
        "    movie = []\n",
        "    for i in range(len(video)):\n",
        "        img = plt.imshow(video[i], animated=True)\n",
        "        plt.axis('off')\n",
        "        movie.append([img])\n",
        "\n",
        "    anime = animation.ArtistAnimation(fig, movie, interval=interval, repeat_delay=1000)\n",
        "    plt.close()\n",
        "    return anime\n",
        "\n",
        "HTML(play_video('result.mp4').to_html5_video()) "
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "[Colaboratory]Tensorflow2-ObjectDetectionAPI-Colab-Hands-On.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}