traffic-signs-detection-using-yolov3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b37a549f",
   "metadata": {
    "papermill": {
     "duration": 0.017257,
     "end_time": "2021-11-28T10:41:27.940906",
     "exception": false,
     "start_time": "2021-11-28T10:41:27.923649",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Traffic Signs Detection using YOLOv3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fec7589",
   "metadata": {
    "papermill": {
     "duration": 0.015733,
     "end_time": "2021-11-28T10:41:27.973685",
     "exception": false,
     "start_time": "2021-11-28T10:41:27.957952",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Important: The following publicly available datasets on kaggle.com are required to run this notebook:\n",
    "- [Models, YOLOv3 weights and config, videos and utils](https://www.kaggle.com/eiriksteira/gtsrb-models-idatt-2502)\n",
    "\n",
    "Resources for how to get started with datasets on kaggle can be found [here](https://www.kaggle.com/docs/datasets).\n",
    "\n",
    "\n",
    "In this notebook, a trained model from the Darknet framework detects traffic signs among 4 categories. Then a trained model in Keras classifies the detected traffic signs into one of 43 classes.\n",
    "\n",
    "The YOLOv3 detector is trained beforehand by using the Darknet framework by running the following command:\n",
    "\n",
    "<code>>> ./darknet detector train ts_data.data yolov3_ts_train.cfg darknet53.conv.74 yolov3.weights -gpus 0</code>\n",
    "\n",
    "Which generates the model weight-files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce85b91f",
   "metadata": {
    "papermill": {
     "duration": 0.015642,
     "end_time": "2021-11-28T10:41:28.007696",
     "exception": false,
     "start_time": "2021-11-28T10:41:27.992054",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "12a4bfbc",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:28.049454Z",
     "iopub.status.busy": "2021-11-28T10:41:28.048679Z",
     "iopub.status.idle": "2021-11-28T10:41:34.221292Z",
     "shell.execute_reply": "2021-11-28T10:41:34.220493Z",
     "shell.execute_reply.started": "2021-11-28T10:32:06.336449Z"
    },
    "papermill": {
     "duration": 6.197624,
     "end_time": "2021-11-28T10:41:34.221474",
     "exception": false,
     "start_time": "2021-11-28T10:41:28.023850",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "import cv2\n",
    "import time\n",
    "from timeit import default_timer as timer\n",
    "import matplotlib.pyplot as plt\n",
    "import pickle\n",
    "\n",
    "from keras.models import load_model\n",
    "\n",
    "import os\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ea84a8",
   "metadata": {
    "papermill": {
     "duration": 0.015509,
     "end_time": "2021-11-28T10:41:34.253611",
     "exception": false,
     "start_time": "2021-11-28T10:41:34.238102",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Loading labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "202df60e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:34.289071Z",
     "iopub.status.busy": "2021-11-28T10:41:34.288347Z",
     "iopub.status.idle": "2021-11-28T10:41:34.303416Z",
     "shell.execute_reply": "2021-11-28T10:41:34.303970Z",
     "shell.execute_reply.started": "2021-11-28T10:32:12.413344Z"
    },
    "papermill": {
     "duration": 0.03476,
     "end_time": "2021-11-28T10:41:34.304166",
     "exception": false,
     "start_time": "2021-11-28T10:41:34.269406",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "labels = pd.read_csv('../input/gtsrb-models-idatt-2502/label_names_yolo_v3.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a87092",
   "metadata": {
    "papermill": {
     "duration": 0.016177,
     "end_time": "2021-11-28T10:41:34.336569",
     "exception": false,
     "start_time": "2021-11-28T10:41:34.320392",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Loading the trained CNN model for Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9dd56ae4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:34.374209Z",
     "iopub.status.busy": "2021-11-28T10:41:34.373477Z",
     "iopub.status.idle": "2021-11-28T10:41:36.422209Z",
     "shell.execute_reply": "2021-11-28T10:41:36.421674Z",
     "shell.execute_reply.started": "2021-11-28T10:32:12.437017Z"
    },
    "papermill": {
     "duration": 2.069709,
     "end_time": "2021-11-28T10:41:36.422369",
     "exception": false,
     "start_time": "2021-11-28T10:41:34.352660",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "User settings:\n",
      "\n",
      "   KMP_AFFINITY=granularity=fine,verbose,compact,1,0\n",
      "   KMP_BLOCKTIME=0\n",
      "   KMP_SETTINGS=1\n",
      "   KMP_WARNINGS=0\n",
      "\n",
      "Effective settings:\n",
      "\n",
      "   KMP_ABORT_DELAY=0\n",
      "   KMP_ADAPTIVE_LOCK_PROPS='1,1024'\n",
      "   KMP_ALIGN_ALLOC=64\n",
      "   KMP_ALL_THREADPRIVATE=128\n",
      "   KMP_ATOMIC_MODE=2\n",
      "   KMP_BLOCKTIME=0\n",
      "   KMP_CPUINFO_FILE: value is not defined\n",
      "   KMP_DETERMINISTIC_REDUCTION=false\n",
      "   KMP_DEVICE_THREAD_LIMIT=2147483647\n",
      "   KMP_DISP_NUM_BUFFERS=7\n",
      "   KMP_DUPLICATE_LIB_OK=false\n",
      "   KMP_ENABLE_TASK_THROTTLING=true\n",
      "   KMP_FORCE_REDUCTION: value is not defined\n",
      "   KMP_FOREIGN_THREADS_THREADPRIVATE=true\n",
      "   KMP_FORKJOIN_BARRIER='2,2'\n",
      "   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_GTID_MODE=3\n",
      "   KMP_HANDLE_SIGNALS=false\n",
      "   KMP_HOT_TEAMS_MAX_LEVEL=1\n",
      "   KMP_HOT_TEAMS_MODE=0\n",
      "   KMP_INIT_AT_FORK=true\n",
      "   KMP_LIBRARY=throughput\n",
      "   KMP_LOCK_KIND=queuing\n",
      "   KMP_MALLOC_POOL_INCR=1M\n",
      "   KMP_NUM_LOCKS_IN_BLOCK=1\n",
      "   KMP_PLAIN_BARRIER='2,2'\n",
      "   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_REDUCTION_BARRIER='1,1'\n",
      "   KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_SCHEDULE='static,balanced;guided,iterative'\n",
      "   KMP_SETTINGS=true\n",
      "   KMP_SPIN_BACKOFF_PARAMS='4096,100'\n",
      "   KMP_STACKOFFSET=64\n",
      "   KMP_STACKPAD=0\n",
      "   KMP_STACKSIZE=8M\n",
      "   KMP_STORAGE_MAP=false\n",
      "   KMP_TASKING=2\n",
      "   KMP_TASKLOOP_MIN_TASKS=0\n",
      "   KMP_TASK_STEALING_CONSTRAINT=1\n",
      "   KMP_TEAMS_THREAD_LIMIT=4\n",
      "   KMP_TOPOLOGY_METHOD=all\n",
      "   KMP_USE_YIELD=1\n",
      "   KMP_VERSION=false\n",
      "   KMP_WARNINGS=false\n",
      "   OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'\n",
      "   OMP_ALLOCATOR=omp_default_mem_alloc\n",
      "   OMP_CANCELLATION=false\n",
      "   OMP_DEFAULT_DEVICE=0\n",
      "   OMP_DISPLAY_AFFINITY=false\n",
      "   OMP_DISPLAY_ENV=false\n",
      "   OMP_DYNAMIC=false\n",
      "   OMP_MAX_ACTIVE_LEVELS=1\n",
      "   OMP_MAX_TASK_PRIORITY=0\n",
      "   OMP_NESTED: deprecated; max-active-levels-var=1\n",
      "   OMP_NUM_THREADS: value is not defined\n",
      "   OMP_PLACES: value is not defined\n",
      "   OMP_PROC_BIND='intel'\n",
      "   OMP_SCHEDULE='static'\n",
      "   OMP_STACKSIZE=8M\n",
      "   OMP_TARGET_OFFLOAD=DEFAULT\n",
      "   OMP_THREAD_LIMIT=2147483647\n",
      "   OMP_WAIT_POLICY=PASSIVE\n",
      "   KMP_AFFINITY='verbose,warnings,respect,granularity=fine,compact,1,0'\n",
      "\n",
      "2021-11-28 10:41:34.507818: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 48, 48, 3)]  0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv_1_3x3 (Conv2D)             (None, 48, 48, 32)   864         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization (BatchNorma (None, 48, 48, 32)   128         conv_1_3x3[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "activation (Activation)         (None, 48, 48, 32)   0           batch_normalization[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "conv_2_7x1 (Conv2D)             (None, 48, 48, 48)   10752       activation[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 48, 48, 48)   192         conv_2_7x1[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, 48, 48, 48)   0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv_3_1x7 (Conv2D)             (None, 48, 48, 48)   16128       activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 48, 48, 48)   192         conv_3_1x7[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, 48, 48, 48)   0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D)    (None, 24, 24, 48)   0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dropout (Dropout)               (None, 24, 24, 48)   0           max_pooling2d[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 24, 24, 48)   192         dropout[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, 24, 24, 48)   0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv__6_1_3x1 (Conv2D)          (None, 24, 24, 64)   9216        activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv__6_2_7x1 (Conv2D)          (None, 24, 24, 64)   21504       activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv__6_3_5x1 (Conv2D)          (None, 24, 24, 64)   15360       activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 24, 24, 64)   256         conv__6_1_3x1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 24, 24, 64)   256         conv__6_2_7x1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, 24, 24, 64)   256         conv__6_3_5x1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 24, 24, 64)   0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_6 (Activation)       (None, 24, 24, 64)   0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_8 (Activation)       (None, 24, 24, 64)   0           batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv__7_1_1x3 (Conv2D)          (None, 24, 24, 64)   12288       activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv__7_2_1x7 (Conv2D)          (None, 24, 24, 64)   28672       activation_6[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv__7_3_1x5 (Conv2D)          (None, 24, 24, 64)   20480       activation_8[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 24, 24, 64)   256         conv__7_1_1x3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, 24, 24, 64)   256         conv__7_2_1x7[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 24, 24, 64)   256         conv__7_3_1x5[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 24, 24, 64)   0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_7 (Activation)       (None, 24, 24, 64)   0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_9 (Activation)       (None, 24, 24, 64)   0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 24, 24, 192)  0           activation_5[0][0]               \n",
      "                                                                 activation_7[0][0]               \n",
      "                                                                 activation_9[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2D)  (None, 12, 12, 192)  0           concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 12, 12, 192)  0           max_pooling2d_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "conv_6_3x3 (Conv2D)             (None, 12, 12, 128)  221184      dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, 12, 12, 128)  512         conv_6_3x3[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "activation_10 (Activation)      (None, 12, 12, 128)  0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv_7_3x3 (Conv2D)             (None, 12, 12, 256)  294912      activation_10[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, 12, 12, 256)  1024        conv_7_3x3[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "activation_11 (Activation)      (None, 12, 12, 256)  0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2D)  (None, 6, 6, 256)    0           activation_11[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 6, 6, 256)    0           max_pooling2d_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, 6, 6, 256)    1024        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_12 (Activation)      (None, 6, 6, 256)    0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "flatten (Flatten)               (None, 9216)         0           activation_12[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 256)          2359552     flatten[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, 256)          1024        dense[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "activation_13 (Activation)      (None, 256)          0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 256)          0           activation_13[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_14 (BatchNo (None, 256)          1024        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_14 (Activation)      (None, 256)          0           batch_normalization_14[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 43)           11051       activation_14[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 3,028,811\n",
      "Trainable params: 3,025,387\n",
      "Non-trainable params: 3,424\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "model_path =  '../input/gtsrb-models-idatt-2502/gtsrb-final.h5'\n",
    "model = load_model(model_path)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99bdbc2d",
   "metadata": {
    "papermill": {
     "duration": 0.016816,
     "end_time": "2021-11-28T10:41:36.456717",
     "exception": false,
     "start_time": "2021-11-28T10:41:36.439901",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Loading YOLOv3 network with OpenCV dnn library"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44714cce",
   "metadata": {
    "papermill": {
     "duration": 0.016654,
     "end_time": "2021-11-28T10:41:36.490395",
     "exception": false,
     "start_time": "2021-11-28T10:41:36.473741",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Loading the trained weights and cfg file into the Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1c5a359c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:36.540404Z",
     "iopub.status.busy": "2021-11-28T10:41:36.539374Z",
     "iopub.status.idle": "2021-11-28T10:41:39.344688Z",
     "shell.execute_reply": "2021-11-28T10:41:39.344079Z",
     "shell.execute_reply.started": "2021-11-28T10:32:14.189555Z"
    },
    "papermill": {
     "duration": 2.837382,
     "end_time": "2021-11-28T10:41:39.344843",
     "exception": false,
     "start_time": "2021-11-28T10:41:36.507461",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "path_to_weights = '../input/gtsrb-models-idatt-2502/yolov3_ts_train_best.weights'\n",
    "path_to_cfg = '../input/gtsrb-models-idatt-2502/yolov3_ts_train.cfg'\n",
    "\n",
    "network = cv2.dnn.readNetFromDarknet(path_to_cfg, path_to_weights)\n",
    "\n",
    "# To enable usage with GPU\n",
    "network.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)\n",
    "network.setPreferableTarget(cv2.dnn.DNN_TARGET_OPENCL_FP16)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae016518",
   "metadata": {
    "papermill": {
     "duration": 0.017184,
     "end_time": "2021-11-28T10:41:39.379258",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.362074",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Getting output layers where detections are made"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "12431289",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:39.420893Z",
     "iopub.status.busy": "2021-11-28T10:41:39.419758Z",
     "iopub.status.idle": "2021-11-28T10:41:39.424983Z",
     "shell.execute_reply": "2021-11-28T10:41:39.424252Z",
     "shell.execute_reply.started": "2021-11-28T10:32:16.819181Z"
    },
    "papermill": {
     "duration": 0.028785,
     "end_time": "2021-11-28T10:41:39.425171",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.396386",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['yolo_82', 'yolo_94', 'yolo_106']\n"
     ]
    }
   ],
   "source": [
    "# Getting names of all YOLOv3 layers\n",
    "layers_all = network.getLayerNames()\n",
    "\n",
    "# YOLOv3 detection layers that are 82, 94 and 106\n",
    "# These are the names of layers for which the output is to be computed\n",
    "layers_names_output = [layers_all[i - 1] for i in network.getUnconnectedOutLayers()]\n",
    "\n",
    "print(layers_names_output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89ac7025",
   "metadata": {
    "papermill": {
     "duration": 0.017019,
     "end_time": "2021-11-28T10:41:39.460581",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.443562",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Setting probability, threshold and colour for the bounding boxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e4215501",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:39.504172Z",
     "iopub.status.busy": "2021-11-28T10:41:39.503140Z",
     "iopub.status.idle": "2021-11-28T10:41:39.505936Z",
     "shell.execute_reply": "2021-11-28T10:41:39.505350Z",
     "shell.execute_reply.started": "2021-11-28T10:32:16.827870Z"
    },
    "papermill": {
     "duration": 0.027315,
     "end_time": "2021-11-28T10:41:39.506080",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.478765",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Minimum probability to eliminate weak detections\n",
    "probability_minimum = 0.2\n",
    "\n",
    "# Setting threshold to filtering weak bounding boxes by non-maximum suppression\n",
    "threshold = 0.2\n",
    "\n",
    "# Generating colours for bounding boxes\n",
    "colours = np.random.randint(0, 255, size=(len(labels), 3), dtype='uint8')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "375258ff",
   "metadata": {
    "papermill": {
     "duration": 0.017072,
     "end_time": "2021-11-28T10:41:39.541197",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.524125",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Reading input video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "88ed1445",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:39.581560Z",
     "iopub.status.busy": "2021-11-28T10:41:39.580853Z",
     "iopub.status.idle": "2021-11-28T10:41:39.685485Z",
     "shell.execute_reply": "2021-11-28T10:41:39.684828Z",
     "shell.execute_reply.started": "2021-11-28T10:32:16.837449Z"
    },
    "papermill": {
     "duration": 0.126891,
     "end_time": "2021-11-28T10:41:39.685683",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.558792",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reading video from a file\n",
    "video = cv2.VideoCapture('../input/gtsrb-models-idatt-2502/traffic-sign-test-video.mp4')\n",
    "\n",
    "# Writer that will be used to write processed frames\n",
    "writer = None\n",
    "\n",
    "# Variables for spatial dimensions of the frames\n",
    "h, w = None, None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12832eb",
   "metadata": {
    "papermill": {
     "duration": 0.017814,
     "end_time": "2021-11-28T10:41:39.722010",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.704196",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Processing the video frames\n",
    "The forward() function of cv2.dnn module returns a nested list containing information about all the detected objects which includes the x and y coordinates of the centre of the object detected, height and width of the bounding box, confidence and scores for all the classes of objects listed in coco.names. The class with the highest score is considered to be the predicted class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3e4c247c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:41:39.773672Z",
     "iopub.status.busy": "2021-11-28T10:41:39.772880Z",
     "iopub.status.idle": "2021-11-28T10:42:29.200654Z",
     "shell.execute_reply": "2021-11-28T10:42:29.199528Z",
     "shell.execute_reply.started": "2021-11-28T10:32:16.943146Z"
    },
    "papermill": {
     "duration": 49.460732,
     "end_time": "2021-11-28T10:42:29.200824",
     "exception": false,
     "start_time": "2021-11-28T10:41:39.740092",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ WARN:0] global /tmp/pip-req-build-0culq997/opencv/modules/dnn/src/dnn.cpp (1422) setUpNet DNN: OpenCL target is not supported with current OpenCL device (tested with GPUs only), switching to CPU.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame number 1 took 0.97270 seconds\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-11-28 10:41:40.976711: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frame number 2 took 0.62464 seconds\n",
      "Frame number 3 took 0.61929 seconds\n",
      "Frame number 4 took 0.94200 seconds\n",
      "Frame number 5 took 0.62285 seconds\n",
      "Frame number 6 took 0.63592 seconds\n",
      "Frame number 7 took 0.63684 seconds\n",
      "Frame number 8 took 0.62177 seconds\n",
      "Frame number 9 took 0.62528 seconds\n",
      "Frame number 10 took 0.62592 seconds\n",
      "Frame number 11 took 0.63127 seconds\n",
      "Frame number 12 took 0.62715 seconds\n",
      "Frame number 13 took 0.62412 seconds\n",
      "Frame number 14 took 0.61850 seconds\n",
      "Frame number 15 took 0.62691 seconds\n",
      "Frame number 16 took 0.61947 seconds\n",
      "Frame number 17 took 0.62048 seconds\n",
      "Frame number 18 took 0.62384 seconds\n",
      "Frame number 19 took 0.62417 seconds\n",
      "Frame number 20 took 0.63115 seconds\n",
      "Frame number 21 took 0.62574 seconds\n",
      "Frame number 22 took 0.62499 seconds\n",
      "Frame number 23 took 0.62690 seconds\n",
      "Frame number 24 took 0.62043 seconds\n",
      "Frame number 25 took 0.61799 seconds\n",
      "Frame number 26 took 0.62202 seconds\n",
      "Frame number 27 took 0.62283 seconds\n",
      "Frame number 28 took 0.61869 seconds\n",
      "Frame number 29 took 0.61684 seconds\n",
      "Frame number 30 took 0.62183 seconds\n",
      "Frame number 31 took 0.62288 seconds\n",
      "Frame number 32 took 0.61947 seconds\n",
      "Frame number 33 took 0.63015 seconds\n",
      "Frame number 34 took 0.62440 seconds\n",
      "Frame number 35 took 0.63747 seconds\n",
      "Frame number 36 took 0.63063 seconds\n",
      "Frame number 37 took 0.62855 seconds\n",
      "Frame number 38 took 0.62601 seconds\n",
      "Frame number 39 took 0.62639 seconds\n",
      "Frame number 40 took 0.62467 seconds\n",
      "Frame number 41 took 0.92603 seconds\n",
      "Frame number 42 took 0.62294 seconds\n",
      "Frame number 43 took 0.62900 seconds\n",
      "Frame number 44 took 0.62182 seconds\n",
      "Frame number 45 took 0.61986 seconds\n",
      "Frame number 46 took 0.61956 seconds\n",
      "Frame number 47 took 0.62376 seconds\n",
      "Frame number 48 took 0.62882 seconds\n",
      "Frame number 49 took 0.67850 seconds\n",
      "Frame number 50 took 0.62387 seconds\n",
      "Frame number 51 took 0.61759 seconds\n",
      "Frame number 52 took 0.62289 seconds\n",
      "Frame number 53 took 0.62055 seconds\n",
      "Frame number 54 took 0.61991 seconds\n",
      "Frame number 55 took 0.62198 seconds\n",
      "Frame number 56 took 0.62137 seconds\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Set default size of plots\n",
    "plt.rcParams['figure.figsize'] = (3, 3)\n",
    "\n",
    "# Variable for counting total amount of frames\n",
    "f = 0\n",
    "\n",
    "# Variable for counting total processing time\n",
    "t = 0\n",
    "\n",
    "# Catch frames in the loop\n",
    "while True:\n",
    "    # Capture frames one-by-one\n",
    "    ret, frame = video.read()\n",
    "\n",
    "    # If the frame was not retrieved\n",
    "    if not ret:\n",
    "        break\n",
    "       \n",
    "    # Get spatial dimensions of the frame for the first time\n",
    "    if w is None or h is None:\n",
    "        h, w = frame.shape[:2]\n",
    "\n",
    "    # Blob from current frame\n",
    "    blob = cv2.dnn.blobFromImage(frame, 1 / 255, (416, 416), swapRB=True, crop=False)\n",
    "\n",
    "    # Forward pass with blob through output layers\n",
    "    network.setInput(blob)\n",
    "    start = time.time()\n",
    "    output_from_network = network.forward(layers_names_output)\n",
    "    end = time.time()\n",
    "\n",
    "    # Increase counters\n",
    "    f += 1\n",
    "    t += end - start\n",
    "\n",
    "    print('Frame number {0} took {1:.5f} seconds'.format(f, end - start))\n",
    "\n",
    "    bounding_boxes = []\n",
    "    confidences = []\n",
    "    class_numbers = []\n",
    "\n",
    "    # Go through all output layers after feed forward pass\n",
    "    for result in output_from_network:\n",
    "        \n",
    "        # Go through all detections from current output layer\n",
    "        for detected_objects in result:\n",
    "            \n",
    "            # 80 classes' probabilities for current detected object\n",
    "            scores = detected_objects[5:]\n",
    "            \n",
    "            # Index of the class with the maximum value of probability\n",
    "            class_current = np.argmax(scores)\n",
    "            \n",
    "            # Get value of probability for defined class\n",
    "            confidence_current = scores[class_current]\n",
    "\n",
    "            # Eliminate weak predictions by minimum probability\n",
    "            if confidence_current > probability_minimum:\n",
    "                \n",
    "                # Scale bounding box coordinates to the initial frame size\n",
    "                box_current = detected_objects[0:4] * np.array([w, h, w, h])\n",
    "\n",
    "                # Top left corner coordinates\n",
    "                x_center, y_center, box_width, box_height = box_current\n",
    "                x_min = int(x_center - (box_width / 2))\n",
    "                y_min = int(y_center - (box_height / 2))\n",
    "\n",
    "                # Add results into prepared lists\n",
    "                bounding_boxes.append([x_min, y_min, int(box_width), int(box_height)])\n",
    "                confidences.append(float(confidence_current))\n",
    "                class_numbers.append(class_current)\n",
    "                \n",
    "\n",
    "    # Non-maximum suppression of given bounding boxes\n",
    "    # which removes redundant overlapping bounding boxes.\n",
    "    results = cv2.dnn.NMSBoxes(bounding_boxes, confidences, probability_minimum, threshold)\n",
    "\n",
    "    is_any_detected_objects_left = len(results) > 0\n",
    "\n",
    "    if is_any_detected_objects_left:\n",
    "        \n",
    "        for i in results.flatten():\n",
    "            \n",
    "            # Bounding box coordinates - width and height\n",
    "            x_min, y_min = bounding_boxes[i][0], bounding_boxes[i][1]\n",
    "            box_width, box_height = bounding_boxes[i][2], bounding_boxes[i][3]\n",
    "            \n",
    "            # Cut fragment with Traffic Sign\n",
    "            c_ts = frame[y_min:y_min + int(box_height), x_min:x_min + int(box_width), :]\n",
    "            \n",
    "            if c_ts.shape[:1] == (0,) or c_ts.shape[1:2] == (0,):\n",
    "                pass\n",
    "            else:\n",
    "                # Get preprocessed blob with Traffic Sign of required shape\n",
    "                blob_ts = cv2.dnn.blobFromImage(c_ts, scalefactor=1, size=(48, 48), swapRB=True, crop=False)\n",
    "                blob_ts = blob_ts.transpose(0, 2, 3, 1)\n",
    "      \n",
    "                # Feed the CNN model to get predicted label among 43 classes\n",
    "                predictions = model.predict(blob_ts)\n",
    "\n",
    "                # Get the class with the maximum value\n",
    "                prediction = np.argmax(predictions)\n",
    "\n",
    "                # Colour for current bounding box\n",
    "                colour_box_current = colours[class_numbers[i]].tolist()\n",
    "\n",
    "                # Draw bounding box on the original current frame\n",
    "                cv2.rectangle(frame, (x_min, y_min),\n",
    "                              (x_min + box_width, y_min + box_height),\n",
    "                              colour_box_current, 2)\n",
    "\n",
    "                # Prepare text with label and confidence for current bounding box\n",
    "                text_box_current = '{}: {:.4f}'.format(labels['SignName'][prediction],\n",
    "                                                       confidences[i])\n",
    "\n",
    "                # Put text with label and confidence on the original image\n",
    "                cv2.putText(frame, text_box_current, (x_min, y_min - 5),\n",
    "                            cv2.FONT_HERSHEY_SIMPLEX, 0.5, colour_box_current, 2)\n",
    "\n",
    "\n",
    "    # Initialize writer only once\n",
    "    if writer is None:\n",
    "        fourcc = cv2.VideoWriter_fourcc(*'mp4v')\n",
    "\n",
    "        # Writecurrent processed frame to the video file\n",
    "        writer = cv2.VideoWriter('result.mp4', fourcc, 25,\n",
    "                                 (frame.shape[1], frame.shape[0]), True)\n",
    "\n",
    "    # Write processed current frame to the file\n",
    "    writer.write(frame)\n",
    "\n",
    "\n",
    "# Release video reader and writer\n",
    "video.release()\n",
    "writer.release()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8878eae",
   "metadata": {
    "papermill": {
     "duration": 0.036007,
     "end_time": "2021-11-28T10:42:29.273115",
     "exception": false,
     "start_time": "2021-11-28T10:42:29.237108",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "238e24e5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:42:29.349754Z",
     "iopub.status.busy": "2021-11-28T10:42:29.349020Z",
     "iopub.status.idle": "2021-11-28T10:42:29.354168Z",
     "shell.execute_reply": "2021-11-28T10:42:29.354669Z",
     "shell.execute_reply.started": "2021-11-28T10:33:05.671843Z"
    },
    "papermill": {
     "duration": 0.046405,
     "end_time": "2021-11-28T10:42:29.354858",
     "exception": false,
     "start_time": "2021-11-28T10:42:29.308453",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of frames 56\n",
      "Total amount of time 35.98161 seconds\n",
      "FPS: 1.6\n"
     ]
    }
   ],
   "source": [
    "print('Total number of frames', f)\n",
    "print('Total amount of time {:.5f} seconds'.format(t))\n",
    "print('FPS:', round((f / t), 1))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a0a963ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-28T10:42:29.428695Z",
     "iopub.status.busy": "2021-11-28T10:42:29.428048Z",
     "iopub.status.idle": "2021-11-28T10:42:29.436263Z",
     "shell.execute_reply": "2021-11-28T10:42:29.435621Z",
     "shell.execute_reply.started": "2021-11-28T10:33:05.680499Z"
    },
    "papermill": {
     "duration": 0.04646,
     "end_time": "2021-11-28T10:42:29.436408",
     "exception": false,
     "start_time": "2021-11-28T10:42:29.389948",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='result.mp4' target='_blank'>result.mp4</a><br>"
      ],
      "text/plain": [
       "/kaggle/working/result.mp4"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Saving locally without committing\n",
    "from IPython.display import FileLink\n",
    "\n",
    "FileLink('result.mp4')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33f69117",
   "metadata": {
    "papermill": {
     "duration": 0.035137,
     "end_time": "2021-11-28T10:42:29.507169",
     "exception": false,
     "start_time": "2021-11-28T10:42:29.472032",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Example results from frame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d36c9963",
   "metadata": {
    "papermill": {
     "duration": 0.035277,
     "end_time": "2021-11-28T10:42:29.577923",
     "exception": false,
     "start_time": "2021-11-28T10:42:29.542646",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<a href=\"https://ibb.co/nRTbx17\"><img src=\"https://i.ibb.co/PtRNqWc/Va-r.png\" alt=\"Va-r\" border=\"0\"></a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 74.684748,
   "end_time": "2021-11-28T10:42:33.080289",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-11-28T10:41:18.395541",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}