notebooks/301-tensorflow-training-openvino/301-tensorflow-training-openvino-nncf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KwQtSOz0VrVX",
    "tags": []
   },
   "source": [
    "# Post-Training Quantization with TensorFlow Classification Model\n",
    "\n",
    "This example demonstrates how to quantize the OpenVINO model that was created in [301-tensorflow-training-openvino.ipynb](301-tensorflow-training-openvino.ipynb), to improve inference speed. Quantization is performed with [Post-training Quantization with NNCF](https://docs.openvino.ai/latest/nncf_ptq_introduction.html). A custom dataloader and metric will be defined, and accuracy and performance will be computed for the original IR model and the quantized model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KwQtSOz0VrVX"
   },
   "source": [
    "## Preparation\n",
    "\n",
    "The notebook requires that the training notebook has been run and that the Intermediate Representation (IR) models are created. If the IR models do not exist, running the next cell will run the training notebook. This will take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide_output"
    ],
    "test_replace": {
     "301-tensorflow-training-openvino.ipynb": "test_301-tensorflow-training-openvino.ipynb"
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "model_xml_path = Path(\"model/flower/flower_ir.xml\")\n",
    "dataset_url = (\n",
    "    \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
    ")\n",
    "data_dir = Path(tf.keras.utils.get_file(\"flower_photos\", origin=dataset_url, untar=True))\n",
    "\n",
    "if not model_xml_path.exists():\n",
    "    print(\"Executing training notebook. This will take a while...\")\n",
    "    %run 301-tensorflow-training-openvino.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xG4gTnxHXn3z"
   },
   "source": [
    "### Imports\n",
    "\n",
    "The Post Training Quantization API is implemented in the `nncf` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Wi2JvOs1Xn3z",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import nncf\n",
    "from openvino.runtime import Core\n",
    "from openvino.runtime import serialize\n",
    "from PIL import Image\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "mgaRN7mTXn30"
   },
   "source": [
    "## Post-training Quantization with NNCF\n",
    "\n",
    "[NNCF](https://github.com/openvinotoolkit/nncf) provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.\n",
    "> **Note**: NNCF Post-training Quantization is available as a preview feature in OpenVINO 2022.3 release.\n",
    "\n",
    "Fully functional support will be provided in the next releases.\n",
    "\n",
    "Create a quantized model from the pre-trained FP32 model and the calibration dataset. The optimization process contains the following steps:\n",
    "1. Create a Dataset for quantization.\n",
    "2. Run nncf.quantize for getting an optimized model.\n",
    "\n",
    "The validation dataset already defined in the training notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "img_height = 180\n",
    "img_width = 180\n",
    "val_dataset = tf.keras.preprocessing.image_dataset_from_directory(\n",
    "  data_dir,\n",
    "  validation_split=0.2,\n",
    "  subset=\"validation\",\n",
    "  seed=123,\n",
    "  image_size=(img_height, img_width),\n",
    "  batch_size=1\n",
    ")\n",
    "\n",
    "for a, b in val_dataset:\n",
    "    print(type(a), type(b))\n",
    "    break"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The validation dataset can be reused in quantization process. But it returns a tuple (images, labels), whereas calibration_dataset should only return images. The transformation function helps to transform a user validation dataset to the calibration dataset."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "def transform_fn(data_item):\n",
    "    \"\"\"\n",
    "    The transformation function transforms a data item into model input data.\n",
    "    This function should be passed when the data item cannot be used as model's input.\n",
    "    \"\"\"\n",
    "    images, _ = data_item\n",
    "    return images\n",
    "\n",
    "\n",
    "calibration_dataset = nncf.Dataset(val_dataset, transform_fn)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Download Intermediate Representation (IR) model."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "ir_model = ie.read_model(model_xml_path)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Use [Basic Quantization Flow](https://docs.openvino.ai/latest/basic_qauntization_flow.html#doxid-basic-qauntization-flow). To use the most advanced quantization flow that allows to apply 8-bit quantization to the model with accuracy control see [Quantizing with accuracy control](https://docs.openvino.ai/latest/quantization_w_accuracy_control.html#)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "quantized_model = nncf.quantize(\n",
    "    ir_model,\n",
    "    calibration_dataset,\n",
    "    subset_size=1000\n",
    ")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Save quantized model to benchmark."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "compressed_model_dir = Path(\"model/optimized\")\n",
    "compressed_model_dir.mkdir(parents=True, exist_ok=True)\n",
    "compressed_model_xml = compressed_model_dir / \"flower_ir.xml\"\n",
    "serialize(quantized_model, str(compressed_model_xml))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Compare Metrics\n",
    "Define a metric to determine the performance of the model.\n",
    "\n",
    "For this demo we define validate function to compute accuracy metrics."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "def validate(model, validation_loader):\n",
    "    \"\"\"\n",
    "    Evaluate model and compute accuracy metrics.\n",
    "\n",
    "    :param model: Model to validate\n",
    "    :param validation_loader: Validation dataset\n",
    "    :returns: Accuracy scores\n",
    "    \"\"\"\n",
    "    predictions = []\n",
    "    references = []\n",
    "\n",
    "    output = model.outputs[0]\n",
    "\n",
    "    for images, target in validation_loader:\n",
    "        pred = model(images)[output]\n",
    "\n",
    "        predictions.append(np.argmax(pred, axis=1))\n",
    "        references.append(target)\n",
    "\n",
    "    predictions = np.concatenate(predictions, axis=0)\n",
    "    references = np.concatenate(references, axis=0)\n",
    "\n",
    "    scores = accuracy_score(references, predictions)\n",
    "\n",
    "    return scores"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Calculate accuracy for the original model and the quantized model."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "original_compiled_model = ie.compile_model(model=ir_model, device_name=\"CPU\")\n",
    "quantized_compiled_model = ie.compile_model(model=quantized_model, device_name=\"CPU\")\n",
    "\n",
    "original_accuracy = validate(original_compiled_model, val_dataset)\n",
    "quantized_accuracy = validate(quantized_compiled_model, val_dataset)\n",
    "\n",
    "print(f\"Accuracy of the original model: {original_accuracy:.3f}\")\n",
    "print(f\"Accuracy of the quantized model: {quantized_accuracy:.3f}\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Compare file size of the models."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "original_model_size = model_xml_path.with_suffix(\".bin\").stat().st_size / 1024\n",
    "quantized_model_size = compressed_model_xml.with_suffix(\".bin\").stat().st_size / 1024\n",
    "\n",
    "print(f\"Original model size: {original_model_size:.2f} KB\")\n",
    "print(f\"Quantized model size: {quantized_model_size:.2f} KB\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "So, we can see that the original and quantized models have similar accuracy with a much smaller size of the quantized model."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DJbP4pdEXn30"
   },
   "source": [
    "## Run Inference on Quantized Model\n",
    "\n",
    "Copy the preprocess function from the training notebook and run inference on the quantized model with Inference Engine. See the [OpenVINO API tutorial](../002-openvino-api/002-openvino-api.ipynb) for more information about running inference with Inference Engine Python API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process_image(imagePath, img_height=180):\n",
    "    # Model input format\n",
    "    n, c, h, w = [1, 3, img_height, img_height]\n",
    "    image = Image.open(imagePath)\n",
    "    image = image.resize((h, w), resample=Image.BILINEAR)\n",
    "\n",
    "    # Convert to array and change data layout from HWC to CHW\n",
    "    image = np.array(image)\n",
    "\n",
    "    input_image = image.reshape((n, h, w, c))\n",
    "\n",
    "    return input_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q8lGxI-1Xn31"
   },
   "outputs": [],
   "source": [
    "# Get the names of the input and output layer\n",
    "# model_pot = ie.read_model(model=\"model/optimized/flower_ir.xml\")\n",
    "input_layer = quantized_compiled_model.input(0)\n",
    "output_layer = quantized_compiled_model.output(0)\n",
    "\n",
    "# Get the class names: a list of directory names in alphabetical order\n",
    "class_names = sorted([item.name for item in Path(data_dir).iterdir() if item.is_dir()])\n",
    "\n",
    "# Run inference on an input image...\n",
    "inp_img_url = (\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/4/48/A_Close_Up_Photo_of_a_Dandelion.jpg\"\n",
    ")\n",
    "directory = \"output\"\n",
    "inp_file_name = \"A_Close_Up_Photo_of_a_Dandelion.jpg\"\n",
    "file_path = Path(directory)/Path(inp_file_name)\n",
    "# Download the image if it does not exist yet\n",
    "if not Path(inp_file_name).exists():\n",
    "    download_file(inp_img_url, inp_file_name, directory=directory)\n",
    "\n",
    "# Pre-process the image and get it ready for inference.\n",
    "input_image = pre_process_image(imagePath=file_path)\n",
    "print(f'input image shape: {input_image.shape}')\n",
    "print(f'input layer shape: {input_layer.shape}')\n",
    "\n",
    "res = quantized_compiled_model([input_image])[output_layer]\n",
    "\n",
    "score = tf.nn.softmax(res[0])\n",
    "\n",
    "# Show the results\n",
    "image = Image.open(file_path)\n",
    "plt.imshow(image)\n",
    "print(\n",
    "    \"This image most likely belongs to {} with a {:.2f} percent confidence.\".format(\n",
    "        class_names[np.argmax(score)], 100 * np.max(score)\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Inference Speed\n",
    "\n",
    "Measure inference speed with the [OpenVINO Benchmark App](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html).\n",
    "\n",
    "Benchmark App is a command line tool that measures raw inference performance for a specified OpenVINO IR model. Run `benchmark_app --help` to see a list of available parameters. By default, Benchmark App tests the performance of the model specified with the `-m` parameter with asynchronous inference on CPU, for one minute. Use the `-d` parameter to test performance on a different device, for example an Intel integrated Graphics (iGPU), and `-t` to set the number of seconds to run inference. See the [documentation](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) for more information.\n",
    "\n",
    "This tutorial uses a wrapper function from [Notebook Utils](https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/utils/notebook_utils.ipynb). It prints the `benchmark_app` command with the chosen parameters.\n",
    "\n",
    "In the next cells, inference speed will be measured for the original and quantized model on CPU. If an iGPU is available, inference speed will be measured for CPU+GPU as well. The number of seconds is set to 15.\n",
    "\n",
    "> **NOTE**: For the most accurate performance estimation, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the available devices on this system\n",
    "print(\"Device information:\")\n",
    "print(ie.get_property(\"CPU\", \"FULL_DEVICE_NAME\"))\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    print(ie.get_property(\"GPU\", \"FULL_DEVICE_NAME\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "# Original model - CPU\n",
    "! benchmark_app -m $model_xml -d CPU -t 15 -api async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "# Quantized model - CPU\n",
    "! benchmark_app -m $compressed_model_xml -d CPU -t 15 -api async"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Benchmark on MULTI:CPU,GPU**\n",
    "\n",
    "With a recent Intel CPU, the best performance can often be achieved by doing inference on both the CPU and the iGPU, with OpenVINO's [Multi Device Plugin](https://docs.openvino.ai/2021.4/openvino_docs_IE_DG_supported_plugins_MULTI.html). It takes a bit longer to load a model on GPU than on CPU, so this benchmark will take a bit longer to complete than the CPU benchmark, when run for the first time. Benchmark App supports caching, by specifying the `--cdir` parameter. In the cells below, the model will cached to the `model_cache` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "# Original model - MULTI:CPU,GPU\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    ! benchmark_app -m $model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "else:\n",
    "    print(\"A supported integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "# Quantized model - MULTI:CPU,GPU\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    ! benchmark_app -m $compressed_model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "else:\n",
    "    print(\"A supported integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the available devices on this system\n",
    "print(\"Device information:\")\n",
    "print(ie.get_property(\"CPU\", \"FULL_DEVICE_NAME\"))\n",
    "if \"GPU\" in ie.available_devices:\n",
    "    print(ie.get_property(\"GPU\", \"FULL_DEVICE_NAME\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Original IR model - CPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_output = %sx benchmark_app -m $model_xml -t 15 -api async\n",
    "# Remove logging info from benchmark_app output and show only the results\n",
    "benchmark_result = benchmark_output[-8:]\n",
    "print(\"\\n\".join(benchmark_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Quantized IR model - CPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_output = %sx benchmark_app -m $compressed_model_xml -t 15 -api async\n",
    "# Remove logging info from benchmark_app output and show only the results\n",
    "benchmark_result = benchmark_output[-8:]\n",
    "print(\"\\n\".join(benchmark_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Original IR model - MULTI:CPU,GPU**\n",
    "\n",
    "With a recent Intel CPU, the best performance can often be achieved by doing inference on both the CPU and the iGPU, with OpenVINO's [Multi Device Plugin](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Running_on_multiple_devices.html). It takes a bit longer to load a model on GPU than on CPU, so this benchmark will take a bit longer to complete than the CPU benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_output = %sx benchmark_app -m $model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "    # Remove logging info from benchmark_app output and show only the results\n",
    "    benchmark_result = benchmark_output[-8:]\n",
    "    print(\"\\n\".join(benchmark_result))\n",
    "else:\n",
    "    print(\"An integrated GPU is not available on this system.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-0eZaw1W5-wt"
   },
   "source": [
    "**Quantized IR model - MULTI:CPU,GPU**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-0eZaw1W5-wt",
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "if \"GPU\" in ie.available_devices:\n",
    "    benchmark_output = %sx benchmark_app -m $compressed_model_xml -d MULTI:CPU,GPU -t 15 -api async\n",
    "    # Remove logging info from benchmark_app output and show only the results\n",
    "    benchmark_result = benchmark_output[-8:]\n",
    "    print(\"\\n\".join(benchmark_result))\n",
    "else:\n",
    "    print(\"An integrated GPU is not available on this system.\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "301-tensorflow-training-openvino-pot.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "interpreter": {
   "hash": "8e25c8ed6cc2cfe6c8620be5042bb64fac4c236f57496fb5eb68e9ea1795f1fe"
  },
  "kernelspec": {
   "display_name": "openvino_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}