notebooks/104-model-tools/104-model-tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "artificial-discretion",
   "metadata": {},
   "source": [
    "# Working with Open Model Zoo Models\n",
    "This tutorial shows how to download a model from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo), convert it to OpenVINO™ IR format, show information about the model, and benchmark the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4dda33e",
   "metadata": {},
   "source": [
    "## OpenVINO and Open Model Zoo Tools\n",
    "\n",
    "OpenVINO and Open Model Zoo tools are listed in the table below.\n",
    "\n",
    "| Tool             | Command             | Description                                             |\n",
    "|:-----------------|:--------------------|:--------------------------------------------------------|\n",
    "| Model Downloader | omz_downloader      | Download models from Open Model Zoo.                    |\n",
    "| Model Converter  | omz_converter       | Convert Open Model Zoo models to OpenVINO's IR format.  |\n",
    "| Info Dumper      | omz_info_dumper     | Print information about Open Model Zoo models.          |\n",
    "| Benchmark Tool   | benchmark_app       | Benchmark model performance by computing inference time.|"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fec90f61-08f3-4417-8a6f-ec61c9e5955b",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "## Preparation\n",
    "\n",
    "### Model Name\n",
    "\n",
    "Set `model_name` to the name of the Open Model Zoo model to use in this notebook. Refer to the list of [public](https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/index.md) and [Intel](https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/intel/index.md) pre-trained models for a full list of models that can be used. Set `model_name` to the model you want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21d6d2be-05a8-4c52-9ad9-af892a76db1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_name = \"resnet-50-pytorch\"\n",
    "model_name = \"mobilenet-v2-pytorch\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0206a22-dc33-4666-ab2b-86386b97caca",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "impressed-uncertainty",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "from IPython.display import Markdown, display\n",
    "from openvino.runtime import Core\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import DeviceNotFoundAlert, NotebookAlert"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "parental-assets",
   "metadata": {
    "tags": [
     "hide"
    ]
   },
   "source": [
    "### Settings and Configuration\n",
    "\n",
    "Set the file and directory paths. By default, this notebook downloads models from Open Model Zoo to the `open_model_zoo_models` directory in your `$HOME` directory. On Windows, the $HOME directory is usually `c:\\users\\username`, on Linux `/home/username`. To change the folder, change `base_model_dir` in the cell below.\n",
    "\n",
    "The following settings can be changed:\n",
    "\n",
    "* `base_model_dir`: Models will be downloaded into the `intel` and `public` folders in this directory.\n",
    "* `omz_cache_dir`: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.\n",
    "* `precision`: If specified, only models with this precision will be downloaded and converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "korean-agency",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model_dir = Path(\"model\")\n",
    "omz_cache_dir = Path(\"cache\")\n",
    "precision = \"FP16\"\n",
    "\n",
    "# Check if an iGPU is available on this system to use with Benchmark App.\n",
    "ie = Core()\n",
    "gpu_available = \"GPU\" in ie.available_devices\n",
    "\n",
    "print(\n",
    "    f\"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "judicial-preview",
   "metadata": {},
   "source": [
    "## Download a Model from Open Model Zoo\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7da8061a-ee82-49dd-a1f5-dd71993e6479",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/openvinotoolkit/open_model_zoo.git\n",
    "%cd open_model_zoo/tools/model_tools\n",
    "!pip install .\n",
    "%cd ../../../"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rising-interval",
   "metadata": {},
   "source": [
    "Specify, display and run the Model Downloader command to download the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df2b0446-0c49-41cf-9f3f-dec1232249f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment the next line to show help in omz_downloader which explains the command-line options.\n",
    "\n",
    "# !omz_downloader --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "556d0c12-15cf-492d-a1ed-41dff5090eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "download_command = (\n",
    "    f\"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}\"\n",
    ")\n",
    "display(Markdown(f\"Download command: `{download_command}`\"))\n",
    "display(Markdown(f\"Downloading {model_name}...\"))\n",
    "! $download_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proprietary-checklist",
   "metadata": {},
   "source": [
    "## Convert a Model to OpenVINO IR format\n",
    "\n",
    "Specify, display and run the Model Converter command to convert the model to OpenVINO IR format. Model conversion may take a while. The output of the Model Converter command will be displayed. When the conversion is successful, the last lines of the output will include: `[ SUCCESS ] Generated IR version 11 model.` For downloaded models that are already in OpenVINO IR format, conversion will be skipped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11fe7461-90db-4585-b55f-b3df42b01274",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment the next line to show Help in omz_converter which explains the command-line options.\n",
    "\n",
    "# !omz_converter --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "engaged-academy",
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_command = f\"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}\"\n",
    "display(Markdown(f\"Convert command: `{convert_command}`\"))\n",
    "display(Markdown(f\"Converting {model_name}...\"))\n",
    "\n",
    "! $convert_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa8d655f-215d-4e3c-adcb-e8fd4a2e8ab4",
   "metadata": {},
   "source": [
    "## Get Model Information\n",
    "\n",
    "The Info Dumper prints the following information for Open Model Zoo models:\n",
    "\n",
    "* Model name\n",
    "* Description\n",
    "* Framework that was used to train the model\n",
    "* License URL\n",
    "* Precisions supported by the model\n",
    "* Subdirectory: the location of the downloaded model\n",
    "* Task type\n",
    "\n",
    "This information can be shown by running `omz_info_dumper --name model_name` in a terminal. The information can also be parsed and used in scripts. \n",
    "\n",
    "In the next cell, run Info Dumper and use `json` to load the information in a dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8247daf-d3c5-4420-b4c8-d305ac4ace5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_info_output = %sx omz_info_dumper --name $model_name\n",
    "model_info = json.loads(model_info_output.get_nlstr())\n",
    "\n",
    "if len(model_info) > 1:\n",
    "    NotebookAlert(\n",
    "        f\"There are multiple IR files for the {model_name} model. The first model in the \"\n",
    "        \"omz_info_dumper output will be used for benchmarking. Change \"\n",
    "        \"`selected_model_info` in the cell below to select a different model from the list.\",\n",
    "        \"warning\",\n",
    "    )\n",
    "\n",
    "model_info"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7ea7e868-fd2d-4d11-9c87-7aa1f1301083",
   "metadata": {},
   "source": [
    "Having information of the model in a JSON file enables extraction of the path to the model directory, and building the path to the OpenVINO IR file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de1a319e-bbef-414c-921d-60938b4a01a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_model_info = model_info[0]\n",
    "model_path = (\n",
    "    base_model_dir\n",
    "    / Path(selected_model_info[\"subdirectory\"])\n",
    "    / Path(f\"{precision}/{selected_model_info['name']}.xml\")\n",
    ")\n",
    "print(model_path, \"exists:\", model_path.exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54e01154-f700-479f-9111-147c95595d46",
   "metadata": {},
   "source": [
    "## Run Benchmark Tool\n",
    "\n",
    "By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput values (frames per second). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "282452e8-24c7-49c0-bdb2-10677971c30f",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment the next line to show Help in benchmark_app which explains the command-line options.\n",
    "# !benchmark_app --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9812b0c8-8cd0-4840-bca3-a28171d055b7",
   "metadata": {
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_command = f\"benchmark_app -m {model_path} -t 15\"\n",
    "display(Markdown(f\"Benchmark command: `{benchmark_command}`\"))\n",
    "display(Markdown(f\"Benchmarking {model_name} on CPU with async inference for 15 seconds...\"))\n",
    "\n",
    "! $benchmark_command"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75891996-cf53-4c76-ad3c-5fb468ccd7bb",
   "metadata": {},
   "source": [
    "### Benchmark with Different Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88d64dd7-789d-4536-ab8f-84999c73afaf",
   "metadata": {},
   "source": [
    "The `benchmark_app` tool displays logging information that is not always necessary. A more compact result is achieved when the output is parsed with `json`.\n",
    "\n",
    "The following cells show some examples of `benchmark_app` with different parameters. Below are some useful parameters:\n",
    "\n",
    "- `-d` A device to use for inference. For example: CPU, GPU, MULTI. Default: CPU.\n",
    "- `-t` Time expressed in number of seconds to run inference. Default: 60.\n",
    "- `-api` Use asynchronous (async) or synchronous (sync) inference. Default: async.\n",
    "- `-b` Batch size. Default: 1.\n",
    "\n",
    "\n",
    "Run `! benchmark_app --help` to get an overview of all possible command-line parameters.\n",
    "\n",
    "In the next cell, define the `benchmark_model()` function that calls `benchmark_app`. This makes it easy to try different combinations. In the cell below that, you display available devices on the system.\n",
    "\n",
    "> **Note**: In this notebook, `benchmark_app` runs for 15 seconds to give a quick indication of performance. For more accurate performance, it is recommended to run inference for at least one minute by setting the `t` parameter to 60 or higher, and run `benchmark_app` in a terminal/command prompt after closing other applications. Copy the **benchmark command** and paste it in a command prompt where you have activated the `openvino_env` environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7742390e-df71-45e1-9572-f3cbaa576ec3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def benchmark_model(model_xml, device=\"CPU\", seconds=60, api=\"async\", batch=1):\n",
    "    ie = Core()\n",
    "    model_path = Path(model_xml)\n",
    "    if (\"GPU\" in device) and (\"GPU\" not in ie.available_devices):\n",
    "        DeviceNotFoundAlert(\"GPU\")\n",
    "    else:\n",
    "        benchmark_command = f\"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}\"\n",
    "        display(Markdown(f\"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**\"))\n",
    "        display(Markdown(f\"Benchmark command: `{benchmark_command}`\"))\n",
    "\n",
    "        benchmark_output = %sx $benchmark_command\n",
    "        print(\"command ended\")\n",
    "        benchmark_result = [line for line in benchmark_output\n",
    "                            if not (line.startswith(r\"[\") or line.startswith(\"      \") or line == \"\")]\n",
    "        print(\"\\n\".join(benchmark_result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "298904f0-638c-4958-876a-3b8c8bd06518",
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "\n",
    "# Show devices available for OpenVINO Runtime\n",
    "for device in ie.available_devices:\n",
    "    device_name = ie.get_property(device, \"FULL_DEVICE_NAME\")\n",
    "    print(f\"{device}: {device_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "486919e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "benchmark_model(model_path, device=\"CPU\", seconds=15, api=\"async\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1d4c745-a4c2-4242-a2e7-fff1eb098b98",
   "metadata": {
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_model(model_path, device=\"AUTO\", seconds=15, api=\"async\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d47e1cf-cb2f-4138-928c-b56891d5c318",
   "metadata": {
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_model(model_path, device=\"GPU\", seconds=15, api=\"async\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a1b0bb2-87c7-44c9-92bf-1d95aa1500a1",
   "metadata": {
    "tags": [],
    "test_replace": {
     "seconds=15": "seconds=3"
    }
   },
   "outputs": [],
   "source": [
    "benchmark_model(model_path, device=\"MULTI:CPU,GPU\", seconds=15, api=\"async\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "bc9b317d09174d2c31f34f506ea6004a876573d0732009f97d44018ebb6e02f6"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}