notebooks/002-openvino-api/002-openvino-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b875b31c",
   "metadata": {},
   "source": [
    "# OpenVINO™ Runtime API Tutorial\n",
    "\n",
    "> **Note:** This notebook has been moved to a new branch named \"latest\". [Click here](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/openvino-api/openvino-api.ipynb) to get the most updated version of the notebook. This branch is deprecated.\n",
    "\n",
    "This notebook explains the basics of the OpenVINO Runtime API.\n",
    "\n",
    "The notebook is divided into sections with headers. The next cell contains global requirements for installation and imports. Each section is standalone and does not depend on any previous sections. All models used in this tutorial are provided as examples. These model files can be replaced with your own models. The exact outputs will be different, but the process is the same. \n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Loading OpenVINO Runtime and Showing Info](#Loading-OpenVINO-Runtime-and-Showing-Info)\n",
    "- [Loading a Model](#Loading-a-Model)\n",
    "    - [OpenVINO IR Model](#OpenVINO-IR-Model)\n",
    "    - [ONNX Model](#ONNX-Model)\n",
    "    - [PaddlePaddle Model](#PaddlePaddle-Model)\n",
    "    - [TensorFlow Model](#TensorFlow-Model)\n",
    "    - [TensorFlow Lite Model](#TensorFlow-Lite-Model)\n",
    "    - [PyTorch Model](#PyTorch-Model)\n",
    "- [Getting Information about a Model](#Getting-Information-about-a-Model)\n",
    "    - [Model Inputs](#Model-Inputs)\n",
    "    - [Model Outputs](#Model-Outputs)\n",
    "- [Doing Inference on a Model](#Doing-Inference-on-a-Model)\n",
    "- [Reshaping and Resizing](#Reshaping-and-Resizing)\n",
    "    - [Change Image Size](#Change-Image-Size)\n",
    "    - [Change Batch Size](#Change-Batch-Size)\n",
    "- [Caching a Model](#Caching-a-Model)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cdfd726a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Required imports. Please execute this cell first.\n",
    "%pip install -q \"openvino>=2023.1.0\" \n",
    "%pip install -q requests tqdm ipywidgets\n",
    "\n",
    "# Fetch `notebook_utils` module\n",
    "import urllib.request\n",
    "urllib.request.urlretrieve(\n",
    "    url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py',\n",
    "    filename='notebook_utils.py'\n",
    ")\n",
    "\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ed058f4",
   "metadata": {},
   "source": [
    "## Loading OpenVINO Runtime and Showing Info\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Initialize OpenVINO Runtime with `ov.Core()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c08b79c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68bc4125",
   "metadata": {},
   "source": [
    "OpenVINO Runtime can load a network on a device. A device in this context means a CPU, an Intel GPU, a Neural Compute Stick 2, etc. The `available_devices` property shows the available devices in your system. The \"FULL_DEVICE_NAME\" option to `core.get_property()` shows the name of the device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d0c94f76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz\n"
     ]
    }
   ],
   "source": [
    "devices = core.available_devices\n",
    "\n",
    "for device in devices:\n",
    "    device_name = core.get_property(device, \"FULL_DEVICE_NAME\")\n",
    "    print(f\"{device}: {device_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58ed5c2d-b8d9-43b7-9fac-716b083c1398",
   "metadata": {},
   "source": [
    "### Select device for inference\n",
    "\n",
    "You can specify which device from available devices will be used for inference using this widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e8a64531-ea36-4766-ab6e-6fa4c1bd4732",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "daf3fdaef8ff41d18fbb309de5091b00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', options=('CPU',), value='CPU')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices,\n",
    "    value=core.available_devices[0],\n",
    "    description='Device:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14d62615",
   "metadata": {},
   "source": [
    "## Loading a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "After initializing OpenVINO Runtime, first read the model file with `read_model()`, then compile it to the specified device with the `compile_model()` method. \n",
    "\n",
    "[OpenVINO™ supports several model formats](https://docs.openvino.ai/2024/openvino-workflow/model-preparation/convert-model-to-ir.html) and enables developers to convert them to its own OpenVINO IR format using a tool dedicated to this task.\n",
    "\n",
    "### OpenVINO IR Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "An OpenVINO IR (Intermediate Representation) model consists of an `.xml` file, containing information about network topology, and a `.bin` file, containing the weights and biases binary data. Models in OpenVINO IR format are obtained by using model conversion API. The `read_model()` function expects the `.bin` weights file to have the same filename and be located in the same directory as the `.xml` file: `model_weights_file == Path(model_xml).with_suffix(\".bin\")`. If this is the case, specifying the weights file is optional. If the weights file has a different filename, it can be specified using the `weights` parameter in `read_model()`.\n",
    "\n",
    "The OpenVINO [Model Conversion API](https://docs.openvino.ai/2024/openvino-workflow/model-preparation.html) tool is used to convert models to OpenVINO IR format. Model conversion API reads the original model and creates an OpenVINO IR model (`.xml` and `.bin` files) so inference can be performed without delays due to format conversion. Optionally, model conversion API can adjust the model to be more suitable for inference, for example, by alternating input shapes, embedding preprocessing and cutting training parts off.\n",
    "For information on how to convert your existing TensorFlow, PyTorch or ONNX model to OpenVINO IR format with model conversion API, refer to the [tensorflow-to-openvino](../101-tensorflow-classification-to-openvino/101-tensorflow-classification-to-openvino.ipynb) and [pytorch-onnx-to-openvino](../102-pytorch-onnx-to-openvino/102-pytorch-onnx-to-openvino.ipynb) notebooks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6ff44409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.xml' already exists.\n",
      "'model/classification.bin' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.bin')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "ir_model_name_xml = 'classification.xml'\n",
    "ir_model_name_bin = 'classification.bin'\n",
    "\n",
    "download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory='model')\n",
    "download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "523978fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "classification_model_xml = \"model/classification.xml\"\n",
    "\n",
    "model = core.read_model(model=classification_model_xml)\n",
    "compiled_model = core.compile_model(model=model, device_name=device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5516e87",
   "metadata": {},
   "source": [
    "### ONNX Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[ONNX](https://onnx.ai/) is an open format built to represent machine learning models. ONNX defines a common set of operators - the building blocks of machine learning and deep learning models - and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers. OpenVINO supports reading models in ONNX format directly,that means they can be used with OpenVINO Runtime without any prior conversion.\n",
    "\n",
    "Reading and loading an ONNX model, which is a single `.onnx` file, works the same way as with an OpenVINO IR model. The `model` argument points to the filename of an ONNX model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "09640278",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/segmentation.onnx' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/segmentation.onnx')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "onnx_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/segmentation.onnx'\n",
    "onnx_model_name = 'segmentation.onnx'\n",
    "\n",
    "download_file(onnx_model_url, filename=onnx_model_name, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "15833f51",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "onnx_model_path = \"model/segmentation.onnx\"\n",
    "\n",
    "model_onnx = core.read_model(model=onnx_model_path)\n",
    "compiled_model_onnx = core.compile_model(model=model_onnx, device_name=device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d6a41d9-a2de-40a8-b925-fe3e817866d4",
   "metadata": {},
   "source": [
    "The ONNX model can be exported to OpenVINO IR with `save_model()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1610b254-8567-4409-9b26-259b8acd10dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov.save_model(model_onnx, output_model=\"model/exported_onnx_model.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9ba7d2-ef7a-46ed-b8a2-5c47f7b97fec",
   "metadata": {},
   "source": [
    "### PaddlePaddle Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[PaddlePaddle](https://www.paddlepaddle.org.cn/documentation/docs/en/guides/index_en.html) models saved for inference can also be passed to OpenVINO Runtime without any conversion step. Pass the filename with extension to `read_model` and exported an OpenVINO IR with `save_model`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b0daa43e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/inference.pdmodel' already exists.\n",
      "'model/inference.pdiparams' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/inference.pdiparams')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paddle_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "paddle_model_name = 'inference.pdmodel'\n",
    "paddle_params_name = 'inference.pdiparams'\n",
    "\n",
    "download_file(paddle_model_url + paddle_model_name, filename=paddle_model_name, directory='model')\n",
    "download_file(paddle_model_url + paddle_params_name, filename=paddle_params_name, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d6524ef0-9a76-4254-8614-3a1e24cc9420",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "paddle_model_path = 'model/inference.pdmodel'\n",
    "\n",
    "model_paddle = core.read_model(model=paddle_model_path)\n",
    "compiled_model_paddle = core.compile_model(model=model_paddle, device_name=device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "74f8948d-0eed-4b3c-98ae-35870ff82625",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov.save_model(model_paddle, output_model=\"model/exported_paddle_model.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d7fcaff-8ed6-4c96-a46d-2ef813f24eb1",
   "metadata": {},
   "source": [
    "### TensorFlow Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "TensorFlow models saved in frozen graph format can also be passed to `read_model`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eb33ec27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.pb' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.pb')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pb_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/classification.pb'\n",
    "pb_model_name = 'classification.pb'\n",
    "\n",
    "download_file(pb_model_url, filename=pb_model_name, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "85446bed-8787-4fab-b64b-03035ad74bfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "tf_model_path = \"model/classification.pb\"\n",
    "\n",
    "model_tf = core.read_model(model=tf_model_path)\n",
    "compiled_model_tf = core.compile_model(model=model_tf, device_name=device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "48f3cd1c-9343-4fdd-8e8d-9226a4956517",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov.save_model(model_tf, output_model=\"model/exported_tf_model.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7ba1b1",
   "metadata": {},
   "source": [
    "### TensorFlow Lite Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[TFLite](https://www.tensorflow.org/lite) models saved for inference can also be passed to OpenVINO Runtime. Pass the filename with extension `.tflite` to `read_model` and exported an OpenVINO IR with `save_model`.\n",
    "\n",
    "This tutorial uses the image classification model [inception_v4_quant](https://tfhub.dev/tensorflow/lite-model/inception_v4_quant/1/default/1). It is pre-trained model optimized to work with TensorFlow Lite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "16ef64c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.tflite' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.tflite')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "tflite_model_url = 'https://www.kaggle.com/models/tensorflow/inception/frameworks/tfLite/variations/v4-quant/versions/1?lite-format=tflite'\n",
    "tflite_model_path = Path('model/classification.tflite')\n",
    "\n",
    "download_file(tflite_model_url, filename=tflite_model_path.name, directory=tflite_model_path.parent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ed67c6c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "model_tflite = core.read_model(tflite_model_path)\n",
    "compiled_model_tflite = core.compile_model(model=model_tflite, device_name=device.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4f73b1ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov.save_model(model_tflite, output_model=\"model/exported_tflite_model.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f77c985-f9c1-4500-bd3e-714890a191b5",
   "metadata": {},
   "source": [
    "### PyTorch Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "[PyTorch](https://pytorch.org/) models can not be directly passed to `core.read_model`. `ov.Model` for model objects from this framework can be obtained using `ov.convert_model` API. You can find more details in [pytorch-to-openvino](../102-pytorch-to-openvino) notebook. In this tutorial we will use [resnet18](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html) model form torchvision library. After conversion model using `ov.convert_model`, it can be compiled on device using `core.compile_model` or saved on disk for the next usage using `ov.save_model`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ffdfb706-256e-47e8-a55c-090d303b6815",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "import torch\n",
    "from torchvision.models import resnet18, ResNet18_Weights\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "pt_model = resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)\n",
    "example_input = torch.zeros((1, 3, 224, 224))\n",
    "ov_model_pytorch = ov.convert_model(pt_model, example_input=example_input)\n",
    "\n",
    "compiled_model_pytorch = core.compile_model(ov_model_pytorch, device_name=device.value)\n",
    "\n",
    "ov.save_model(ov_model_pytorch, \"model/exported_pytorch_model.xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ebee450",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Getting Information about a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The OpenVINO Model instance stores information about the model. Information about the inputs and outputs of the model are in `model.inputs` and `model.outputs`. These are also properties of the `CompiledModel` instance. While using `model.inputs` and `model.outputs` in the cells below, you can also use `compiled_model.inputs` and `compiled_model.outputs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e8a70ab1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.xml' already exists.\n",
      "'model/classification.bin' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.bin')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "ir_model_name_xml = 'classification.xml'\n",
    "ir_model_name_bin = 'classification.bin'\n",
    "\n",
    "download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory='model')\n",
    "download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory='model')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edc79b32",
   "metadata": {},
   "source": [
    "### Model Inputs\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Information about all input layers is stored in the `inputs` dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5571614d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Output: names[input, input:0] shape[1,3,224,224] type: f32>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "classification_model_xml = \"model/classification.xml\"\n",
    "model = core.read_model(model=classification_model_xml)\n",
    "model.inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebb1299",
   "metadata": {},
   "source": [
    "The cell above shows that the loaded model expects one input with the name _input_. If you loaded a different model, you may see a different input layer name, and you may see more inputs. You may also obtain info about each input layer using `model.input(index)`, where index is a numeric index of the input layers in the model. If a model has only one input, index can be omitted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6cf48354",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = model.input(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f754e26-2223-4b97-9181-46434aa8fbe7",
   "metadata": {},
   "source": [
    "It is often useful to have a reference to the name of the first input layer. For a model with one input, `model.input(0).any_name` gets this name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2202aec4-d68f-4cfa-8b8e-3340ec085a8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'input'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_layer.any_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea90446",
   "metadata": {},
   "source": [
    "The next cell prints the input layout, precision and shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bd0bc0c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input precision: <Type: 'float32'>\n",
      "input shape: [1,3,224,224]\n"
     ]
    }
   ],
   "source": [
    "print(f\"input precision: {input_layer.element_type}\")\n",
    "print(f\"input shape: {input_layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d189a73c",
   "metadata": {},
   "source": [
    "This cell shows that the model expects inputs with a shape of [1,3,224,224], and that this is in the `NCHW` layout. This means that the model expects input data with the batch size of 1 (`N`), 3 channels (`C`) , and images with a height (`H`) and width (`W`) equal to 224. The input data is expected to be of `FP32` (floating point) precision."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "155cf48e",
   "metadata": {},
   "source": [
    "### Model Outputs\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4583eb0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Output: names[MobilenetV3/Predictions/Softmax] shape[1,1001] type: f32>]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "classification_model_xml = \"model/classification.xml\"\n",
    "model = core.read_model(model=classification_model_xml)\n",
    "model.outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a189c02f",
   "metadata": {},
   "source": [
    "Model output info is stored in `model.outputs`. The cell above shows that the model returns one output, with the `MobilenetV3/Predictions/Softmax` name. Loading a different model will result in different output layer name, and more outputs might be returned. Similar to input, you may also obtain information about each output separately using `model.output(index)`\n",
    "\n",
    "Since this model has one output, follow the same method as for the input layer to get its name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "88fbbd06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'MobilenetV3/Predictions/Softmax'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_layer = model.output(0)\n",
    "output_layer.any_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16ad0240",
   "metadata": {},
   "source": [
    "Getting the output precision and shape is similar to getting the input precision and shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0ee5e14a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output precision: <Type: 'float32'>\n",
      "output shape: [1,1001]\n"
     ]
    }
   ],
   "source": [
    "print(f\"output precision: {output_layer.element_type}\")\n",
    "print(f\"output shape: {output_layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2739f5bb",
   "metadata": {},
   "source": [
    "This cell shows that the model returns outputs with a shape of [1, 1001], where 1 is the batch size (`N`) and 1001 is the number of classes (`C`). The output is returned as 32-bit floating point."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "021708ab",
   "metadata": {},
   "source": [
    "## Doing Inference on a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "> **NOTE** this notebook demonstrates only the basic synchronous inference API. For an async inference example, please refer to [Async API notebook](../115-async-api/115-async-api.ipynb)\n",
    "\n",
    "The diagram below shows a typical inference pipeline with OpenVINO\n",
    "\n",
    "![image.png](https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/a91bc582-165b-41a2-ab08-12c812059936)\n",
    "\n",
    "Creating OpenVINO Core and model compilation is covered in the previous steps. The next step is preparing inputs. You can provide inputs in one of the supported format: dictionary with name of inputs as keys and `np.arrays` that represent input tensors as values, list or tuple of `np.arrays` represented input tensors (their order should match with model inputs order). If a model has a single input, wrapping to a dictionary or list can be omitted.   To do inference on a model, pass prepared inputs into compiled model object obtained using `core.compile_model`. The inference result represented as dictionary, where keys are model outputs and `np.arrays` represented their produced data as values.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d64830f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Install opencv package for image handling\n",
    "%pip install -q opencv-python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec2eac8",
   "metadata": {},
   "source": [
    "**Load the network**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6263a7d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.xml' already exists.\n",
      "'model/classification.bin' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.bin')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "ir_model_name_xml = 'classification.xml'\n",
    "ir_model_name_bin = 'classification.bin'\n",
    "\n",
    "download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory='model')\n",
    "download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "298c80b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "classification_model_xml = \"model/classification.xml\"\n",
    "model = core.read_model(model=classification_model_xml)\n",
    "compiled_model = core.compile_model(model=model, device_name=device.value)\n",
    "input_layer = compiled_model.input(0)\n",
    "output_layer = compiled_model.output(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173cd1c9",
   "metadata": {},
   "source": [
    "**Load an image and convert to the input shape**\n",
    "\n",
    "To propagate an image through the network, it needs to be loaded into an array, resized to the shape that the network expects, and converted to the input layout of the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1f23c43a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data/coco_hollywood.jpg' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(663, 994, 3)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cv2\n",
    "\n",
    "image_filename = download_file(\n",
    "    \"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/coco_hollywood.jpg\",\n",
    "    directory=\"data\"\n",
    ")\n",
    "image = cv2.imread(str(image_filename))\n",
    "image.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf541d8",
   "metadata": {},
   "source": [
    "The image has a shape of (663,994,3). It is 663 pixels in height, 994 pixels in width, and has 3 color channels. A reference to the height and width expected by the network is obtained and the image is resized to these dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c9f97da2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(224, 224, 3)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# N,C,H,W = batch size, number of channels, height, width.\n",
    "N, C, H, W = input_layer.shape\n",
    "# OpenCV resize expects the destination size as (width, height).\n",
    "resized_image = cv2.resize(src=image, dsize=(W, H))\n",
    "resized_image.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "436a6d74",
   "metadata": {},
   "source": [
    "Now, the image has the width and height that the network expects. This is still in `HWC` format and must be changed to `NCHW` format. First, call the `np.transpose()` method to change to `CHW` and then add the `N` dimension (where `N`= 1) by calling the `np.expand_dims()` method. Next, convert the data to `FP32` with `np.astype()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d09b7275",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 3, 224, 224)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "input_data = np.expand_dims(np.transpose(resized_image, (2, 0, 1)), 0).astype(np.float32)\n",
    "input_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af110efb",
   "metadata": {},
   "source": [
    "**Do inference**\n",
    "\n",
    "Now that the input data is in the right shape, run inference. The `CompiledModel` inference result is a dictionary where keys are the Output class instances (the same keys in `compiled_model.outputs` that can also be obtained with `compiled_model.output(index)`) and values - predicted result in `np.array` format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "098c8cb2",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# for single input models only\n",
    "result = compiled_model(input_data)[output_layer]\n",
    "\n",
    "# for multiple inputs in a list\n",
    "result = compiled_model([input_data])[output_layer]\n",
    "\n",
    "# or using a dictionary, where the key is input tensor name or index\n",
    "result = compiled_model({input_layer.any_name: input_data})[output_layer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "978a131e",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "You can also create `InferRequest` and run `infer` method on request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bf94022c",
   "metadata": {},
   "outputs": [],
   "source": [
    "request = compiled_model.create_infer_request()\n",
    "request.infer(inputs={input_layer.any_name: input_data})\n",
    "result = request.get_output_tensor(output_layer.index).data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53bf7c61",
   "metadata": {},
   "source": [
    "The `.infer()` function sets output tensor, that can be reached, using `get_output_tensor()`. Since this network returns one output, and the reference to the output layer is in the `output_layer.index` parameter, you can get the data with `request.get_output_tensor(output_layer.index)`. To get a numpy array from the output, use the `.data` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "4a0f63b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 1001)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5834ee",
   "metadata": {},
   "source": [
    "The output shape is (1,1001), which is the expected output shape. This shape indicates that the network returns probabilities for 1001 classes. To learn more about this notion, refer to the [hello world notebook](../001-hello-world/001-hello-world.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec6a9be1",
   "metadata": {},
   "source": [
    "## Reshaping and Resizing\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "### Change Image Size\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4239b10c",
   "metadata": {},
   "source": [
    "Instead of reshaping the image to fit the model, it is also possible to reshape the model to fit the image. Be aware that not all models support reshaping, and models that do, may not support all input shapes. The model accuracy may also suffer if you reshape the model input shape.\n",
    "\n",
    "First check the input shape of the model, then reshape it to the new input shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e16cd8c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/segmentation.xml' already exists.\n",
      "'model/segmentation.bin' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/segmentation.bin')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "ir_model_name_xml = 'segmentation.xml'\n",
    "ir_model_name_bin = 'segmentation.bin'\n",
    "\n",
    "download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory='model')\n",
    "download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "bc1a69f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~ ORIGINAL MODEL ~~~~\n",
      "input shape: [1,3,512,512]\n",
      "output shape: [1,1,512,512]\n",
      "~~~~ RESHAPED MODEL ~~~~\n",
      "model input shape: [1,3,544,544]\n",
      "compiled_model input shape: [1,3,544,544]\n",
      "compiled_model output shape: [1,1,544,544]\n"
     ]
    }
   ],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "segmentation_model_xml = \"model/segmentation.xml\"\n",
    "segmentation_model = core.read_model(model=segmentation_model_xml)\n",
    "segmentation_input_layer = segmentation_model.input(0)\n",
    "segmentation_output_layer = segmentation_model.output(0)\n",
    "\n",
    "print(\"~~~~ ORIGINAL MODEL ~~~~\")\n",
    "print(f\"input shape: {segmentation_input_layer.shape}\")\n",
    "print(f\"output shape: {segmentation_output_layer.shape}\")\n",
    "\n",
    "new_shape = ov.PartialShape([1, 3, 544, 544])\n",
    "segmentation_model.reshape({segmentation_input_layer.any_name: new_shape})\n",
    "segmentation_compiled_model = core.compile_model(model=segmentation_model, device_name=device.value)\n",
    "# help(segmentation_compiled_model)\n",
    "print(\"~~~~ RESHAPED MODEL ~~~~\")\n",
    "print(f\"model input shape: {segmentation_input_layer.shape}\")\n",
    "print(\n",
    "    f\"compiled_model input shape: \"\n",
    "    f\"{segmentation_compiled_model.input(index=0).shape}\"\n",
    ")\n",
    "print(f\"compiled_model output shape: {segmentation_output_layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2104cef6",
   "metadata": {},
   "source": [
    "The input shape for the segmentation network is [1,3,512,512], with the `NCHW` layout: the network expects 3-channel images with a width and height of 512 and a batch size of 1. Reshape the network with the `.reshape()` method of `IENetwork` to make it accept input images with a width and height of 544. This segmentation network always returns arrays with the input width and height of equal value. Therefore, setting the input dimensions to 544x544 also modifies the output dimensions. After reshaping, compile the network once again."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "249d697a",
   "metadata": {},
   "source": [
    "### Change Batch Size\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ded79c8f",
   "metadata": {},
   "source": [
    "Use the `.reshape()` method to set the batch size, by increasing the first element of `new_shape`. For example, to set a batch size of two, set `new_shape = (2,3,544,544)` in the cell above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a49d65c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input shape: [2,3,544,544]\n",
      "output shape: [2,1,544,544]\n"
     ]
    }
   ],
   "source": [
    "import openvino as ov\n",
    "\n",
    "segmentation_model_xml = \"model/segmentation.xml\"\n",
    "segmentation_model = core.read_model(model=segmentation_model_xml)\n",
    "segmentation_input_layer = segmentation_model.input(0)\n",
    "segmentation_output_layer = segmentation_model.output(0)\n",
    "new_shape = ov.PartialShape([2, 3, 544, 544])\n",
    "segmentation_model.reshape({segmentation_input_layer.any_name: new_shape})\n",
    "segmentation_compiled_model = core.compile_model(model=segmentation_model, device_name=device.value)\n",
    "\n",
    "print(f\"input shape: {segmentation_input_layer.shape}\")\n",
    "print(f\"output shape: {segmentation_output_layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40c2f3d2",
   "metadata": {},
   "source": [
    "The output shows that by setting the batch size to 2, the first element (`N`) of the input and output shape has a value of 2. Propagate the input image through the network to see the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0eb487fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input data shape: (2, 3, 544, 544)\n",
      "result data data shape: [2,1,544,544]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "segmentation_model_xml = \"model/segmentation.xml\"\n",
    "segmentation_model = core.read_model(model=segmentation_model_xml)\n",
    "segmentation_input_layer = segmentation_model.input(0)\n",
    "segmentation_output_layer = segmentation_model.output(0)\n",
    "new_shape = ov.PartialShape([2, 3, 544, 544])\n",
    "segmentation_model.reshape({segmentation_input_layer.any_name: new_shape})\n",
    "segmentation_compiled_model = core.compile_model(model=segmentation_model, device_name=device.value)\n",
    "input_data = np.random.rand(2, 3, 544, 544)\n",
    "\n",
    "output = segmentation_compiled_model([input_data])\n",
    "\n",
    "print(f\"input data shape: {input_data.shape}\")\n",
    "print(f\"result data data shape: {segmentation_output_layer.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9657dc5-9713-4d2b-a324-c8cd6195e79a",
   "metadata": {},
   "source": [
    "## Caching a Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "For some devices, like GPU, loading a model can take some time. Model Caching solves this issue by caching the model in a cache directory. If `core.compile_model(model=net, device_name=device_name, config=config_dict)` is set, caching will be used. This option checks if a model exists in the cache. If so, it loads it from the cache. If not, it loads the model regularly, and stores it in the cache, so that the next time the model is loaded when this option is set, the model will be loaded from the cache.\n",
    "\n",
    "In the cell below, we create a *model_cache* directory as a subdirectory of *model*, where the model will be cached for the specified device. The model will be loaded to the GPU. After running this cell once, the model will be cached, so subsequent runs of this cell will load the model from the cache.\n",
    "\n",
    "*Note: Model Caching is also available on CPU devices*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "342c456a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'model/classification.xml' already exists.\n",
      "'model/classification.bin' already exists.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ea/work/openvino_notebooks/notebooks/002-openvino-api/model/classification.bin')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ir_model_url = 'https://storage.openvinotoolkit.org/repositories/openvino_notebooks/models/002-example-models/'\n",
    "ir_model_name_xml = 'classification.xml'\n",
    "ir_model_name_bin = 'classification.bin'\n",
    "\n",
    "download_file(ir_model_url + ir_model_name_xml, filename=ir_model_name_xml, directory='model')\n",
    "download_file(ir_model_url + ir_model_name_bin, filename=ir_model_name_bin, directory='model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1d235185-18f7-4cf0-8cb2-1ecba279318a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading the network to the CPU device took 0.14 seconds.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "from pathlib import Path\n",
    "\n",
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "cache_path = Path(\"model/model_cache\")\n",
    "cache_path.mkdir(exist_ok=True)\n",
    "# Enable caching for OpenVINO Runtime. To disable caching set enable_caching = False\n",
    "enable_caching = True\n",
    "config_dict = {\"CACHE_DIR\": str(cache_path)} if enable_caching else {}\n",
    "\n",
    "classification_model_xml = \"model/classification.xml\"\n",
    "model = core.read_model(model=classification_model_xml)\n",
    "\n",
    "start_time = time.perf_counter()\n",
    "compiled_model = core.compile_model(model=model, device_name=device.value, config=config_dict)\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Loading the network to the {device.value} device took {end_time-start_time:.2f} seconds.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48e0a860-c93c-4b93-a684-f53cd66ec2e3",
   "metadata": {},
   "source": [
    "After running the previous cell, we know the model exists in the cache directory. Then, we delete the compiled model and load it again. Now, we measure the time it takes now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b5a7686b-b9e0-44a6-8b6e-5b299d085eac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading the network to the CPU device took 0.06 seconds.\n"
     ]
    }
   ],
   "source": [
    "del compiled_model\n",
    "start_time = time.perf_counter()\n",
    "compiled_model = core.compile_model(model=model, device_name=device.value, config=config_dict)\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Loading the network to the {device.value} device took {end_time-start_time:.2f} seconds.\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ae617ccb002f72b3ab6d0069d721eac67ac2a969e83c083c4321cfcab0437cd1"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "openvino_notebooks": {
   "imageUrl": "",
   "tags": {
    "categories": [
     "First Steps"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Image Classification",
     "Image Segmentation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "7a51ab3fb0d84150a6ab2259c02898ab": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "9fb0508a99f546e1b2d4d2eab1ad5dd5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "daf3fdaef8ff41d18fbb309de5091b00": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DropdownModel",
      "state": {
       "_options_labels": [
        "CPU"
       ],
       "description": "Device:",
       "index": 0,
       "layout": "IPY_MODEL_9fb0508a99f546e1b2d4d2eab1ad5dd5",
       "style": "IPY_MODEL_7a51ab3fb0d84150a6ab2259c02898ab"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}