notebooks/cv2PYNQ - Get Started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cv2PYNQ: Get Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This jupyter notebook serves as a quick start guide of the cv2PYNQ library.  \n",
    "It demonstrates its capabilities as well as the limitations and what to pay attention to.  \n",
    "This notebook was created based on [this](https://github.com/Xilinx/PYNQ-ComputerVision/tree/master/notebooks/computer_vision) template."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Include cv2PYNQ\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import cv2pynq as cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The video subsystem with HDMI \n",
    "The library uses the video subsystem from the base PYNQ design.\n",
    "If you want to learn all about its capabilities, use the notebooks \n",
    "https://github.com/Xilinx/PYNQ/tree/master/boards/Pynq-Z1/base/notebooks/video\n",
    "provided by Xilinx as an introduction.  \n",
    "You can access the video subsystem simply with *cv2.video*  \n",
    "It contains the HDMI-in and HDMI-out interfaces.  \n",
    "CAUTION: hdmi_in.start() will take some time and will fail if no incoming video signal is detected.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VideoMode: width=1920 height=1080 bpp=8\n"
     ]
    }
   ],
   "source": [
    "hdmi_in = cv2.video.hdmi_in\n",
    "hdmi_out = cv2.video.hdmi_out\n",
    "\n",
    "hdmi_in.configure(cv2.PIXEL_GRAY)\n",
    "hdmi_out.configure(hdmi_in.mode)\n",
    "\n",
    "hdmi_in.start()\n",
    "hdmi_out.start()\n",
    "\n",
    "print(hdmi_in.mode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the original OpenCV Sobel 5x5 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second using OpenCV:  2.7751992189247194\n"
     ]
    }
   ],
   "source": [
    "import cv2 as openCV\n",
    "import time\n",
    "\n",
    "iterations = 10\n",
    "\n",
    "start = time.time()\n",
    "for i in range(iterations):\n",
    "    inframe = hdmi_in.readframe()\n",
    "    outframe = hdmi_out.newframe()\n",
    "    openCV.Sobel(inframe,-1,1,0,ksize=5,dst=outframe)\n",
    "    inframe.freebuffer()\n",
    "    hdmi_out.writeframe(outframe)\n",
    "end = time.time()\n",
    "print(\"Frames per second using OpenCV:  \" + str(iterations / (end - start)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the cv2PYNQ Sobel 5x5 in the Programmable Logic\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second using cv2PYNQ:  59.75132486181549\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "iterations = 10\n",
    "\n",
    "start = time.time()\n",
    "for i in range(iterations):\n",
    "    inframe = hdmi_in.readframe()\n",
    "    outframe = hdmi_out.newframe()\n",
    "    cv2.Sobel(inframe,-1,1,0,ksize=5,dst=outframe)\n",
    "    inframe.freebuffer()\n",
    "    hdmi_out.writeframe(outframe)\n",
    "end = time.time()\n",
    "print(\"Frames per second using cv2PYNQ:  \" + str(iterations / (end - start)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cv2PYNQ and continous memory\n",
    "\n",
    "The video subsystem returns images as [contiguous memory arrays](https://pynq.readthedocs.io/en/latest/pynq_libraries/xlnk.html).  \n",
    "This allows the cv2PYNQ library to stream the data directly through the hardware.  \n",
    "If the image is a normal numpy ndarray and no destination is given, the library must execute two copy functions.  \n",
    "This results in a perspicuous drop of the framerate but is still faster than the software version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second using cv2PYNQ without CMA:  16.418144610118855\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "image = np.ndarray(shape=(1080,1920),dtype=np.uint8) \n",
    "\n",
    "iterations = 10\n",
    "\n",
    "start = time.time()\n",
    "for i in range(iterations):\n",
    "    sobel = cv2.Sobel(image,-1,1,0,ksize=5)\n",
    "end = time.time()\n",
    "print(\"Frames per second using cv2PYNQ without CMA:  \" + str(iterations / (end - start)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The solution to this problem is allocating contiguous memory arrays and use them as images.  \n",
    "Don't forget to free them after use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frames per second using cv2PYNQ with CMA:  65.67885150201688\n"
     ]
    }
   ],
   "source": [
    "from pynq import Xlnk\n",
    "xlnk = Xlnk()\n",
    "\n",
    "image_buffer  = xlnk.cma_array(shape=(1080,1920), dtype=np.uint8)\n",
    "return_buffer = xlnk.cma_array(shape=(1080,1920), dtype=np.uint8)\n",
    "\n",
    "iterations = 10\n",
    "\n",
    "start = time.time()\n",
    "for i in range(iterations):\n",
    "    cv2.Sobel(image_buffer,-1,1,0,ksize=5,dst=return_buffer)\n",
    "end = time.time()\n",
    "print(\"Frames per second using cv2PYNQ with CMA:  \" + str(iterations / (end - start)))\n",
    "\n",
    "image_buffer.close()\n",
    "return_buffer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up HDMI drivers\n",
    "\n",
    "NOTE: This is needed to reset the HDMI drivers in a clean state. If this is not run, subsequent executions of this notebook may show visual artifacts on the HDMI out (usually a shifted output image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdmi_out.close()\n",
    "hdmi_in.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up cv2PYNQ\n",
    "NOTE: This cleanup is needed because the library allocates contiguous memory and must free it. Otherwise, it may allocate all the available contiguous memory after including it a few times. The only solution is a reboot of the device, therefore do the cleanup ;) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv2.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}