53 Web Scraping with Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:44.826995Z",
     "start_time": "2019-02-18T15:07:44.817022Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From Tree to HTML\n",
    "\n",
    "Here you are given the chance to create your own bit of HTML code (as a python string). More specifically, below is an HTML tree image and you will finish the missing code within the string html which produces this HTML tree.\n",
    "\n",
    "html_tree_exercise.png\n",
    "\n",
    "To note:\n",
    "\n",
    "    We have started the string html for you, to help nudge you in the correct direction.\n",
    "    The spacing we use in the portion of the string we provide (e.g., indenting <head> two spaces more than <html>) isn't necessary, but we did so just to make it easier to read.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:45.711520Z",
     "start_time": "2019-02-18T15:07:45.708506Z"
    }
   },
   "outputs": [],
   "source": [
    "html = '''\n",
    "<html>\n",
    "  <head>\n",
    "    <title>Intro HTML</title>\n",
    "  </head>\n",
    "  <body>\n",
    "    <p>Hello World!</p>\n",
    "    <p>Enjoy DataCamp!</p>\n",
    "  </body>\n",
    "</html>\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where am I?\n",
    "\n",
    "In this exercise, you will navigate to a specific element using your new knowledge of XPath notation.\n",
    "\n",
    "Consider the HTML code:\n",
    "\n",
    "    <html>\n",
    "      <body>\n",
    "        <div>\n",
    "          <p>Good Luck!</p>\n",
    "          <p>Not here...</p>\n",
    "        </div>\n",
    "        <div>\n",
    "          <p>Where am I?</p>\n",
    "        </div>\n",
    "      </body>\n",
    "    </html>\n",
    "\n",
    "Your job will be to create an XPath string using only single forward-slashes which navigates to the paragraph p element which contains the text \"Where am I?\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:48.295255Z",
     "start_time": "2019-02-18T15:07:48.291266Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in the blank\n",
    "xpath = '/html/body/div[2]/p[1]'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## It's Time to P\n",
    "\n",
    "In the lecture, we learned how to use double forward-slashes to navigate to all future generations. In this exercise, you will select all paragraph p elements within the HTML. Because we want you to navigate to all paragraph elements, it is not important that you know what the HTML code is, since the task can be accomplished with a simple XPath string using the double forward-slash notation you have learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:49.698151Z",
     "start_time": "2019-02-18T15:07:49.694160Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in the blank\n",
    "xpath = '//p'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A classy span\n",
    "\n",
    "Although we haven't yet gone deep into XPath, one thing we can do is select elements by their attributes using an XPath. For example, if we want to direct to the div element within the HTML document whose id attribute is \"uid\", then we could write the XPath string '//div[@id=\"uid\"]'. The first part of this string, //div, first looks at all div elements in the HTML document. Then, using the brackets, we specify that we want only the div element with a specific id attribute (in this case uid). To note, the phrase @id=\"uid\" in the brackets would be read as \"attribute id equals uid\".\n",
    "\n",
    "In this exercise, you will select all span elements whose class attribute equals \"span-class\". (Note: span is just another possible tag-name)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:50.962898Z",
     "start_time": "2019-02-18T15:07:50.958908Z"
    }
   },
   "outputs": [],
   "source": [
    "# Fill in the blank\n",
    "xpath = '//span[@class=\"span-class\"]'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XPaths and Selectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Body Appendages\n",
    "\n",
    "We have loaded the HTML from a secret website and have used it to create a function how_many_elements(). The way this function works is that you pass it an XPath string and it will print out the number of elements the XPath you wrote has selected. For example, by running the code how_many_elements('//*') in the console will print out the total number of elements the HTML document has (try it!).\n",
    "\n",
    "Your job in this exercise is to create an XPath string which can be used to direct to all child elements the body (regardless of tag type). To note, you can first test your solution with how_many_elements() to find the total number of children in the body element if you wish.\n",
    "\n",
    "Note that the exercises in this chapter may take some time to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:07:53.221990Z",
     "start_time": "2019-02-18T15:07:53.210001Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'how_many_elements' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-37-e39e63dba318>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out the number of elements selected\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mhow_many_elements\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'how_many_elements' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an XPath string to direct to children of body element\n",
    "xpath = '/html/body/*'\n",
    "\n",
    "# Print out the number of elements selected\n",
    "how_many_elements( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:01:39.969680Z",
     "start_time": "2019-02-18T15:01:39.960724Z"
    }
   },
   "source": [
    "## Choose DataCamp!\n",
    "\n",
    "In this exercise, we want to give you the opportunity to create your own XPath string to achieve a certain task; the task is to select the paragraph element containing the text \"Choose DataCamp!\".\n",
    "\n",
    "Consider the following HTML:\n",
    "\n",
    "    <html>\n",
    "      <body>\n",
    "        <div>\n",
    "          <p>Hello World!</p>\n",
    "          <div>\n",
    "            <p>Choose DataCamp!</p>\n",
    "          </div>\n",
    "        </div>\n",
    "        <div>\n",
    "          <p>Thanks for Watching!</p>\n",
    "        </div>\n",
    "      </body>\n",
    "    </html>\n",
    "\n",
    "We have created the function print_element_text() for you, which will print the text contained in your element (if it contains any). Feel free to use this function to check if your solution is correct!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:01.789498Z",
     "start_time": "2019-02-18T15:08:01.777528Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_element_text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-38-fe6f9a4d9b9a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out the element text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprint_element_text\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_element_text' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an XPath string to the desired paragraph element\n",
    "xpath = '/html/body/div[1]/div/p'\n",
    "\n",
    "# Print out the element text\n",
    "print_element_text( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where it's @\n",
    "\n",
    "In this exercise, you'll begin to write an XPath string using attributes to achieve a certain task; that task is to select the paragraph element containing the text \"Thanks for Watching!\". We've already created most of the XPath string for you.\n",
    "\n",
    "Consider the following HTML:\n",
    "\n",
    "    <html>\n",
    "      <body>\n",
    "        <div id=\"div1\" class=\"class-1\">\n",
    "          <p class=\"class-1 class-2\">Hello World!</p>\n",
    "          <div id=\"div2\">\n",
    "            <p id=\"p2\" class=\"class-2\">Choose DataCamp!</p>\n",
    "          </div>\n",
    "        </div>\n",
    "        <div id=\"div3\" class=\"class-2\">\n",
    "          <p class=\"class-2\">Thanks for Watching!</p>\n",
    "        </div>\n",
    "      </body>\n",
    "    </html>\n",
    "\n",
    "We have created the function print_element_text() for you, which will print any text contained in your element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:03.404280Z",
     "start_time": "2019-02-18T15:08:03.394297Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_element_text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-39-ca202c419d2d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out selection text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprint_element_text\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_element_text' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an Xpath string to select desired p element\n",
    "xpath = '//*[@id=\"div3\"]/p'\n",
    "\n",
    "# Print out selection text\n",
    "print_element_text( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check your Class\n",
    "\n",
    "This exercise is to emphasize that when you use an XPath to select an element by its class attribute without using the contains() function, you match the class exactly. Your job is to fill in the blank below and finish the variable xpath directing to the specified element.\n",
    "\n",
    "Consider the following HTML:\n",
    "\n",
    "    <html>\n",
    "      <body>\n",
    "        <div id=\"div1\" class=\"class-1\">\n",
    "          <p class=\"class-1 class-2\">Hello World!</p>\n",
    "          <div id=\"div2\">\n",
    "            <p id=\"p2\" class=\"class-2\">Choose DataCamp!</p>\n",
    "          </div>\n",
    "        </div>\n",
    "        <div id=\"div3\" class=\"class-2\">\n",
    "          <p class=\"class-2\">Thanks for Watching!</p>\n",
    "        </div>\n",
    "      </body>\n",
    "    </html>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:07.038359Z",
     "start_time": "2019-02-18T15:08:07.023389Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_element_text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-40-71a870d4bddf>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out select text\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprint_element_text\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_element_text' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an XPath string to select p element by class\n",
    "xpath = '//p[@class=\"class-1 class-2\"]'\n",
    "\n",
    "# Print out select text\n",
    "print_element_text( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper(link) Active\n",
    "\n",
    "One of the most important attributes to extract for \"web-crawling\" is the hyperlink url (href attribute) within an a tag. Here, you will extract such a hyperlink! We have created the function print_attribute to print out the data extracted from your XPath, so you can test your XPath strings in the console, if you like.\n",
    "\n",
    "The exercise refers to the following HTML source code:\n",
    "\n",
    "    <html>\n",
    "      <body>\n",
    "        <div id=\"div1\" class=\"class-1\">\n",
    "          <p class=\"class-1 class-2\">Hello World!</p>\n",
    "          <div id=\"div2\">\n",
    "            <p id=\"p2\" class=\"class-2\">Choose \n",
    "                <a href=\"http://datacamp.com\">DataCamp!</a>!\n",
    "            </p>\n",
    "          </div>\n",
    "        </div>\n",
    "        <div id=\"div3\" class=\"class-2\">\n",
    "          <p class=\"class-2\">Thanks for Watching!</p>\n",
    "        </div>\n",
    "      </body>\n",
    "    </html>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:11.290471Z",
     "start_time": "2019-02-18T15:08:11.275509Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_attribute' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-41-4204b271f5d9>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out the selection(s); there should be only one\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mprint_attribute\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_attribute' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an xpath to the href attribute\n",
    "xpath = '//p[@id=\"p2\"]/a/@href'\n",
    "\n",
    "# Print out the selection(s); there should be only one\n",
    "print_attribute( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Secret Links\n",
    "\n",
    "We have loaded the HTML from a secret website and have used it to create the functions how_many_elements() and preview(). The function how_many_elements() allows you to pass in an XPath string and it will print out the number of elements the XPath you wrote has selected. The function preview() allows you to pass in an XPath string and it will print out the first few elements you've selected.\n",
    "\n",
    "Your job in this exercise is to create an XPath which directs to all href attribute values of the hyperlink a elements whose class attributes contain the string \"course-block\". If you do it correctly, you should find that you have selected 40 elements with your XPath string and that it previews links (with some repetition)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:12.772701Z",
     "start_time": "2019-02-18T15:08:12.760734Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'how_many_elements' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-42-5618dcd61078>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out how many elements are selected\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mhow_many_elements\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;31m# Preview the selected elements\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[0mpreview\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'how_many_elements' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an xpath to the href attributes\n",
    "xpath = '//a[contains(@class,\"course-block\")]/@href'\n",
    "\n",
    "# Print out how many elements are selected\n",
    "how_many_elements( xpath )\n",
    "# Preview the selected elements\n",
    "preview( xpath )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XPath Chaining\n",
    "\n",
    "Selector and SelectorList objects allow for chaining when using the xpath method. What this means is that you can apply the xpath method over once you've already applied it. For example, if sel is the name of our Selector, then\n",
    "\n",
    "    sel.xpath('/html/body/div[2]')\n",
    "\n",
    "is the same as\n",
    "\n",
    "    sel.xpath('/html').xpath('./body/div[2]')\n",
    "\n",
    "or is the same as\n",
    "\n",
    "    sel.xpath('/html').xpath('./body').xpath('./div[2]')\n",
    "\n",
    "The only catch is that you need to \"glue together\" the XPath pieces by using a period at the start of each subsequent XPath string (notice the periods we added to the XPath strings in our examples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:14.425448Z",
     "start_time": "2019-02-18T15:08:14.411485Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Chain together xpath methods to select desired p element\n",
    "sel.xpath( '//div' ).xpath( './span/p[3]' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Divvy Up This Exercise\n",
    "\n",
    "We have pre-loaded an HTML into the string variable html. In this two part problem you will use this html variable as the HTML document to set up a Selector object with, and create a SelectorList which selects all div elements; then, you will check your understanding of what happens within the SelectorList."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:18.568086Z",
     "start_time": "2019-02-18T15:08:18.561122Z"
    }
   },
   "outputs": [],
   "source": [
    "from scrapy import Selector\n",
    "\n",
    "# Create a Selector selecting html as the HTML document\n",
    "sel = Selector( text = html )\n",
    "\n",
    "# Create a SelectorList of all div elements in the HTML document\n",
    "divs = sel.xpath( '//div' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requesting a Selector\n",
    "\n",
    "We have pre-loaded the URL for a particular website in the string variable url and use the requests library to put the content from the website into the string variable html. Your task is to create a Selector object sel using the HTML source code stored in html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:20.132020Z",
     "start_time": "2019-02-18T15:08:20.128031Z"
    }
   },
   "outputs": [],
   "source": [
    "url = 'https://assets.datacamp.com/production/repositories/2560/datasets/19a0a26daa8d9db1d920b5d5607c19d6d8094b3b/all_short'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:21.471316Z",
     "start_time": "2019-02-18T15:08:21.391991Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 1020 elements in the HTML document.\n",
      "You have found:  1020\n"
     ]
    }
   ],
   "source": [
    "# Import a scrapy Selector\n",
    "from scrapy import Selector\n",
    "\n",
    "# Import requests\n",
    "import requests\n",
    "\n",
    "# Create the string html containing the HTML source\n",
    "html = requests.get( url ).content\n",
    "\n",
    "# Create the Selector object sel from html\n",
    "sel = Selector( text = html )\n",
    "\n",
    "# Print out the number of elements in the HTML document\n",
    "print( \"There are 1020 elements in the HTML document.\")\n",
    "print( \"You have found: \", len( sel.xpath('//*') ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSS Locators, Chaining, and Responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The (X)Path to CSS Locators\n",
    "\n",
    "Many people prefer using CSS Locator notation to XPath notation. As we will see later, it often makes attribute selection very easy. To help get you more comfortable going back and forth between XPath and CSS Locator strings, we give you a chance in this exercise to do some direct \"translation\" between the two.\n",
    "\n",
    "Note that the exercises in this chapter may take some time to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:48.272683Z",
     "start_time": "2019-02-18T15:08:48.268673Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create the XPath string equivalent to the CSS Locator \n",
    "xpath = '/html/body/span[1]//a'\n",
    "\n",
    "# Create the CSS Locator string equivalent to the XPath\n",
    "css_locator = 'html > body > span:nth-of-type(1) a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:08:49.884935Z",
     "start_time": "2019-02-18T15:08:49.879950Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create the XPath string equivalent to the CSS Locator \n",
    "xpath = '//div[@id=\"uid\"]/span//h4'\n",
    "\n",
    "# Create the CSS Locator string equivalent to the XPath\n",
    "css_locator = 'div#uid > span h4'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get an \"a\" in this Course\n",
    "\n",
    "We have loaded the HTML from a secret website which you will use to set up a Selector object and the function how_many_elements(). When passing this function a CSS Locator string, it will print out the number of elements that the CSS Locator you wrote has selected.\n",
    "\n",
    "In the second part of this problem, we want you to create a CSS Locator string which will select a certain collection of elements as described here: Select the hyperlink (a element) children of all div elements belonging to the class \"course-block\" (that is, any div element with a class attribute such that \"course-block\" is one of the classes assigned). The number of such elements is 11, so you can check your solution with how_many_elements if you choose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:09:02.388964Z",
     "start_time": "2019-02-18T15:09:02.375003Z"
    }
   },
   "outputs": [],
   "source": [
    "from scrapy import Selector\n",
    "\n",
    "# Create a selector from the html (of a secret website)\n",
    "sel = Selector( text = html )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:09:09.909677Z",
     "start_time": "2019-02-18T15:09:09.885177Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'how_many_elements' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-50-db92312066ba>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;31m# Print the number of selected elements.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mhow_many_elements\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mcss_locator\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'how_many_elements' is not defined"
     ]
    }
   ],
   "source": [
    "from scrapy import Selector\n",
    "\n",
    "# Create a selector from the html (of a secret website)\n",
    "sel = Selector( text = html )\n",
    "\n",
    "# Fill in the blank\n",
    "css_locator = 'div.course-block > a'\n",
    "\n",
    "# Print the number of selected elements.\n",
    "how_many_elements( css_locator )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The CSS Wildcard\n",
    "\n",
    "You can use the wildcard * in CSS Locators too! In fact, we can use it in a similar way, when we want to ignore the tag type. For example:\n",
    "\n",
    "    The CSS Locator string '*' selects all elements in the HTML document.\n",
    "    The CSS Locator string '*.class-1' selects all elements which belong to class-1, but this is unnecessary since the string '.class-1' will also do the same job.\n",
    "    The CSS Locator string '*#uid' selects the element with id attribute equal to uid, but this is unnecessary since the string '#uid' will also do the same job.\n",
    "\n",
    "In this exercise, we want you to work by analogy with the wildcard character you know from XPath notation to discover how to select all the children of a certain element in CSS Locator notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:09:24.066595Z",
     "start_time": "2019-02-18T15:09:24.062609Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create the CSS Locator to all children of the element whose id is uid\n",
    "css_locator = \"#uid > *\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You've been `href`ed\n",
    "\n",
    "In a previous exercise, you created a CSS Locator string to select the hyperlink (a element) children of all div elements belonging to the class \"course-block\". Here we have created a SelectorList called course_as having selected those hyperlink children.\n",
    "\n",
    "Now, we want you to fill in the blank below to extract the href attribute values from these elements. This is another example of chaining, as we've seen in a previous exercise.\n",
    "\n",
    "The point here is that we can chain together calls to the methods css and xpath, and combine them! We help nudge you in the correct direction by giving you the solution if we chain with another call to the css method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:10:05.051342Z",
     "start_time": "2019-02-18T15:10:05.035384Z"
    }
   },
   "outputs": [],
   "source": [
    "from scrapy import Selector\n",
    "\n",
    "# Create a selector object from a secret website\n",
    "sel = Selector( text = html )\n",
    "\n",
    "# Select all hyperlinks of div elements belonging to class \"course-block\"\n",
    "course_as = sel.css( 'div.course-block > a' )\n",
    "\n",
    "# Selecting all href attributes chaining with css\n",
    "hrefs_from_css = course_as.css( '::attr(href)' )\n",
    "\n",
    "# Selecting all href attributes chaining with xpath\n",
    "hrefs_from_xpath = course_as.xpath( './@href' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Top Level Text\n",
    "\n",
    "This exercise will have you write an XPath and CSS Locator string to direct to the text of a specific paragraph p element. The p element in the HTML is uniquely defined by its id attribute, which is \"p3\". With this small piece of information, you should be able to create the desired strings; however, we have preloaded the variable html with a string containing the HTML in which this link belongs, if you want to peruse it.\n",
    "\n",
    "In this exercise, you will only be selecting the text within the element, which does not include the text in future generations of the element. We have created a function print_results for you to compare which elements your strings direct to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:10:16.539346Z",
     "start_time": "2019-02-18T15:10:16.524388Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_results' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-53-877baad04f58>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;31m# Print the text from our selections\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m \u001b[0mprint_results\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcss_locator\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_results' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an XPath string to the desired text.\n",
    "xpath = '//p[@id=\"p3\"]/text()'\n",
    "\n",
    "# Create a CSS Locator string to the desired text.\n",
    "css_locator = 'p#p3::text'\n",
    "\n",
    "# Print the text from our selections\n",
    "print_results( xpath, css_locator )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All Level Text\n",
    "\n",
    "This exercise is similar to the previous, but differs in that you will be selecting text from multiple generations of a given element.\n",
    "\n",
    "You will write an XPath and CSS Locator strings to direct to the text of a specific paragraph p element. The p element in the HTML is uniquely defined by its id attribute, which is \"p3\". With this small piece of information, you should be able to create the desired strings; however, we have preloaded the variable html with a string containing the HTML in which this link belongs, if you want to peruse it.\n",
    "\n",
    "In this exercise, you will only be selecting the text within the element which includes all text within the future generations. We have created a function print_results for you to compare which elements your strings direct to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:10:31.171838Z",
     "start_time": "2019-02-18T15:10:31.159868Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'print_results' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-54-23b9db178024>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;31m# Print the text from our selections\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 8\u001b[1;33m \u001b[0mprint_results\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mxpath\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcss_locator\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'print_results' is not defined"
     ]
    }
   ],
   "source": [
    "# Create an XPath string to the desired text.\n",
    "xpath = '//p[@id=\"p3\"]//text()'\n",
    "\n",
    "# Create a CSS Locator string to the desired text.\n",
    "css_locator = 'p#p3 ::text'\n",
    "\n",
    "# Print the text from our selections\n",
    "print_results( xpath, css_locator )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reveal By Response\n",
    "\n",
    "We have pre-loaded a Response object, named response with the content from a secret website. Your job is to figure out the URL and the title of the website using the response variable. You learned how to find the URL in the last lesson. To find the website title, what you need to know is:\n",
    "\n",
    "    The title is the text from the title element\n",
    "    The title element is a child of the head element, which is a child of the html root element.\n",
    "\n",
    "To note: the html root element only has one child head element, and the head element only has one child title element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:11:25.114559Z",
     "start_time": "2019-02-18T15:11:25.101570Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'response' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-55-5da56a8e719c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Get the URL to the website loaded in response\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mthis_url\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Get the title of the website loaded in response\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mthis_title\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m'/html/head/title/text()'\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextract_first\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'response' is not defined"
     ]
    }
   ],
   "source": [
    "# Get the URL to the website loaded in response\n",
    "this_url = response.url\n",
    "\n",
    "# Get the title of the website loaded in response\n",
    "this_title = response.xpath( '/html/head/title/text()' ).extract_first()\n",
    "\n",
    "# Print out our findings\n",
    "print_url_title( this_url, this_title )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Responding with Selectors\n",
    "\n",
    "Something that we should emphasize at this point about the relationship between a Selector and Response objects is that both objects return a SelectorList when using the xpath or css methods to direct to elements. In this exercise, we'll prove it to you, by having you find all hyperlink elements belonging to the class course-block__link (notice the double underscore!) and looking at the object that is produced when doing so.\n",
    "\n",
    "We have pre-loaded both a Response object named response and a Selector object named sel with the content from the same \"secret\" website. Once you complete the task of creating a CSS Locator, you will compare both the output from response.css and selector.css to see that they are effectively the same!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:11:37.971332Z",
     "start_time": "2019-02-18T15:11:37.957355Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'response' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-56-7ab7845aa976>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Select the hyperlink elements from response and sel\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[0mresponse_as\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcss\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mcss_locator\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[0msel_as\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcss\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mcss_locator\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'response' is not defined"
     ]
    }
   ],
   "source": [
    "# Create a CSS Locator string to the desired hyperlink elements\n",
    "css_locator = 'a.course-block__link'\n",
    "\n",
    "# Select the hyperlink elements from response and sel\n",
    "response_as = response.css( css_locator )\n",
    "sel_as = sel.css( css_locator )\n",
    "\n",
    "# Examine similarity\n",
    "nr = len( response_as )\n",
    "ns = len( sel_as )\n",
    "for i in range( min(nr, ns, 2) ):\n",
    "  print( \"Element %d from response: %s\" % (i+1, response_as[i]) )\n",
    "  print( \"Element %d from sel: %s\" % (i+1, sel_as[i]) )\n",
    "  print( \"\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting from a Selection\n",
    "\n",
    "In this exercise, you will find the text from an h4 element within a particular div element. It will occur in steps where the first step is selecting a family of div elements, and the second step is narrowing in on the first one, from which we will grab the h4 element text. This process of progressively narrowing in on elements (e.g., first to the div elements, then to the h4 element) is another example of \"chaining\", even if it doesn't look exactly the same as we've seen it before.\n",
    "\n",
    "Along the way in this exercise, there is a variable first_div set up for you to use. Think carefully about what type of object first_div is!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:11:48.558644Z",
     "start_time": "2019-02-18T15:11:48.544676Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'response' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-57-08961d268a18>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Select all desired div elements\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mdivs\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcss\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m'div.course-block'\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Take the first div element\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mfirst_div\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdivs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'response' is not defined"
     ]
    }
   ],
   "source": [
    "# Select all desired div elements\n",
    "divs = response.css( 'div.course-block' )\n",
    "\n",
    "# Take the first div element\n",
    "first_div = divs[0]\n",
    "\n",
    "# Extract the text from the h4 element in first_div\n",
    "h4_text = first_div.css('h4::text').extract_first()\n",
    "\n",
    "# Print out the text\n",
    "print( \"The text from the h4 element is:\", h4_text )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Titular\n",
    "\n",
    "Similar to the work given in the previous lesson, we will have you use a pre-loaded Response object, named response to scrape the course titles from the (shortened version of the) DataCamp course directory https://www.datacamp.com/courses/all. To successfully do so, you only need to know the following\n",
    "\n",
    "    The course titles are the text from all the h4 elements within the HTML document.\n",
    "\n",
    "We ask you to extract these course titles here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:12:07.781398Z",
     "start_time": "2019-02-18T15:12:07.764443Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'response' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-58-b8e938379f5b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Create a SelectorList of the course titles\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mcrs_title_els\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcss\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m'h4::text'\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Extract the course titles\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mcrs_titles\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mcrs_title_els\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextract\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'response' is not defined"
     ]
    }
   ],
   "source": [
    "# Create a SelectorList of the course titles\n",
    "crs_title_els = response.css( 'h4::text' )\n",
    "\n",
    "# Extract the course titles \n",
    "crs_titles = crs_title_els.extract()\n",
    "\n",
    "# Print out the course titles \n",
    "for el in crs_titles:\n",
    "  print( \">>\", el )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping with Children\n",
    "\n",
    "We did a cute trick in the lesson to calculate how many children there were of one of the div elements belonging to the class course-block. Here we ask you to find the number of children of a mystery element (already stored within a Selector object, so you can use the xpath or css method).\n",
    "\n",
    "To be explicit, we have created the Selector object mystery in the following way:\n",
    "\n",
    "    We first loaded a Response variable using a secret website as the input.\n",
    "    Then we used a call to the xpath method to create a SelectorList of elements (but we won't say which ones)\n",
    "    Finally, we let mystery be the first Selector object of this SelectorList.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:12:18.231139Z",
     "start_time": "2019-02-18T15:12:18.219171Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'mystery' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-59-166218ccf1f6>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Calculate the number of children of the mystery element\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mhow_many_kids\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mmystery\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m'./*'\u001b[0m \u001b[1;33m)\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;31m# Print out the number\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m \u001b[1;34m\"The number of elements you selected was:\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhow_many_kids\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'mystery' is not defined"
     ]
    }
   ],
   "source": [
    "# Calculate the number of children of the mystery element\n",
    "how_many_kids = len( mystery.xpath( './*' ) )\n",
    "\n",
    "# Print out the number\n",
    "print( \"The number of elements you selected was:\", how_many_kids )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spiders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inheriting the Spider\n",
    "\n",
    "When learning about scrapy spiders, we saw that the main portion of the code for us to adjust is the class for the spider. To help build some familiarity of the class, you will complete a short piece of code to complete a toy-model of the spider class code. We've omitted the code that would actually run the spider, only including the pieces necessary to create the class.\n",
    "\n",
    "As mentioned in the lesson, a class is roughly a collection of related variables and functions housed together. Sometimes one class likes to use methods from another class, and so we will inherit methods from a different class. That's what we do in the spider class.\n",
    "\n",
    "We wrote the function inspect_class to look at the your class once you're done, if you'd like to test your solution!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:12:42.516786Z",
     "start_time": "2019-02-18T15:12:42.502823Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_class' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-60-2e81f9d0fa6e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;31m# Inspect Your Class\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 15\u001b[1;33m \u001b[0minspect_class\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mYourSpider\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_class' is not defined"
     ]
    }
   ],
   "source": [
    "# Import scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the spider class\n",
    "class YourSpider(scrapy.Spider):\n",
    "  name = \"your_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests(self):\n",
    "    pass\n",
    "  # parse method\n",
    "  def parse(self, response):\n",
    "    pass\n",
    "  \n",
    "# Inspect Your Class\n",
    "inspect_class(YourSpider)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hurl the URLs\n",
    "\n",
    "In the next lesson we will talk about the start_requests method within the spider class. In this quick exercise, we ask you to change around a variable within the start_requests method which foreshadows some of what we will be learning in the next lesson. Basically, we want you to start becoming comfortable turning some of the wheels within a spider class; in this case, making a list of urls within the start_requests method.\n",
    "\n",
    "We've written a function inspect_class which will print out the list of elements you have in the urls variable within the start_requests method.\n",
    "\n",
    "Note: in the next several exercises, you will write code to complete your spider class, but the code does not yet include the pieces to actually run the spider; that will come at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:14:18.677912Z",
     "start_time": "2019-02-18T15:14:18.661955Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_class' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-61-2e4f1d07e692>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     16\u001b[0m \u001b[1;31m# Inspect Your Class\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 17\u001b[1;33m \u001b[0minspect_class\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mYourSpider\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_class' is not defined"
     ]
    }
   ],
   "source": [
    "# Import scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the spider class\n",
    "class YourSpider( scrapy.Spider ):\n",
    "  name = \"your_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests( self ):\n",
    "    urls = [\"https://www.datacamp.com\", \"https://scrapy.org\"]\n",
    "    for url in urls:\n",
    "      yield url\n",
    "  # parse method\n",
    "  def parse( self, response ):\n",
    "    pass\n",
    "  \n",
    "# Inspect Your Class\n",
    "inspect_class( YourSpider )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Self Referencing is Classy\n",
    "\n",
    "You probably have noticed that within the spider class, we always input the argument self in the start_requests and parse methods (just look in the sample code in this exercise!). This allows us to reference between methods within the class. That is, if we want to refer to the method parse within the start_requests method, we would need to write self.parse rather than just parse; what writing self does is tell the code: \"Look in the same class as start_requests for a method called parse to use.\"\n",
    "\n",
    "In this exercise you will get a chance to play with this \"self referencing\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:14:49.608876Z",
     "start_time": "2019-02-18T15:14:49.593916Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_class' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-62-130b7d319e31>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     16\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     17\u001b[0m \u001b[1;31m# Inspect Your Class\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 18\u001b[1;33m \u001b[0minspect_class\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mYourSpider\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_class' is not defined"
     ]
    }
   ],
   "source": [
    "# Import scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the spider class\n",
    "class YourSpider( scrapy.Spider ):\n",
    "  name = \"your_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests( self ):\n",
    "    self.print_msg( \"Hello World!\" )\n",
    "  # parse method\n",
    "  def parse( self, response ):\n",
    "    pass\n",
    "  # print_msg method\n",
    "  def print_msg( self, msg ):\n",
    "    print( \"Calling start_requests in YourSpider prints out:\", msg )\n",
    "  \n",
    "# Inspect Your Class\n",
    "inspect_class( YourSpider )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting with Start Requests\n",
    "\n",
    "In the last lesson we learned about setting up the start_requests method within a scrapy spider. Here we have another toy-model spider which doesn't actually scrape anything, but gives you a chance to play with the start_requests method. What we want is for you to start becomming familiar with the arguments you pass into the scrapy.Request call within start_requests.\n",
    "\n",
    "As before, we have created the function inspect_class to examine what you are yielding in start_requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:14:59.784930Z",
     "start_time": "2019-02-18T15:14:59.766978Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_class' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-63-c22efa140b0f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     14\u001b[0m \u001b[1;31m# Inspect Your Class\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 15\u001b[1;33m \u001b[0minspect_class\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mYourSpider\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_class' is not defined"
     ]
    }
   ],
   "source": [
    "# Import scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the spider class\n",
    "class YourSpider( scrapy.Spider ):\n",
    "  name = \"your_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests( self ):\n",
    "    yield scrapy.Request( url = \"https://www.datacamp.com\", callback = self.parse )\n",
    "  # parse method\n",
    "  def parse( self, response ):\n",
    "    pass\n",
    "  \n",
    "# Inspect Your Class\n",
    "inspect_class( YourSpider )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pen Names\n",
    "\n",
    "In this exercise, we have set up a spider class which, when finished, will retrieve the author names from a shortened version of the DataCamp course directory. The URL for the shortened version is stored in the variable url_short. Your job will be to create the list of extracted author names in the parse method of the spider.\n",
    "\n",
    "Two things you should know:\n",
    "\n",
    "    You will be using the response object and the css method here.\n",
    "    The course author names are defined by the text within the paragraph p elements belonging to the class course-block__author-name\n",
    "\n",
    "You can inspect the spider using the function inspect_spider() that we built for you -- it will print out the author names you find!\n",
    "\n",
    "Note that this and the remaining exercises in this chapter may take some time to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:15:12.622838Z",
     "start_time": "2019-02-18T15:15:12.606902Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_spider' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-64-41c58388c23a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     16\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     17\u001b[0m \u001b[1;31m# Inspect the spider\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 18\u001b[1;33m \u001b[0minspect_spider\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mDCspider\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_spider' is not defined"
     ]
    }
   ],
   "source": [
    "# Import the scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the Spider class\n",
    "class DCspider( scrapy.Spider ):\n",
    "  name = 'dcspider'\n",
    "  # start_requests method\n",
    "  def start_requests( self ):\n",
    "    yield scrapy.Request( url = url_short, callback = self.parse )\n",
    "  # parse method\n",
    "  def parse( self, response ):\n",
    "    # Create an extracted list of course author names\n",
    "    author_names = response.css( 'p.course-block__author-name::text' ).extract()\n",
    "    # Here we will just return the list of Authors\n",
    "    return author_names\n",
    "  \n",
    "# Inspect the spider\n",
    "inspect_spider( DCspider )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawler Time\n",
    "\n",
    "This will be your first chance to play with a spider which will crawls between sites (by first collecting links from one site, and following those links to parse new sites). This spider starts at the shortened DataCamp course directory, then extracts the links of the courses in the parse method; from there, it will follow those links to extract the course descriptions from each course page in the parse_descr method, and put these descriptions into the list course_descrs. Your job is to complete the code so that the spider runs as desired!\n",
    "\n",
    "We have created a function inspect_spider which will print out one of the course descriptions you scrape (if done correctly)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:15:23.505493Z",
     "start_time": "2019-02-18T15:15:23.491533Z"
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'inspect_spider' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-65-ec104b910618>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     25\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     26\u001b[0m \u001b[1;31m# Inspect the spider\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 27\u001b[1;33m \u001b[0minspect_spider\u001b[0m\u001b[1;33m(\u001b[0m \u001b[0mDCdescr\u001b[0m \u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'inspect_spider' is not defined"
     ]
    }
   ],
   "source": [
    "# Import the scrapy library\n",
    "import scrapy\n",
    "\n",
    "# Create the Spider class\n",
    "class DCdescr( scrapy.Spider ):\n",
    "  name = 'dcdescr'\n",
    "  # start_requests method\n",
    "  def start_requests( self ):\n",
    "    yield scrapy.Request( url = url_short, callback = self.parse )\n",
    "  \n",
    "  # First parse method\n",
    "  def parse( self, response ):\n",
    "    links = response.css( 'div.course-block > a::attr(href)' ).extract()\n",
    "    # Follow each of the extracted links\n",
    "    for link in links:\n",
    "      yield response.follow( url = link, callback = self.parse_descr )\n",
    "      \n",
    "  # Second parsing method\n",
    "  def parse_descr( self, response ):\n",
    "    # Extract course description\n",
    "    course_descr = response.css( 'p.course__description::text' ).extract_first()\n",
    "    # For now, just yield the course description\n",
    "    yield course_descr\n",
    "\n",
    "\n",
    "# Inspect the spider\n",
    "inspect_spider( DCdescr )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time to Run\n",
    "\n",
    "In the last lesson, we went through creating an entire web-crawler to access course information from each course in the DataCamp course directory. However, the lesson seemed to stop without a climax, because we didn't play with the code after finishing the parsing methods.\n",
    "\n",
    "The point of this exercise is to remedy that!\n",
    "\n",
    "The code we give you to look at in this and the next exercise is long, because its the entire spider that took us the lesson to create! However, don't be intimidated! The point of these two exercises is to give you a very easy task to complete, with the hope that you will look at and run the code for this spider. That way, even though it is long, you will have a grasp of it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:15:46.485776Z",
     "start_time": "2019-02-18T15:15:45.258974Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-18 17:15:45 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)\n",
      "2019-02-18 17:15:45 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.7 |Anaconda, Inc.| (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1a  20 Nov 2018), cryptography 2.4.2, Platform Windows-10-10.0.17763-SP0\n",
      "2019-02-18 17:15:45 [scrapy.crawler] INFO: Overridden settings: {}\n",
      "2019-02-18 17:15:46 [scrapy.middleware] INFO: Enabled extensions:\n",
      "['scrapy.extensions.corestats.CoreStats',\n",
      " 'scrapy.extensions.telnet.TelnetConsole',\n",
      " 'scrapy.extensions.logstats.LogStats']\n",
      "2019-02-18 17:15:46 [scrapy.middleware] INFO: Enabled downloader middlewares:\n",
      "['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n",
      " 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n",
      " 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n",
      " 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n",
      " 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n",
      " 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\n",
      " 'scrapy.downloadermiddlewares.stats.DownloaderStats']\n",
      "2019-02-18 17:15:46 [scrapy.middleware] INFO: Enabled spider middlewares:\n",
      "['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n",
      " 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n",
      " 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n",
      " 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n",
      " 'scrapy.spidermiddlewares.depth.DepthMiddleware']\n",
      "2019-02-18 17:15:46 [scrapy.middleware] INFO: Enabled item pipelines:\n",
      "[]\n",
      "2019-02-18 17:15:46 [scrapy.core.engine] INFO: Spider opened\n",
      "2019-02-18 17:15:46 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n",
      "2019-02-18 17:15:46 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023\n",
      "2019-02-18 17:15:46 [scrapy.core.engine] ERROR: Error while obtaining start requests\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\Admin\\Anaconda3\\lib\\site-packages\\scrapy\\core\\engine.py\", line 127, in _next_request\n",
      "    request = next(slot.start_requests)\n",
      "  File \"<ipython-input-66-8dcd9216d1cd>\", line 12, in start_requests\n",
      "    yield scrapy.Request(url = url_short,\n",
      "NameError: name 'url_short' is not defined\n",
      "2019-02-18 17:15:46 [scrapy.core.engine] INFO: Closing spider (finished)\n",
      "2019-02-18 17:15:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats:\n",
      "{'finish_reason': 'finished',\n",
      " 'finish_time': datetime.datetime(2019, 2, 18, 15, 15, 46, 457835),\n",
      " 'log_count/DEBUG': 1,\n",
      " 'log_count/ERROR': 1,\n",
      " 'log_count/INFO': 7,\n",
      " 'start_time': datetime.datetime(2019, 2, 18, 15, 15, 46, 389020)}\n",
      "2019-02-18 17:15:46 [scrapy.core.engine] INFO: Spider closed (finished)\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'previewCourses' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-66-8dcd9216d1cd>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     37\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     38\u001b[0m \u001b[1;31m# Print a preview of courses\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 39\u001b[1;33m \u001b[0mpreviewCourses\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdc_dict\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'previewCourses' is not defined"
     ]
    }
   ],
   "source": [
    "# Import scrapy\n",
    "import scrapy\n",
    "\n",
    "# Import the CrawlerProcess: for running the spider\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "# Create the Spider class\n",
    "class DC_Chapter_Spider(scrapy.Spider):\n",
    "  name = \"dc_chapter_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests(self):\n",
    "    yield scrapy.Request(url = url_short,\n",
    "                         callback = self.parse_front)\n",
    "  # First parsing method\n",
    "  def parse_front(self, response):\n",
    "    course_blocks = response.css('div.course-block')\n",
    "    course_links = course_blocks.xpath('./a/@href')\n",
    "    links_to_follow = course_links.extract()\n",
    "    for url in links_to_follow:\n",
    "      yield response.follow(url = url,\n",
    "                            callback = self.parse_pages)\n",
    "  # Second parsing method\n",
    "  def parse_pages(self, response):\n",
    "    crs_title = response.xpath('//h1[contains(@class,\"title\")]/text()')\n",
    "    crs_title_ext = crs_title.extract_first().strip()\n",
    "    ch_titles = response.css('h4.chapter__title::text')\n",
    "    ch_titles_ext = [t.strip() for t in ch_titles.extract()]\n",
    "    dc_dict[ crs_title_ext ] = ch_titles_ext\n",
    "\n",
    "# Initialize the dictionary **outside** of the Spider class\n",
    "dc_dict = dict()\n",
    "\n",
    "# Run the Spider\n",
    "process = CrawlerProcess()\n",
    "process.crawl(DC_Chapter_Spider)\n",
    "process.start()\n",
    "\n",
    "# Print a preview of courses\n",
    "previewCourses(dc_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataCamp Descriptions\n",
    "\n",
    "Like the previous exercise, the code here is long since you are working with an entire web-crawling spider! But again, don't let the amount of code intimidate you, you have a handle on how spiders work now, and you are perfectly capable to complete the easy task for you here!\n",
    "\n",
    "As in the previous exercise, we have created a function previewCourses which lets you preview the output of the spider, but you can always just explore the dictionary dc_dict too after you run the code.\n",
    "\n",
    "In this exercise, you are asked to create a CSS Locator string direct to the text of the course description. All you need to know is that from the course page, the course description text is within a paragraph p element which belongs to the class course__description (two underlines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:16:01.484292Z",
     "start_time": "2019-02-18T15:16:01.428403Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-18 17:16:01 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)\n",
      "2019-02-18 17:16:01 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.7 |Anaconda, Inc.| (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1a  20 Nov 2018), cryptography 2.4.2, Platform Windows-10-10.0.17763-SP0\n",
      "2019-02-18 17:16:01 [scrapy.crawler] INFO: Overridden settings: {}\n",
      "2019-02-18 17:16:01 [scrapy.middleware] INFO: Enabled extensions:\n",
      "['scrapy.extensions.corestats.CoreStats',\n",
      " 'scrapy.extensions.telnet.TelnetConsole',\n",
      " 'scrapy.extensions.logstats.LogStats']\n",
      "2019-02-18 17:16:01 [scrapy.middleware] INFO: Enabled downloader middlewares:\n",
      "['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n",
      " 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n",
      " 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n",
      " 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n",
      " 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n",
      " 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\n",
      " 'scrapy.downloadermiddlewares.stats.DownloaderStats']\n",
      "2019-02-18 17:16:01 [scrapy.middleware] INFO: Enabled spider middlewares:\n",
      "['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n",
      " 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n",
      " 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n",
      " 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n",
      " 'scrapy.spidermiddlewares.depth.DepthMiddleware']\n",
      "2019-02-18 17:16:01 [scrapy.middleware] INFO: Enabled item pipelines:\n",
      "[]\n",
      "2019-02-18 17:16:01 [scrapy.core.engine] INFO: Spider opened\n",
      "2019-02-18 17:16:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n",
      "2019-02-18 17:16:01 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024\n"
     ]
    },
    {
     "ename": "ReactorNotRestartable",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mReactorNotRestartable\u001b[0m                     Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-67-e471bc29ff27>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     39\u001b[0m \u001b[0mprocess\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mCrawlerProcess\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     40\u001b[0m \u001b[0mprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcrawl\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mDC_Description_Spider\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 41\u001b[1;33m \u001b[0mprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     42\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     43\u001b[0m \u001b[1;31m# Print a preview of courses\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\scrapy\\crawler.py\u001b[0m in \u001b[0;36mstart\u001b[1;34m(self, stop_after_crawl)\u001b[0m\n\u001b[0;32m    289\u001b[0m         \u001b[0mtp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madjustPoolsize\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmaxthreads\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msettings\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'REACTOR_THREADPOOL_MAXSIZE'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    290\u001b[0m         \u001b[0mreactor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0maddSystemEventTrigger\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'before'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'shutdown'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstop\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 291\u001b[1;33m         \u001b[0mreactor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# blocking call\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    292\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    293\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_get_dns_resolver\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mrun\u001b[1;34m(self, installSignalHandlers)\u001b[0m\n\u001b[0;32m   1264\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1265\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1266\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstartRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1267\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmainLoop\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1268\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mstartRunning\u001b[1;34m(self, installSignalHandlers)\u001b[0m\n\u001b[0;32m   1244\u001b[0m         \"\"\"\n\u001b[0;32m   1245\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_installSignalHandlers\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1246\u001b[1;33m         \u001b[0mReactorBase\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstartRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1247\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1248\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mstartRunning\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    752\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mReactorAlreadyRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    753\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_startedBefore\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 754\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mReactorNotRestartable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    755\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_started\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    756\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_stopped\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mReactorNotRestartable\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Import scrapy\n",
    "import scrapy\n",
    "\n",
    "# Import the CrawlerProcess: for running the spider\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "# Create the Spider class\n",
    "class DC_Description_Spider(scrapy.Spider):\n",
    "  name = \"dc_chapter_spider\"\n",
    "  # start_requests method\n",
    "  def start_requests(self):\n",
    "    yield scrapy.Request(url = url_short,\n",
    "                         callback = self.parse_front)\n",
    "  # First parsing method\n",
    "  def parse_front(self, response):\n",
    "    course_blocks = response.css('div.course-block')\n",
    "    course_links = course_blocks.xpath('./a/@href')\n",
    "    links_to_follow = course_links.extract()\n",
    "    for url in links_to_follow:\n",
    "      yield response.follow(url = url,\n",
    "                            callback = self.parse_pages)\n",
    "  # Second parsing method\n",
    "  def parse_pages(self, response):\n",
    "    # Create a SelectorList of the course titles text\n",
    "    crs_title = response.xpath('//h1[contains(@class,\"title\")]/text()')\n",
    "    # Extract the text and strip it clean\n",
    "    crs_title_ext = crs_title.extract_first().strip()\n",
    "    # Create a SelectorList of course descriptions text\n",
    "    crs_descr = response.css('p.course__description::text')\n",
    "    # Extract the text and strip it clean\n",
    "    crs_descr_ext = crs_descr.extract_first().strip()\n",
    "    # Fill in the dictionary\n",
    "    dc_dict[crs_title_ext] = crs_descr_ext\n",
    "\n",
    "# Initialize the dictionary **outside** of the Spider class\n",
    "dc_dict = dict()\n",
    "\n",
    "# Run the Spider\n",
    "process = CrawlerProcess()\n",
    "process.crawl(DC_Description_Spider)\n",
    "process.start()\n",
    "\n",
    "# Print a preview of courses\n",
    "previewCourses(dc_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capstone Crawler\n",
    "\n",
    "This exercise gives you a chance to show off what you've learned! In this exercise, you will write the parse function for a spider and then fill in a few blanks to finish off the spider. On the course directory page of DataCamp, each listed course has a title and a short course description. This spider will be used to scrape the course directory to extract the course titles and short course descriptions. You will not need to follow any links this time. Everything you need to know is:\n",
    "\n",
    "    The course titles are defined by the text within an h4 element whose class contains the string block__title (double underline).\n",
    "    The short course descriptions are defined by the text within a paragraph p element whose class contains the string block__description (double underline).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:16:13.843687Z",
     "start_time": "2019-02-18T15:16:13.837683Z"
    }
   },
   "outputs": [],
   "source": [
    "# parse method\n",
    "def parse(self, response):\n",
    "  # Extracted course titles\n",
    "  crs_titles = response.xpath('//h4[contains(@class,\"block__title\")]/text()').extract()\n",
    "  # Extracted course descriptions\n",
    "  crs_descrs = response.xpath('//p[contains(@class,\"block__description\")]/text()').extract()\n",
    "  # Fill in the dictionary\n",
    "  for crs_title, crs_descr in zip(crs_titles, crs_descrs):\n",
    "    dc_dict[crs_title] = crs_descr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-18T15:16:59.665915Z",
     "start_time": "2019-02-18T15:16:59.615079Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-18 17:16:59 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: scrapybot)\n",
      "2019-02-18 17:16:59 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.6.7 |Anaconda, Inc.| (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.1a  20 Nov 2018), cryptography 2.4.2, Platform Windows-10-10.0.17763-SP0\n",
      "2019-02-18 17:16:59 [scrapy.crawler] INFO: Overridden settings: {}\n",
      "2019-02-18 17:16:59 [scrapy.middleware] INFO: Enabled extensions:\n",
      "['scrapy.extensions.corestats.CoreStats',\n",
      " 'scrapy.extensions.telnet.TelnetConsole',\n",
      " 'scrapy.extensions.logstats.LogStats']\n",
      "2019-02-18 17:16:59 [scrapy.middleware] INFO: Enabled downloader middlewares:\n",
      "['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\n",
      " 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\n",
      " 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\n",
      " 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\n",
      " 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\n",
      " 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\n",
      " 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\n",
      " 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\n",
      " 'scrapy.downloadermiddlewares.stats.DownloaderStats']\n",
      "2019-02-18 17:16:59 [scrapy.middleware] INFO: Enabled spider middlewares:\n",
      "['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\n",
      " 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\n",
      " 'scrapy.spidermiddlewares.referer.RefererMiddleware',\n",
      " 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\n",
      " 'scrapy.spidermiddlewares.depth.DepthMiddleware']\n",
      "2019-02-18 17:16:59 [scrapy.middleware] INFO: Enabled item pipelines:\n",
      "[]\n",
      "2019-02-18 17:16:59 [scrapy.core.engine] INFO: Spider opened\n",
      "2019-02-18 17:16:59 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\n",
      "2019-02-18 17:16:59 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6025\n"
     ]
    },
    {
     "ename": "ReactorNotRestartable",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mReactorNotRestartable\u001b[0m                     Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-69-f3545dd9aae0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     25\u001b[0m \u001b[0mprocess\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mCrawlerProcess\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     26\u001b[0m \u001b[0mprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcrawl\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mYourSpider\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 27\u001b[1;33m \u001b[0mprocess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     28\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     29\u001b[0m \u001b[1;31m# Print a preview of courses\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\scrapy\\crawler.py\u001b[0m in \u001b[0;36mstart\u001b[1;34m(self, stop_after_crawl)\u001b[0m\n\u001b[0;32m    289\u001b[0m         \u001b[0mtp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0madjustPoolsize\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmaxthreads\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msettings\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'REACTOR_THREADPOOL_MAXSIZE'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    290\u001b[0m         \u001b[0mreactor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0maddSystemEventTrigger\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'before'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'shutdown'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstop\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 291\u001b[1;33m         \u001b[0mreactor\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# blocking call\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    292\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    293\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_get_dns_resolver\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mrun\u001b[1;34m(self, installSignalHandlers)\u001b[0m\n\u001b[0;32m   1264\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1265\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1266\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstartRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1267\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmainLoop\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1268\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mstartRunning\u001b[1;34m(self, installSignalHandlers)\u001b[0m\n\u001b[0;32m   1244\u001b[0m         \"\"\"\n\u001b[0;32m   1245\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_installSignalHandlers\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0minstallSignalHandlers\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1246\u001b[1;33m         \u001b[0mReactorBase\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstartRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1247\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1248\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\twisted\\internet\\base.py\u001b[0m in \u001b[0;36mstartRunning\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    752\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mReactorAlreadyRunning\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    753\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_startedBefore\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 754\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0merror\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mReactorNotRestartable\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    755\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_started\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    756\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_stopped\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mReactorNotRestartable\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Import scrapy\n",
    "import scrapy\n",
    "\n",
    "# Import the CrawlerProcess\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "\n",
    "# Create the Spider class\n",
    "class YourSpider(scrapy.Spider):\n",
    "  name = 'yourspider'\n",
    "  # start_requests method\n",
    "  def start_requests(self):\n",
    "    yield scrapy.Request(url = url_short, callback = self.parse)\n",
    "      \n",
    "  def parse(self, response):\n",
    "    # My version of the parser you wrote in the previous part\n",
    "    crs_titles = response.xpath('//h4[contains(@class,\"block__title\")]/text()').extract()\n",
    "    crs_descrs = response.xpath('//p[contains(@class,\"block__description\")]/text()').extract()\n",
    "    for crs_title, crs_descr in zip(crs_titles, crs_descrs):\n",
    "      dc_dict[crs_title] = crs_descr\n",
    "    \n",
    "# Initialize the dictionary **outside** of the Spider class\n",
    "dc_dict = dict()\n",
    "\n",
    "# Run the Spider\n",
    "process = CrawlerProcess()\n",
    "process.crawl(YourSpider)\n",
    "process.start()\n",
    "\n",
    "# Print a preview of courses\n",
    "previewCourses(dc_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}