notebooks/Lab1 - Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Goals of the Laboratory\n",
    "In this introductory laboratory:\n",
    "\n",
    "1. Acquire basic knowledge about Python and Matplotlib\n",
    "2. Gain familiarity with Juypter Notebooks\n",
    "3. Gain familiarity with the PySpark API\n",
    "\n",
    "To achieve such goals, we will go through the following steps:\n",
    "\n",
    "1. In section 1, **IPython** and **Jupyter Notebooks** are introduced to help students understand the environment used to work on projects, including those that are part of the CLOUDS course.\n",
    "\n",
    "2. In section 2, we briefly overview **Python** and its syntax. In addition, we cover **Matplotlib**, a very powerful library to plot figures in Python. Finally, we introduce **Pandas**, a python library that is very helpful when manipulating data.\n",
    "\n",
    "3. In section 3 we cover the **PySpark** APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Python + Pandas + Matplotlib: A great environment for Data Science\n",
    "\n",
    "This section aims to help students gain a basic understanding of the python programming language and some of its libraries, including `Pandas` or `Matplotlib`. \n",
    "\n",
    "When working with a small dataset (one that can comfortably fit into a single machine), Pandas and Matplotlib, together with Python are valid alternatives to other popular tools such as R and Matlab. Using such libraries allows to inherit from the simple and clear Python syntax, achieve very good performance, enjoy superior memory management,  error handling, and good package management \\[[1](http://ajminich.com/2013/06/22/9-reasons-to-switch-from-matlab-to-python/)\\].\n",
    "\n",
    "\n",
    "## 2.1. Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "#### Question 1.1\n",
    "<div class=\"alert alert-info\">\n",
    "Write a function `checkSquareNumber` to check if a integer number is a square number or not. For example, 16 and 9 are square numbers. 15 isn't square number.\n",
    "Requirements:\n",
    "\n",
    "- Input: an integer number\n",
    "\n",
    "- Output: `True` or `False`\n",
    "\n",
    "HINT: If the square root of a number is an integer number, it is a square number.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "def checkSquareNumber(x):\n",
    "    # calculate the square root of x\n",
    "    root=math.sqrt(x)\n",
    "    # return True if square root is integer, \n",
    "    if root%1==0 :\n",
    "        return True\n",
    "    # otherwise, return False\n",
    "    return False\n",
    "\n",
    "print(checkSquareNumber(16))\n",
    "print(checkSquareNumber(250))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.2\n",
    "<div class=\"alert alert-info\">\n",
    "A list `list_numbers` which contains the numbers from 1 to 9999 can be constructed from: \n",
    "\n",
    "```python\n",
    "list_numbers = range(0, 10000)\n",
    "```\n",
    "\n",
    "Extract the square numbers in `list_numbers` using function `checkSquareNumber` from question 1.1. How many elements in the extracted list ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521, 1600, 1681, 1764, 1849, 1936, 2025, 2116, 2209, 2304, 2401, 2500, 2601, 2704, 2809, 2916, 3025, 3136, 3249, 3364, 3481, 3600, 3721, 3844, 3969, 4096, 4225, 4356, 4489, 4624, 4761, 4900, 5041, 5184, 5329, 5476, 5625, 5776, 5929, 6084, 6241, 6400, 6561, 6724, 6889, 7056, 7225, 7396, 7569, 7744, 7921, 8100, 8281, 8464, 8649, 8836, 9025, 9216, 9409, 9604, 9801]\n",
      "100\n"
     ]
    }
   ],
   "source": [
    "list_numbers = range(0,10000)\n",
    "square_numbers = list(filter(lambda x: checkSquareNumber(x), list_numbers))\n",
    "print(square_numbers)\n",
    "print(len(square_numbers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1.3\n",
    "<div class=\"alert alert-info\">\n",
    "Using array slicing, select the elements of the list square_numbers, whose index is from 5 to 20 (zero-based index).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]\n"
     ]
    }
   ],
   "source": [
    "print(square_numbers[5:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Numpy\n",
    "Numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.\n",
    "### 2.2.1. Array\n",
    "A numpy array is a grid of values, all of **the same type**, and is indexed by a tuple of nonnegative integers. Thanks to the same type property, Numpy has the benefits of [locality of reference](https://en.wikipedia.org/wiki/Locality_of_reference). Besides, many other Numpy operations are implemented in C, avoiding the general cost of loops in Python, pointer indirection and per-element dynamic type checking. So, the speed of Numpy is often faster than using built-in datastructure of Python. When working with massive data with computationally expensive tasks, you should consider to use Numpy. \n",
    "\n",
    "The number of dimensions is the `rank` of the array; the `shape` of an array is a tuple of integers giving the size of the array along each dimension.\n",
    "\n",
    "We can initialize numpy arrays from nested Python lists, and access elements using square brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type of rank1_array: <class 'numpy.ndarray'>\n",
      "shape of rank1_array: (3,)\n",
      "elements in rank1_array: 1 2 3\n",
      "shape of rank2_array: (2, 3)\n",
      "1 2 4\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Create a rank 1 array\n",
    "rank1_array = np.array([1, 2, 3])\n",
    "print(\"type of rank1_array:\", type(rank1_array))\n",
    "print(\"shape of rank1_array:\", rank1_array.shape)\n",
    "print(\"elements in rank1_array:\", rank1_array[0], rank1_array[1], rank1_array[2])\n",
    "\n",
    "# Create a rank 2 array\n",
    "rank2_array = np.array([[1,2,3],[4,5,6]])\n",
    "print(\"shape of rank2_array:\", rank2_array.shape)\n",
    "print(rank2_array[0, 0], rank2_array[0, 1], rank2_array[1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.2. Array slicing\n",
    "Similar to Python lists, numpy arrays can be sliced. The different thing is that you must specify a slice for each dimension of the array because arrays may be multidimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2 3]\n",
      " [6 7]]\n",
      "value at row 0, column 1: 2\n",
      "the second row of m_array: [5 6 7 8]\n",
      "[3 8]\n"
     ]
    }
   ],
   "source": [
    "m_array = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n",
    "\n",
    "# Use slicing to pull out the subarray consisting of the first 2 rows\n",
    "# and columns 1 and 2\n",
    "b = m_array[:2, 1:3]\n",
    "print(b)\n",
    "\n",
    "# we can only use this syntax with numpy array, not python list\n",
    "print(\"value at row 0, column 1:\", m_array[0, 1])\n",
    "\n",
    "# Rank 1 view of the second row of m_array  \n",
    "print(\"the second row of m_array:\", m_array[1, :])\n",
    "\n",
    "# print element at position (0,2) and (1,3)\n",
    "print(m_array[[0,1], [2,3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.3. Boolean array indexing\n",
    "We can use boolean array indexing to check whether each element in the array satisfies a condition or use it to do filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[False False False  True]\n",
      " [ True  True  True  True]\n",
      " [ True  True  True  True]] \n",
      "\n",
      "[ 4  5  6  7  8  9 10 11 12] \n",
      "\n",
      "[ 4  5  6  7  8  9 10 11 12] \n",
      "\n",
      "[ 4  6  8 10 12]\n"
     ]
    }
   ],
   "source": [
    "m_array = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n",
    "\n",
    "# Find the elements of a that are bigger than 2\n",
    "# this returns a numpy array of Booleans of the same\n",
    "# shape as m_array, where each value of bool_idx tells\n",
    "# whether that element of a is > 3 or not\n",
    "bool_idx = (m_array > 3)\n",
    "print(bool_idx , \"\\n\")\n",
    "\n",
    "# We use boolean array indexing to construct a rank 1 array\n",
    "# consisting of the elements of a corresponding to the True values\n",
    "# of bool_idx\n",
    "print(m_array[bool_idx], \"\\n\")\n",
    "\n",
    "# We can combine two statements\n",
    "print(m_array[m_array > 3], \"\\n\")\n",
    "\n",
    "# select elements with multiple conditions\n",
    "print(m_array[(m_array > 3) & (m_array % 2 == 0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.4. Datatypes\n",
    "Remember that the elements in a numpy array have the same type. When constructing arrays, Numpy tries to guess a datatype when you create an array However, we can specify the datatype explicitly via an optional argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "int64\n",
      "float64\n"
     ]
    }
   ],
   "source": [
    "# let Numpy guess the datatype\n",
    "x1 = np.array([1, 2])\n",
    "print(x1.dtype)\n",
    "\n",
    "# force the datatype be float64\n",
    "x2 = np.array([1, 2], dtype=np.float64)\n",
    "print(x2.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.5. Array math\n",
    "Similar to Matlab or R, in Numpy, basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x + y = [[ 6.  8.]\n",
      " [10. 12.]] \n",
      "\n",
      "np.add(x, y)= [[ 6.  8.]\n",
      " [10. 12.]] \n",
      "\n",
      "x * y = [[ 5. 12.]\n",
      " [21. 32.]] \n",
      "\n",
      "np.multiply(x, y)= [[ 5. 12.]\n",
      " [21. 32.]] \n",
      "\n",
      "x*2= [[2. 4.]\n",
      " [6. 8.]] \n",
      "\n",
      "x.dot(y)= [[19. 22.]\n",
      " [43. 50.]] \n",
      "\n",
      "np.dot(x, y)= [[19. 22.]\n",
      " [43. 50.]] \n",
      "\n",
      "np.sqrt(x)= [[1.         1.41421356]\n",
      " [1.73205081 2.        ]] \n",
      "\n",
      "[[1. 3.]\n",
      " [2. 4.]]\n",
      "[1 2 3]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([[1,2],[3,4]], dtype=np.float64)\n",
    "y = np.array([[5,6],[7,8]], dtype=np.float64)\n",
    "# mathematical function is used as operator\n",
    "print(\"x + y =\", x + y, \"\\n\")\n",
    "\n",
    "# mathematical function is used as function\n",
    "print(\"np.add(x, y)=\", np.add(x, y), \"\\n\")\n",
    "\n",
    "# Unlike MATLAB, * is elementwise multiplication\n",
    "# not matrix multiplication\n",
    "print(\"x * y =\", x * y , \"\\n\")\n",
    "print(\"np.multiply(x, y)=\", np.multiply(x, y), \"\\n\")\n",
    "print(\"x*2=\", x*2, \"\\n\")\n",
    "\n",
    "# to multiply two matrices, we use dot function\n",
    "print(\"x.dot(y)=\", x.dot(y), \"\\n\")\n",
    "print(\"np.dot(x, y)=\", np.dot(x, y), \"\\n\")\n",
    "\n",
    "# Elementwise square root\n",
    "print(\"np.sqrt(x)=\", np.sqrt(x), \"\\n\")\n",
    "\n",
    "# transpose the matrix\n",
    "print(x.T)\n",
    "\n",
    "# Note that taking the transpose of a rank 1 array does nothing:\n",
    "v = np.array([1,2,3])\n",
    "print(v.T)  # Prints \"[1 2 3]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Given a 2D array:\n",
    "\n",
    "```\n",
    " 1  2  3  4\n",
    " 5  6  7  8 \n",
    " 9 10 11 12\n",
    "13 14 15 16\n",
    "```\n",
    "\n",
    "\n",
    "#### Question 2.1\n",
    "<div class=\"alert alert-info\">\n",
    "Print the all odd numbers in this array using `Boolean array indexing`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1  3  5  7  9 11 13 15]\n"
     ]
    }
   ],
   "source": [
    "array_numbers = np.array([\n",
    "        [1, 2, 3, 4],\n",
    "        [5, 6, 7, 8],\n",
    "        [9, 10, 11, 12],\n",
    "        [13, 14, 15, 16]\n",
    "    ])\n",
    "\n",
    "print(array_numbers[array_numbers%2!=0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.2\n",
    "<div class=\"alert alert-info\">\n",
    "Extract the second row and the third column in this array using `array slicing`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5 6 7 8]\n",
      "[ 3  7 11 15]\n"
     ]
    }
   ],
   "source": [
    "print(array_numbers[1, :])\n",
    "print(array_numbers[:, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.3\n",
    "<div class=\"alert alert-info\">\n",
    "Calculate the sum of diagonal elements.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n"
     ]
    }
   ],
   "source": [
    "sum = 0\n",
    "for i in range(0, array_numbers.shape[0]):\n",
    "    sum += array_numbers[i, i]\n",
    "    \n",
    "print(sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2.4\n",
    "<div class=\"alert alert-info\">\n",
    "Print elementwise multiplication of the first row and the last row using numpy's functions.\n",
    "\n",
    "Print the inner product of these two rows.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13 28 45 64]\n",
      "150\n"
     ]
    }
   ],
   "source": [
    "print(array_numbers[0,:] * array_numbers[array_numbers.shape[0]-1,:])\n",
    "print(array_numbers[0,:].dot(array_numbers[array_numbers.shape[0]-1,:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Matplotlib\n",
    "\n",
    "As its name indicates, Matplotlib is a plotting library. It provides both a very quick way to visualize data from Python and publication-quality figures in many formats. The most important function in matplotlib is `plot`, which allows you to plot 2D data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGuRJREFUeJzt3XmYFeWZ9/HvDYoLLqDgEhEakXED\n5MVWceJEVFAWF8wYE2ecoIkhEUwm0bkUfcc1MYMzMb7RDChRRjQSnWh8JY0bQTOMxq1RFESNihBh\nQBo3EBWEvuePuzrddGj6dJ+lzqn+fa6rL86pU1J31ZEfD0/V8zzm7oiISOXrlHYBIiJSGAp0EZGM\nUKCLiGSEAl1EJCMU6CIiGaFAFxHJCAW6iEhGKNBFRDJCgS4ikhHblfJgPXr08KqqqlIeUkSk4s2f\nP3+Nu/dsbb+SBnpVVRW1tbWlPKSISMUzs2W57KcuFxGRjFCgi4hkhAJdRKSY3n4b7r23JIdSoIuI\nFNLmzVBTA7ffHu+rqqCuriSHVqCLiBTC6tUweTL06wenngo//Sm4gxlceGFJSmg10M1supmtNrNF\nTbZdbWYrzGxB8jO6uGWKiJSxadOgVy+47DI44AD49a9hwYII8xLKpYV+BzByK9tvdPfByc9DhS1L\nRKSMffwx3HorLErauUccARdcAIsXw+OPw5lnwvbbl7ysVp9Dd/d5ZlZV/FJERMrc4sUwdSrMmAHr\n1sGVV8KAARHoRxyRdnV5DSy60My+DtQCF7v7BwWqSUSk/IwdCw8+CF26wFlnwYQJMHRo2lVtob03\nRacC/YDBwErghpZ2NLPxZlZrZrV1JbrTKyKSt3feabyxCXD00XD99bB8Odx1FxxzTMn7yFtj3lDs\ntnaKLpcadx/Qls+aq66udg39F5GyVV8Pc+fClCkwa1aE+YIFMGhQqmWZ2Xx3r25tv3a10M1s3yZv\nzwAWtbSviEhF+OMf4eCD4aST4Mkn4ZJLYMmS1MO8LVrtQzezXwHDgB5mthy4ChhmZoMBB5YC3y5i\njSIixVFbG8+Pjx4NffrAQQfBVVfFUyo77JB2dW2Wy1MuZ29l8+1FqEVEpPg+/TSG4k+ZAs8/D4cd\nBqNGRYD/9rdpV5cXjRQVkY5j+nTYbz8477x4lvzmm+Gpp8ru5mZ7lXQ+dBGRktq0CWbPjmfEe/WC\nffeFESPikcMvfSkzQd5ALXQRyZ5Vq+C662IY/tixcMcdsX3UqOhuOe64zIU5qIUuIlniDl//Otxz\nT7TOR4yAm26CU05Ju7KSUAtdRCrb2rVw//3x2gy6doXvfhdefx0eeyxa6Nt1jLZrxzhLEcmel1+O\nJ1V++UtYvz6eI+/fH265Je3KUqMWuohUltdeg2OPhcMPj0myvvIVeO65CPMOTi10ESl/y5bFqj/V\n1bDPPjHT4Q03wLnnwh57pF1d2VCgi0h5qq+HRx+NbpXZsyPMn3sOunWDl15Ku7qypC4XESk/M2dG\nF8ro0TGa8/LL4b770q6q7KmFLiLpc4dnn43Jsbp1gw0bYiDQj38MZ5wRc5BLq9RCF5H0rF8Pt90W\nIzmPOSZuckL0jf/Xf8FXv6owbwMFuoiU3ubN8P3vx7wq3/pWDAKaOhW+8Y34PIOjOEtBXS4iUhqf\nfw4vvBAr/3TuDK++Gn3kEybAF7+oEC8ABbqIFNf//A/84hcwbVrMPf6nP8UkWQ8/DJ3USVBIupoi\nUhxvvBELRfTuDddcEwOBHngA9torPleYF5xa6CJSOB9+CO+/H7McdukC//3fcNFF8O1vQ79+aVeX\neQp0Ecnfiy/GAKCZM2HYsBgI1KcPrFjRYSbGKge60iLSfg8+CJMnwzPPwE47wd//PVxwQePnCvOS\n0tUWkbZ5660Y9LPDDvGkygcfwM9+FvOQd+uWdnUdmu5KiEjrNm+OBZRHjYoh+b/5TWy/6KII9e99\nT2FeBloNdDObbmarzWxRk217mNkcM3sj+bV7ccsUkVRs3Aj/8i9xQ/O002IO8iuvjCXcIG586vnx\nspFLC/0OYGSzbZOAue7eH5ibvBeRLHCHpUvj9fbbw913R6Dfd19sv/pq+MIXUixQWtJqH7q7zzOz\nqmabTweGJa9nAL8HLi1gXSJSauvWRXhPmRLzj69YAbvsEjc8d9kl7eokB+3tQ9/b3Vcmr1cBexeo\nHhEptaVL4cILY16VCy6IYfk/+UnjEyoK84qR91Mu7u5m5i19bmbjgfEAvXv3zvdwIlIIGzdGi3zP\nPWMg0G23wVlnxbwqRx+tfvEK1d4W+rtmti9A8uvqlnZ092nuXu3u1T179mzn4USkIN55B664Iobj\nX3xxbBsyBFauhDvvhKFDFeYVrL2BPgsYl7weBzxYmHJEpCh+//tYKKKqCq67Do48Ev7u7xo/764H\n1bKg1S4XM/sVcQO0h5ktB64CJgP/aWbfBJYBZxWzSBFphw8+iGfDzWJE51NPwaWXwvjxEeySOebe\nYvd3wVVXV3ttbW3JjifSIT3/fDypcs89UFMDJ54Yk2bttFOM7pSKY2bz3b26tf009F8kC9avh2uv\njQBfvBi6do1l3Pr0ic81irNDUKCLVKL33oNHHoml28aNi9b33XfHIssTJ8I558Buu6VdpZSYAl2k\nUrz6avSF19TA009DfT0cdVQEeqdO8PbbMbJTOixNziVSrj79FB57LIbiA/zrv8Jll8X2f/5nePbZ\nCPYGCvMOTy10kXKyfHksDlFTA3PnRni//DIMHBiTYl13neZRkRYp0EXStHkzbNgAO+8Mc+bASSfF\n9r594fzzYcwY+Ku/atwmsg0KdJFS+/BDePTRaIk//DD84Adw+eUxSvP66+GUU+CQQzRiU9pMgS5S\nKu6xQMTvfhct8z32gNGjY+4UgF13hUsuSbdGqWgKdJFi2LAB5s2LvvB3341BPmZw4IExd8qYMdEi\n79w57UolQxToIoX02GNwyy3RH/7xx7DjjjBiRDwvvt128POfp12hZJgeWxRpr/p6qK2Fa66B1cmE\no2+8EUPvzzkn1uB87z2YNatxbnGRItL/ZSJt8ckncUOzpgYeeghWrYqulCFD4NRTY+KrCRN0Q1NS\noUAXac1bb8Fnn8Fhh0FdHXz5y7D77jByZDyRMnIk9OgR+2pwj6RIgS7S3Oefw5NPNg7wef31CPH7\n74/Jrp55JlrkCm8pMwp0EYjZCrt2jdfHHx9zh3fpAsOGRRfKKac07tvwmKFImVGgS8fkDi+91NgK\nX7w4bmzusEMszfZP/wTDh2uBZKkoCnTpeB58MKaYXbEi3h95ZIT4hg0R6GeckW59Iu2kQJdsW7as\nsRV+8cWxes8XvhDdJmPGxEjNffZJu0qRglCgS/asXw8/+lGE+KJFsa1fv5hDBaJFfv/96dUnUiQK\ndKl8H3wQq/ds3BiLPey4I9x5Jxx0ENxwQ+OMhXo2XDJOgS6V6bXXYgRmTQ384Q8x2VV1dQR6586x\nek+XLmlXKVJSeQ39N7OlZrbQzBaYWW2hihL5C599FrMUNqzec/31cOmlsG5drOLz9NPxfHgDhbl0\nQIVooR/v7msK8PuIbGnFihheX1MTYf7JJ42r91xxBfzwh9CrV9pVipQNdblI+aivj0cHd9ppy9V7\n+vSB886LvvD+/WPbAQekV6dImcp3tkUHHjOz+WY2vhAFSQfz0Ufw61/DuefG44M33hjbhw6FyZPj\nKZW3345pZ0eNihueIrJV+bbQj3X3FWa2FzDHzF5z93lNd0iCfjxA79698zycZIZ7tLjnzIm5wrt3\nj8A+8sj4fNddo49cRHKWV6C7+4rk19Vm9gBwFDCv2T7TgGkA1dXVns/xpEJt3Bir98yeHav3zJwZ\njxBWVcVgnzFj4JhjNGe4SJ7a/SfIzLoCndx9XfL6JODaglUmlW/OHLj11ljFZ926GFY/fHjj6j1T\npqRdoUim5NMk2ht4wGKwxnbATHd/pCBVSeWpr4cXX4xW+He+A3vtFdPOPv00nH12tMJPPLFxRkMR\nKbh2B7q7LwEOL2AtUmk+/TRa3zU1EeQrV0ZXyuDBcNppsXrPxIkaoSlSIuq0lNa5w5/+FM+AL1wI\nAwZEYK9eDWPHwm67wcknRyt81KhonYMG94iUmAJdtvThhzE3St++EeQnnAAvvABr1zbu893vRqD3\n6RPD7qurtXqPSBlQoHd0s2ZFP3dD6/udd2KVnieeaHwS5dBDYdCgGKE5YEC0yBscc0xalYtIMwr0\nrHOH5csjrBtCe926CHKAqVNh7lw4+GD4m7+J4G54FhzgP/4jnbpFpM0U6Fmydm2MrFy0CM4/Hzp1\nipuSU6c27rP//nHTsr4+Pp8xA7p1U3+3SAYo0CvRpk3RHdK5c7Suf/azaH0vW9a4z/DhMd/JmWdG\nN8nAgfHTrduWv1fDDUwRqXgK9HK3dm3ceFy4sPHn1Vdj9sFjj43P33or5j4ZPz5Ce9AgaJhm4YQT\n4kdEMk+BXi4+/hheeaWxn/vLX46bk4sWxaOAEGthDhwII0ZAz56x7YwztKixiAAK9NLbvBnefDP6\nr/v3hzVrYsHiJUsa9+naFQ45JAJ98OB44mTgQNhzz9TKFpHyp0AvhZtuimHxL78MixfH6jvnnQfT\np0dI//Vfx/uGfu6qqgh8gJ13jmAXEWmFAr0QPvkkukua9nPvvTfcfXd8PmVKzPs9cCBMmBC/Hn10\nfGYGd92VXu0ikhkK9LbYvDm6RhYujHlLJk6M7aeeCo8/Hq932imeKhkypPG/mz9fk1KJSNGZe+mm\nKK+urvba2gpZS3rNmugOMYPbb49pYF95JVrjEFPBrlsXQ94feSS2DxwYjwp27pxu7SKSKWY2392r\nW9tPLXSI4e6PP974hMnChbBqVYyw3G8/+PzzWEHnW99qHAJ/6KGN85eMHJlu/SIidKRAr6+HpUu3\nHAJ/xRURzvPmxZqWO+4YQX3yybF9hx3iv/3Od+JHRKSMZTPQ338/ArtXL+jXL/qwhw2LZ70b9O0b\nrfCBA2H0aHjtNTjwQHWXiEjFquxAd48+7vXr4ZprGrtLVqyIz6+9Nlrhfftu+VjgYYdFF0qD7t3j\nR0SkglXOTdGGBRaa9nMffzzcfHM8fdKjRzy/3RDagwbFkyYNIypFRCpU9m6KHndc9IFDLKwwaFD8\nQHSTrFmj7hIR6dAqJ9CnTIHdd//LBRYaKMxFpIOrnEBvmKBKRES2qlPaBYiISGEo0EVEMqKkT7mY\nWR2wrNUdt64HsKaA5aRJ51J+snIeoHMpV/mcSx93b/WRvZIGej7MrDaXx3Yqgc6l/GTlPEDnUq5K\ncS7qchERyQgFuohIRlRSoE9Lu4AC0rmUn6ycB+hcylXRz6Vi+tBFRGTbKqmFLiIi26BAFxHJiLIK\ndDObbmarzWxRC5+bmd1kZm+a2ctmNmRr+5WDHM5lmJl9ZGYLkp8rS11jLsxsfzN7wswWm9krZvaP\nW9mnIr6XHM+lUr6XHc3sOTN7KTmXa7ayzw5mdm/yvTxrZlWlr7R1OZ7LuWZW1+R7OT+NWnNhZp3N\n7EUzq9nKZ8X9Tty9bH6ALwFDgEUtfD4aeBgwYCjwbNo153Euw4CatOvM4Tz2BYYkr3cF/ggcWonf\nS47nUinfiwG7JK+3B54FhjbbZwJwS/L6a8C9adedx7mcC/w87VpzPJ+LgJlb+/+o2N9JWbXQ3X0e\n8P42djkduNPDM0A3M9u3NNW1TQ7nUhHcfaW7v5C8Xge8CuzXbLeK+F5yPJeKkFzrhiW4tk9+mj/h\ncDowI3l9H3CimVmJSsxZjudSEcysFzAGuK2FXYr6nZRVoOdgP+CdJu+XU6F/IBPHJP/MfNjMDku7\nmNYk/zz8P0QLqqmK+162cS5QId9L8k/7BcBqYI67t/i9uPsm4CNgz9JWmZsczgXgb5MuvfvMbP8S\nl5ir/wdcAtS38HlRv5NKC/QseYGYn+Fw4Gbg/6dczzaZ2S7A/cD33X1t2vXko5VzqZjvxd03u/tg\noBdwlJkNSLum9srhXH4LVLn7IGAOja3csmFmpwCr3X1+WjVUWqCvAJr+zdwr2VZx3H1twz8z3f0h\nYHsz65FyWVtlZtsTAXi3u/9mK7tUzPfS2rlU0vfSwN0/BJ4ARjb76M/fi5ltB+wOvFfa6tqmpXNx\n9/fcfUPy9jbgiFLXloMvAqeZ2VLgHuAEM/tls32K+p1UWqDPAr6ePFUxFPjI3VemXVR7mNk+DX1n\nZnYU8V2U3R+2pMbbgVfd/act7FYR30su51JB30tPM+uWvN4JGAG81my3WcC45PWZwOOe3I0rJ7mc\nS7N7MqcR9z/Kirtf5u693L2KuOH5uLuf02y3on4nZbVikZn9injKoIeZLQeuIm6Q4O63AA8RT1S8\nCXwCnJdOpa3L4VzOBC4ws03Ap8DXyvEPG9Hq+AdgYdLHCXA50Bsq7nvJ5Vwq5XvZF5hhZp2Jv3T+\n091rzOxaoNbdZxF/ed1lZm8SN+i/ll6525TLuXzPzE4DNhHncm5q1bZRKb8TDf0XEcmISutyERGR\nFijQRUQyQoEuIpIRJb0p2qNHD6+qqirlIUVEKt78+fPXeA5ripY00KuqqqitrS3lIUVEKp6ZLctl\nP3W5iIhkhAJdRCQjFOgiIhlRViNFRSSbqibNTruE1C2dPKbox1ALXUQkI1oNdNvKUmpmdrWZrWiy\nHNTo4pYpIiKtyaWFfgd/OS0nwI3uPjj5eaiwZYmISFu1GuhZWUpNRCTr8ulDvzBZDmq6mXUvWEUi\nItIu7Q30qUA/YDCwErihpR3NbLyZ1ZpZbV1dXTsPJyIirWlXoLv7u8kagPXAL4CjtrHvNHevdvfq\nnj1bnYpARETaqV2B3mw5qDOARS3tKyIipdHqwKIWllIbZmaDAQeWAt8uYo0iIpKDVgPd3c/eyubb\ni1CLiIjkQSNFRUQyQoEuIpIRCnQRkYxQoIuIZIQCXUQkIxToIiIZoUAXEckIBbqISEYo0EVEMkKB\nLiKSEQp0EZGMUKCLiGSEAl1EJCMU6CIiGaFAFxHJCAW6iEhGKNBFRDJCgS4ikhEKdBGRjFCgi4hk\nRKuBbmbTzWy1mS1qsm0PM5tjZm8kv3YvbpkiItKaXFrodwAjm22bBMx19/7A3OS9iIikqNVAd/d5\nwPvNNp8OzEhezwDGFrguERFpo/b2oe/t7iuT16uAvQtUj4iItFPeN0Xd3QFv6XMzG29mtWZWW1dX\nl+/hRESkBe0N9HfNbF+A5NfVLe3o7tPcvdrdq3v27NnOw4mISGvaG+izgHHJ63HAg4UpR0RE2iuX\nxxZ/BTwNHGRmy83sm8BkYISZvQEMT96LiEiKtmttB3c/u4WPTixwLSIikgeNFBURyYhWW+giAlWT\nZqddQqqWTh6TdgmSA7XQRUQyQoEuIpIRCnQRkYxQoIuIZIQCXUQkIxToIiIZoUAXEckIBbqISEYo\n0EVEMkKBLiKSEQp0EZGMUKCLiGSEAl1EJCMU6CIiGaFAFxHJCAW6iEhGKNBFRDJCgS4ikhEKdBGR\njMhrTVEzWwqsAzYDm9y9uhBFiYhI2xVikejj3X1NAX4fERHJg7pcREQyIt9Ad+AxM5tvZuO3toOZ\njTezWjOrraury/NwIiLSknwD/Vh3HwKMAiaa2Zea7+Du09y92t2re/bsmefhRESkJXkFuruvSH5d\nDTwAHFWIokREpO3aHehm1tXMdm14DZwELCpUYSIi0jb5POWyN/CAmTX8PjPd/ZGCVCUiIm3W7kB3\n9yXA4QWsRURE8lCI59ClAlRNmp12CalaOnlM2iWIFJ2eQxcRyQgFuohIRijQRUQyQoEuIpIRCnQR\nkYxQoIuIZIQCXUQkIxToIiIZoUAXEckIBbqISEYo0EVEMkKBLiKSEQp0EZGMUKCLiGSEAl1EJCMU\n6CIiGaFAFxHJiIpZsUgr7mjFHRHZNrXQRUQyIq9AN7ORZva6mb1pZpMKVZSIiLRduwPdzDoD/w6M\nAg4FzjazQwtVmIiItE0+LfSjgDfdfYm7bwTuAU4vTFkiItJW+QT6fsA7Td4vT7aJiEgKiv6Ui5mN\nB8Ynbz82s9eLfcwi6QGsSevgdn1aRy4YXb/86PrlJ9XrB3lfwz657JRPoK8A9m/yvleybQvuPg2Y\nlsdxyoKZ1bp7ddp1VCpdv/zo+uWno1y/fLpcngf6m1lfM+sCfA2YVZiyRESkrdrdQnf3TWZ2IfAo\n0BmY7u6vFKwyERFpk7z60N39IeChAtVS7iq+2yhlun750fXLT4e4fubuadcgIiIFoKH/IiIZUTGT\nc+XDzPYE5iZv9wE2A3XJ+6OSgVGFPN5XgauAg4Eh7r6gkL9/qaVw/X4KjAE2AG8A33D3jwp5jFJK\n4fr9GDgFcGAVMM7dVxXyGKVW6mvY5LiXApOB7u7+YTGOUUgdrsvFzK4GPnb3nxTxGIcCm4DpwIWV\nHuhNlej6nQzMTW683wB85u7/t1jHK6USXb/d3H1t8voi4AB3v7BYxyu1UlzD5DhVwBRgEDCgEgK9\nQ3e5mNmPkyd1Gt5fb2YTzWy4mT1hZg8nk4/9u5lZss8oM3vazF4ws3vNrGvz39fdF7v7H0t5Lmko\n4vV71N03JW+fIcY4ZE4Rr9/aJm93JlrqmVSsa5i4EbikFOdRKB060IkW9Dj482RjXwFmJp8dDVxA\nTDx2CHC6me0FTAJOdPchwMvAP5a66DJS1OuX/AH8BvBwsU4gZUW7fmY22cyWA2cBVxfxHNJWlGto\nZn8LLHH3RUU/gwLqEH3oLXH3N81snZkNJIbWPufuHyR/kT/j7ksBzOwe4NjkPzsU+EOyTxfgyZIX\nXiZKcP2uJP5pfU+RTiFVxbx+7j4JmGRmVwATgB8W81zSUoxraGa7EC3z4SU5iQLq0IGeuB04F6gC\nbm2yvfk/Ux0w4BF3/4eSVFYZinL9zOybwEnAiQWpsnwV+/+/u4HfkNFATxT6Gh4I9AUWJqG/D/Cy\nmR3h7nXb+O9S19G7XADuB04FBgO/a7J9qJn1Tv4Zdxbxt/gfgOPM7AAAM+tqZv1LXXCZKfj1M7Mx\nwA+A09z9s2KfQMqKcf2abjsdeK1YxZeJgl5Dd1/g7nu5e5W7VxFPCg0q9zAHtdBx98/MbB6wyt3r\nm3z0HHAL0I/4n2SWu3vScrzXYv4agMuJR+v+zMy+QtxQ6Qk8ajExUCYXBS3G9SMWTukEzE1aSE+5\n+8RinkdainT9/s3MDgTqgbeJfuTMKtI1rEgd7rHF5sysE7AAGOvuS5Jtw4nHDcemWlwF0PXLj65f\n/nQNG3XoLpfkRspbRJ/akrTrqTS6fvnR9cufruGWOnwLXUQkKzp0C11EJEsU6CIiGaFAFxHJCAW6\niEhGKNBFRDJCgS4ikhH/C1+M5kW1dXvwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.plot([1, 2, 3, 4], [1, 4, 9, 16], 'r--')\n",
    "\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.bar([1, 2, 3, 4], [1, 4, 9, 16], align='center')\n",
    "# labels of each column bar\n",
    "x_labels = [\"Type 1\", \"Type 2\", \"Type 3\", \"Type 4\"]\n",
    "# assign labels to the plot\n",
    "plt.xticks([1, 2, 3, 4], x_labels)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Question 3\n",
    "Given a list of numbers from 0 to 9999.\n",
    "\n",
    "\n",
    "#### Question 3.1\n",
    "<div class=\"alert alert-info\">\n",
    "Calculate the histogram of numbers divisible by 3, 7, 11 in the list respectively.\n",
    "\n",
    "( Or in other words, how many numbers divisible by 3, 7, 11 in the list respectively ?)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3334, 1429, 910]\n"
     ]
    }
   ],
   "source": [
    "arr = np.array(range(0,10000))\n",
    "divisors = [3, 7, 11]\n",
    "histogram = list([arr[arr%divisors[0]==0].shape[0],arr[arr%divisors[1]==0].shape[0],arr[arr%divisors[2]==0].shape[0]])\n",
    "print(histogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3.2\n",
    "<div class=\"alert alert-info\">\n",
    "Plot the histogram in a line chart.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt0lPW97/H3dyYJgQAJIeEa5Bqw\nQLkGklqB1l0r9rKxtWVxESnXdrX0tnZ7jrZn1e6eso6t7rqsrfuUcBERULe7+9Tdu7VWtJZAAohy\nkQByFUJCCIFASDLzO3/MY5oqkACTPHP5vNbKYvKbZyafWYuVT+b5Ps8z5pxDRESST8DvACIi4g8V\ngIhIklIBiIgkKRWAiEiSUgGIiCQpFYCISJJSAYiIJCkVgIhIklIBiIgkqRS/A1xNTk6OGzRokN8x\nRETiSllZWZVzLre17WK6AAYNGkRpaanfMURE4oqZHW7LdtoFJCKSpFQAIiJJSgUgIpKkVAAiIklK\nBSAikqRUACIiSUoFICKSpBKyAEJhx//57R6O11z0O4qISMxKyAI4Un2BDVuOMHvFZk6cVQmIiFxO\nQhbA4JwM1i0qpLqugTnFJVTU1vsdSUQk5iRkAQCMG5DF2oWTOFVbz5zizVSeu+R3JBGRmJKwBQAw\ncWA2axZM5p2aeuau3Mzp8yoBEZF3JXQBAEwenM2qLxRwpPoCc1eWcKauwe9IIiIxIeELAOCWoTkU\n31vAwao67llVwtkLjX5HEhHxXVIUAMCU/Fx+MW8i5RXnuXd1CbX1KgERSW5JUwAAHx3Ri8fnTmD3\niVrmr97C+UtNfkcSEfFNUhUAwMdG9uax2RPYeewsC9ZsoU4lICJJKukKAGD66D78dNZ4th2pYdHa\nrVxsCPkdSUSkwyVlAQB8ckxffjJzLFvermbxk1upb1QJiEhySdoCAJgxrj8PfW4srx04zdJ1ZSoB\nEUkqSV0AAHdPzONHnx3Dpn2VfHn9Nhqawn5HEhHpEElfAAAzJw1g+WdG8+e9p1i2YRuNIZWAiCQ+\nFYBnbuFA/vWfR/HH3RV8/entNKkERCTBpfgdIJbMv2UQjaEwP/zNHoKB13lk5lhSgupIEUlMKoD3\nWDxlCE1hx4O/20tqwHjo82MJBszvWCIiUacCuIwvTRtKUyjMw3/cRzBg/OjuMQRUAiKSYFQAV7Ds\ntnwaQ45HXywnJRhg+V2jVQIiklBUAFfxjY/l0xQO8/OXDpASMH4wYxRmKgERSQwqgKswM7718RE0\nhhwrNh0kJWh871MjVQIikhBUAK0wM+6/82YaQ2HW/PUQqcEA9995s0pAROKeCqANzCJ/+YfCkXcC\nqcHIOwOVgIjEMxVAG5kZ3//0KBpDzpsJBPjm7cP9jiUict1aPcvJzNLNbIuZvW5mu8zsX731wWZW\nYmb7zewZM0vz1jt53+/37h/U4rnu99bfMrM72utFtZdAwFh+12g+PzGPR18s52d/Lvc7kojIdWvL\naa6XgNucc2OBccB0MysCfgQ84pwbBpwBFnnbLwLOeOuPeNthZiOBWcAoYDrwuJkFo/liOkIgYDx4\n9xg+O74/D/9xH//35QN+RxIRuS6tFoCLOO99m+p9OeA24DlvfS1wl3d7hvc93v3/ZJGd5TOAp51z\nl5xzbwP7gclReRUdLOidIfzpsf148Hd7WfnKQb8jiYhcszbNALy/1MuAYcDPgQNAjXPu3c9TPAb0\n9273B44COOeazOws0NNb39ziaVs+puXPWgosBbjpppuu8eV0nGDAeGTmWELhyLWDUoMB5t8yyO9Y\nIiJt1qYrnTnnQs65cUAekb/ab26vQM65Fc65AudcQW5ubnv9mKhICQZ4dNZ4bh/Zmwee38X6ksN+\nRxIRabNrutSlc64GeAn4EJBlZu++g8gDjnu3jwMDALz7M4HTLdcv85i4lRoM8LM547nt5l5897/e\n5JmtR/yOJCLSJm05CijXzLK8252B24E9RIrgc95m84Ffebef977Hu//Pzjnnrc/yjhIaDOQDW6L1\nQvzUKSXI43MnMHV4Lvf98g3+s+yY35FERFrVlhlAX2CtNwcIAM86535tZruBp83sh8B2YJW3/Spg\nnZntB6qJHPmDc26XmT0L7AaagK845xLmQ3jTU4OsmDeRxWtL+fZzr5MSNGaMe9+IQ0QkZljkj/PY\nVFBQ4EpLS/2OcU0uNoRY8MQWtrxdzWOzJ/DJMX39jiQiScbMypxzBa1tp4+7irLOaUFWzZ/ExIE9\n+NrT2/n9myf9jiQiclkqgHaQ0SmFNQsmMyYvk69u3Mafdlf4HUlE5H1UAO2ka6cU1i6czMi+3fny\n+m289NYpvyOJiPwDFUA76p6eypMLCxnepytfXFfGK+WVfkcSEWmmAmhnmV1SWbewkCE5GSxeW8pr\nB6r8jiQiAqgAOkSPjDTWLy5kYM8uLHqilJKDp/2OJCKiAugoPbt2Yv3iIvplpbPgia2UHa72O5KI\nJDkVQAfK7daJjUuK6N09nfmrt7LjaI3fkUQkiakAOliv7ulsWFJIdkYa81aV8Maxs35HEpEkpQLw\nQd/MzmxYUkj39FTuWVXCrndUAiLS8VQAPsnr0YWnlxaRkRbknpUl7D1Z63ckEUkyKgAfDcjuwoYl\nRaSlBJhbXEJ5xTm/I4lIElEB+GxQTgYblxQRCBhzVpZwoPJ86w8SEYkCFUAMGJLblY1LCgmHHXOK\nN3Ooqs7vSCKSBFQAMWJYr25sWFJEQ1OYOcWbOVp9we9IIpLgVAAxZESfbjy1uJC6hhCzVmzmeM1F\nvyOJSAJTAcSYUf0yeWpRIbX1jcxesZkTZ1UCItI+VAAx6IN5maxbVEh1XQNzikuoqK33O5KIJCAV\nQIwaNyCLtQsncaq2njnFm6k8d8nvSCKSYFQAMWziwGzWLJjMOzWREjh9XiUgItGjAohxkwdns+oL\nBRw9c4G5K0s4U9fgdyQRSRAqgDhwy9AcVt47iYNVddyzqoSzFxr9jiQiCUAFECduzc9hxbyJlFec\n597VJdTWqwRE5MaoAOLIR0b04vG5E9h9opb5q7dwTiUgIjdABRBnPjayN4/NnsDOY2dZ+MRW6i41\n+R1JROKUCiAOTR/dh5/OGs+2IzUsWruViw0hvyOJSBxSAcSpT47py09mjmXL29UsfnIr9Y0qARG5\nNiqAODZjXH8e+txYXjtwmqXrylQCInJNVABx7u6Jefzos2PYtK+SL6/fxqUmlYCItI0KIAHMnDSA\n5Z8ZzZ/3nmLZhu00hsJ+RxKROKACSBBzCwfygxmjeGF3BV9/ejtNKgERaUWK3wEkeu790CAamsL8\n8Dd7CAZe55GZY0kJquNF5PJa/e1gZgPM7CUz221mu8zs6976983suJnt8L4+0eIx95vZfjN7y8zu\naLE+3Vvbb2b3tc9LSm6Lpwzhvjtv5r9ff4dvP7eTUNj5HUlEYlRb3gE0Af/inNtmZt2AMjN7wbvv\nEefcwy03NrORwCxgFNAP+JOZDffu/jlwO3AM2GpmzzvndkfjhcjffWnaUJpCYR7+4z5SAsaP7h5D\nIGB+xxKRGNNqATjnTgAnvNvnzGwP0P8qD5kBPO2cuwS8bWb7gcneffudcwcBzOxpb1sVQDtYdls+\njSHHoy+WkxIMsPyu0SoBEfkH17SD2MwGAeOBEm9pmZntNLPVZtbDW+sPHG3xsGPe2pXW3/szlppZ\nqZmVVlZWXks8eY9vfCyfr3x0KBu3HOGB53fhnHYHicjftbkAzKwr8J/AN5xztcC/A0OBcUTeIfxb\nNAI551Y45wqccwW5ubnReMqkZWZ86+MjWDp1COs2H+YHv96tEhCRZm06CsjMUon88l/vnPslgHOu\nosX9xcCvvW+PAwNaPDzPW+Mq69JOzIz777yZxlCYNX89RGowwP133oyZdgeJJLtWC8AivylWAXuc\ncz9psd7Xmw8AfAZ407v9PLDBzH5CZAicD2wBDMg3s8FEfvHPAuZE64XIlZkZ3/vUSEJhx4pNB0kJ\nGN++Y4RKQCTJteUdwIeBecAbZrbDW/sOMNvMxgEOOAR8EcA5t8vMniUy3G0CvuKcCwGY2TLgD0AQ\nWO2c2xXF1yJXYWZ8/9OjaAw5Hv/LAVKDAb55+/DWHygiCastRwG9SuSv9/f67VUesxxYfpn1317t\ncdK+AgFj+V2jaQqFefTFclKDxrLb8v2OJSI+0ZnASSYQMB68ewyhsIucJxAM8KVpQ/2OJSI+UAEk\noWDAeOjzY2kMOx783V5SAsbiKUP8jiUiHUwFkKSCAeORmWMJhSPXDkoNBph/yyC/Y4lIB1IBJLGU\nYIBHZ42nMbSNB57fRUrQmFs40O9YItJBdKnIJJcaDPCzOeO57eZefPe/3uSZrUf8jiQiHUQFIHRK\nCfL43AlMHZ7Lfb98g+fKjvkdSUQ6gApAAEhPDbJi3kQ+PDSHbz/3Or/aoZO0RRKdCkCapacGKb63\ngMLB2XzzmR38ZueJ1h8kInFLBSD/oHNakFXzJzFxYA++9vR2fv/mSb8jiUg7UQHI+2R0SmHNgsmM\nycvkqxu38afdFa0/SETijgpALqtrpxTWLpzMyL7d+fL6bbz01im/I4lIlKkA5Iq6p6fy5MJChvfp\nyhfXlfFKuT6gRySRqADkqjK7pLJuYSFDcjJYvLaU1w5U+R1JRKJEBSCt6pGRxvrFhQzs2YVFT5RS\ncvC035FEJApUANImPbt2Yv3iIvplpbPgia2UHa72O5KI3CAVgLRZbrdObFxSRO/u6cxfvZUdR2v8\njiQiN0AFINekV/d0NiwpJDsjjXmrSnjj2Fm/I4nIdVIByDXrm9mZjUuLyOycyj2rStj1jkpAJB6p\nAOS69M/qzMYlRWSkBblnZQl7T9b6HUlErpEKQK7bgOwubFhSRFpKgLnFJZRXnPM7kohcAxWA3JBB\nORlsXFJEIGDMLi7hQOV5vyOJSBupAOSGDcntysYlhTjnmFO8mUNVdX5HEpE2UAFIVAzr1Y0NS4po\naAozp3gzR6sv+B1JRFqhApCoGdGnG08tLqSuIcSsFZs5dkYlIBLLVAASVaP6ZfLUokJq6xuZU1zC\nibMX/Y4kIlegApCo+2BeJusWFXKmroE5xSVU1Nb7HUlELkMFIO1i3IAsnlg4iVO19cwp3kzluUt+\nRxKR91ABSLuZODCbNQsm805NpAROn1cJiMQSFYC0q8mDs1n1hQKOnrnA3JUlnKlr8DuSiHhUANLu\nbhmaw8p7J3Gwqo57VpVw9kKj35FEBBWAdJBb83NYMW8i5RXnuXd1CbX1KgERv7VaAGY2wMxeMrPd\nZrbLzL7urWeb2QtmVu7928NbNzP7qZntN7OdZjahxXPN97YvN7P57feyJBZ9ZEQvHp87gd0napm/\negvnVAIivmrLO4Am4F+ccyOBIuArZjYSuA940TmXD7zofQ9wJ5DvfS0F/h0ihQE8ABQCk4EH3i0N\nSR4fG9mbx2ZPYOexsyxYs5W6S01+RxJJWq0WgHPuhHNum3f7HLAH6A/MANZ6m60F7vJuzwCedBGb\ngSwz6wvcAbzgnKt2zp0BXgCmR/XVSFyYProPP501nu1Ha1j4xFYuNoT8jiSSlK5pBmBmg4DxQAnQ\n2zl3wrvrJNDbu90fONriYce8tSutSxL65Ji+/GTmWLYeqmbxk1upb1QJiHS0NheAmXUF/hP4hnPu\nHz79wznnABeNQGa21MxKzay0srIyGk8pMWrGuP489LmxvHbgNEvXlakERDpYmwrAzFKJ/PJf75z7\npbdc4e3awfv3lLd+HBjQ4uF53tqV1v+Bc26Fc67AOVeQm5t7La9F4tDdE/P40WfHsGlfJV9ev41L\nTSoBkY7SlqOADFgF7HHO/aTFXc8D7x7JMx/4VYv1e72jgYqAs96uoj8AHzezHt7w9+PemiS5mZMG\nsPwzo/nz3lMs27CdxlDY70giSSGlDdt8GJgHvGFmO7y17wAPAs+a2SLgMDDTu++3wCeA/cAFYAGA\nc67azP43sNXb7gfOueqovAqJe3MLBxIKO773q118beN2Hps9npSgTlMRaU8W2X0fmwoKClxpaanf\nMaQDrXzlID/8zR4+PbYfj8wcqxIQuQ5mVuacK2htu7a8AxDpMIunDKEp7Hjwd3tJCRgPf34swYD5\nHUskIakAJOZ8adpQmkJhHv7jPoIB48d3jyGgEhCJOhWAxKRlt+XTGHI8+mI5qUFj+V0fVAmIRJkK\nQGLWNz6WT1M4zM9fOkBKIMAPZowiclCaiESDCkBilpnxrY+PoCnk+MWmg6QEje99aqRKQCRKVAAS\n08yM++68mYZQmDV/PURqMMD9d96sEhCJAhWAxDyzyF/+obBjxaaDpASMb98xQiUgcoNUABIXzIzv\nf3oUjSHH4385QGowwDdvH+53LJG4pgKQuBEIGMvvGk1TKNx8dNCy2/L9jiUSt1QAElcCAePBu8cQ\nCjse/uM+UoIBvjRtqN+xROKSCkDiTjBgPPT5sTS2OGN48ZQhfscSiTsqAIlLwYDxyMyxhMJhfvib\nPaQGA8y/ZZDfsUTiigpA4lZKMMCjs8bTFNrGA8/vIiVozC0c6HcskbihSy1KXEsNBvjZnAncdnMv\nvvtfb/LM1iN+RxKJGyoAiXtpKQEenzuBqcNzue+Xb/Bc2TG/I4nEBRWAJIT01CAr5k3kw0Nz+PZz\nr/OrHe/7tFEReQ8VgCSM9NQgxfcWUDg4m28+s4Pf7DzhdySRmKYCkITSOS3IqvmTmDiwB197eju/\nf/Ok35FEYpYKQBJORqcU1iyYzJi8TL66cRt/2l3hdySRmKQCkITUtVMKaxdOZmTf7nx5/TZeeuuU\n35FEYo4KQBJW9/RUnlxYyPA+XfniujI27av0O5JITFEBSELL7JLKuoWFDM3typInS3ltf5XfkURi\nhgpAEl6PjDSeWjSZgT27sGhtKSUHT/sdSSQmqAAkKfTs2on1i4vol5XOgie2Una42u9IIr5TAUjS\nyO3WiY1LiujdPZ35q7ey/cgZvyOJ+EoFIEmlV/d0NiwpJDsjjXtXb+GNY2f9jiTiGxWAJJ2+mZ3Z\nuLSIzM6p3LOqhF3vqAQkOakAJCn1z+rMxiVFZKQFuWdlCXtP1vodSaTDqQAkaQ3I7sKGJUWkpQSY\nW1xCecU5vyOJdCgVgCS1QTkZbFxSRCBgzC4u4UDleb8jiXQYFYAkvSG5Xdm4pBBwzCnezKGqOr8j\niXQIFYAIMKxXN9YvLqKhKcyc4s0crb7gdySRdtdqAZjZajM7ZWZvtlj7vpkdN7Md3tcnWtx3v5nt\nN7O3zOyOFuvTvbX9ZnZf9F+KyI0Z0acbTy0upK4hxKwVmzl2RiUgia0t7wCeAKZfZv0R59w47+u3\nAGY2EpgFjPIe87iZBc0sCPwcuBMYCcz2thWJKaP6ZfLUokJq6xuZU1zCibMX/Y4k0m5aLQDn3Cag\nrefNzwCeds5dcs69DewHJntf+51zB51zDcDT3rYiMeeDeZmsW1TImboG5hSXUFFb73ckkXZxIzOA\nZWa209tF1MNb6w8cbbHNMW/tSusiMWncgCyeWDiJU7X1zCneTOW5S35HEom66y2AfweGAuOAE8C/\nRSuQmS01s1IzK62s1PXbxT8TB2azZsFk3qmJlMDp8yoBSSzXVQDOuQrnXMg5FwaKieziATgODGix\naZ63dqX1yz33CudcgXOuIDc393riiUTN5MHZrPpCAUfPXGDuyhLO1DX4HUkkaq6rAMysb4tvPwO8\ne4TQ88AsM+tkZoOBfGALsBXIN7PBZpZGZFD8/PXHFuk4twzNYeW9kzhYVcc9q0o4e6HR70giUdGW\nw0A3An8DRpjZMTNbBPzYzN4ws53AR4FvAjjndgHPAruB3wNf8d4pNAHLgD8Ae4BnvW1F4sKt+Tms\nmDeR8orzzFtdQm29SkDinznn/M5wRQUFBa60tNTvGCLNXtxTwZeeKmN0/0yeWDCZzM6pfkcSeR8z\nK3POFbS2nc4EFrkG//SB3jw2ewI7j51l0vI/MW9VCcWbDvLWyXPE8h9TIpejdwAi12HH0Rr++/V3\n2LSvkvJTkQvI9emezpT8HKYOz2VKfg5ZXdJ8TinJqq3vAFI6IoxIohk3IItxA7IAeKfmIq+UV7Jp\nXxV/2HWS/yg7RsBgTF4WU4fnMm14DmPzskgJ6g23xBa9AxCJoqZQmNePnWXTvko2lVfy+tEawg66\np6dwa34OU/NzmTo8l35Znf2OKgmsre8AVAAi7ajmQgOv7q+KFMK+Kk56l5UY1qsrU/NzmTYil8LB\n2aSnBn1OKolEBSASY5xzlJ86z6Z9lby8r5KSt6tpaArTKSXA5MHZTBseeXeQ36srZuZ3XIljKgCR\nGHexIUTJ26fZtK+KTeWV7PeGyX0z05t3Fd06LIfMLjrUVK6NCkAkzhyvuejtKqrk1f1VnKtvImAw\ndkBW87uDsXlZBAN6dyBXpwIQiWORYXINL79VycvlVew8VoNzkNk5lVuH5TB1eORw076ZGibL+6kA\nRBLImboWw+TySipqI1cmHd67a/PuoskaJotHBSCSoJxzvFVxrvnIoi1vV9MQigyTC4f0ZJp37sHQ\nXA2Tk5UKQCRJXGwIsfnt07z8VuTdwcHKOgD6ZaYz1ZsdfHhYjq5blERUACJJ6tiZC5Eji/ZV8tf9\nVZy71EQwYIwbkOXtLsphjIbJCU0FICI0hsLsOFrTfHTRzuNncQ6yurw7TM5l2vBcendP9zuqRJEK\nQETep7quofm6RZvKK5s/63hE725MHZ7DtOG9KBjUQ8PkOKcCEJGrcs6x9+S55jOTSw+doSEUJj01\nQNGQns1HFw3NzdAwOc6oAETkmlxoaGLzwdPN84ODVZFhcv+szs1XNb1lWA7d0zVMjnUqABG5IUer\nL/CyNzt47cBpznvD5PEtzkz+YP9MAhomxxwVgIhETWMozPYjNby87xSb9lXxxvGzAPToksqt+blM\nzc9h2vBcemmYHBNUACLSbk6fv8Sr+6u8dwhVVJ2PDJNv7tOt+d1BwaAedErRMNkPKgAR6RDhsGPP\nydrm2UHp4WoaQ47OqUGKhvz9MteDczRM7igqABHxRd2lyDD53fnBodMXAMjrERkmT83P5cPDetJN\nw+R2owIQkZhw5PQFXi73hsn7q6hrCBEMGBNv6tF8VdPR/TRMjiYVgIjEnIamMNuOnGm+qumbx2sB\nyM5IY4r3mclThufQq5uGyTdCBSAiMa/y3CVe3R8ZJL9SXknV+QYAPtC3u3dmci4FA7NJSwn4nDS+\nqABEJK6Ew47dJ2rZVF7Jy29VUnb4DE1hR5e0IB8a0rP5yqaDenbRMLkVKgARiWvnLzXxtwOnm3cX\nHfaGyQOyO0eOLMrP5ZZhOXTtlOJz0tijAhCRhHKoqo5N5X8/M/lCQ4iUgDFhYA/vQ3ByGdm3u4bJ\nqABEJIE1NIUpO3ym+VDT3Sciw+Se7w6Th+cyJT+X3G6dfE7qDxWAiCSNU+fqebU8ciLaK+VVnK6L\nDJNH9evefO7BxIE9kmaYrAIQkaQUDjt2veMNk/dVss0bJmekBfnQ0J7NZyYP7Jnhd9R2owIQEQHO\n1TfytwPemcnllRytvgjAwJ5dmj/z4ENDeybUMDlqBWBmq4FPAaecc6O9tWzgGWAQcAiY6Zw7Y5Fj\nsx4FPgFcAL7gnNvmPWY+8L+8p/2hc25ta+FUACISTc45Dp2+0PwRma8dOM3FxhCpQWPiwB7Nu4vi\nfZgczQKYCpwHnmxRAD8Gqp1zD5rZfUAP59z/NLNPAF8lUgCFwKPOuUKvMEqBAsABZcBE59yZq/1s\nFYCItKdLTSHKDp3xLlVRxR5vmJzTtRNTvWHyrfk55HSNr2FyVHcBmdkg4NctCuAt4CPOuRNm1hf4\ni3NuhJn9wru9seV27345577orf/DdleiAhCRjnSqtp5XyiOXuX51fxXV3jB5dP/uzbuLJg7sQWow\ntofJbS2A693p1ds5d8K7fRLo7d3uDxxtsd0xb+1K6yIiMaNX93TunpjH3RPzCIcdb75z1ttdVMUv\nNh3k8b8coGunFD40NHJm8rT8XG7q2cXv2NfthqcezjlnZlGbJJvZUmApwE033RStpxURuSaBgDEm\nL4sxeVksuy2f2vpGXtt/uvlktBd2VwAwqGeX5iOLiob0JCOOhsnXm7TCzPq22AV0yls/DgxosV2e\nt3acyG6glut/udwTO+dWACsgsgvoOvOJiERV9/RUpo/uw/TRfXDOcbCqrnmY/GzpMdb+7TCpQaNg\nYHbk3cHwXD7Qt1tMX7foemcADwGnWwyBs51z/8PMPgks4+9D4J865yZ7Q+AyYIL3lNuIDIGrr/Zz\nNQMQkXhwqSlE6aHIZa5f3lfJ3pPnAMjt1okp3ucl3zosh54dNEyO5lFAG4n89Z4DVAAPAP8PeBa4\nCThM5DDQau8w0J8B04kcBrrAOVfqPc9C4Dve0y53zq1pLZwKQETiUUVtvXcRuypeLa/kzIVGzOCD\n/TObh8njb8pqt2GyTgQTEYkBobDjjeNnm3cXbT9aQyjs6OYNk6eNiJx7MCA7esNkFYCISAw6e7GR\nvx2o8i5kV8XxmsiZyUNyMrzPPMihaEhPuqRd/zBZBSAiEuOccxyorGv+zIPNB09T3xgmLRjg46N6\n87M5E1p/ksto7/MARETkBpkZw3p1ZVivriy8dTD1jSG2Hqpm077KDrlyqQpARCRGpKcGmZIf+SyD\njhDb5zOLiEi7UQGIiCQpFYCISJJSAYiIJCkVgIhIklIBiIgkKRWAiEiSUgGIiCSpmL4UhJlVErna\n6PXKAaqiFEfkvfT/S9rTjfz/Guica/VsspgugBtlZqVtuR6GyPXQ/y9pTx3x/0u7gEREkpQKQEQk\nSSV6AazwO4AkNP3/kvbU7v+/EnoGICIiV5bo7wBEROQKEq4AzCzdzLaY2etmtsvM/tXvTJI4zGyE\nme1o8VVrZt/wO5fENzNbbWanzOzNFmuf936Hhc2sXY4GSrhdQGZmQIZz7ryZpQKvAl93zm32OZok\nGDMLAseBQufcjZyvIknOzKYC54EnnXOjvbUPAGHgF8C3nHNR/3zchPtEMBdptPPet6neV2K1nMSK\nfwIO6Je/3Cjn3CYzG/SetT0C236tAAAA7klEQVQQ+djI9pJwu4Ag8peZme0ATgEvOOdK/M4kCWkW\nsNHvECLXKyELwDkXcs6NA/KAyWY22u9MkljMLA34Z+A//M4icr0SsgDe5ZyrAV4CpvudRRLOncA2\n51yF30FErlfCFYCZ5ZpZlne7M3A7sNffVJKAZqPdPxLnEvEooDHAWiBIpOCedc79wN9UkkjMLAM4\nAgxxzp31O4/EPzPbCHyEyBVAK4AHgGrgMSAXqAF2OOfuiOrPTbQCEBGRtkm4XUAiItI2KgARkSSl\nAhARSVIqABGRJKUCEBFJUioAEZEkpQIQEUlSKgARkST1/wE0lVYV3BVoKgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# simple line chart\n",
    "plt.plot(histogram)\n",
    "x_indexes = [0,1,2]\n",
    "x_names = list([\"3\",\"7\",\"11\"])\n",
    "plt.xticks(x_indexes, x_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 3.3\n",
    "<div class=\"alert alert-info\">\n",
    "Plot the histogram in a bar chart.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAEKxJREFUeJzt3G+onvV9x/H3x0TbYsuM65m4JEzX\nZnO2rKk7REfHcBU1WlgsrEUf1CBCuqHQQjcW+2C2doKFtYLDylLMGsemy/oHQ5vNZVYoPvDPSZda\noxVP/TMTojlt1FZkDuN3D87P9tbmeO5zcnLu6O/9gov7ur7X77ru78VNzif39edOVSFJ6s8xo25A\nkjQaBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqdmDYAkb09yX5IfJNmd5POt/rUkjyfZ1abVrZ4kNySZ\nTPJAkjMG9rU+yaNtWn/kDkuSNJulQ4x5CfhwVb2Q5Fjg7iT/3tb9VVV9/XXjLwBWtelM4CbgzCQn\nAlcD40ABO5Nsq6pnF+JAJElzM+s3gJr2Qls8tk1v9PTYOuCWtt09wAlJTgbOB3ZU1YH2R38HsPbw\n2pckzdcw3wBIsgTYCbwXuLGq7k3yF8C1Sf4GuBPYWFUvAcuBpwY239NqM9Vf/14bgA0Axx9//B+c\ndtppcz4oSerZzp07f1JVY7ONGyoAquogsDrJCcC3krwfuAp4GjgO2AT8NXDN/Fv+xXttavtjfHy8\nJiYmDneXktSVJE8OM25OdwFV1XPAXcDaqtrXTvO8BPwjsKYN2wusHNhsRavNVJckjcAwdwGNtf/5\nk+QdwLnAj9p5fZIEuAh4sG2yDbi03Q10FvB8Ve0D7gDOS7IsyTLgvFaTJI3AMKeATga2tOsAxwBb\nq+rbSb6bZAwIsAv48zZ+O3AhMAm8CFwGUFUHknwBuL+Nu6aqDizcoUiS5iJH889Bew1AkuYuyc6q\nGp9tnE8CS1KnDABJ6pQBIEmdMgAkqVMGgCR1aqgngd+sTtn4nVG38Jb1xHUfGXULkg6T3wAkqVMG\ngCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBI\nUqcMAEnqlAEgSZ2aNQCSvD3JfUl+kGR3ks+3+qlJ7k0ymeRfkxzX6m9ry5Nt/SkD+7qq1R9Jcv6R\nOihJ0uyG+QbwEvDhqvoAsBpYm+Qs4IvA9VX1XuBZ4PI2/nLg2Va/vo0jyenAxcD7gLXAV5IsWciD\nkSQNb9YAqGkvtMVj21TAh4Gvt/oW4KI2v64t09afkyStfltVvVRVjwOTwJoFOQpJ0pwNdQ0gyZIk\nu4D9wA7gx8BzVfVyG7IHWN7mlwNPAbT1zwO/Plg/xDaD77UhyUSSiampqbkfkSRpKEMFQFUdrKrV\nwAqm/9d+2pFqqKo2VdV4VY2PjY0dqbeRpO7N6S6gqnoOuAv4Q+CEJEvbqhXA3ja/F1gJ0Nb/GvDT\nwfohtpEkLbJh7gIaS3JCm38HcC7wMNNB8Gdt2Hrg9ja/rS3T1n+3qqrVL253CZ0KrALuW6gDkSTN\nzdLZh3AysKXdsXMMsLWqvp3kIeC2JH8L/Ddwcxt/M/BPSSaBA0zf+UNV7U6yFXgIeBm4oqoOLuzh\nSJKGNWsAVNUDwAcPUX+MQ9zFU1X/C3xshn1dC1w79zYlSQvNJ4ElqVMGgCR1ygCQpE4ZAJLUKQNA\nkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSp\nUwaAJHXKAJCkThkAktQpA0CSOmUASFKnZg2AJCuT3JXkoSS7k3yq1T+XZG+SXW26cGCbq5JMJnkk\nyfkD9bWtNplk45E5JEnSMJYOMeZl4DNV9f0k7wJ2JtnR1l1fVX83ODjJ6cDFwPuA3wT+K8nvtNU3\nAucCe4D7k2yrqocW4kAkSXMzawBU1T5gX5v/eZKHgeVvsMk64Laqegl4PMkksKatm6yqxwCS3NbG\nGgCSNAJzugaQ5BTgg8C9rXRlkgeSbE6yrNWWA08NbLan1WaqS5JGYOgASPJO4BvAp6vqZ8BNwHuA\n1Ux/Q/jSQjSUZEOSiSQTU1NTC7FLSdIhDBUASY5l+o//P1fVNwGq6pmqOlhVrwBf5ZenefYCKwc2\nX9FqM9Vfo6o2VdV4VY2PjY3N9XgkSUMa5i6gADcDD1fVlwfqJw8M+yjwYJvfBlyc5G1JTgVWAfcB\n9wOrkpya5DimLxRvW5jDkCTN1TB3AX0I+ATwwyS7Wu2zwCVJVgMFPAF8EqCqdifZyvTF3ZeBK6rq\nIECSK4E7gCXA5qravYDHIkmag2HuArobyCFWbX+Dba4Frj1EffsbbSdJWjw+CSxJnTIAJKlTBoAk\ndcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKn\nDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjo1awAkWZnkriQPJdmd5FOtfmKSHUkeba/L\nWj1JbkgymeSBJGcM7Gt9G/9okvVH7rAkSbMZ5hvAy8Bnqup04CzgiiSnAxuBO6tqFXBnWwa4AFjV\npg3ATTAdGMDVwJnAGuDqV0NDkrT4Zg2AqtpXVd9v8z8HHgaWA+uALW3YFuCiNr8OuKWm3QOckORk\n4HxgR1UdqKpngR3A2gU9GknS0OZ0DSDJKcAHgXuBk6pqX1v1NHBSm18OPDWw2Z5Wm6kuSRqBoQMg\nyTuBbwCfrqqfDa6rqgJqIRpKsiHJRJKJqamphdilJOkQhgqAJMcy/cf/n6vqm638TDu1Q3vd3+p7\ngZUDm69otZnqr1FVm6pqvKrGx8bG5nIskqQ5GOYuoAA3Aw9X1ZcHVm0DXr2TZz1w+0D90nY30FnA\n8+1U0R3AeUmWtYu/57WaJGkElg4x5kPAJ4AfJtnVap8FrgO2JrkceBL4eFu3HbgQmAReBC4DqKoD\nSb4A3N/GXVNVBxbkKCRJczZrAFTV3UBmWH3OIcYXcMUM+9oMbJ5Lg5KkI8MngSWpUwaAJHXKAJCk\nThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqU\nASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqdmDYAkm5PsT/LgQO1zSfYm2dWmCwfWXZVk\nMskjSc4fqK9ttckkGxf+UCRJczHMN4CvAWsPUb++qla3aTtAktOBi4H3tW2+kmRJkiXAjcAFwOnA\nJW2sJGlEls42oKq+l+SUIfe3Dritql4CHk8yCaxp6yar6jGAJLe1sQ/NuWNJ0oI4nGsAVyZ5oJ0i\nWtZqy4GnBsbsabWZ6pKkEZlvANwEvAdYDewDvrRQDSXZkGQiycTU1NRC7VaS9DrzCoCqeqaqDlbV\nK8BX+eVpnr3AyoGhK1ptpvqh9r2pqsaranxsbGw+7UmShjCvAEhy8sDiR4FX7xDaBlyc5G1JTgVW\nAfcB9wOrkpya5DimLxRvm3/bkqTDNetF4CS3AmcD706yB7gaODvJaqCAJ4BPAlTV7iRbmb64+zJw\nRVUdbPu5ErgDWAJsrqrdC340kqShDXMX0CWHKN/8BuOvBa49RH07sH1O3akrp2z8zqhbeMt64rqP\njLoFHYV8EliSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqU\nASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHVq1gBIsjnJ\n/iQPDtROTLIjyaPtdVmrJ8kNSSaTPJDkjIFt1rfxjyZZf2QOR5I0rGG+AXwNWPu62kbgzqpaBdzZ\nlgEuAFa1aQNwE0wHBnA1cCawBrj61dCQJI3GrAFQVd8DDryuvA7Y0ua3ABcN1G+pafcAJyQ5GTgf\n2FFVB6rqWWAHvxoqkqRFNN9rACdV1b42/zRwUptfDjw1MG5Pq81UlySNyGFfBK6qAmoBegEgyYYk\nE0kmpqamFmq3kqTXmW8APNNO7dBe97f6XmDlwLgVrTZT/VdU1aaqGq+q8bGxsXm2J0mazXwDYBvw\n6p0864HbB+qXtruBzgKeb6eK7gDOS7KsXfw9r9UkSSOydLYBSW4FzgbenWQP03fzXAdsTXI58CTw\n8TZ8O3AhMAm8CFwGUFUHknwBuL+Nu6aqXn9hWZK0iGYNgKq6ZIZV5xxibAFXzLCfzcDmOXUnSTpi\nZg0ASZrJKRu/M+oW3rKeuO4jR/w9/CkISeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBI\nUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1\nygCQpE4ZAJLUqcMKgCRPJPlhkl1JJlrtxCQ7kjzaXpe1epLckGQyyQNJzliIA5Akzc9CfAP4k6pa\nXVXjbXkjcGdVrQLubMsAFwCr2rQBuGkB3luSNE9H4hTQOmBLm98CXDRQv6Wm3QOckOTkI/D+kqQh\nHG4AFPCfSXYm2dBqJ1XVvjb/NHBSm18OPDWw7Z5We40kG5JMJJmYmpo6zPYkSTNZepjb/1FV7U3y\nG8COJD8aXFlVlaTmssOq2gRsAhgfH5/TtpKk4R3WN4Cq2tte9wPfAtYAz7x6aqe97m/D9wIrBzZf\n0WqSpBGYdwAkOT7Ju16dB84DHgS2AevbsPXA7W1+G3BpuxvoLOD5gVNFkqRFdjingE4CvpXk1f38\nS1X9R5L7ga1JLgeeBD7exm8HLgQmgReByw7jvSVJh2neAVBVjwEfOET9p8A5h6gXcMV830+StLB8\nEliSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIA\nJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHVq0QMgydokjySZTLJx\nsd9fkjRtUQMgyRLgRuAC4HTgkiSnL2YPkqRpi/0NYA0wWVWPVdX/AbcB6xa5B0kSsHSR32858NTA\n8h7gzMEBSTYAG9riC0keWaTeRu3dwE9G3cSw8sVRd3BUeNN8Zn5ev9DLZ/Zbwwxa7ACYVVVtAjaN\nuo/FlmSiqsZH3YeG52f25uNn9lqLfQpoL7ByYHlFq0mSFtliB8D9wKokpyY5DrgY2LbIPUiSWORT\nQFX1cpIrgTuAJcDmqtq9mD0cxbo77fUW4Gf25uNnNiBVNeoeJEkj4JPAktQpA0CSOmUAjFiStye5\nL8kPkuxO8vlR96SZJfndJLsGpp8l+fSo+9JrJdmcZH+SBwdqH2v/xl5J4q2geA1g5JIEOL6qXkhy\nLHA38KmqumfErWkW7adN9gJnVtWTo+5Hv5Tkj4EXgFuq6v2t9nvAK8A/AH9ZVRMjbPGocNQ9CNab\nmk7gF9risW0yld8czgF+7B//o09VfS/JKa+rPQww/X8ugaeAjgpJliTZBewHdlTVvaPuSUO5GLh1\n1E1I82UAHAWq6mBVrWb6yeg1Sd4/6p70xtqDjH8K/Nuoe5HmywA4ilTVc8BdwNpR96JZXQB8v6qe\nGXUj0nwZACOWZCzJCW3+HcC5wI9G25WGcAme/tGbnHcBjViS3we2MP3TGMcAW6vqmtF2pTeS5Hjg\nf4DfrqrnR92PflWSW4Gzmf7552eAq4EDwN8DY8BzwK6qOn9UPR4NDABJ6pSngCSpUwaAJHXKAJCk\nThkAktQpA0CSOmUASFKnDABJ6tT/A/vsgWscIaAgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# char chart with x-lables\n",
    "x_indexes = range(0,3)\n",
    "x_names = list([\"3\",\"7\",\"11\"])\n",
    "plt.bar( x_indexes, histogram, align='center')\n",
    "plt.xticks(x_indexes, x_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4. Pandas\n",
    "\n",
    "Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. Indeed, it is great for data manipulation, data analysis, and data visualization.\n",
    "\n",
    "### 2.4.1. Data structures\n",
    "Pandas introduces two useful (and powerful) structures: `Series` and `DataFrame`, both of which are built on top of NumPy.\n",
    "\n",
    "#### Series\n",
    "A `Series` is a one-dimensional object similar to an array, list, or even column in a table. It assigns a *labeled index* to each item in the Series. By default, each item will receive an index label from `0` to `N-1`, where `N` is the number items of `Series`.\n",
    "\n",
    "We can create a Series by passing a list of values, and let pandas create a default integer index.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s['Col1']= 3.0 \n",
      "\n",
      "s[['Col1', 'Col3', 4.5]]= Col1    3.000000\n",
      "Col3    1.414259\n",
      "dtype: float64 \n",
      "\n",
      "Col1    3.000000\n",
      "Col3    1.414259\n",
      "5       8.000000\n",
      "dtype: float64 \n",
      "\n",
      "Col1       15.0\n",
      "Col2      -10.0\n",
      "Col3       15.0\n",
      "4.1    -65545.0\n",
      "5          15.0\n",
      "dtype: float64 \n",
      "\n",
      "Col1       150.0\n",
      "Col2      -100.0\n",
      "Col3       150.0\n",
      "4.1    -655450.0\n",
      "5          150.0\n",
      "dtype: float64 \n",
      "\n",
      "Col1    2.250000e+02\n",
      "Col2    1.000000e+02\n",
      "Col3    2.250000e+02\n",
      "4.1     4.296147e+09\n",
      "5       2.250000e+02\n",
      "dtype: float64 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "s = pd.Series({\n",
    "        'Col1': 3, 'Col2': -10, \n",
    "        'Col3': 1.414259, \n",
    "        4.1: -65545, \n",
    "        5: 8\n",
    "    })\n",
    "\n",
    "# get element which has index='Col1'\n",
    "print(\"s['Col1']=\", s['Col1'], \"\\n\")\n",
    "\n",
    "# get elements whose index is in a given list\n",
    "print(\"s[['Col1', 'Col3', 4.5]]=\", s[['Col1', 'Col3'， 4.5]], \"\\n\") #Not existing element 4.5 prints NaN\n",
    "\n",
    "# use boolean indexing for selection\n",
    "print(s[s > 0], \"\\n\")\n",
    "\n",
    "# modify elements on the fly using boolean indexing\n",
    "s[s > 0] = 15\n",
    "\n",
    "print(s, \"\\n\")\n",
    "\n",
    "# mathematical operations can be done using operators and functions.\n",
    "print(s*10,  \"\\n\")\n",
    "print(np.square(s), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DataFrame\n",
    "A DataFrame is a tabular data structure comprised of rows and columns, akin to database table, or R's data.frame object. In a loose way, we can also think of a DataFrame as a group of Series objects that share an index (the column names).\n",
    "\n",
    "We can create a DataFrame by passing a dict of objects that can be converted to series-like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>team</th>\n",
       "      <th>wins</th>\n",
       "      <th>losses</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2013</td>\n",
       "      <td>Manchester United</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014</td>\n",
       "      <td>Chelsea</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015</td>\n",
       "      <td>Asernal</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2013</td>\n",
       "      <td>Liverpool</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014</td>\n",
       "      <td>West Ham</td>\n",
       "      <td>11</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2015</td>\n",
       "      <td>Newcastle</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2013</td>\n",
       "      <td>Machester City</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2014</td>\n",
       "      <td>Tottenham</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year               team  wins  losses\n",
       "0  2013  Manchester United    11       5\n",
       "1  2014            Chelsea     8       8\n",
       "2  2015            Asernal    10       6\n",
       "3  2013          Liverpool    15       1\n",
       "4  2014           West Ham    11       5\n",
       "5  2015          Newcastle     6      10\n",
       "6  2013     Machester City    10       6\n",
       "7  2014          Tottenham     4      12"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'year': [2013, 2014, 2015, 2013, 2014, 2015, 2013, 2014],\n",
    "        'team': ['Manchester United', 'Chelsea', 'Asernal', 'Liverpool', 'West Ham', 'Newcastle', 'Machester City', 'Tottenham'],\n",
    "        'wins': [11, 8, 10, 15, 11, 6, 10, 4],\n",
    "        'losses': [5, 8, 6, 1, 5, 10, 6, 12]}\n",
    "football = pd.DataFrame(data, columns=['year', 'team', 'wins', 'losses'])\n",
    "football"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JOIN\n",
    "An important feature that Pandas supports is `JOIN`. Very often, the data comes from multiple sources, in multiple files. For example, we have 2 CSV files, one contains the information of Artists, the other contains information of Songs. If we want to query the artist name and his/her corresponding songs, we have to do joining two dataframe.\n",
    "\n",
    "Similar to SQL, in Pandas, you can do inner join, left outer join, right outer join and full outer join. Let's see a small example. Assume that we have two dataset of singers and songs. The relationship between two datasets is maintained by a constrain on `singer_code`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   singer_code singer_name\n",
      "0            0    singer_a\n",
      "1            1    singer_b\n",
      "2            2    singer_c\n",
      "3            3    singer_d\n",
      "4            4    singer_e\n",
      "\n",
      "\n",
      "   singer_code song_name\n",
      "0            2    song_f\n",
      "1            2    song_g\n",
      "2            3    song_h\n",
      "3            4    song_i\n",
      "4            5    song_j\n"
     ]
    }
   ],
   "source": [
    "singers = pd.DataFrame({'singer_code': range(5), \n",
    "                           'singer_name': ['singer_a', 'singer_b', 'singer_c', 'singer_d', 'singer_e']})\n",
    "songs = pd.DataFrame({'singer_code': [2, 2, 3, 4, 5], \n",
    "                           'song_name': ['song_f', 'song_g', 'song_h', 'song_i', 'song_j']})\n",
    "print(singers)\n",
    "print('\\n')\n",
    "print(songs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>singer_code</th>\n",
       "      <th>singer_name</th>\n",
       "      <th>song_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>singer_d</td>\n",
       "      <td>song_h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>singer_e</td>\n",
       "      <td>song_i</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   singer_code singer_name song_name\n",
       "0            2    singer_c    song_f\n",
       "1            2    singer_c    song_g\n",
       "2            3    singer_d    song_h\n",
       "3            4    singer_e    song_i"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inner join\n",
    "pd.merge(singers, songs, on='singer_code', how='inner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>singer_code</th>\n",
       "      <th>singer_name</th>\n",
       "      <th>song_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>singer_a</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>singer_b</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>singer_d</td>\n",
       "      <td>song_h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>singer_e</td>\n",
       "      <td>song_i</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   singer_code singer_name song_name\n",
       "0            0    singer_a       NaN\n",
       "1            1    singer_b       NaN\n",
       "2            2    singer_c    song_f\n",
       "3            2    singer_c    song_g\n",
       "4            3    singer_d    song_h\n",
       "5            4    singer_e    song_i"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# left join\n",
    "pd.merge(singers, songs, on='singer_code', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>singer_code</th>\n",
       "      <th>singer_name</th>\n",
       "      <th>song_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>singer_d</td>\n",
       "      <td>song_h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>singer_e</td>\n",
       "      <td>song_i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>song_j</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   singer_code singer_name song_name\n",
       "0            2    singer_c    song_f\n",
       "1            2    singer_c    song_g\n",
       "2            3    singer_d    song_h\n",
       "3            4    singer_e    song_i\n",
       "4            5         NaN    song_j"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# right join\n",
    "pd.merge(singers, songs, on='singer_code', how='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>singer_code</th>\n",
       "      <th>singer_name</th>\n",
       "      <th>song_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>singer_a</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>singer_b</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_f</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>singer_c</td>\n",
       "      <td>song_g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>singer_d</td>\n",
       "      <td>song_h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>singer_e</td>\n",
       "      <td>song_i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>song_j</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   singer_code singer_name song_name\n",
       "0            0    singer_a       NaN\n",
       "1            1    singer_b       NaN\n",
       "2            2    singer_c    song_f\n",
       "3            2    singer_c    song_g\n",
       "4            3    singer_d    song_h\n",
       "5            4    singer_e    song_i\n",
       "6            5         NaN    song_j"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# outer join (full join)\n",
    "pd.merge(singers, songs, on='singer_code', how='outer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aggregate data by each group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             singer_name  song_name\n",
      "singer_code                        \n",
      "0                      1          0\n",
      "1                      1          0\n",
      "2                      2          2\n",
      "3                      1          1\n",
      "4                      1          1\n",
      "\n",
      "\n",
      "singer_code\n",
      "0    0\n",
      "1    0\n",
      "2    2\n",
      "3    1\n",
      "4    1\n",
      "Name: song_name, dtype: int64\n",
      "\n",
      "\n",
      "singer_code\n",
      "0    0\n",
      "1    0\n",
      "3    1\n",
      "4    1\n",
      "2    2\n",
      "Name: song_name, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "data = pd.merge(singers, songs, on='singer_code', how='left')\n",
    "\n",
    "# count the values of each column in group\n",
    "print(data.groupby('singer_code').count())\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# count only song_name\n",
    "print(data.groupby('singer_code').song_name.count())\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# count song name but ignore duplication, and order the result\n",
    "print(data.groupby('singer_code').song_name.nunique().sort_values(ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "We have two datasets about music: [song](https://github.com/michiard/AML-COURSE/blob/master/data/song.tsv) and [album](https://github.com/michiard/AML-COURSE/blob/master/data/album.tsv).\n",
    "\n",
    "In the following questions, you **have to** use Pandas to load data and write code to answer these questions.\n",
    "\n",
    "\n",
    "#### Question 4.1\n",
    "<div class=\"alert alert-info\">\n",
    "Load both dataset into two dataframes and print the information of each dataframe\n",
    "\n",
    "**HINT**: \n",
    "\n",
    "- You can click button `Raw` on the github page of each dataset and copy the URL of the raw file.\n",
    "- The dataset can be load by using function `read_table`. For example: `df = pd.read_table(raw_url, sep='\\t')`\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            Singer                  Song  Album Length\n",
      "0  Michael Jackson                 2 bad    1.0   4:07\n",
      "1  Michael Jackson           Unbreakable    2.0   6:26\n",
      "2  Michael Jackson       Don't Walk Away    2.0   4:25\n",
      "3  Michael Jackson         Break of Dawn    2.0   5:33\n",
      "4     Taylor Swift          All Too Well    5.0   5:29\n",
      "5     Taylor Swift             Bad Blood    7.0   3:19\n",
      "6     Taylor Swift      Back to December    6.0   4:54\n",
      "7  Michael Jackson          Human Nature    4.0   4:06\n",
      "8  Michael Jackson          Baby Be Mine    4.0   4:20\n",
      "9  Michael Jackson  What More Can I Give    NaN   3:36 \n",
      "\n",
      "   Album code                                 Album name  Year\n",
      "0           1  HIStory: Past, Present and Future, Book I  1995\n",
      "1           2                                 Invincible  2001\n",
      "2           3                                        Bad  1986\n",
      "3           4                                   Thriller  1982\n",
      "4           5                                        Red  2012\n",
      "5           6                                  Speak Now  2010\n",
      "6           7                                       1989  2014\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "songdb_url = 'https://raw.githubusercontent.com/DistributedSystemsGroup/Algorithmic-Machine-Learning/master/data/song.tsv'\n",
    "albumdb_url = 'https://raw.githubusercontent.com/DistributedSystemsGroup/Algorithmic-Machine-Learning/master/data/album.tsv'\n",
    "song_df = pd.read_table(songdb_url, sep='\\t')\n",
    "album_df = pd.read_table(albumdb_url, sep='\\t')\n",
    "\n",
    "print(song_df,\"\\n\")\n",
    "print(album_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.2\n",
    "<div class=\"alert alert-info\">\n",
    "How many albums in this datasets ?\n",
    "\n",
    "How many songs in this datasets ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of albums: 7\n",
      "number of songs: 10\n"
     ]
    }
   ],
   "source": [
    "print(\"number of albums:\", album_df[\"Album name\"].count())\n",
    "print(\"number of songs:\", song_df[\"Song\"].count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.3\n",
    "<div class=\"alert alert-info\">\n",
    "How many distinct singers in this dataset ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number distinct singers: 2\n"
     ]
    }
   ],
   "source": [
    "print(\"number distinct singers:\", len(song_df.groupby(\"Singer\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.4\n",
    "<div class=\"alert alert-info\">\n",
    "Is there any song that doesn't belong to any album ?\n",
    "\n",
    "Is there any album that has no song ?\n",
    "\n",
    "**HINT**: \n",
    "\n",
    "- To join two datasets on different key names, we use `left_on=` and `right_on=` instead of `on=`.\n",
    "- Funtion `notnull` and `isnull` help determining the value of a column is missing or not. For example:\n",
    "`df['song'].isnull()`.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fulldf = pd.merge(song_df, album_df, how='outer', left_on='Album', right_on='Album code')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Singer</th>\n",
       "      <th>Song</th>\n",
       "      <th>Album</th>\n",
       "      <th>Length</th>\n",
       "      <th>Album code</th>\n",
       "      <th>Album name</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Michael Jackson</td>\n",
       "      <td>What More Can I Give</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3:36</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Singer                  Song Album Length  Album code Album name  \\\n",
       "9  Michael Jackson  What More Can I Give   NaN   3:36         NaN        NaN   \n",
       "\n",
       "   Year  \n",
       "9   NaN  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fulldf[fulldf['Song'].notnull() & fulldf['Album'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Singer</th>\n",
       "      <th>Song</th>\n",
       "      <th>Album</th>\n",
       "      <th>Length</th>\n",
       "      <th>Album code</th>\n",
       "      <th>Album name</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Bad</td>\n",
       "      <td>1986.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Singer Song Album Length  Album code Album name    Year\n",
       "10    NaN  NaN   NaN    NaN         3.0        Bad  1986.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fulldf[fulldf['Song'].isnull() & fulldf['Album code'].notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 4.5\n",
    "<div class=\"alert alert-info\">\n",
    "How many songs in each albums of Michael Jackson ?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Album\n",
       "1.0    1\n",
       "2.0    3\n",
       "4.0    2\n",
       "Name: Song, dtype: int64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fulldf[fulldf['Singer']=='Michael Jackson'].groupby(\"Album\").Song.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. PySpark\n",
    "\n",
    "Spark is an alternative framework to Hadoop MapReduce, designed to make it easier and quicker to build and run distributed data manipulation algorithms. Spark comes with a library for machine learning (MLLib) and graph algorithms, and also supports real-time streaming and SQL syntax, via Spark Streaming and SparkSQL, respectively. Spark exposes the Spark programming model to Java, Scala, or Python. In Python, we use the PySpark API to interact with Spark.\n",
    "\n",
    "As discussed in the CLOUDS lectures, every Spark application has a Spark driver. It is the program that declares the transformations and actions on RDDs of data and submits such requests to the cluster manager. Actually, the driver is the program that creates the `SparkContext`, connecting to a given cluster manager such as  Spark Master, YARN or others. The executors run user code, run computations and can cache data for your application. The `SparkContext` will create a job that is broken into stages. The stages are broken into tasks which are scheduled by the SparkContext on an executor.\n",
    "\n",
    "When starting PySpark with command `pyspark` or using a well configured notebook (such as this one), `SparkContext` is created automatically in variable `sc`. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://10.0.0.25:4040\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v2.2.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>spark://spark-master0-2919-prod2:7077</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>pyspark-shell</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        "
      ],
      "text/plain": [
       "<SparkContext master=spark://spark-master0-2919-prod2:7077 appName=pyspark-shell>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PySpark uses PySpark RDDs which  are just RDDs of Python objects: like Python lists, they can store objects with mixed types (actually all the objects are instances of `PyObject`).\n",
    "\n",
    "When PySpark is started, it also starts a JVM, which is accessible through a socket. PySpark uses `Py4J` to handle this communication. The JVM works as the actual Spark driver, and loads a `JavaSparkContext` that communicates with the Spark executors across the cluster. Python API calls to the Spark Context object are then **translated into Java API calls** to the JavaSparkContext. For example, the implementation of PySpark's `sc.textFile()` dispatches a call to the `.textFile` method of the `JavaSparkContext`, which ultimately communicates with the Spark executor JVMs to load the text data from HDFS. \n",
    "\n",
    "![](http://i.imgur.com/YlI8AqEl.png)\n",
    "\n",
    "The Spark executors on the cluster start a Python interpreter for each core, with which they communicate data through a pipe when they need to execute user-code. A Python RDD in the local PySpark client corresponds to a `PythonRDD` object in the local JVM. The data associated with the RDD actually lives in the Spark JVMs as Java objects. For example, running `sc.textFile()` in the Python interpreter will call the `JavaSparkContexts` `textFile` method, which loads the data as Java String objects in the cluster.\n",
    "\n",
    "\n",
    "When an API call is made on the `PythonRDD`, any associated code (e.g., Python lambda function) **is serialized and distributed to the executors**. The data is then converted from Java objects to a Python-compatible representation (e.g., pickle objects) and streamed to executor-associated Python interpreters through a pipe. Any necessary Python processing is executed in the interpreter, and the resulting data is stored back as an RDD (as pickle objects by default) in the JVMs. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with HDFS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 6 items\r\n",
      "-rw-r--r--   3 user14 supergroup     244438 2017-03-29 14:30 airports.csv\r\n",
      "-rw-r--r--   3 user14 supergroup      43758 2017-03-29 14:31 carriers.csv\r\n",
      "drwxr-xr-x   - user14 supergroup          0 2017-03-22 15:10 lastfm_model.spark\r\n",
      "drwxr-xr-x   - user14 supergroup          0 2017-03-22 15:10 lastfm_modelWUA.spark\r\n",
      "drwxr-xr-x   - user14 supergroup          0 2017-03-20 15:17 lastfm_modelWithoutUnknowns.spark\r\n",
      "-rw-r--r--   3 user14 supergroup     428796 2017-03-29 14:50 plane-data.csv\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs dfs -ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 items\r\n",
      "-rw-r--r--   3 root supergroup 3000000000 2016-03-30 08:36 /datasets/gutenberg/gutenberg_small.txt\r\n",
      "-rw-r--r--   3 root supergroup   31457280 2017-02-28 13:34 /datasets/gutenberg/gutenberg_tiny.txt\r\n"
     ]
    }
   ],
   "source": [
    "! hdfs dfs -ls /datasets/gutenberg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.1\n",
    "<div class=\"alert alert-info\">\n",
    "Using the hdfs command, inspect the file we've used before, located in /datasets/gutenberg/gutenberg_tiny.txt\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many HDFS blocks constitute our file?\n",
    "    * The file is constituted of just one block. In fact the file is of ~32MB and in a HDFS a typical dimension of a block is bigger than 64MB.\n",
    "\n",
    "* Where are they located?\n",
    "* What is the replication factor of our file?\n",
    "    * The only block is replicated 3 times, as expected, in the locations [10.0.0.5:50010, 10.0.0.11:50010, 10.0.0.14:50010]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to namenode via http://hdfs-namenode.zoe:50070\r\n",
      "FSCK started by user14 (auth:SIMPLE) from /10.0.0.25 for path /datasets/gutenberg/gutenberg_tiny.txt at Fri Oct 27 11:54:06 UTC 2017\r\n",
      "/datasets/gutenberg/gutenberg_tiny.txt 31457280 bytes, 1 block(s):  OK\r\n",
      "0. BP-1699471697-10.0.0.2-1454672112057:blk_1074378409_637598 len=31457280 repl=3 [10.0.0.5:50010, 10.0.0.11:50010, 10.0.0.14:50010]\r\n",
      "\r\n",
      "Status: HEALTHY\r\n",
      " Total size:\t31457280 B\r\n",
      " Total dirs:\t0\r\n",
      " Total files:\t1\r\n",
      " Total symlinks:\t\t0\r\n",
      " Total blocks (validated):\t1 (avg. block size 31457280 B)\r\n",
      " Minimally replicated blocks:\t1 (100.0 %)\r\n",
      " Over-replicated blocks:\t0 (0.0 %)\r\n",
      " Under-replicated blocks:\t0 (0.0 %)\r\n",
      " Mis-replicated blocks:\t\t0 (0.0 %)\r\n",
      " Default replication factor:\t3\r\n",
      " Average block replication:\t3.0\r\n",
      " Corrupt blocks:\t\t0\r\n",
      " Missing replicas:\t\t0 (0.0 %)\r\n",
      " Number of data-nodes:\t\t11\r\n",
      " Number of racks:\t\t1\r\n",
      "FSCK ended at Fri Oct 27 11:54:06 UTC 2017 in 1 milliseconds\r\n",
      "\r\n",
      "\r\n",
      "The filesystem under path '/datasets/gutenberg/gutenberg_tiny.txt' is HEALTHY\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs fsck /datasets/gutenberg/gutenberg_tiny.txt -files -blocks -locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of lines in the input file is: 745613\n"
     ]
    }
   ],
   "source": [
    "# each line is stored as an element in 'input_file' - a PythonRDD.\n",
    "input_file = sc.textFile(\"/datasets/gutenberg/gutenberg_tiny.txt\")\n",
    "num_lines = input_file.count()\n",
    "print(\"The number of lines in the input file is:\", num_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.2\n",
    "<div class=\"alert alert-info\">\n",
    "Using the Apache Spark Web UI, as explained above, answer the following questions:\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many tasks were launched?\n",
    "    * 2 tasks\n",
    "* What is the duration of each task?\n",
    "    * 2 seconds each\n",
    "* How many input bytes where processed by each task? How does this relate to the input file size?\n",
    "    * Each task processed about half of the file (15 MB and 15.1 MB).\n",
    "* How many workers in total does your cluster have? How many workers were involved in your \"line count\" job?\n",
    "    * Our cluster has 4 workers, but only 2 of them have been involved in the line count job, one for each task.\n",
    "* Given that we know how many HDFS blocks compose our input file (from Question 3.1), explain the number of tasks your job is broken into.\n",
    "    * The job is broken in two tasks because the block is broken in two partition, since it uses the default partition setting which equals = 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3.1. Wordcount example\n",
    "In the example below, we are interested in the top-10 words in terms of frequency of occurrence. To do so, we use a small text file as an input, and we wish to plot the term frequency of such top-10 words using Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = (\n",
    "            # read the text file\n",
    "            sc.textFile(\"/datasets/gutenberg/gutenberg_small.txt\").repartition(16)\n",
    "            \n",
    "            # construct words from lines\n",
    "            .flatMap(lambda line: line.split())\n",
    "            \n",
    "            # map each word to (word, 1)\n",
    "            .map(lambda x: (x, 1))\n",
    "    \n",
    "            # reduce by key: accumulate sum the freq of the same word\n",
    "            .reduceByKey(lambda freq1, freq2: freq1 + freq2)\n",
    "            \n",
    "            # swap (word, freq) to (freq, word)\n",
    "            .map(lambda x: (x[1], x[0]))\n",
    "    \n",
    "            # sort result by key DESC\n",
    "            .sortByKey(False)\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3.3\n",
    "<div class=\"alert alert-info\">\n",
    "Using both the terminal and the Apache Spark Web UI, as explained above, answer the following questions:\n",
    "</div>\n",
    "\n",
    "* What is the size of the input file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to namenode via http://hdfs-namenode.zoe:50070\r\n",
      " Total size:\t3000000000 B\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs fsck /datasets/gutenberg/gutenberg_small.txt -files -blocks -locations | grep \"Total size\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many blocks the input file is broken into? What is the block size?\n",
    "    * The blocks size is 134217728 B except for the last block that is smaller: 47209984 B."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting to namenode via http://hdfs-namenode.zoe:50070\r\n",
      " Total blocks (validated):\t23 (avg. block size 130434782 B)\r\n"
     ]
    }
   ],
   "source": [
    "!hdfs fsck /datasets/gutenberg/gutenberg_small.txt -files -blocks -locations  | grep \"Total blocks\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* How many tasks execute in parallel?\n",
    "   * 4 tasks are executed in parallel during the first two stages. 3 tasks are executed in parallel during the last stage.\n",
    "* How many jobs were launched upon the execution of the code above? Why (note that there are no actions!)?\n",
    "   * 2 Jobs have been launched. Even if the sorting operation is listed among the trasformation, it still fire a job cluster. This is a known behaviour, explained by the fact that the operation depends on a partitioner which samples input RDD to compute partition boundaries.\n",
    "* What does a **\"skipped\"** stage mean?\n",
    "   * It means that the RDD produced by the stage is already cached, as result of previous executions, and thus it can be skipped.\n",
    "* What is the number of shuffled bytes? How does this compare to the number of input bytes?\n",
    "   * The number of shuffled bytes between the first two stages is 2.1 GB while the number of shuffled bytes between the last two stages is 369.6 MB. The total number of shuffled bytes is ~2.47 GB with respect to the 2.8 GB of input data. The main reduction in the volume of data happens between the last two stages thanks to the reduceByKey transformation.\n",
    "* Do you think Spark is doing a good job in balancing the load across the workers? What can go wrong with load balancing?\n",
    "   * Yes, in fact during all the stages the number of data processed by each worker is almost the same. If the load balance is not good, it may happens that a machine works more than the others that will idle in the meanwhile, wasting resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(29556020, 'the'), (18043957, 'of'), (14914748, 'and'), (12892052, 'to'), (9779124, 'a'), (8568854, 'in'), (5143447, 'that'), (4824456, 'was'), (4156700, 'I'), (3915358, 'with')]\n"
     ]
    }
   ],
   "source": [
    "# the top-10 words are collected and sent back to the driver:\n",
    "top10 = words.take(10)\n",
    "print(top10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The action `collect` gathers all elements of the RDD (that reside on multiple machines) to the driver (which is running in a single machine), and cast it as a list.\n",
    "\n",
    "**ATTENTION** collecting an RDD in the driver can be problematic: indeed, an RDD can be very big in size (this is why they are distributed across several machines in the first place!) and thus it could deplete the RAM available in the machine running the driver!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# collect results from executors to the driver\n",
    "# results = words.collect()\n",
    "# If you want to have a look at the results, don't print them all, for otherwise the notebook size will be too big"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that there are two kinds of functions in Spark: **transformations** and **actions**. All functions `map`, `flatMap`, `reduceByKey`, `sortByKey` are transformation functions. They are not executed right away when called. Indeed, Spark is lazily evaluated, so nothing gets executed unless the driver invokes actions such as `count`, `take`, `collect`...\n",
    "\n",
    "RDD transformations allow us to create dependencies between RDDs. Each RDD in the lineage chain (string of dependencies) has a function for calculating its data and has a pointer (dependency) to its parent RDD. Every time we use an RDD, dependencies are computed again from the beginning, which can be costly. Fortunately, we can use the function `cache` to instruct Spark to checkpoint in RAM (but eventually also on disk) a particular RDD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAEGCAYAAAB4lx7eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHS5JREFUeJzt3XmUVOWd//H3F7oVwQVxQRFkFxUXcGFR0dJxAaI4LjNE\nx5gQFZcgRo+/o+NJYk/iGZM5MT/cwTUaN9xHDWqcaAtuiLLIIiqyCBoxE0BAYlj8zh9PNbRld1d1\nU1VP3arP65w6favrVt2PgN/+9nOf+1xzd0REJLlaxQ4gIiJbR4VcRCThVMhFRBJOhVxEJOFUyEVE\nEk6FXEQk4YpWyM3sHjNbbmazc9j3d2Y2I/34wMxWFiOjiEgSWbHmkZvZEGAtcL+7H9iM940B+rn7\n+QULJyKSYEXryN19CvCtztrMeprZ82b2jplNNrM+Dbz1bODhooQUEUmgqsjHvwO40N0XmNlA4Dbg\nn+peNLOuQDfg5TjxRERKX7RCbmbbA4OBx8ys7tvbZOz2feAx1zoCIiKNitmRtwJWuXv/JvYZCVxS\npDwiIonU5Bi5mbUxs6lmNtPM5phZTSP73WRmH5nZLDNrqjBv5u6rgUVmdmb6M8zMDqr3mfsCO7v7\nWzn/14iIVKAmC7m7fw0c6+79gH7A0PRY9mZmNhzo5e69gdHA7Q19lpk9DLwB9DGzpWY2Cvg34Dwz\nmwnMAUbUe8tIdJJTRCSrrEMr7r4uvbkNUA18k7HLCOC+9L5Tzay9mXV09+UZn3NWI4cY1shx/yNb\nNhERyWH6oZm1SnfMy4E/ufu0jF32ApbWe74M6Jy/iCIi0pSshdzdv0kPrXQGBppZ3wZ2s8y35SOc\niIhkl/OsFXf/0sxeAYYCc+u99CnQpd7zzunvfYuZqbiLiLSAu2c2y9+SbdbKrmbWPr29HXAC8H7G\nbs8A56b3GUSYUricBrh7ST2uvfba6BmUqbxyKZMy5fuRi2wd+Z7AfWbWmlD0J7r7JDO7MF2YJ6Sf\nDzezBcBXwKicjiwiInnRZCF399nAIQ18f0LG8zF5ziUiIjkq6nrkn39ezKNll0qlYkf4DmXKXSnm\nUqbcKFN+FXMZW7/2WqempiiHExEpC2aGZznZWdRC3rGjs2QJbLttUQ4pIpJ4uRTyog6tHHQQTJxY\nzCOKiJS/ohbyyy6DG2+EIv0SICJSEYpayIcNgzVr4PXXi3lUEZHyVtRC3qoVXHopjBtXzKOKiJS3\nop7sdHfWrIGuXWHGjPBVREQaV3InOwF22AF+9CO49dZiH1lEpDwVvSMHWLgQBgyAJUugXbuiHF5E\nJJFKsiMH6NEDjjoK/vCHGEcXESkvUQo5bJmK+E3m/YZERKRZohXyVAq22QZeeilWAhGR8hCtkJtt\n6cpFRKTlopzsrPP112EK4uTJ0KdPUWKIiCRKyZ7srNOmDVxwAdx8c8wUIiLJFrUjB/jsMzjggDAl\nsX37okQREUmMku/IATp1gqFD4e67YycREUmm6B05wNtvw8iRsGABtG5dlDgiIomQiI4cwlWee+wB\nzzwTO4mISPKURCEHTUUUEWmpkinkZ5wRhlZmzoydREQkWUqmkFdXwyWXqCsXEWmukjjZWed//xd6\n94YPPoDddy9KLBGRkpaYk511dt0VzjwTJkyInUREJDlKqiMHmD0bTjoJFi8Oi2qJiFSyxHXkAAce\nCPvtB489FjuJiEgylFwhhzAVcdw4KNIvCyIiidZkITezLmb2ipnNNbM5Zja2gX1SZvalmc1IP362\ntaG+9z1YsQLefHNrP0lEpPxVZXl9A3C5u880s+2Bd83sJXd/P2O/V919RL5CtW4NY8eGqYhHHJGv\nTxURKU9NduTu/rm7z0xvrwXeBzo1sGuTA/EtMWpUuHvQ0qX5/mQRkfKS8xi5mXUD+gNTM15y4Agz\nm2Vmk8xs/3wE23FH+MEP4Lbb8vFpIiLlK6fph+lhlVrgOnd/OuO1HYBN7r7OzIYBN7r7Pg18Rk7T\nD+tbsAAGD4YlS6Bt22a9VUSkLOQy/TDbGDlmVg08ATyQWcQB3H1Nve3nzew2M+vg7isy962pqdm8\nnUqlSKVSTR67Vy8YNAgeeABGj86WVEQk+Wpra6mtrW3We5rsyM3MgPuAv7n75Y3s0xH4wt3dzAYA\nj7p7twb2a3ZHDvDnP4fpiLNnhxs2i4hUknx05EcC5wDvmdmM9PeuAfYGcPcJwJnAxWa2EVgHfH+r\nUmc47rhQwP/8Zzj++Hx+sohIeSi5S/Qbcued4aYTzz6b51AiIiUul448EYV83Tro2hXeeCOsjigi\nUikSudZKQ9q2hfPPh5tvjp1ERKT0JKIjh3Bh0MEHw6JFsNNOeQwmIlLCyqYjB+jSBU48Ee69N3YS\nEZHSkpiOHMIiWuecAx9+GNZjEREpd2XVkUO4OGiXXeCPf4ydRESkdCSqkJuFi4N0g2YRkS0SNbQC\nsH49dOsGL7wABx209blEREpZ2Q2tQLiP5yWXwE03xU4iIlIaEteRA/z1r7DPPvDRR7Drrnn5SBGR\nklSWHTnAbrvBaafBHXfETiIiEl8iO3KAWbPCvT0XLYLq6rx9rIhISSnbjhzCVZ69esHjj8dOIiIS\nV2ILOcBPf6qpiCIiiS7kp5wCX3wBUzPvIioiUkESXchbt4YxY9SVi0hlS+zJzjqrVkGPHuFWcHvt\nlfePFxGJqqxPdtZp3x7OPhtuvz12EhGROBLfkQN88AEMGQJLlsB22xXkECIiUVRERw7Qpw8cfjg8\n9FDsJCIixVcWhRy2rIpYpF8wRERKRtkU8hNOgI0bobY2dhIRkeIqm0JuBmPHaiqiiFSesjjZWeer\nr6Br13CBUM+eBT2UiEhRVMzJzjrt2sF558Ett8ROIiJSPGXVkQN88gn07w+LF8MOOxT8cCIiBVVx\nHTnA3nvDccfB738fO4mISHGUXUcO8NprMGpUuFCoVdn9qBKRSlKRHTnAkUfCjjvCpEmxk4iIFF6T\nhdzMupjZK2Y218zmmNnYRva7ycw+MrNZZta/MFFzZ7blAiERkXKXrSPfAFzu7n2BQcBPzGy/+juY\n2XCgl7v3BkYDJbF81ciRMGcOzJ0bO4mISGE1Wcjd/XN3n5neXgu8D3TK2G0EcF96n6lAezPrWICs\nzbLttnDRRXDTTbGTiIgUVs5j5GbWDegPZN6PZy9gab3ny4DOWxssHy66CB59FFasiJ1ERKRwqnLZ\nycy2Bx4HLkt35t/ZJeN5g9NTampqNm+nUilSqVROIVuqY0cYMQLuvBOuuqqghxIRyYva2lpqm7lo\nVNbph2ZWDTwHPO/u4xp4fTxQ6+6PpJ/PB45x9+UZ+xVt+mF906fDqafCwoVQXV30w4uIbJWtnn5o\nZgbcDcxrqIinPQOcm95/ELAqs4jHdMgh0L07PPVU7CQiIoXRZEduZkcBk4H32DJccg2wN4C7T0jv\ndwswFPgKGOXu0xv4rCgdOcATT8Dvfgevvx7l8CIiLZZLR16WV3Zm2rgxrIb4xBNw2GFRIoiItEjF\nXtmZqaoKxozRBUIiUp4qoiMHWLkSevSAefNgzz2jxRARaRZ15PXsvDOcdRbcXhLXnYqI5E/FdOQA\n8+fDMcfAkiXQpk3UKCIiOVFHnmHffcNNJx55JHYSEZH8qahCDltWRYz8y4GISN5UXCE/6SRYtw6m\nTImdREQkPyqukLdqBWPHwrjGrlMVEUmYijrZWWftWujaFd59F7p1i51GRKRxOtnZiO23hwsugHPO\ngQ8/jJ1GRGTrVGQhB7juOjjjDDjiCPjVr+Af/4idSESkZSq2kFdVweWXh2Vup02Dfv1g8uTYqURE\nmq8ix8gzuYdlbseOhaFD4b/+Czp0iJ1KRERj5Dkzg9NPD+uwtG0L++8PDzygueYikgzqyBvw9ttw\n4YWwyy5hbZbevWMnEpFKpY68hQYMCOPmw4fD4MHhxOj69bFTiYg0TIW8EVVVcMUVYa751KnhZKiu\nBhWRUqShlRy4w5NPhnVahg2D3/xGJ0NFpDg0tJInZmHO+dy5Yfnbvn3hwQd1MlRESoM68haYOhVG\nj4aOHcPJ0J49YycSkXKljrxABg6Ed96BE08M2//5nzoZKiLxqJC3UHU1XHllKOivvx5uWPHaa7FT\niUgl0tBKHrjD44/DT38KJ58Mv/51uEeoiMjW0tBKkZjBv/xLuDK0qiqcDH34YZ0MFZHiUEdeAG+9\nFU6GduoEt90GPXrETiQiSaWOPJJBg8KFRMcdF64S/fWvYcOG2KlEpFypIy+wRYvgkktg2TKYMCGs\nfy4ikqtcOnIV8iJwh8ceC+ufjxgB118P7dvHTiUiSZCXoRUzu8fMlpvZ7EZeT5nZl2Y2I/34WUsD\nlysz+Nd/DVeGmoVlcidO1MlQEcmPrB25mQ0B1gL3u/uBDbyeAq5w9xFZPqdiO/JMb74ZToZ27hxO\nhnbvHjuRiJSqvHTk7j4FWJntWM0JVukGDw63mEul4PDDwx2JdDJURFoqH7NWHDjCzGaZ2SQz2z8P\nn1n2qqvhqqvCTSxefhkOPTRMWxQRaa6qPHzGdKCLu68zs2HA08A+De1YU1OzeTuVSpFKpfJw+GTr\n0QOefz6MmZ9+Opx0Urh3aP/+sZOJSAy1tbXU1tY26z05zVoxs27Asw2NkTew7yLgUHdfkfF9jZFn\nsXIl3Hor3HlnWFnxootg5Eho1y52MhGJpSgXBJlZRzOz9PYAwg+HFVneJg3YeWf42c9g4UK49lp4\n+mnYe2+49FKYMyd2OhEpVbnMWnkYOAbYFVgOXAtUA7j7BDP7CXAxsBFYR5jB8p3RXnXkLfPJJ3DX\nXXD33WF2y0UXwZlnhhtciEj50wVBZWTDBnjuuXB16LvvwrnnhimMffrETiYihaS1VspIdTWcdhq8\n8EK4Q9E228DRR4f1XB59VDe2EKlk6sgTbP16eOqp0KXPmwejRsEFF2i1RZFyoo68zG2zTZjV8vLL\n8OqrobAPHAhDh4YCv3Fj7IQiUgzqyMvM11+HuxWNHx9WXjzvPDj//DD7RUSSRx15BWrTBs45J9w/\n9MUXw9z0fv3glFPgj3+ETZtiJxSRfFNHXgG++ipcOTp+PCxfHsbRzzsP9twzdjIRyUYduQDhytAf\n/zis6/LUU7B0aVhK94wz4KWX4JtvYicUka2hjrxCrV4NDz0UuvS1a8Oc9FGjYLfdYicTkfp0QZBk\n5R7mpU+YELr1YcPC1aNHHx1ugiEicamQS7OsXAl/+EPo0t3hwgvDFaQdOsROJlK5VMilRdzDrJfx\n48NMl9NPh+uug06dYicTqTw62SktYgZDhsCDD8KCBbDHHnDwwWGJXU1fFCk96sglJ3PnhrHz9evD\neHq/frETiVQGdeSSN337hmUARo+GE0+EK68Ms11EJD4VcslZq1bhQqI5c+CLL0Jxf/bZ2KlEREMr\n0mIvvxyGWw44AG66CTp3jp1IpPxoaEUK6rjj4L334KCDwpj5uHFacVEkBnXkkhcffBC689Wrw8nQ\nww6LnUikPKgjl6Lp0ycMtVx2GZx8cvi6enXsVCKVQYVc8sYsXAk6d26Y0dK3Lzz5ZLjASEQKR0Mr\nUjCTJ4fhlp494ZZboGvX2IlEkkdDKxLV0UfDzJkwaBAceij89rewYUPsVCLlRx25FMWCBXDJJeHG\nFhMmhOIuItmpI5eS0atXuPXc1VfDaaeFor5qVexUIuVBhVyKxgzOOgvmzQt3Jdp//3ALOv2iJrJ1\nNLQi0bzxRljzfK+94LbboEeP2IlESo+GVqSkHXEETJ8Oxx4LAwbA9deH1RVFpHnUkUtJWLQIxoyB\nJUvCDS2OOip2IpHSkJeO3MzuMbPlZja7iX1uMrOPzGyWmfVvSVipbN27w3PPQU0NjBwJF1wAK1bE\nTiWSDLkMrdwLDG3sRTMbDvRy997AaOD2PGWTCmMGZ54ZToa2aROuDH3gAZ0MFckmayF39ynAyiZ2\nGQHcl953KtDezDrmJ55Uop12gptvhv/+b7jhBjjhBPjww9ipREpXPk527gUsrfd8GaCVqWWrDRgA\n06bB8OHhxOgvfwn/+EfsVCKlpypPn5M5EN/gL8M1NTWbt1OpFKlUKk+Hl3JVVQVXXBGGXMaODTeB\nHj8e9E9HylVtbS21tbXNek9Os1bMrBvwrLsf2MBr44Fad38k/Xw+cIy7L8/YT7NWZKs9/XQo6Mce\nCz/8YbjUv23b2KlECqdY88ifAc5NH3AQsCqziIvkyz//c1gmt2dP+MUvYPfdw7DL1VfDpEnw5Zex\nE4oUX9aO3MweBo4BdgWWA9cC1QDuPiG9zy2EmS1fAaPcfXoDn6OOXPLu73+HqVPDkrmTJ4ft3r3D\nyotDhoTH7rvHTinScrl05LogSMrK+vXhatG6wv7667DHHqGw1z26dImdUiR3KuRS8TZtgtmztxT2\nyZOhXbtvF/ZevcIcdpFSpEIuksE93Ci6rqi/+ips3Pjtwt63L7TSKkRSIlTIRbJwD+u71O/Y//a3\nsNZLXWHv3z9MgxSJQYVcpAX+8heYMmVLYV+8GAYP3nICdcCAsISASDGokIvkwYoV4aRpXWGfOxcO\nOWRLxz54MOywQ+yUUq5UyEUKYO1aePPNLYX93XfD3Y6OPhouuiicPBXJFxVykSL4+uuwJsyLL4bl\nA375S7j4Ys2EkfxQIRcpsvnz4dxzYeed4e67obOWj5OtpFu9iRTZvvuGe5EedVQYR3/wQa2nLoWn\njlykQKZPhx/8APbbD26/HXbbLXYiSSJ15CIRHXJIOBHavXtYfveZZ2InknKljlykCKZMCcvuplIw\nbhzsuGPsRJIU6shFSsSQITBrFlRXw0EHwSuvxE4k5UQduUiRTZoEo0eHux5dfz1st13sRFLK1JGL\nlKDhw+G992D58jCOPm1a7ESSdOrIRSKaODHcuu7CC+HnPw9DLyL1qSMXKXEjR8LMmWF2y6BBYR0X\nkeZSIReJbM894bnnwjotqRT89rfhhhgiudLQikgJWbgQfvSjsP3730OPHjHTSCnQ0IpIwvToEaYm\nnnoqDBwId9yhS/wlO3XkIiVq7tywAFfHjnDXXdCpU+xEEoM6cpEE69sX3noLDj883G5u4sTYiaRU\nqSMXSYBp00J3fvDBcOutsMsusRNJsagjFykThx8eVlPcc89wif+kSbETSSlRRy6SMK+8AqNGwYkn\nwg036H6h5U4duUgZOvbYcIn/pk1hqGXKlNiJJDZ15CIJ9uyz4fL+s8+G666DNm1iJ5J8U0cuUuZO\nOSV054sXw6GHhnF0qTxZC7mZDTWz+Wb2kZld1cDrKTP70sxmpB8/K0xUEWnIrrvCY4/BNdfA0KHw\nq1/Bxo2xU0kxNTm0YmatgQ+A44FPgWnAWe7+fr19UsAV7j6iyQNpaEWk4JYtg/POg5Ur4f77w82g\nJdnyMbQyAFjg7ovdfQPwCHBqQ8dqYUYRyaPOneGFF8KslqOOghtvhG++iZ1KCq0qy+t7AUvrPV8G\nDMzYx4EjzGwWoWu/0t3n5S+iiDSHGVx8MZxwQrhP6M9/Dh06QPv24bHzztm36563axc+T0pbtkKe\ny1jIdKCLu68zs2HA08A+W51MRLZKr17w2mvw5ZewalUYblm16rvbH33U8PdXroT162GnnXIr/g1t\nb7tt7D+FypCtkH8KdKn3vAuhK9/M3dfU237ezG4zsw7uviLzw2pqajZvp1IpUqlUCyKLSK7MthTX\nbt2a//7167P/IFiypPEfBK1bN1zkd9sNuneHnj3Dio/du+vepXVqa2upra1t1nuyneysIpzs/Cfg\nM+BtvnuysyPwhbu7mQ0AHnX3bg18lk52ilQQd/j73xsu8MuXw6JFYf31hQvD9MkOHUJRryvu9R97\n7FG5Qzy5nOxssiN3941mNgZ4EWgN3O3u75vZhenXJwBnAheb2UZgHfD9vKQXkUQzg7ZtwyPbEryb\nNsFnn20p7AsXhpO2H38ctr/6KnTt9Yt7XcHv1k3dvK7sFJGSt2bNlg6+rrjXPZYsCatBNtbNd+yY\n7G4+l45chVxEEm3TJvj0028X9/oFf9267xb3+t18qS9roEIuIhVv9erGu/lPPglXxvboAV27hhk6\n228fVpSs/7Wx7bZtoVWBFzpRIRcRaUJdN//xx6Gor14Na9eGx5o1TW+vWQNffx2KeS5FP9cfENtu\n++2hIBVyEZEC2rQpDN1kK/qZPwCaen3jxm8X+PnzVchFRBJlw4ZvF/e+fVXIRUQSTeuRi4hUABVy\nEZGEUyEXEUk4FXIRkYRTIRcRSTgVchGRhFMhFxFJOBVyEZGEUyEXEUk4FXIRkYRTIRcRSTgVchGR\nhFMhFxFJOBVyEZGEUyEXEUk4FXIRkYRTIRcRSTgVchGRhFMhFxFJOBVyEZGEUyEXEUk4FXIRkYTL\nWsjNbKiZzTezj8zsqkb2uSn9+iwz65//mCIi0pgmC7mZtQZuAYYC+wNnmdl+GfsMB3q5e29gNHB7\ngbLmXW1tbewI36FMuSvFXMqUG2XKr2wd+QBggbsvdvcNwCPAqRn7jADuA3D3qUB7M+uY96QFUIp/\nccqUu1LMpUy5Uab8ylbI9wKW1nu+LP29bPt03vpoIiKSi2yF3HP8HGvh+0REZCuZe+M118wGATXu\nPjT9/N+Bb9z9N/X2GQ/Uuvsj6efzgWPcfXnGZ6m4i4i0gLtnNsvfUpXl/e8Avc2sG/AZMBI4K2Of\nZ4AxwCPpwr8qs4jnEkRERFqmyULu7hvNbAzwItAauNvd3zezC9OvT3D3SWY23MwWAF8BowqeWkRE\nNmtyaEVEREpfwa/szOWComIzs3vMbLmZzY6dpY6ZdTGzV8xsrpnNMbOxJZCpjZlNNbOZ6Uw1sTPV\nMbPWZjbDzJ6NnQXAzBab2XvpTG/HzlPHzNqb2eNm9r6ZzUsPf8bM0yf9Z1T3+LJE/q1fnv43PtvM\nHjKzbUsg02XpPHPM7LImd3b3gj0IwzELgG5ANTAT2K+Qx8wx1xCgPzA7dpZ6mfYA+qW3twc+KJE/\nq7bpr1XAW8DA2JnSea4AHgSeiZ0lnWcR0CF2jgZy3Qf8uN7f4U6xM9XL1gr4C9Alco69gIXAtunn\nE4EfRs50ADAbaJOuoy8BPRvbv9AdeS4XFBWdu08BVsbOUZ+7f+7uM9Pba4H3gU5xU4G7r0tvbkP4\nYfxNxDgAmFlnYDhwF9+d+hpTKWXBzHYChrj7PRDOebn7l5Fj1Xc88LG7L826Z+FVAW3NrApoC3wa\nOc++wFR3/9rdNwGvAqc3tnOhC3kuFxRJhvQsof7A1LhJwMxamdlMYDnwJ3efFjsT8P+B/0cJ/FCp\nx4H/MbN3zOyC2GHSugN/NbN7zWy6md1pZm1jh6rn+8BDsUO4+6fADcAnhNl5q9z9f+KmYg4wxMw6\npP/OvkcTF1oWupDrTGozmdn2wOPAZenOPCp3/8bd+xH+EQ00s74x85jZycAX7j6D0uqAj3T3/sAw\n4CdmNiR2IEKXeQhwm7sfQphVdnXcSIGZbQOcAjxWAll2Jiw10o3wW/D2ZvZvMTO5+3zgN8CfgOeB\nGTTRuBS6kH8KdKn3vAuhK5cGmFk18ATwgLs/HTtPfelfyV8hLKAW0xHACDNbBDwMHGdm90fOhLv/\nJf31r8BThGHF2JYBy+r9FvU4obCXgmHAu+k/r9iOBxa5+9/cfSPwJOHfWVTufo+7H+buxwCrCOfN\nGlToQr75gqL0T+CRhAuIJIOZGXA3MM/dx8XOA2Bmu5pZ+/T2dsAJhLH7aNz9Gnfv4u7dCb+av+zu\n58bMZGZtzWyH9HY74ETCiaqo3P1zYKmZ7ZP+1vHA3IiR6juL8IO4FCwBBpnZdun/D48H5kXOhJnt\nnv66N3AaTQxDZbuyc6t4IxcUFfKYuTCzh4FjgF3MbCnwC3e/N3KsI4FzgPfMbEb6e//u7i9EzLQn\ncF96OeNWwER3nxQxT0NKYfiuI/BUqAFUAQ+6+5/iRtrsUuDBdCP1MSVwwV76h93xQEmcS3D3t83s\ncWA6sDH99Y64qQB43Mx2ATYAl7j76sZ21AVBIiIJp1u9iYgknAq5iEjCqZCLiCScCrmISMKpkIuI\nJJwKuYhIwqmQi4gknAq5iEjC/R+7ckvjzdSswQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8440036898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot word frequency\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the frequencies from the result\n",
    "frequencies = [x[0] for x in top10]\n",
    "\n",
    "# plot the frequencies\n",
    "plt.plot(frequencies)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Night flights example\n",
    "We have a CSV file which contains the information about flights that took place in the US in 1994.\n",
    "The data in this file has 29 columns such as `year`, `month`, `day_of_month`, `scheduled_departure_time`,..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Year,Month,DayofMonth,DayOfWeek,DepTime,CRSDepTime,ArrTime,CRSArrTime,UniqueCarrier,FlightNum,TailNum,ActualElapsedTime,CRSElapsedTime,AirTime,ArrDelay,DepDelay,Origin,Dest,Distance,TaxiIn,TaxiOut,Cancelled,CancellationCode,Diverted,CarrierDelay,WeatherDelay,NASDelay,SecurityDelay,LateAircraftDelay\r\n",
      "1994,1,7,5,858,900,954,1003,US,227,NA,56,63,NA,-9,-2,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,8,6,859,900,952,1003,US,227,NA,53,63,NA,-11,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,10,1,935,900,1023,1003,US,227,NA,48,63,NA,20,35,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,11,2,903,900,1131,1003,US,227,NA,148,63,NA,88,3,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,12,3,933,900,1024,1003,US,227,NA,51,63,NA,21,33,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,13,4,NA,900,NA,1003,US,227,NA,NA,63,NA,NA,NA,CLT,ORF,290,NA,NA,1,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,14,5,903,900,1005,1003,US,227,NA,62,63,NA,2,3,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,15,6,859,900,1004,1003,US,227,NA,65,63,NA,1,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "1994,1,17,1,859,900,955,1003,US,227,NA,56,63,NA,-8,-1,CLT,ORF,290,NA,NA,0,NA,0,NA,NA,NA,NA,NA\r\n",
      "cat: Unable to write to output stream.\r\n"
     ]
    }
   ],
   "source": [
    "#  quick look on the data\n",
    "! hdfs dfs -cat /datasets/airline/1994.csv | head -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Questions:\n",
    "\n",
    "- How many night flights do we have in our data ?\n",
    "- How many night flights per unique carrier ? Plot the top-5 of them, in terms of volume"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read the data\n",
    "data = sc.textFile('/datasets/airline/1994.csv')\n",
    "\n",
    "# extract information about scheduled departure time and carrier\n",
    "def extract_CRSDepTime_Carier(line):\n",
    "    cols = line.split(\",\")\n",
    "    return (int(cols[5]), cols[8])\n",
    "\n",
    "header = data.first()\n",
    "\n",
    "# remove header\n",
    "data_without_header = data.filter(lambda line: line != header)\n",
    "\n",
    "# create a new RDD with only scheduled departure time and carrier information\n",
    "# cache it for later usage\n",
    "newdata = (\n",
    "            data_without_header\n",
    "               .map(extract_CRSDepTime_Carier)\n",
    "               .cache()\n",
    "          )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(2030, 'US'), (2030, 'US'), (2030, 'US')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select the night flights\n",
    "night_flights = newdata.filter(lambda f: f[0] > 1800).cache()\n",
    "night_flights.take(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1078203\n"
     ]
    }
   ],
   "source": [
    "# filter and count the night flights\n",
    "num_night_flights = night_flights.count()\n",
    "print(num_night_flights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `groupBy` to cluster flights which belong to the same carrier into a group. In this example, to select the top-5 carriers, we use a method based on the function `takeOrder`. This function takes the top-`k` objects ordered by an index: the trick is that we instruct it to use the cumulative counts as the key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('DL', 208224), ('US', 170397), ('AA', 142832), ('WN', 124024), ('UA', 113640)]\n"
     ]
    }
   ],
   "source": [
    "# group by carrier\n",
    "night_flights_by_carrier = night_flights.groupBy(lambda x: x[1]).mapValues(lambda flights: len(flights))\n",
    "\n",
    "# take top 5 carriers\n",
    "top5_carriers = night_flights_by_carrier.takeOrdered(5, key=lambda x: -x[1])\n",
    "\n",
    "print(top5_carriers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEACAYAAABcXmojAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE7xJREFUeJzt3XuM3eV95/H3p3HCshsUxzRLuJjLqs5u3CBx2UIu1XYq\nBDhqFxwtG0y3weq6F8WbEmW30kKqjY2ymy2VUkpahUpbEi7bUFBpE1CJsSGZ3WQTcIKgMXUIRsG7\ntgGDTEwSrdTFzXf/OI8fjgZ7ZhjP+MyM3y/paJ7z/V3meWZGv8/5/Z7zO5OqQpIkgJ8adQckSfOH\noSBJ6gwFSVJnKEiSOkNBktQZCpKkbtJQSLI8yVeT/G2SJ5Jc0+obk+xO8lh7vH9om+uS7EjyZJJL\nhurnJ9nWlt00VD8uyV2t/nCSM4aWrU3yVHtcPbtDlyRNlMnuU0jyduDtVfV4kjcDjwKrgQ8CP6qq\nP5iw/krgC8DPAacCDwIrqqqSbAU+UlVbk9wPfKaqNiVZD7yrqtYnuRL4QFWtSbIM+BZwftv9o8D5\nVbV/FscvSRoy6ZlCVT1fVY+39o+B7zI42APkEJtcDtxZVa9U1U7gaeDCJCcDJ1TV1rbe7QzCBeAy\n4LbWvge4qLUvBTZX1f4WBFuAVa9zfJKk12HacwpJzgTOBR5upd9O8jdJbkmytNVOAXYPbbabQYhM\nrO/h1XA5FdgFUFUHgJeTnDjJviRJc2RaodAuHf0F8NF2xnAzcBZwDvAc8Ok566Ek6ahZMtUKSd7I\n4LLOf6+qLwJU1QtDy/8UuK893QMsH9r8NAav8Pe09sT6wW1OB55NsgR4S1XtS7IHGBvaZjnwlUP0\nzw9vkqQZqKrXTANM9e6jALcA26vqD4fqJw+t9gFgW2vfC6xJ8qYkZwErgK1V9TzwwyQXtn1+CPjS\n0DZrW/sK4KHW3gxckmRpkrcCFwMPHGZgi/axYcOGkffB8Tk2x7f4Hocz1ZnC+4BfBb6T5LFW+zhw\nVZJzgAKeAX6rHZy3J7kb2A4cANbXq999PXArcDxwf1VtavVbgDuS7AD2AWvavl5K8kkG70ACuL58\n55EkzalJQ6Gqvs6hzya+PMk2nwI+dYj6o8DZh6j/HYO3uB5qX58HPj9ZHyVJs8c7mue5sbGxUXdh\nTi3m8S3msYHjW6wmvXltIUhSC30MknS0JaFe70SzJOnYYihIkjpDQZLUGQqSpM5QkCR1hoIkqTMU\nJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkK\nkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEndklF3YDFI\nMuouzEhVjboLkuYZQ2HWLLQD7MIMMklzy8tHkqRu0lBIsjzJV5P8bZInklzT6suSbEnyVJLNSZYO\nbXNdkh1JnkxyyVD9/CTb2rKbhurHJbmr1R9OcsbQsrXtezyV5OrZHbokaaKpzhReAT5WVT8LvBv4\nd0neCVwLbKmqdwAPteckWQlcCawEVgGfzasX3G8G1lXVCmBFklWtvg7Y1+o3Aje0fS0DPgFc0B4b\nhsNHkjT7Jg2Fqnq+qh5v7R8D3wVOBS4Dbmur3Qasbu3LgTur6pWq2gk8DVyY5GTghKra2ta7fWib\n4X3dA1zU2pcCm6tqf1XtB7YwCBpJ0hyZ9pxCkjOBc4FHgJOqam9btBc4qbVPAXYPbbabQYhMrO9p\nddrXXQBVdQB4OcmJk+xLkjRHpvXuoyRvZvAq/qNV9aPht2BWVSUZ6VtvNm7c2NtjY2OMjY2NrC+S\nNB+Nj48zPj4+5XpThkKSNzIIhDuq6outvDfJ26vq+XZp6IVW3wMsH9r8NAav8Pe09sT6wW1OB55N\nsgR4S1XtS7IHGBvaZjnwlUP1cTgUJEmvNfEF8/XXX3/I9aZ691GAW4DtVfWHQ4vuBda29lrgi0P1\nNUnelOQsYAWwtaqeB36Y5MK2zw8BXzrEvq5gMHENsBm4JMnSJG8FLgYemKy/kqQjk8nuak3y88D/\nBL7Dq3dnXQdsBe5m8Ap/J/DBNhlMko8D/xY4wOBy0wOtfj5wK3A8cH9VHXx763HAHQzmK/YBa9ok\nNUl+Dfh4+77/uaoOTkgP97FGfWfuIOcW3s1ro/65SRqdJFTVa+5inTQUFgJDYaYMBelYdrhQ8I5m\nSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaC\nJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqVsy6g5o/ksy6i7MSFWNugvSgmMo\naJoW2gF2YQaZNGpePpIkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMU\nJEndlKGQ5HNJ9ibZNlTbmGR3ksfa4/1Dy65LsiPJk0kuGaqfn2RbW3bTUP24JHe1+sNJzhhatjbJ\nU+1x9ewMWZJ0ONM5U/g8sGpCrYA/qKpz2+PLAElWAlcCK9s2n82rH7F5M7CuqlYAK5Ic3Oc6YF+r\n3wjc0Pa1DPgEcEF7bEiydIbjlCRNw5ShUFVfA35wiEWH+hjKy4E7q+qVqtoJPA1cmORk4ISq2trW\nux1Y3dqXAbe19j3ARa19KbC5qvZX1X5gC68NJ0nSLDqSOYXfTvI3SW4ZegV/CrB7aJ3dwKmHqO9p\nddrXXQBVdQB4OcmJk+xLkjRHZhoKNwNnAecAzwGfnrUeSZJGZkb/ZKeqXjjYTvKnwH3t6R5g+dCq\npzF4hb+ntSfWD25zOvBskiXAW6pqX5I9wNjQNsuBrxyqPxs3buztsbExxsbGDrWaJB2zxsfHGR8f\nn3K9TOdfFiY5E7ivqs5uz0+uquda+2PAz1XVr7SJ5i8wmBg+FXgQ+JmqqiSPANcAW4G/Bj5TVZuS\nrAfOrqoPJ1kDrK6qNW2i+dvAeQzmLx4FzmvzC8N9q1H/28XBXPrC+89k0/25LfbxSceiJFTVa+aG\npzxTSHIn8AvATyfZBWwAxpKcw+BI8QzwWwBVtT3J3cB24ACwfuiIvR64FTgeuL+qNrX6LcAdSXYA\n+4A1bV8vJfkk8K223vUTA0GSNLumdaYwn3mmMFOeKUjHssOdKXhHsySpMxQkSd2M3n0kLSav3nS/\nsHh5THPBUJCAhThnIs0FLx9JkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNB\nktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnq/H8K0iK3EP+JkP9AaHQMBemYsJAOsgsvxBYTLx9JkjpD\nQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6rxPQdKC5s15s8tQkLQIzN+D7GvN7xDz8pEkqTMUJEmd\noSBJ6gwFSVJnKEiSOkNBktRNGQpJPpdkb5JtQ7VlSbYkeSrJ5iRLh5Zdl2RHkieTXDJUPz/Jtrbs\npqH6cUnuavWHk5wxtGxt+x5PJbl6doYsSTqc6ZwpfB5YNaF2LbClqt4BPNSek2QlcCWwsm3z2bx6\nZ8nNwLqqWgGsSHJwn+uAfa1+I3BD29cy4BPABe2xYTh8JEmzb8pQqKqvAT+YUL4MuK21bwNWt/bl\nwJ1V9UpV7QSeBi5McjJwQlVtbevdPrTN8L7uAS5q7UuBzVW1v6r2A1t4bThJkmbRTOcUTqqqva29\nFziptU8Bdg+ttxs49RD1Pa1O+7oLoKoOAC8nOXGSfUmS5sgRTzTX4EM8FtI95pKkw5jpZx/tTfL2\nqnq+XRp6odX3AMuH1juNwSv8Pa09sX5wm9OBZ5MsAd5SVfuS7AHGhrZZDnzlUJ3ZuHFjb4+NjTE2\nNnao1STpmDU+Ps74+PiU62U6n9aX5Ezgvqo6uz3/fQaTwzckuRZYWlXXtonmLzCYGD4VeBD4maqq\nJI8A1wBbgb8GPlNVm5KsB86uqg8nWQOsrqo1baL528B5DD5B6lHgvDa/MNy3GvUnDg7m0hfayVKm\n/UmNjm8+Wszjm/7YYPGPb856kVBVr/l0vinPFJLcCfwC8NNJdjF4R9DvAXcnWQfsBD4IUFXbk9wN\nbAcOAOuHjtjrgVuB44H7q2pTq98C3JFkB7APWNP29VKSTwLfautdPzEQJEmza1pnCvOZZwoztZhf\naYLjG1pzwY3PM4Wj0ovDnCl4R7MkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSp\nMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLU\nGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnq\nDAVJUndEoZBkZ5LvJHksydZWW5ZkS5KnkmxOsnRo/euS7EjyZJJLhurnJ9nWlt00VD8uyV2t/nCS\nM46kv5KkyR3pmUIBY1V1blVd0GrXAluq6h3AQ+05SVYCVwIrgVXAZ5OkbXMzsK6qVgArkqxq9XXA\nvla/EbjhCPsrSZrEbFw+yoTnlwG3tfZtwOrWvhy4s6peqaqdwNPAhUlOBk6oqq1tvduHthne1z3A\nRbPQX0nSYczGmcKDSb6d5Dda7aSq2tvae4GTWvsUYPfQtruBUw9R39PqtK+7AKrqAPBykmVH2GdJ\n0mEsOcLt31dVzyV5G7AlyZPDC6uqktQRfo8pbdy4sbfHxsYYGxub628pSQvK+Pg44+PjU66Xqtk5\nZifZAPwY+A0G8wzPt0tDX62qf5bkWoCq+r22/iZgA/C/2zrvbPWrgH9RVR9u62ysqoeTLAGeq6q3\nTfi+NVtjmKnB1Mho+/D6hen+3BzffLSYxzf9scHiH9+c9SKhqiZe/p/55aMk/zDJCa39j4BLgG3A\nvcDattpa4IutfS+wJsmbkpwFrAC2VtXzwA+TXNgmnj8EfGlom4P7uoLBxLUkaY4cyeWjk4C/am8g\nWgL8WVVtTvJt4O4k64CdwAcBqmp7kruB7cABYP3QS/z1wK3A8cD9VbWp1W8B7kiyA9gHrDmC/kqS\npjBrl49GxctHM7WYLz+A4xtac8GNz8tHR6UXs335SJK0+BgKkqTOUJAkdYaCJKkzFCRJnaEgSeoM\nBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWG\ngiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpD\nQZLUGQqSpM5QkCR18z4UkqxK8mSSHUn+46j7I0mL2bwOhSRvAP4YWAWsBK5K8s7R9upoGx91B+bY\n+Kg7MIfGR92BOTY+6g7MsfFRd2Ak5nUoABcAT1fVzqp6Bfhz4PIR9+koGx91B+bY+Kg7MIfGR92B\nOTY+6g7MsfFRd2Ak5nsonArsGnq+u9UkSXNgvodCjboDknQsSdX8Pe4meTewsapWtefXAT+pqhuG\n1pm/A5CkeayqMrE230NhCfA94CLgWWArcFVVfXekHZOkRWrJqDswmao6kOQjwAPAG4BbDARJmjvz\n+kxBknR0zfeJ5mNGkr9P8liSJ5I8nuTfJ0lbNpbkvlH3caaSnJlk24TaxiT/Icm7kzzSxr49yYZR\n9fNIJFmd5CdJ/umE+jmtfumo+jZTSW5M8tGh5w8k+W9Dzz+d5GNtfB8Zqv9xkrVHu78zMdnfZmsv\nSfJikv86mh4efYbC/PF/q+rcqnoXcDHwfmBBHiCn6eAp6q3Ar1fVucDPAnePrEdH5irga+3rdOoL\nwdeB9wIk+SngRAY3kR70HuAbwAvANUne2OoL/fLDcP8vZjCv+a9H1JejzlCYh6rqReA3gY9Mte4i\n8I+B5wFqYMHNGSV5M/A+4NeBNUP1AFcAvwZcnOS40fRwxr7J4MAPg8B+AvhRkqVtLO8EXgJeBB4C\nFsTZwTQdDIargJuA/5PkPZOsv2gYCvNUVT0DvCHJ20bdlzl2I/C9JH+Z5DcX4IETBnfZf7mqdgD7\nkpzX6u8Fvl9V32dwe+wvjah/M1JVzwIHkixnEA7fZPAOwPcA/xzYBvy/tvrvA7/TzigWhST/gME7\nH+8F7mRhnu29bovmF6h57XCXE6qqPsngALMZ+BVg01Hr1ey5Crirte/i1YPH4eoLyTcYhNt7GYTC\nN1v7PQwuLwH9RcwjDH6HC8lkl7p+GfhqVf0d8JfA6oPzfIvZvH5L6rEsyT8B/r6qXlwEf4f7gLdO\nqJ0IfB+gvZL+kzaJ+WKSt1bVD45yH2ckyTLgF4F3tRsp3wD8pH2i778CLkvyu0CAZUneXFU/Hl2P\nX7f/xeDS2NkMzgx2Ab8DvAx8jsG4DvoU8BfA/zjKfTwSh/rbXAY8wyDE35fkmaH6RcCDR697R59n\nCvNQu2T0J8Afjbovs6EdBJ9L8ovQD6SXAl9P8ktDr77eARwA9o+mpzNyBXB7VZ1ZVWdV1enATuB3\ngcer6vRWP5PBq80PjK6rM/INBq+Y97U5nx8AS3l1krmrqu8B24F/yQKZbD7M3+Yq4HHg54Hl7fd3\nFoM5voV4tve6GArzx/EH35IKbAE2VdX1bVkBFyXZNfS4cHRdnZGrgf+U5DEGk5Ib2yWHXwWebPXb\ngX9TC+vmmTXAX02o3QOcdZj6GhaWJxic1T08VPsOsL+qXmrPh39f/wU47Sj1bba85m8TOAd4qH06\n80H3Ar889C6rRcmb1yRJnWcKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLU/X+UzNO2\n3YI5gQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fcd01f8dd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the number of flights which will be used as y-values\n",
    "num_flights = [ x[1] for x in top5_carriers]\n",
    "\n",
    "# extract the carriers' names\n",
    "carrier_names = [x[0] for x in top5_carriers]\n",
    "\n",
    "# create `virtual indexes for carriers which will be used as x-values`\n",
    "carrier_indexes = range(0, len(carrier_names))\n",
    "\n",
    "# plot\n",
    "plt.bar(carrier_indexes, num_flights, align=\"center\")\n",
    "\n",
    "# put x-labels for the plot\n",
    "plt.xticks(carrier_indexes, carrier_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5\n",
    "\n",
    "\n",
    "### Question 5.1\n",
    "<div class=\"alert alert-info\">\n",
    "Compute how many flights have a scheduled departure time later than 09:00 and before 14:00.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1625828\n"
     ]
    }
   ],
   "source": [
    "# read the data\n",
    "data = sc.textFile('/datasets/airline/1994.csv')\n",
    "\n",
    "# extract information about scheduled departure time and carrier\n",
    "def extract_CRSDepTime_Carier(line):\n",
    "    cols = line.split(\",\")\n",
    "    return (int(cols[5]), cols[16])\n",
    "\n",
    "header = data.first()\n",
    "\n",
    "# remove header\n",
    "data_without_header = data.filter(lambda x: x != header)\n",
    "\n",
    "# create RDD with only scheduled departure time and carrier information\n",
    "# cache it for later usages\n",
    "newdata = (\n",
    "            data_without_header\n",
    "               .map(extract_CRSDepTime_Carier)\n",
    "               .cache()\n",
    "          )\n",
    "\n",
    "flights = newdata.filter(lambda x: x[0] > 900 and x[0] < 1400).cache()\n",
    "\n",
    "print(flights.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5.2\n",
    "<div class=\"alert alert-info\">\n",
    "Compute the cumulative number flights that have a scheduled departure time after 09:00 and before 14:00, for each source airport (origin). Plot the top-5 of such airports.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEACAYAAABcXmojAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFDFJREFUeJzt3X+QXeV93/H3x8gQsAmyWo/4/SNjUaPUTUEdZGhsb4OL\nNZkaaIeAmNZRa6WZMU3tum2mwp0pop1JTdoMxm0gTUyMYMYUxbT+URNAIV47NZblAnZwACM6loOE\nJagI2IlbDwrf/nGffXQtr6TlrlZ3d/N+zdzZc7/nOWe/j/bu/dxzzr2rVBWSJAG8ZtwNSJLmD0NB\nktQZCpKkzlCQJHWGgiSpMxQkSd0hQyHJbyfZk+SxodqyJFuSPJXkgSRLh9Zdl2R7kieTXDpUX5Xk\nsbbu5qH6cUnubvWtSc4aWreufY+nkvz8kZuyJOlgDnek8HFgzQG1DcCWqjoXeLDdJ8lK4GpgZdvm\nliRp29wKrK+qFcCKJFP7XA/sbfWbgBvbvpYB/wa4sN2uHw4fSdLcOGQoVNUfAH9yQPkyYFNb3gRc\n0ZYvB+6qqperagfwNLA6ySnAiVW1rY27Y2ib4X3dA1zSlt8FPFBVL1bVi8AWfjScJElH2CjXFJZX\n1Z62vAdY3pZPBXYOjdsJnDZNfVer074+A1BV+4CXkvylQ+xLkjSHZnWhuQZ/I8O/kyFJi8SSEbbZ\nk+TkqtrdTg091+q7gDOGxp3O4BX+rrZ8YH1qmzOBZ5MsAU6qqr1JdgETQ9ucAfz+dM0kMZQk6VWq\nqkxXH+VI4TPAura8DvjUUH1tkmOTnAOsALZV1W7gu0lWtwvP7wE+Pc2+rmRw4RrgAeDSJEuTvAH4\n28D9h5jcorxdf/31Y+/B+Tk/57f4bodyyCOFJHcB7wD+cpJnGLwj6MPA5iTrgR3AVe2J+fEkm4HH\ngX3AtbX/u18L3A4cD9xbVfe1+m3AnUm2A3uBtW1fLyT5d8BX27gbanDBWZI0hw4ZClV1zUFWvfMg\n438F+JVp6g8Db5mm/gNaqEyz7uMM3hIrSTpK/ETzPDYxMTHuFuaU81vYnN/ilMOdX5rvktRCn4Mk\nHU1JqCN4oVmStEgZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdaP8\nJzs6wOC/iVh4/JtRkg5kKBwxC+0JdmEGmaS55ekjSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQk\nSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqS\npM5QkCR1I4dCkg8m+UaSx5J8IslxSZYl2ZLkqSQPJFk6NP66JNuTPJnk0qH6qraP7UluHqofl+Tu\nVt+a5KzRpylJmomRQiHJacA/BVZV1VuAY4C1wAZgS1WdCzzY7pNkJXA1sBJYA9ySJG13twLrq2oF\nsCLJmlZfD+xt9ZuAG0fpVbOXZMHdJI1mNqePlgAnJFkCnAA8C1wGbGrrNwFXtOXLgbuq6uWq2gE8\nDaxOcgpwYlVta+PuGNpmeF/3AJfMolfNWi2gm6RRjRQKVbUL+DXgjxmEwYtVtQVYXlV72rA9wPK2\nfCqwc2gXO4HTpqnvanXa12fa99sHvJRk2Sj9SpJmZskoGyV5A4NX8mcDLwG/k+QfDI+pqkpyVF62\nbdy4sS9PTEwwMTFxNL6tJC0Ik5OTTE5OzmjsSKEAvBP4VlXtBUjy34CLgN1JTq6q3e3U0HNt/C7g\njKHtT2dwhLCrLR9Yn9rmTODZdorqpKp6YbpmhkNBkvTDDnyxfMMNNxx07KjXFL4NvDXJ8e2C8TuB\nx4HPAuvamHXAp9ryZ4C1SY5Ncg6wAthWVbuB7yZZ3fbzHuDTQ9tM7etKBheuJUlzaKQjharaluST\nwCPAvvb1N4ETgc1J1gM7gKva+MeTbGYQHPuAa6tq6tTStcDtwPHAvVV1X6vfBtyZZDuwl8G7myRJ\ncyj7n5sXpiQ17jkMDnIW2r9jmOm/28Kb38znJv1FlISqmva9236iWZLUGQqSpM5QkCR1hoIkqTMU\nJEmdoSBJ6kb9RLO0aCzEv6rqW241VwwFCVhon8OQ5oqnjyRJnaEgSeoMBUlSZyhIkjpDQZLUGQqS\npM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJ\nUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSN3IoJFma5JNJnkjyeJLVSZYl2ZLk\nqSQPJFk6NP66JNuTPJnk0qH6qiSPtXU3D9WPS3J3q29Nctbo05QkzcRsjhRuBu6tqvOAvwY8CWwA\ntlTVucCD7T5JVgJXAyuBNcAtSdL2cyuwvqpWACuSrGn19cDeVr8JuHEWvUqSZmCkUEhyEvC2qvpt\ngKraV1UvAZcBm9qwTcAVbfly4K6qermqdgBPA6uTnAKcWFXb2rg7hrYZ3tc9wCWj9CpJmrlRjxTO\nAZ5P8vEkjyT5rSSvA5ZX1Z42Zg+wvC2fCuwc2n4ncNo09V2tTvv6DAxCB3gpybIR+5UkzcCoobAE\nuAC4paouAP6MdqpoSlUVULNrT5J0NC0ZcbudwM6q+mq7/0ngOmB3kpOranc7NfRcW78LOGNo+9Pb\nPna15QPrU9ucCTybZAlwUlW9MF0zGzdu7MsTExNMTEyMOC1JWnwmJyeZnJyc0dgMXtC/ekm+CPxC\nVT2VZCNwQlu1t6puTLIBWFpVG9qF5k8AFzI4LfR7wJuqqpJ8BXg/sA34HPDRqrovybXAW6rqfUnW\nAldU1dpp+qhR53CkDK6ZL7SDojDTf7eFN7+Zzw0W//ykAyWhqjLtulmEwk8BHwOOBf438I+AY4DN\nDF7h7wCuqqoX2/gPAe8F9gEfqKr7W30VcDtwPIN3M72/1Y8D7gTOB/YCa9tF6gP7MBRGYij00Yt8\nftKB5iQU5gtDYVSGQh+9yOcnHehQoeAnmiVJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1o/6Z\nC0kLwP6/UL+w+DmM8TEUpEVvoT3BLswgWyw8fSRJ6gwFSVJnKEiSOkNBktQZCpKkzncfSVqwfMvt\nkWcoSFrg5u8T7PTmd5B5+kiS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqS\npM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpm1UoJDkmyaNJPtvu\nL0uyJclTSR5IsnRo7HVJtid5MsmlQ/VVSR5r624eqh+X5O5W35rkrNn0Kkk6vNkeKXwAeByodn8D\nsKWqzgUebPdJshK4GlgJrAFuSZK2za3A+qpaAaxIsqbV1wN7W/0m4MZZ9ipJOoyRQyHJ6cDPAh8D\npp7gLwM2teVNwBVt+XLgrqp6uap2AE8Dq5OcApxYVdvauDuGthne1z3AJaP2KkmamdkcKdwE/DLw\nylBteVXtact7gOVt+VRg59C4ncBp09R3tTrt6zMAVbUPeCnJsln0K0k6jJFCIcnfAZ6rqkfZf5Tw\nQ6qq2H9aSZK0ACwZcbuLgcuS/CzwY8CPJ7kT2JPk5Kra3U4NPdfG7wLOGNr+dAZHCLva8oH1qW3O\nBJ5NsgQ4qapemK6ZjRs39uWJiQkmJiZGnJYkLT6Tk5NMTk7OaGwGL+hHl+QdwL+sqncn+VUGF4dv\nTLIBWFpVG9qF5k8AFzI4LfR7wJuqqpJ8BXg/sA34HPDRqrovybXAW6rqfUnWAldU1dppvn/Ndg6z\nNbhmvtAOisJM/90W3vxmPjdY3PNbeHMD53cUOkioqmnP8ox6pHCgqRl+GNicZD2wA7gKoKoeT7KZ\nwTuV9gHXDj2TXwvcDhwP3FtV97X6bcCdSbYDe4EfCQRJ0pE16yOFcfNIYVSL+dWYRwp95IKbGzi/\no9DBIY4U/ESzJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwF\nSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaC\nJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVI3UigkOSPJ\n55P8UZJvJHl/qy9LsiXJU0keSLJ0aJvrkmxP8mSSS4fqq5I81tbdPFQ/Lsndrb41yVmzmagk6fBG\nPVJ4GfhgVf0k8FbgnyQ5D9gAbKmqc4EH232SrASuBlYCa4BbkqTt61ZgfVWtAFYkWdPq64G9rX4T\ncOOIvUqSZmikUKiq3VX1tbb8p8ATwGnAZcCmNmwTcEVbvhy4q6perqodwNPA6iSnACdW1bY27o6h\nbYb3dQ9wySi9SpJmbtbXFJKcDZwPfAVYXlV72qo9wPK2fCqwc2iznQxC5MD6rlanfX0GoKr2AS8l\nWTbbfiVJB7dkNhsneT2DV/EfqKrv7T8jBFVVSWqW/c3Ixo0b+/LExAQTExNH49tK0oIwOTnJ5OTk\njMamarTn7SSvBf4H8LtV9ZFWexKYqKrd7dTQ56vqzUk2AFTVh9u4+4DrgW+3Mee1+jXA26vqfW3M\nxqrammQJ8J2qeuM0fdSoczhSBmE43h5evTDTf7eFN7+Zzw0W9/wW3tzA+R2FDhKqKtOtG/XdRwFu\nAx6fCoTmM8C6trwO+NRQfW2SY5OcA6wAtlXVbuC7SVa3fb4H+PQ0+7qSwYVrSdIcGulIIclPA18E\n/pD9MX0dsA3YDJwJ7ACuqqoX2zYfAt4L7GNwuun+Vl8F3A4cD9xbVVNvbz0OuJPB9Yq9wNp2kfrA\nXjxSGMlifjXmkUIfueDmBs7vKHRwiCOFkU8fzReGwqgW8y+eodBHLri5gfM7Ch0c6dNHkqTFyVCQ\nJHWGgiSpMxQkSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKkzFCRJnaEgSeoMBUlSZyhI\nkjpDQZLUGQqSpM5QkCR1hoIkqTMUJEmdoSBJ6gwFSVJnKEiSOkNBktQZCpKkzlCQJHWGgiSpMxQk\nSZ2hIEnqDAVJUmcoSJI6Q0GS1BkKkqTOUJAkdYaCJKmb96GQZE2SJ5NsT/Kvxt2PJC1m8zoUkhwD\n/GdgDbASuCbJeePt6miaHHcDc2xy3A3MsclxNzDHJsfdwBybHHcDYzGvQwG4EHi6qnZU1cvAfwUu\nH3NPR9HkuBuYY5PjbmCOTY67gTk2Oe4G5tjkuBsYi/keCqcBzwzd39lqkqQ5MN9DocbdgCT9RZKq\n+fu8m+StwMaqWtPuXwe8UlU3Do2ZvxOQpHmqqjJdfb6HwhLgm8AlwLPANuCaqnpirI1J0iK1ZNwN\nHEpV7UvyS8D9wDHAbQaCJM2deX2kIEk6uub7heZFK8npST6d5KkkTyf5SJLXJplI8lKSR5M8keQ/\nDG3zD5M8n+SRtt19SS4a5zwOlOTPW+/fSPK1JP88Sdq64bk9muSBJCcl2Tu0/UVJXklyarv/Q+vn\nmyRXtH7/SpKtbV7fTvJcW34kyVlJdiRZNu5+X40k/7r9HL/e5vL77ev2JC8O/RwvSjKZZNW4ez6U\ngz02k7xraC7fax+WfTTJpvbz+6m2/ZIkf5rk7w/t8+Ekf318s5oDVeXtKN+AMLg+sq7dfw3wMeBX\ngXcAn231HwOeAC5u99cBHx3azwTwHeDN457TUE/fG1p+I7CFwZsFpvr9zDTbPAac15b/BfC/gJ9r\n998F3DvueR1ivncDX5ia43Q/p1b7FrBs3P2+inldBDwEvLbdXwac3Jb7Y3Ro/OeBC8bd92HmdNDH\n5sHmAfwn4H1teRXwMPDr7f7rgD+hnXFZLDePFMbjZ4D/W1WbAKrqFeCDwHuBE6YGVdX/A74GnNpK\nabep9ZPAbwK/eFS6fpWq6nkGvf3SUHm6dzw8BFzcli8CPjJ0/2LgS3PV42wkeT3wN4FfANYOr2L6\neS4kJwP/pwYfGqWqXqiq3W3dQp/bwR6bU4bnd+Bj8zeAqSODC4GHqyXEYmEojMdPMnjF0VXV94A/\nBt40VUvyBmAF8MWpYdPs6xHgzXPT5uxV1beAY5K8sZXeNnSofl2rfYn9v3g/AfwO8Dfa/YsZ/GLO\nR5cDv1tV24G9SS5o9cXwJPEAcEaSbyb59SRvH3dDR9o0j82+amh5OBQuZvC7+IP2gmDevmCZDUNh\nPA73pPG2JF9j8Anu+6vquUOMXWg/wz+oqvPb7d+32kPAxUnOBnZU1Q+AJHkdcAHwlfG0eljXMDh9\nRPt6TVteDK+k/4zB6ZJfBJ4H7k6ybrxdHX1V9W3g2CTLGZym/SbwVWA1gyOHRRcK8/otqYvY48CV\nw4UkPw6cCTzN4Inz3e1JcmuSzVX19YPs6/y2v3kpyU8Af15Vz7frzT+iqp5OshR4N/uPCh5mcDpt\nR1V9/6g0+yq0i8Z/C/ir7QOUxwCvAL881saOoHZa8wvAF5I8xuBayabxdnXkDD82DzP0IeAqBtfv\nALYCP83g9NGX567D8VhorzIXhap6EDghyXug/zXYXwM+Dnx/aNwO4MPA1J8M/6Fn1STvAP4x8Ftz\n3/Wr1w7Lf4PBxbrD2Qp8gP2/ZF8G/hnwP+emu1m7Erijqs6uqnOq6kxgR5K3cfAjwQVzBJHk3CQr\nhkrnAzsOt9ncdXRkvcrH5kMMHotTL1i+DPw88J122ndRMRTG5+8CP5fkKQaf2v4+8KG2bvhJ5b8A\nb09yZqtf3c7HfxPYAPy9dkg7Xxw/9bY/Bu/uuK+qbmjrioM/YX4JOJ3BO49gEBLnMH+vJ6wF/vsB\ntXvYfwppunn+YZJn2u0/zml3s/d64PYkf5Tk6wyuW21s6w72c/zc0Pzunmb9uP3IYxP4tzPY7iHg\nbNoLlnbB/TXM38fmrPjhNUlS55GCJKkzFCRJnaEgSeoMBUlSZyhIkjpDQZLUGQqSpM5QkCR1/x9O\nc/XfPEyjXAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f3de398a160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flights_per_carrier = flights.groupBy(lambda x :x[1]).mapValues(lambda x : len(x))\n",
    "\n",
    "# take top 5 source airports\n",
    "top5_source_airport = flights_per_carrier.takeOrdered(5, key=lambda x: -x[1])\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# extract the number of flights which will be used as y-values\n",
    "# This is called list comprehension\n",
    "num_flights = [ x[1] for x in top5_source_airport]\n",
    "\n",
    "# create `virtual indexes for carriers which will be used as x-values`\n",
    "airport_indexes = range(0, len(top5_source_airport))\n",
    "\n",
    "# plot\n",
    "plt.bar(airport_indexes, num_flights, align=\"center\")\n",
    "\n",
    "# extract the carriers' names\n",
    "airport_names = [ x[0] for x in top5_source_airport]\n",
    "\n",
    "# put x-labels for the plot\n",
    "plt.xticks(airport_indexes, airport_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "In this lecture, we gained familiarity with the Jupyter Notebook environment, the Python programming language and its modules. In particular, we covered the Python syntax, Numpy - the core library for scientific computing, Matplotlib - a module to plot graphs, Pandas - a data analysis module. Besides, we started to gain practical experience with PySpark, using, for the moment, small data sets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}