S06a_TidyData.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute before using this notebook if using google colab\n",
    "\n",
    "kernel = str(get_ipython())\n",
    "\n",
    "if 'google.colab' in kernel:    \n",
    "    !wget https://raw.githubusercontent.com/fredzett/rmqa/master/utils.py -P local_modules -nc \n",
    "    !npx degit fredzett/rmqa/data data\n",
    "    import sys\n",
    "    sys.path.append('local_modules')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 475,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import statsmodels.api as sm\n",
    "from patsy import dmatrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have been mostly using the following modules:\n",
    "\n",
    "- `scipy`: for probabilties and simulations\n",
    "- `numpy`: for numerical operations\n",
    "- `matplotlib`: for plotting (note: partly disguised by helper functions provided for this class (e.g. `plot_line`) which abstract away use of matplotlib)\n",
    "\n",
    "We have only recently introduced `pandas` as a new module to work with dataframes. When referring to dataframes we mean data objects similar to an **excel sheet** where we would typically have:\n",
    "\n",
    "- header describing the column data\n",
    "- each column data could be of different types (numeric, qualtiative, text etc.)\n",
    "- each row is labelled\n",
    "\n",
    "**Dataframes helpful for reserach**  \n",
    "Working with dataframes is helpful in quantitative research as our data typically doesn't fully consist of numerical data and we also want to give our variables explanatory names. \n",
    "\n",
    "**Organization of data is important**  \n",
    "Quantitative research involves many steps. The data analysis part involves e.g.: \n",
    "\n",
    "- generating data: e.g. via survey, web scraping, gathering from different sources etc.\n",
    "\n",
    "- preprocessing data: i.e. preparing the data for analysis; e.g. cleaning the data, filtering data, rearranging data, combining data sets etc.\n",
    "\n",
    "- exploring data: plotting raw data, calculating descriptive statistics etc.\n",
    "\n",
    "- analyzing data: e.g. running a regression analysis\n",
    "\n",
    "- presenting data: e.g. plotting data or creating summary tables\n",
    "\n",
    "Most university courses focus on the generation and analysis part of the data analysis part. Up unitl now this was also our focus (and will continue to be). However, in practical research one of the most **time consuming** steps is the **preprocessing** step. Some [authors](https://www.wiley.com/en-us/Exploratory+Data+Mining+and+Data+Cleaning-p-9780471268512) claim that this part of the data analysis pipeline takes up about $80$% of the work. \n",
    "\n",
    "\n",
    "To this end, the this lecture has two purposes:\n",
    "\n",
    "1. explain how to work with `pandas`\n",
    "\n",
    "2. explain how to set up a dataframe best suited for quantitative analysis and statistics\n",
    "\n",
    "Both shall help you to be able to prepare data for your research. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we want to understand how to best set up a dataframe to serve your research / analysis purposes, let's have a look at the `pandas` module in detail. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd # import pandas module and use it by prefix \"pd\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating pandas dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create pandas dataframes in two basic fashions:\n",
    "\n",
    "1. from scratch using other datatypes\n",
    "\n",
    "2. from reading in a file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating dataframe from scratch using numpy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1,  2,  3],\n",
       "       [ 1,  2,  4],\n",
       "       [35,  7,  3]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.array(((1,2,3),(1,2,4),(35,7,3)))\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0  1  2\n",
       "0   1  2  3\n",
       "1   1  2  4\n",
       "2  35  7  3"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also give both index and columns specific names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>H1</th>\n",
       "      <th>H2</th>\n",
       "      <th>H3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   H1 H2   H3\n",
       "a   1  A  NaN\n",
       "b   2  B  NaN\n",
       "c   3  C  NaN"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(data, index=(\"a\",\"b\",\"c\"), columns=(\"H1\",\"H2\",\"H3\"))\n",
    "df # a common general name for a variable holding a dataframe is \"df\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating dataframe from scratch using dictionaries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>H1</th>\n",
       "      <th>H2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   H1 H2\n",
       "0   1  A\n",
       "1   2  B\n",
       "2   3  C"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\"H1\":[1,2,3], \"H2\":[\"A\",\"B\",\"C\"]}\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Index and columns names**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access both index and columns using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['A', 'B', 'C'], dtype='object')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.index = [\"A\",\"B\",\"C\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>New1</th>\n",
       "      <th>New2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   New1 New2\n",
       "A     1    A\n",
       "B     2    B\n",
       "C     3    C"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['New1', 'New2'], dtype='object')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>New1</th>\n",
       "      <th>New2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>2</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C</th>\n",
       "      <td>3</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   New1 New2\n",
       "A     1    A\n",
       "B     2    B\n",
       "C     3    C"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = [\"New1\", \"New2\"]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating dataframe from files**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In research it is common that we have data and need to load it into a dataframe. This can be easily done using the \n",
    "\n",
    "- `pd.read_` commmand\n",
    "\n",
    "which gives you a range of file types that you can read from\n",
    "\n",
    "![pd.read](https://www.dropbox.com/s/4zx16ag09vcjt6t/pdread.png?dl=1)\n",
    "\n",
    "Let's look at an example and read in the file `Auto.csv` which is in the folder `data` which is located in the same folder as this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>27.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>44.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>vw pickup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>32.0</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge rampage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>31.0</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>392 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0    18.0          8         307.0         130    3504          12.0    70   \n",
       "1    15.0          8         350.0         165    3693          11.5    70   \n",
       "2    18.0          8         318.0         150    3436          11.0    70   \n",
       "3    16.0          8         304.0         150    3433          12.0    70   \n",
       "4    17.0          8         302.0         140    3449          10.5    70   \n",
       "..    ...        ...           ...         ...     ...           ...   ...   \n",
       "387  27.0          4         140.0          86    2790          15.6    82   \n",
       "388  44.0          4          97.0          52    2130          24.6    82   \n",
       "389  32.0          4         135.0          84    2295          11.6    82   \n",
       "390  28.0          4         120.0          79    2625          18.6    82   \n",
       "391  31.0          4         119.0          82    2720          19.4    82   \n",
       "\n",
       "     origin                       name  \n",
       "0         1  chevrolet chevelle malibu  \n",
       "1         1          buick skylark 320  \n",
       "2         1         plymouth satellite  \n",
       "3         1              amc rebel sst  \n",
       "4         1                ford torino  \n",
       "..      ...                        ...  \n",
       "387       1            ford mustang gl  \n",
       "388       2                  vw pickup  \n",
       "389       1              dodge rampage  \n",
       "390       1                ford ranger  \n",
       "391       1                 chevy s-10  \n",
       "\n",
       "[392 rows x 9 columns]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fpath = \"./data/Auto.csv\"\n",
    "df = pd.read_csv(fpath)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All `pd.read_` functions provide a wide variety of loading options. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pd.read_csv? # see load in options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example I could read in the same file with only a specific set of columns (columns 2 and 6). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>displacement</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>307.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>350.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>318.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>304.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>302.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>140.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>97.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>135.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>120.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>119.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>392 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     displacement  year\n",
       "0           307.0    70\n",
       "1           350.0    70\n",
       "2           318.0    70\n",
       "3           304.0    70\n",
       "4           302.0    70\n",
       "..            ...   ...\n",
       "387         140.0    82\n",
       "388          97.0    82\n",
       "389         135.0    82\n",
       "390         120.0    82\n",
       "391         119.0    82\n",
       "\n",
       "[392 rows x 2 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(fpath, usecols=[2,6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways how to work with data stored in dataframes. Note that one way is to convert the data into numpy arrays and work with them the usual way. This has the advantage of working with data objects we already are familar with. It has the disadvantage that we loose some information (such as header) and functionality (e.g. it is much harder to work with non-numerical data). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "      <th>horsepower_per_weight</th>\n",
       "      <th>horsepower_per_lbs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "      <td>0.037100</td>\n",
       "      <td>0.037100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "      <td>0.044679</td>\n",
       "      <td>0.044679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "      <td>0.043655</td>\n",
       "      <td>0.043655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "      <td>0.043694</td>\n",
       "      <td>0.043694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "      <td>0.040591</td>\n",
       "      <td>0.040591</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0  18.0          8         307.0         130    3504          12.0    70   \n",
       "1  15.0          8         350.0         165    3693          11.5    70   \n",
       "2  18.0          8         318.0         150    3436          11.0    70   \n",
       "3  16.0          8         304.0         150    3433          12.0    70   \n",
       "4  17.0          8         302.0         140    3449          10.5    70   \n",
       "\n",
       "   origin                       name  horsepower_per_weight  \\\n",
       "0       1  chevrolet chevelle malibu               0.037100   \n",
       "1       1          buick skylark 320               0.044679   \n",
       "2       1         plymouth satellite               0.043655   \n",
       "3       1              amc rebel sst               0.043694   \n",
       "4       1                ford torino               0.040591   \n",
       "\n",
       "   horsepower_per_lbs  \n",
       "0            0.037100  \n",
       "1            0.044679  \n",
       "2            0.043655  \n",
       "3            0.043694  \n",
       "4            0.040591  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head() # .head() shows first 5 rows of data (note that .tail() shows last five rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_numpy = df.values # .values extracts the numpy data underneath the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18.0, 8, 307.0, 130, 3504, 12.0, 70, 1,\n",
       "        'chevrolet chevelle malibu', 0.037100456621004564,\n",
       "        0.037100456621004564],\n",
       "       [15.0, 8, 350.0, 165, 3693, 11.5, 70, 1, 'buick skylark 320',\n",
       "        0.04467912266450041, 0.04467912266450041],\n",
       "       [18.0, 8, 318.0, 150, 3436, 11.0, 70, 1, 'plymouth satellite',\n",
       "        0.043655413271245634, 0.043655413271245634],\n",
       "       [16.0, 8, 304.0, 150, 3433, 12.0, 70, 1, 'amc rebel sst',\n",
       "        0.043693562481794346, 0.043693562481794346],\n",
       "       [17.0, 8, 302.0, 140, 3449, 10.5, 70, 1, 'ford torino',\n",
       "        0.04059147579008408, 0.04059147579008408]], dtype=object)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_numpy[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting columns**\n",
    "\n",
    "Let's continue and work with the dataframes directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      18.0\n",
       "1      15.0\n",
       "2      18.0\n",
       "3      16.0\n",
       "4      17.0\n",
       "       ... \n",
       "387    27.0\n",
       "388    44.0\n",
       "389    32.0\n",
       "390    28.0\n",
       "391    31.0\n",
       "Name: mpg, Length: 392, dtype: float64"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"mpg\"] # selecting one column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>27.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>44.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>32.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>28.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>31.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>392 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  year\n",
       "0    18.0    70\n",
       "1    15.0    70\n",
       "2    18.0    70\n",
       "3    16.0    70\n",
       "4    17.0    70\n",
       "..    ...   ...\n",
       "387  27.0    82\n",
       "388  44.0    82\n",
       "389  32.0    82\n",
       "390  28.0    82\n",
       "391  31.0    82\n",
       "\n",
       "[392 rows x 2 columns]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[[\"mpg\", \"year\"]] # selecting two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "      <th>horsepower_per_lbs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "      <td>0.037100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "      <td>0.044679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "      <td>0.043655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "      <td>0.043694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "      <td>0.040591</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0  18.0          8         307.0         130    3504          12.0    70   \n",
       "1  15.0          8         350.0         165    3693          11.5    70   \n",
       "2  18.0          8         318.0         150    3436          11.0    70   \n",
       "3  16.0          8         304.0         150    3433          12.0    70   \n",
       "4  17.0          8         302.0         140    3449          10.5    70   \n",
       "\n",
       "   origin                       name  horsepower_per_lbs  \n",
       "0       1  chevrolet chevelle malibu            0.037100  \n",
       "1       1          buick skylark 320            0.044679  \n",
       "2       1         plymouth satellite            0.043655  \n",
       "3       1              amc rebel sst            0.043694  \n",
       "4       1                ford torino            0.040591  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# adding a new column by selecting two columns and applying calculation\n",
    "df[\"horsepower_per_lbs\"] = df[\"horsepower\"] / df[\"weight\"]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selecting columns using `loc` and `iloc`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commonly we want to select subsets of a dataframe (i.e. we want to select rows and columns). This can be done via \n",
    "\n",
    "- `iloc`: selecting as it we were working with numpy array (i.e. use numerical indexing)\n",
    "- `loc`: selecting using explicit index and/or column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "      <th>horsepower_per_lbs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "      <td>0.037100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "      <td>0.044679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "      <td>0.043655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "      <td>0.043694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "      <td>0.040591</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0  18.0          8         307.0         130    3504          12.0    70   \n",
       "1  15.0          8         350.0         165    3693          11.5    70   \n",
       "2  18.0          8         318.0         150    3436          11.0    70   \n",
       "3  16.0          8         304.0         150    3433          12.0    70   \n",
       "4  17.0          8         302.0         140    3449          10.5    70   \n",
       "\n",
       "   origin                       name  horsepower_per_lbs  \n",
       "0       1  chevrolet chevelle malibu            0.037100  \n",
       "1       1          buick skylark 320            0.044679  \n",
       "2       1         plymouth satellite            0.043655  \n",
       "3       1              amc rebel sst            0.043694  \n",
       "4       1                ford torino            0.040591  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `iloc` indexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   displacement  horsepower  weight  acceleration\n",
       "1         350.0         165    3693          11.5\n",
       "2         318.0         150    3436          11.0\n",
       "3         304.0         150    3433          12.0"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[1:4,2:6] # selecting subset consisting of row 1 to row 3 and columns 2 to columns 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>86</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>52</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>84</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>79</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>82</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>392 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     horsepower  weight  acceleration  year\n",
       "0           130    3504          12.0    70\n",
       "1           165    3693          11.5    70\n",
       "2           150    3436          11.0    70\n",
       "3           150    3433          12.0    70\n",
       "4           140    3449          10.5    70\n",
       "..          ...     ...           ...   ...\n",
       "387          86    2790          15.6    82\n",
       "388          52    2130          24.6    82\n",
       "389          84    2295          11.6    82\n",
       "390          79    2625          18.6    82\n",
       "391          82    2720          19.4    82\n",
       "\n",
       "[392 rows x 4 columns]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[:,3:7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acceleration</th>\n",
       "      <th>origin</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10.5</td>\n",
       "      <td>1</td>\n",
       "      <td>3449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>13.5</td>\n",
       "      <td>3</td>\n",
       "      <td>2330</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    acceleration  origin  weight\n",
       "2           11.0       1    3436\n",
       "4           10.5       1    3449\n",
       "5           10.0       1    4341\n",
       "70          13.5       3    2330"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[[2,4,5,70],[5,7,4]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `loc` indexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cylinders</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>392 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     cylinders  year  origin\n",
       "0            8    70       1\n",
       "1            8    70       1\n",
       "2            8    70       1\n",
       "3            8    70       1\n",
       "4            8    70       1\n",
       "..         ...   ...     ...\n",
       "387          4    82       1\n",
       "388          4    82       2\n",
       "389          4    82       1\n",
       "390          4    82       1\n",
       "391          4    82       1\n",
       "\n",
       "[392 rows x 3 columns]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[:, [\"cylinders\", \"year\", \"origin\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8</td>\n",
       "      <td>429.0</td>\n",
       "      <td>198</td>\n",
       "      <td>4341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8</td>\n",
       "      <td>454.0</td>\n",
       "      <td>220</td>\n",
       "      <td>4354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>440.0</td>\n",
       "      <td>215</td>\n",
       "      <td>4312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>455.0</td>\n",
       "      <td>225</td>\n",
       "      <td>4425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8</td>\n",
       "      <td>390.0</td>\n",
       "      <td>190</td>\n",
       "      <td>3850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>8</td>\n",
       "      <td>383.0</td>\n",
       "      <td>170</td>\n",
       "      <td>3563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>8</td>\n",
       "      <td>340.0</td>\n",
       "      <td>160</td>\n",
       "      <td>3609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>8</td>\n",
       "      <td>400.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3761</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    cylinders  displacement  horsepower  weight\n",
       "1           8         350.0         165    3693\n",
       "2           8         318.0         150    3436\n",
       "3           8         304.0         150    3433\n",
       "4           8         302.0         140    3449\n",
       "5           8         429.0         198    4341\n",
       "6           8         454.0         220    4354\n",
       "7           8         440.0         215    4312\n",
       "8           8         455.0         225    4425\n",
       "9           8         390.0         190    3850\n",
       "10          8         383.0         170    3563\n",
       "11          8         340.0         160    3609\n",
       "12          8         400.0         150    3761"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[1:12, \"cylinders\":\"weight\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Using more sophisticated indexing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "      <th>horsepower_per_lbs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "      <td>0.037100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "      <td>0.044679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "      <td>0.043655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "      <td>0.043694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "      <td>0.040591</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0  18.0          8         307.0         130    3504          12.0    70   \n",
       "1  15.0          8         350.0         165    3693          11.5    70   \n",
       "2  18.0          8         318.0         150    3436          11.0    70   \n",
       "3  16.0          8         304.0         150    3433          12.0    70   \n",
       "4  17.0          8         302.0         140    3449          10.5    70   \n",
       "\n",
       "   origin                       name  horsepower_per_lbs  \n",
       "0       1  chevrolet chevelle malibu            0.037100  \n",
       "1       1          buick skylark 320            0.044679  \n",
       "2       1         plymouth satellite            0.043655  \n",
       "3       1              amc rebel sst            0.043694  \n",
       "4       1                ford torino            0.040591  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>horsepower</th>\n",
       "      <th>year</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>130</td>\n",
       "      <td>70</td>\n",
       "      <td>3504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>165</td>\n",
       "      <td>70</td>\n",
       "      <td>3693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>150</td>\n",
       "      <td>70</td>\n",
       "      <td>3436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150</td>\n",
       "      <td>70</td>\n",
       "      <td>3433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140</td>\n",
       "      <td>70</td>\n",
       "      <td>3449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>142</td>\n",
       "      <td>79</td>\n",
       "      <td>4054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>125</td>\n",
       "      <td>79</td>\n",
       "      <td>3605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>150</td>\n",
       "      <td>79</td>\n",
       "      <td>3940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>125</td>\n",
       "      <td>79</td>\n",
       "      <td>3900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>330</th>\n",
       "      <td>132</td>\n",
       "      <td>80</td>\n",
       "      <td>2910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>102 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     horsepower  year  weight\n",
       "0           130    70    3504\n",
       "1           165    70    3693\n",
       "2           150    70    3436\n",
       "3           150    70    3433\n",
       "4           140    70    3449\n",
       "..          ...   ...     ...\n",
       "288         142    79    4054\n",
       "289         125    79    3605\n",
       "290         150    79    3940\n",
       "296         125    79    3900\n",
       "330         132    80    2910\n",
       "\n",
       "[102 rows x 3 columns]"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df[\"horsepower\"]>120, [\"horsepower\", \"year\", \"weight\"]] # for columns horsepower, year and weight, select all rows where horespower > 120"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations on dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is designed to work with numpy which means that most numpy functions work on pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      2.890372\n",
       "1      2.708050\n",
       "2      2.890372\n",
       "3      2.772589\n",
       "4      2.833213\n",
       "         ...   \n",
       "387    3.295837\n",
       "388    3.784190\n",
       "389    3.465736\n",
       "390    3.332205\n",
       "391    3.433987\n",
       "Name: mpg, Length: 392, dtype: float64"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take the natural logarithm of \"mpg\"\n",
    "np.log(df[\"mpg\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also conduct regular operations on dataframes. Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       5.538462\n",
       "1       4.800000\n",
       "2       6.000000\n",
       "3       4.923077\n",
       "4       5.913043\n",
       "         ...    \n",
       "387     6.506024\n",
       "388     6.616541\n",
       "389    10.158730\n",
       "390     5.714286\n",
       "391     6.078431\n",
       "Length: 392, dtype: float64"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_calc = (4 * df[\"mpg\"]) / (df[\"acceleration\"] + df[\"origin\"])\n",
    "random_calc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns and row wise operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12</td>\n",
       "      <td>49</td>\n",
       "      <td>40</td>\n",
       "      <td>91</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>77</td>\n",
       "      <td>40</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>75</td>\n",
       "      <td>67</td>\n",
       "      <td>88</td>\n",
       "      <td>64</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>87</td>\n",
       "      <td>22</td>\n",
       "      <td>25</td>\n",
       "      <td>98</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>96</td>\n",
       "      <td>82</td>\n",
       "      <td>56</td>\n",
       "      <td>50</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>89</td>\n",
       "      <td>91</td>\n",
       "      <td>75</td>\n",
       "      <td>76</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>40</td>\n",
       "      <td>75</td>\n",
       "      <td>4</td>\n",
       "      <td>71</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>98</td>\n",
       "      <td>60</td>\n",
       "      <td>18</td>\n",
       "      <td>33</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>53</td>\n",
       "      <td>62</td>\n",
       "      <td>18</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>82</td>\n",
       "      <td>43</td>\n",
       "      <td>86</td>\n",
       "      <td>3</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    a   b   c   d   e\n",
       "0  12  49  40  91  84\n",
       "1  21   1  77  40  92\n",
       "2  75  67  88  64  95\n",
       "3  87  22  25  98  77\n",
       "4  96  82  56  50   3\n",
       "5  89  91  75  76  93\n",
       "6  40  75   4  71  43\n",
       "7  98  60  18  33  17\n",
       "8   8  53  62  18  64\n",
       "9  82  43  86   3  29"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create random dataframe\n",
    "rnd_data = np.random.randint(1,100, (10,5))\n",
    "df_rnd = pd.DataFrame(rnd_data,columns=list(\"abcde\"))\n",
    "df_rnd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add row 8 from all other rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20</td>\n",
       "      <td>102</td>\n",
       "      <td>102</td>\n",
       "      <td>109</td>\n",
       "      <td>148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>29</td>\n",
       "      <td>54</td>\n",
       "      <td>139</td>\n",
       "      <td>58</td>\n",
       "      <td>156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>83</td>\n",
       "      <td>120</td>\n",
       "      <td>150</td>\n",
       "      <td>82</td>\n",
       "      <td>159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>95</td>\n",
       "      <td>75</td>\n",
       "      <td>87</td>\n",
       "      <td>116</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>104</td>\n",
       "      <td>135</td>\n",
       "      <td>118</td>\n",
       "      <td>68</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>97</td>\n",
       "      <td>144</td>\n",
       "      <td>137</td>\n",
       "      <td>94</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>48</td>\n",
       "      <td>128</td>\n",
       "      <td>66</td>\n",
       "      <td>89</td>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>106</td>\n",
       "      <td>113</td>\n",
       "      <td>80</td>\n",
       "      <td>51</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>16</td>\n",
       "      <td>106</td>\n",
       "      <td>124</td>\n",
       "      <td>36</td>\n",
       "      <td>128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>90</td>\n",
       "      <td>96</td>\n",
       "      <td>148</td>\n",
       "      <td>21</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b    c    d    e\n",
       "0   20  102  102  109  148\n",
       "1   29   54  139   58  156\n",
       "2   83  120  150   82  159\n",
       "3   95   75   87  116  141\n",
       "4  104  135  118   68   67\n",
       "5   97  144  137   94  157\n",
       "6   48  128   66   89  107\n",
       "7  106  113   80   51   81\n",
       "8   16  106  124   36  128\n",
       "9   90   96  148   21   93"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd + df_rnd.loc[8,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Substract column \"b\" from all columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-37</td>\n",
       "      <td>0</td>\n",
       "      <td>-9</td>\n",
       "      <td>42</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>76</td>\n",
       "      <td>39</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>-3</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>65</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>76</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>-26</td>\n",
       "      <td>-32</td>\n",
       "      <td>-79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>-16</td>\n",
       "      <td>-15</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-35</td>\n",
       "      <td>0</td>\n",
       "      <td>-71</td>\n",
       "      <td>-4</td>\n",
       "      <td>-32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>-42</td>\n",
       "      <td>-27</td>\n",
       "      <td>-43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-45</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>-35</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>43</td>\n",
       "      <td>-40</td>\n",
       "      <td>-14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    a  b   c   d   e\n",
       "0 -37  0  -9  42  35\n",
       "1  20  0  76  39  91\n",
       "2   8  0  21  -3  28\n",
       "3  65  0   3  76  55\n",
       "4  14  0 -26 -32 -79\n",
       "5  -2  0 -16 -15   2\n",
       "6 -35  0 -71  -4 -32\n",
       "7  38  0 -42 -27 -43\n",
       "8 -45  0   9 -35  11\n",
       "9  39  0  43 -40 -14"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd.subtract(df_rnd[\"b\"], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregating data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also - as with numpy arrays - aggregate data using pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12</td>\n",
       "      <td>49</td>\n",
       "      <td>40</td>\n",
       "      <td>91</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>77</td>\n",
       "      <td>40</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>75</td>\n",
       "      <td>67</td>\n",
       "      <td>88</td>\n",
       "      <td>64</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>87</td>\n",
       "      <td>22</td>\n",
       "      <td>25</td>\n",
       "      <td>98</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>96</td>\n",
       "      <td>82</td>\n",
       "      <td>56</td>\n",
       "      <td>50</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>89</td>\n",
       "      <td>91</td>\n",
       "      <td>75</td>\n",
       "      <td>76</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>40</td>\n",
       "      <td>75</td>\n",
       "      <td>4</td>\n",
       "      <td>71</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>98</td>\n",
       "      <td>60</td>\n",
       "      <td>18</td>\n",
       "      <td>33</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>53</td>\n",
       "      <td>62</td>\n",
       "      <td>18</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>82</td>\n",
       "      <td>43</td>\n",
       "      <td>86</td>\n",
       "      <td>3</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    a   b   c   d   e\n",
       "0  12  49  40  91  84\n",
       "1  21   1  77  40  92\n",
       "2  75  67  88  64  95\n",
       "3  87  22  25  98  77\n",
       "4  96  82  56  50   3\n",
       "5  89  91  75  76  93\n",
       "6  40  75   4  71  43\n",
       "7  98  60  18  33  17\n",
       "8   8  53  62  18  64\n",
       "9  82  43  86   3  29"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    60.8\n",
       "b    54.3\n",
       "c    53.1\n",
       "d    54.4\n",
       "e    59.7\n",
       "dtype: float64"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd.mean() # take mean over each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    55.2\n",
       "1    46.2\n",
       "2    77.8\n",
       "3    61.8\n",
       "4    57.4\n",
       "5    84.8\n",
       "6    46.6\n",
       "7    45.2\n",
       "8    41.0\n",
       "9    48.6\n",
       "dtype: float64"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd.mean(axis=1) # taken mean over each row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    527\n",
       "b    365\n",
       "c    348\n",
       "d    324\n",
       "e    314\n",
       "dtype: int64"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_rnd[df_rnd[\"a\"]>50].sum() # take sum over each column only taking rows into account where column a > 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling missing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In real life you will have to handle missing data (in python called `nan`, i.e. not a number). It is important to be able to deal with missing data, e.g.\n",
    "\n",
    "- filter out all observation where certain variable data is missing (e.g. in a survey people didn't answer certain questions)\n",
    "\n",
    "- specify how to deal with missing data when calculating descriptive statistics (e.g. the mean)\n",
    "\n",
    "Let's look at how to deal with missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b     c     d\n",
       "0  1.0  5.0   8.0  12.0\n",
       "1  NaN  6.0   9.0  13.0\n",
       "2  3.0  NaN  10.0  14.0\n",
       "3  4.0  NaN   NaN  15.0"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.array(((1,np.nan, 3,4), (5,6,np.nan, np.nan), (8, 9, 10, np.nan), (12,13,14,15))).T\n",
    "df = pd.DataFrame(data, columns=list(\"abcd\"))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Determine nans**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       a      b      c      d\n",
       "0  False  False  False  False\n",
       "1   True  False  False  False\n",
       "2  False   True  False  False\n",
       "3  False   True   True  False"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull() # True if nan, False if not nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       a      b      c     d\n",
       "0   True   True   True  True\n",
       "1  False   True   True  True\n",
       "2   True  False   True  True\n",
       "3   True  False  False  True"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.notnull() # True if not nan, False if nan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filtering out nans**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b    c     d\n",
       "0  1.0  5.0  8.0  12.0"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna() # drop all rows that contain nans (equal to df.dropna(axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      d\n",
       "0  12.0\n",
       "1  13.0\n",
       "2  14.0\n",
       "3  15.0"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(axis=1) # drop all columns that contain nans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b     c     d\n",
       "0  1.0  5.0   8.0  12.0\n",
       "2  3.0  NaN  10.0  14.0"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(subset=[\"a\", \"c\"]) # drow all rows that contain nans in columns \"a\" and/or \"b\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b     c     d\n",
       "0  1.0  5.0   8.0  12.0\n",
       "1  NaN  6.0   9.0  13.0\n",
       "2  3.0  NaN  10.0  14.0"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(thresh=3) # drop all rows that have less than 3 non-nan values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      d\n",
       "0  12.0\n",
       "1  13.0\n",
       "2  14.0\n",
       "3  15.0"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(thresh=4, axis=1) # drop all columns that have less than 4 non-nan values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Filling nan values**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice you don't always have to drop missing data points but could instead impute values for these missing data points. \n",
    "\n",
    "For example in stock market data you may have missing data for certain days (out of many days) because the data provider was experiencing a technical issue. In that specific case it may be perfectly reasonable to fill the missing data using some sort of data imputation technique. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stock1</th>\n",
       "      <th>Stock2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-01</th>\n",
       "      <td>0.649239</td>\n",
       "      <td>0.956359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-02</th>\n",
       "      <td>0.100613</td>\n",
       "      <td>0.749957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-03</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.899354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-04</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.374508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-05</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.913792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0.303548</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.207133</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.486524</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.667178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.607004</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Stock1    Stock2\n",
       "2020-01-01  0.649239  0.956359\n",
       "2020-01-02  0.100613  0.749957\n",
       "2020-01-03  0.883636  0.899354\n",
       "2020-01-04       NaN  0.374508\n",
       "2020-01-05  0.867854  0.913792\n",
       "2020-01-06  0.303548  0.540079\n",
       "2020-01-07  0.207133       NaN\n",
       "2020-01-08  0.486524  0.704109\n",
       "2020-01-09  0.096799  0.667178\n",
       "2020-01-10  0.607004  0.134332"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create random stock data\n",
    "stock1 = np.random.random(10)\n",
    "stock2 = np.random.random(10)\n",
    "df = pd.DataFrame({\"Stock1\": stock1, \"Stock2\":stock2})\n",
    "df.iloc[3,0] = np.nan\n",
    "df.iloc[6,1] = np.nan\n",
    "df.index = pd.date_range(start=\"01/01/2020\", periods=len(stock1))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stock1</th>\n",
       "      <th>Stock2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-01</th>\n",
       "      <td>0.649239</td>\n",
       "      <td>0.956359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-02</th>\n",
       "      <td>0.100613</td>\n",
       "      <td>0.749957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-03</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.899354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-04</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.374508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-05</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.913792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0.303548</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.207133</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.486524</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.667178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.607004</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Stock1    Stock2\n",
       "2020-01-01  0.649239  0.956359\n",
       "2020-01-02  0.100613  0.749957\n",
       "2020-01-03  0.883636  0.899354\n",
       "2020-01-04  0.883636  0.374508\n",
       "2020-01-05  0.867854  0.913792\n",
       "2020-01-06  0.303548  0.540079\n",
       "2020-01-07  0.207133  0.540079\n",
       "2020-01-08  0.486524  0.704109\n",
       "2020-01-09  0.096799  0.667178\n",
       "2020-01-10  0.607004  0.134332"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace with previous value\n",
    "df.fillna(method=\"ffill\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stock1</th>\n",
       "      <th>Stock2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-01</th>\n",
       "      <td>0.649239</td>\n",
       "      <td>0.956359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-02</th>\n",
       "      <td>0.100613</td>\n",
       "      <td>0.749957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-03</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.899354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-04</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.374508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-05</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.913792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0.303548</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.207133</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.486524</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.667178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.607004</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Stock1    Stock2\n",
       "2020-01-01  0.649239  0.956359\n",
       "2020-01-02  0.100613  0.749957\n",
       "2020-01-03  0.883636  0.899354\n",
       "2020-01-04  0.867854  0.374508\n",
       "2020-01-05  0.867854  0.913792\n",
       "2020-01-06  0.303548  0.540079\n",
       "2020-01-07  0.207133  0.704109\n",
       "2020-01-08  0.486524  0.704109\n",
       "2020-01-09  0.096799  0.667178\n",
       "2020-01-10  0.607004  0.134332"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace with next value\n",
    "df.fillna(method=\"bfill\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stock1</th>\n",
       "      <th>Stock2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-01</th>\n",
       "      <td>0.649239</td>\n",
       "      <td>0.956359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-02</th>\n",
       "      <td>0.100613</td>\n",
       "      <td>0.749957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-03</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.899354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-04</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.374508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-05</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.913792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0.303548</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.207133</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.486524</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.667178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.607004</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Stock1    Stock2\n",
       "2020-01-01  0.649239  0.956359\n",
       "2020-01-02  0.100613  0.749957\n",
       "2020-01-03  0.883636  0.899354\n",
       "2020-01-04  0.000000  0.374508\n",
       "2020-01-05  0.867854  0.913792\n",
       "2020-01-06  0.303548  0.540079\n",
       "2020-01-07  0.207133  0.000000\n",
       "2020-01-08  0.486524  0.704109\n",
       "2020-01-09  0.096799  0.667178\n",
       "2020-01-10  0.607004  0.134332"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace with specific value\n",
    "df.fillna(value=0) # replace by zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Stock1</th>\n",
       "      <th>Stock2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-01</th>\n",
       "      <td>0.649239</td>\n",
       "      <td>0.956359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-02</th>\n",
       "      <td>0.100613</td>\n",
       "      <td>0.749957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-03</th>\n",
       "      <td>0.883636</td>\n",
       "      <td>0.899354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-04</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.374508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-05</th>\n",
       "      <td>0.867854</td>\n",
       "      <td>0.913792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0.303548</td>\n",
       "      <td>0.540079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.207133</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.486524</td>\n",
       "      <td>0.704109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0.096799</td>\n",
       "      <td>0.667178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.607004</td>\n",
       "      <td>0.134332</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Stock1    Stock2\n",
       "2020-01-01  0.649239  0.956359\n",
       "2020-01-02  0.100613  0.749957\n",
       "2020-01-03  0.883636  0.899354\n",
       "2020-01-04  0.096799  0.374508\n",
       "2020-01-05  0.867854  0.913792\n",
       "2020-01-06  0.303548  0.540079\n",
       "2020-01-07  0.207133  0.134332\n",
       "2020-01-08  0.486524  0.704109\n",
       "2020-01-09  0.096799  0.667178\n",
       "2020-01-10  0.607004  0.134332"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace with specific value using a function\n",
    "df.fillna(value=df.min()) # replace with minimum value of each stock"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing quantitative analysis and statistics we are frequently applying calculations on filter and aggregations on data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a simple example dataframe: in the below dataframe we may be interested in the sum of data per key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>data</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>c</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key  data\n",
       "0   a     1\n",
       "1   b     2\n",
       "2   c     3\n",
       "3   a     4\n",
       "4   b     5\n",
       "5   c     6"
      ]
     },
     "execution_count": 260,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\"key\":list(\"abcabc\"), \"data\":range(1,7)}\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This could be calculated easily with what we have seen before. We could simply apply three filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 7, 9)"
      ]
     },
     "execution_count": 263,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1 = df.loc[df[\"key\"]==\"a\",\"data\"].sum()\n",
    "s2 = df.loc[df[\"key\"]==\"b\",\"data\"].sum()\n",
    "s3 = df.loc[df[\"key\"]==\"c\",\"data\"].sum()\n",
    "s1, s2, s3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is tedious (although we could have made the code less repetitive and verbose using a for-loop) and sometimes not practical.\n",
    "\n",
    "A very powerful feature of pandas is the **groupby** operation / concept which we will introduce in the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "key\n",
       "a    5\n",
       "b    7\n",
       "c    9\n",
       "Name: data, dtype: int64"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"key\")[\"data\"].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way this function works is illustrated in the following figure\n",
    "\n",
    "![groupby](https://www.dropbox.com/s/rpg9grsod2dhrh1/pdgroupby.png?dl=1)\n",
    "\n",
    "(note: the example is taken from [Jake VanderPlas' book](https://github.com/jakevdp/PythonDataScienceHandbook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"split-apply-combine\" mechanic can be very helpful when grouping and aggregating data. \n",
    "\n",
    "Let's take our `car dataset` and apply some more methods. For example let's say we want to understand if \n",
    "\n",
    "- mean `mpg` differs \n",
    "- when conditioned on \n",
    "- origin and number of cylinders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 332,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>27.0</td>\n",
       "      <td>4</td>\n",
       "      <td>140.0</td>\n",
       "      <td>86</td>\n",
       "      <td>2790</td>\n",
       "      <td>15.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford mustang gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>44.0</td>\n",
       "      <td>4</td>\n",
       "      <td>97.0</td>\n",
       "      <td>52</td>\n",
       "      <td>2130</td>\n",
       "      <td>24.6</td>\n",
       "      <td>82</td>\n",
       "      <td>2</td>\n",
       "      <td>vw pickup</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>32.0</td>\n",
       "      <td>4</td>\n",
       "      <td>135.0</td>\n",
       "      <td>84</td>\n",
       "      <td>2295</td>\n",
       "      <td>11.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge rampage</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>28.0</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>79</td>\n",
       "      <td>2625</td>\n",
       "      <td>18.6</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>31.0</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>82</td>\n",
       "      <td>2720</td>\n",
       "      <td>19.4</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "387  27.0          4         140.0          86    2790          15.6    82   \n",
       "388  44.0          4          97.0          52    2130          24.6    82   \n",
       "389  32.0          4         135.0          84    2295          11.6    82   \n",
       "390  28.0          4         120.0          79    2625          18.6    82   \n",
       "391  31.0          4         119.0          82    2720          19.4    82   \n",
       "\n",
       "     origin             name  \n",
       "387       1  ford mustang gl  \n",
       "388       2        vw pickup  \n",
       "389       1    dodge rampage  \n",
       "390       1      ford ranger  \n",
       "391       1       chevy s-10  "
      ]
     },
     "execution_count": 332,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(fpath)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "origin  cylinders\n",
       "1       4            28.013043\n",
       "        6            19.645205\n",
       "        8            14.963107\n",
       "2       4            28.106557\n",
       "        5            27.366667\n",
       "        6            20.100000\n",
       "3       3            20.550000\n",
       "        4            31.595652\n",
       "        6            23.883333\n",
       "Name: mpg, dtype: float64"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mpg_conditioned = df.groupby([\"origin\", \"cylinders\"])[\"mpg\"].mean()\n",
    "mpg_conditioned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the *as_index=False* statement to return a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "mpg_conditioned = df.groupby([\"origin\", \"cylinders\"],as_index=False)[\"mpg\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the groupby method to quickly explore our dataset when combining with other methods. Most importantly\n",
    "\n",
    "- aggregate\n",
    "- transform\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Groupby and aggregate**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">mpg</th>\n",
       "      <th colspan=\"2\" halign=\"left\">cylinders</th>\n",
       "      <th colspan=\"2\" halign=\"left\">displacement</th>\n",
       "      <th colspan=\"2\" halign=\"left\">horsepower</th>\n",
       "      <th colspan=\"2\" halign=\"left\">weight</th>\n",
       "      <th colspan=\"2\" halign=\"left\">acceleration</th>\n",
       "      <th colspan=\"2\" halign=\"left\">year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.033469</td>\n",
       "      <td>6.440384</td>\n",
       "      <td>6.277551</td>\n",
       "      <td>1.655996</td>\n",
       "      <td>247.512245</td>\n",
       "      <td>98.376347</td>\n",
       "      <td>119.048980</td>\n",
       "      <td>39.897790</td>\n",
       "      <td>3372.489796</td>\n",
       "      <td>795.346690</td>\n",
       "      <td>14.990204</td>\n",
       "      <td>2.736020</td>\n",
       "      <td>75.591837</td>\n",
       "      <td>3.660368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>27.602941</td>\n",
       "      <td>6.580182</td>\n",
       "      <td>4.161765</td>\n",
       "      <td>0.506975</td>\n",
       "      <td>109.632353</td>\n",
       "      <td>22.693220</td>\n",
       "      <td>80.558824</td>\n",
       "      <td>20.157871</td>\n",
       "      <td>2433.470588</td>\n",
       "      <td>491.813867</td>\n",
       "      <td>16.794118</td>\n",
       "      <td>3.087813</td>\n",
       "      <td>75.676471</td>\n",
       "      <td>3.422669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30.450633</td>\n",
       "      <td>6.090048</td>\n",
       "      <td>4.101266</td>\n",
       "      <td>0.590414</td>\n",
       "      <td>102.708861</td>\n",
       "      <td>23.140126</td>\n",
       "      <td>79.835443</td>\n",
       "      <td>17.819199</td>\n",
       "      <td>2221.227848</td>\n",
       "      <td>320.497248</td>\n",
       "      <td>16.172152</td>\n",
       "      <td>1.954937</td>\n",
       "      <td>77.443038</td>\n",
       "      <td>3.650595</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              mpg           cylinders           displacement             \\\n",
       "             mean       std      mean       std         mean        std   \n",
       "origin                                                                    \n",
       "1       20.033469  6.440384  6.277551  1.655996   247.512245  98.376347   \n",
       "2       27.602941  6.580182  4.161765  0.506975   109.632353  22.693220   \n",
       "3       30.450633  6.090048  4.101266  0.590414   102.708861  23.140126   \n",
       "\n",
       "        horsepower                  weight             acceleration            \\\n",
       "              mean        std         mean         std         mean       std   \n",
       "origin                                                                          \n",
       "1       119.048980  39.897790  3372.489796  795.346690    14.990204  2.736020   \n",
       "2        80.558824  20.157871  2433.470588  491.813867    16.794118  3.087813   \n",
       "3        79.835443  17.819199  2221.227848  320.497248    16.172152  1.954937   \n",
       "\n",
       "             year            \n",
       "             mean       std  \n",
       "origin                       \n",
       "1       75.591837  3.660368  \n",
       "2       75.676471  3.422669  \n",
       "3       77.443038  3.650595  "
      ]
     },
     "execution_count": 300,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"origin\").aggregate([np.mean, np.std]) # group by origin and calculate mean and standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>horsepower</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>origin</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.033469</td>\n",
       "      <td>230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>27.602941</td>\n",
       "      <td>133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30.450633</td>\n",
       "      <td>132</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              mpg  horsepower\n",
       "origin                       \n",
       "1       20.033469         230\n",
       "2       27.602941         133\n",
       "3       30.450633         132"
      ]
     },
     "execution_count": 302,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"origin\").aggregate({\"mpg\":np.mean, \n",
    "                                \"horsepower\":max})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Groupby and transform**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {},
   "outputs": [],
   "source": [
    "def df_plus_mean(dframe):\n",
    "    return dframe + dframe.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 333,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>47.033469</td>\n",
       "      <td>10.277551</td>\n",
       "      <td>387.512245</td>\n",
       "      <td>205.048980</td>\n",
       "      <td>6162.489796</td>\n",
       "      <td>30.590204</td>\n",
       "      <td>157.591837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>71.602941</td>\n",
       "      <td>8.161765</td>\n",
       "      <td>206.632353</td>\n",
       "      <td>132.558824</td>\n",
       "      <td>4563.470588</td>\n",
       "      <td>41.394118</td>\n",
       "      <td>157.676471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>389</th>\n",
       "      <td>52.033469</td>\n",
       "      <td>10.277551</td>\n",
       "      <td>382.512245</td>\n",
       "      <td>203.048980</td>\n",
       "      <td>5667.489796</td>\n",
       "      <td>26.590204</td>\n",
       "      <td>157.591837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>48.033469</td>\n",
       "      <td>10.277551</td>\n",
       "      <td>367.512245</td>\n",
       "      <td>198.048980</td>\n",
       "      <td>5997.489796</td>\n",
       "      <td>33.590204</td>\n",
       "      <td>157.591837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391</th>\n",
       "      <td>51.033469</td>\n",
       "      <td>10.277551</td>\n",
       "      <td>366.512245</td>\n",
       "      <td>201.048980</td>\n",
       "      <td>6092.489796</td>\n",
       "      <td>34.390204</td>\n",
       "      <td>157.591837</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           mpg  cylinders  displacement  horsepower       weight  \\\n",
       "387  47.033469  10.277551    387.512245  205.048980  6162.489796   \n",
       "388  71.602941   8.161765    206.632353  132.558824  4563.470588   \n",
       "389  52.033469  10.277551    382.512245  203.048980  5667.489796   \n",
       "390  48.033469  10.277551    367.512245  198.048980  5997.489796   \n",
       "391  51.033469  10.277551    366.512245  201.048980  6092.489796   \n",
       "\n",
       "     acceleration        year  \n",
       "387     30.590204  157.591837  \n",
       "388     41.394118  157.676471  \n",
       "389     26.590204  157.591837  \n",
       "390     33.590204  157.591837  \n",
       "391     34.390204  157.591837  "
      ]
     },
     "execution_count": 333,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"origin\").transform(df_plus_mean).tail() # add mean per column per groupy to columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.033469387755098"
      ]
     },
     "execution_count": 335,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"origin\"]==1][\"mpg\"].mean() # mean of 20.033 is added for mpg in first row, because it is origin == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas dataframes can be plotted quite easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f9918230f10>"
      ]
     },
     "execution_count": 337,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOx9d5wkV3X1eVXV3TM9eXdng3ZX2l3lLKGVCEKAEhaIJJKNP2OCMQ7AJww2CCxbYGzDZ6ITMiIYYcAgTJCREEGRJCTtKq60Kwlpc5zdndjTsep9f1TdqlevXqWenpnumTq/3/52pqe7+nWo+06de+69jHOODBkyZMjQudDmewEZMmTIkGFmyAJ5hgwZMnQ4skCeIUOGDB2OLJBnyJAhQ4cjC+QZMmTI0OEw5vLJli1bxtetWzeXT5khQ4YMHY/Nmzcf5pwPh/19TgP5unXrsGnTprl8ygwZMmToeDDGdkb9PZNWMmTIkKHDkQXyDBkyZOhwZIE8Q4YMGTocWSDPkCFDhg5HFsgzZMiQocORBfIMGTJk6HBkgTxDhgwZOhxZIM+QIcO849BkBT99/MB8L6NjkQXyDBkyzDt+/4v34Z3/tRm1hjXfS+lIZIE8Q4YM846dR0oAAI5s0E0zyAJ5hgwZ5h00qCwbWNYcskCeIUOGtkEWyJtDFsgzZMgw76D4bWWRvCkkDuSMMZ0x9hBj7Bbn948wxvYyxh52/r189paZIUOGhQwaAp8F8uaQpo3t1QC2AugXbvss5/xTrV1ShgwZFiusLI43hUSMnDG2BsCVAL40u8vJkCHDYgTFb54x8qaQVFr5HIAPAJBNnu9mjD3KGPsKY2yotUvLkCHDYgHF74yRN4fYQM4YewWAQ5zzzdKfrgdwPIBzAOwH8OmQx7+TMbaJMbZpZGRkpuvNkCHDAkamkTeHJIz8QgCvYoztAPAtAJcwxr7OOT/IOTc55xaALwK4QPVgzvkNnPONnPONw8OhI+cyZMiQwRfIaw0L6665Fd+4L3LKWQYkCOSc8w9xztdwztcB+D0Ad3LO/4Axtkq421UAtszSGjNkyLBIwDlw15OHUGtYmKzUAQCf+smT87yq9sdMfOT/xBh7jDH2KICLAfxFi9aUIUOGRYote8fxtv98AB+75QkwxgAgK9pPgFSBnHN+N+f8Fc7Pb+acn8k5P4tz/irO+f7ZWWKGDBkWC6pO06xf/vYwmHNbp8rmtz66H+/+5oNz8lxZZWeGDBnaBtM1EwCwd7Tc8ZbEd33zQdzy6Nzw2yyQZ8iQoW1QqjYAADXTgul4ETszjM8tskCeIUOGtsGUE8gBuMnOLJLHIwvkGTK0IaxFWhkzXfMC+cGJKoAsjidBFsgzZGgzbN0/gQ0f/hHu2nZovpcy5yhVTffnmmknPjtVIyfMxfqzQJ4hQ5vhoV1jAICfLMIZlqK0Uq3bQb2zwzhcrX82kQXyDBnaDLpzVs5FAGg3TFW8QF5pECOfr9W0BmbGyDNkWHzQNfu0XIyBvFQLMvJOhzUH86SzQJ4hQ5vB0OxSmMYiDOQ+aYUYeYeLKxkjz5BhEUJzAvlcBIB2Q0kI5BXSyDv8bcg08gwZFiGIkZtmh0ewJiC6VjxG3tnIAnmGDIsQ+mJm5LWGu5FRIO/0SJ4F8gwZFiFcRr4ANPL7nj2Cr927I/H9S9UGBrpzAIBqg+yHnf0+zMWwjDTDlzNkyDAH0BZQsvN3b/gNAOAPn78u0f3rJkd/dw5HSjVU6wvEfpgx8gwZFh80pw/3Yi3T7w8w8s5GFsgzZFiEoJLuxlwYkOcIaTal/i5bKPAYeWeH8rmQVrJAniFDm4FO+4WgkRPKKYp7PEa+MFwrcyGRZYE8Q4Y2AzHQhRTIaWBEEhRzOgBPWul0zIVEljiQM8Z0xthDjLFbnN+XMMZ+xhh72vl/aPaWmSHD4gEpKgspkJdTBPIuN5AvkGRnm0krVwPYKvx+DYA7OOcnArjD+T1DhgwzhOVq5B0ewQRM1xvxd3JQMOywRBp5p6Ntkp2MsTUArgTwJeHmVwO40fn5RgCvae3SMmRYnFiIGnkaaSVnaGAMqCwYaWX2nyMpI/8cgA8AEJe0gnO+HwCc/5erHsgYeydjbBNjbNPIyMiMFpshw2LAQtTI00grOmPI6drCYeTtIK0wxl4B4BDnfHMzT8A5v4FzvpFzvnF4eLiZQ2TIsKhA8XshlOhTlWoaRq5pDHldWzDJTnMOKHkSRn4hgFcxxnYA+BaASxhjXwdwkDG2CgCc/xffXKoMGWYBlsDIv7t5D+7ffnSeV9Q8cs6UDHEWZxx0xmDoDJWFwsjbQVrhnH+Ic76Gc74OwO8BuJNz/gcA/hfAW5y7vQXAzbO2ygwZFhGIiDdMjvd/5xG88Qv3zu+CZoC8QYE8hbSi2RvAwmHkbSCtROATAC5njD0N4HLn9wwZMswQ1gLSyD1G3oy0kozK/t0Pn8Apf3Mbtu6faGqNs422q+zknN/NOX+F8/MRzvmlnPMTnf879/ovQ4Y2Al9AGnletzXyclPSSrLgv2nnUVTqFr7+m51NrXG20e6MPEOGDLOAhcTIGUuf7NQ127WS9OVT29s0zpi5RFu4VjJkyDC38DTyzk/2UeOvNIGcOfbDxM/hTFJqV9/5XEx6ygJ5hgxtBmLkC4CQu0E2nY8cyDmSTKLncDaLjJFnyJChbUDnfS1hsq+dQW0GSmk0ci0lI3eeI02HxWaw43DJp9v/9tAkxst17Bmdxs4jpVBNfy6aZmUTgjJkaDMQI68tBGnFeQ1pNiVNY6kYOeUSZtN3blocL/nU3bjklOX4ylvPx/bDJVz2mZ/77kN/Czw2Y+QZMiw+qE77Tk181q30m5LerEbuMOJv3rcLP96yP8Uq41F31n/nNrvucf94OXCfu59U10RmrpUMGRYhVL7jNJWR7QRi5PUUgVxLKa2YgrTCOceHv/8Y/vTrD6ZbaAzk9auq7uVPzelO0H4+8gwZMsw+VASuXRN5MnYcLuEb99l+bsvi7mupN5IHM5uRN5fs3H00yJRbgYbkPEkil5D1Un7sbCAL5BkytBlUMypLHRLIr/r8r/DX398C0+K+fur37ziKddfcisf2jMceQ9cYjCaSnZW6iYf3jLnHaCWCjDx5cM4YeYYMixCq875TpJXR6ToAOIE8qD/85tkjvt+rDRPrrrnVdxuV6IuICpyeRm5hy157o1g71J1+8SG47uYtuOAf7/DdlkQqoq2kLZpmZciQYW6h1sg7g5ETGpaFukJSkJnyRDm4QamklSgpwxQSqkdLNQBQPnezuPHeYOl/JYELx1FWMtdKhgyLESry2XmBnCvdGnIg5wqPjq4hIK1EOT9ECWe8bF8RzOYVTN20UFF8HnK8Zg4nb6vhyxkyZJgbqDTy6WpnSCsE0+TKFgNyIFcxZ40FpZWoQG5alst+xx1pp1XFQarXMF6uJ2sHQIw8C+QZMiw+qDXyzmPk5CE3hOAtB3JVoZCuKAiKGkTdsDh6C3Zt41jZllYqdaslTHjvWNAFMzZdS+Qi8jTyLJBnyND22LJ3HH/1nUdadgmt1Mhnufy81WhYlstmu/O6e7vO4gO5pnCtRL23psXR5wRyklaA1jTRevZwKXDbn3/jQXzyJ0+GPuazP3sKtz9xcE418qxEP0OGGeKPbnwAByeqeN9LT8KqgZm7JVQxSyW3zBfqpoWpSgNDPfnQ+zRMz35YzOuYrNjSkBaQVhSMXFHZGcnITY7eLgMYB8amvUA+XTNRzM8sxI0LxwOAwWIO4+V65Hr++Y6nAQDdOXsDyxh5hgwdADpPZbbZ/PGCJ347lei/76ZHcO7Hfha5uZgWd22BYjCV7eGqKUC6xtyBFIQoL3bDslxpRTxeK4qo5Pf9hjdvxMvOWJXosfR1aItkJ2OsizF2P2PsEcbY44yxjzq3f4Qxtpcx9rDz7+WzvtoMGdoQ7ona2hoU/3O0TxzHDx/ZByB6TQ2Lu2y7KEgrMgFXSSuMBV0rYQyYqkd7u3KBv7Ui4SnLIjmdYagYfiWi2tzaRVqpAriEcz7FGMsB+CVj7Dbnb5/lnH9q9paXIUP7g05UVf+NZqBicO0krRDqpgVd05V/a1iWy2b9gdz/JiWVVsJYLb33vQXvOfq7DExUGi1h5PLz5nQNQz3BTYOgcuG0BSPnNqacX3POv/b7Vs0zKnUT37p/V1uecBlmF3SiqioZmzqe4ivUTtIKIdLbbXqVnV05L8jKzDqNa4Vzjpse2I2SYMWkNZC0AgCDDmNuhdNHZtOGztzjqyAmWEkOitLTW4VEGjljTGeMPQzgEICfcc7vc/70bsbYo4yxrzDGhkIe+07G2CbG2KaRkZEWLbv98InbtuGa7z2Gu59cuK8xgxp0nrYq2Kr04DaM45HNoEyLu+y0R9DI5fdI1d5W1f3QtCxs3jmKD3z3UfzNzVu8NTjHE4PrUNFmzEmHN0dByciL4YxcLBSifb1tKjs55ybn/BwAawBcwBg7A8D1AI4HcA6A/QA+HfLYGzjnGznnG4eHh1u07PYD9SduxZcnQ2eBglOrArnqKHPReCkJxCvOqCsQ234YlFbk4J9UWjEtbzPbdWRaOJ79eBrADAADTlB/21cfwLYDE5GvJw4ym85pWqRGLg63cEf2tQsjJ3DOxwDcDeAKzvlBJ8BbAL4I4IJZWF/HgE7iVnddy9D+IMbVskCuYuRtQsknKp6sIQa5yUodb/iPX3t/E6QV0UcuB/8w10pQWrFcO58omdAa+roM1yUiMuYPfvexZC8sBPJnmjMYBiMYuZhgtdzvxYyWkAhJXCvDjLFB5+duAJcB2MYYEz04VwHYonr8YgFdRqZpiJ9hYYCCbKsuodtZWjk8VXV/Ftn0f/5qBx7YMer+7rcfBjXyLXvH8drP/woTZb9PG7BL9IPJTs/OJwZLCrSGpqFg2I8RGfMju8dmdJUsfxZGLCMXA7n6GLOBJK6VVQBuZIzpsAP/TZzzWxhj/8UYOwf2leAOAH8ye8tsfxDTMFI0xM+wMOAmtVrUcc/its1NdEC0i7QyMukFcpGt/ugx/2i1hiUyckEjd17Tx255Ag/uGsOy3kLgOVTDlxuW5V7tisnOhhvI7cdU6laAMVfrli/hGofth0tYv6zHeY3+v+V1zbcxyVBZHluVBI9CbCDnnD8K4FzF7W+elRV1KOgkbpPzLcMcotXJTs5pukz7BfJRp00s4LfayRKJ2MZWxcjzDntWOUvs7ofBgiDuPIVoK6SNwdC94D9UzOPPXnI8Jit1fP03u1BpmBhAuBwi4q4nD+Ft//kA/u33z8UrzjomyMh15k7+UUHF/ttCWsmQDPQFnYvdN0N7olXSCuccYqpF11jbBHKxl4m4ccmbWMPkIT5yvwQ5pejqqOp+WDe5K2GJfWfofBN19a6chg9ecQrOWWsb6SYrDXzslieUA5NlPHPIdlo/tGtM+bpog3nwby7HGzeuCTxeFcjbLtmZIRyUPZ+L+XwZ2hP/s3kPHto1Gn9HCd9/aA827/QeZ3EOTWB9OmNto5GPCYFc1MjlgGcKlZ3dCh85Bd2SIpCrpJVqw1K6gxqCRm5o9mMKhu78b/9+22P78eVfbscnfxze6IpAVwrkbw8kO53nWNKTR7+impRcKyJpbxv7YYZ40GXkXJj/M7QnvnnfLlz1+V/H31HCX3z7Ebzueu9xFvdX+zPWPq6VMEYuX4k2LLFplugjt+9HgVolrWiMBaSVcs1USldEnHSNuUGYAjjp4tsOTAJIlr+iK4GqU9gjP6fY9Iuez7dOh5GLfXcyRt5BoA88ySy/DBmiIDNyjbWntCIGb/lrb2vkTq+VgkIjj5BWdMXMzmrDVDJbU0h2Ut/zQs4f0B/YcRSA3+MdhgAjj3jfVQ41klbEgJ8x8g5C3cqklXbAtgMTWHfNre4Q3iSwLI5X/OsvcJvkvJgLqDzjdrLT+93WyOdwUREY90krokYuMXLTq+wUy+cbkk1XNZJNlFZIlqnUQxg5aeRCspOkFWLkhxynzX3bj+AXT0dXXpMzhipOo9h0UkbeNiX6GeJBX9B27ImxmPCzxw8CAG7bkjwoVxomtuydwJMHJ2drWaFQ2dU45z5Gx1j7fK8mQqWVlBq5Yb++sFFvJIP0OGx+6/5JPLHfq9K0JL08p2mu7i5LLISDE1W8+cv3R/ZDovO4Wvc0ciOkyE++agDUGvlcSCvZYIkWgZKd9cy1Mq+gAJhG4aKTbz50aJW0QBr5bVdfhJHJKt7z3w+1TTO28XLd7S4oyojye9ewOOqOPNFTCNfIVdCYFySpKvSrv97hu0+p1kBfV87dGFQJ0jDv+ES5gYGQ6kxi+MTITc7RldOVn5NcfQp40krDd7WSMfKOgWs/zKSVeQVdGqcJfHTyzYWWKaNUVdjVHI381FX9eNFJw20lrYxN190inihG3jAtd2an0kceEcjFoFzMqbkmTRxqCD5yYvG0gciMnLAvwoZIVwjki7csHtp2I6c4Pn2XxGZgc5HfyAJ5i9DIkp1tAdIm07AgN5DPw0enst9xwFd0orH52WRUGC/XsbTXLlGvR7BOGiyR1zWfA0f2katgz+y0H2XoTKlFu8xX8JG//Ey7a8iaoSIAL+kJAFdfeiJeeMIyAMA+xUBlAp2/lOxspJRWVD3QM0beQaAvQLtombOJasPEB/7nERycqMTe92iphvd9+2FMVoI9NWYDrrSSipE77Et4zP7xMv7qO4+4NrTZgjKQSwVBjLG2kFYsi2OiUsfSnnhGbjrSiqEzFMVkp6K53AnLe/GW5x/n/i52P9Q15tPYCVXJ521oDG9+3nF45LqX4tildiDvMrzHHbe0iE+/8WwAwL7x8O8tMXwK5JaUrxCh3GAUTcCyZGcHobGIfOS3P3EIN23ag7/74ROx9/3UT5/E9x7ai1senRtHCMmWafTuisIz/Lc3P47vbN6Dnz91uKXrk1FSuDYsC8GCoDa40CvVGuAc7tBlYsOq95p85DldQ2/BwAN/fRmOH+5xS+rFRywp5vHRV5/h/i7aDxlj6MoFw5TImOkxjDFfO1uRkfcUDCzrLcDQGPZHMHKSRGoCMQubxfq8DUtx2qp+aV3BjX8urtKzQN4i0Jd6MUgraTS/vaP2SRPVMa6V0Jtg5KJDwb3NCRRhl9UE1YSbNBA1cpEFirFjNqUVu0thstdAZIUYskxe3vqCdbj45GH3bzXTcpn1cF8BBUN37ysGf3KwEETXis6gZOQ16QpYJdUUBEbekzegawwr+ruwPwEj9wqCwltTr+jvwiffcJbvNtVbmcS/PlNkgbwFsIRpKItBWnFfYYJGjweck2aupAHGmnCtOCetuEFRwiyqGpBzjpOuvS3070kgSiukr3L4GTmbxYKgl3zqLpz397cnui9tJiQpEHmh7/zKgS585a3nO7dZqDcsn7PD0JnrWhFfj6w160Jxj8aYUtqgzVcl1YjHIVBR0rK+Ao4Ijb9kEBEr10RpJfTugeeV/fTA3AybyeyHLYA4p0/li11ooKCcpGEvOQQqs6w1E+jdTyOtVOuKQJ6gd04rYqtoayvX7S59MiPXNTZrXTV3H41vJEWwpEDukhfndp3Z8kZOZz5phaBrzA284hWGrDVrzN688roGTWPKz7JmEmNOduVERUnFnI6yQs4ikH2YNtgoaQVA4G8qaXUuAnnGyFsA8dIp6WVqp+KJfRN4dI9dNRnVzhOwAz7ZxIjhAMDOIyX8+pnZ0Z5Nwf+bFBVXWvFuc5lZxEnYCpYs9hqhKkfOIZXot8eVHpFNsvW5TaxMPyvWNQbTImlFYOQaU7Z7XtLj70lO3ytDZ9AZU36WJEPVzXBGLoIskN15PXIoM62v7FSSmhHJTgCBv8mfU8HQsOPIND53+1PYfrgUucaZIGPkLcDotHepttCTnS//l18kvq/INkVW8uJP3g0A2PGJK1u2LoKrwTbhIxeZHwWIKDbVio9alFYowAQ08hm2sX3n1zaBA/jiH24MvQ/nPHZjpoBacBk5yRt+GcrQNNRN7kgrfkau6mC4vC84XAKwdW9N8zYKEUHXSjQnpQHQ3Xk9cnMWc1xTlQZM07Yfnr12EEt7gnkemZHLgbyY11FtWPjc7U/jvOOG3IEVrUYWyFuAHcJOu5j6kceNJxWHDUSdPK0EBeBUrhVFQdBcMXJZWgFUjHxm0spPnzgYe59SzfT1RFGB3tMAI3cWR2u2A7YVkFZyuoZSw3694nu3vD8ikIcw8qrkWonrbEgaeTGn49mREs7+6E9x87suxDopsIrS6Fi5ZjNyxnDzuy5UHjeokcuB3MDotG297Yl5f2eCJDM7uxhj9zPGHmGMPc4Y+6hz+xLG2M8YY087/w/N2irbHHTJlDe0RVXZGaeRi4G8OkeBvOFKK8kfUxGq+NzjWMTIwzfmVksrb/iPezFRqatdK7N8pTcakQAkmG4gd1wrErsmnZo08rokrYiM3PIx8i7l8+V0ZgdylUZOjNxMppHTmkliGS/X8YOH9wbuJzLy0el6ZGUnEJRW5CvynoLfOTNbSKKRVwFcwjk/G8A5AK5gjD0PwDUA7uCcnwjgDuf3RYlnD5cwVMxhuLewKJKdSVGbB0buBZfkV0YqRk5rn21pRa4EfPLApJKRz3aZ99h0fMFWwLVC9sMwjbxhwRAYuSEkO8X3bkUEIxeDvwiVjzwJxPmhqoKehi+Q24w86thx0or4fGJQbzViAzm3MeX8mnP+cQCvBnCjc/uNAF4zKytsc0xVG/jJlgNYv6wHhs4WlbQSp6mKVZFz4aUFhKZHjWaSnUEfeXQgn3lwrZkW+rq8kz2nazYjF+6jNTkh6MB4JTGTF/M8YSC3kl18E7QfUsAjjbxhcZ+10KeR83hGbjiMXJV3SquRE8S+L6LPnFA3varasekaTMvfG16G/LRyHUmP8Hxx0tVMkOjVM8Z0xtjDAA4B+Bnn/D4AKzjn+wHA+X/5rK2yjXHdzY/jSKmGk1f2+exViwGx0kp9Hhi5VNCRBBWF/ZBaCqh6ZxB4C/amasPEsJDsMy0ryMi19JvG0VINz/v4HfiHW7dG3o+kjySBnGKUrjHknGANCPZDza+Ry9KKoWnu+SHWFSzrVReLLespYLCYay0jz4mBPBj+6qblbiyjJVvmipRWpCAvB3Jx4yjOs7QCzrnJOT8HwBoAFzDGzoh7DIEx9k7G2CbG2KaRkeim7p2IfWNlFAwNH3r5qchp2oK3H/oQc+6IHeDmwksLeMmqNBWX8livasP0GHnEhkDB9a0vWNfMUgHY61zR14V/vOpMAM6QYdlH3oS0QlbGH8UMy6CS9kTSiuUlNSlYi7cTKzYcjVyWVsJcK0ZIA63r/+A5+NtXnoYLnWZX4nHIR06BU9VSFgA+88az8eGXn+L+3i0EVpWuXjctLOnJgzGbkTfMdNJKLRDIo6WcViHVkTnnYwDuBnAFgIOMsVUA4Px/KOQxN3DON3LONw4PD89wue2H0ekaLjpxGP1dOfsLvIg0chYTyUVGPleBnIJLNU0gl3zk5H0HkiU7Nww3bymrNiwUchpOWN7rrIHD4qrKznTHpaA6EdOsrK8reSC3BOZt6MybU+tq5PTczB0skZc0cgq8Frelhl984OLQ51vaW0B/Vw7/8nvn4vefe6x7e17X3M+MWH+YzPfa56zBO190vPu7yJDloAvYDL+Q0zDQnbOTnTy6IEhOdlbrli/wz6Yu7ltH3B0YY8OMsUHn524AlwHYBuB/AbzFudtbANw8W4tsZ4yX6xhymtQbuub2YM7gVd8ZGlMGxNko26+7rDpFslMq0RfllGj7of1/XK4gCrWGhYKhuSd/3bQcT7d3H62J4cskd0QVvwDeRdVUNTkj1zUvWANigNfc/0kjD3WtcI7BYg5rlxRjn7c7r+PkFX3u74Wc5gbhuskjW+LK8AVyxXek7vSHGSrm7WRnjGtF/lvNtHySzWzKKSKSPMsqADcyxnTYgf8mzvktjLF7AdzEGPsjALsAvGEW19m2GJ2uud3gDOFyczEg7rUSaxrozikDomnxRJPN04CkLVUXujDIyU5RxoiyTdJGlFCeVaLasJA3dDfgmRYP9FrRmygIShr46bhJcju0OTDGYOiam+x0vdyC/ZB6rfhcK7rgWolJIsoQA2Ze19wgXJOKjuIgukhUm33d5OhyGPnYdB0mD7Ju37pkjdzZmGkDFTeO2UQS18qjnPNzOedncc7P4Jz/nXP7Ec75pZzzE53/j87+ctsLlbqJSt1ydUb70rFzGflje8ax7ppbfQVOUQh7rXduO4h119zqDr0d6M4ppZWkZfRHpqpYd82tuOepYI7lBR+/A//0423u716yM35D/fVvD2PdNbdiz+g0AC+oifptEkaeJiABwB9+5X6848ZN9jrrpsTIuTMhyLt/M02zkrpV6DNIIgnS5qAz5iu3pw1dE5KdDYujJrFlQ9OEzTJ5gtJ+rBDIDc39fOWEahziGHnDZeQ5jE7XbB95xOFl14rNyOcmwelbx5w8ywIFZfqpRWtO7+xk5/ce2gMAuH1rfCUgEN6y90u/2A4AeHDXKACgP4SRJ714eXzfhHPcZwN/2zdewefvfsb9nRhfkmTnV361AwDw1EHbXasK5Ek08rSM/OdPjbjvcc20kDc0N+CRRi5PCEp7oZd0k6TjJrHN0ttCGrnn2bdvFzsW/uLpwzg8VUVekla8gi2/fBQHkdnnDU3otZKSkeeiNfKayWFotrQyNl2Pl1YCrhUu9UFvE0aeIRyjJVtXJI280+2HdEIkfQ1hgZxOFkqgDRZzvsQnIannnk6kJKyxkSLZKZ//Ko9zlP3QEqQGQtqgXq37NfKGFdTI50JaSXIlKbpWcoKVUBy3BgAPORs4gEBBEB2DxyQRZYiMvGDoAiNPp5F3J2LkDIOORh5nP1T9bT408iyQzwBjDiMfdBl5Z7tW3KEMCYNAWMDvEsqgAb9GLiY4k7LMNOuqp/CRy5IIrYeeJ69rkfZDeilyFWYaVB1GbgibVSsqOxMzcldaScLIPdeKj11LGrn4MfmaZvlY/Aw0ckNMdjYvrai+I3VBWpmumSjXzchAzhgLXFn4pZWMkbc9qBnOUA9p5FpHV3bmBOdEEgeRR4wAACAASURBVMQxcjGQk0buG9ibMNgYAluNg5fstGJdMYEWpJwScfbvPQUdlQSMXDxMmuDEOXdcK7rLXBuWpdTI0/KDpJsx3S1RslN0rehCQRAxdWfRN7z5PPcxeamNraiRRyURZfgYua65Sei00kox5zFktWvFTsAPOgaGo1O12M9UvrLwM/IskM8JJit1fOh7j2I8gY9WhqyRt9pHfrRUw4e//xhuemA3vvfgnpYdNwyGoNMmQdjluCet2O9Pf5cdyDnnPl0y6fOkYeRiL4+44BTWJ4MCejFvpE52piHk9F4UREZuUUGQoC2z9FbNpHyCJJgk31ufa0VZEGSv+aWnr8Tlp62wb/MVBNlyDOfcca0kW6N8nJnYD6OklZs27cbesTLyDiMHgMlqIzYpK29I8qzQucCiD+Rf/MV2/Pf9u/GN+3emfiz1kqYeCmJToFbgn368Dd+8bxc+8N1H8b6bHmnZccMgOieSIOxyvFuQVvK6hu68DovbgUu08yWVC3QhyMWvKZnjBAjq2XKyc0lPHpOVRqje7GnkwbUmAem8BUlasTgUvVZmWVpJkuwUXSuCTKIqk6dzQgyyvU7ib6raiNWeZfhcK7qc7Ex+nLyh4d4PXYLjlhYDyc4P/M+j9nPpzDdjNk7Ll1+GKK3kdQ1vu3Advvtnz0+8xmaw6PuR00Rtcfp2UtSEExGwWUMrXSsyA7Ws6GklM4XnZU72GmohAb/LYeQWt98b+r1St3xJyKSMnGKS6v2QIQakUrWB/q7wzzVsugv9v2qgC4/tHcfR6RqW9QY79Hk+cpE9pwjkdTGQC4lmSXZgjKWaQSq+BhGq7w/dL8nm7XOt+OyHwUBObg0xyNJ7eHiqBpMHC6k2X3tZ6HdC1sirTfrIAWDVQDd68kaos8nQNAwWve9N3DkXJa1ojOG6V56ean3NYNEzcvI6p01SATajsq1Y9tuY01lLKzvlNdH8y9kCVebNlJF35fxWMfq9Ujd9J09yHdfP/NzHK1inuHZxQr0K8vurCuQAcHBCPXV9ptIKJdvyhuYWRpmkkQtnpq41Ia0o7q+6ognbJFUQXStiPkiWVgBPUhA3bmoONjJZdVwr/uMv7S1geX9IJ0Q52Skw8mZ6mBRyWqizaaraSMfIZWlFWM8Min5TIQvkTiAXR24lRS3QS6K1jFz+EszmzD/AS0wlTXaGSR1iDCkYmquZV+pmU4zcCgk2qseLt8V9pvIJSsGP/l850A3A+44E16VIdqa4YqLqv4KhKwqCZuhaiXlv3Ntc+2E610pXTkO5bmKiUsd3Nu9xbvfOhT4nkIsTkMRAPhPXSsEXyNNp5ARRnpGx++i0P5DHSDeyRCRKK80QxGawoAP5vrEy7n3mSOR9iG3FsTcVqnXTl9hotY98rgM5BaGkATbsRBDZY16QVsoyI5+hjqtaZ9200O/09y5VG6jUzdAOgHHSyjGDNjscmQgJ5M5yRIkgzWk74bh6VAVBvnWGTMmJAslOLxQ6B6p08FQl+oJrZbCYx2ipjmu++yh+7lTcihsjMXJxMyVpZWSy4lx1pNDIhWA91JPH0VINlbqZWiMniPKMjD2jZXTndZdZxzHygLQixYS5wIIO5F/6xXa8/asPhCarTIvjqDPiqlRLz8irDT8j7ynYE7pbxcplDXH/uPoSv1WgtymxtBKipYsvv2DoAiO3fN7dxEUrbn8O+fnV8sGAo2+WaiZufngv/vwbD2LXkenY53FlBueHFc5l/qHJMGklyMjTxFvqTFgwNPcYDdMK9FrRtPQzO+k1XH3ZifjbV5xm36bUzb3njYNYADVYzGFsuub7TorMtUfByIeKeegaw+GpGiwrXfGUKK2cf9wS1EwLD+8eQ920QtvgRkFk9QSaVPSOi9YDgKuTp3WtLOstuEVeS0N6rbcaCzrZWa43UK6b2D9RwerB7sDfxcvVpqQVpwUp4bilPTAtjj2j5cBQ12Ygf33SdsBLC8tlpEl95NFuDsBmPvQelWumj60nZeRmCCNXJjtNC4PdeexGGaVqA8+M2FcxquIPMXjpmjfkl6a2F/M6BrpzCaQV75NK8xlNlO3vXMHQwRhzZ11yyUeusfSDJXx6th5+pdUUI2e2q6NUM32vVwy2F6xbAgD4ndNXurfpGsPSnjxGJquwOEcu4VQfeizh/PVLwBhw37NHUTf9U4iSQiwqenDXKM5cPQCNMbxx4xq87UI7kNubUTW1j7y/y8DjH/0dcKAp2acZLOhATpdO20dKsxLIqw1/g5wNTvDefrjUmkAufX9mewCv62CIeJ6c0Ic6TFcV31efRt4w7SGB0vPFISwhpwo+dZO7DqSpagPPOoFcfV9v/cW8rpwKv7yvEJ/s1MTbUgTyiietAJ40p9LIk2568jqoChOI1sgTNc0SXCuuz1ro3S6ued2yHuz4xJWBYwz3FTAyVXWn0yeF2CVzoDuHU1b2Y9POo81LK45GvnX/BF77+V/jjy9aj4bUV4UslHGxWGbsuq41dZUwEyxoaYUCzvbDU8q/i+fGVDMauSStrBcCeSsQcFXM8gBeS2KkKoizEcNOfl8gzwn2w5opSSvp1iUHY1XQNC0vkE/XGu5nr9bTvduKeV2QcLwg2NdlhPb0VvVaSSWtlD1pBaBkOYdlQepHzlI3zXKHPTjFO0Dw/eOcu+dAPYWPXNOY25aCqnft9ccH1GW9BRyeqgaGZ8RBnsl5+jH92Lp/MnVlJ6Fg6Kg1LPdqa9uByUCDLKrKjNPy5QuLJO9Dq7GwAzkx8sO2Pjpda7izGAF/IJhuSiP3JzuX9OTR32W0JJBPVOqBRlPNxPGx6VpsJ8CK4z6gYJe0zUAYIxdvzuseIz8wUfExuKTP4zJlaeNQBeeGZaG/22ZSE+UGdh2ddm6PZuQ9ecOTVgQ2m4twN6h85Gk2298esjeZvFuHYA/v5gg24mrWfmgzcnXFrvhrmspOklYA/wSiOHcHYDeYG5uup6/slO582qp+HJ6q4vBUrTnXiiOtuJuTk1AWNwyx0C8KsrQyVwlOEQtaWiENjFjZtT/Ygn1jZXzrnXaVVRqrmvL4EiNnjNmXlEdmHsjP+shPA7c1I62c83c/w0tPW4Eb/nBj6H1e8++/wrYDk/jAFScDiE52ioGK9Fw5KSsGnYHunMvIP/rDJ3DMgOcTTipD0N0CPvIQlp3XNRTzOp48OCn0A1FPgyEUCzomnE1G1JfzwpAAGa7UILz+NAH3Bw/vAyAy8jCNPL20Ig5KDmPk4vufykeueYlA8buSpBhqsJjH2HQNfV1GqoAn3/fUVf3uz834yMmLLhYzyZZI6lyY1kc+H4x8QQfyuhvI7cD62J5x3xdW/O5ONamR90sVoct6wzXVmSLtyUzJvJ8+Ed1ffNuBSQBisjP8eTjneNXZx2D1UDeuv/sZO3Aaahvff771fJy1ZsDHcvYJLoek5h5Vn/CwdTYcF0NPwcDje8eF22OklZwBi1d8z0dBMDQX4AZ8ca1JXpEfeZ+0YgU1ci39zE7vqkLsVRNu30zjI9cYc6diiUgSmAe6c5ioNFA3rVQj8uRJUqeu8ka/NRM4bfuh6cuHyBOrqDo1VlphDN053W0JMR+MfEFLK3RJvHu0jErdxM4j0z7vKPdJKzPXyAE4tqz0DbiSIO3ldSnla6JzOeqkNi2OY5cUXR1a7U22C0IuPmU5lvYWfPKTfKxk61JLPmH2Q0Nj6Mnr0qYRvG9NYuTu8zgB3oiRVuiQYkBq5qopmOyUNfJmmmZ5ASqsV414yPSulWDrgyQBlZj82HQ9VTsDOTgOdOeE0XLNFQTVTe7zxssaORkZkvjIxQESbRnIGWNrGWN3Mca2MsYeZ4xd7dz+EcbYXsbYw86/l8/+ctOBTlTT4rhv+1G7aZNwUnol1s0x8pqkkQNwh7bOBtIGCZKLukICaeD4CaxodpLKO3lUMowlDUYohFz6JpVWaDnyssLK0A2dBbrOJdHI3WQnsTQnkIdXsHqM/E9ffHzg9iQ4bVW/MGGKOf3IuaSRpy8IEiWDsMEc4hVeMkburEezGagsaSQJYBTIj5ZqgSRhFGSrImPMbc7WbIk+AEw5cpq9iVq+oE3Hj9tvNM3/fWvXZGcDwPs556cCeB6AdzHGTnP+9lnO+TnOvx/N2iqbRN20sMS5BLzTGa3lc004X+TegoFStZGa9VQbViBIUUP6JIMN0iJtnRElcMXxVlGg1x+b3NOYa/lSBQCZ2YRdQqfttSJDDkycczdh1ZP3J6rCqkAJxbzuJTtJX3a83eE2S/t/TWO45mWn4H2Xn+S7PQl+dPVF7qZIOq08WEJXSCvbD5eweWf4mFxRMvAGPqg1co0l+yxENw9jLNBoLolUMthtn48NK2WJviKRSq6SZuyHy/vsXA3ls+w2CP7NSCxki1yb5p8EpKfZoVqEJMOX93POH3R+ngSwFcDq2V5YK1BvcJy0ohcAcMe2QwAgMXL7i9nXlYPF4z8wGTVFICdb1snX/hhPHZxsat1hVXZpNxqyVCYN5HQyh21CctMkQB3I41qU5oVy9CSQA9Duo9O47DP3YNdRf1KZrg5yOkOvU6ZPfn6lDCNsBD0FI+gj1+wrj3qotOLXyJP0TY/6DHO6hrqpGiwRLDT61zuexl9+59HQY1kqRi4nO53f84aW2rUCwG2FkAYDYlfBJke9Eeh73Yy0QnUlz4zYRgh69eLziM3eopDTNbe3TNhaZxup3gHG2DoA5wK4z7np3YyxRxljX2GMDYU85p2MsU2MsU0jI8Ep6LOJmmlhuK8LQ8Uc9ozanQPFyTF0TtHOnmRgrwi5IAiAr/3l9x7c2/S6VUib7Jx2pJXuhFNKzBhGLhaEUFJIFQDkwQgyqGw5bdMswgM7juK3h6bwm2f9jJSSTV05He+6+Hj88UXr8a6Lj3eeSzVo17utK6d7TbMELdjQtdACKdlHTi85SjKKes3EyC0e34+8VGtEOq1EC6URYz/M61oiHzk9nl6nnOhPgkHhMTNxrQBAN111NRHI1wzZgZwsoATNF8iFQrYIfPCKU/DBl52iPMZcIfE7wBjrBfBdAO/lnE8AuB7A8QDOAbAfwKdVj+Oc38A538g53zg8PNyCJSeH3auYuYU67u3OCUwnB+llVTOdHFJtmAF9TuyaNqhICCU6bsiVQVqddCplILdcRh7NQBmLliwsKzpB5AbypBq59BzkQpJPQgpsPQUD5x23BH995Wk445gBANEaOQ12oLuI+nI+Qlqh5ROzpNccGcgj/mY4ejwHj5VWqg0r9HMCQpKdIT78vKGD8/jvl8jyAUT2eg/DoHB+pGkMqCrnJwKWb0JaWTXQBV1jbgsHgo+ROyStXIve5J63YSnOO87jsW3LyBljOdhB/Buc8+8BAOf8IOfc5JxbAL4I4ILZW2ZzqJu29LF+Wa/vdmKc9L0lVp20WRQgzlsMulbk50mLMEaett9GKaVGTk8bF8g1xoQZk8E1mTy62GNpj92cKO2kd8KzIYGccgJi4ilK7qg37NtWD3ZD07yEohjI7WEh0YycXqvmBvKI1yK8tVed61coDSfhZld2ijmG4AZQlRqQyfC/BvV7QFem9B2OK9CSpZW+JqQVUY5J41pRsVxPI0/PyA1dw0qh9znJZyLzpyHicYxcRru6VhiALwPYyjn/jHD7KuFuVwHY0vrlzQxUvrth2GbklPikQEWBhLSwMC1UBbKJyfZDkZE3614JY+Rpy7SpNW9XwkBuJZVWhARaWCOmqMtLameauGmWzMgdFnVA8utTTqBXsIK5k3eUPnILv7txLf73PS/02LTFg5WdccnONNKK87f/e8kJ+OTrz/L9zZ5Mry4IkrX1mmmhUg8fMG0KnxWtL9B0jKQVI/w9Ut2fPttmpBVD0JNn2qubvtfN9jVZPeT1X1J5wLuF1hJp0K6M/EIAbwZwiWQ1/CfG2GOMsUcBXAzgL2ZzoUlx57aD+B+n0T2NgSJp5eQVdhGBG8hdRkKMPHmkdMe8KeyHhGb95Cqm1Z3TU2vkJDUkZeQUsOj5946V8fHbtgZseYyJczQVyc4YR8KylBq5+LI556GVs5QT8DkIQtjoV3+1HUdKNXTndfQWDLcxksm5L6kbJa3IMzvpPYm60qBj93fnAgGIuh/KPnKdqaQV+zMK3WSEKkxx062bFspOYDJlRh4XyCU3UjPSCuCdMzPVkomRJ+3WKUNk5ESeVMnOuNmvMuZDI4+9NuKc/xLqfvltZzcEgLd/dRMA4PXnrXGqDjU8b8NSXHbqcjx3/VLc++wRhbTiaOQpGLk3ONcfJLvzOt50wbH47/t3Nc/IFesQmzolBRUEFRIGcroioUKJ99/0MH7z7FFccfpKnHvskE93NUJ0VwABGxcAfPy1Z+LAeAVPHZzEC09chi/8/NmmXCv7xiuYrplYPdiNvWP+0XdT0jBsAKHl6R/54RMAPDYqDtUQtWBD11z9WH5Ncq+VZNKKX2cWYU+ZNwMaucoeSIFHlXAH/MlOeq7th0v4oxs34Zy1g/jBuy70uVaA+MZZJuc+OaQZaQXwzpmZxjsiKM0U8wHeFTrgBWsxCJ+/bgkuP20FPui0rkiKrES/heCco+ZIK0t68vjSW87Hbc6kGGIzFCDoEi0NIxfnLcr4+GvPxJ7RaYw2zciD6xBdFUlBjDypbVFkd5W66Uo5lDvwu1aiNXKZkL/pgmPdn3c7jawSV3YK63/ywAQA4OSVfYFATjmBopDcVZWni58zeZDFRKWoBXuFTxZ0zR8wZWmFzt8k0ooqkOc05pToyxq5/bNYKETfkWrdAhRjLlX9yL/+m50AgId3j/nWmdQOalnqQq+rzl2N9152YuRjRYiVrGlB4+IAL4lfbjKQL1UEcj8j1/HFiB5FYciaZrUQ9EUXM9r0BSJGzt1Anm7osHiMsKrFoWLe7byXFiqNujuvI8XyAIiBPPw+Yf02KnVTsBhavvtqMa4VLjE3GcR60lZ2AsCTB+wE50kr+nCnUxsA2EGm5Grk0YxclLzEYhx6PWK7Vgr0NdMK5BoCyc4E0op4VSPDKwjya+SuZMPhDiym70iYx9l3VeE8foczJYk2OlkjjyMy8lUJvXfdeR3HLU3ef582jrQa+W1XX4TlQiCn/v+iEyYNlgjTeyquRj7zYh655e5cYMEG8iPOCDcxo02XdNUQ10oal4m7UYQG8hxGS81KK8GTcybSShTTEl+z+HO5brrvHTF1sbIzSiOPG6zrbQKJXobviuKpg5MoGBqOW1r03adhcZ/9kKByrYwJkldOCiqW5fVrEf8eJiEBHmNOIq1EMnK3ICjYa4VeAz2OviNhcqB4VSEHp3LddKtggeQauSytuBtASneWK2elDORix0MA+D/PPQ7L+7tw+akrUh2HIDJyL5A3dSgf2tK10qk45DgaxEBLP5O+aEnJnqaSnQp9ErBZwkSl0dT8TvnkZMxmMc32WolKkvoCuXAiV+re5JWGJK2IGrnatRKd8KETuBnXyrYDk1i/rCdwJfS1e3e4XR7F5K7rWhGOIUpe9NrcYOlIK5oUyJUVrJb/sXqC12VKjxHhZ+RBaUW8gnGllRBrnHhVIcoF5x47CM7hc7zkE9oPufS55qWNPikKbiBP9bAANI3hd05f2XRycUmPx+6pqrsVjDwL5C0ETf7wM3L64jkaufP9o2Rgmi9klEYOeH7yiUpzfc5FnLyiz2llmi6Qk686SiMXA4HIrCp1M1CGbwpSQliRCUCulfB1JXF3+I4n3G374SmsGeoOeIf//tat2LxzFMW87juxVYxcTELTz7K0QkHZiOgpI8/sdO2HkdIKfI8RYThj9Cwe7LUC+CUyL5CHM3J6nBhY1g7ZVzLTtYb7eeYdMhLXAVGWVjaus4tgXnX2MZGPkzETjbyVUCU7W5GobFf7YUfCZeR6ckbejLQSppHT5X2zs0AJzzl2ED9+74scC1q6QE7HiWLy4nOFaeRuJayg7+Yikp1xvVZc5po0kAv3q9Qt9HXlQjdQ2cGgcteI0gr9LOr2puUFmbzLyMOlFdm1EvUxWRHSiuGW6HO1tELdKU1vIEKYRm4K1bXic60atDOj0zUvmZ2PkI98x5SuFI5bas/lvDSltEGfXZp+5LMBUVoRk8PNwr0yywL5zEGWKGLk4gnvMXLSfP23p3OtxARyx8tcamKEnLihyBNMmjlO1MPCArmokdPGJ5akhw0rAOxilKiTlK5em22a1VPQE09O1zRmV0YK6xSlFfeSWpAvxKZVkdJKiI88UlqJCuS6Zo964wATHL+aJK2IV45RVbj0PovPdcyAXQRTrpuBFhVx3/+049nCUHAZ+cyPNRPI3RuBmbFp+q5kjLwFoCKFQxMKacWRUGRG3pT9sB6d7KRG880xco9leZ340k+JkQufVBA3japPWvE0cipRFrsChvmzAXKthK8rScATIT9FT8FIVZZNI9QIo9M16BrDH1+0Hte9yu7ILPrIRQkhSloRk79AssrOKNdK1Kg3AODUQkGo/I3qy+PKQ8LBVg4IjFyyH/70iYO4/u5nQteu8tI3g7zrI59fRq7S1pPMHQ2D7ICaSyy4QE7WqoOTtrQi9iqmL2y1oa5sSyKt1E0LU9WGy4rCkp29rrSSzuNqWRwjztUE4Dk7NJa+14rslxdRqZuo1P1906k3DWAzNvKKk09X1ISjepjEuVaStHsVEWDkeSPVMAHxasayODbvGMWSnjz++srTsMphqF6Jvr05UdIrmbRC/5O00hwj1zUGkzRy4e+ytOJn5OEth+kYqmrM6Voj4Fq5/u5n8P9+vC1y7a0Ivs3aD2cDD//t5Xjdc9a4v6fp/yKDYk1KvtUSLLhATic9MXKftEJdDgM+ckp2xn8E7/7mgzjjup+g6miTYdIKlYmnZeTX3/MMPi+wovOOGwRgf8GalVZUj/uLbz+M//vfD/lYeM20XG3fTnY6jLzuf7/8/cjVGnmUk0Dsa5IE8v1sRq4+vup2Q/Om/Hzrgd3YtHPU7ffirkm4SrBL0Z3HSl5637qkJlLeBhX+WqK02C5n7mOlYfo1csl3n4SRi3kK0ddcFIpoZB85IWwjkkv0mwV9Ru0QyAeLeV+F6kxkkReeaHd3baaJ10yx4HzkdKJMVGwd1Jfs1P2BnL7IaaSVnzxOk4aiNXKXkaesOrvnSa9n+4/fexE2OJ0bZyatBP/2yO4xLOnN+65C6g2OnoKOoyU7kNP5TBl97+pA7KinsuV5OrgKM5ZW8rryZPn1NZcobxcZ+ZEpe4P/1zed67uPT1oR/NKyl161Ls9HTrfHu1ZUAfG8Y4dwvXPQKI1cZOFRjFyV7KRAPl0z0d/tZ+SEqWoDfYo+KqrWC82AvjvzrZETxM1/Jn1SPvn6s/CeS05Qau+zjQUXyIl5TZTtQJ5TJTul7odusjOFa2XcOX6rNXJyFQDAKSu9AgiNJWewBPl1Eso1E/vGKygWjAAjHyzaTLVSN933kpwRnuMiWiO3OFf2jyYwxpRTb8IgB/yegqHcQI8Z7A7cBnjtYQG4QyKOH/ZXIvqSnZbKR67OBQCqNrZJpJXg3y48YRkKhoZqw1Jq5LQJVEPyGiLEqyKRZVJZ++1bD+Lmh/cBCH6Hx6brykAe1544KegKYb5dKwRx858JI+/K6TjJacw312iTPbF18Bi5HUDFD4kxhrxzogAeozJ0O7Ck8ZEfdao2wzRykijSDnWWL/kJusZSdz8MS3ZS98ByzZQYueUytkrdctl2pe6vEGVxGnkCLVVnLNa3TKDkHwXv3pTJTpGRN0wLhsYCQcTtfkiM3A3kEdKKJJPIAVeFKGmlO6/jdeetQVdOwymr/Js4IDLy+EAuMnLNx8jt7yUFcfE2ApEUGXFdLZOCgmU7SCuAvw3ufHvbm8WCY+RyYJFtagVdcy9HuaBxRvWdVoHK78MYecHQoGvMLcpJijAdXGMsFSMXvcYymdzhDGaoNvzJzqrTZCyvayjXTddXTIGc9gM9ViOPP0nTbEyUPO0tGKg2aijmdd+VFhBdJWhozH0tDYu7l/b+x0uuFSYz8nBpxQ3kGt0eIa1EJDsB4B+vOhP/eNWZ/rXJGrnwmUX6yBXPUVRMixIbUQHh7Zdb5VqhwJk2eT9bEPsxdWogX7CMnJA3/B9MIRdk5JrGUNA1d2JMEhydrsHQWOgHzxhDT17Hv9/1DL7482cTHzesk5tqbmMUxE1JTl7RhJ1yzQz4yHWNoZDTfNJKWZJWNE3s861mqnEnhK7Fb0w/3nIAl376btRNC5rGUHTkKlWyM4qh67rHyOumpZR9dCFYigHLa5qllpAAgDmHS9J6wK0LSDMdp2lpxX/bO164HgVDC3SmHJKaTo2V1T2CrBa5VuTWD/ON3AJg5AsukMuX63LHOrJ4Af7udTlDc0v3k2C0VAtNdMrP/Q8/2pr4uOTZlttnppVWRDdDYMKOE8grDQuTFY99cYdJd+d0VOqmUD3oL9FnYj/yUPth9PpsF070ff7m5i14ZqSEPaNlaMwrsuotGCjo/s81qkBIdK00zBBGHuIj95pmqXzkzmMDlZ3xPvI0+nBAWvG5Vkx8+4Fd+LOvb/Y9RryqAIAdn7gS177iNDDGUFScE+KV5awzcncYdPo+RLMBv0bemSGxM1cdATloiS1NAf9JLU5Bz+ksNSOP8zKPhWiNUSjXTJyysg+Xn+Yve7alleTHERm5HGspkJuSZx2wA1pXTkelbrkJwnI9KEW5TbPC7IcxgSpJ7xhqWfrMyBQ0xty8Q7GgIyddaamCM8GnkVuWcjSYmKgU15+kslMuzRbf7ycPTPoCe5SPPGr94vM9smfM/dvWA5P44Hcfw21bDvjWKDb+ktEtaeK6c0VKuPYHW/DI7jH5YbHN0JLCLbJKa8OaJYgyXbs4adIidtmMsbWMsbsYY1sZY48zxq52bl/CGPsZY+xp5/+huGPNBeRALidyNM07IcQCl7yhP29LlwAAIABJREFUparsPDpVC010EiiRuKw3eb/kSsNSTvTRtXSaosjawqbQA/bEHd/zMLt7YLnmaeRVV1qx7+ObzB7iWkmkkcecyF4gL0EXArkq2RklrfhcKyZHTlWM42rkfubpVXaqcwFAsGkWva5f/fYwfudzP8dNm3YHHpMmYIjdD7fsHXerL3M6w/3bj7r3E5OUlhXeE17WyTUWzPXcoJADxdYFM4HX/6Y9GLlfI+/MSJ5k1Q0A7+ecnwrgeQDexRg7DcA1AO7gnJ8I4A7n93mHGBzyuhb4ghqa1w7W60RnB4Jqii9WqWYmri5cu6QYfycHlZqJ7pyaMaYpCBJlInEDGJuu4WiphnVOP+8DciDXGLpyGiqNoEYutmAl54pyZmcCv7GWwLVCm7Bp2U2kegs6mLPRyDaxSI1cdq2oGLmQqDQF5plPwMgpXsr2wz2j9iCHB3aMeo9pojkTvdTvbN6Df7njafd2mkVLEAN5lAwSDOQs8F3eMBwcFCHLNc3CcN/TNmHkLbIfzidiIxHnfD/n/EHn50kAWwGsBvBqADc6d7sRwGtma5FpIAaWnkKQ2WrMC0g+Rq5riXzk4uccp5ET0hQIVBqmcup92ja2FaEXjKitExunJv0HxivuhCTAfi8KDiM3hUD+z7c/7bpdxNJvJSO3gqPeZBgJkp1iwzFNYyjmDfTkDceHLgfy8CcUe63UQ1wrYrWpZXmDJYyIQC7P7JTbzXYrqnvFJmhJQWv7wj3Puj3XX3zSMM5d678I9jHyiKuibkUgp+/yFaevBKBuVyGW/c8EbrKzTTRysX/9fAxObgVSXUcwxtYBOBfAfQBWcM73A3awB7A85DHvZIxtYoxtGhkZUd2lZbAs7tMnZVkFkBi5cGmcVFoRT8CCgjmL+NQbzgaQLjtfrpnKqfeqSepRII28O6f72qoSAz9+2K4Y3TdexlKhwb6uMSwp5jEyVXVPtIMTVXz29qdwzfceA+AFrpyQOBYR18aWnicueSsGQJ0xXHbqCrxh4xrlfdMwcpVrpVuoeGxYlmA/TC6tuD1RnD9QMBRrCUyenpHLVxAFQ8NX33Y+1g+3hpGLyc6rLzsRg8WccqCxxVvEyGnYR5sw8t4WlejPJxIHcsZYL4DvAngv53wi6eM45zdwzjdyzjcODw83s8bEoJOkx/miqliaJrAzr3sdEvnIOee+Ezqulerrz1uDC9YvScU8Qhk5C+r/USCNvDunS0MV7JOdKkg5B5YJPmJNY1g/3IPdR6dda5u8wYnJvbDhy/HJzvjKTrHhGGMMV5yxEte98nTlfaM1cs3zkYe4Vga77TzGWLnuazGQ0+KlFXo/5Ek+VEMgBkWrCUZ+ztrBwG2MMRzrSHb0PRwX3CaiPCSjOyfljRjw9685E6eu6sf6ZT0oOj1fZLTORx6+Oc4HxCrWBW0/ZIzlYAfxb3DOv+fcfJAxtsr5+yoAh8IeP1eggEVJMdUX2RAkCpFRJXGtyJ7duGQnPV8a5lGpB4f8Ao60kiaQOzbG7rwcyG2P8KoBrxXAMqHBvs4Y1i/tQd3k2OkM65WJsybIDspRb1aCgiCWgJEL0kpccjBKWhEZuS2tBA822GOfzGPTNV9lJ80njRr1RgFclwI5MfGSgpGnCRhywQ59D6klATFKPyO3QlsJBzRyjeGC9Utw29UXoSunoyuvK+sZWuVayUXUIMwHRGdbK6445gNJXCsMwJcBbOWcf0b40/8CeIvz81sA3Nz65aUDnaz0wag+FJGRi4wqb+ixjFyuokuS7DR0LZXNqlIzfZo1IUngE0GX9V053Wd/G5uuoSunYaDbC95LBVeN7jBywGtDIEPUhNXJznh3g5bAtSIGwLiNIZKR6946bWkleKy+ggFDYxidrgXa8OZ09WZsV7B6v8uFO7R+WfJI8npkXHvlqejv8jPpE5b3IqczfPRVp7vPc8uj+1B3qnrDPNGqZKf8d5mR7z46jYd3j7XEtULOkHaxH4rv60z6kc8nkjDyCwG8GcAljLGHnX8vB/AJAJczxp4GcLnz+7yCAjQxFGXPZ+YxKdFHntcZag0LddPCPU+ptXz5y50k2ZnTWKTN6tfPHPaxn0ojRCNPmewk1lbM674NYHS6jqFi3vccS4X+LhpjATdEYC3CwAJ1gEugkSeoVBWllZkEcr9GrpZWGGMYLOYwOl0PrD+nazg8VcO37t+Fx/aMu7fLCUW5RJ/WPzJZDXzn0l7Cv+OiDXjkupcCAN53+UkAbMLy9D+8HK88+xj05HXctGk33v3Nh/CFe56J7EApJztlwtOd0wOtJV75b7+0X2MLGCu1je0rtEeHEFEj71RGHvtOcs5/CSDs1V3a2uXMDK60kg8P5HZBUHB0md1My8Tn73oGn739KXzjHc/FhScs8z22IvV+TsbIw5nn7qPT+P0v3oerzl2Nz/7uOWiYFuomV0orLG1BUMPTyMWnH5uuYbCY97H+FX1istOeZdgdopPaa4l+baYVX7nYndcxFTF0w7S47/nj7L1hSVBAdq1Y6M2pv/aDxbwtrQiuFcAO5N99cA++++AebFjWgzv/8iUAgj1lZPshMfKGxbHjSAkbhnvdatZmAgZjDDs+caXybwPdOewZLQOwxxyaER0oZUYuL6U7bwQaZ1G1584jJcwULzlpGNe98jS8YePaGR+rFRBJzYLWyDsFFKBJI1d9KHaSzf5ZlFaKeQPTNRP7x+2T4ZmRqcBjiTnTpVgyjTzcDUMa8BP77NxxRQi+MnQtef9uwGPkXTndp63bjDznY2UbHAcLYAcjxpjSuineB/BXyYrgnMdq2scuKUYGBXnWaRQT3PGJK/Ha54QH8gAjDzlZh4o5jJbqSmmFMCoMbuZcHpJMgdz+fVqoNbjL6TPv6eqhy20K/YLFNefkLsLeM9nNJZ8nxZyOsvT+r3b0+GdGZh7IGWN424XrA1XX8wWRdCx410ongAJ0byF8JqCheb5qUa/sLRgoVRuu5/vIVFAfJoZ42jG2BzuJXdHWZ9UB2K0mdNZDG0WYRp6qaZaT7Czmdd/jRqdrGCrm0SVsQqKUQsmsqE2KTvyw6swkrpUNy2xnTNh4vWmHrdPl90wu6cUNpx5SEAQAA9153PvsEWw7MOkLbqLWLF6VydKKLkkrU9UGTl3ZhxOX9+LObbb/u5lkZxKI7Y9Hp2uR8pZMFOT3tluhkR8j9MlfyMgYeRuAGDmxIDUjF+2H9m2M2cVDpZrpfoF3HZ0OPJZK1amYZt9YOXZNovVNhryhUDI1TFrhPLohk2+tArsXg+3YdB2DEiMXf6bNRdZRRdDbakS4OeIC77plPbC4+n0GPMcHMc2ZnF8+14pphTpchooeqxXdGaKEVq6b7muWk53E7Oi5StUGegoGzlw9gB2Hp53HzE4gf/7xS92fRyarkVWYQdeK/+/dgmtlbLqGWsNyN9wvv8XfzG2hoV2GXaRFRwTyo6Uath8u2XMGIzLdsiNA9UUWKwrFk6qYN2Ba3C2YefZw8BLSZeSpArna2QHAtTsmCeRJBxZXG3ag2eFYB7tymrthWRbHmMPIxUSteDlJz6O6KiC4djsnQE7XGr7PRdaOVaCrgO2K9xnwPNiDRQrkzZ9gorumYfHQxOgRwaUzLThm5MA/6QwtCSQ73e6H9u+lmoli3kBfl+F2mWymRD8JLj3Vq8f7xdOH8fi+iYimWTGMPOcF8pf98y/whXueQblu4orTV+LSU/3N3DK0B9pDpIrBp3/6JH74yD5MVBp4xwvX49pXnKa8HzFt8t2evXYgcB+xx4foIye9jhJGKv2WLqtPXmmPcxKnuITBCLGuAV71pdwuNsy1AtgsPupDO/naH/t+z+melDRVa8DidnCkYHz8cI80sd0J5AmkFUPXMFVt4Hn/eAc++urTcdW5tk5tJdDI1y21A3mYTk6MnHplz2QIgdi62NbI1YsT5QPxSkG+/2SljiU9eXDu17plH3mp2kBvQUdfVw5T1Qa40+uc1tRKnLyiD68/bw1+8+wR9zsctukHNHKF/XC6bmK61sD+8QqePDiJct1UDqXI0B7oiEDeWzDc0W1f+83O0EBObOfklX34wbsuxOnHBAOtipEz5l1u7nVYtmrWJhXZ9HflcNvVF2HNkHpGpIicHp7srEmj2MqR0or9f5p49rnfPQdb90+4x5cZ/0/e+yKsHOiSGLn9f1Jp5eBEBROVBp455AXkqEQbgXIREyGtfsm6N+AwclVQuv/Dl4YOVhDhc61ESCsffvmpeM6xQ3jfTY9g91HvakueRiQycjEgy90PSVrp7zZgcZuh057eapsbYwyfesPZ+JP/2uQG8i17x5X3jfORdzltHeiK88B4BeWaha4skLctOkJaERlEWHIM8Bi5oTGcs3YwdKK6Zz/0JzsBr3ijbvJA8KDnzhsaTl3VrxxQK8OIKHypBxh5dLJTvG8SXHzycp9tkUqiqaT75JV9GOjOSV5of7JT5ZUXC4IOO0lhv5sjvgLQboJl5yVUcKUVJ+CrXvby/q5EnSX9/cjVPnLA/p5dedYqAP5+7nIB0QTJJIFkpyyt2IGcvieTlbonrczSmff2C9e7VbuHpF7zhIC0Iq2FAv1ln/k5AGD/eAWVurq+YaGg019bRzDyKCucCO+yNfwssbsI2j+L0kqPwgpVqZu+24n9Je16CNiVYmEVbF7SLJ6Ri9JKEmjMLrwQ+5hTd8fAUAaRkUvJTprq7js2SSsacyWQMV+fj2R9q3scp5AKdNzBCEaeFN05HaVaw+mVY0VOgSkYOo5bWsRV5652b5MJgcfI/cmxvi4DGrPb1zZMC5W6hR5HI6fHzZZrhfDcDUtx74cuxRWf+zleeppaz45j5HJQOzhRgcV5xwe7KNz5ly/2XYV1GjokkCdbpsjIwyAmH0UfubhZLO8r4NBkFWUpkBMjVw1+CENO00IrO+l4jQAjVzXNcthewqKgwWIemsbsPubO6ySWmZfGpOlakFV2OZtVIacDFdnTbf8v2vhERp60S15vwQhn5I60Qs2sZhLIh/sKqNQtTFUbaJg8si8LANzzVxf7fidphYqkKJBzacPq68rhOccO4c4nD+FtF64HAPR3e4x8olxvukQ/LX783heF/q0YaJolbezSxkXfzyi5rdOxaqAbqwbipdJ2RUdIK3Igf8N//DqkotAOVFGX9eLINDoEkxg5XZrKvVVII4/reijC0O0rALqkfvW//wo3P7wXgCrZ6TW6Cq7b/j+KkYvWRNHtQbZF2jjkQGb394Z7f8DbTKJkHnHDpK6KnPNE0gpgM0OZkX//oT34y+88ItgPneESM0h2Lu+3k9+HJquho96iQNIK9aTxHCjBIHjJqcuxZe8Enj1sF5T1deV8jLyZ7oetRrEQLa2E5S0WMiPvdHREIO+VvngP7BjF2HSwYIeIbxwjp8DJRUYu6PBU6RgM5BYYi+60J8Md3mvZgfSR3WO4+lsPA/A0a8sN5E41pkK6SWI/FCUQcnuI1YYk5cjJO8DrwUDPQ95pVWGQaD8k0Och9+iOQk/B8PXqBoBfPn0E33twD8amayjmdff503R+lLGiz96YD05UQke9RYE+wyVOl8iJsmg/9N/3zNW2U4p6svR1GW4l8ESl7kkr8+hXXtqTx98KhgH5s/rd89fi/15yQuBxC5mRdzo6IpCrBkSMKiZ9k2QSxXbEgiDxMldk5Keusu2Fcm+VWsNCXtdSFQ14sy0tN4FHDyeGbEoauZKRu4m08IAmJuiouEWsNnSloQhGSm8dBS/V1YfoWiGQtJLGXteTDzZnKlVti+TW/ZMo5g03MTkT+yExcqoRSMvIaQ3dOR0FQ8PdTx1Cw7QCGjkALHc2DSpltwM5JTsbs1ainwaMMbz9hevd3+VNpadg4H0vPTnwuIyRty86IpCrejKoGXkyjVzuR65r/t4iJy63A7lcplxtWKkSneJa6iZ32ac8B5KkHlcjV7BgYk1REoM4cHnQYeRitSFdASgZuXM/2jDi+nuL/9trt1Cpm4E5llGwk53+95h6rDy+bxy9Bd3dUGamkdvBda9jywtzrYSBPq+8oWGwmMNDu8Zw62P7bY1ceitpYPRvD9nSSn9XTnCtNNxEcDtVEIYthb679PpVuZsM7YGOCOSqZKeakZO1KzoINSS3CGN+5kmapkpaySdolCVCDEQ0KUYO5CIjzxuacv1J7IdqRu5Z4lxpJarlq9AQKwwUhOTjUI8P8ThR6FVIK6SZl2p2otkbQtB8IO/vMlAwNOxzGqKFdQUMg3h1cuPbLwBgVxur5mIOFnPI6xp+O0IauYGunAZDY5is1GFa7dfPI2xTefBvL8cj170UKwbszSkrCGpfdIZrRfEFGlUxcjOekeuC/VD0kYtfZmIe8pSUasNMz8hp0KxpuUGLGDFp2hSkqnVLqY8D3uYU1cq2Kmw8g65Gbv9ucu4+n0oukTVyWrcqfNIx5YA0Wqq7V09JNPJi3sB0tYGfPzWC8XIdvz00hd2jngWsJ2+4G8pMkp2MMazo73ILZdIycrp/TtfcitTpmqlsRcAYw3BfwS0s6+uyq2j7ugxMVOroyRuz7lhJihvffoGbeFeBJKFV/d3YfbScaeRtjM4I5AIjP3F5L54+NKWWVhJ4dMWCIDkx96YLjsWZqwdcp0alEdTI4wYuy3ClFYu7ljqZkQN2Mq9SN0NPFnpJUVqxmOx8gdNESeyRTc+XN4LvD13N0NUIrVGlydP7+/wNS/GbZ4/gtFX9+OkTBzFermM1ty1cSVwrvU6jss/e/hQe2jUW+HuPIK3MdCrYhuEebN4xCiC9Rk5ryBkaCoYGxuxN3pLa2BKW94uB3H4/1y4pYtv+SQwW865mP9948UnDePFJ8XN0Vzourkwjb190hLTSndPBmG0L/OlfvAiGxnwFKARPI49K5jFlP3IA+Phrz8TvP/dYl5FXAozcSmU9FNfiZ+Skm3vRqdqwUK6rBy8DyQqCKJn5lbduxLnHDjmvjZg8TyStrF9mO3aiWCsd83XnrcEvP3gJ/vxi2+EgNs9KWhAEeP3YVX93pZUZMHIAuOSU5Zik9z+ltEEOnpxuX7kVc7rDyNWtCEgn7855G9FLTl6OTTtHcfvWg7jk5OWBx7QzyI6baeTtiyQzO7/CGDvEGNsi3PYRxtheafTb7C1Ss0+eYl73jeSS0UjgmDA0r0BG9JGLcAN5Q5HsTPlldqUVi7v6b85l5F5wKtfNyDJoMSCHwZNOghNPTIv7WgyEgboSEmtVXQHIwYvsoaWal+xMogMXnUAuV45S8O7JGy1JdgJ2uwJCateK27bAKQzKGyjXTcdHHrz/in478IkjxC49xXv+i0/prEBOQ577ujriAn5RIskn81UA/wbga9Ltn+Wcf6rlKwpBT8Fw9VcaySXDTGg/NC3uFK6oS8kpmMrJzlrDjLTuqeAxcu5a7YjVi31jKnUT5Xr4RkGvKSqeeZWn3ho9SSZZsnOZU/RCrFVFhOX3jJh1qeqVoCdxZcj1AYSTVvTh8X0T6CkYqfXsMIg9WdLUAdj3t98v+iyLeXuCThgjP2ftIL52706MCL1OzlozgL9/zRmoNSxcdGK8nNFOeO1zVmOoJ+9uUBnaD7FRiXP+cwBH52AtkegtGG7AGCrm3GTnockKvrNpN3YeKeEXTx0GEG8/BOzAFnYieslOP1OsNqORO0Hj8FQV922330ZixKLLxGPkIclOSlo6kXzzzqPKZCzgT2a6SVLOUSP7YUQglx0p6mSn/z0jn3+p2nADfxLXSo+iPgAAzjjGLqoR7YetADHKKOlNBQr89BkU87qb7FRtWC9RSCeMMfzB847D21+4vu1cK3Ho68rhVWcfM9/LyBCBmZwl72aMPepIL0Nhd2KMvZMxtokxtmlkRD2dPgnWL+vBOueyf2lPwWU7/3z70/ir/3kUr/q3X+F7D9kZ+LhkJ2AX6IQNQNA1hryuBaSVWhMaOQWBf779adzy6H4A3nPWBUY+XbUDeZgOKSYtD05U8Pr/uBc/kBwHakYuBPII10pe13D2Gq9/O/Vyf9kZK93b3vy84+xjSu8vuYpKVVMoslK+DB9WO22ADY1h3VKPMZ+xuh9LevI4bmlPSwP52iH7OdKyfFoDBW0ahRZ2RbekJ4+B7hzOOy70tMiQoaVoVvS6HsDHYBO2jwH4NIC3q+7IOb8BwA0AsHHjxqaFzhv+0BsxddyyIu7cdgimxd3CC3Hqd5z9ELBdEJaldh0AdjAMMt5mXCv2/Z/Y7yX0yDUjMvKxci1SI9cFZv3MoSlwbnuZ5fUB/rJ60X/uuVaCr2Hrx67w/X7C8j489fcvw84jJfzrnb+FxoC/e/XpuO6VwV7whq6hK6eh5MgNQDLXyunHDGDztZchb2goGDpe+tl7sOPINJb0FPCbD12KnM5cy2ArsGaoG0/sn0jUw1yErKkTIze6WKiV8IG/vqzjmHeGzkVTgZxzfpB+Zox9EcAtLVtRCMSTYsOyHtRMC/vGyspRYZGMnImMPHxAbVdOd6UKQq1hRQ4lVoE2FbHwhZix6FoZna5HulY0ISDTGDq54ZQrrRhBaYUKgjSmfn9Ut+UNzfOVa3ZrgjA225O3W9KSIyipV3qpMDSYXnuxoLuvISoxmxbHOjr5oYnKjI7TnTMwWiqjt2CEXnm0ct0ZMsShqW8bY2yV8OtVALaE3Xc2QBa5x/aO49Bk1Z2hSYhLdgIOIw+RVgD/3EJCtWGmtx8q7k9ulVqDu1a1sekaKnUrPJALjHx7SCB3pRUjmOwk10paqYKuKOLYJfUW91wrqZ4GgCdhiNp51NVVWrzn0hPx2uesxhvOW9vU4+mrQtKK7SPPWHeG+UcsI2eM/TeAlwBYxhjbA+A6AC9hjJ0DW1rZAeBPZnGNAZBF7rYtBwDYg2dF6SLOfgh4jDzsPOzKaRidrmPP6DT6u3Po78rNKNkpggJuzbQw7PQ+Hy3VUamZoUOP6UqiUre8QO5sNLWGBUNjysEXoiRTM63UTFHXiZHHB/Lxct0tg2+mepHWJj5U1RemWQx05/CZN56T+nFyUZTtI7cTu5l6kqEdEBvIOedvUtz85VlYS2Is681jsJjDDx/ZBwC47NQVuP7uZ1wfeRTrJGZrUt/skIDT35XDPU+N4IX/7y70Fgxsuvay5pKdCocESSr1hoViXkdfwcDRkj3IItRH7hzm/3zpPpfFl5yBvhd/6m687cJ1no9cCH5MSHbWzWYKmuzH6zEJwp68jrueHMFdT9oJ7bQSFGBb9DbvHHVLw4H0fVFmEwxesjOqIChDhrlGRzr8GWP4z7eej6cPTWGomMfZawfxnT99PlYPdmPPWDmyAs3wSSvhI8k+/toz8dDuMWzeMYpvb9qNw1PVphi5eHVwy3teiM/87Ck8uMsuFa+b9vEGe3LYun8SDYtjzZB6BqUYMGgW41S1gYMTVewdK+PBXaM4wenaKAZrb7I7mpRWPI08CmQNfc6xg3jLC9bhJSen90p/6GWn4orTV+LklX3e87fIR94K0Edg+8izQJ6hfdCRgRwAzj12yC1Dp98BeyBvFORkZ9iJeOKKPpy4wh5O/O1Nu3FkqgbT4oExaXEg++Gy3jzOWD2ADct6cN+zRwDY0kpvl4GhYt4N7iQbBdat2HGma6Y7iebZkRLWLikib/j7pYsaed3kqaUVCuBx0god98ITluHV56yOvG/UMZ67Yan0/PMfKOWiqGJedweFZEnNDO2ARfct1KVkZ1yyiibtHHScDuk1cvv+FKBzhiYkO22GPFjMu7LQhmF1IJc3nGW9BZSqDVcv33GkhGo92C9dkzTytFWNJKnEJTvJChm2ETWLdkgmcqcsilbS7RZAmRkjz9AWWLSBvGFZoQUdIqivtxvImxws4QZyXUPNtNyJ7nldc5+jmNdd/VuG3PPkzNX9KNUa2O5MoqnULew8Ugpo016PlhlKKzEbwKFJ+/1pdSBvB6gYOWBLW1kgz9AOWLSB3OIcphWvcVJf77+5+XEA6f3BFDipKjXvBMQ3fuFeTFUbyOnMZf3rlvaEMtBxoUnYQHcOa4aK2H20jC/9crur3W47MBnYaChu37RpN372xMHUG5FqGpDyfsy/YS0keM2yHJ+78//+8fK8jmzLkIHQsRp5s/AYOXd85NH3p2n09FhVH40orOgv4P2Xn4TXPWcNAG8jeMDpjT3cV8Crz1mNiUodLz1tRehxXnzyMN5zyQk4Z+0gxst1PHlw0v3btVeeho/d8gT2j1dw9tpB3+NoY/h+gvYFKhAjj3OPfOkt5+Oep0bcja+V+KfXnxWoFZhLvOm5x+JIqYY/f4ndrvcFxy/D752/FtWGhSvPXBXz6AwZZh+LLpCLFZIW57Gl5KIU8fHXnonVTkvPpGCM4T2Xnqg8HmAXN52xeiDW35zTNbxfGIj7r3c8DQDYeNwQ3vaCdfjUT55EuW5ig8SIiSlTZenhqWDXyLj16xqL3QBOWN6LE5b3pjp2UrxxY3MFPK1CwdB97/1wXwGfeN1Z87iiDBn8WHTSCjFMu5VtusIVOUg2g2Agb+6Y1Mt7RX8XNI250o18PPn10eSaNNC18NL8DBkyzD8WXSAXBy1E+chVaIX+K2vsYS6VOFDv9WEnObohLJBr8uPS9y0zEjDyDBkyzB8WbSC3OI/staLCkp6Z679isU6USyUOhybsoiBq9r8+ISNvBobG2qrC8v+3d38xUt1lGMe/zy7/GiBbKNuyAUJpQ4qVNLgiIWlDWrUUqBGa3lQlIU0TbjTRVGPAJtrGGzXReGM0VVuJf2hM1JT0SgKa3hhbsCxdgluoQsSSXRRNqRpb4fVifkOn2zmzu8PZ86f7fJLNnPPbs3uevJl998xvzpxjZu804+bIr77ZeTk6Xmul1Ze2reGV0TdyOae5dWrlwcHlXf/OT21cyeGRMR4cbHz45p41N3L07D/eNU/deiR9z2393NXF3Wnplj8JAAAGaElEQVRm9fb4iNyswmZsI7989VZvEzeo3ZtuzW3/zamVdSuu56s71nb9e1Ytmc/hz999df2DKxexf/fGd23X7L/9C+fy9MMbutqX58jNqm3GvV5+xxz5lXymHqai+WZr8/6j060nh/O7PUduVm0zu5FPcmolT80bOs/PuPFw3pqN/FrOuOntUSWueWJm7c28Rp4a28V/vcnI6KXCj8ib53Nn3Xg4b81/XNdyRD67t2fKNyw2s+LMuL/OZmN79OdDnP37vwufMnhf+oTi5vcvnWDLfNywYA6zesS6cZ/4nIqBvnks7et8VUkzK8+MfbOz6X9dnFd9LdYu6+PEE/ddvX73dBvou46hr2y+pv09/fCHfHEoswqb8Ihc0lOSxiQNt4wtlnRQ0qn0uKjT76iS8Y389f+8lbHl9Cmqiee1v7mzeqd81UQzK85k/jp/BGwZN7YHOBQRq4FDab0WqtDIzczyNGEjj4jngYvjhrcD+9LyPmBHzrmmTe+4KYJL4+5Eb2ZWN92+Xr4pIs4DpMepXdu1RJ3u52lmVkfTPvEpabekI5KOXLhwYbp3N6GlffP43s5B9m5dU3YUM7NcdNvIRyUNAKTHsawNI+LJiFgfEev7+6d+nY/psGXtADs3riw7hplZLrpt5AeAXWl5F/BsPnGK07xdl5lZ3U3m9MP9wO+A2ySdk/QI8DXgXkmngHvTeq1U4e7sZmZ5mPAE44j4RMa3PpJzFjMz68KM+2Rnq+98ctBTLGZWezO6kd9/h++Abmb1589dm5nVnBu5mVnNuZGbmdWcG7mZWc25kZuZ1ZwbuZlZzbmRm5nVnBu5mVnNKaK4e1ZKugCc7fLHlwB/yzFOnpytO87WHWfrTp2zrYyIzMvHFtrIr4WkIxGxvuwc7Thbd5ytO87WnfdyNk+tmJnVnBu5mVnN1amRP1l2gA6crTvO1h1n6857Nltt5sjNzKy9Oh2Rm5lZG27kZmY1V4tGLmmLpBFJpyXtqUCeM5JelnRM0pE0tljSQUmn0uOigrI8JWlM0nDLWGYWSXtTHUck3Vdwrscl/TXV7ZikbUXnSvtaIek3kk5KOiHps2m8CnXLylZ67STNk/SCpKGU7Yk0XoW6ZWUrvW5pX72SXpL0XFrPt2YRUekvoBd4FbgFmAMMAbeXnOkMsGTc2DeAPWl5D/D1grJsAgaB4YmyALen+s0FVqW69haY63HgC222LSxX2t8AMJiWFwKvpAxVqFtWttJrBwhYkJZnA78HNlakblnZSq9b2t+jwM+A59J6rjWrwxH5BuB0RPwpIt4EngG2l5ypne3AvrS8D9hRxE4j4nng4iSzbAeeiYj/RsSfgdM06ltUriyF5UrZzkfEH9LyJeAksIxq1C0rW5Yis0VEvJFWZ6evoBp1y8qWpbBskpYD9wM/GLf/3GpWh0a+DPhLy/o5Oj+xixDAryUdlbQ7jd0UEeeh8ccI3FhauuwsVajlZyQdT1MvzZeTpeWSdDPwARpHcJWq27hsUIHapSmCY8AYcDAiKlO3jGxQft2+DXwRuNIylmvN6tDI1Was7HMm74yIQWAr8GlJm0rOM1ll1/K7wK3AOuA88M00XkouSQuAXwCfi4jXO23aZmxa87XJVonaRcTliFgHLAc2SFrbYfMqZCu1bpI+BoxFxNHJ/kibsQlz1aGRnwNWtKwvB14rKQsAEfFaehwDfkXjpc+opAGA9DhWXsLMLKXWMiJG0x/bFeD7vP2SsfBckmbTaJQ/jYhfpuFK1K1dtirVLuX5J/BbYAsVqVu7bBWo253AxyWdoTEt/GFJPyHnmtWhkb8IrJa0StIc4CHgQFlhJM2XtLC5DGwGhlOmXWmzXcCz5SSEDlkOAA9JmitpFbAaeKGoUM0nbvIAjboVnkuSgB8CJyPiWy3fKr1uWdmqUDtJ/ZKuT8vXAR8F/kg16tY2W9l1i4i9EbE8Im6m0bsOR8RO8q7ZdL1Lm/M7vttovHv/KvBYyVluofGu8hBwopkHuAE4BJxKj4sLyrOfxkvGt2j8N3+kUxbgsVTHEWBrwbl+DLwMHE9P2IGic6V93UXj5epx4Fj62laRumVlK712wB3ASynDMPDliZ77FchWet1a9nc3b5+1kmvN/BF9M7Oaq8PUipmZdeBGbmZWc27kZmY150ZuZlZzbuRmZjXnRm5mVnNu5GZmNfd/KD9KM9uwSFAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[\"mpg\"].plot(kind=\"line\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 347,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAETCAYAAADNpUayAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUp0lEQVR4nO3dfbRddX3n8fenCSKIo8EEhicJtlQFraARH+pYWnDAhxamHVpo1cBg0874gFO7KkydYu2wStfoWP/QtikiQS0Og1qitZaYNiBteQjgKBgZ0ARICeSiWB4XNvidP/ZOPbncm5t7z31Ifvf9Wuuus/fv99t7f885937OPvvcs3eqCklSW35srguQJE0/w12SGmS4S1KDDHdJapDhLkkNMtwlqUGGu2ZVkuOTbJ7rOvYESS5J8j/maNt/nWT5dI/V7Fk41wVoZiXZBBwMHFxVDwy0fw14CXBEVW2am+q0u6qq18/EWM0e99znh43AGdtnkrwY2GfuypkeSZrZOdld7ks65kIDfBLnh08Cbx2YXw5cOjggyd5JPpjk7iT3J/nTJPv0fYuSfDHJSJIH++lDB5Zdl+QPkvx9koeTXJVk8c4KSvKeJFuTbEly1kD7s5Jc2m/rriTv2x42Sc7st/HhJN8D3p/kJ5JcneSfkzyQ5H8PrOsFSdYk+V6S25P88kDfJf19XNPXfHWSwwf6X53kxn69NyZ5dd/+s0m+MTDuK0luGJi/Nsmp/fTBST7b35eNSd41MO79Sa5I8qkkDwFnjvNQLR6rxiQfTfKhUY/pF5K8e5zHe8z70/etS3JBkr8HHgOe17e9re9fkORD/eO7Mck7ktT2F6RRY8/sH4MP9r8rG5O4Zz8Xqsqfhn+ATcCJwO3AC4EFwD3A4UABS/txfwysBvYHngl8AfjDvu85wC8B+/Z9/wf4y4FtrAO+Dfwk3TuCdcCF49RzPLAN+ACwF/AGukBZ1PdfClzZb2cp8P+As/u+M/tl30l3SHEf4DLgd+l2VJ4OvKYf+4z+fp7Vj30p8ABwdN9/CfAw8Fpgb+AjwLV93/7Ag8Bb+mXP6Oef02/jcWBx33cfcG9f7z5933P6em4Cfg94GvA84DvASf023g/8C3BqP3afMR6rndV4XL/dH+vnF/eP44FjrGfc+zPw/N0NHN3379W3va3v/03gm8ChwCLgK3S/OwsHln/bwHP0L8Cv0/2u/ee+zsz138J8+5nzAvyZ4Sf4R+H+PuAPgZOBNf0fcdEFaIBHgR8fWO5VwMZx1nkM8ODA/DrgfQPz/wX48jjLHt8H4MKBtq3AK/sweAI4aqDvN4B1/fSZwN2j1ncpsBI4dFT7rwBfHdX2Z8D5/fQlwGcG+vYDngQO60PwhlHL/iNwZj/9VeAX+5qvAi7vH9efBb7ej3nFGLWeB3yin34/cM0Ez924NfbzG4DX9dPvAL40znomuj/rgA+M6h8M7L8FfmOg70R2Hu53Dozdtx/7b+f6b2G+/ewWx/k0Kz4JXAMcwahDMsASuj/Cm5Jsbwtd2JJkX+DDdAG2qO9/ZpIFVfVkP3/fwPoeowui8Xy3qraNMX4x3V7uXQN9dwGHDMzfM2pdvwP8AXBDkgeBD1XVxXTvTF6R5PsDYxfSPQ5PWVdVPdIf6jm4/xmsYXQdV9O9SG3upx8EfobuhenqfszhwMGjtr+A7oVhvPsylvFqvAdYBbyZ7sX6zXR79mOZ6P5MVMvBo/onqvtffxeq6rH+d2pnvw+aAYb7PFFVdyXZSHcY5OxR3Q/Q7U0fXVX/NMbi7wGeD7yiqu5LcgxwC90LwHR6gO4t/eF0hwEAngsM1rTDaUyr6j66QwAkeQ3wlSTX0AXQ1VX1up1s77DtE0n2ozt8cW//c/iosc8FvtxPXw18iO5QxoV04f7ndOH+0X7MPXTvfI7cyfZ35ZSs49UI8Cng1iQvoTvk9pfjrGOi+zNRLVvoDsk8pSbtvvxAdX45G/i5qnp0sLGqfkgXTh9OcgBAkkOSnNQPeSZd+H8/yf7A+TNRXP8u4HLggiTP7D88/C26EBtTktMGPtx9kC6kngS+CPxkkrck2av/eXmSFw4s/oYkr0nyNLq9/+ur6h7gS/2yv5pkYZJfAY7q1wnwD3QvdsfRHe64jf6dAt27I4AbgIeSvDfJPv2Hki9K8vJJPizj1UhVbQZupHs38tmqenycdUx0fyZyOXBO/zvxbOC9k7wPmgOG+zxSVd+uqvXjdL8XuBO4rv/vja/QBRh0H7buQ7dnfR077vFNt3fSHf//DnAt8BfAxTsZ/3Lg+iSP0H0gfE5Vbayqh4F/D5xOt+d6H/BHdB9MbvcXdC9U3wNeBvwaQFV9F3gT3TuW79Id+nlT9d8T6F8cbwZuq6of9Ov6R+Cuqtraj3kS+Hm6zyc20j12FwHPmuTjMWaNA1YBL2bHw007mOj+7II/p/ts4et079i+RPfB9pM7W0hzK/2HHtK8kuQSYHNVvW+uaxlGktfSvbNZ2r8Dm41tvh7406oafahHuxH33KU9VJK9gHOAi2Yy2PvDSm/oD+kcQvdO4vMztT1ND8Nd2gP1nx18HziI7rDZjG4O+H26zzRuofsXzN+b4W1qSB6WkaQGuecuSQ0y3CWpQbvFl5gWL15cS5cunesyJGmPctNNNz1QVUvG6tstwn3p0qWsXz/ev19LksaSZPRpJf6Vh2UkqUGGuyQ1yHCXpAZNGO5JLk53xZxbB9r+Z5JvJfl6ks/3JxPa3ndekjvTXfnmpLHXKkmaSbuy534J3Xm8B60BXlRVP0V3pZzzAJIcRXeipqP7ZT6WZMG0VStJ2iUThntVXUN3RrrBtqsGLrZwHT861/MpdFeOeaKqNtKdZfC4aaxXkrQLpuOY+38C/rqfPoQdr9KymR2v9iJJmgVDhXuS36U7r/OntzeNMWzMk9ckWZFkfZL1IyMjw5QhSRplyl9iSrKc7gIAJ9SPzj62mR0vwXUoP7ok2A6qaiXdhY1ZtmzZrJ69bOm5fzWbm5t1my5841yXIGmOTWnPPcnJdFfu+YWqemygazVwepK9kxwBHEl3uTFJ0iyacM89yWV0V3pfnGQz3Yn6z6O7XNma/srm11XVb1bVbUkup7u48Tbg7f3lxiRJs2jCcK+qM8Zo/vhOxl8AXDBMUZKk4fgNVUlqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ2aMNyTXJxka5JbB9r2T7ImyR397aKBvvOS3Jnk9iQnzVThkqTx7cqe+yXAyaPazgXWVtWRwNp+niRHAacDR/fLfCzJgmmrVpK0SyYM96q6BvjeqOZTgFX99Crg1IH2z1TVE1W1EbgTOG6aapUk7aKpHnM/sKq2APS3B/TthwD3DIzb3Lc9RZIVSdYnWT8yMjLFMiRJY5nuD1QzRluNNbCqVlbVsqpatmTJkmkuQ5Lmt6mG+/1JDgLob7f27ZuBwwbGHQrcO/XyJElTMdVwXw0s76eXA1cOtJ+eZO8kRwBHAjcMV6IkabIWTjQgyWXA8cDiJJuB84ELgcuTnA3cDZwGUFW3Jbkc+CawDXh7VT05Q7VLksYxYbhX1RnjdJ0wzvgLgAuGKUqSNBy/oSpJDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDRoq3JP81yS3Jbk1yWVJnp5k/yRrktzR3y6armIlSbtmyuGe5BDgXcCyqnoRsAA4HTgXWFtVRwJr+3lJ0iwa9rDMQmCfJAuBfYF7gVOAVX3/KuDUIbchSZqkhVNdsKr+KckHgbuBx4GrquqqJAdW1ZZ+zJYkB0xTrRIAS8/9q7kuYUZtuvCNc12CGjDMYZlFdHvpRwAHA89I8uZJLL8iyfok60dGRqZahiRpDFPecwdOBDZW1QhAks8BrwbuT3JQv9d+ELB1rIWraiWwEmDZsmU1RB2S9hC+65o9wxxzvxt4ZZJ9kwQ4AdgArAaW92OWA1cOV6IkabKGOeZ+fZIrgJuBbcAtdHvi+wGXJzmb7gXgtOkoVJK064Y5LENVnQ+cP6r5Cbq9eEnSHPEbqpLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkho0VLgneXaSK5J8K8mGJK9Ksn+SNUnu6G8XTVexkqRdM+ye+0eAL1fVC4CXABuAc4G1VXUksLaflyTNoimHe5J/A7wW+DhAVf2gqr4PnAKs6oetAk4dtkhJ0uQMs+f+PGAE+ESSW5JclOQZwIFVtQWgvz1gGuqUJE3CMOG+EHgp8CdVdSzwKJM4BJNkRZL1SdaPjIwMUYYkabRhwn0zsLmqru/nr6AL+/uTHATQ324da+GqWllVy6pq2ZIlS4YoQ5I02pTDvaruA+5J8vy+6QTgm8BqYHnfthy4cqgKJUmTtnDI5d8JfDrJ04DvAGfRvWBcnuRs4G7gtCG3IUmapKHCvaq+Biwbo+uEYdYrSRqO31CVpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUoKHDPcmCJLck+WI/v3+SNUnu6G8XDV+mJGkypmPP/Rxgw8D8ucDaqjoSWNvPS5Jm0VDhnuRQ4I3ARQPNpwCr+ulVwKnDbEOSNHnD7rn/MfA7wA8H2g6sqi0A/e0BQ25DkjRJUw73JG8CtlbVTVNcfkWS9UnWj4yMTLUMSdIYhtlz/2ngF5JsAj4D/FySTwH3JzkIoL/dOtbCVbWyqpZV1bIlS5YMUYYkabQph3tVnVdVh1bVUuB04G+r6s3AamB5P2w5cOXQVUqSJmUm/s/9QuB1Se4AXtfPS5Jm0cLpWElVrQPW9dPfBU6YjvVKkqbGb6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lq0JTDPclhSf4uyYYktyU5p2/fP8maJHf0t4umr1xJ0q4YZs99G/Ceqnoh8Erg7UmOAs4F1lbVkcDafl6SNIumHO5VtaWqbu6nHwY2AIcApwCr+mGrgFOHLVKSNDnTcsw9yVLgWOB64MCq2gLdCwBwwHRsQ5K064YO9yT7AZ8F3l1VD01iuRVJ1idZPzIyMmwZkqQBQ4V7kr3ogv3TVfW5vvn+JAf1/QcBW8datqpWVtWyqlq2ZMmSYcqQJI0yzH/LBPg4sKGq/tdA12pgeT+9HLhy6uVJkqZi4RDL/jTwFuAbSb7Wt/034ELg8iRnA3cDpw1XoiRpsqYc7lV1LZBxuk+Y6nolScPzG6qS1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaNGPhnuTkJLcnuTPJuTO1HUnSU81IuCdZAHwUeD1wFHBGkqNmYluSpKeaqT3344A7q+o7VfUD4DPAKTO0LUnSKAtnaL2HAPcMzG8GXjE4IMkKYEU/+0iS22eolt3BYuCB2dpY/mi2tjRv+PztuVp/7g4fr2Omwj1jtNUOM1UrgZUztP3dSpL1VbVsruvQ1Pj87bnm83M3U4dlNgOHDcwfCtw7Q9uSJI0yU+F+I3BkkiOSPA04HVg9Q9uSJI0yI4dlqmpbkncAfwMsAC6uqttmYlt7iHlx+KlhPn97rnn73KWqJh4lSdqj+A1VSWqQ4S5JDTLcJalBhrs0IMkLkpyQZL9R7SfPVU3adUmOS/LyfvqoJL+V5A1zXddc8APVWZTkrKr6xFzXobEleRfwdmADcAxwTlVd2ffdXFUvncv6tHNJzqc7n9VCYA3dt+LXAScCf1NVF8xddbPPcJ9FSe6uqufOdR0aW5JvAK+qqkeSLAWuAD5ZVR9JcktVHTunBWqn+ufvGGBv4D7g0Kp6KMk+wPVV9VNzWuAsm6nTD8xbSb4+Xhdw4GzWoklbUFWPAFTVpiTHA1ckOZyxT6mh3cu2qnoSeCzJt6vqIYCqejzJD+e4tllnuE+/A4GTgAdHtQf4h9kvR5NwX5JjquprAP0e/JuAi4EXz21p2gU/SLJvVT0GvGx7Y5JnAYa7hvZFYL/tATEoybrZL0eT8FZg22BDVW0D3prkz+amJE3Ca6vqCYCqGgzzvYDlc1PS3PGYuyQ1yH+FlKQGGe6S1CDDXRolyZeSPHuCMR9IcuJs1SRNlsfcpV6S0P1NzLv/rFB73HPXvNJ/Hf3W/ufdSZYm2ZDkY8DNwGFJNiVZ3I//70m+lWRNksuS/HbffkmS/9hPb0ry+0luTvKNJC+Yu3sodQx3zRtJXgacRfe19FcCvw4sAp4PXFpVx1bVXQPjlwG/BBwL/CKws2txPtCfnuBPgN+emXsg7TrDXfPJa4DPV9Wj/TdRPwf8O+CuqrpunPFXVtXjVfUw8IWdrPtz/e1NwNJprFmaEsNd88l4pxB4dJLjx/JEf/skfjlQuwHDXfPJNcCpSfZN8gzgPwBf3cn4a4GfT/L0/hTAb5yNIqXp4B6G5o2qujnJJcANfdNFPPUcQIPjb0yyGvi/wF3AeuCfZ7pOaTr4r5DSTiTZrz+B2L50e/4rqurmua5Lmoh77tLOrUxyFPB0YJXBrj2Fe+6S1CA/UJWkBhnuktQgw12SGmS4S1KDDHdJapDhLkkN+v8V7UTycOREkAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.groupby(\"origin\")[\"horsepower\"].mean().plot(kind=\"bar\", title=\"Mean horsepower by origin\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 363,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEICAYAAABI7RO5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xb1fn48c9jW957ryTOdBbZC8JKIawyS8veUEoLpXTQPejg25YfUKClUEbLaFktK7RhhQx2SMgiy47jJI4TJ97b8jy/P+614yHZsi3JdvK8Xy+9ZN977rlHiuJHZ4sxBqWUUspTAUNdAKWUUiOLBg6llFL9ooFDKaVUv2jgUEop1S8aOJRSSvWLBg6llFL9ooHjGCUij4rIL7yU12gRqRWRQPv31SJykzfytvN7U0Su9VZ+/bjv70SkVEQOuTh3qogU+rtMI5GIPCUivxuie3v82Rmqz9lIFDTUBVDeJyJ7gRSgBWgFtgPPAI8ZY9oAjDG39COvm4wxK9ylMcYUAJGDK3XH/e4CJhhjruqU/9neyLuf5RgFfB8YY4wp9vf9lXf057MzFJ+zkUprHEev84wxUcAY4A/Aj4AnvX0TETlav3yMAcr8ETSOpvdwuLwWsejfNx/RN/YoZ4ypMsYsAy4FrhWR6dC1+UBEEkXkvyJSKSLlIvKBiASIyLPAaOANuynqhyKSJSJGRG4UkQJgZadjnf9ojBeRz0SkSkReF5F4+149mnhEZK+InC4iZwE/BS6177fZPt/R9GWX6+cisk9EikXkGRGJsc+1l+NaESmwm5l+5u69EZEY+/oSO7+f2/mfDrwLpNvleKqXPL5vl6NIRK7vK2/73HUi8pGI/ElEyoG7RGSCiKyx369SEXmxU16TReRd+98mR0Qu6XTuKbvZ8V0RqbHzGNPp/Akiss7Od52InGAfXyIiX3RKt0JEPuv0+4cicqH9c7qIvGy/lj0icnundHeJyH9E5J8iUg1c5+atSnRVRhF5WETu6/aeviEid7h5v12+HvvcahG5W0Q+AuqBcd0+O4Eicp/9/u4Rkds6f267pb3Ofg/uFZEKO73WSNoZY/RxlD2AvcDpLo4XAN+0f34K+J398++BRwGH/TgJEFd5AVmAwWr6igDCOh0LstOsBg4A0+00LwP/tM+dChS6Ky9wV3vaTudXYzWXAdwA5AHjsJrHXgGe7Va2x+1yzQQagSlu3qdngNeBKPvaXOBGd+Xsdu2pWE2Bv7Hfs3Ow/ljFeZD3dfa138ZqLg4Dngd+hvVlLhQ40U4bAewHrrfTzgFKgWmd/h1rgJOBEOBB4EP7XDxQAVxtX3u5/XuCfY8GINE+dwg4aJc3zD6XYJfnc+CXQLD9vucDZ3b692oGLrTThrl4r3or4wL7vgH274n2+5jiIh+3r6fT56QAmGafd9D1s3MLVrNtJhAHrKDn5/amTv9GzcDXgUDgm3Y5Zaj/fw+Hh9Y4ji0Hsf7zddcMpGG15zcbYz4w9v+eXtxljKkzxjS4Of+sMWarMaYO+AVwidid54N0JXC/MSbfGFML/AS4rFtt59fGmAZjzGZgM1YA6cIuy6XAT4wxNcaYvcB9WH+UPNUM/MZ+z5YDtUC2h3kfNMb82RjTYr+HzVjNY+nGGKcx5kM73bnAXmPMP+y0G7AC8Vc75fU/Y8z7xphGrOBzvFh9NF8GdhljnrWvfR7YidWM6QTWY/0xnwdsAT4EFgOL7OvKgPlAkjHmN8aYJmNMPlZgvqzT/T8xxrxmjGnr5fPgsozGmM+AKuA0O91lwGpjzGEXebh9PZ3SPGWM2Wafb+52/SXAg8aYQmNMBVYTbm/2GWMeN8a0Ak9j/R9J6eOaY4IGjmNLBlDu4vj/w/oW/46I5IvIjz3Ia38/zu/D+vaX6FEpe5du59c57yC6/ofuPAqqHtcd94lY36C755XRj7KUGWNaXNzLk7y7v38/BAT4TES2icgN9vExwEKxmhErRaQSK3imusrLDqblWO9T9/eqeznWYNWcTrZ/Xg2cYj/WdLp/erf7/5Su73dfn4XeygjWH+X2wRBXAc+6yaOv19NXWdK7ne+r3B2fI2NMvf2jVwaBjHTDoiNL+Z6IzMf6D/Zh93PGmBqsEUTfF5FpwCoRWWeMeQ+rKu9KXzWSUZ1+Ho31jboUqAPCO5UrEEjqR74Hsf6Ydc67BTiM1QThqVKOfMvf3imvA/3IYzB5d3mdxphDWM0iiMiJwAoReR/rj9saY8zSXu7X8V6LSCRWrfIgPd+r9nK8Zf+8BqsmVID17bsCqzbRCDxsp9kP7DHGTOzl/p4sse2ujAD/BLaKyExgCvCamzz6ej19laWIrp+RUe4Sqt5pjeMoJyLRInIu8AJW38EXLtKca3fOClCNNYS31T59GKtdu7+uEpGpIhKO1Q/wH7vKnwuEisiXRcQB/Byr3bvdYSBL3I+IeR74roiMtf8A/R/wYrdv/n2yy/IScLeIRNmdtd/D+iM2KAPJW0S+JiLtf9QqsP4AtgL/BSaJyNUi4rAf80VkSqfLzxGRE0UkGPgtsNYYsx9Ybl97hYgEicilwFQ7T4CPgWysfobPjDHbsGs4wPt2ms+AahH5kYiE2R3M0+0vIv3hrowYYwqBdVg1jZd7ae7q6/X05SXgOyKSISKxWCMN1QBo4Dh6vSEiNVjfGH8G3I/VwerKRKyOwlrgE+CvxpjV9rnfAz+3myl+0I/7P4vVKXoIqyP2drBGeQHfAp7A+gZeB3QeZfVv+7lMRDa4yPfvdt7vA3sAJ1Yn80B8275/PlZN7Dk7f2/ob97zgbUiUgssA75jjNlj1wbPwGr7P4j1fv6RrsH2OeBXWM0/c7GasrD7KM7Fqk2WYTWHnWuMKbXP1wEbgG3GmCY7r0+w2vaL7TStWH0Is7De71Ksf7uYfr4fLsvYydPAcbhvpurz9XjgceAdrP6cjViBqH2uk+qH9pEzSqkRSKyhwoXGmJ8PdVkGQ0ROxqqRZRl7kqof7nk28Kgxpnvzl+qD1jiUUkPKbrL8DvCEL4OG3dR2jt3MlYFVA3rVV/c7mmngUEoNGbuvphJrqOsDvr4d8GusPqSNwA6s+Smqn7SpSimlVL9ojUMppVS/HBPzOBITE01WVtZQF0MppUaUzz//vNQYk9T9+DEROLKysli/fv1QF0MppUYUEek+Ux/QpiqllFL95NPAISJnibUMdJ6r9Y/E8pB9fouIzLGPjxKRVSKyw1635zudrrlLRA6IyCb7cY4vX4NSSqmufNZUZa9B9DCwFGtm8DoRWWaM2d4p2dlYs5YnYi1z8Ij93AJ83xizQUSigM9F5N1O1/7JGHOvr8qulFLKPV/2cSwA8uxlmBGRF4ALOLLoG/bvz9hLeH8qIrEikmaMKcJakAxjTI2I7MBaoG87Sik1QjQ3N1NYWIjT6RzqovQqNDSUzMxMHA6HR+l9GTgy6LpscSFWbaKvNBnYQQOsXd2A2cDaTuluE5FrsPYT+L69tn4XInIzcDPA6NGjB/oalFJqwAoLC4mKiiIrKwtrDdHhxxhDWVkZhYWFjB071qNrfNnH4epd6j7bsNc09uqnLwN3GGOq7cOPAOOxFl0rwloWumcmxjxmjJlnjJmXlNRjNJlSSvmc0+kkISFh2AYNABEhISGhX7UiXwaOQrqud5/JkfX3+0xjr1/zMvAvY8wr7QmMMYeNMa32mjaPYzWJKaXUsDScg0a7/pbRl4FjHTDR3jchGGtZ6GXd0iwDrrFHVy0CqowxRfa+EE8CO4wx93e+QETSOv16EbDVdy9B+ZqzxckLO18gvzJ/qIuilPKQzwKHvbHObcDbWIuJvWSM2SYit4jILXay5Vj7FeRh1R6+ZR9fjLU/85dcDLu9R0S+EJEtwBLgu756Dcr37ll3D3evvZub3rmJhhZ3+/copQbjhhtuIDk5menTp3slP5/OHDfGLMcKDp2PPdrpZwPc6uK6D3Hd/4Ex5movF1MNkarGKpbtXsboqNEU1BSwYt8Kzht/3lAXS6mjznXXXcdtt93GNddc45X8dOa4GjLrDq2jsbWR3y7+LYlhibxf+H7fFyml+u3kk08mPj7ea/kdE2tVqeFpbdFawoLCOC7pOE5IP4EPD3yIMWZEdCYq1V+/fmMb2w9W952wH6amR/Or86Z5NU9PaI1DDZntZduZnjgdR4CD6YnTKXeWc6ju0FAXSynVB61xqCHRZtrIq8zjwgkXAjA9weq021a2jbTItN4uVWpEGoqaga9ojUMNiYO1B6lvqWdC3ASAjue8yryhLJZSygMaONSQaA8QE2MnAhAWFEZ6RDp7qvYMZbGUOipdfvnlHH/88eTk5JCZmcmTTz45qPy0qUoNifbAMSF2QsexsTFjNXAo5QPPP/+8V/PTGocaEoU1hSSEJhAZHNlxbGzMWPZW76XNtA1hyZRSfdHAoYbEgdoDpEemdzk2JnoMDS0NlDaUDlGplFKe0MChhsTB2oM9Akf77wdru6+FqZQaTjRwKL9rM20U1RX1DBwRGjiUGgk0cCi/K20opbmtmYyIjC7HO2ocdRo4lBrONHAov2uvUXSvcYQ7wokNidUah1LDnAYO5XcHag8APQMHQFpEmtY4lPKi/fv3s2TJEqZMmcK0adN48MEHB52nzuNQfldUZ20pnxbRc2mRjMgMdlft9neRlDpqBQUFcd999zFnzhxqamqYO3cuS5cuZerUqQPOU2scyu9K6kuIdEQS7gjvcS41IpXi+uIhKJVSR6e0tDTmzJkDQFRUFFOmTOHAgQODylNrHMrvShpKSAxLdHkuKTyJuuY66prriHBE+LlkSvnQmz+GQ194N8/U4+DsP3icfO/evWzcuJGFCxcO6rZa41B+V1JfQlJ4kstzSWFJHWmUUt5TW1vLxRdfzAMPPEB0dPSg8tIah/K7koYSZiTNcHkuOTy5I01WTJYfS6WUj/WjZuBtzc3NXHzxxVx55ZV85StfGXR+WuNQfmWMobShtKNm0V17TUT7OZTyDmMMN954I1OmTOF73/ueV/LUwKH8qqa5hsbWxo6aRXfJYXaNo74Edi6Hj/8Czip/FlGpo8pHH33Es88+y8qVK5k1axazZs1i+fLlg8pTm6qUX5XWWwsYuuscj3BEEBYURvHeVbD+NevgtlfhhrchUD+uSvXXiSeeiDHGq3lqjUP5VUmD1entrqlKREgOTaCk8FMYezJc+CgcWA8bnvJjKZVSvdHAofyqPXAkhruucQAktUGxtMFpd8HMyyBjHnzyMLTpPh1KDQcaOJRftTdVuatxACTVllESHAYZc0AE5t8E5flWzUMpNeQ0cCi/Km0oJSQwhEhHpOsE9eUk1RRTEhhAR6vs5HMgMBi2veavYiqleqGBQ/lVRWMF8aHxiIjrBPs+IrG1Badppb6l3joWGgPjToXcN/1VTKVULzRwKL+qbKwkNiTWfYI9H5BgAgEobyg/cnzcEqu5qnK/j0uolOqLBg7lV5XOPgLHvo+IT5gEQJmz7MjxcadYz3vW+LB0Sh19nE4nCxYsYObMmUybNo1f/epXg85TA4fyq4rGCmJD3QSOpnoo3k5CirUcSVlDp8CRPBUikmDP+34opVJHj5CQEFauXMnmzZvZtGkTb731Fp9++umg8tTAofyq16aq4u1g2khIs5aA7lLjEIFRC6FQR1Yp1R8iQmSkNRilubmZ5uZm932MHvLpVFwROQt4EAgEnjDG/KHbebHPnwPUA9cZYzaIyCjgGSAVaAMeM8Y8aF8TD7wIZAF7gUuMMRW+fB3KO1raWqhpqiEuJM51gqLNAMRlLoJN3QIHQMZc2PlfqC+H8Hgfl1Yp7/rjZ39kZ/lOr+Y5OX4yP1rwoz7Ttba2MnfuXPLy8rj11luH77LqIhIIPAycDUwFLheR7ltOnQ1MtB83A4/Yx1uA7xtjpgCLgFs7Xftj4D1jzETgPft3NQJUNVprTsWExLhOULQZQmNxxI8jJiSma1MVWIED4OAGH5ZSqaNPYGAgmzZtorCwkM8++4ytW7cOKj9f1jgWAHnGmHwAEXkBuADY3inNBcAzxlpI5VMRiRWRNGNMEVAEYIypEZEdQIZ97QXAqfb1TwOrgb5DrhpylY2VAMSF9lLjSJsJIsSHxlPuLO96Pn02IHBgA0w43beFVcrLPKkZ+FpsbCynnnoqb731FtOnTx9wPr7s48gAOo+dLLSP9SuNiGQBs4G19qEUO7BgP7tcZlVEbhaR9SKyvqRENwUaDiqcVouiyz6OtjYo2Qkp0wBICE3oWeMIjYakbO3nUKofSkpKqKy0vrQ1NDSwYsUKJk+ePKg8fRk4XPW+dF+isdc0IhIJvAzcYYyp7s/NjTGPGWPmGWPmJSW5X95C+U97U5XLwFG1H1qckGgNxU0IS+hZ4wBInwMHN/qymEodVYqKiliyZAkzZsxg/vz5LF26lHPPPXdQefqyqaoQGNXp90zgoKdpRMSBFTT+ZYx5pVOaw+3NWSKSBuiOPyNERaNV43DZVFW6y3q2A0d8aHzPznGA1Omw+TmoLYFI/UKgVF9mzJjBxo3e/bLlyxrHOmCiiIwVkWDgMmBZtzTLgGvEsgiosgOCAE8CO4wx97u45lr752uB1333EpQ3tfdxuOwcL821nttrHKEJ1DTV0NTa1DWd3ZRF8TZfFVMp1QefBQ5jTAtwG/A2sAN4yRizTURuEZFb7GTLgXwgD3gc+JZ9fDFwNfAlEdlkP86xz/0BWCoiu4Cl9u9qBKh0VhIaGEpYUFjPk6W5EBoLEdZy6wlhCQA9m6tS7A69wxo4lBoqPp3HYYxZjhUcOh97tNPPBrjVxXUf4rr/A2NMGXCad0uq/KHXWeOlu6zahj0xKT7UmqdR5iwjNSL1SLqIRIhM0cChRgxjzKAn3Plaf3cI1Jnjym+qGqvczxovze1opoJONY4GFx3kKdPg8ODGoSvlD6GhoZSVlXl961ZvMsZQVlZGaGiox9foJs7KbyoaK1wHDmcV1BVD4oSOQ51rHD2kTIO1j0Fri+5Droa1zMxMCgsLGe5TAkJDQ8nMzPQ4vf6vU35T1VhFRkT3qTxAxT7rOW5sx6GEUKvG0WMuB1j9HK2NUJYHyYMbj66ULzkcDsaOHdt3whFGm6qU31Q4K1yPqKqy54DGHhmZHe4IJywozPVcjvaRVdpcpdSQ0MCh/KKlrYXqpmrXczgqC6zn2DFdDrudy5EwEZAjcz+UUn6lgUP5RXWTNfHfZY2jcj8EhUF4QpfDCWEulh0BcIRC3BgozfFFUZVSfdDAofyi0mkvcOhqSfWqAquZqtuQxYTQBNc1DoDEbCjJ9XYxlVIe0MCh/KJ91rjLeRyVBRA7usdhtzUOgKRJVud4W6s3i6mU8oAGDuUX7etUuRyOW7kfYkb1OJwQmkBlYyWtroJDYrY1sqpyn7eLqpTqgwYO5Rdum6oaa6Gh3G2No820dQSdLtonC2pzlVJ+p4FD+YXbBQ47huK6CBy9zeVIsgOHdpAr5XcaOJRfVDZWEhIY0nOBw0o7cLhqqrKXHXHZQR4WBxHJR1bVVUr5jQYO5ReVjZXEhsT2XOytvY/CRY0jMcxaKdd9B7mOrFJqKGjgUH5R6ax0v/NfYLC14m03vTZVASROtJqqhvECckodjTRwKL9wu6R65X6IyYSAnh/FCEcEIYEhvc/lcFZB3fBeQE6po40GDuUXbpdUryxw2b8BICLWJMDe5nIAlGgHuVL+pIFD+YXbJdWr9ndZ3LC7hLDeZo+3j6zSfg6l/EkDh/K51rZWqhtdLHDY7ITawz0WN+ys1xpHVDo4wq0Z5Eopv9HAoXyuuqkag+lZ46gqtJ7dNFWBVeMobSh1fTIgABImaI1DKT/TwKF8zu1yI1Xty6n3HIrbLiEsgYrGCtfLjoA9ssr18ur/zf8vj25+lPrm+n6XWSnlnu4AqHyuqrEKcBE4Ovbh6KXGEWotO1LZWNkxIbCLxEmw9RVobgDHkcmFKwtW8pMPfgJAUV0Rvz7h14N7EUqpDlrjUD5X4bRrHN2H41buBwm0+irc6HX2OFg1DgyU53c5/OjmR8mKzuLS7EtZlrfMfT+JUqrfNHAon2uvcfRY4LCyAKIzINB9xbfPSYAJE63nTv0ce6r2sKN8B5dmX8pl2ZfRYlp4Z987A38BSqkuNHAon3Pfx9H7UFzwoMaRMMF6Lj0ysuq9gvcAWDpmKRPiJpAekc7aorUDKLlSyhUNHMrnKp2VBAcEu17gsJeOcegUONzVOILDrVFZnWoc6w6tY2LcRFIirGVM5qfOZ/3h9bSZtoG/CKVUBw0cyudcLnDY2gw1B3sdigsQ5YgiOCC49z6KxIlQZo2sam5rZmPxRuYmz+04PTdlLlWNVeyt2juYl6GUsmngUD7ncp2q6gNg2vpsqhKR3mePg9XPUboLjGFn2U4aWhqYlzqv4/SUhCkA5FTo0iRKeYMGDuVzVY1VLjrG3W/g1F2vs8fBqnE01ULNIbaWbQVgZtLMjtPjY8YTFBBETrkGDqW8QQOH8rkKZ0XPnf/a53D00VQFfaxXBfaQXKA0l5zyHKKDo0kJP7JMuyPQwbiYcVrjUMpLNHAon6tsrOy5TlXVfkCsJdX7kBDWV43DXuywbBe7KnaRHZ/dY8Oo7Lhscst1aRKlvEEDh/Kp1rZWqpuqXdc4olIhKKTPPBLDEilzltHS1uI6QVQaBEfSWpLLrspdZMdl90gyLnYcxQ3FuvyIUl7g08AhImeJSI6I5InIj12cFxF5yD6/RUTmdDr3dxEpFpGt3a65S0QOiMgm+3GOL1+DGpyaphraTJvryX8eNFMBpISn0Gba3C92KAIJEygs3U5DSwOT4ib1SDIqyrpXQU1Bv8qvlOrJZ4FDRAKBh4GzganA5SIytVuys4GJ9uNm4JFO554CznKT/Z+MMbPsx3KvFlx5VWVjJeBiuZGqvudwtEuNSAXgcP1h94kSJ5JTbe1fPim+Z+AYE20t3V5QrYFDqcHyZY1jAZBnjMk3xjQBLwAXdEtzAfCMsXwKxIpIGoAx5n2g3IflU37QETg6zxpva7WWVO9jKG679o7uw3W9BY5J5LZUESABjI8Z3+O01jiU8h5fBo4MYH+n3wvtY/1N48ptdtPW30Ukru/kaqi0L3DYpamq5hC0tXi3xpEwgZxgB1nhaYQGhfY4HeGIIDEsUWscSnmBLwOHuDhmBpCmu0eA8cAsoAi4z+XNRW4WkfUisr6kpKSvsiofaa9xdOkcr7K/K8R4Fjiig6MJDQztu8YRHMykYBfb09pGR41mn92cpZQaOF8GjkKgc1tEJnBwAGm6MMYcNsa0GmPagMexmsRcpXvMGDPPGDMvKSmp34VX3tEeOLoMx/VgH47ORISUiBQO1R9ym6YmKoWDjiCyjcNtmlFRoyisKfTonkop93wZONYBE0VkrIgEA5cBy7qlWQZcY4+uWgRUGWOKesu0vQ/EdhGw1V1aNfQqGytxBDgIDwrvdNDzyX/tUsJTeq1x5NZaeU5qcD/cNj0ynZKGEppbmz2+r1KqJ58FDmNMC3Ab8DawA3jJGLNNRG4RkVvsZMuBfCAPq/bwrfbrReR54BMgW0QKReRG+9Q9IvKFiGwBlgDf9dVrUIPncoHDygIIT7RWtvVQakRqr30cuRXW5L5JVe7TpEWkYTC995Uopfrk061j7aGyy7sde7TTzwa41c21l7s5frU3y6h8q8LpYoFDD/bh6C4lPIWS+hJa21oJDAjscT6nPIcYcZBSmg/GWHM7ukmLtCqrRXVFZEb1PWNdKeWazhxXPuV2gUMPR1S1S4tMo8W0UNLgeqBDbkUuk8JSkOY6qHbdTZYWcSRwKKUGzqPAISIvi8iXRUQDjeqXisZuCxwaY9U4+tG/AZ3mYbgYTtva1mqtURXXcxvZztrngxTVauBQajA8DQSPAFcAu0TkDyIy2YdlUkeRHjWO2mJocULsmH7lMzrKqqG4msBXUFOAs9VJdqq9eVNZXo80AKFBocSHxmuNQ6lB8ihwGGNWGGOuBOYAe4F3ReRjEbleRNyPf1THtDbTRmVjZdcaR/uIqrj+BY6U8BQcAQ6XgaN9n43s1PkQHOW2xgFWc9WhOvfDepVSffO46UlEEoDrgJuAjcCDWIHkXZ+UTI147QscdllupNKegNfPPo7AgEAyozLZX72/x7mcihyCJIjxcROsvTlKd7nNJy0iTWscSg2Sp30crwAfAOHAecaY840xLxpjvg1E+rKAauTqWG6ky+S/gQUOsPo59te4CBzlOYyNHUtwYHCfgSM1IpWiuiKsAX1KqYHwtMbxhDFmqjHm9+0T9EQkBMAYM6/3S9WxyuWs8Yp99hyOiH7nNzpqNAU1BT3+6OeU5zA5zu52S5wI1YXQVOcyj7SINBpaGqhqrOr3/ZVSFk8Dx+9cHPvEmwVRRx+XCxxWFvS7f6PduNhxNLQ0UFh7ZNmQkvoSihuKmRxvB44Ee2SVmw7ylAh7pV2dBKjUgPUaOEQkVUTmAmEiMltE5tiPU7GarZRyy+06VQNopgKYEj8FgJ3lOzuObS7ZDMCs5FnWgfZtZN00V7UPyXU3H0Qp1be+Zo6fidUhngnc3+l4DfBTH5VJHSUqGq0aR0fneFubNYdjynkDym9i3EQCJZAdZTtYOmYpABuLNxIcENwRVIgfB4jbwJEUbi14WVxfPKAyKKX6CBzGmKeBp0XkYmPMy34qkzpKVDorCQkMISwozDpQewhamwZc4wgJDGFc7Dh2lO/oOLapeBPTE6fjCLRHhTtCraawMjeBI0wDh1KD1WvgEJGrjDH/BLJE5Hvdzxtj7ndxmVKAVePossBhhT2iaoB9HAAzEmfwzt53aGlrobqpmq1lW/nGjG90TZQw0e1cjuDAYOJC4iip16YqpQaqr87x9qEvkUCUi4dSblU4K9zswzHwwHFixonUNNewqXgTHxR+QJtp45RRp3RNlDgJynZbTWMuJIUnaY1DqUHoq6nqb/bzr/1THHU0qWis6Daiyq5x9HOdqs4WpS3CEeBg+Z7l5FbkkhmZydT4qV0TJU6A5nqoPuByFd6k8CSKGzRwKDVQnk4AvEdEokXEISLviUipiFzl68KpkSxb8aEAACAASURBVK3SWdl1SfXKfRCZavVDDFBkcCQXTbiIf+f+m80lm7lu2nVd9/qATiOr3C92qE1VSg2cp/txnGGM+aGIXIS13evXgFXAP31WMjXi9ahxVOwbcMd4Z3fMvYPmtmYSwhL4WvbXeiboPJdjwmk9TieFJVHmLKOlrYWgAJ9uSaPUUcnT/zXtCxmeAzxvjCnv8S1PqU6a25qpaarpWuMo3wNZiwedd1RwFL9Z/Jsexwsr6nl4VR4f7iphOeFs/ewTRk26msy4rlOOksOTaTNtlDWUdUwIVEp5ztOZ42+IyE5gHvCeiCQBTt8VS4107Ut6dNQ4mhuspUDix/vkfh/nlXL2gx/w6sYDzMiMozR0DKZkF2c/8AFrcrs2SyWHJwM6CVCpgfJ0WfUfA8cD84wxzUAdcIEvC6ZGtkqnNWu8o8ZRvsd6TvB+4PiisIobnl5HekwY7373FB6+cg5js2exIKqMzPhwvv7Metbml3Wkb58EqMuOKDUw/dnRbwpwqYhcA3wVOMM3RVJHg/ZZ4/Eh8daB8t3Wc/w4r96nxtnMbc9vID48mH99fSGj4u1mqcSJBNUV8dw108iMC+O25zdSWtsIdFp2RDvIlRoQT0dVPQvcC5wIzLcfuiqucqt9gcOOGkeZHTi8XOO4560cCisaeOjy2SRGhhw5YXeQxzXs469XzqGqoZkfv7zFOhYSR6AE6lwOpQbI087xecBUo5sYKA91LHDY3sdRlmctpx4a08tV/bPzUDX/WruPa47PYl5WfNeTHUNy85g8YzZ3npHN3ct38N6Ow5w2JYWEsAQNHEoNkKdNVVuBVF8WRB1dOmoc7QscludDwgSv3uOPb+4kKtTBHadP7HkyfixIQMdcjusWZzEhOZJfv7GdppY2ay6Hdo4rNSCeBo5EYLuIvC0iy9ofviyYGtkqGyuJdEQeWXywbLdXm6m2H6xmVU4JN588jtjw4J4JgkIgLqtjsUNHYAA/PWcyBeX1vLKhkKQwXXZEqYHytKnqLl8WQh19Kho7rVPVWGutjOvFjvFH1uwmMiSIqxb1su5VQtdtZJdkJzMzM4a/rMrjjJOTWH94vdfKo9SxxNPhuGuAvYDD/nkdsMGH5VIjXKWz8kj/Rnm+9eylGsf+8nr+t+UgVy4cTUyYw33CxIlW34q92KGIcMfpkyisaKC0IpTqpmqcLTodSan+8nRU1deB/wB/sw9lAK/5qlBq5Ct3lh8ZUdW+ZpSX+jie/8xaZfe6xVm9J0ycCC1OqCroOHRqdhJT06LZuKcV0CG5Sg2Ep30ctwKLgWoAY8wuINlXhVIjX2Vj5ZGO8eIdIIFHRjoNQnNrG//+vJAl2cmkxYT1njh5mvV8eFvHIRHhusVZHCy3hu7qJECl+s/TwNFojGlq/0VEggAdmqvcqmzs1FRVvMNqpgoK6f0iD7y3o5iSmkYuX+DBYokpUwGBQ190OXz+zHSiHQmALjui1EB4GjjWiMhPgTARWQr8G3jDd8VSI1l9cz0NLQ1HOsdLdkDyFK/k/dL6/aREh3BqdlLfiYMjrOaqboEj1BHIV2dae3jklhZ6pVxKHUs8DRw/BkqAL4BvAMuBn/uqUGpkK3Na60IlhCVAU721TlXS4ANHRV0T7+eWcOHsDIICPfzoph4HRVt6HL7++CmYNgcf780fdLmUOtZ4NBzXGNMmIq8BrxljtG6velXWYAeO0AQozQGMV2ocb207REub4bwZ6Z5flHocbH0ZGiog7MjeIOmx4YRKHLllB2htMwQG6DYBSnmq169tYrlLREqBnUCOiJSIyC89yVxEzhKRHBHJE5Efu8n/Ifv8FhGZ0+nc30WkWES2drsmXkTeFZFd9nNc93zV0Cp3lgMQHxYPxTutg8lTe7nCM29sPsi4xAimpUd7flHqDOu5W3MVQEZUKk5TwYd5pYMum1LHkr7q+3dgjaaab4xJMMbEAwuBxSLy3d4uFJFA4GHgbGAqcLmIdP/rcTYw0X7cDDzS6dxTwFkusv4x8J4xZiLwnv27GkY6mqpCE6B4OwQGD3ryX3G1k0/yyzh3ZnrPrWJ700vgmJSYQZCjmpfW7x9U2ZQ61vQVOK4BLjfG7Gk/YIzJB66yz/VmAZBnjMm3R2S9QM89PC4AnjGWT4FYEUmz7/M+UO4i3wuAp+2fnwYu7KMcys+6NFUd2gJJkyFwcFu0Lv+iCGPgvBlp/bswMgmi0lz2c6RFpBLgqOHdbYeoqGtycbFSypW+AofDGNOjHm/3c/QyZRewJgl2/ipXaB/rb5ruUowxRXY5inAzn0REbhaR9SKyvqREu2X8qayhjOjgaBwBQXBwE6TPHnSeb2wpYnJqFBNTovp/cepxLmscSeFJtNFME7W8vunAoMuo1LGir8DR29ewvr6iuWpP6D73w5M0A2KMecwYM88YMy8pyYOhm8prypxlxIfGQ8VecFYOOnCU1TayoaCCs6YPcIHmtJlQshOa6rocbt9CdnxqK69tOjioMip1LOkrcMwUkWoXjxrguD6uLQRGdfo9E+j+v9OTNN0dbm/Osp91idNhpqyhzBqKe3CjdWCQgWN1TgnGwGmTUwaWQeZ8MK1HymNr3wlw4YRANu2vZG9pnaurlVLd9Bo4jDGBxphoF48oY0xfTVXrgIkiMlZEgoHLgO5LsS8DrrFHVy0CqtqboXqxDLjW/vla4PU+0is/K3eWW/0bBzdaHeODHFG1MqeYpKiQ/o2m6ixzvvW8/7Muh9v3Hp+Q3oYIvK61DqU80p89x/vFGNMC3Aa8DewAXjLGbBORW0TkFjvZciAfyAMeB77Vfr2IPA98AmSLSKGI3Gif+gOwVER2AUvt39UwUubsVONImQ5BLvbL8FBzaxvv55Zw6qQkAgY61yI8HuLHQ+G6LoeTw6ymqkYqWJAVz+ubD6CbXCrVt8ENdemDMWY5VnDofOzRTj8brAUUXV17uZvjZcBpXiym8qKm1iZqmmqID46FA5/D7KsHld+GfRXUOFv40uRBrqk5agHkrQBjwB7O6wh0EB8aT3F9MRfOzuAnr3zB1gPVHJfpve1tlToa+azGoY5N7ZP/Eprqobkexhw/qPxW5hTjCBROnJg4uIJlzoe6EqvDvpOksCRK6ks4Z3oajkDhNR1dpVSfNHAor+qYw1Fp9xeMPmFQ+a3aWcz8rHiiQvvqUuvDqAXWc7d+juTwZIrri4kJd3BqdjJvbD5Ia5s2VynVGw0cyqs6Zo2X7rZmi0cNcCQUUFhRT+7h2sE3U4HVQR8aA3vf73o4PLljT44LZ2VQXNPIJ7vLBn8/pY5iGjiUV7XXOOKLtsCYwdc2AJZ4I3AEBMLYk2H3aqufw5Ycnky5s5zm1mZOm5JMZEiQNlcp1QcNHMqrOmocdRUw9pRB5bVyZzGj48MZlxjhjaLBuCVQXWjtQ25rnwRY2lBKqCOQs6an8tbWQzibW71zT6WOQho4lFeVNpQSIQ7CDTD+SwPOp6GplY93l/Glycn9W9SwN+OXWM+7V3Ucag8c7c1VF8xKp7axhfd26LxSpdzRwKG8qri+mKS2Nmu2eMTAR0J9kl9KY0ubd5qp2sWPg7gs2L2y41B74CiutwLFCeMTSY4K4dWN2lyllDsaOJRXldQWkeysg4lLB5XPqp0lhDkCWTg23ksls008E/JXQWMNcCRwtO89HhggXDg7g9U5xZTVNnr33kodJTRwKK8qqSkkqbUVJpw+4DyMMazcWcziCYmEOgK9WDpg2kXQ4oTctwGIC4kjOCCYQ3WHOpJ8ZU4GLW2G/27pa/UbpY5NGjiU1xhjKG6qIjkgFDLmDTifXcW1HKhs8M4w3O5GLbT259j2KgAiQlpkGgdrj6xTNTk1milp0byyodD791fqKKCBQ3lNVc1BmjEkJ02DgIF/tFZ2DMP1wXL4AQEw9ULY9Q7UWVvNpEekU1TXtXZx8ZwMNhdWkVdc6/0yKDXCaeBQXlOc8wYASWNOHFQ+K3cWMzk1irSYMG8Uq6e510JrE2x4BoD0yHQO1HbtDD9/VjoBAq9u1FqHUt1p4FBeU5Jn9Rskj1o84Dyq6pv5fF+Fb5qp2iVPsSYDrnsSWppIj0yn3FmOs8V5JElUKCdNTOK1jQdp0yVIlOpCA4fyjqZ6ig9ZGyUlRQxwpz7g/V0ltLYZ3wYOgBO+Y00G/PwfpEVY+5gfrOu6H8dX5mRwoLKBtXvKfVsWpUYYDRzKO/JWUCzWbOv2DZIGYtXOYmLDHcweHeetkrk24TSr1rH6D6QHhgNQVNu1n+OMqalEhQTx4roC35ZFqRFGA4fyjm2vUBISQUxwDCGBIQPKoq3NsDq3hFMmJRE40E2bPCUCZ98DzfVkfPAgQI9+jrDgQC6ak8HyLw5RXtfk2/IoNYJo4FCD11QHuW9THJM+qNrG5sJKyuuafN9M1S55Cpz1e5J2ryEIoaimZ0f4VYvG0NTaxr/X7/dPmZQaATRwqMHLfRua6ykJDe+YiT0Qq3YWEyBw8kQfDMN1Z94NBC75OSnNzRz44nmo7NosNSkligVZ8Tz3WYF2kitl08ChBm/bqxCZQnFL/aACx8qcYmaPjiMuYuB7lA/IKXeSHjeBouZaeGQxbH6hy9LrVy4azb6yej7MK/V5UeoaW9h5qJqdh6p1hV41bPl0z3F1DGisgV3v0Dz7KkpK3+0YodRfxdVOth6o5s4zs71cQM+kp8zkk5ZqCBoDr34DcpbDeQ9CWBxnTU8lISKYpz7ey8mTfFMbWr+3nL+u3s0Hu0pobrWCliNQOHNaKt/+0kSyU6N8cl+lBkIDhxqcnLegxcmh8adgSt8ZcOBYlWPPFs/2U/9GNxmRGZQ4K2i8+k1C1v4NVt4N5Xvg2mWEhMVx7QlZ3P9uLjuKqpmSFu21+9Y3tXD3/3bwr7UFJEYGc/3isczIjKHNwIZ9Fby8oZC3th7ijtMncuuSCd5bYl6pQdCmKjU4216FqHQORVtbxKZFDixwrNxZTFpMKFPShuab9ejo0RgM+2sPwonfhcueg5Kd8MJV0NrCtcdnEREcyCOrd3vtnpX1TVz22Kc891kBN588jg9++CV+es4Uzp2Rzvkz07nr/GmsuXMJZx+Xxr3v5HLHi5toaW3z2v2VGigNHGrgGmshbwVMvYCiBmsjpIHUOBpbWvlwVymnZntx06Z+yorOAmBf9T7rwKQz4LyHYN+HsOaPxIQ7uGrRGP675SD7yuoGfb+KuiaufGItO4tqePzqefz0nCmEBfdcCTg+IpiHLpvFnWdm8/qmg3z/35tp1U56NcQ0cKiBy18FrY0w+csdk+dSBzBrfG1+OXVNrZw+ZWiaqcCqcQDsrd575OCsy2HGpfDhn6AklxtPHIsjMID7380d1L3Kahu5/PFP2VVcy2PXzOX0qSm9phcRbl0yoSN4/N/yHYO6v1KDpYFDDVzOmxAaA6MXUVRXREJowoAm/7234zChjgAWTxj4joGDFRUcRUJowpEaR7sz7gZHOLz1I5KjQ/n6SeN4fdNBNhZUDOg+pbWNXPH4WvaU1vHktfM4tR99OrcumcB1J2Tx5Id7ePlzXXxRDR0NHGpg2lqt+RsTz4BAB0V1RQNqpjLG8N7OYhaP98GmTf2UFZPVM3BEJsEpP7S2m933Cd88dTxJUSH8+o3t/W4yKq5xcvljn1JQXs8/rpvPSQOYr/KzL09h0bh4fvLqF2w9UNXv65XyBg0camAK10N9KUw6C8AKHAPoGM89XEthRQOnTem9ucYfsqKzujZVtZt3A0Qkwfv3EBESxE/Onsym/ZU8+WG+x3kfrnZy2WOfcqCygX9cP58TBli7cgQG8PAVc4gNc3DHi5toaNK5Hsr/NHCogcl9EwKCYMLpGGMoqi0aUP/Gih1Wp7rflhnpxZjoMZQ7y6lq7PZNPjgcTvi2Ves4sIGLZmdwxtQU7n0716Mmq31ldXz10Y85XOXk6RsWsGhcwqDKmRAZwn2XzCSvuFb7O9SQ0MChBibnLRhzAoTFUtlYibPVSXpEer+zWbmzmOkZ0aTGhPqgkP0zJnoMQM/mKoC514MjAtY9gYjwh4tnkBoTyk1Pr2d3iftdAj/fV85XH/2EWmcLz9+8iPlZ8V4p60kTk7jpxLE8++k+3rODr1L+ooFD9V91EZTsgAlLATq2Xe1vH0dZbSMbCio4bfLQN1MBTIidAMCuil09T4ZGw8xLYevLUF9OfEQw/7h+Pgb4yl8/ZuXOrn+865taeGBFLpf87VNCHQH8+5bjmZEZ69Xy3nlWNpNTo/jRy1soq230at5K9UYDh+q/PWus53GnAlBoryqbHtm/GsfqnBKMgdOGcBhuZ5lRmYQHhZNTkeM6wfyboMUJG/8JwPikSF771mJSokO44an1nP+XD/nZq19w63MbWPR/7/HAil2cc1wa/7v9JCYke39iY0hQIH+6dBbVDS387NWtGKPzO5R/+DRwiMhZIpIjInki8mMX50VEHrLPbxGROX1dKyJ3icgBEdlkP87x5WtQLuSvhvAESJkOQEGNtaJs+1wIT7238zDJUSFMT4/xdgkHJEACmBg3kZxyN4EjZRqMWgQbnu5YBHF0QjhvfPtE7jpvKoEBwv++KGJTQSVLp6by8jdP4M+XzyY61OGzMk9Ji+Z7Z0zirW2HeG3Tgb4vUMoLfLZWlYgEAg8DS4FCYJ2ILDPGbO+U7Gxgov1YCDwCLPTg2j8ZY+71VdlVL4yB/DXW7nkB1veO/TX7iQ+NJ8IR4XE2zuZW1uSUcP6sdAJ8vWlTP2THZfPmnjcxxriexT7rCnjjdjiwATLnAtY3/+sWj+W6xWP9XFrL108ax4rth/nl69tYODaB9NiwISmHOnb4ssaxAMgzxuQbY5qAF4ALuqW5AHjGWD4FYkUkzcNr1VAo3QU1BzuaqcAKHKOj+lfb+GBXKXVNrZw1fWBrW/lKdnw2Nc01PfYf7zDtQggKhc3P+7dgvQgMEO67ZCatbYY7/7NZ9w1RPufLwJEBdN42rdA+5kmavq69zW7a+ruIuNycWkRuFpH1IrK+pKRkoK9BdZe/2noee0rHoYLqgn43U735RRExYQ5OGD+4oaneNiluEgA7y3e6ThAaA5O/DFv/Ay3DZzvZMQkR/PzLU/kor4xnP3UxKkwpL/Jl4HDV/tD9q5C7NL1d+wgwHpgFFAH3ubq5MeYxY8w8Y8y8pCQ/7ih3tNuzBmLHQLzVLONscXK4/jCZUZkeZ9HU0sa7Ow5z+pQUHIHDa3zG5PjJBAUEsblks/tEMy+HhgrY9Y7/CuaByxeM4tTsJH7/5o5ehwgrNVi+/F9bCIzq9Hsm0L3+7y6N22uNMYeNMa3GmDbgcaxmLeUPrS2w54MuzVQHaq0O2f40VX28u5QaZwvnHNf/CYO+FhoUytSEqWwq3uQ+0bglEJE8rJqrwFoM8Z6LZxDqCOR7L23WJdiVz/gycKwDJorIWBEJBi4DlnVLswy4xh5dtQioMsYU9Xat3QfS7iJgqw9fg+qsaBM0VsG4rs1U0L/A8eYXh4gMCeLEiUO3qGFvZifNZmvpVhpb3cyNCAyCGZdYa3XVl/u3cH1Ijg7ldxdOZ/P+Sq/uHaJUZz4LHMaYFuA24G1gB/CSMWabiNwiIrfYyZYD+UAeVu3hW71da19zj4h8ISJbgCXAd331GlQ3+aus5879G/ZQ3FFRo1xd0UNLaxvvbD/EaVOSCQka2kUN3ZmdPJvmtma2l213n2jGJdDWDNte8V/BPNS+EdSD7+3ShRCVT/h061hjzHKs4ND52KOdfjbArZ5eax+/2svFVJ7KXwOpx0HEkZrC/pr9RDmiiAnxbC7G2j3lVNQ3c/b04ddM1W5W8iwA1h1ax+zk2a4Tpc6ApCmw+UVrYuAw85sLprF2TxnffXETb3z7xCFfeVgdXYZXz6QavprqYf/aLrUNgPyqfMbGjvV4577XNh4gKiSoX/tQ+FtCWAJT4qfwQeEH7hOJWEuQFH4GZcOvSSg2PJh7vjqTXcW13Pu2mwmNSg2QBg7lmYJPoLXJ6hjuZHflbsbHjPcoC2dzK29uPcRZ01OH/TfgU0adwpbSLVQ4e1n99rhLAIEtL/mtXP1xyqQkrlo0mic+3MPb2w4NdXHUUUQDh/LMnjUQ4IAxx3ccqnBWUO4sZ3ysZ4FjxY7D1Da2cNHs7tN5hp+TM06mzbTxfuH77hPFZMDYk2DLix1LkAw3P//yVGZmxvD9lzaTV6xDdJV3aOBQnslfDaMWQPCRZUXyq6yNjDwNHK9tPEBqdCgLB7kfhT9MS5xGRmQG/83/b+8JZ1wGFXugcJ1/CtZPoY5AHrlqLiFBAdz87HpqnM1DXSR1FNDAofpWXw5FW3r0b+yutNr2PWmqKq9rYnVOCRfMSidwGK1N5U6ABHDB+AtYW7SWotoi9wmnnAdBYbD5Bf8Vrp/SY8N4+Mo57Cur5/bnN+r8DjVoGjhU3/asAQyM79m/ER4U7tHOf29sPkhLm+GCWcO/mard+RPOR0R4IaeXoBAaDZPPsYblDqMlSLpbNC6B31wwjVU5JfzidV2CXQ2OBg7Vt92rICQa0ud0PVy1m/Gx4/scUWWM4fnPCjguI4ap6dG+LKlXZURmcOaYM3kx58We28l2NuOyYbkESXdXLhzDrUvG8/xn+/nzyryhLo4awTRwqL7lr4ask6wZ0zZjDLsqdnnUv7GhoJKdh2q4YmH/FkIcDm487kbqm+t5dPOj7hON/xJEJMGW4dtc1e4HZ2TzlTkZ3P9uLv9aq4shqoHRwKF6V54Plfu6rE8FcLj+MOXOcqbET+kzi+fWFhARHMj5M/u/J/lQy47P5pLsS3hu53NsK9vmOlFgEEz/6rBcgqQ7EeEPX5nBkuwkfvbqVl5at7/vi5TqRgOH6l37Murd+jd2lO0AYGrC1F4vr6pv5r9bDnLB7AwiQny6UIHP3D7ndhLDEvnB6h9Q3VTtOtHsq6x5Lpv+5d/CDUBwUACPXDWXUyYl8aNXtvCfzwuHukhqhNHAoXqXvxqiMyBhQpfDO8p3ECABHftXuPPKxkIaW9q4YsHIa6ZqFx0czX2n3MehukP8YPUPaGp10QmeOh1GHw/rnoS24T9qKdQRyN+unsuJExK58z+bdQ8P1S8aOJR7ba3W+lTjllhLbHSyo2wHY6PHEu4Id3t5a5vhHx/tZfboWKZnDI99xQdqVvIsfnn8L/mk6BN+8sFPaG1r7Zlo/k3WnI7d7/m/gAMQ6gjk8WvmcdrkZH7x2lbufzdXR1spj2jgUO4VbQJnZY/+DYDt5duZktB7/8Y72w5RUF7PzSeN8035/OyiiRfx/bnf55197/C7tb/r+Ud2yvnWPh3rnhiaAg5AqCOQR6+ayyXzMnnovV387LWtOs9D9WlkNjor/8h9B5Ae/Rsl9SUU1xf32TH+2Af5jI4P54xpw3cl3P66bvp1VDVV8cQXTxAdHM1353Za1T8oGOZeC+/fC6V5kDjBfUbDSFBgAH+8eAYJkSE8sno3+8rq+Mvlc4iLCB7qoqlhSmscyr3ctyBzfpdl1AE2Fm8Ejiw/7sr6veVsLKjkppPGjoiZ4v1x++zbuWTSJfx969958osnu55ccDMEhcBHfxqawg2QiPCjsyZzz1dnsG5PBef95UO2H3QzEEAd8zRwKNeqi6ymquyzepzaWLyR0MDQXmscf1qRS0JEMF+d6/le5COFiPDThT/l7KyzeWDDA7yU02l13MhkmHONtQRJ5cgb6nrJvFG8+I1FNLe2cdFfP+LZT/Zqv4fqQQOHci33Let50tk9Tm0o3sCMpBk4Ah0uL/00v4yP8sr45qnjCQ8+OltDAwMCufukuzkp4yTuXnt311V0T7jdev74oaEp3CDNHh3HG98+kYXjEvjF69u48en1lNS42UZXHZM0cCjXct+C2NGQ3LVWUd9cT055jtud8Ywx3P9OLinRIVy1aIw/SjpkHAEO7j3lXrLjsrlzzZ3kVuRaJ2JHwczL4POnoGJkDnNNjgrlqevm86vzpvJhXimn3beaZz/ZS2ub1j6UBg7lSmONNX9j0tk9huGuP7yeVtPK3JS5Li9dlVPMZ3vLuXXJhGG/WZM3hDvC+fOX/kykI5Lb3ruN0oZS68SpPwUJgJW/HdoCDkJAgHD94rEsv/0kpmfE8IvXt3HBwx/yUV6pNl8d4zRwqJ52/g9anDD94h6nPjrwEWFBYcxJmdPjXGNLK795YzvjkyK4bP7InfDXXykRKTx02kNUNlbynVXfwdnitDZ5Ov5W+OLfcGDDUBdxUCYkR/KvmxbylytmU1bbxJVPrOVrj37Cqp3FtGkN5JikgUP19MV/IGa0tXFTNx8d/Ij5qfMJCQzpce6JD/awt6yeu86fRnDQsfXRmpYwjd+f+Hu2lGzhlx/9kjbTBovvgPBEWH6nNZlyBBMRzp2RzqofnMpvL5jGgcoGrn9qHSf/v1X8ZeUu9pTWDXURlR8dW/+7Vd/qymD3Sjju4h7NVPtr9rOveh+L0xf3uGxfWR1/WZnHmdNSOGlikr9KO6ycNuY07phzB2/ufZOHNjxk7dVx1u/hwPoRNSmwN6GOQK4+Pos1dy7hL1fMZnR8OPe+k8uSe1dz2n2ruft/23lraxGHqpxDXVTlQ0fnkBc1cFteANMKx32tx6kV+1YAcFLmSV2Ot7YZvvfSZoIChbvOn+aXYg5XN0y/gQO1B3hy65OkR6ZzyXFfs4bmvvcbyD7H6jg/CgQHBXDujHTOnZFOYUU97+0oZsWOwzz98T4e/2APAMlRIUxIjmRcUgTjEq3n8UmRZMSGEXCUze051mjgUEcYA+v/DqMWQkrPAPDmnjc5LvE4RkV1/eP3t/d38/m+Ch64dBZpMWH+Ku2w1D7Ho7i+mLvX3k1yeDKnnvsn+Ovx8MrNcO0bXfY1ORpkxoVz7QlZXHtCFo0trWw/WM3Ggkq2Hqwiv6SOZZsOUu1sKxNAAwAADQ1JREFU6UgfEhTA2MQIxidHMj4xguzUaOaOiSM1JnQIX4Xqj6PrE6wGZ88aKMuDk+/scWpv1V52lO/gznldz328u5T73snlnONSuWDWyNtvwxeCAoK45+R7uOHtG/jh+z/kiTOeYMa5f4JXb4Y1f4Av/Xyoi+gzIUGBzB4dx+zRcR3HjDGU1TWxu7iW/NI68ktq2V1Sx9YDVbz5RRHt/esZsWEsGBvPaVOSOXlSEtGhrucJqaGngUMd8eED1k52Uy/sceqVvFcIkADOzDqz49j+8npu/dcGxiVG8MeLZ/S5heyxJNwRzl9O+wtXL7+aW1bcwhNnPMHUWVdZ61hlLoBJZwx1Ef1GREiMDCExMoSF4xK6nHM2t5JzqIb1+yr4fF85q3OKeXXjARyBwvHjE7l4TgZnTks9JoZ2jyRyLIzHnjdvnlm/fv1QF2N42/8ZPLkUlv4WFt/e5VRDSwOn//t0FqYt5P5T7weguNrJZY99SmltI8tuO5GsxIihKPWwd7D2INe/dT21zbX8/UsPk/3qbVCWD9cvh3T3a30dq1rbDBsKKlix/TD/3VLEgcoGokKCOHdmOlcvGjOi9qw/GojI58aYed2P66gqZfVtrPg1hCfAvBt6nF6Wt4zqpmqumHwFACU1jVzxxFoOVTv5+3XzNWj0Ij0ynSfOfIJwRzg3rfo2O8/5A4TFwXOXQPmeoS7esBMYIMzPiucn50zhgx8u4bmvL2TptBRe23iAcx76gEv/9glvbS3Spd+HmNY4lDXq59VvwLkPwLzru5yqb67n3FfPJSMyg2fOfobcw7Xc+PQ6Smsbeer6BSzq1vSgXCuoLuCGt2+gtrmWB2d/n4XL7oSgMLh2GSROHOriDXtV9c28tH4/T3+yl8KKBjJiw7jm+DFcOn8UseG6/LuvuKtxaOA41lXuh7+dBPHj4cZ3IaBrJfSBzx/gya1P8vRZT7P3gLVTXERIEI9fM4+Zo2KHqNAj06G6Q3xzxTfZW72X30y7mfNW3GvNlbn0nzB60VAXb0RobTOs2HGYf3y0h0/zywkJCuA8uxlLP4/ep4FDA0dPzip4+nwoz4ebV0PC+C6n1x1ax03v3MRpmV+mev9XWLHjMHPHxPHwFXN06OQAVTdVc8eqO/j/7Z17bFzFFYe/s+td24mdOM7D2HkQh4S3CUTCBEIRqkgxphVFRSVUlBRQERXQVlUloEgtlWhLo7YSVVGBpkgojUiRWkqoUFsIpUhENKF5OSYkOG0ejpc4D2zHseO1d0//uGNns97dZI039zo+nzWeufO4/vncsc/M3bl3Nn6ykTtmL+XRpnWUdB6Am38CV39zmOM2srMj1sWq9/fyl80H6IknqJs5mbsXz6GxrppyW5E1KpjjMMdxKt3tsOZr0LYZ7lw9bN+NHUd2cO/f7odEGYd3PUhEJvC9pRdy3/Xn3sZMZ5uB5ADPbnmWlU0rmVs+hx8cV67d/R7MuRYafwHnXe63xDHFsRP9vLr5AH94fy+7DnYTLQpx44XT+eLCGj5/8QzKim3x6EjxxXGISAPwDBAGVqrq02nl4sobgR7gG6q6KVdbEakE/gjMBfYAX1XVT3PpMMeRQjIB216Bt37kzTi+shIu+ZJXlFSaYx08v2kN7x5ZSWKglP7Wb3HnlQt55PPzmTHJZhmjyfq29Tz1/lPsP7af68rmsnxvM4u7jhC66Fa4+j6ovfGce1iwkKgqm/Z18PrWNt5oitF+rI+ikLBwdgVLLpjK1bWVXFYzmUrbEveMOeuOQ0TCwC5gKdAKbATuUtUPU+o0Ao/gOY5rgGdU9ZpcbUVkBXBUVZ8WkceAKar6aC4t49ZxqELvp9DdzkCsicS+DRTtfJ1wd4zjU+vYsPApPtJqWo58woeHWtjfsx0t20QoepTigfl8fd4T3FN/he09XUD6En2s3rGaVR+u4nDvYaaGSvhcdxd1Pd1cKCVUVS9i2uzriEy/BKacD2VVUDzJHMppSCaVjXuO8q9dh3hv9xGaWjuGHjSsnlzCgqpyZk8pZdaUCdRUlFA5MUpFaZSKCREmlUYojYSJhGXcP5vkh+O4FnhSVW92x48DqOrPUuo8D7yjqi+7453AjXiziYxtB+uoakxEql37i3JpGanj+PW6j1m7tW1o7wEd+jYUnVqG97/aO9aT6TQTZ2qj7kiHnR9WJFZQq62EUIQkghJCCbn0b6ZG2FwSRoGkeG0VUNGhdBIhIRCniD6K6CcMKBJO3dlNmDuxjnsuu4s7Lr513P/RnE3iiThv73ubdfvWsb5tPV3xU/f7LkkmiapSrEpUvWk4EkIARBBSr9XJo9R46UCYh+Ip9/6XfBuuurtAv1Gw6OztZ/uBTprbOmlu62L3oW5aP+2lo6c/Z7toUYhoODQUi3i2HPzbEPFCSGQo37smp9bzk5/eXkd9beWI2mZzHIUctswEUjddbsWbVZyuzszTtK1S1RiAcx4zMv1wEXkAeABgzpyR7Q0xo7yYi6rK3QlPRkOdZuhnkXacUj5UJlnqniwfSg/1NS9RFJvHsfhEVDyXgYRIEkIlBCJEo4eokl7vHClfShGJUDHJUAmJ6GSSxRVEiiJEw0WUFIWZWByhqqyCWZPOo6ashrppdZRHy0dkK+OzEQ1HaahtoKG2AVUldjxGS0cL7T3tHOrcR093jL7eI8Tjx+kb6CWZiKPJATfSUFD1Bh+DMSmDDxQUppeUQ1HKm4snjJ+l1JNLIyyZP40l86edkn/sRD+xzhN09PTT0ROno7efrt5+TvQniA8k6UskiQ94oT+RdIO8lMGeO06qppTpKYNMv5lYPPpP3RfScWRytemmzFbnTNrmRFVfAF4Ab8aRT9tBltXPYVl9EDYkei5naea9+IyxiohQU1ZDTZm9+6vQlJdEbAXWCCjk2r9WIPU1qrOAtjOsk6vtQXeLChe3j6JmwzAM4zQU0nFsBBaISK2IRIFlwNq0OmuBe8RjMdDpbkPlarsWWO7Sy4HXCvg7GIZhGGkU7FaVqg6IyMPA3/E+y3tRVZtF5EFX/hzwBt6Kqha85bj35mrrTv008IqI3A/sA4bvOGQYhmEUDHsA0DAMw8iIvR3XMAzDGBXMcRiGYRh5YY7DMAzDyAtzHIZhGEZejIsPx0XkELA3JWsacNgnOfliWkefsaITTGuhGCta/dZ5vqpOT88cF44jHRH5INNKgSBiWkefsaITTGuhGCtag6rTblUZhmEYeWGOwzAMw8iL8eo4XvBbQB6Y1tFnrOgE01ooxorWQOocl59xGIZhGCNnvM44DMMwjBFijsMwDMPIi3HlOETkSRE5ICJbXGhMKXtcRFpEZKeI3OynTqenwWlpcXurBwoR2SMiTc6OH7i8ShF5U0Q+dvEUn7S9KCLtIrI9JS+rNj+vfRatgeunIjJbRP4pIjtEpFlEvuPyA2fXHFoDZVcRKRGRDSKy1en8scsPnE2HoarjJgBPAt/PkH8psBUoBmqB3UDYR51hp2EeEHXaLvXbfmka9wDT0vJWAI+59GPAz33SdgOwCNh+Om1+X/ssWgPXT4FqYJFLlwO7nJ7A2TWH1kDZFW+n0zKXjgD/BhYH0abpYVzNOHJwG7BGVftU9X94+4PU+6inHmhR1f+qahxY4zQGnduAl1z6JeDLfohQ1XeBo2nZ2bT5eu2zaM2Gb1pVNaaqm1z6GLADmEkA7ZpDazZ80aoe3e4w4oISQJumMx4dx8Miss3dIhicAs4E9qfUaSV3Rys0QdOTCQX+ISL/EZEHXF6Vejs44uIZvqkbTjZtQbV1YPupiMwFrsIbIQfarmlaIWB2FZGwiGzB2wL7TVUNvE3hHHQcIvKWiGzPEG4DfgtcAFwJxIBfDjbLcCo/1ykHTU8mlqjqIuAW4CERucFvQSMkiLYObD8VkTLgT8B3VbUrV9UMeX5rDZxdVTWhqlcCs4B6Ebk8R3XfbTpIwbaO9QtVvelM6onI74C/usNWYHZK8SygbZSl5UPQ9AxDVdtc3C4ir+JNmQ+KSLWqxkSkGm8UFRSyaQucrVX14GA6SP1URCJ4/4hXq+qfXXYg7ZpJa1DtCqCqHSLyDtBAQG2ayjk348iFuwiD3A4MrmRZCywTkWIRqQUWABvOtr4UNgILRKRWRKLAMqcxEIjIRBEpH0wDX8Cz5Vpguau2HHjNH4UZyaYtaNc+kP1URAT4PbBDVX+VUhQ4u2bTGjS7ish0Ealw6VLgJuAjAmjTYfjxibxfAVgFNAHb3EWoTil7Am+Vwk7glgBobcRbDbIbeMJvPWna5uGt7tgKNA/qA6YC64CPXVzpk76X8W5F9OON0u7Ppc3Pa59Fa+D6KXA93m2RbcAWFxqDaNccWgNlV+AKYLPTsx34ocsPnE3Tg71yxDAMw8iLcXWryjAMw/jsmOMwDMMw8sIch2EYhpEX5jgMwzCMvDDHYRiGYeSFOQ7DMAwjL8xxGIZhGHnxf5E7RJNn/i9NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.groupby(\"origin\")[\"horsepower\"].plot(kind=\"kde\", title=\"Distribution of horsepower by origin\", legend=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pivot tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pivot tables work similar to excel pivot tables and are helpful for more complex conditional filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 367,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130</td>\n",
       "      <td>3504</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15.0</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3693</td>\n",
       "      <td>11.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18.0</td>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3436</td>\n",
       "      <td>11.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth satellite</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150</td>\n",
       "      <td>3433</td>\n",
       "      <td>12.0</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>amc rebel sst</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17.0</td>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140</td>\n",
       "      <td>3449</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>ford torino</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "0  18.0          8         307.0         130    3504          12.0    70   \n",
       "1  15.0          8         350.0         165    3693          11.5    70   \n",
       "2  18.0          8         318.0         150    3436          11.0    70   \n",
       "3  16.0          8         304.0         150    3433          12.0    70   \n",
       "4  17.0          8         302.0         140    3449          10.5    70   \n",
       "\n",
       "   origin                       name  \n",
       "0       1  chevrolet chevelle malibu  \n",
       "1       1          buick skylark 320  \n",
       "2       1         plymouth satellite  \n",
       "3       1              amc rebel sst  \n",
       "4       1                ford torino  "
      ]
     },
     "execution_count": 367,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(fpath)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we want to calculate average weight conditioned by number of cylinders and origin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>origin</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cylinders</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2398.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2427.391304</td>\n",
       "      <td>2338.295082</td>\n",
       "      <td>2153.492754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3103.333333</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3218.547945</td>\n",
       "      <td>3382.500000</td>\n",
       "      <td>2882.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4114.718447</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "origin               1            2            3\n",
       "cylinders                                       \n",
       "3                  NaN          NaN  2398.500000\n",
       "4          2427.391304  2338.295082  2153.492754\n",
       "5                  NaN  3103.333333          NaN\n",
       "6          3218.547945  3382.500000  2882.000000\n",
       "8          4114.718447          NaN          NaN"
      ]
     },
     "execution_count": 372,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot_table(values=\"weight\",index=\"cylinders\", columns=\"origin\",aggfunc=np.mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we want to apply multiple filter. Here `query` can be handy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>19.2</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>105</td>\n",
       "      <td>3535</td>\n",
       "      <td>19.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac phoenix lj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>20.5</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>95</td>\n",
       "      <td>3155</td>\n",
       "      <td>18.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>20.2</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85</td>\n",
       "      <td>2965</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>ford fairmont (auto)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>20.5</td>\n",
       "      <td>6</td>\n",
       "      <td>225.0</td>\n",
       "      <td>100</td>\n",
       "      <td>3430</td>\n",
       "      <td>17.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth volare</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>19.4</td>\n",
       "      <td>6</td>\n",
       "      <td>232.0</td>\n",
       "      <td>90</td>\n",
       "      <td>3210</td>\n",
       "      <td>17.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>20.6</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>105</td>\n",
       "      <td>3380</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>buick century special</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>257</th>\n",
       "      <td>20.8</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85</td>\n",
       "      <td>3070</td>\n",
       "      <td>16.7</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>mercury zephyr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>18.6</td>\n",
       "      <td>6</td>\n",
       "      <td>225.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3620</td>\n",
       "      <td>18.7</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge aspen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>18.1</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>120</td>\n",
       "      <td>3410</td>\n",
       "      <td>15.1</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord d/l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>17.7</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3445</td>\n",
       "      <td>13.4</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>buick regal sport coupe (turbo)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>17.0</td>\n",
       "      <td>6</td>\n",
       "      <td>163.0</td>\n",
       "      <td>125</td>\n",
       "      <td>3140</td>\n",
       "      <td>13.6</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>volvo 264gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>16.2</td>\n",
       "      <td>6</td>\n",
       "      <td>163.0</td>\n",
       "      <td>133</td>\n",
       "      <td>3410</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 604sl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "250  19.2          6         231.0         105    3535          19.2    78   \n",
       "251  20.5          6         200.0          95    3155          18.2    78   \n",
       "252  20.2          6         200.0          85    2965          15.8    78   \n",
       "254  20.5          6         225.0         100    3430          17.2    78   \n",
       "255  19.4          6         232.0          90    3210          17.2    78   \n",
       "256  20.6          6         231.0         105    3380          15.8    78   \n",
       "257  20.8          6         200.0          85    3070          16.7    78   \n",
       "258  18.6          6         225.0         110    3620          18.7    78   \n",
       "259  18.1          6         258.0         120    3410          15.1    78   \n",
       "261  17.7          6         231.0         165    3445          13.4    78   \n",
       "273  17.0          6         163.0         125    3140          13.6    78   \n",
       "275  16.2          6         163.0         133    3410          15.8    78   \n",
       "\n",
       "     origin                             name  \n",
       "250       1               pontiac phoenix lj  \n",
       "251       1                 chevrolet malibu  \n",
       "252       1             ford fairmont (auto)  \n",
       "254       1                  plymouth volare  \n",
       "255       1                      amc concord  \n",
       "256       1            buick century special  \n",
       "257       1                   mercury zephyr  \n",
       "258       1                      dodge aspen  \n",
       "259       1                  amc concord d/l  \n",
       "261       1  buick regal sport coupe (turbo)  \n",
       "273       2                      volvo 264gl  \n",
       "275       2                    peugeot 604sl  "
      ]
     },
     "execution_count": 376,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Standard option\n",
    "df.loc[(df[\"cylinders\"]==6) & (df[\"year\"]==78)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>19.2</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>105</td>\n",
       "      <td>3535</td>\n",
       "      <td>19.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>pontiac phoenix lj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>20.5</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>95</td>\n",
       "      <td>3155</td>\n",
       "      <td>18.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>20.2</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85</td>\n",
       "      <td>2965</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>ford fairmont (auto)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>20.5</td>\n",
       "      <td>6</td>\n",
       "      <td>225.0</td>\n",
       "      <td>100</td>\n",
       "      <td>3430</td>\n",
       "      <td>17.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>plymouth volare</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>19.4</td>\n",
       "      <td>6</td>\n",
       "      <td>232.0</td>\n",
       "      <td>90</td>\n",
       "      <td>3210</td>\n",
       "      <td>17.2</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>20.6</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>105</td>\n",
       "      <td>3380</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>buick century special</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>257</th>\n",
       "      <td>20.8</td>\n",
       "      <td>6</td>\n",
       "      <td>200.0</td>\n",
       "      <td>85</td>\n",
       "      <td>3070</td>\n",
       "      <td>16.7</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>mercury zephyr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>18.6</td>\n",
       "      <td>6</td>\n",
       "      <td>225.0</td>\n",
       "      <td>110</td>\n",
       "      <td>3620</td>\n",
       "      <td>18.7</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>dodge aspen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>18.1</td>\n",
       "      <td>6</td>\n",
       "      <td>258.0</td>\n",
       "      <td>120</td>\n",
       "      <td>3410</td>\n",
       "      <td>15.1</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>amc concord d/l</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>17.7</td>\n",
       "      <td>6</td>\n",
       "      <td>231.0</td>\n",
       "      <td>165</td>\n",
       "      <td>3445</td>\n",
       "      <td>13.4</td>\n",
       "      <td>78</td>\n",
       "      <td>1</td>\n",
       "      <td>buick regal sport coupe (turbo)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>17.0</td>\n",
       "      <td>6</td>\n",
       "      <td>163.0</td>\n",
       "      <td>125</td>\n",
       "      <td>3140</td>\n",
       "      <td>13.6</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>volvo 264gl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>16.2</td>\n",
       "      <td>6</td>\n",
       "      <td>163.0</td>\n",
       "      <td>133</td>\n",
       "      <td>3410</td>\n",
       "      <td>15.8</td>\n",
       "      <td>78</td>\n",
       "      <td>2</td>\n",
       "      <td>peugeot 604sl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      mpg  cylinders  displacement  horsepower  weight  acceleration  year  \\\n",
       "250  19.2          6         231.0         105    3535          19.2    78   \n",
       "251  20.5          6         200.0          95    3155          18.2    78   \n",
       "252  20.2          6         200.0          85    2965          15.8    78   \n",
       "254  20.5          6         225.0         100    3430          17.2    78   \n",
       "255  19.4          6         232.0          90    3210          17.2    78   \n",
       "256  20.6          6         231.0         105    3380          15.8    78   \n",
       "257  20.8          6         200.0          85    3070          16.7    78   \n",
       "258  18.6          6         225.0         110    3620          18.7    78   \n",
       "259  18.1          6         258.0         120    3410          15.1    78   \n",
       "261  17.7          6         231.0         165    3445          13.4    78   \n",
       "273  17.0          6         163.0         125    3140          13.6    78   \n",
       "275  16.2          6         163.0         133    3410          15.8    78   \n",
       "\n",
       "     origin                             name  \n",
       "250       1               pontiac phoenix lj  \n",
       "251       1                 chevrolet malibu  \n",
       "252       1             ford fairmont (auto)  \n",
       "254       1                  plymouth volare  \n",
       "255       1                      amc concord  \n",
       "256       1            buick century special  \n",
       "257       1                   mercury zephyr  \n",
       "258       1                      dodge aspen  \n",
       "259       1                  amc concord d/l  \n",
       "261       1  buick regal sport coupe (turbo)  \n",
       "273       2                      volvo 264gl  \n",
       "275       2                    peugeot 604sl  "
      ]
     },
     "execution_count": 377,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Option using query function\n",
    "df.query(\"cylinders==6 & year == 78\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tidy data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please refer to the great article by Hadley Wickham [Tidy Data](https://vita.had.co.nz/papers/tidy-data.pdf) for a detailed discussion on tidy data.\n",
    "\n",
    "According to Wickham, **tidy data** is a standard way of mapping the meaning of a dataset to its structure. A dataset is messy or tidy depending on how rows, columns and tables are matched up with observations, variables and types. \n",
    "\n",
    "In tidy data:\n",
    "\n",
    "1. each variable forms a column\n",
    "\n",
    "2. each observation forms a unit\n",
    "\n",
    "3. each type of observational unit forms a table "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is this important: because when a dataframe is not tidy it is much harder to analyse and explore the data. \n",
    "\n",
    "**Example** (from Wickham)\n",
    "\n",
    "There are different ways how to structure data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Treatment_A</th>\n",
       "      <th>Treatment_B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>John</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mary</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Name  Treatment_A  Treatment_B\n",
       "0  John          NaN            2\n",
       "1  Jane         16.0            1\n",
       "2  Mary          3.0            1"
      ]
     },
     "execution_count": 441,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\"Name\":(\"John\",\"Jane\",\"Mary\"),\"Treatment_A\":(np.nan, 16,3), \"Treatment_B\": (2,1,1)}\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Treatment_A</th>\n",
       "      <th>Treatment_B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>John</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jane</th>\n",
       "      <td>16.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mary</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Treatment_A  Treatment_B\n",
       "                              \n",
       "John          NaN            2\n",
       "Jane         16.0            1\n",
       "Mary          3.0            1"
      ]
     },
     "execution_count": 436,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df.set_index(\"Name\")\n",
    "df1.index.name = \" \"\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 437,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>John</th>\n",
       "      <th>Jane</th>\n",
       "      <th>Mary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Treatment_A</th>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Treatment_B</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             John  Jane  Mary\n",
       "Treatment_A   NaN  16.0   3.0\n",
       "Treatment_B   2.0   1.0   1.0"
      ]
     },
     "execution_count": 437,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df1.T\n",
    "df2.columns.name = \"\"\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>treatment</th>\n",
       "      <th>treatment_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>John</td>\n",
       "      <td>Treatment_A</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Treatment_A</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mary</td>\n",
       "      <td>Treatment_A</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>John</td>\n",
       "      <td>Treatment_B</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Treatment_B</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Mary</td>\n",
       "      <td>Treatment_B</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Name    treatment  treatment_result\n",
       "0  John  Treatment_A               NaN\n",
       "1  Jane  Treatment_A              16.0\n",
       "2  Mary  Treatment_A               3.0\n",
       "3  John  Treatment_B               2.0\n",
       "4  Jane  Treatment_B               1.0\n",
       "5  Mary  Treatment_B               1.0"
      ]
     },
     "execution_count": 448,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.melt(df, id_vars=\"Name\", var_name=\"treatment\", value_name=\"treatment_result\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing quantitative or statistical analysis you would typically want to conduct analysis on the two variables \"treatment\" and \"treatment_result\" which is why the third option is much more helpful when conducting analysis and is considered to be **tidy**:\n",
    "\n",
    "- each variable forms a column: we have three variables name, treatment and treatment result\n",
    "- each observation forms a row: we have 6 observations for 3 persons two treatments\n",
    "- (number 3: not relevant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following examples are taken from [Daniel Chen](https://github.com/chendaniely/pydatadc_2018-tidy/tree/master/data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example using a bigger data set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>religion</th>\n",
       "      <th>&lt;$10k</th>\n",
       "      <th>$10-20k</th>\n",
       "      <th>$20-30k</th>\n",
       "      <th>$30-40k</th>\n",
       "      <th>$40-50k</th>\n",
       "      <th>$50-75k</th>\n",
       "      <th>$75-100k</th>\n",
       "      <th>$100-150k</th>\n",
       "      <th>&gt;150k</th>\n",
       "      <th>Don't know/refused</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Agnostic</td>\n",
       "      <td>27</td>\n",
       "      <td>34</td>\n",
       "      <td>60</td>\n",
       "      <td>81</td>\n",
       "      <td>76</td>\n",
       "      <td>137</td>\n",
       "      <td>122</td>\n",
       "      <td>109</td>\n",
       "      <td>84</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Atheist</td>\n",
       "      <td>12</td>\n",
       "      <td>27</td>\n",
       "      <td>37</td>\n",
       "      <td>52</td>\n",
       "      <td>35</td>\n",
       "      <td>70</td>\n",
       "      <td>73</td>\n",
       "      <td>59</td>\n",
       "      <td>74</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Buddhist</td>\n",
       "      <td>27</td>\n",
       "      <td>21</td>\n",
       "      <td>30</td>\n",
       "      <td>34</td>\n",
       "      <td>33</td>\n",
       "      <td>58</td>\n",
       "      <td>62</td>\n",
       "      <td>39</td>\n",
       "      <td>53</td>\n",
       "      <td>54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Catholic</td>\n",
       "      <td>418</td>\n",
       "      <td>617</td>\n",
       "      <td>732</td>\n",
       "      <td>670</td>\n",
       "      <td>638</td>\n",
       "      <td>1116</td>\n",
       "      <td>949</td>\n",
       "      <td>792</td>\n",
       "      <td>633</td>\n",
       "      <td>1489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Don’t know/refused</td>\n",
       "      <td>15</td>\n",
       "      <td>14</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>10</td>\n",
       "      <td>35</td>\n",
       "      <td>21</td>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Evangelical Prot</td>\n",
       "      <td>575</td>\n",
       "      <td>869</td>\n",
       "      <td>1064</td>\n",
       "      <td>982</td>\n",
       "      <td>881</td>\n",
       "      <td>1486</td>\n",
       "      <td>949</td>\n",
       "      <td>723</td>\n",
       "      <td>414</td>\n",
       "      <td>1529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Hindu</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>34</td>\n",
       "      <td>47</td>\n",
       "      <td>48</td>\n",
       "      <td>54</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Historically Black Prot</td>\n",
       "      <td>228</td>\n",
       "      <td>244</td>\n",
       "      <td>236</td>\n",
       "      <td>238</td>\n",
       "      <td>197</td>\n",
       "      <td>223</td>\n",
       "      <td>131</td>\n",
       "      <td>81</td>\n",
       "      <td>78</td>\n",
       "      <td>339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Jehovah's Witness</td>\n",
       "      <td>20</td>\n",
       "      <td>27</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>21</td>\n",
       "      <td>30</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Jewish</td>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>30</td>\n",
       "      <td>95</td>\n",
       "      <td>69</td>\n",
       "      <td>87</td>\n",
       "      <td>151</td>\n",
       "      <td>162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Mainline Prot</td>\n",
       "      <td>289</td>\n",
       "      <td>495</td>\n",
       "      <td>619</td>\n",
       "      <td>655</td>\n",
       "      <td>651</td>\n",
       "      <td>1107</td>\n",
       "      <td>939</td>\n",
       "      <td>753</td>\n",
       "      <td>634</td>\n",
       "      <td>1328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Mormon</td>\n",
       "      <td>29</td>\n",
       "      <td>40</td>\n",
       "      <td>48</td>\n",
       "      <td>51</td>\n",
       "      <td>56</td>\n",
       "      <td>112</td>\n",
       "      <td>85</td>\n",
       "      <td>49</td>\n",
       "      <td>42</td>\n",
       "      <td>69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Muslim</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>23</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Orthodox</td>\n",
       "      <td>13</td>\n",
       "      <td>17</td>\n",
       "      <td>23</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>47</td>\n",
       "      <td>38</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Other Christian</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>18</td>\n",
       "      <td>14</td>\n",
       "      <td>12</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Other Faiths</td>\n",
       "      <td>20</td>\n",
       "      <td>33</td>\n",
       "      <td>40</td>\n",
       "      <td>46</td>\n",
       "      <td>49</td>\n",
       "      <td>63</td>\n",
       "      <td>46</td>\n",
       "      <td>40</td>\n",
       "      <td>41</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Other World Religions</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>217</td>\n",
       "      <td>299</td>\n",
       "      <td>374</td>\n",
       "      <td>365</td>\n",
       "      <td>341</td>\n",
       "      <td>528</td>\n",
       "      <td>407</td>\n",
       "      <td>321</td>\n",
       "      <td>258</td>\n",
       "      <td>597</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   religion  <$10k  $10-20k  $20-30k  $30-40k  $40-50k  \\\n",
       "0                  Agnostic     27       34       60       81       76   \n",
       "1                   Atheist     12       27       37       52       35   \n",
       "2                  Buddhist     27       21       30       34       33   \n",
       "3                  Catholic    418      617      732      670      638   \n",
       "4        Don’t know/refused     15       14       15       11       10   \n",
       "5          Evangelical Prot    575      869     1064      982      881   \n",
       "6                     Hindu      1        9        7        9       11   \n",
       "7   Historically Black Prot    228      244      236      238      197   \n",
       "8         Jehovah's Witness     20       27       24       24       21   \n",
       "9                    Jewish     19       19       25       25       30   \n",
       "10            Mainline Prot    289      495      619      655      651   \n",
       "11                   Mormon     29       40       48       51       56   \n",
       "12                   Muslim      6        7        9       10        9   \n",
       "13                 Orthodox     13       17       23       32       32   \n",
       "14          Other Christian      9        7       11       13       13   \n",
       "15             Other Faiths     20       33       40       46       49   \n",
       "16    Other World Religions      5        2        3        4        2   \n",
       "17             Unaffiliated    217      299      374      365      341   \n",
       "\n",
       "    $50-75k  $75-100k  $100-150k  >150k  Don't know/refused  \n",
       "0       137       122        109     84                  96  \n",
       "1        70        73         59     74                  76  \n",
       "2        58        62         39     53                  54  \n",
       "3      1116       949        792    633                1489  \n",
       "4        35        21         17     18                 116  \n",
       "5      1486       949        723    414                1529  \n",
       "6        34        47         48     54                  37  \n",
       "7       223       131         81     78                 339  \n",
       "8        30        15         11      6                  37  \n",
       "9        95        69         87    151                 162  \n",
       "10     1107       939        753    634                1328  \n",
       "11      112        85         49     42                  69  \n",
       "12       23        16          8      6                  22  \n",
       "13       47        38         42     46                  73  \n",
       "14       14        18         14     12                  18  \n",
       "15       63        46         40     41                  71  \n",
       "16        7         3          4      4                   8  \n",
       "17      528       407        321    258                 597  "
      ]
     },
     "execution_count": 462,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"./data/pew.csv\", sep=\";\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table may be useful for presentation purposes and may be data efficient. But it is not tidy.\n",
    "\n",
    "Why is it not tidy?\n",
    "\n",
    "A good proxy to find out if the data is stored in a tidy format is to think about a model you want to build (e.g. a regression model) and aks yourself what possible variables for your models would be. \n",
    "\n",
    "In the above case it is unlikely that we would want to treat \"<$10k\" as a variable. Instead we would likely want to use \"income type\" as a variable. So the data set actually consists of three variables:\n",
    "\n",
    "- religion\n",
    "- income\n",
    "- count (number of people)\n",
    "\n",
    "With the data structure above we are not able to analyze the data effiently. \n",
    "\n",
    "For example: the following analysis would already be quite tedious:\n",
    "\n",
    "- calculate sum of counts per religion\n",
    "\n",
    "- run a model on different variables\n",
    "\n",
    "The format the above table is also called **wide format**.\n",
    "\n",
    "In order to make it tidy we need to put it into a **long format** using `pd.melt`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 499,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>religion</th>\n",
       "      <th>income</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Agnostic</td>\n",
       "      <td>&lt;$10k</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Atheist</td>\n",
       "      <td>&lt;$10k</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Buddhist</td>\n",
       "      <td>&lt;$10k</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Catholic</td>\n",
       "      <td>&lt;$10k</td>\n",
       "      <td>418</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Don’t know/refused</td>\n",
       "      <td>&lt;$10k</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>175</th>\n",
       "      <td>Orthodox</td>\n",
       "      <td>Don't know/refused</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>Other Christian</td>\n",
       "      <td>Don't know/refused</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>Other Faiths</td>\n",
       "      <td>Don't know/refused</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>Other World Religions</td>\n",
       "      <td>Don't know/refused</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179</th>\n",
       "      <td>Unaffiliated</td>\n",
       "      <td>Don't know/refused</td>\n",
       "      <td>597</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>180 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                  religion              income  count\n",
       "0                 Agnostic               <$10k     27\n",
       "1                  Atheist               <$10k     12\n",
       "2                 Buddhist               <$10k     27\n",
       "3                 Catholic               <$10k    418\n",
       "4       Don’t know/refused               <$10k     15\n",
       "..                     ...                 ...    ...\n",
       "175               Orthodox  Don't know/refused     73\n",
       "176        Other Christian  Don't know/refused     18\n",
       "177           Other Faiths  Don't know/refused     71\n",
       "178  Other World Religions  Don't know/refused      8\n",
       "179           Unaffiliated  Don't know/refused    597\n",
       "\n",
       "[180 rows x 3 columns]"
      ]
     },
     "execution_count": 499,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tidy = df.melt(id_vars=\"religion\", var_name=\"income\", value_name=\"count\")\n",
    "df_tidy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now already answer the first question really easy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 501,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "religion\n",
       "Agnostic                    826\n",
       "Atheist                     515\n",
       "Buddhist                    411\n",
       "Catholic                   8054\n",
       "Don’t know/refused          272\n",
       "Evangelical Prot           9472\n",
       "Hindu                       257\n",
       "Historically Black Prot    1995\n",
       "Jehovah's Witness           215\n",
       "Jewish                      682\n",
       "Mainline Prot              7470\n",
       "Mormon                      581\n",
       "Muslim                      116\n",
       "Orthodox                    363\n",
       "Other Christian             129\n",
       "Other Faiths                449\n",
       "Other World Religions        42\n",
       "Unaffiliated               3707\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 501,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_tidy.groupby(\"religion\")[\"count\"].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although this would also be achievable using the wide format it is (i) much verbose and (ii) also would need change of code if a new income category was introduced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**More complex example**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at case and death number from an ebola pandemic in 2014/2015. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 704,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"https://raw.githubusercontent.com/chendaniely/pydatadc_2018-tidy/master/data/country_timeseries.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 705,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Day</th>\n",
       "      <th>Cases_Guinea</th>\n",
       "      <th>Cases_Liberia</th>\n",
       "      <th>Cases_SierraLeone</th>\n",
       "      <th>Cases_Nigeria</th>\n",
       "      <th>Cases_Senegal</th>\n",
       "      <th>Cases_UnitedStates</th>\n",
       "      <th>Cases_Spain</th>\n",
       "      <th>Cases_Mali</th>\n",
       "      <th>Deaths_Guinea</th>\n",
       "      <th>Deaths_Liberia</th>\n",
       "      <th>Deaths_SierraLeone</th>\n",
       "      <th>Deaths_Nigeria</th>\n",
       "      <th>Deaths_Senegal</th>\n",
       "      <th>Deaths_UnitedStates</th>\n",
       "      <th>Deaths_Spain</th>\n",
       "      <th>Deaths_Mali</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/5/2015</td>\n",
       "      <td>289</td>\n",
       "      <td>2776.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10030.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1786.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2977.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/4/2015</td>\n",
       "      <td>288</td>\n",
       "      <td>2775.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9780.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1781.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2943.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2015</td>\n",
       "      <td>287</td>\n",
       "      <td>2769.0</td>\n",
       "      <td>8166.0</td>\n",
       "      <td>9722.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1767.0</td>\n",
       "      <td>3496.0</td>\n",
       "      <td>2915.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/2/2015</td>\n",
       "      <td>286</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8157.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3496.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12/31/2014</td>\n",
       "      <td>284</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>8115.0</td>\n",
       "      <td>9633.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1739.0</td>\n",
       "      <td>3471.0</td>\n",
       "      <td>2827.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  Day  Cases_Guinea  Cases_Liberia  Cases_SierraLeone  \\\n",
       "0    1/5/2015  289        2776.0            NaN            10030.0   \n",
       "1    1/4/2015  288        2775.0            NaN             9780.0   \n",
       "2    1/3/2015  287        2769.0         8166.0             9722.0   \n",
       "3    1/2/2015  286           NaN         8157.0                NaN   \n",
       "4  12/31/2014  284        2730.0         8115.0             9633.0   \n",
       "\n",
       "   Cases_Nigeria  Cases_Senegal  Cases_UnitedStates  Cases_Spain  Cases_Mali  \\\n",
       "0            NaN            NaN                 NaN          NaN         NaN   \n",
       "1            NaN            NaN                 NaN          NaN         NaN   \n",
       "2            NaN            NaN                 NaN          NaN         NaN   \n",
       "3            NaN            NaN                 NaN          NaN         NaN   \n",
       "4            NaN            NaN                 NaN          NaN         NaN   \n",
       "\n",
       "   Deaths_Guinea  Deaths_Liberia  Deaths_SierraLeone  Deaths_Nigeria  \\\n",
       "0         1786.0             NaN              2977.0             NaN   \n",
       "1         1781.0             NaN              2943.0             NaN   \n",
       "2         1767.0          3496.0              2915.0             NaN   \n",
       "3            NaN          3496.0                 NaN             NaN   \n",
       "4         1739.0          3471.0              2827.0             NaN   \n",
       "\n",
       "   Deaths_Senegal  Deaths_UnitedStates  Deaths_Spain  Deaths_Mali  \n",
       "0             NaN                  NaN           NaN          NaN  \n",
       "1             NaN                  NaN           NaN          NaN  \n",
       "2             NaN                  NaN           NaN          NaN  \n",
       "3             NaN                  NaN           NaN          NaN  \n",
       "4             NaN                  NaN           NaN          NaN  "
      ]
     },
     "execution_count": 705,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is not really well structured for analysis given cases and death for each country is organized as columns. However, typically we would ask questions such as:\n",
    "\n",
    "- what is number of death per country\n",
    "- what is average case increase per country \n",
    "- etc.\n",
    "\n",
    "Clearly the data is not tidy as we have the following variables in the data set:\n",
    "\n",
    "- date\n",
    "- day\n",
    "- indicator (case or death)\n",
    "- country"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be very difficult to calculate simple statistics such as aggregate deaths by country."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make the dataframe tidy to be able to do these kind of calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 706,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Day</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/5/2015</td>\n",
       "      <td>289</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2776.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/4/2015</td>\n",
       "      <td>288</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2775.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2015</td>\n",
       "      <td>287</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2769.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/2/2015</td>\n",
       "      <td>286</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12/31/2014</td>\n",
       "      <td>284</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2730.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  Day      variable   value\n",
       "0    1/5/2015  289  Cases_Guinea  2776.0\n",
       "1    1/4/2015  288  Cases_Guinea  2775.0\n",
       "2    1/3/2015  287  Cases_Guinea  2769.0\n",
       "3    1/2/2015  286  Cases_Guinea     NaN\n",
       "4  12/31/2014  284  Cases_Guinea  2730.0"
      ]
     },
     "execution_count": 706,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df.melt(id_vars=[\"Date\",\"Day\"])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are almost there. We only need to look more carefully at column \"variable\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 707,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Day</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "      <th>type</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1/5/2015</td>\n",
       "      <td>289</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2776.0</td>\n",
       "      <td>Cases</td>\n",
       "      <td>Guinea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1/4/2015</td>\n",
       "      <td>288</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2775.0</td>\n",
       "      <td>Cases</td>\n",
       "      <td>Guinea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1/3/2015</td>\n",
       "      <td>287</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2769.0</td>\n",
       "      <td>Cases</td>\n",
       "      <td>Guinea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1/2/2015</td>\n",
       "      <td>286</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cases</td>\n",
       "      <td>Guinea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12/31/2014</td>\n",
       "      <td>284</td>\n",
       "      <td>Cases_Guinea</td>\n",
       "      <td>2730.0</td>\n",
       "      <td>Cases</td>\n",
       "      <td>Guinea</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  Day      variable   value   type country\n",
       "0    1/5/2015  289  Cases_Guinea  2776.0  Cases  Guinea\n",
       "1    1/4/2015  288  Cases_Guinea  2775.0  Cases  Guinea\n",
       "2    1/3/2015  287  Cases_Guinea  2769.0  Cases  Guinea\n",
       "3    1/2/2015  286  Cases_Guinea     NaN  Cases  Guinea\n",
       "4  12/31/2014  284  Cases_Guinea  2730.0  Cases  Guinea"
      ]
     },
     "execution_count": 707,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[[\"type\", \"country\"]] = df[\"variable\"].str.split(\"_\",expand=True) # split column \"variables\" by \"_\" and store it in two columns\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 708,
   "metadata": {},
   "outputs": [],
   "source": [
    "df  = df.drop(labels=\"variable\", axis=1) # drop columns \"variable\" which is not needed anymore\n",
    "df = df.dropna() # drop nans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 709,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country\n",
       "Guinea          136547.0\n",
       "Liberia         283031.0\n",
       "Mali                80.0\n",
       "Nigeria            869.0\n",
       "Senegal             27.0\n",
       "SierraLeone     271533.0\n",
       "Spain               19.0\n",
       "UnitedStates        74.0\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 709,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"country\"])[\"value\"].sum()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}