Projects/step_7/DL fundametals/Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### In this Guided Project, we'll:\n",
    "\n",
    "* explore why image classification is a hard task\n",
    "* observe the limitations of traditional machine learning models for image classification\n",
    "* train, test, and improve a few different deep neural networks for image classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll build models that can classify handwritten digits. Before the year 2000, institutions like the United States Post Office used handwriting recognition software to read addresses, zip codes, and more. One of their approaches, which consists of pre-processing handwritten images then feeding to a neural network model is detailed in this [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.852.5499&rep=rep1&type=pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is image classifcation a hard task?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* First, each image in a training set is high dimensional. **Each pixel in an image is a feature and a separate column.** This means that a 128 x 128 image has 16384 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Second, **images are often downsampled to lower resolutions and transformed to grayscale (no color)**. This is a limitation of compute power unfortunately. The resolution of a 8 megapixel photo has 3264 by 2448 pixels, for a total of 7,990,272 features (or about 8 million). Images of this resolution are usually scaled down to between 128 and 512 pixels in either direction for significantly faster processing. **This often results in a loss of detail that's available for training and pattern matching.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Third, the **features in an image don't have an obvious linear or nonlinear relationship** that can be learned with a model like linear or logistic regression. In grayscale, each pixel is just represented as a brightness value ranging from 0 to 256."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is deep learning effective in image classification?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep learning is effective in image classification because of the **`models' ability to learn hierarchical representations`.** At a high level, an effective deep learning model learns intermediate representations at each layer in the model and uses them in the prediction process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In the first hidden layer the network learned to represent edges and specific features of faces. In the second hidden layer, the weights seemed to represent higher level facial features like eyes and noses. Finally, the weights in the last hidden layer resemble faces that could be matched against. **Each successive layer uses weights from previous layers to try to learn more complex representations.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Working With Image Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scikit-learn contains a number of datasets pre-loaded with the library, within the namespace of **sklearn.datasets**. The **load_digits()** function returns a copy of the hand-written digits dataset from UCI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Because dataframes are a tabular representation of data, each image is represented as a row of pixel values. To visualize an image from the dataframe, we need to reshape the image back to its original dimensions (28 x 28 pixels). To visualize the image, we need to reshape these pixel values back into the 28 by 28 and plot them on a coordinate grid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO:\n",
    "* Import load_digits() from the sklearn.datasets package.\n",
    "* Transform the NumPy 2D array into a pandas dataframe.\n",
    "* Use matplotlib to visualize some of the images in the dataset.\n",
    "  * Generate a scatter plot grid, with 2 rows and 4 columns.\n",
    "  * In the first row:\n",
    "    * Display the images corresponding to rows 0, 100, 200, and 300.\n",
    "  * In the second row:\n",
    "    * Display the images corresponding to rows 1000, 1100, 1200, and 1300."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'target_names', 'images', 'DESCR'])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "digits_data=load_digits()\n",
    "digits_data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels=pd.Series(digits_data['target'])\n",
    "features=pd.DataFrame(digits_data['data'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    1\n",
       "2    2\n",
       "3    3\n",
       "4    4\n",
       "dtype: int32"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "      <th>60</th>\n",
       "      <th>61</th>\n",
       "      <th>62</th>\n",
       "      <th>63</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    0    1    2     3     4     5    6    7    8    9   ...   54   55   56  \\\n",
       "0  0.0  0.0  5.0  13.0   9.0   1.0  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0   \n",
       "1  0.0  0.0  0.0  12.0  13.0   5.0  0.0  0.0  0.0  0.0  ...  0.0  0.0  0.0   \n",
       "2  0.0  0.0  0.0   4.0  15.0  12.0  0.0  0.0  0.0  0.0  ...  5.0  0.0  0.0   \n",
       "\n",
       "    57   58    59    60    61   62   63  \n",
       "0  0.0  6.0  13.0  10.0   0.0  0.0  0.0  \n",
       "1  0.0  0.0  11.0  16.0  10.0  0.0  0.0  \n",
       "2  0.0  0.0   3.0  11.0  16.0  9.0  0.0  \n",
       "\n",
       "[3 rows x 64 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because dataframes are a tabular representation of data, each image is represented as a row of pixel values. To visualize an image from the dataframe, we need to reshape the image back to its original dimensions (28 x 28 pixels). To visualize the image, we need to reshape these pixel values back into the 28 by 28 and plot them on a coordinate grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1e3f32162e8>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPgAAAD8CAYAAABaQGkdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAACtlJREFUeJzt3V9onfUdx/HPZ1HZ/FOsazekqYsBKchgtoaCFITVZdQpuospLShMBr1SlA2s7m53eiPuYghSdYKd0lQFEacTVJywOZO226ypo60dzapryir+GaxUv7vIKXRdtjzp+T1/ztf3C4L5c8jve4jvPs85OXl+jggByOlLbQ8AoD4EDiRG4EBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiZ9XxTZctWxYjIyN1fOtWHTt2rNH1ZmZmGltryZIlja01PDzc2FpDQ0ONrdWkgwcP6ujRo17odrUEPjIyosnJyTq+dasmJiYaXW/Lli2NrTU+Pt7YWvfdd19jay1durSxtZo0NjZW6XacogOJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQWKXAbW+w/a7tfbbvqXsoAGUsGLjtIUm/kHStpMslbbJ9ed2DAehflSP4Wkn7IuJARByX9JSkG+sdC0AJVQJfIenQKR/P9D4HoOOqBD7fX6z818XUbW+2PWl7cnZ2tv/JAPStSuAzklae8vGwpMOn3ygiHo6IsYgYW758ean5APShSuBvSbrM9qW2z5G0UdJz9Y4FoIQF/x48Ik7Yvl3SS5KGJD0aEXtqnwxA3ypd8CEiXpD0Qs2zACiMV7IBiRE4kBiBA4kROJAYgQOJETiQGIEDiRE4kFgtO5tk1eROI5L03nvvNbZWk9syXXTRRY2ttX379sbWkqSbbrqp0fUWwhEcSIzAgcQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEisys4mj9o+YvvtJgYCUE6VI/gvJW2oeQ4ANVgw8Ih4XdI/GpgFQGE8BgcSKxY4WxcB3VMscLYuArqHU3QgsSq/JntS0u8krbI9Y/tH9Y8FoIQqe5NtamIQAOVxig4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgO/ddHU1FRjazW5lZAk7d+/v7G1RkdHG1trfHy8sbWa/P9DYusiAA0icCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgsSoXXVxp+1Xb07b32L6zicEA9K/Ka9FPSPpJROy0fYGkKdsvR8Q7Nc8GoE9V9iZ7PyJ29t7/WNK0pBV1Dwagf4t6DG57RNJqSW/O8zW2LgI6pnLgts+X9LSkuyLio9O/ztZFQPdUCtz22ZqLe1tEPFPvSABKqfIsuiU9Imk6Ih6ofyQApVQ5gq+TdKuk9bZ3996+V/NcAAqosjfZG5LcwCwACuOVbEBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4kNvB7kx07dqyxtdasWdPYWlKz+4U16corr2x7hC8MjuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGJVLrr4Zdt/sP3H3tZFP2tiMAD9q/JS1X9JWh8Rn/Qun/yG7V9HxO9rng1An6pcdDEkfdL78OzeW9Q5FIAyqm58MGR7t6Qjkl6OCLYuAgZApcAj4rOIuELSsKS1tr85z23YugjomEU9ix4RH0p6TdKGWqYBUFSVZ9GX276w9/5XJH1H0t66BwPQvyrPol8s6XHbQ5r7B2F7RDxf71gASqjyLPqfNLcnOIABwyvZgMQIHEiMwIHECBxIjMCBxAgcSIzAgcQIHEiMrYsWYXx8vLG1MmvyZ7Z06dLG1uoijuBAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJEbgQGKVA+9dG32Xba7HBgyIxRzB75Q0XdcgAMqrurPJsKTrJG2tdxwAJVU9gj8o6W5Jn9c4C4DCqmx8cL2kIxExtcDt2JsM6JgqR/B1km6wfVDSU5LW237i9BuxNxnQPQsGHhH3RsRwRIxI2ijplYi4pfbJAPSN34MDiS3qii4R8ZrmdhcFMAA4ggOJETiQGIEDiRE4kBiBA4kROJAYgQOJETiQ2MBvXdTk1jRTU//3720GWpPbCU1OTja21s0339zYWl3EERxIjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxAgcSKzSK9l6V1T9WNJnkk5ExFidQwEoYzEvVf12RBytbRIAxXGKDiRWNfCQ9BvbU7Y31zkQgHKqnqKvi4jDtr8m6WXbeyPi9VNv0At/syRdcsklhccEcCYqHcEj4nDvv0ckPStp7Ty3YesioGOqbD54nu0LTr4v6buS3q57MAD9q3KK/nVJz9o+eftfRcSLtU4FoIgFA4+IA5K+1cAsAArj12RAYgQOJEbgQGIEDiRG4EBiBA4kRuBAYgQOJDbwWxeNjo42tlaTW+5I0sTERMq1mrRly5a2R2gVR3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgMQIHEiNwILFKgdu+0PYO23ttT9u+qu7BAPSv6ktVfy7pxYj4ge1zJJ1b40wAClkwcNtLJF0t6YeSFBHHJR2vdywAJVQ5RR+VNCvpMdu7bG/tXR8dQMdVCfwsSWskPRQRqyV9Kume029ke7PtSduTs7OzhccEcCaqBD4jaSYi3ux9vENzwf8Hti4CumfBwCPiA0mHbK/qfeoaSe/UOhWAIqo+i36HpG29Z9APSLqtvpEAlFIp8IjYLWms5lkAFMYr2YDECBxIjMCBxAgcSIzAgcQIHEiMwIHECBxIjMCBxNibbBHuv//+xtaSmt1Xa2ysuRcqTk1NNbbWFx1HcCAxAgcSI3AgMQIHEiNwIDECBxIjcCAxAgcSI3AgsQUDt73K9u5T3j6yfVcTwwHoz4IvVY2IdyVdIUm2hyT9TdKzNc8FoIDFnqJfI2l/RPy1jmEAlLXYwDdKenK+L7B1EdA9lQPvbXpwg6SJ+b7O1kVA9yzmCH6tpJ0R8fe6hgFQ1mIC36T/cXoOoJsqBW77XEnjkp6pdxwAJVXdm+yfkr5a8ywACuOVbEBiBA4kRuBAYgQOJEbgQGIEDiRG4EBiBA4k5ogo/03tWUmL/ZPSZZKOFh+mG7LeN+5Xe74REQv+VVctgZ8J25MR0dwGWQ3Ket+4X93HKTqQGIEDiXUp8IfbHqBGWe8b96vjOvMYHEB5XTqCAyisE4Hb3mD7Xdv7bN/T9jwl2F5p+1Xb07b32L6z7ZlKsj1ke5ft59uepSTbF9reYXtv72d3Vdsz9aP1U/Tetdb/orkrxsxIekvSpoh4p9XB+mT7YkkXR8RO2xdImpL0/UG/XyfZ/rGkMUlLIuL6tucpxfbjkn4bEVt7Fxo9NyI+bHuuM9WFI/haSfsi4kBEHJf0lKQbW56pbxHxfkTs7L3/saRpSSvanaoM28OSrpO0te1ZSrK9RNLVkh6RpIg4PshxS90IfIWkQ6d8PKMkIZxke0TSaklvtjtJMQ9KulvS520PUtiopFlJj/Uefmy1fV7bQ/WjC4F7ns+leWrf9vmSnpZ0V0R81PY8/bJ9vaQjETHV9iw1OEvSGkkPRcRqSZ9KGujnhLoQ+Iyklad8PCzpcEuzFGX7bM3FvS0islyRdp2kG2wf1NzDqfW2n2h3pGJmJM1ExMkzrR2aC35gdSHwtyRdZvvS3pMaGyU91/JMfbNtzT2Wm46IB9qep5SIuDcihiNiRHM/q1ci4paWxyoiIj6QdMj2qt6nrpE00E+KVrpscp0i4oTt2yW9JGlI0qMRsaflsUpYJ+lWSX+2vbv3uZ9GxAstzoSF3SFpW+9gc0DSbS3P05fWf00GoD5dOEUHUBMCBxIjcCAxAgcSI3AgMQIHEiNwIDECBxL7NyyRs2/TGgiSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "first_image=features.iloc[0]\n",
    "np_image=first_image.values\n",
    "\n",
    "np_image=np_image.reshape(8,8)\n",
    "\n",
    "plt.imshow(np_image,cmap='gray_r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAADiCAYAAABupy2YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEnhJREFUeJzt3X+MV2V2x/HPcWREAREyQKOoA6K0kq5WpljrrsUYDdtNA91df3W3ibYNa1KbatqEjWmqJt1WkyZC0127xEVMu66r2wKmYf1RI2OT3TXOrLgKFQM4yuwkAiouUhXR0z9gzBSGeZ7vfO8z93uY9+uf+cGZ+xw+czl8udxnrrm7AABxnFR3AwCAxjC4ASAYBjcABMPgBoBgGNwAEAyDGwCCYXADQDAMbgAIhsENAMGcXOKgHR0d3tnZ2fRx3n333WRNf39/sub000/PWm/27NnJmra2tqxjjaSvr0979+610Xxtu53iEzWp6R5y2MRTkjWT536Qdaz9W8fuNcJ+vbvX3Wc0+nVjmW0Ov6A9q+6Ukw4law6++mmz7UgafbZSdfnayemx9enc9Pk2f+K+pnsZtOXA9GTNhB0fjvjrH+qADvpHWXMha3Cb2RJJqyS1SXrA3e8Zqb6zs1M9PT05hx7RY489lqxZsWJFsubqq6/OWu+ee0b8bUmSpk2blnWskXR1dX32fqPZTtQkXWpXNd1DjrZ585M1n//B5qxjdX/u1GbbyfZf/qM3Bt9vJN+xzDbHwe+cm1XXOeWdZM3A7+xvth1Jo89Wqi7fto6ZyZoPvpM+355dsKHpXgYt+OnXkjWzv7JlxF9/3p/JXi/515KZtUn6tqQvSrpQ0o1mdmH2Cjgusi2LfMsh23rl/Pt1kaTt7r7T3Q9KekTS0rJtjRtkWxb5lkO2NcoZ3GdJ2jXk4/4jn0PzyLYs8i2HbGuUM7iHu1h+zM+CNbPlZtZjZj179uxpvrPxoeFsP9ZHY9DWCSOZL9mOGudujXIGd7+ks4d8PFvSwNFF7r7a3bvcvWvGjFH9p/N41HC2E5S+0wOfSeZLtqPGuVujnMH9gqTzzWyOmbVLukHS42XbGjfItizyLYdsa5S8HdDdD5nZrZKe1OHbfta4+8j3tSAL2ZZFvuWQbb2y7uN2942SNhbu5Rg592i//vrryZqcjTySNH16+ib6Rx99NFlz7bXXZq0n1Zdtjm1/lr5nfeXUn2cdq1uXN9vOqLRqvp/+3m8la757wT9nHWvrwVnJmvs1L+tYjagr288//Uay5sJTf5msOe+Ht2Std8Vl6b+PZp7+ftaxqsKWdwAIhsENAMEwuAEgGAY3AATD4AaAYBjcABAMgxsAgmFwA0AwRZ6Ak6O3tzdZk7O5ZseOHcmauXPnZvWU88CFnL4b2YBTl3dvuixZs+P6+5M1i+7466z1Ohakf9j/J1u2ZR3rRHDob9J5XDAh72kx33gt/XCCdqU3rUTx4zsXJ2u61z2frJmnn2Wtd/4v0k952n5v+keRV/k94BU3AATD4AaAYBjcABAMgxsAgmFwA0AwDG4ACIbBDQDBMLgBIJjaNuDkPJXmkksuSdbkbq7JsXDhwsqO1er+7e5/TNbc/OaSZE3Hj9MboCRp44tPJWu+8OffSNaclrGxom6vrelK1ry+4IHK1tv9q8nJmtmVrVa/qs6BnKcQSdJXp6afRNS97tRm22kIr7gBIBgGNwAEw+AGgGAY3AAQDIMbAIJhcANAMAxuAAiGwQ0AwbT0BpycJ9JUKaenadOmjUEnzfnfP7w0WXPBhM3Jmrf+9Mxkzf/8w5SsnnIMXGHJmnnrKltuVHKyfX3Jd5M1N7/5hWTNg+f8d1ZPi856M1kzkHWkGNoWzE/WrNy4JlmT82fgsPSTiHI2XV3wJz2Z66XxihsAgmFwA0AwDG4ACIbBDQDBMLgBIBgGNwAEw+AGgGAY3AAQDIMbAIKpbedkzg7E3t7eStbK2REpST096Z1N1113XbPtFJfzaKcFX/9asuZb6zcka5ZNej+rpxxnPueVHauUie8cTNYs+Gk62zP/qT292A/ydk727Z+erGnX/qxjRfBxx2nJmh+9l37s4R0d27LWO++HtyRrfuOencmaT7JWy5M1uM2sT9L+I2sfcvf0/k5kIduyyLccsq1PI6+4r3T3vcU6Gd/ItizyLYdsa8A1bgAIJndwu6SnzKzXzJYPV2Bmy82sx8x69uzZU12HJ76Gsv1YH41xe+GNmC/ZNoVztya5l0oud/cBM5sp6Wkze9Xdnxta4O6rJa2WpK6urtb/X6bW0VC2p9t0sm3MiPmSbVM4d2uS9Yrb3QeOvN0taZ2kRSWbGk/ItizyLYds65Mc3GY2ycymDL4v6RpJr5RubDwg27LItxyyrVfOpZJZktaZ2WD9w+7+RNGuxg+yLYt8yyHbGiUHt7vvlHRR1QvPnTs3WZOzIeaxxx6rpCbXihUrKjtWqWxzzP7KlmTN/ZqXrNn6iw+y1vveM1cma+at+1nWsXKVyPek7heTNbO7q1wx7S87n0nW5HwvG1HnuZvzPfjX/7gqWfP938y77Xze7enzssrNNTm4HRAAgmFwA0AwDG4ACIbBDQDBMLgBIBgGNwAEw+AGgGAY3AAQTG1PwMnZgHPvvfcma3I2xHR15d1oX9UTd3CsyW/yGqFRV25ZWtmx2vVGZceK4I+/nN6U9Oxf/O4YdFIGf5oAIBgGNwAEw+AGgGAY3AAQDIMbAIJhcANAMAxuAAiGwQ0AwZh79Q9eNrM90v+7479D0t7KFyqvVN/nuvuM0Xwh2WYZVb7DZCvFzLflspU4dzNkZ1tkcB+ziFmPu+dtX2whEfqO0ONwovQdpc+hovQcpc+jtULfXCoBgGAY3AAQzFgN7tVjtE7VIvQdocfhROk7Sp9DRek5Sp9Hq73vMbnGDQCoDpdKACCY4oPbzJaY2TYz225m3yy9XhXMrM/MXjazzWbWU3c/xxMxWylGvmRbVsR8WynbopdKzKxN0muSrpbUL+kFSTe6+9Zii1bAzPokdbl7y95jGjVbqfXzJduyoubbStmWfsW9SNJ2d9/p7gclPSKpusd6jG9kWw7ZlkW+TSo9uM+StGvIx/1HPtfqXNJTZtZrZsvrbuY4omYrtX6+ZFtW1HxbJtvSz5y0YT4X4TaWy919wMxmSnrazF519+fqbuooUbOVWj9fsi0rar4tk23pV9z9ks4e8vFsSQOF12yauw8cebtb0jod/qddqwmZrRQiX7ItK2S+rZRt6cH9gqTzzWyOmbVLukHS44XXbIqZTTKzKYPvS7pG0iv1djWscNlKYfIl27LC5dtq2Ra9VOLuh8zsVklPSmqTtMbdt5RcswKzJK0zM+lwPg+7+xP1tnSsoNlKAfIl27KC5ttS2bJzEgCCYeckAATD4AaAYBjcABAMgxsAgmFwA0AwDG4ACIbBDQDBMLgBIBgGNwAEw+AGgGAY3AAQDIMbAIJhcANAMAxuAAiGwQ0AwTC4ASAYBjcABMPgBoBgGNwAEAyDGwCCYXADQDAMbgAIhsENAMEwuAEgGAY3AATD4AaAYBjcABAMgxsAgmFwA0AwDG4ACObkEgdtt1N8oiY1fRw7Od3ejF9/P1mzZ/vUrPX8w4+y6pr1oQ7ooH9ko/najo4O7+zsbLqHbdu2JWvefz+dba7Jkycna+bPn1/JWr29vXvdfUajX1dVtrt27UrW7N69u+l1BuVkO2fOnGRNe3t7sma02UrV5fvWW28la3LyPfXUU7PWy8mura0t61gj6evr0969e7PmQtbgNrMlklZJapP0gLvfM1L9RE3SpXZVzqFH1NYxM1mzfP1PkjWrl30pa71PtqSHWRWe92c+e7/RbDs7O9XT09N0D4sXL07WdHd3N73OoIULFyZrNm3aVMlaZvbGkPez860q29tuuy1Zs2rVqqbXGZST7dq1a5M1OUN1tNkOHr+KfFeuXFlJzcUXX5y1Xk52Z5xxRtaxRtLV1ZVdm7xUYmZtkr4t6YuSLpR0o5ldOOru8BmyLYt8yyHbeuVc414kabu773T3g5IekbS0bFvjBtmWRb7lkG2Ncgb3WZKGXrTrP/I5NI9syyLfcsi2RjmDe7iL5X5MkdlyM+sxs56PNTb/yXcCaDjbPXv2jEFbJ4xkvmQ7apy7NcoZ3P2Szh7y8WxJA0cXuftqd+9y964JOqWq/k50DWc7Y8ao/kN/vErmS7ajxrlbo5zB/YKk881sjpm1S7pB0uNl2xo3yLYs8i2HbGuUvB3Q3Q+Z2a2SntTh237WuPuW4p2NA2RbFvmWQ7b1yrqP2903StpYuJdjfPJW+ib6ZZPSm0T+/renZ603rYbTrq5sb7rppmTNXXfdlazJucdVkjZv3pys2bdvX7Km0ftlq8435/eRcz/6+vXrkzVTp+ZtHMv5Xr700kvJmkY3x5Q4d3POp5zzMuc+7pzzTcq7Lz/3z0FV2PIOAMEwuAEgGAY3AATD4AaAYBjcABAMgxsAgmFwA0AwDG4ACKbIE3DG0voD6ad/HPiDX2Uda9raJpsJJGfTRo6czQlS3oMbqvhh9KXl/PD9nE06OXI2kUh5G0kuuuiiZtsZEzkbk3I24OSc3319femGlNfTWOMVNwAEw+AGgGAY3AAQDIMbAIJhcANAMAxuAAiGwQ0AwTC4ASCY8BtwVvVdlazZctn3s471+7OuSdbkPJUngg0bNiRrnn322WRNzpNVcuuqerpMBDn533777VnHWrp0abKm0afb1OXmm29O1uQ+GSglwoav4+EVNwAEw+AGgGAY3AAQDIMbAIJhcANAMAxuAAiGwQ0AwTC4ASCY8BtwTv676cma9Q+kn5IjSb/8o3nJml+7r/U34ORsZFm2bFkla+Vs/pDynlqS83SZE0VObnfeeWfWse6+++5kTc7TdHKfZlRS7vlUhdwn4LTieckrbgAIhsENAMEwuAEgGAY3AATD4AaAYBjcABAMgxsAgmFwA0AwDG4ACCZr56SZ9UnaL+kTSYfcvatkU404qfvFZM1f/efXs451xfUvJ2sG7ss6VLYS2ebs9LrvvvRvZO3atcma9evX57RUm1Y+d1NydptKed+Dffv2NdnNsSJnK+Xnm1s3lhrZ8n6lu+8t1sn4RrZlkW85ZFsDLpUAQDC5g9slPWVmvWa2vGRD4xDZlkW+5ZBtTXIvlVzu7gNmNlPS02b2qrs/N7TgyDduuSRN1GkVt3lCayjbc845p44eIxsxX7JtCuduTbJecbv7wJG3uyWtk7RomJrV7t7l7l0TdEq1XZ7AGs12xowZY91iaKl8yXb0OHfrkxzcZjbJzKYMvi/pGkmvlG5sPCDbssi3HLKtV86lklmS1pnZYP3D7v5E0a7GD7Iti3zLIdsaJQe3u++UdNEY9DLukG1Z5FsO2darpR9d1v/vC5I13/rchmTN1g8+yFrvjo5tyZrz7rslWTN3/UcjF/T8JKufkjZt2pSsqerxZuNNTrbvvfdeZevlPIJr8eLFla0XQc5j2HIfXXbGGWcka3IeF1jlI9C4jxsAgmFwA0AwDG4ACIbBDQDBMLgBIBgGNwAEw+AGgGAY3AAQTEtvwDn3bw8la1atvCpZ0znlnaz1Xvv458maHdf/S/o4Xz4w4q9/9Uv1/9z5DRvSG5da8ckfEeTk1t3dnayZOnVq1noPPfRQsuZE2oCzcuXKZM2qVasqW2/OnDmVHOfBBx8c8dfffvvt7GPxihsAgmFwA0AwDG4ACIbBDQDBMLgBIBgGNwAEw+AGgGAY3AAQjLl79Qc12yPpjSGf6pBU/66TxpXq+1x3H9Ujr8k2y6jyHSZbKWa+LZetxLmbITvbIoP7mEXMety9q/hCFYvQd4QehxOl7yh9DhWl5yh9Hq0V+uZSCQAEw+AGgGDGanCvHqN1qhah7wg9DidK31H6HCpKz1H6PFrtfY/JNW4AQHW4VAIAwRQf3Ga2xMy2mdl2M/tm6fWqYGZ9ZvaymW02s566+zmeiNlKMfIl27Ii5ttK2Ra9VGJmbZJek3S1pH5JL0i60d23Flu0AmbWJ6nL3Vv2HtOo2Uqtny/ZlhU131bKtvQr7kWStrv7Tnc/KOkRSUsLrzlekG05ZFsW+Tap9OA+S9KuIR/3H/lcq3NJT5lZr5ktr7uZ44iardT6+ZJtWVHzbZlsSz9z0ob5XITbWC539wEzmynpaTN71d2fq7upo0TNVmr9fMm2rKj5tky2pV9x90s6e8jHsyUNFF6zae4+cOTtbknrdPifdq0mZLZSiHzJtqyQ+bZStqUH9wuSzjezOWbWLukGSY8XXrMpZjbJzKYMvi/pGkmv1NvVsMJlK4XJl2zLCpdvq2Vb9FKJux8ys1slPSmpTdIad99Scs0KzJK0zsykw/k87O5P1NvSsYJmKwXIl2zLCppvS2XLzkkACIadkwAQDIMbAIJhcANAMAxuAAiGwQ0AwTC4ASAYBjcABMPgBoBg/g+29Ve+dDRIZAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "f,ax=plt.subplots(2,4)\n",
    "\n",
    "ax[0,0].imshow(features.iloc[0].values.reshape(8,8),cmap='gray_r')\n",
    "ax[0,1].imshow(features.iloc[100].values.reshape(8,8))\n",
    "ax[0,2].imshow(features.iloc[200].values.reshape(8,8))\n",
    "ax[0,3].imshow(features.iloc[300].values.reshape(8,8))\n",
    "\n",
    "ax[1,0].imshow(features.iloc[1000].values.reshape(8,8))\n",
    "ax[1,1].imshow(features.iloc[1200].values.reshape(8,8),cmap='gray_r')\n",
    "ax[1,2].imshow(features.iloc[1300].values.reshape(8,8),cmap='gray_r')\n",
    "ax[1,3].imshow(features.iloc[1400].values.reshape(8,8),cmap='gray_r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. K-Nearest Neighbors Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While linear and logistic regression models make assumptions about the linearity between the features and the output labels, the k-nearest neighbors algorithm make no such assumption. This allows them to capture nonlinearity in the data. If you recall, `k-nearest neighbors don't have a specific model representation (hence why it's referred to as an algorithm and not a model)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Split up the data into train and test sets.\n",
    "  * Setup a pipeline for testing and training k-nn models.\n",
    "  * Write a function named train() that uses KNeighborsClassifer for training k-nearest neighbors models.\n",
    "  * Write a function named test() that tests the model.\n",
    "  * Write a function named cross_validate() that performs 4-fold cross validation using train() and test().\n",
    "* Experiment with different values for k and plot the resulting classification accuracies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_features,test_features,train_labels,test_labels=train_test_split(features,labels,test_size=0.5,random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "def train(nneighbors,train_features,train_labels):\n",
    "    knn=KNeighborsClassifier(n_neighbors=nneighbors)\n",
    "    knn.fit(train_features,train_labels)\n",
    "    return knn\n",
    "    \n",
    "    \n",
    "def test(model,test_features,test_labels):\n",
    "    prediction=model.predict(test_features)\n",
    "    train_test_df=pd.DataFrame()\n",
    "    train_test_df['predicted_labels']=prediction\n",
    "    train_test_df['correct_lables']=test_labels\n",
    "    accuracy=sum(train_test_df['predicted_labels']==train_test_df['correct_lables'])/len(train_test_df)\n",
    "    return accuracy\n",
    "\n",
    "\n",
    "def cross_validate(k):\n",
    "    fold_accuracies=[]\n",
    "    kf=KFold(n_splits=4,random_state=2)\n",
    "    for train_index,test_index in kf.split(features):\n",
    "        train_features,test_features=features.loc[train_index],features.loc[test_index]\n",
    "        train_labels,test_labels=labels.loc[train_index],labels.loc[test_index]\n",
    "        model=train(k,train_features,train_labels)\n",
    "        accuracy=test(model,test_features,test_labels)\n",
    "        fold_accuracies.append(accuracy)\n",
    "        return fold_accuracies\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1e3f35d5240>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VOXZ//HPlYQEgbAmbAmyCUIEZAkURYRqbUHrAuhTXHCtSyuKbW2F+vi0P1ofbdVaXFqLispT1yqi4oItguKGBMIOAQSFBJCwCrKEhOv3xxzaaRrNACFnkvm+X695ZeY+9zlznYHMN2e9zd0RERFJCrsAERGJDwoEEREBFAgiIhJQIIiICKBAEBGRgAJBREQABYKIiAQUCCIiAigQREQkkBJLJzMbAkwAkoHH3P3uctPbApOATGAbcJm7F5rZt4H7o7p2AUa6+1Qzaw88BzQF5gOj3L3km+rIyMjwdu3axbRiIiISMW/evC3unllZP6vs1hVmlgysBM4CCoG5wMXuviyqz9+Aae7+lJmdAVzl7qPKLacpsBrIdvc9ZvYCMMXdnzOzR4CF7v7nb6olNzfX8/LyKlsnERGJYmbz3D23sn6x7DLqB6x29zXBX/DPAeeX65MDzAiez6xgOsCFwJtBGBhwBvBiMO0p4IIYahERkWMklkDIAtZHvS4M2qItBEYEz4cB6WbWrFyfkcCzwfNmwA53L/2GZYqISDWKJRCsgrby+5luBQaZWT4wCCgCDn3ZY2atgO7A9MNY5qF5rzOzPDPLKy4ujqFcERE5ErEEQiHQJup1NrAhuoO7b3D34e7eC7g9aNsZ1eW/gJfd/UDwegvQ2MwOHdT+j2VGLXuiu+e6e25mZqXHRERE5AjFEghzgU5m1t7MUons+nk1uoOZZZjZoWWNI3LGUbSL+dfuIjxyJHsmkeMKAFcArxx++SIiUlUqDYRgP/9oIrt7lgMvuPtSMxtvZucF3QYDBWa2EmgB3HlofjNrR2QL491yi74N+KmZrSZyTOHxo1oTERE5KpWedhpPdNqpiMjhq8rTTmu81xdt5JUFRWGXISIS12K6Urkmc3deml/IOys2M3vVFv7feSdRP63Wr7aIyGGr9VsIZsbEUX24+cxOvDS/kHMffJ+lG3ZWPqOISIKp9YEAkJKcxE/P6swzP+zPVyWlDHv4Q574YC016fiJiMixlhCBcMgpHZvx5pjTOb1zBv/vtWVcOzmPbV994/30REQSRkIFAkDT+qk8enkuvzo3h/dWbmHohPf46NOtYZclIhK6hAsEiBxXuGpAe6b8+FTqp6ZwyWMf84e3CygtOxh2aSIioUnIQDikW1YjXrvpNEb0zuaBd1Zz8aMfU7Rjb9hliYiEIqEDAaB+Wgr3XnQyE0b2ZPnGXZw9YTZvLdkUdlkiItUu4QPhkPN7ZvH6zafRtlk9bvjrPP576mL2HSgLuywRkWqjQIjStll9XrzhVK47vQN//XgdFzz8Aau+2BV2WSIi1UKBUE5qShK/PLsrT17Vl+Jd+zn3ofd59pN1umZBRGo9BcLXGHxic94cM5Dctk0ZN2Uxo5/JZ+feA5XPKCJSQykQvkHzhnWZfHU/bhvShelLN3HOA7OZv2572GWJiBwTCoRKJCUZPxrckRduOAWAix75iD/NWs3Bg9qFJCK1iwIhRr2Pb8LrNw9kSLeW/P6tAkZNmsPmL/eFXZaISJVRIByGRsfV4aGLe3H38O7M+3w7QyfMZmbB5rDLEhGpEgqEw2RmjOx3PK+NPo3M9DSuemIuv522jJJS3fZCRGo2BcIR6tQinak3DmBU/7Y89v5aLnzkQz7b8lXYZYmIHLGYAsHMhphZgZmtNrOxFUxva2YzzGyRmc0ys+yoaceb2dtmttzMlplZu6D9STNba2YLgkfPqlqp6lK3TjK/uaAbj1zWh8+37uGcB2YzNV9DdYpIzVRpIJhZMvAwMBTIAS42s5xy3e4FJrt7D2A8cFfUtMnAPe7eFegHRO90/7m79wweC45iPUI1pFtL3hgzkJzWDbnl+QX87IWFfLW/NOyyREQOSyxbCP2A1e6+xt1LgOeA88v1yQFmBM9nHpoeBEeKu/8dwN13u/ueKqk8zmQ1Po5nr+3PzWd2Ykp+ZKjOJUUaqlNEao5YAiELWB/1ujBoi7YQGBE8Hwakm1kzoDOww8ymmFm+md0TbHEccmewm+l+M0s7wnWIG9FDde4pKWP4nz5k0vsaqlNEaoZYAsEqaCv/DXcrMMjM8oFBQBFQCqQAA4PpfYEOwJXBPOOALkF7U+C2Ct/c7DozyzOzvOLi4hjKDd8pHZvxxpiBnN45g/HTlvHDpzRUp4jEv1gCoRBoE/U6G9gQ3cHdN7j7cHfvBdwetO0M5s0PdjeVAlOB3sH0jR6xH3iCyK6p/+DuE909191zMzMzD3P1wnNoqM5fn5vD7FWRoTo//HRL2GWJiHytWAJhLtDJzNqbWSowEng1uoOZZZjZoWWNAyZFzdvEzA59k58BLAvmaRX8NOACYMnRrEg8MjOuHNCel288lfppKVz62Bwem70m7LJERCpUaSAEf9mPBqYDy4EX3H2pmY03s/OCboOBAjNbCbQA7gzmLSOyu2iGmS0msvvp0WCep4O2xUAG8NsqW6s4c1LrRrw2+jSGnNSS376+nBfmrq98JhGRamY16YBnbm6u5+XlhV3GESspPcg1T83lg9Vb+MuoXM7KaRF2SSKSAMxsnrvnVtZPVypXo9SUJB65rA/dsxsz+pn5fLJ2W9gliYj8kwKhmtVPS+GJK/uS1eQ4rnlqLss3fhl2SSIigAIhFE3rpzL56n7UT03hikmfsH5brbxWT0RqGAVCSLKb1GPyNf3YX3qQUY/PYcvu/WGXJCIJToEQos4t0pl0ZS6bvtzHlU98wq59GrNZRMKjQAhZn7ZN+fOlfVi+cRfX/9889peWhV2SiCQoBUIc+HaX5txzYQ8+/HQrP3l+AWUar1lEQpASdgESMbx3Ntu+KuG3ry+nSb0l/PaCbkQu4hYRqR4KhDjyw4EdKN69n7+8u4aMBmn85KzOYZckIglEgRBnxg7pwrbdJUyYsYqMBqmMOqVd2CWJSIJQIMQZM+Ou4d3ZvucA//PqUprUT+X7PVqHXZaIJAAdVI5DKclJPHRJL3LbNuEnzy/g/VW6bbaIHHsKhDhVt04yj13el46ZDbj+//JYVLgj7JJEpJZTIMSxRvXq8NTV/WhSP5Urn5jLmuLdYZckIrWYAiHOtWhYl8lX98OAUY9/whdf7gu7JBGppRQINUCHzAY8eVU/duwp4fLHP2HnHt3iQkSqngKhhuie3YiJl+eydstXXPPUXPaW6BYXIlK1FAg1yIATMrj/Bz2Zt247o5+Zz4Gyg2GXJCK1iAKhhjmnRyvGn9+NGSs2M27KYmrSEKgiEt9iCgQzG2JmBWa22szGVjC9rZnNMLNFZjbLzLKjph1vZm+b2XIzW2Zm7YL29mY2x8xWmdnzZpZaVStV243q35ZbvtOJF+cVcvdbK8IuR0RqiUoDwcySgYeBoUAOcLGZ5ZTrdi8w2d17AOOBu6KmTQbucfeuQD9gc9D+O+B+d+8EbAeuOZoVSTRjzuzEqP5t+cu7a3j0vTVhlyMitUAsWwj9gNXuvsbdS4DngPPL9ckBZgTPZx6aHgRHirv/HcDdd7v7HovcxvMM4MVgnqeAC45qTRKMmfHr807inO6tuPON5bw0rzDskkSkhoslELKA9VGvC4O2aAuBEcHzYUC6mTUDOgM7zGyKmeWb2T3BFkczYIe7l37DMqUSyUnGH35wMgNOaMYvXlrEOyu+CLskEanBYgmEim7KX/5I5q3AIDPLBwYBRUApkZvnDQym9wU6AFfGuMzIm5tdZ2Z5ZpZXXFwcQ7mJJS0lmb+MyiWnVUN+/PR85n2+LeySRKSGiiUQCoE2Ua+zgQ3RHdx9g7sPd/dewO1B285g3vxgd1MpMBXoDWwBGptZytctM2rZE909191zMzMzD2PVEkeDtBSeuKovrRodx1VPzKVg066wSxKRGiiWQJgLdArOCkoFRgKvRncwswwzO7SsccCkqHmbmNmhb/IzgGUeOVdyJnBh0H4F8MqRr4ZkNEhj8tX9qFsnmcsnzaFw+56wSxKRGqbSQAj+sh8NTAeWAy+4+1IzG29m5wXdBgMFZrYSaAHcGcxbRmR30QwzW0xkV9GjwTy3AT81s9VEjik8XmVrlaDaNK3H5Gv6sbekjMsf/4Stu/eHXZKI1CBWky5sys3N9by8vLDLiHtzP9vGZY/NoUvLdJ65tj/10zQOkkgiM7N57p5bWT9dqVwL9W3XlIcv6c2SDV9yw1/nUVKqW1yISOUUCLXUd3JacNfw7sxetYWf/W0hBw/WnC1BEQmH9iXUYv+V24ZtX5Vw95sraFqvDr8+7yQi1wSKiPwnBUItd/3pHdi6ez+Pzl5LRoM0bjqzU9gliUicUiDUcmbGuKFd2bq7hPv+vpKmDVK59Fttwy5LROKQAiEBJCUZv7uwB9v3lHDH1CU0rZfK0O6twi5LROKMDioniDrJSfzp0j70bNOYMc8t4K0lG8MuSUTijAIhgRyXmsykK/vStVU6N/x1PndMXcK+AxqKU0QiFAgJpnG9VP52w6lcd3oH/u/jz7ng4Q9Y9YXufSQiCoSElJqSxC/P7sqTV/WleNd+zn3ofZ77ZJ2G4xRJcAqEBDb4xOa8OWYguW2bMnbKYkY/m8+X+w6EXZaIhESBkOCaN6zL5Kv7cduQLkxfsomzJ8xm/rrtYZclIiFQIAhJScaPBnfkhRtOAeCiRz7iT7NW63YXIglGgSD/1Pv4Jrx+80CGdGvJ798q4PJJn7B5176wyxKRaqJAkH/T6Lg6PHRxL+4e3p28z7cx9I+zmVWwOeyyRKQaKBDkP5gZI/sdz2ujTyMzPY0rn5jLna8v0220RWo5BYJ8rU4t0pl64wBG9W/Lo7PXcuEjH/LZlq/CLktEjhEFgnyjunWS+c0F3Xjksj58vnUP5zwwm6n5RWGXJSLHgAJBYjKkW0veGDOQnNYNueX5BfzshYV8tb807LJEpArFFAhmNsTMCsxstZmNrWB6WzObYWaLzGyWmWVHTSszswXB49Wo9ifNbG3UtJ5Vs0pyrGQ1Po5nr+3PzWd2Ykp+Iec++D5LinaGXZaIVJFKA8HMkoGHgaFADnCxmeWU63YvMNndewDjgbuipu11957B47xy8/08atqCI18NqS4pyUn89KzOPPPD/uwpKWP4nz5k0vtrddsLkVogli2EfsBqd1/j7iXAc8D55frkADOC5zMrmC61zCkdm/HGmIGc3jmD8dOW8cOn8tj2VUnYZYnIUYglELKA9VGvC4O2aAuBEcHzYUC6mTULXtc1szwz+9jMLig3353Bbqb7zSytojc3s+uC+fOKi4tjKFeqS9P6qTx6eS6/OjeH2au2MHTCe3z06dawyxKRIxRLIFQ0Knv5/QO3AoPMLB8YBBQBh444Hu/uucAlwB/NrGPQPg7oAvQFmgK3VfTm7j7R3XPdPTczMzOGcqU6mRlXDWjPlB+fSv3UFC557GP+8HYBpWW6ZkGkpoklEAqBNlGvs4EN0R3cfYO7D3f3XsDtQdvOQ9OCn2uAWUCv4PVGj9gPPEFk15TUUN2yGvHaTacxonc2D7yzmpETP6Zox96wyxKRwxBLIMwFOplZezNLBUYCr0Z3MLMMMzu0rHHApKC9yaFdQWaWAQwAlgWvWwU/DbgAWHL0qyNhqp+Wwr0XncyEkT1ZsWkXQ//4nobqFKlBKg0Edy8FRgPTgeXAC+6+1MzGm9mhs4YGAwVmthJoAdwZtHcF8sxsIZGDzXe7+7Jg2tNmthhYDGQAv62idZKQnd8zi9dvPo12GfW54a/z+e+pizVUp0gNYDXpdMHc3FzPy8sLuwyJUUnpQe59u4CJ763hxBbpPHhJLzq3SA+7LJGEY2bzgmO530hXKssxEz1U55bd+znvofd5VkN1isQtBYIcc4NPbM6bt0SG6hw3ZTGjn8ln514N1SkSbxQIUi2ap0cN1bk0MlTnvM81VKdIPFEgSLWJHqrTDP7rLx/x8EwN1SkSLxQIUu2ih+q8Z3oBoybNYfOXGqpTJGwKBAnFoaE6fzeiO/M+387QCbOZqaE6RUKlQJDQmBk/6PuvoTqvemIuv52moTpFwqJAkNBFD9X52PtrGfFnDdUpEgYFgsSF6KE6122LDNX5cn5h2GWJJBQFgsSV6KE6f/L8Qn76wgIN1SlSTRQIEneih+qcml/E9zVUp0i1UCBIXPrnUJ3X9mdvMFTn4xqqU+SYUiBIXOvfoRlvBkN1/mbaMq55Ko+tu/eHXZZIraRAkLjXJBiq89fn5vD+qi0MnTCbDz/dEnZZIrWOAkFqBDPjygHtefnGU2lQN4VLH5vDfRqqU6RKKRCkRjmpdSOm3XQaF/bO5sF3VvODiR9TuH1P2GWJ1AoKBKlx6qWmcE8wVGfBpl2cPWG2huoUqQIKBKmxDg3V2T4YqvP2lzVUp8jRUCBIjda2WX3+dsOpXH96B56es47zH/qAlV/sCrsskRoppkAwsyFmVmBmq81sbAXT25rZDDNbZGazzCw7alqZmS0IHq9Gtbc3szlmtsrMnjez1KpZJUk0qSlJjDu7K09d3Y+tX0WG6nxmjobqFDlclQaCmSUDDwNDgRzgYjPLKdftXmCyu/cAxgN3RU3b6+49g8d5Ue2/A+53907AduCao1gPEQZ1zuSNMQPp264pv3x5MTc+M19DdYochli2EPoBq919jbuXAM8B55frkwPMCJ7PrGD6vzEzA84AXgyangIuiLVoka/TPL0uT13Vj7FDu/D20i80VKfIYYglELKA9VGvC4O2aAuBEcHzYUC6mTULXtc1szwz+9jMDn3pNwN2uPuhu5ZVtEwAzOy6YP684uLiGMqVRJeUZNwwqCN/u+EUkpL+NVRnmYbqFPlGsQSCVdBW/jfrVmCQmeUDg4Ai4NCX/fHungtcAvzRzDrGuMxIo/tEd89199zMzMwYyhWJ6BUM1Xl291aRoTofn8MXGqpT5GvFEgiFQJuo19nAhugO7r7B3Ye7ey/g9qBt56Fpwc81wCygF7AFaGxmKV+3TJGq0LBuHR4Y2ZPfj+hB/rodkaE6V2ioTpGKxBIIc4FOwVlBqcBI4NXoDmaWYWaHljUOmBS0NzGztEN9gAHAMo+c/jETuDCY5wrglaNdGZGKmBn/1bcNr900gObpaVz15Fx+M20Z+0t1zYJItEoDIdjPPxqYDiwHXnD3pWY23swOnTU0GCgws5VAC+DOoL0rkGdmC4kEwN3uviyYdhvwUzNbTeSYwuNVtE4iFTqheWSozitOacvjwVCdazVUp8g/WU06Vzs3N9fz8vLCLkNqgbeXbuIXLy3iQOlBfnXuSQzrnUWdZF2nKbWTmc0LjuV+cz8FgiSqDTv2csvzC/hk7Taa1U/l3JNbM7x3Ft2zGhE5M1qkdlAgiMSg7KDzzorNvJxfyD+Wbaak7CAnNG/AsF5ZXNAri6zGx4VdoshRUyCIHKadew7w+uKNvJxfyNzPtmMG/ds3Y3jvLIZ2b0WDtJTKFyIShxQIIkdh3dY9vJxfxJT8Qj7fuoe6dZL43kktGdYri9NOyCBFxxukBlEgiFQBd2f+uh1MmV/ItEUb2bn3AJnpaZx/cmuG984mp3XDsEsUqZQCQaSK7S8tY+aKzUyZX8TMgs0cKHO6tExneO8szu+ZRYuGdcMuUaRCCgSRY2j7VyVMW7SBl+YXsWD9DpIMBpyQwYje2Xz3pBbUS9XxBokfCgSRarKmeDcv5xfxcn4Rhdv3Uj81mSHdWjG8dxb9OzQjOUmnsEq4FAgi1ezgQWfuZ9t4Ob+I1xdtZNf+Ulo1qsv5PbMY0TuLTi3Swy5REpQCQSRE+w6U8Y/lXzBlfhHvriym7KDTLashw3tlc17P1mQ0SAu7REkgCgSROLFl935eXbCBl/OLWFy0k+QkY1DnTIb1yuKsnBbUrZMcdolSyykQROLQyi92MWV+Ea8sKGLjzn2kp6VwTo9WDOuVRd92TUnS8QY5BhQIInGs7KDz8ZqtTJlfxJtLNrKnpIwTW6Tzx5E96dpK1zZI1VIgiNQQe0pKeXPxJu5+awU79x7gjnO6cln/trrBnlSZWANB19+LhKxeagoj+mTz5piBnNqxGXe8spQb/jqPHXtKwi5NEowCQSROZDRIY9IVfbn97K68s2IzZ0+YzSdrt4VdliQQBYJIHElKMq49vQMv/ehU6qQkMXLiR0z4xyrKDtacXbtScykQROJQj+zGTLvpNM47uTX3/2Mllzz6MRt37g27LKnlYgoEMxtiZgVmttrMxlYwva2ZzTCzRWY2y8yyy01vaGZFZvZQVNusYJkLgkfzo18dkdojvW4d7v9BT+696GQWF+1k6ITZ/H3ZF2GXJbVYpYFgZsnAw8BQIAe42MxyynW7F5js7j2A8cBd5ab/Bni3gsVf6u49g8fmw65epJYzMy7sk820m04jq/FxXDs5j1+/upR9B8rCLk1qoVi2EPoBq919jbuXAM8B55frkwPMCJ7PjJ5uZn2AFsDbR1+uSGLqkNmAKT8+lasGtOPJDz9j+J8+5NPi3WGXJbVMLIGQBayPel0YtEVbCIwIng8D0s2smZklAfcBP/+aZT8R7C66w3TStcg3SktJ5lfnnsTjV+Sycedezn3wff6Wt56adC2RxLdYAqGiL+ry/wNvBQaZWT4wCCgCSoEfA2+4+3r+06Xu3h0YGDxGVfjmZteZWZ6Z5RUXF8dQrkjtdmbXFrw55nR6ZDfi5y8u4pbnF7Br34Gwy5JaIJZAKATaRL3OBjZEd3D3De4+3N17AbcHbTuBU4DRZvYZkeMMl5vZ3cH0ouDnLuAZIrum/oO7T3T3XHfPzczMPJx1E6m1Wjaqy9M/7M/PzurMaws38P0H32fh+h1hlyU1XCyBMBfoZGbtzSwVGAm8Gt3BzDKC3UMA44BJAO5+qbsf7+7tiGxFTHb3sWaWYmYZwbx1gO8DS6pkjUQSRHKScdOZnXjh+lMoLXNG/PlDJr73KQdr8DUL7s6Sop3cO72ApRt2hl1Owql0nD93LzWz0cB0IBmY5O5LzWw8kOfurwKDgbvMzIH3gBsrWWwaMD0Ig2TgH8CjR74aIokrt11T3rh5ILe9tIj/fWMF76/eyn0XnUxmes0Zc2Hjzr1Mzd/AlPmFrNocOVg+Y8Vmpt10mkacq0a6uZ1ILeHuPD1nHb+Ztiy4huFkBnaK392su/eX8taSTbycX8iHn27FHfq0bcKwXpFzVv576hLuu+hkRvTJrmRJUhnd7VQkQRVs2sXoZ+azung315/ekZ99tzN1kuPjpgRlB50PVm9hyvxCpi/9gr0HymjT9DiG98pmWK8s2mXUByLDkV7wpw/Ysms/79w6WIMIHSUFgkgC21tSxvhpy3j2k3X0bNOYBy/uRZum9UKrZ8WmL5kyv4ip+UVs3rWf9LopfL9Ha0b0zqJP2yYV3ur7w0+3cMmjcxg7tAs3DOoYQtW1hwJBRHh90UbGTlkEDv87vDvnnty62t578659vLpgAy/NL2L5xi9JSTIGn5jJ8N7ZnNGleUx/9V/95FzmfraN937+bZrUT62GqmsnBYKIALB+2x7GPJfP/HU7GNm3Df9zbg71Uis9n+SI7C0p4+1lm5gyv4jZq4o56HBydiOG9cri3JNb06zB4R3oLti0i6ET3uOqAe254/vl75gjsYo1EI7N/woRiRttmtbj+etP4f6/r+TP737K3M+28dAlvatsqM6DB52P10aGA31rySZ27y+ldaO6/GhwR4b1yuaE5g2OeNkntkznoj5tmPzRZ1xxSjuObxbebq9EoC0EkQTyweot3PL8AnbuPcB/n9OVUUcxVOfqzbv+eVxgw859NEhLYWi3lgzvnc232jclqYpOF/3iy30MumcmZ+W05MGLe1XJMhONdhmJSIW27N7PrX9byKyCYr6b04LfX9iDxvVi2z+/dfd+Xlu4gSn5RSwq3EmSwemdMxnWK4vv5rTkuNRjczbQfW8X8OA7q3nlxgGc3KbxMXmP2kyBICJf6+BBZ9IHa/ndWyvIbJDGH0f2ol/7phX23XegjBnLN/NyfiGzCoopPejktGrI8N5ZnNezNc3T6x7zenftO8Dge2ZxQvMGPHdd/yPeqklUOoYgIl8rKcn44cAO9GvflJufzWfkxI8Yc2ZnRp9xAslJhruT9/l2pswvZNqijezaV0qLhmlcc1p7hvXOokvLqjn+EKv0unUY851O/M8rS5lZsJkzurSo1vdPFNpCEElwu/eXcsfUJbycX0S/9k3p374pLy8oYv22vRxXJ5mh3VoyrHcWp3bMCPU2EgfKDvLd+98jJcl4c8xAUuLkYruaQFsIIhKTBmkp3P+Dnpx2QgZ3vLKEuZ9tY0DHDG45szNDurWkflp8fE3USU7iF987kR89PZ+X5hfyg77Hh11SrRMf/9IiEroRfbL5dpfmlJYdpHnDY39c4EgM6daS3sc35r63V3Luya2P2fUUiUrbXCLyT03rp8ZtGEBkjOlfnt2Vzbv28/jstWGXU+soEESkRslt15TvndSCR979lC2794ddTq2iQBCRGucXQ7qwr/QgD8xYFXYptYoCQURqnI6ZDbi4XxuembOONcW7wy6n1lAgiEiNNObMzqSmJHHP9IKwS6k1FAgiUiNlpqdx/ekdeXPJJuZ9vj3scmoFBYKI1Fg/HNiezPQ07npjOTXpItt4FVMgmNkQMysws9VmNraC6W3NbIaZLTKzWWaWXW56QzMrMrOHotr6mNniYJkPmG5OIiKHqX5aCj89qzN5n29n+tIvwi6nxqs0EMwsGXgYGArkABebWfmRKu4FJrt7D2A8cFe56b8B3i3X9mfgOqBT8Bhy2NWLSMK7qE9kzIXfv7WCA2UHwy6nRotlC6EfsNrd17h7CfAccH65PjnAjOD5zOjpZtYHaAG8HdXWCmjo7h95ZDtvMnDBEa+FiCSslOQkxg7pwpotX/Hc3PVhl1OjxRIIWUD0p1wYtEVbCIwIng95AGSrAAAJMElEQVQD0s2smZklAfcBP69gmYWVLFNEJCZndm1Ov/ZNmfCPlezeXxp2OTVWLIFQ0b798kdvbgUGmVk+MAgoAkqBHwNvuHv52I5lmZGOZteZWZ6Z5RUXF8dQrogkmkO3tNiyu4SJ760Ju5waK5Y7QxUCbaJeZwMboju4+wZgOICZNQBGuPtOMzsFGGhmPwYaAKlmthuYECzna5cZteyJwESI3P46lpUSkcTTs01jzunRikffW8Nl3zo+ru/JFK9i2UKYC3Qys/ZmlgqMBF6N7mBmGcHuIYBxwCQAd7/U3Y9393ZEtiImu/tYd98I7DKz/sHZRZcDr1TNKolIovrF906k9OBB7v+HbmlxJCoNBHcvBUYD04HlwAvuvtTMxpvZeUG3wUCBma0kcgD5zhje+0fAY8Bq4FPgzcMvX0TkX9o2q89l/dvy/Nx1rPpiV9jl1DgaMU1EapVtX5Uw6Pcz+VaHpjx2Rd+wy4kLsY6YpiuVRaRWaVo/lR99uyP/WL6Zj9dsDbucGkWBICK1ztUD2tOqUV3d0uIwKRBEpNapWyeZn57VmYWFO3l98cawy6kxFAgiUisN751Nl5bp/P6tAkpKdUuLWCgQRKRWSk4yxp3dlXXb9vD0nM/DLqdGUCCISK11eqcMTjshgwdmrGLn3gNhlxP3FAgiUmuZGWOHdmH7ngM88u6nYZcT9xQIIlKrdctqxLBeWUx6fy0bduwNu5y4pkAQkVrvZ9/tjDv84e8rwy4lrikQRKTWy25SjysHtOOl+YUs3/hl2OXELQWCiCSEGwefQMO6dbj7zRVhlxK3FAgikhAa1avDTWecwLsri3l/1Zawy4lLCgQRSRijTmlLdpPjuOvN5Rw8qFtalKdAEJGEkZaSzM+/dyJLN3zJKwuLwi4n7igQRCShnNujNd2yGnLv9JXsO1AWdjlxRYEgIgklKcn45dCuFO3Yy+SPPgu7nLiiQBCRhHPqCRkMPjGTh95ZzY49JWGXEzcUCCKSkMYO7cLu/aU8PHN12KXEDQWCiCSkLi0bcmGfbJ768HPWb9sTdjlxIaZAMLMhZlZgZqvNbGwF09ua2QwzW2Rms8wsO6p9npktMLOlZnZD1DyzgmUuCB7Nq261REQq95OzOpOUBPe9XRB2KXGh0kAws2TgYWAokANcbGY55brdC0x29x7AeOCuoH0jcKq79wS+BYw1s9ZR813q7j2Dx+ajXBcRkcPSqtFxXHNae6Yu2MDiwp1hlxO6WLYQ+gGr3X2Nu5cAzwHnl+uTA8wIns88NN3dS9x9f9CeFuP7iYhUm+sHdaRp/VT+V+Mvx/QFnQWsj3pdGLRFWwiMCJ4PA9LNrBmAmbUxs0XBMn7n7hui5nsi2F10h5nZEa2BiMhRaFi3DjefcQIfrdnKrJXFYZcTqlgCoaIv6vIxeiswyMzygUFAEVAK4O7rg11JJwBXmFmLYJ5L3b07MDB4jKrwzc2uM7M8M8srLk7sfywROTYu+VZb2jarx91vrKAsgW9pEUsgFAJtol5nA9F/5ePuG9x9uLv3Am4P2naW7wMsJfLlj7sXBT93Ac8Q2TX1H9x9orvnuntuZmZmTCslInI4UlOS+MX3ulDwxS5eml8YdjmhiSUQ5gKdzKy9maUCI4FXozuYWYaZHVrWOGBS0J5tZscFz5sAA4ACM0sxs4ygvQ7wfWBJVayQiMiROLt7S3q2acwf3l7J3pLEvKVFpYHg7qXAaGA6sBx4wd2Xmtl4Mzsv6DaYyBf9SqAFcGfQ3hWYY2YLgXeBe919MZEDzNODYwsLiOxierTqVktE5PCYGb88uyubvtzHpA/Whl1OKKwmHVXPzc31vLy8sMsQkVrs2sl5fPTpVt79+WCaNUgLu5wqYWbz3D23sn46DVREJMptQ7qw90AZD76TeLe0UCCIiEQ5oXkDftC3DX/9+HM+2/JV2OVUKwWCiEg5t3ynE6kpSdwzPbFuaZESdgEiIvGmeXpdrh3YgQkzVlHwh3crvBiruj1+RV+Ob1bvmL6HAkFEpALXD+rAzr0H2LxrX9ilAJFrJY41BYKISAXqpabw6/NOCruMaqVjCCIiAigQREQkoEAQERFAgSAiIgEFgoiIAAoEEREJKBBERARQIIiISKBG3f7azIqBz8Ou4yhlAFvCLiJO6LP4d/o8/p0+j3852s+irbtXOuRkjQqE2sDM8mK5L3ki0Gfx7/R5/Dt9Hv9SXZ+FdhmJiAigQBARkYACofpNDLuAOKLP4t/p8/h3+jz+pVo+Cx1DEBERQFsIIiISUCBUAzNrY2YzzWy5mS01szFh1xQPzCzZzPLNbFrYtYTNzBqb2YtmtiL4f3JK2DWFxcx+EvyeLDGzZ82sbtg1VSczm2Rmm81sSVRbUzP7u5mtCn42ORbvrUCoHqXAz9y9K9AfuNHMckKuKR6MAZaHXUScmAC85e5dgJNJ0M/FzLKAm4Fcd+8GJAMjw62q2j0JDCnXNhaY4e6dgBnB6yqnQKgG7r7R3ecHz3cR+WXPCreqcJlZNnAO8FjYtYTNzBoCpwOPA7h7ibvvCLeqUKUAx5lZClAP2BByPdXK3d8DtpVrPh94Knj+FHDBsXhvBUI1M7N2QC9gTriVhO6PwC+Ag2EXEgc6AMXAE8EutMfMrH7YRYXB3YuAe4F1wEZgp7u/HW5VcaGFu2+EyB+YQPNj8SYKhGpkZg2Al4Bb3P3LsOsJi5l9H9js7vPCriVOpAC9gT+7ey/gK47RLoF4F+wbPx9oD7QG6pvZZeFWlTgUCNXEzOoQCYOn3X1K2PWEbABwnpl9BjwHnGFmfw23pFAVAoXufmir8UUiAZGIvgOsdfdidz8ATAFODbmmePCFmbUCCH5uPhZvokCoBmZmRPYPL3f3P4RdT9jcfZy7Z7t7OyIHDN9x94T9K9DdNwHrzezEoOlMYFmIJYVpHdDfzOoFvzdnkqAH2Mt5FbgieH4F8MqxeJOUY7FQ+Q8DgFHAYjNbELT90t3fCLEmiS83AU+bWSqwBrgq5HpC4e5zzOxFYD6Rs/PySbArls3sWWAwkGFmhcCvgLuBF8zsGiKhedExeW9dqSwiIqBdRiIiElAgiIgIoEAQEZGAAkFERAAFgoiIBBQIIiICKBBERCSgQBAREQD+P+V1T+z7+kY9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "k_vs_accuracy=[]\n",
    "\n",
    "for k in range(1,11):\n",
    "    knn_fold_accuracies=cross_validate(k)\n",
    "    k_vs_accuracy.append(np.mean(knn_fold_accuracies))\n",
    "    \n",
    "plt.plot(range(1,11),k_vs_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Neural Network With One Hidden Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a few downsides to using k-nearest neighbors:\n",
    "\n",
    "* high memory usage (for each new unseen observation, many comparisons need to be made to seen observations)\n",
    "* no model representation to debug and explore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * Use the [MLPClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html) package from scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO:\n",
    "* Using 4-fold cross validation:\n",
    "\n",
    "  * Train and test a neural network model using a single hidden layer of 8 neurons.\n",
    "  * Train and test a neural network model using a single hidden layer of 16 neurons.\n",
    "  * Train and test a neural network model using a single hidden layer of 32 neurons.\n",
    "  * Train and test a neural network model using a single hidden layer of 64 neurons.\n",
    "  * Train and test a neural network model using a single hidden layer of 128 neurons.\n",
    "  * Train and test a neural network model using a single hidden layer of 256 neurons.\n",
    "* Create a new Markdown cell summarizing what you saw.\n",
    "\n",
    "  * Rank the performance of the models by the overall accuracies.\n",
    "  * Which models, if any, start to overfit?\n",
    "    * For each model, you could compute and visualize the performance for both the train and test sets to understand how they diverged. Models that overfit tend to perform well on the train set but poorly on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "# 50% Train / test validation\n",
    "def train_nn(neuron_arch, train_features, train_labels):\n",
    "    mlp = MLPClassifier(hidden_layer_sizes=neuron_arch)\n",
    "    mlp.fit(train_features, train_labels)\n",
    "    return mlp\n",
    "\n",
    "def test_nn(model, test_features, test_labels):\n",
    "    predictions = model.predict(test_features)\n",
    "    train_test_df = pd.DataFrame()\n",
    "    train_test_df['correct_label'] = test_labels\n",
    "    train_test_df['predicted_label'] = predictions\n",
    "    overall_accuracy = sum(train_test_df[\"predicted_label\"] == train_test_df[\"correct_label\"])/len(train_test_df)    \n",
    "    return overall_accuracy\n",
    "\n",
    "def cross_validate_nn(neuron_arch):\n",
    "    fold_accuracies = []\n",
    "    kf = KFold(n_splits = 4, random_state=2)\n",
    "    for train_index, test_index in kf.split(features):\n",
    "        train_features, test_features = features.loc[train_index], features.loc[test_index]\n",
    "        train_labels, test_labels = labels.loc[train_index], labels.loc[test_index]\n",
    "       \n",
    "        model = train_nn(neuron_arch, train_features, train_labels)\n",
    "        overall_accuracy = test_nn(model, test_features, test_labels)\n",
    "        fold_accuracies.append(overall_accuracy)\n",
    "    return fold_accuracies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.8992687453600594,\n",
       " 0.9254355357584756,\n",
       " 0.9259824300915614,\n",
       " 0.9410034644889878,\n",
       " 0.948236822568671]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neuron_arch=[8,16,32,64,128]\n",
    "neuron_vs_accuracy=[]\n",
    "\n",
    "for neuron in neuron_arch:\n",
    "    fold_accuracies=cross_validate_nn(neuron)\n",
    "    neuron_vs_accuracy.append(np.mean(fold_accuracies))\n",
    "    \n",
    "neuron_vs_accuracy    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1e3f48a1c18>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAEICAYAAAByPazKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucXHV9//HXe69JdjcJue1CEghyCRfFAClorUqFIlArAqKgCHjDemmr1T4E688L1mL706pttT5QqQgqYryUnyKoVGu9k4giEYIhXLKE3SQEsptN9v75/XHObE4me5lNNpk5O+/n4zGPnTnnzMx3vntm3vP9nu/5jiICMzMzq3w15S6AmZmZlcahbWZmlhMObTMzs5xwaJuZmeWEQ9vMzCwnHNpmZmY54dA2s9yQ9F1JV0zB4yyTFJLqpqJcozz+eyR9bpz1j0g6a4x1Z0hqPxDlsvxzaOdI+kbvl7SgaPlv0g+gZWUq15GShiV9uhzPn1fph3NI+lTR8p9IurJMxdpv+xs6aeA9LGmHpHZJXy2si4hzI+LGqSnpPpfvA5JuHmV5SDoaICL+MSLecPBLN75sGS2fHNr58zBwaeGGpGcBM8tXHAAuB54CLpHUeDCf+EC1lA6iHuDyg/GFKw91lbaiXwOcFRHNwErgrvKWyqZKHvbBSufQzp+bSEKy4Argi9kNJDVK+qikxyR1SvqMpJnpukMkfVvSFklPpdeXZO77I0kfkvRTSd2Svlfcsh/F5cB7gQHgL4rKcqKk70valpblPeny2rRF9VD6PGskLR2t2zIt0xvS61emZfu4pG3AByQdJem/JT0paaukL0mam7n/UknfSF/zk5L+Pa2jbemXnsJ2iyTtkrRwlPp8WtIzM8sWptsukrQgrcen08f8X0mlvreeBr4AvH+sDSS9TtL96f/rTklHpMv3pa5qJL1X0qOSNkv6oqQ5RY93RbrvbJX095nHPk3Sakld6f/yX0p5gZPcp/4IuDMiHgKIiI6IuH6c1/eTdF9/Km2dn5vZ9khJP06f8weSPqVRWsjptnMkfV7SE5Iel/QPkmpLeX1jPN4erXFJr0nr/MlsnabrZkr6Qvoafp/WQXb9YZK+nu6/D0v666LnuTX9P3ZLWitp5T6Ud8z3kKS/k/T1ou3/TdIn0utj1t1o++Bky2Z7cmjnzy+A2ZKOT98YrwSKP4j+CTgWWAEcDSwG3peuqwH+EzgCOBzYBfx70f1fBbwWWAQ0AO8aqzCSng8sAW4BbiXzhUJSC/AD4A7gsLQshVbT35L0GJwHzAZeB+ws4fUDnA5sSMv3YUDAdelzHA8sJf1wSOvo28CjwDKSurglIvrSMl+WedxLgR9ExJbsk6XbfoNMDwfwCuB/ImIz8E6gHVgItALvASYzP/CHgYskLS9eIell6eNdmD7+/wJfmcRjF9fVlenlT4FnAM3s/f//E2A5cCbwPknHp8s/CXwyImYDR5H8v0tV6j71C5Keh7+TtLKE4DwdWAcsAP4Z+Lwkpeu+DPwKmE+yP7xmnMe5ERgk2UdPBs4GpqR7W9IJwH+kz39YWp4lmU3eT1KfRwEvJvkiXrhvDfD/gN+S7LtnAm+X9OLM/V9Ksi/PBW5j7/9nScVkjPcQyefLOZkQryP53LkpXT9R3RXvg7Y/IsKXnFyAR4CzSFq11wHnAN8H6khCYhnJm68HOCpzv+cCD4/xmCuApzK3fwS8N3P7LcAd45Tpc8C3Ms8zACxKb18K3DPG/dYB54+yfFn6WuqKyvSG9PqVwGMT1NPLCs+blmlL9vEy250ObARq0turgVeM8ZhnARsyt38KXJ5evxb4L+DoSf4/zwDa0+v/DHw1vf4T4Mr0+neB12fuU0Py5eaIfakrki9Nb8ncXp7+z+oyj7cks/5XwCXp9R8DHwQWlPq69nGfejXJl70e4Eng6nFe3/rMullp+dtIvpAOArMy628Gbi7ez0i+aPUBMzPbXgr8cIzyfQDoJ+klyV6isA+k2xSe630kXxQL929K739WensDcE5m/VWZ/eL0Uf6H1wD/mXmeH2TWnQDsGqduR8pY6nsosx++Mb3+EuD36fVx6260fdCX/bv4+EI+3UTyAXokRV3jJK2xWcCa3Q0OBBS6q2YBHycJ/EPS9S2SaiNiKL3dkXm8nSStsb0o6XK/mPRbdUT8XNJjJK2qT5B8W39ojNcw3rqJbCwqxyLgX4HnAy0kwfZU5nkejYjB4geJiF9K6gFeKOkJkpbCbWM8538DMyWdTlI/K4Bvpuv+L8mH5/fSOr8+Ij4yydf0T8BDkp5dtPwI4JOSPpZZJpJW16YSHndj0e3DSHodCh5ld3AVjPX/fz3JF5QHJD0MfDAivl1CGcZ7zL1ExJeAL0mqJwmPL0m6JyLuHO9xI2JnWv/NJC3vbRGR7b3ZSLI/FDsCqAeeyLxnati77rJujYhsLw2SxupdOSz7WBHRI+nJsdaz5//nCOAwSU9nltWS9LgUFNftDEl1o+3zY5ngPQRJa/rNwGdJeqcKrexS6m68erRJcvd4DkXEoyQD0s4j6bbN2krS5X1iRMxNL3MiGdQDSVfucuD0SLo5X5AuF5N3AUnX9qcldUjqIAmTQhf5RpIuv9GMta4n/Tsrs6ytaJviD8fr0mUnpa/pMna/no3A4Rp7AMyN6favAVZFRO9oG0XEMEl38KUkX0q+HRHd6bruiHhnRDyD5Jj+30o6c4znG1VEPEnyRedDRas2Am/K/C/nRsTMiPgZ+1ZXm0g+aAsKLdLOEsr4h4i4lKSb85+AVZKaJrrfvoqIgYj4GnAv8MyJti/yBDAv/ZJaMFpgQ1LHfSQ9CIU6nh0RJ06+1GOWZeS50zLNH2s9yf8kW7aHi/7/LRFx3hSVrWC89xDAt4CTlIzreAnwpUz5Jqo7/5TkFHJo59frgRdFRE92YRounwU+nn57RtLizDGwFpJQf1rSPMYZAFWCK4AbgGeRtDxXAM8DVigZ4PVtoE3S25UM5mpJW6qQdKt/SNIxSpwkaX4kx5MfBy5TMljtdYwd/AUtwI70NS0G/i6z7lckH4ofkdQkaYak52XW30Ty5eMy9u61KPZlkmN5r06vAyDpJZKOTo+ldgFD6WWy/gX4Y5JjigWfAa6RdGL6XHMkXQywj3X1FeAdSgZpNQP/SNItP2GrTNJlkham+1ih5bcvr3O857hS0p+n+0qNkoFlJwK/nMzjpF9sV5MMvmuQ9FyKBklmtn0C+B7wMUmz0+c9StIL9/PlFKwCXiLpTyQ1kPRWZD97byX5Hx+iZFDoX2XW/QrokvRuJQPWaiU9U9Ieg9UmqSF9HxQutYz/HiL9MruKdJxARDyWLj/QdWdFHNo5FREPRcTqMVa/G1gP/EJSF8nxwcIgp0+QnCK2lWTQzx378vzpG/tM4BORjPAtXNakj3lF2hL9M5IPyw7gDyQDoCAJqFtJ3vBdwOfZferaG0k+NJ4k+cD+2QTF+SBwCrAd+A6Z3oe0y/8vSLq+HyMZMPbKzPp24NckrYFsl+NeIuKXJK3bw0iO8RUcQ1LHO4CfA5+OiB+l9fRdpSPmJxIRXSTHtudlln2TpFV7S/q/vA84N3O3ydbVDew+vPIw0MueITGec4C1knaQDEq7ZKyeif3QRTLw7jGSLwb/DLw5In6yD4/1apIxDU8C/wB8laRVOJrLSQbI/Z6kW3gVcOg+POdeImIt8FaSwHsiffzseewfJOkSf5jk/XBT5r6F/XdFun4ryRfeOftRpLUkX9wLl9cyznso40aSL+g3FS0/YHVne1OEey6sukm6AdgUEe8td1nswFEyScsDEbE/vUtVS9LhwANAW/oF08rAA9GsqimZ1ORCklNVbBpJu5C3kbRQzwbOByY7QNAYOfXsb0lGwTuwy8ihbVVL0oeAdwDXRcTD5S6PTbk2km7e+STd0W+OiHvKW6T8SQcbdpJ04Z9T5uJUPXePm5mZ5YQHopmZmeVExXWPL1iwIJYtW1buYpiZmR00a9as2RoRCyfaruJCe9myZaxePdaZTGZmZtOPpEcn3srd42ZmZrnh0DYzM8sJh7aZmVlOOLTNzMxywqFtZmaWEw5tMzOznHBom5mZ5UTFnadtZmZWDhFB3+AwPX2D7Chcegfp6R+ku3eQnr4hdvQNsKNviPNXHMZRC5sPehkd2mZmlmt9g0NJoPbuDtuevkG6C6GbXu9Jb+/o3718R9+e9xkYmvj3OCQ48bDZDm0zM6sO/YPDu8M1bc3uFbpjBGt2+56+IfqHhkt6zubGOpoaa2lurEsuM+qY1zSL5hl1I8uaGutomVFHU0Pd6Msb65hVX0tNjQ5wDY3OoW1mZiXpL+o63qsFm1m+Y6/QTbqWCy3iUoO2qaGWpsY9A3TpvFm0pEFaHKzZQM5eL2fQTiWHtpnZNDYwNFxCa3b3sdodY4Rwd98g/YOlBe2shj1bs00NdSw5pIHmxpai1mwtzTPqaW6spbmxnqbG2pHWbHNjcr/pELRTyaFtZlZhBoeG6ekborvQMi0EavHx2exgqeLlaej2lRi0M+tri1qttRw2d2YSqGmQtkzQmm1Kg7bWQXvAOLTNzKbA0HDsEaJ7dBWP0modq0t5R98gvQOlB+3uY61J6/bQOTN2dxXPqKO5oW7v0C3qUm5qqKWu1mcA54FD28yqViFox221jjvaeGjkuO2ugaGSnnNGfc1ex2DbZs8YtzU7WuvWQVudHNpmlitDw0FP/xgt2Eyw7tmazQyCyoRzqUHbWFez57HWxjpaW2bQtGCUEcZ7tGYzx2rTvw5a2x8ObTM74IbToB1z8FPvAD39Q3uf4jPK4Kmd/aUFbUNdzV6t1oUtjSxb0JTeHn3wU3bwVGF5vYPWKoRD28xGNTwc7BwYZcKKMc6dzW5X3NXcU2rQ1tbs7v5NW63zmxo4Yn5TGrKjdx0Xh3NTYx0NdQ5am34c2mZVoG9wiIc297Cus4tNT/dO2JotdCOXor5We7VO5zU1cPi8WaNOTDFWa7apsZbGutoDXBNm+ebQNptGhoeDx5/exQMd3azr6Er/dvPw1h4Gh3dPz1hXoz1P12ms45CmBpZkJ62YoDXbPMNBa3awObTNcuqpnv6RcF7X2c0DHd082NG9R1f0kkNmclxbC2ef2Mryttkc19bC4fNm0VhXg+Rzac3yxqFtVuF6B4ZYv3nHXq3nzd19I9vMnVXP8tYWXn7qEpa3zWZ5WwvHtjbTMqO+jCU3s6nm0DarEMPDwWPbdo6E8rrOJKAf2dpDoWe7oa6GYxY18yfHLOC4tpaR1vOilka3nM2qgEPbrAy27uhjXUf37u7tjm4e7Nwxct6wBIfPm8Xy1hZe8qxDR1rPy+bP8nm+ZlXMoW12AO3qH+LBzu7dAd2ZBPTWHf0j28xvamB5WwuXnLZ0pPV8bGszsxr89jSzPflTwWwKDA0HjzzZs1fr+dFtO4m0a3tGfQ3Htrbwp8sXsbythePS1vPClsbyFt7McsOhbTYJEcGW7r6R486F1vMfOneM/JpSjWDZ/CaOP3Q2Lzt58Ujr+fB5s/zrR2a2XxzaZmPo6RtkXdq1nQR00np+aufAyDYLWxo5rq2F1zzniJHW8zGtzcyo97nLZjb1HNpW9QaHhnl4a89ereeN23aNbDOroZZjW1t48YltLG9rGQnoeU0NZSy5mVUbh7ZVjYigo6t39ylVaUA/tHkH/UNJ13ZtjThyQRMnLZnLK05dOhLOSw6ZSY27ts2szBzaNi119Q7w4MigsN3d2129u+fTbps9g+VtLbzgmAUjreejFzV7Wk4zq1gObcu1/sFhNmzdkRm1nVwef3p313ZLYx3HtrXwkmcflgwKa01az3NmebYwM8sXh7blQkTyQxjF4fzQlh0jP4RRXyuOWtjMqUccwqtOPzwdtd3C4rkzPVuYmU0LDm2rONt3DiQjtTt3B/SDHd10Z34qcvHc5Icwzjx+9znPRy5o8m8om9m05tC2sukbTH4IIzsobF1HNx1dvSPbzJlZz/K2Fi44ZXEazi0c29riH8Iws6rk0LYDbng4aH9q18h5zg907v6N56G0a7uhtoajFzXzx0fN3+OUqtbZ/iEMM7MCh7ZNqW09/SPhXGg9P9jZzc7MbzwfPm8Wy9taOPeZbSOt52Xzm/xDGGZmE3Bo237r6h3gO/c+wao17ax59KmR5fOaGlje2sIrVi4dGRR2bGsLTY3e7czM9kVJn56SzgE+CdQCn4uIjxStPwK4AVgIbAMui4j2zPrZwP3ANyPibVNUdiujoeHgZw9tZdWadu64r4O+wWGOXtTMu84+lmcvnZv8EEazu7bNzKbShKEtqRb4FPBnQDtwt6TbIuL3mc0+CnwxIm6U9CLgOuA1mfUfAv5n6opt5bJhyw6+/ut2vvHrx3liey+zZ9Rx8colvPzUpTx7yRyHtJnZAVRKS/s0YH1EbACQdAtwPpAN7ROAd6TXfwh8q7BC0qlAK3AHsHIKymwHWVfvALen3d+rH32KGsELjl3I3//58Zx1fKt/HMPM7CApJbQXAxszt9uB04u2+S1wEUkX+gVAi6T5wFPAx0ha3WeO9QSSrgKuAjj88MNLLbsdQEPDwc8fepJVazZyx9oOegeS7u+rzz2OC05eTOvsGeUuoplZ1SkltEfr74yi2+8C/l3SlcCPgceBQeAtwO0RsXG8btOIuB64HmDlypXFj20H0cNbe/j6mna+8et2NqXd3y8/1d3fZmaVoJTQbgeWZm4vATZlN4iITcCFAJKagYsiYruk5wLPl/QWoBlokLQjIq6ektLblOjOjP7Odn+/x93fZmYVpZTQvhs4RtKRJC3oS4BXZTeQtADYFhHDwDUkI8mJiFdntrkSWOnArgyjdX8ftbDJ3d9mZhVswtCOiEFJbwPuJDnl64aIWCvpWmB1RNwGnAFcJylIusffegDLbPvB3d9mZvmliMo6hLxy5cpYvXp1uYsxrYzV/f3yU5e4+9vMrAJIWhMRE55h5amppqnh4eBno3R/v/uc47jwFHd/m5nlkUN7mnlkaw+rMt3fLTPquOiUJbz81CWsWDrX3d9mZjnm0J4GunsHuP13Sff33Y8k3d/PP2Yh15x3PH92gru/zcymC4d2Tg0PBz/f8CSr1rTz3fue2KP7+4KTF9M2x93fZmbTjUM7Zx7Z2sPXf93O19e4+9vMrNo4tHPA3d9mZgYO7Yrl7m8zMyvm0K4whe7vb/z6cR5/epe7v83MbIRDuwKM1f199bnHufvbzMxGOLTL6L7Ht/P5nzw80v39DHd/m5nZOBzaZfLNe9p596rf0Vhfw4WnLOFid3+bmdkEHNoH2fBw8PEfPMi//fd6nvOMeXzmslOZO6uh3MUyM7MccGgfRL0DQ7zza7/lO/c+wStWLuEfXvYsGupqyl0sMzPLCYf2QbK5u5c3fnEN97Y/zXvOO443Pv8Z7go3M7NJcWgfBL/f1MUbbrybp3YO8JnLTuXFJ7aVu0hmZpZDDu0D7K77O/nrr9xDy4x6vvaXz+WZi+eUu0hmZpZTDu0DJCL4/E8e5sO3388zD5vD565Y6d+wNjOz/eLQPgAGhoZ533+t5Su/eoxzTmzj469cwcwGT5BiZmb7x6E9xbbvHOAtX17DT9c/yVvOOIp3nb2cmhoPODMzs/3n0J5Cj2zt4XU33s3GbTv56MXP5uWnLil3kczMbBpxaE+RX254kjfdvAaAm19/Oqc/Y36ZS2RmZtONQ3sKrFrTzjXfuJel82ZxwxV/xLIFTeUukpmZTUMO7f0wPBx89Hvr+PSPHuJ5R8/n0686lTmz6stdLDMzm6Yc2vtoV/8Q7/jqb7hjbQevOv1wPvjSE6mv9ZSkZmZ24Di090FnVy9vuHE1923azv95yQm87nnLPCWpmZkdcA7tSbrv8e284cbVdPcO8LnLV3Lm8a3lLpKZmVUJh/YkfG9tB39zy284ZFY9q978xxx/6OxyF8nMzKqIQ7sEEcFn/3cD1333AU5aMpfPXn4qi1o8JamZmR1cDu0J9A8O83++dR9fXb2RPz/pUD528bOZUe8pSc3M7OBzaI/j6Z39/OXNa/jFhm389YuO5u1nHespSc3MrGwc2mPYsGUHr79xNY8/tYtPvHIFLzt5cbmLZGZmVc6hPYqfP/Qkf3nzGupqxJffeDorl80rd5HMzMwc2qN5+1fvYX5TAze+7jSWzptV7uKYmZkB4Cm8ivQODNHZ1ceFpyx2YJuZWUVxaBfZ3NUHQOtsn9JlZmaVpaTQlnSOpHWS1ku6epT1R0i6S9K9kn4kaUm6fIWkn0tam6575VS/gKnW0dULOLTNzKzyTBjakmqBTwHnAicAl0o6oWizjwJfjIiTgGuB69LlO4HLI+JE4BzgE5LmTlXhD4TONLTb5ji0zcysspTS0j4NWB8RGyKiH7gFOL9omxOAu9LrPyysj4gHI+IP6fVNwGZg4VQU/EAphHarZzwzM7MKU0poLwY2Zm63p8uyfgtclF6/AGiRND+7gaTTgAbgoeInkHSVpNWSVm/ZsqXUsh8QnV29zKivYfZMD6w3M7PKUkpojzYFWBTdfhfwQkn3AC8EHgcGRx5AOhS4CXhtRAzv9WAR10fEyohYuXBheRviHV19tM6e4Z/aNDOzilNKc7IdWJq5vQTYlN0g7fq+EEBSM3BRRGxPb88GvgO8NyJ+MRWFPpA6u3o9CM3MzCpSKS3tu4FjJB0pqQG4BLgtu4GkBZIKj3UNcEO6vAH4Jskgta9NXbEPHIe2mZlVqglDOyIGgbcBdwL3A7dGxFpJ10p6abrZGcA6SQ8CrcCH0+WvAF4AXCnpN+llxVS/iKkSEXR29dI2u7HcRTEzM9tLSaOtIuJ24PaiZe/LXF8FrBrlfjcDN+9nGQ+arl2D9A4Mu6VtZmYVyTOiZXR2e2IVMzOrXA7tjI7tDm0zM6tcDu2MwhSmbQ5tMzOrQA7tjM1paC/yQDQzM6tADu2Mjq5e5s6qZ0Z9bbmLYmZmtheHdkZnV5/nHDczs4rl0M7o7Oql1b/uZWZmFcqhneGJVczMrJI5tFODQ8Ns6e7z6V5mZlaxHNqpJ3v6GQ6fo21mZpXLoZ3yxCpmZlbpHNqpTk+sYmZmFc6hnSqEdqsHopmZWYVyaKc6u/qorRHzmx3aZmZWmRzaqY6uXhY2N1Jbo3IXxczMbFQO7ZQnVjEzs0rn0E51dvXS2uKucTMzq1wO7VRnVx9tbmmbmVkFc2gDvQNDbN814HO0zcysojm0yZ7u5dA2M7PK5dAmOxuaj2mbmVnlcmgDnd19gGdDMzOzyubQBjrTlvYih7aZmVUwhzbJMe2Z9bXMnlFX7qKYmZmNyaFNMhta6+xGJM+GZmZmlcuhDWzu6vPIcTMzq3gObQotbYe2mZlVtqoP7Yigs6vXs6GZmVnFq/rQ3r5rgL7BYRZ53nEzM6twVR/anV3pOdpuaZuZWYWr+tDu8BSmZmaWE1Uf2oV5xz0bmpmZVTqHdjob2kIf0zYzswrn0O7u5ZBZ9cyory13UczMzMZV9aHdsd0Tq5iZWT6UFNqSzpG0TtJ6SVePsv4ISXdJulfSjyQtyay7QtIf0ssVU1n4qbC52xOrmJlZPkwY2pJqgU8B5wInAJdKOqFos48CX4yIk4BrgevS+84D3g+cDpwGvF/SIVNX/P3Xsb3Xv6NtZma5UEpL+zRgfURsiIh+4Bbg/KJtTgDuSq//MLP+xcD3I2JbRDwFfB84Z/+LPTUGh4bZuqPPI8fNzCwXSgntxcDGzO32dFnWb4GL0usXAC2S5pd4XyRdJWm1pNVbtmwptez7beuOfobDv6NtZmb5UEpoj/Z7lVF0+13ACyXdA7wQeBwYLPG+RMT1EbEyIlYuXLiwhCJNDZ+jbWZmeVJXwjbtwNLM7SXApuwGEbEJuBBAUjNwUURsl9QOnFF03x/tR3mnlGdDMzOzPCmlpX03cIykIyU1AJcAt2U3kLRAUuGxrgFuSK/fCZwt6ZB0ANrZ6bKKsLkQ2nM8EM3MzCrfhKEdEYPA20jC9n7g1ohYK+laSS9NNzsDWCfpQaAV+HB6323Ah0iC/27g2nRZRejo6qW2RsxvcmibmVnlK6V7nIi4Hbi9aNn7MtdXAavGuO8N7G55V5TOrj4WtTRSWzPaoXczM7PKUtUzonV29XrkuJmZ5UbVh3abJ1YxM7OcqOrQTmZDc0vbzMzyoWpDe1f/EF29gw5tMzPLjaoN7U6fo21mZjlT9aHt2dDMzCwvqja0d8+G5oFoZmaWD1Ub2pu7+gBoneOWtpmZ5UPVhnZHVy8z62tpaSxpfhkzM7Oyq9rQ7uzqpW3ODCTPhmZmZvlQ1aG9qMXHs83MLD+qOLT7aPPxbDMzy5GqDO2IoKPLs6GZmVm+VGVob981QP/gsEPbzMxypSpD2+dom5lZHlVlaHem52h7NjQzM8uT6gzt7Z533MzM8qc6QzvtHl/k7nEzM8uRqgztjq5eDplVT2NdbbmLYmZmVrKqDO3Orj53jZuZWe5UaWj7HG0zM8ufqg1tjxw3M7O8qbrQHhwaZuuOPp+jbWZmuVN1ob11Rz/D4d/RNjOz/Km60B6ZDa3FoW1mZvlSdaFdOEfbv/BlZmZ5U7Wh7YlVzMwsb6oytGtrxIImh7aZmeVL1YV2x/Y+FrU0UlOjchfFzMxsUqoutDd3e2IVMzPLp6oL7Y7tvT5H28zMcqnqQtuzoZmZWV5VVWjv6h+iq3eQRQ5tMzPLoaoK7ZFztB3aZmaWQ1UV2iOzoTm0zcwsh0oKbUnnSFonab2kq0dZf7ikH0q6R9K9ks5Ll9dLulHS7yTdL+maqX4Bk7F7NjQPRDMzs/yZMLQl1QKfAs4FTgAulXRC0WbvBW6NiJOBS4BPp8svBhoj4lnAqcCbJC2bmqJP3u7Z0NzSNjOz/CmlpX0asD4iNkREP3ALcH7RNgHMTq/PATZlljdJqgNmAv1A136Xeh91dvUxq6GWlsa6chXBzMxsn5US2ouBjZnb7emyrA8Al0lqB24H/ipdvgroAZ4AHgM+GhHb9qfA+6OjK5lYRfJsaGZmlj+lhPZoCRdFty8FvhARS4DzgJsk1ZC00oeAw4AjgXeE/tcHAAAGmUlEQVRKesZeTyBdJWm1pNVbtmyZ1AuYjM1dnljFzMzyq5TQbgeWZm4vYXf3d8HrgVsBIuLnwAxgAfAq4I6IGIiIzcBPgZXFTxAR10fEyohYuXDhwsm/ihIVWtpmZmZ5VEpo3w0cI+lISQ0kA81uK9rmMeBMAEnHk4T2lnT5i5RoAp4DPDBVhZ+MiKCzq8/naJuZWW5NGNoRMQi8DbgTuJ9klPhaSddKemm62TuBN0r6LfAV4MqICJJR583AfSTh/58Rce8BeB0TenrnAP2Dwx45bmZmuVXSMOqIuJ1kgFl22fsy138PPG+U++0gOe2r7Dq7PRuamZnlW9XMiNaxvTAbmgeimZlZPlVNaG/u6gM8hamZmeVX1YR2x8hsaG5pm5lZPlVNaHd29TKvqYHGutpyF8XMzGyfVFVoL2pxK9vMzPKrikK7j7Y5Pp5tZmb5VTWh3dHVS2uLQ9vMzPKrKkJ7cGiYrTv6aHVL28zMcqwqQnvLjj4ifI62mZnlW1WEdmd6jrZnQzMzszyritDePRuaQ9vMzPKrKkJ7c7dD28zM8q8qQrtjey91NWJ+U0O5i2JmZrbPqiK0O7v6WNTSSE2Nyl0UMzOzfVYlod3r39E2M7Pcq5rQ9shxMzPLu6oI7Y6uXp+jbWZmuTftQ3tn/yDdvYOeDc3MzHJv2od2YWIVzztuZmZ5VwWhnZyj7V/4MjOzvKua0PYxbTMzy7sqCm23tM3MLN+mfWh3bO9jVkMtzY115S6KmZnZfpn2od3ZnZyjLXk2NDMzy7fpH9rbe1nk49lmZjYNTP/Q7vZsaGZmNj1M69COCDq7+jwIzczMpoVpHdp9g8Oc+8w2ViydW+6imJmZ7bdpPaR6Rn0tn7zk5HIXw8zMbEpM65a2mZnZdOLQNjMzywmHtpmZWU44tM3MzHLCoW1mZpYTDm0zM7OccGibmZnlhEPbzMwsJxQR5S7DHiRtAR4dZ5MFwNaDVJzpynU4NVyPU8P1ODVcj1OjXPV4REQsnGijigvtiUhaHREry12OPHMdTg3X49RwPU4N1+PUqPR6dPe4mZlZTji0zczMciKPoX19uQswDbgOp4brcWq4HqeG63FqVHQ95u6YtpmZWbXKY0vbzMysKjm0zczMciI3oS3pHEnrJK2XdHW5y5Mnkh6R9DtJv5G0Ol02T9L3Jf0h/XtIuctZaSTdIGmzpPsyy0atNyX+Nd0/75V0SvlKXlnGqMcPSHo83Sd/I+m8zLpr0npcJ+nF5Sl1ZZG0VNIPJd0vaa2kv0mXe3+chHHqMTf7Yy5CW1It8CngXOAE4FJJJ5S3VLnzpxGxInP+4dXAXRFxDHBXetv29AXgnKJlY9XbucAx6eUq4D8OUhnz4AvsXY8AH0/3yRURcTtA+r6+BDgxvc+n0/d/tRsE3hkRxwPPAd6a1pX3x8kZqx4hJ/tjLkIbOA1YHxEbIqIfuAU4v8xlyrvzgRvT6zcCLytjWSpSRPwY2Fa0eKx6Ox/4YiR+AcyVdOjBKWllG6Mex3I+cEtE9EXEw8B6kvd/VYuIJyLi1+n1buB+YDHeHydlnHocS8Xtj3kJ7cXAxsztdsavaNtTAN+TtEbSVemy1oh4ApIdGVhUttLly1j15n108t6Wdt3ekDk843qcgKRlwMnAL/H+uM+K6hFysj/mJbQ1yjKfq1a650XEKSRdZm+V9IJyF2ga8j46Of8BHAWsAJ4APpYudz2OQ1Iz8HXg7RHRNd6moyxzPaZGqcfc7I95Ce12YGnm9hJgU5nKkjsRsSn9uxn4Jkn3Tmehuyz9u7l8JcyVserN++gkRERnRAxFxDDwWXZ3OboexyCpniRovhQR30gXe3+cpNHqMU/7Y15C+27gGElHSmogGRhwW5nLlAuSmiS1FK4DZwP3kdTfFelmVwD/VZ4S5s5Y9XYbcHk6avc5wPZCt6Xtrej46gUk+yQk9XiJpEZJR5IMpPrVwS5fpZEk4PPA/RHxL5lV3h8nYax6zNP+WFfOJy9VRAxKehtwJ1AL3BARa8tcrLxoBb6Z7KvUAV+OiDsk3Q3cKun1wGPAxWUsY0WS9BXgDGCBpHbg/cBHGL3ebgfOIxmoshN47UEvcIUaox7PkLSCpKvxEeBNABGxVtKtwO9JRvq+NSKGylHuCvM84DXA7yT9Jl32Hrw/TtZY9XhpXvZHT2NqZmaWE3npHjczM6t6Dm0zM7OccGibmZnlhEPbzMwsJxzaZmZmOeHQNjMzywmHtpmZWU78f7kzBAtdbuJLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nn_one_neurons = [\n",
    "    (8,),\n",
    "    (16,),\n",
    "    (32,),\n",
    "    (64,),\n",
    "    (128,),\n",
    "    (256,)\n",
    "]\n",
    "nn_one_accuracies = []\n",
    "\n",
    "for n in nn_one_neurons:\n",
    "    nn_accuracies = cross_validate_nn(n)\n",
    "    nn_mean_accuracy = np.mean(nn_accuracies)\n",
    "    nn_one_accuracies.append(nn_mean_accuracy)\n",
    "\n",
    "plt.figure(figsize=(8,4))\n",
    "plt.title(\"Mean Accuracy vs. Neurons In Single Hidden Layer\")\n",
    "\n",
    "x = [i[0] for i in nn_one_neurons]\n",
    "plt.plot(x, nn_one_accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "It looks like adding more neurons to the single hidden layer helped massively improved simple accuracy from approximately 86% to approximately 94%. Simple accuracy computes the number of correct classifications the model made, but doesn't tell us anything about false or true positives or false or true negatives.\n",
    "\n",
    "Given that k-nearest neighbors achieved approximately 96% accuracy, there doesn't seem to be any advantages to using a single hidden layer neural network for this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Neural Network With Two Hidden Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1e3f40f7278>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAEICAYAAABLWh2RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8FuWd///XOwkEMAgKiMpRESt4Qk09tFXcqq26LVY8rMfqaldtf363lbKLWmtbrdp66narPdiWbu22taBbpepWtwLa2nrAlgCiYAQ1gAIFQc4hyef3x0z05iaBGxIyuZP38/G4H5nDNTOf656ZfOa6Zu77VkRgZmZmxask6wDMzMysZZzMzczMipyTuZmZWZFzMjczMytyTuZmZmZFzsnczMysyDmZm5m1EkmvSzqumXmnSqrexrIPSLph10VnHZmTeQck6Q1JtZL65k2fKSkkDc0orv0kNUj6fhbbL1aSTkz327150/8k6dKMwmqxtF6LdmK5CyWtTV8b0mOqcXztrog13e47kj6WN+0qSX9oHI+IYRHxl10Vw87Ij9E6JifzjmshcH7jiKRDge7ZhQPAZ4F3gfMklbflhiWVteX2doF1wGfb4kKsvb9XEfHLiKiIiArgNGBJ43g6zdoRSSWSnGt2Mb/BHdcvSJJno0uA+3MLSCqXdKektyQtlfRDSd3TeXtIelTScknvpsMDc5adLulmSc9KWiPpyfyegCZ8FrgB2Ax8Oi+WgyX9n6SVaSzXp9NLJV2fdl+ukfSSpEGShqat1bKcdUyX9Ll0+NI0tu9IWgl8XdIwSVMlrZD0d0m/lNQ7Z/lBkv4nrfMKSfek79HK9GKosdxeaYuwXxPv5ypJh+RM65eW3UtS3/R9XJWu84878E9uFfBfwNeaKyDpMkmvpPvrCUlD0uk7816VSLpB0puSlkm6X1KvvPVdkh47f5f0lZx1Hy1phqT30n15dyEV3Mljqqn1fF7S5JzxGkn354wvk3RQOjxa0l8lrZb0nKQP7+j28rb9futd0m7pMbZK0mzgiLyyR0uqSuv630DXvPlnSpqVLv9HSSPztnONpDlp7L+UtMXyBcZ7paRX0xiqJV2WM69a0ik5493SbY1Ix4+X9Hwa318lfTSn7HOSbpL0PLAe2FfSvyjpNVwjaYGkc3Y0Xmuek3nH9Rywu6QRkkqBfwL+O6/Mt4EDgVHAAcAA4MZ0XgnwM2AIMBjYANyTt/wFwD8De5H8IxrfXDCSjgcGAg8Ak8i50JDUE/gD8Htg3zSWp9LZ40h6GE4HdgcuI/nnUIhjgAVpfLcAAm5LtzECGAR8PY2hFHgUeBMYSvJePBARm9KYL8pZ7/nAHyJiee7G0rL/Q06PCHAu8HRELAO+DCwC+gH9geuBHfk+5VuAsyR9KH+GpM+k6xubrv+PwK93YN3579Wl6esfgP2BCrbe/x8DPgScBNzY+E8e+C7w3YjYHRhGsr8LVfAxtQ1PA6MBJO0P1AEnpOMjgYiIVyXtBfwO+BbQB/gh8HjjRUsruAXYm+R4GkPyfpLG0Q14GPgRsCfwv2mZxvnHAt8neS/6kFycP6wte03OJnnvDyDZfxfsRIxvk/Ru7A5cBdwr6eB03v1sedyfAcyPiFeU9BA9DHwljf+GNL49cspfRHKe9wQ2AXcAJ0VET+B4YM5OxGvNiQi/OtgLeAM4meQEuw04Ffg/oIwkeQwlSWzrgGE5yx0HLGxmnaOAd3PGpwM35Ix/Afj9NmL6CfBwznY2A3ul4+cDf2tmuXnAGU1MH5rWpSwvps+lw5cCb23nffpM43bTmJbnri+n3DFADVCSjs8Azm1mnScDC3LGnwU+mw7fBDwCHLCD+/NEYFE6fDvwm3T4T8Cl6fD/ApfnLFNCctEzZGfeK5KLqS/kjH8o3WdlOesbmDP/BeC8dPgZ4BtA30LrtTPHVFPryJm+DBiZ1u0/gVlp3J8HJqVl/gV4Jm+5vzXWo4l1vgOsIeklaXytJ7mwyy3zsXR4CXBizrx/BarT4U+Qd64Bf22sP8mF9Ffy5r8JHJOznbNz5v0n8B/NxH1VbozbeT9/D1yZc46tBnqk448C/5oOfw34cd6yTwP/lA4/B1yfM2+P9P06A+i2I8e/X4W93DLv2H5BcrV+KXld7CSttx7AS2k32SqSE7kfgKQekn6UdrO+R/IPunfagm30Ts7wepLW21aUdN2fA/wSIJIHhN7ig5bEIOD1ZuqwrXnbU5MXx15KnhhenNbpv4HGbtxBwJsRUZe/koh4nuTCZ3TaPXsAMKWZbU4Fuks6Ju3mHgX8Np13B1ANPJl2M167E3X6NvBJSYfnTR8CfDdnX64kuWAbUOB6a/LG9yVJHo3eJEnk/XOmNbf/Lyfp8XlV0ouSPlVgDNta5456hiTRn0CSZKaTtNZHp+OwdR1Jx7f1np0WEb0bXyQ9R1uRJJL3Kvd9zd3WviS9NPnbbjQEuL5xf6b7tF9ebC1+rySNkfSCkts+q4CPk54TEfEGycXNGUpuKX2cpJeqMb6L8uKrTOvV6P26R8S7wIUkFzTvSJoi6YAdjdea52TegUXEmyQPwp1O0v2b6+8kXecH5/xz6hUfPED0ZZLW2DGRdJeekE7XToRyJkk33vfTe33vkPxTauxqryHpjm1Kc/PWpX975EzbO69Mfhf2bem0w9I6XcQH9akBBqv5h79+npa/GHgwIjY2VSgiGki6lc8nuVh5NCLWpPPWRMSXI2J/kmcGxkk6qZntNSkiVgD/AdycN6uGpEXVO+fVPSL+zM69V0tI/mE3GkzSXb20gBhfi4jzSbrKvw08KGm37S3Xyp4mSebHkyT2xq73xuQOW9cRknoubunGI2mOLiO5SMxdd6O3SW475W+7UQ1wY97+7BER+efxTkv3yWSSY2mv9OJkKlue443H/XnA1EhuFzXG95O8+HaLiO/kLLvFMRURj0XESSQJ/y3gB61VF3My7wwuBz4eEetyJ6ZJ58fAd9J7h0gaIOmTaZGeJMl+laQ92caDVwW4BJgIHErSUh0FfBQYpeTBskeBvSV9SclDZD0lHZMu+xPgZknDlThMUp9I7lcvJmkdlKYP7jR3QdCoJ7A2rdMA4N9y5r1A8g/2W+mDS91yH+gh6eU4k+QfW34vR75fkTyjcGE6DICkT0k6IG21vQfUp68ddTfwEZL7/o1+CFzXeL9TUq/GB4x28r36NXCNko8TVgC3knTvb9VzkU/SRZL6pcfYqnTyztSzJZ4mub1Um9b/aeAskvvwL6dlpgBHSDpbUpmkz5Ik1N+3UgyTgK+k+2IIyW2DRs8A3ZR8bKxM0vnAYTnz7wP+n6TK9LivSFvRuRdkO6IkPaYbX+Ukn27pQnLR0SBpDMkFUK4HSZ6N+DxbHvc/B86RdFJ6THVPh/MvEoH3/7f8Yxr/JpLzsK2PiQ7NybyDi4jXI2JGM7MnkHT7Ppd2O/+BpDUOSeuvO0kL/jl28h9cmjRPIrmf907O66V0nZekLddTSFqr7wCvkTx4BUnimgQ8SZIAf8oHH7H7F5KEvAI4GPjzdsL5BnAkyX3Ax8jprYiI+nT7B5C0GhaRJOTG+YtI7mkGycNlzcrplt+X5F52o+Ek7/Fa4C/A9yNievo+/a/SJ/i3JyLeI7l3vmfOtN+StIIfSPflHJIHmxrt6Hs1keQC5hmS3p2NwP8rJD6SJPqyks98f5fkHnSTPRm70GySe/zPAETE30lak39MW81ExFKSh86+QvK+XA18KiJWNbnGHXcDyfnzFsnx9n4yjIgNJBeHXyD5uOY/kjyM1zj/WZIu6R+RXBDNJ+np2ZEHJnP9A8nFeeNrXfqejE+3u4LkGZLHcxdKz83fkfSkTcmZvoDk4ugbaR3fBL5I8zmlFLiO5PxeAXyYwo8nK4DS49rMtkPSRJLPNPtbuqzTkHQrSTf857KOxZrXrr8cwqy9SD+KM5a8zwqbdWTpg2+XkrTarR1zN7vZdki6maTb+o6IWJh1PGZtQdLVJB9znRwRL2Qcjm2Hu9nNzMyKnFvmZmZmRa6o7pn37ds3hg4dmnUYZmZmbeKll176e0T02165okrmQ4cOZcaM5j5lZWZm1rFIyv+Wwia5m93MzKzIOZmbmZkVOSdzMzOzIudkbmZmVuSczM3MzIqck7mZmVmRKyiZSzpV0jxJ1ZKubWL+EElPSZolabqkgTnz6iXNTF9TcqZL0i2S5kt6RdK/tk6VzMzMOpftfs5cUilwL8lPVC4CXpQ0JSLm5hS7E7g/In4u6ePAbcDF6bwNETGqiVVfCgwCDoqIhsbf1DbL2qr1tTzwYg31DUF5WQnlXUrplve3vKyEbs38LS8roazUnV5m1nYK+dKYo4Hq9PdrkfQAcAaQm8xHAtekw9OAhwtY7+eBCyKiASAilhUatNmuEhF8eVIVT73assOxrERbJfquzVwYJH9L6FZWSnmXEsrLSunWzN/y5i4muiQXEV1LS5DUSu+GmRWLQpL5AKAmZ3wRcExemSqSH6r/LnAm0FNSn4hYAXSTNAOoA74VEY2JfhjwT5LOBJYD/xoRr+VvXNIVwBUAgwcPLrhiZjtj8oxFPPXqMm74xxFcdOwQNtU1sGlzPZvqGtiY/t1UV8/GzU38fb/clvNyl238u3p9LcvqGraat3FzPQ0t+O0jiWZ6C5q4MMi5QGi2lyHnImNb6ywvK6GkxBcRZlkpJJk3dYbm/7sZD9wj6VLgGWAxSfIGGBwRSyTtD0yVNDsiXgfKgY0RUSlpLDAROH6rDUXcB9wHUFlZ6Z94s12mZuV6bnp0LsfuvyeXfXQ/SkpEty6l0L1Lm8axub7h/YuIjY1/t3GBsPXFRjq8uYGNdcnfxmXX19axcl3uutL1bW6gtr6hRXF3LS3Ju0Bo/iKi+QuE5m9jvL+OvF4N39IwKyyZLyK5t91oILAkt0BELAHGAkiqAM6KiNU584iIBZKmA0cAr6frfShdxW+Bn+10LcxaqKEhGD+5CoA7zzk801Zml9ISupSWUFHetj+d0NAQW/U8bNWrsMUFwtY9Drm9F5vyLkBWrKtrtnxLfom5tIlbGrkXBdt+vqHxIuOD2xXbut2RO923NKw9KeS/xYvAcEn7kbS4zwMuyC0gqS+wMr3/fR1JKxtJewDrI2JTWuajwO3pYg8DH0/Ljgbmt7w6ZjvnZ39+g+cXruT2sw5j4B49sg4nEyUlonvXUrp3LW3T7UYEm+vj/YuEQi4M3r/YyLm4+ODvlr0VazbWsXzNJmqbuK1R15J7GjR9S6PJWxS5tzUKeC5iW+ssLyul1Lc0LM92k3lE1Em6GngCKAUmRsTLkm4CZkTEFOBE4DZJQdLN/v+li48AfiSpgeRjcN/KeQr+W8AvJV0DrAU+14r1MitY9bK13P77VznpoL04p3Lg9hewViWJrmWia1kJdGvbbdc13tJo6lmInNscTfY+5N/W2Or5hwZWb9jc5HMVm+padkujS6m2cYFQ2DMTWz070eTFx5brKiuReyPaKUVL+rfaWGVlZfgnUK011dU3cNYP/sxbK9fzxDUnsFfPNs4m1ik1NAS19Q1b9Ths89bF+xcXW/dG5PdWbGudLemMKBGFPVTZxIVBcxcIhT2o2XlvaUh6KSIqt1euqH7P3Ky1fX/661QtWs29FxzpRG5tpqREdCspTR6wpO0esIwI6tJnI5p/FqLwC4MtLzYaWLupjhVra7dYV+PzFS19wDL3AckduUVR0LMT21hnsdzScDK3TmvO4tX851OvccaoffnHw/bJOhyzXU4SXUqVyQOW9Q1BbSEf8Wz2WYgPLgyauvXx3oa6ZtfZEmXpp1oKvUA4bFBvLj52SCu9azsQZ5tv0awd2Li5nmt+M5M+FV25acwhWYdj1uGVZviAZW39lh/Z3OKjmdv4REbTFxtbPk+xasPmD56F2FyfbNTJ3Kxt3P1/83lt2Vr+658/TK8ebfs5cjNrO5LSLvhSdu/Wcc91f9uCdTovLFzJj/+4gAuOGcyJH/JPAphZ8XMyt05l3aY6xk+uYtAePfjK6SOyDsfMrFW4m906lVsef4Wad9cz6crj2K2NHwAyM9tV3DK3TmPavGX86vm3uOL4/fnw0D2zDsfMrNU4mVunsGp9LRMenMWB/Su45pQDsw7HzKxVuZ/ROoUbH3mZletqmXjph9Mv6jAz6zjcMrcO77FZbzOlagn/etJwDhnQK+twzMxanZO5dWjL1mzkhodnc/jAXnzhxGFZh2Nmtks4mVuHFRFc99Bs1tfWc9e5oygr9eFuZh2T/7tZhzVpRg1PvbqMCacexAF7VWQdjpnZLuNkbh1Szcr13PS7uRy3fx8u/cjQrMMxM9ulnMytw2loCMZPrkISd5xzGCVF8hOGZmY7y8ncOpyf/fkNnl+4khs/NZKBe/TIOhwzs13Oydw6lOpla7j9969y8oi9OKdyYNbhmJm1CSdz6zA21zcwblIVPbqWcuvYQ5HcvW5mnYO/Ac46jO9Pe51Zi1bz/QuPZK+e3bIOx8yszbhlbh3C7EWr+d7U1zhj1L6cfug+WYdjZtamnMyt6G3cXM+4STPpU9GVm8YcknU4ZmZtzt3sVvTu/r/5vLZsLT+/7Gh69eiSdThmZm2uoJa5pFMlzZNULenaJuYPkfSUpFmSpksamDOvXtLM9DWliWW/J2lty6phndXzC1bw4z8u4MJjBjP6wH5Zh2NmlonttswllQL3AqcAi4AXJU2JiLk5xe4E7o+In0v6OHAbcHE6b0NEjGpm3ZVA75ZUwDqvtZvqGP9gFYP26MH1p4/IOhwzs8wU0jI/GqiOiAURUQs8AJyRV2Yk8FQ6PK2J+VtJLxLuAP698HDNPnDLY6+w6N0N3HXu4exW7jtGZtZ5FZLMBwA1OeOL0mm5qoCz0uEzgZ6S+qTj3STNkPScpM/kLHM1MCUi3t7WxiVdkS4/Y/ny5QWEa53BtHnL+PULb3HF8fvz4aF7Zh2OmVmmCmnONPXNG5E3Ph64R9KlwDPAYqAunTc4IpZI2h+YKmk2sAE4BzhxexuPiPuA+wAqKyvzt2ud0Kr1tUx4cBYH9q/gmlMOzDocM7PMFZLMFwGDcsYHAktyC0TEEmAsgKQK4KyIWJ0zj4hYIGk6cARJMj8AqE6/pauHpOqIOKBFtbFO4cZHXmblulomXvphunUpzTocM7PMFdLN/iIwXNJ+kroC5wFbPJUuqa+kxnVdB0xMp+8hqbyxDPBRYG5EPBYRe0fE0IgYCqx3IrdCPDprCVOqlvDFk4ZzyIBeWYdjZtYubDeZR0Qdyf3tJ4BXgEkR8bKkmySNSYudCMyTNB/oD9ySTh8BzJBURfJg3LfynoI3K9iy9zZyw8NzOHxQbz5/4rCswzEzazcKegQ4Ih4HHs+bdmPO8IPAg00s92fg0ALWX1FIHNZ5RQTX/s9sNtTWc9c5h1NW6i8vNDNr5P+IVhQmzahh6qvLmHDqQRywl6/9zMxyOZlbu1ezcj03/W4ux+3fh0s/MjTrcMzM2h0nc2vXGhqC8ZOrkMQd5xxGSYl/o9zMLJ+TubVrE59dyPMLV3Ljp0cycI8eWYdjZtYuOZlbu1W9bA23PzGPk0fsxTlHDdz+AmZmnZSTubVLm+sbGDepit26lnLr2ENJv1zIzMya4F+nsHbp+9NeZ9ai1fzgwiPZq2e3rMMxM2vX3DK3dmf2otV8b+prfGbUvpx26D5Zh2Nm1u45mVu7snFzPeMmzaRvRTnfGHNI1uGYmRUFd7Nbu3LXk/N4bdlafn7Z0fTq0SXrcMzMioJb5tZuPL9gBT/500IuPGYwow/sl3U4ZmZFw8nc2oW1m+oY/2AVg/bowfWnj8g6HDOzouJudmsXbnnsFRa9u4HJVx7HbuU+LM3MdoRb5pa5aa8u49cvvMUVJ+xP5dA9sw7HzKzoOJlbplatr2XCQ7P4UP+ejDvlwKzDMTMrSu7PtEx99ZGXWbmulomXfpjystKswzEzK0pumVtmHp21hN9VLeGLJw3nkAG9sg7HzKxoOZlbJpa9t5EbHp7D4YN68/kTh2UdjplZUXMytzYXEUx4aBYbN9dz97mHU1bqw9DMrCX8X9Ta3G9erGHavOVMOPUghvWryDocM7Oi52Rubapm5XpufnQux+3fh0uOG5p1OGZmHYKTubWZhoZg/OQqJHHHOYdRUuLfKDczaw1O5tZmJj67kOcXruTGT49k4B49sg7HzKzDKCiZSzpV0jxJ1ZKubWL+EElPSZolabqkgTnz6iXNTF9Tcqb/Ml3nHEkTJfknsjqw15au4fYn5nHyiP6cc9TA7S9gZmYF224yl1QK3AucBowEzpc0Mq/YncD9EXEYcBNwW868DRExKn2NyZn+S+Ag4FCgO/C5na+GtWeb6xsYN6mKivIybht7KJK7183MWlMhLfOjgeqIWBARtcADwBl5ZUYCT6XD05qYv5WIeDxSwAuAm2sd1L3Tqpm9eDW3fOYQ+vUszzocM7MOp5BkPgCoyRlflE7LVQWclQ6fCfSU1Ccd7yZphqTnJH0mf+Vp9/rFwO+b2rikK9LlZyxfvryAcK09mb1oNfdMreYzo/bltEP3yTocM7MOqZBk3lSfaOSNjwdGS/obMBpYDNSl8wZHRCVwAfAfkvK/7uv7wDMR8cemNh4R90VEZURU9uvXr4Bwrb3YuLmecZNm0reinG+MOSTrcMzMOqxCfmhlETAoZ3wgsCS3QEQsAcYCSKoAzoqI1TnziIgFkqYDRwCvp2W/BvQDrmxRLaxduuvJeby2bC33X3Y0vXr4+UYzs12lkJb5i8BwSftJ6gqcB0zJLSCpr6TGdV0HTEyn7yGpvLEM8FFgbjr+OeCTwPkR0dAalbH247kFK/jJnxZy0bGDOeFA96iYme1K203mEVEHXA08AbwCTIqIlyXdJKnx6fQTgXmS5gP9gVvS6SOAGZKqSB6M+1ZEzE3n/TAt+5f0Y2s3tlalLFtrN9UxfnIVg/fswfWnj8g6HDOzDq+g3zOPiMeBx/Om3Zgz/CDwYBPL/Znko2dNrdO/pd5B3fLYXBav2sDkK4+jR1fvZjOzXc3fAGetatqry/j1CzVcccL+VA7dM+twzMw6BSdzazXvrqtlwkOz+FD/now75cCswzEz6zTcB2qt5quPzOHd9bX87J8/THlZadbhmJl1Gm6ZW6v4XdUSHp31Nl88aTgH79sr63DMzDoVJ3NrsWXvbeSrj8xh1KDeXDU6/zuBzMxsV3MytxaJCCY8NIuNm+u569zDKSv1IWVm1tb8n9da5Dcv1jBt3nImnHoQw/pVZB2OmVmn5GRuO61m5XpufnQuHxnWh0uOG5p1OGZmnZaTue2Uhobgy5OrKJG445zDKSnxb5SbmWXFydx2ysRnF/LCwpXc+OmRDOjdPetwzMw6NSdz22GvLV3D7U/M4+QR/Tn7qIFZh2Nm1uk5mdsO2VzfwLhJVVSUl3Hb2EOR3L1uZpY1fwOc7ZB7p1Uze/FqfnDhkfTrWZ51OGZmhlvmtgNmLVrF96ZWc+YRAzjt0H2yDsfMzFJO5laQjZvrGTepin4V5Xx9zMFZh2NmZjnczW4FufOJeVQvW8v9lx1Nr+5dsg7HzMxyuGVu2/XcghX89NmFXHTsYE44sF/W4ZiZWR4nc9umtZvqGD+5isF79uD600dkHY6ZmTXB3ey2Tbc8NpclqzYw+arj6NHVh4uZWXvklrk1a+qrS/n1CzVcccIwjhqyZ9bhmJlZM5zMrUnvrqtlwkOzOWjvnlxzyvCswzEzs21wv6k16auPzGHV+lr+658/THlZadbhmJnZNrhlblv5XdUSHp31Nl88aTgH79sr63DMzGw7Ckrmkk6VNE9StaRrm5g/RNJTkmZJmi5pYM68ekkz09eUnOn7SXpe0muSfiOpa+tUyVpi6Xsb+eojcxg1qDdXjR6WdThmZlaA7SZzSaXAvcBpwEjgfEkj84rdCdwfEYcBNwG35czbEBGj0teYnOnfBr4TEcOBd4HLW1APawURwYSHZrFxcz13n3s4ZaXuuDEzKwaF/Lc+GqiOiAURUQs8AJyRV2Yk8FQ6PK2J+VtQ8lNbHwceTCf9HPhMoUHbrvHAizVMn7eca089iP37VWQdjpmZFaiQZD4AqMkZX5ROy1UFnJUOnwn0lNQnHe8maYak5yQ1Juw+wKqIqNvGOq0N1axczzcfnctHhvXhs8cNzTocMzPbAYUk86Z+sDryxscDoyX9DRgNLAYaE/XgiKgELgD+Q9KwAteZbFy6Ir0YmLF8+fICwrUd1dAQfHlyFSUSd5xzOCUl/o1yM7NiUkgyXwQMyhkfCCzJLRARSyJibEQcAXwlnba6cV76dwEwHTgC+DvQW1JZc+vMWfd9EVEZEZX9+vl7wXeFic8u5IWFK/namIMZ0Lt71uGYmdkOKiSZvwgMT58+7wqcB0zJLSCpr6TGdV0HTEyn7yGpvLEM8FFgbkQEyb31s9NlLgEeaWllbMfNX7qG25+Yxykj+3PWkb7TYWZWjLabzNP72lcDTwCvAJMi4mVJN0lqfDr9RGCepPlAf+CWdPoIYIakKpLk/a2ImJvOmwCMk1RNcg/9p61UJyvQ5voGxk2aSUV5GbeNPZTkuUQzMys2BX0DXEQ8DjyeN+3GnOEH+eDJ9NwyfwYObWadC0ielLeM3DO1mjmL3+OHFx1J34ryrMMxM7Od5A8Sd1KzFq3inmnVnHnEAE49ZJ+swzEzsxZwMu+ENm6uZ9ykKvpVlPP1MQdnHY6ZmbWQf2ilE7rziXlUL1vLLy4/ml7du2QdjpmZtZBb5p3McwtW8NNnF3LxsUM4frg/6mdm1hE4mXciazfVMX5yFUP27MF1px+UdThmZtZK3M3eiXzz0bksWbWByVcdR4+u3vVmZh2FW+adxNRXl/LAizVcccIwjhqyZ9bhmJlZK3Iy7wTeXVfLhIdmc9DePbnmlOFZh2NmZq3Mfa2dwA2PzGHV+lp+/s9HU15WmnU4ZmbWytwy7+CmVC3hsVlv86WTD2TkvrtnHY6Zme0CTuYd2NL3NvLVh+dwxODeXHnC/lmHY2Zmu4iTeQcVEUx4aBab6uq565zDKSv1rjYz66j8H741AWdmAAARvElEQVSDeuDFGqbPW851p41g/34VWYdjZma7kJN5B/TWivV889G5fPSAPlx87JCswzEzs13MybyDqW8Ixk+uokTijrMPp6TEv1FuZtbROZl3MBP/tJAX3ljJ18YczL69u2cdjpmZtQEn8w5k/tI13PHkPE4Z2Z+zjhyQdThmZtZGnMw7iM31DYybNJOK8jJuG3sokrvXzcw6C38DXAdxz9Rq5ix+jx9edBR9K8qzDsfMzNqQW+YdQFXNKu6ZVs3YIwZw6iF7Zx2OmZm1MSfzIrdxcz3jJs1kr57lfG3MwVmHY2ZmGXA3e5G744l5vL58Hb+4/Gh6de+SdThmZpYBt8yL2HMLVjDx2YVcfOwQjh/eL+twzMwsI07mRWrNxs2Mn1zFkD17cN3pB2UdjpmZZaigZC7pVEnzJFVLuraJ+UMkPSVplqTpkgbmzd9d0mJJ9+RMO1/S7HSZ30vq2/LqdB7ffPQVlqzawF3njqJHV98tMTPrzLabzCWVAvcCpwEjgfMljcwrdidwf0QcBtwE3JY3/2bg6Zx1lgHfBf4hXWYWcPXOVqKzmfrqUn4zo4YrRw/jqCF7ZB2OmZllrJCW+dFAdUQsiIha4AHgjLwyI4Gn0uFpufMlHQX0B57MKa/0tZuSbzfZHViyUzXoZN5dV8uEh2Zz0N49+dLJw7MOx8zM2oFCkvkAoCZnfFE6LVcVcFY6fCbQU1IfSSXAXcC/5RaOiM3A54HZJEl8JPDTpjYu6QpJMyTNWL58eQHhdmw3PDKHVetrufvcUZSXlWYdjpmZtQOFJPOmvhc08sbHA6Ml/Q0YDSwG6oAvAI9HRO7FAJK6kCTzI4B9SbrZr2tq4xFxX0RURkRlv36d+4ntKVVLeGzW23zp5AMZue/uWYdjZmbtRCFPTi0CBuWMDySvSzwilgBjASRVAGdFxGpJxwHHS/oCUAF0lbQWeChd7vV0mUnAVg/W2QeWvreRrz48hyMG9+bKE/bPOhwzM2tHCknmLwLDJe1H0uI+D7ggt0D6JPrKiGggaWFPBIiIC3PKXApURsS1kvYFRkrqFxHLgVOAV1qhPh1SRPDvD85iU109d587irJSf6LQzMw+sN2sEBF1JE+aP0GScCdFxMuSbpI0Ji12IjBP0nySh91u2c46lwDfAJ6RNAsYBdy607Xo4H79Qg1Pz1/OdaeNYL++u2UdjpmZtTOKyL/93X5VVlbGjBkzsg6jTb21Yj2nfvcZjhjcm19cdgwlJf5pUzOzzkLSSxFRub1y7q9tx+obgvGTqyiVuOPsw53IzcysSf7qsHZs4p8W8sIbK7nrnMPZt3f3rMMxM7N2yi3zdmr+0jXc8cQ8PjGyP2OPzP9Yv5mZ2QeczNuhzfUNjJs0k57dyrh17KEkX5JnZmbWNHezt0Pfm1rNnMXv8cOLjqJvRXnW4ZiZWTvnlnk7U1WzinunVTP2iAGcesjeWYdjZmZFwMm8Hdm4uZ5xk2ayV89yvjbm4KzDMTOzIuFu9nbkjifm8frydfz35cfQq3uXrMMxM7Mi4ZZ5O/GX11fw0z8t5LPHDeFjw/tmHY6ZmRURJ/N2YM3GzYyfXMXQPj249rSDsg7HzMyKjLvZ24FvPvoKb6/ewOSrPkKPrt4lZma2Y9wyz9jUV5fymxk1XDV6GEcN2SPrcMzMrAg5mWdo5bpa/v3B2Ry0d0++ePLwrMMxM7Mi5T7djEQEX314Dqs31PKLy4+mvKw065DMzKxIuWWekSlVS3hs9tt86eQDGbHP7lmHY2ZmRczJPANL39vIjY+8zBGDe3PlCftnHY6ZmRU5J/M2FhH8+4Oz2FRXz93njqKs1LvAzMxaxpmkjf36hRqenr+c608fwX59d8s6HDMz6wCczNvQmyvW8c3H5vKxA/py0TFDsg7HzMw6CCfzNlLfEIyfXEVpibj97MMoKfFvlJuZWevwR9PayE//tIAX33iXu845nH17d886HDMz60DcMm8D85eu4c4n5vOJkf0Ze+SArMMxM7MOxsl8F9tc38C4STPp2a2MW8ceiuTudTMza10FJXNJp0qaJ6la0rVNzB8i6SlJsyRNlzQwb/7ukhZLuidnWldJ90maL+lVSWe1vDrtz/emVjNn8XvcOvZQ+laUZx2OmZl1QNtN5pJKgXuB04CRwPmSRuYVuxO4PyIOA24CbsubfzPwdN60rwDLIuLAdL3584teVc0q7p1WzdgjB/DJg/fOOhwzM+ugCmmZHw1UR8SCiKgFHgDOyCszEngqHZ6WO1/SUUB/4Mm8ZS4jTfoR0RARf9/x8NuvjZvrGTdpJnv1LOdrnz4463DMzKwDKySZDwBqcsYXpdNyVQGN3eRnAj0l9ZFUAtwF/FtuYUm908GbJf1V0mRJ/ZvauKQrJM2QNGP58uUFhNs+3P77eby+fB13nH04vbp3yTocMzPrwApJ5k09sRV54+OB0ZL+BowGFgN1wBeAxyOiJq98GTAQeDYijgT+QtJVv/WGIu6LiMqIqOzXr18B4WbvL6+vYOKzC7nkuCF8bHjfrMMxM7MOrpDPmS8CBuWMDwSW5BaIiCXAWABJFcBZEbFa0nHA8ZK+AFQAXSWtBa4D1gO/TVcxGbi8JRVpL9Zs3Mz4yVXs13c3rj1tRNbhmJlZJ1BIMn8RGC5pP5IW93nABbkFJPUFVkZEA0minggQERfmlLkUqIyIa9Px3wEnAlOBk4C5LaxLu3Dzo3N5e/UGHvz8R+je1b9RbmZmu952u9kjog64GngCeAWYFBEvS7pJ0pi02InAPEnzSR52u6WAbU8Avi5pFnAx8OWdiL9d+cPcpUyasYirRg/jyMF7ZB2OmZl1EorIv/3dflVWVsaMGTOyDqNJK9fV8onvPEPfiq48cvVHKS9zq9zMzFpG0ksRUbm9cv5u9lYQEXz14Tms3lDLLy4/2onczMzalL/OtRVMqVrCY7Pf5ppTDmTEPrtnHY6ZmXUyTuYt9M7qjdz4yMscObg3V54wLOtwzMysE3Iyb4GIYMJDs6ita+Cuc0dR6t8oNzOzDDiZt8CvXniLp+cv57rTD2K/vrtlHY6ZmXVSTuY76c0V67jlsVf42AF9ueiYIVmHY2ZmnZiT+U6obwjGT66itETcfvZhlLh73czMMuSPpu2En/5pAS++8S53n3s4+/bunnU4ZmbWybllvoPmvbOGO5+YzycP7s+ZR+T/eJyZmVnbczLfAbV1DYybNJOe3cq49cxDkdy9bmZm2XM3+w64Z+prvLzkPX508VH0qSjPOhwzMzPALfOCVdWs4t7przP2yAF88uC9sw7HzMzsfU7mBdi4uZ5rJs2kf89yvvbpg7MOx8zMbAvuZi/At3//KguWr+OXnzuGXt27ZB2OmZnZFtwy344/v/53fvbsG1xy3BA+ekDfrMMxMzPbipP5NqzZuJl/mzyL/fruxrWnjcg6HDMzsya5m30bbn50Lm+v3sCDn/8I3bv6N8rNzKx9csu8GX+Yu5RJMxbx+ROHceTgPbIOx8zMrFlO5k1Yua6Wa/9nNiP22Z0vnnRg1uGYmZltk7vZ80QENzw8m9UbavnF5UfTtczXO2Zm1r45U+WZUrWEx2e/wzWnHMiIfXbPOhwzM7PtcjLP8c7qjXz14TkcObg3V54wLOtwzMzMCuJknooIJjw0i831wV3njqLUv1FuZmZFoqBkLulUSfMkVUu6ton5QyQ9JWmWpOmSBubN313SYkn3NLHsFElzdr4KreNXL7zF0/OXc/3pB7Ff392yDsfMzKxg203mkkqBe4HTgJHA+ZJG5hW7E7g/Ig4DbgJuy5t/M/B0E+seC6zdibhb1Zsr1nHLY69w/PC+XHTskKzDMTMz2yGFtMyPBqojYkFE1AIPAGfklRkJPJUOT8udL+kooD/wZO4CkiqAccA3dy701lHfEHx5UhWlJeL2sw/zb5SbmVnRKSSZDwBqcsYXpdNyVQFnpcNnAj0l9ZFUAtwF/FsT6705nbd+WxuXdIWkGZJmLF++vIBwd8xP/riAGW++yzfGHMw+vbq3+vrNzMx2tUKSeVNN1cgbHw+MlvQ3YDSwGKgDvgA8HhG5FwNIGgUcEBG/3d7GI+K+iKiMiMp+/foVEG7h5r2zhruenM8nD+7PmUfkX5+YmZkVh0K+NGYRMChnfCCwJLdARCwBxsL73ednRcRqSccBx0v6AlABdJW0FngTOErSG2kMe0maHhEntrA+Bauta2DcpJns3r2MW8881N3rZmZWtApJ5i8CwyXtR9LiPg+4ILeApL7AyohoAK4DJgJExIU5ZS4FKiOi8Wn4H6TThwKPtmUiB7hn6mu8vOQ97rv4KPpUlLflps3MzFrVdrvZI6IOuBp4AngFmBQRL0u6SdKYtNiJwDxJ80kedrtlF8Xbavbp3Z3PHjeETxy8d9ahmJmZtYgi8m9/t1+VlZUxY8aMrMMwMzNrE5JeiojK7ZXzN8CZmZkVOSdzMzOzIudkbmZmVuSczM3MzIqck7mZmVmRczI3MzMrck7mZmZmRc7J3MzMrMgV1ZfGSFpO8r3uxaAv8Pesg2hlrlNxcJ2KQ0erU0erD7SPOg2JiO3+ylhRJfNiImlGId/aU0xcp+LgOhWHjlanjlYfKK46uZvdzMysyDmZm5mZFTkn813nvqwD2AVcp+LgOhWHjlanjlYfKKI6+Z65mZlZkXPL3MzMrMg5mZuZmRU5J/NWIOlDkmbmvN6T9CVJX5e0OGf66VnHui2SJkpaJmlOzrQ9Jf2fpNfSv3uk0yXpPyVVS5ol6cjsIm9eM3W6Q9Krady/ldQ7nT5U0oac/fXD7CJvXjN1avZYk3Rdup/mSfpkNlE3r5n6/CanLm9ImplOL5Z9NEjSNEmvSHpZ0hfT6UV7Pm2jTkV7Pm2jTsV3PkWEX634AkqBd4AhwNeB8VnHtAOxnwAcCczJmXY7cG06fC3w7XT4dOB/AQHHAs9nHf8O1OkTQFk6/O2cOg3NLddeX83UqcljDRgJVAHlwH7A60Bp1nXYXn3y5t8F3Fhk+2gf4Mh0uCcwP90XRXs+baNORXs+baNORXc+uWXe+k4CXo+IYvmmuvdFxDPAyrzJZwA/T4d/DnwmZ/r9kXgO6C1pn7aJtHBN1SkinoyIunT0OWBgmwfWAs3sp+acATwQEZsiYiFQDRy9y4LbCduqjyQB5wK/btOgWigi3o6Iv6bDa4BXgAEU8fnUXJ2K+Xzaxn5qTrs9n5zMW995bPmP5+q0+2liY5dakekfEW9DcuADe6XTBwA1OeUWse2ToL26jKRF1Gg/SX+T9LSk47MKaic1dawV+346HlgaEa/lTCuqfSRpKHAE8Dwd5HzKq1Ouoj2fmqhTUZ1PTuatSFJXYAwwOZ30A2AYMAp4m6S7sKNQE9OK6nOOkr4C1AG/TCe9DQyOiCOAccCvJO2eVXw7qLljrdj30/lseXFcVPtIUgXwEPCliHhvW0WbmNYu91NzdSrm86mJOhXd+eRk3rpOA/4aEUsBImJpRNRHRAPwY9pJd8wOWtrY3Zf+XZZOXwQMyik3EFjSxrHtNEmXAJ8CLoz0ZljadbYiHX6J5H7YgdlFWbhtHGtFu58klQFjgd80TiumfSSpC0mC+GVE/E86uajPp2bqVNTnU1N1Ksbzycm8dW3Risi753UmMGerJdq/KcAl6fAlwCM50z+bPoV7LLC6sfuwvZN0KjABGBMR63Om95NUmg7vDwwHFmQT5Y7ZxrE2BThPUrmk/Ujq9EJbx7eTTgZejYhFjROKZR+l9/p/CrwSEXfnzCra86m5OhXz+bSNOhXf+ZT1E3gd5QX0AFYAvXKm/QKYDcwiOQj2yTrO7dTh1yRdSptJrkAvB/oATwGvpX/3TMsKuJfkans2UJl1/DtQp2qS+14z09cP07JnAS+TPK36V+DTWce/A3Vq9lgDvpLup3nAaVnHX0h90un/BVyVV7ZY9tHHSLpfZ+UcZ6cX8/m0jToV7fm0jToV3fnkr3M1MzMrcu5mNzMzK3JO5mZmZkXOydzMzKzIOZmbmZkVOSdzMzOzIudkbmZmVuSczM3MzIrc/w9kxtHhWiT8QgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nn_two_neurons = [\n",
    "    (64,64),\n",
    "    (128, 128),\n",
    "    (256, 256)\n",
    "]\n",
    "nn_two_accuracies = []\n",
    "\n",
    "for n in nn_two_neurons:\n",
    "    nn_accuracies = cross_validate_nn(n)\n",
    "    nn_mean_accuracy = np.mean(nn_accuracies)\n",
    "    nn_two_accuracies.append(nn_mean_accuracy)\n",
    "\n",
    "plt.figure(figsize=(8,4))\n",
    "plt.title(\"Mean Accuracy vs. Neurons In Two Hidden Layers\")\n",
    "\n",
    "x = [i[0] for i in nn_two_neurons]\n",
    "plt.plot(x, nn_two_accuracies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.943783716901757, 0.9560331601088841, 0.9554813165058154]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nn_two_accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary**\n",
    "\n",
    "Using 2 hidden layers improved our simple accuracy to 95%. While I'd traditionally be worried about overfitting, using 4-fold cross validation also gives me a bit more assurance that the model is generalizing to achieve the extra 1% in simple accuracy over the single hidden layer networks we tried earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Neural Network With Three Hidden Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "# 50% Train / test validation\n",
    "def train_nn(neuron_arch, train_features, train_labels):\n",
    "    mlp = MLPClassifier(hidden_layer_sizes=neuron_arch)\n",
    "    mlp.fit(train_features, train_labels)\n",
    "    return mlp\n",
    "\n",
    "def test(model, test_features, test_labels):\n",
    "    predictions = model.predict(test_features)\n",
    "    train_test_df = pd.DataFrame()\n",
    "    train_test_df['correct_label'] = test_labels\n",
    "    train_test_df['predicted_label'] = predictions\n",
    "    overall_accuracy = sum(train_test_df[\"predicted_label\"] == train_test_df[\"correct_label\"])/len(train_test_df)    \n",
    "    return overall_accuracy\n",
    "\n",
    "def cross_validate_six(neuron_arch):\n",
    "    fold_accuracies = []\n",
    "    kf = KFold(n_splits = 6, random_state=2)\n",
    "    for train_index, test_index in kf.split(features):\n",
    "        trrain_features, tesst_features = features.loc[train_index], features.loc[test_index]\n",
    "        trrain_labels, tesst_labels = labels.loc[train_index], labels.loc[test_index]\n",
    "       \n",
    "        model = train_nn(neuron_arch, trrain_features, trrain_labels)\n",
    "        overall_accuracy = test(model, tesst_features, tesst_labels)\n",
    "        fold_accuracies.append(overall_accuracy)\n",
    "    return fold_accuracies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n",
      "C:\\Users\\krishna\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\multilayer_perceptron.py:562: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1e3f3858630>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAEICAYAAAByPazKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VOX5xvHvQ0iAsAUBUXZUVFARJAIuVVu1orZSxSogAgJi69Jqta1au2Gtrdqq/blUUFB2qdUWrTtqXSpLAEERQQSFgEhk37M9vz/OiQxDQgZIcmaS+3NduXK2mXnmzZm5533PmRNzd0RERCT51Yq6ABEREUmMQltERCRFKLRFRERShEJbREQkRSi0RUREUoRCW0REJEUotEUk6ZnZ52Z2TtR1lMXM/m5mv97Hejezo8pYN8TM3q286qQ6UWingPANK9/MmsUt/yB8M2gfUV0dzKzYzB6J4vFTlZmdFf7dHo5b/q6ZDYmorIMWPq/cA7jdFWa2NfzZEe5TJfNbK6PW/azvSTP7Q9yy9uHfsDaAu//I3e+MpsLSxdco1YNCO3UsB/qXzJjZCUC96MoBYBCwAehnZnWq8oGrwRvRNmBQVXzgSva2cveJ7t7A3RsA5wOrS+bDZfsl2Z9vTaK/RcVTaKeO8QQhWWIwMC52AzOrY2b3mdkKM/sqHLKrF65rYmYvmFmemW0Ip1vH3PYtM7vTzN4zsy1m9mp8z74Ug4A7gALg+3G1HGdmr5nZ+rCW28PlaWZ2u5l9Fj7OHDNrU1qvIKxpeDg9JKztfjNbD/zOzI40szfMbJ2ZfW1mE80sK+b2bczs2fA5rzOzh8I2Wh9+6CnZ7tCwh9e8lPbcaGbHxyxrHm57qJk1C9txY3if75hZoq+pjcCTwG/L2sDMhprZovDv9YqZtQuXH0hb1TKzO8zsCzNba2bjzKxx3P0NDvedr83sVzH33cPMcsxsc/i3/GsiT/AA96l96WpmC8xsk5k9bWZ1w8c5y8xyzeyXZrYGGBsu/54Fo1Ebzex/ZtYlpraWZvbPcN9YbmY/OYi69uqNm9nPzexLM1ttZkPjtm1qZtPC9pwFHBm3/tiY185iM7ss7nEeNrP/hG0608z2uH2C9fYws/fDtvkyfG1khOseNrO/xG3/vJndGE6X2XZm9jsze8bMJpjZZmDIge4/UjqFduqYATQys05mlgZcDkyI2+bPwNFAV+AooBXwm3BdLYI3s3ZAW2AH8FDc7QcAVwGHAhnALWUVY2bfAloDU4CpxHygMLOGwOvAy0DLsJbp4eqfEYwYXAA0AoYC2xN4/gA9gWVhfXcBBtwdPkYnoA3wu7CGNOAF4AugPUFbTHH3XWHNA2Putz/wurvnxT5YuO2zxIxwAJcB/3X3tcDNQC7QHGgB3A7sz3WB7wL6mtkx8SvM7Afh/V0S3v87wOT9uO/4thoS/nwbOAJowN5//9OBY4Czgd+YWadw+YPAg+7eiCBgpu5HHQnvUwm4DOgNdAC6EDyfEocBhxDs3yPM7CRgDHAN0BR4DJgWfhCrBTwPzCfYL84GbjSz8w6itm+YWW+C53ku0BGIPxb/MLATOJxg/x8ac9v6wGvAJII26w88YmbHxdy+P/B7oAmwlODvu7+KgJuAZsApBG1wbbjuKaB/yQfQ8IPW2cDkBNuuD/AMkAVM5OD2H4nn7vpJ8h/gc4IX/h0EIdWb4IVdmyAk2hME2DbgyJjbnQIsL+M+uwIbYubfAu6Imb8WeHkfNT0O/CvmcQqAQ8P5/sC8Mm63GOhTyvL24XOpHVfT8HB6CLCinHb6QcnjhjXlxd5fzHY9gZVArXA+B7isjPs8B1gWM/8eMCicHgn8GzhqP/+eZwG54fQ9wNPh9LvAkHD6JWBYzG1qEXy4aXcgbUXwoenamPljwr9Z7Zj7ax2zfhbQL5x+myAkmiX6vA5knyrtPuJeAwNj5u8B/h5zm3ygbsz6R4E7S9n3zgz//vHtcxswtoyaniQI2Y0xP5tj/wbhNn8Ip8cAf4q5/dHhtkcBaWG7Hxuz/o/Au+H05cA7cY//GPDbmMd5PGbdBcAnZdS9136yj3a/EXguZn4RcG44fT3wYsxrp8y2I/jQ/Hbc+oT2H/0k9qOedmoZT9BzGULc0DhBbywTmBMOeW0k6Ok2BzCzTDN7LBwe3UzwQsoKe6Ql1sRMbyfoje3FgiH3HxJ8isbd3wdWhLVB0OP9rIznsK915VkZV8ehZjbFzFaFz2kCQc+h5HG+cPfC+Dtx95kEH3DONLNjCd5Mp5XxmG8A9cyspwXD012B58J19xL0dF41s2VmdusBPKc/A+eZ2Ylxy9sBD8b8LdcTfDBrleD9roybb0kw6lDiC4LAbhGzrKy//zCC4PnEzGab2fcSrGFf93kg9nVfee6+M2a+HXBzSfuFbdiGoB3aAS3j1t3Onm0R7z53zyr5Iejpl6Ule7Z/bLs3J2j3sta3A3rG1XYFwUhCiYNuUzM72oJDO2vC184f2f3agaC3XTIaNZDgvaekvvLaLn7fO5j9R+IotFOIu39BcELaBQTDtrG+JhjyPi7mzaWx7z6R52aC3lVPD4apzgiX2wGUcjHB0PYj4Yt+DUGYlAyRryTuOF2MstZtC39nxiw7LG6b+KHnu8NlXcLnNJDdz2cl0NbKPhGm5E3pSuCZuDf83Q/oXkwwnNef4EPJC+6+JVy3xd1vdvcjCI7p/8zMzi7j8Url7uuAB4D4M49XAtfEBoW713P3/3FgbbWa4A23RFugEPgqgRo/dff+BMO1fwaeCYdxk0n8810J3BXXfpnuPjlctzxuXUN3v6CCavmS4ANCibYx03kE7V7W+pUEh19ia2vg7j+uoNpKPAp8AnQMXzu3s+d7wQSgT/hhshPwr5j6ymu7Pf4WKbL/pAyFduoZBnzH3bfFLgzDZTRwv5kdCmBmrWKONTUkCPWNZnYI+zgBKgGDCYYATyDoeXYFTiM4UegEgmPJh5nZjeExxIZm1jO87ePAnWbW0QJdzKypB8eTVwEDLThZbShlB3+JhsDW8Dm1An4es24WwZvnn8ysvpnVNbPTYtaPJ/jwMZC9Ry3iTSIYtrwinAa+OdHpKDMzguHSovBnf/0VOJXgzbHE34HbSo5lmlljM/shwAG21WTgJgu+pteAoGf1dGkjEfHMbKCZNQ/3sY3h4gN5nlVpNPCjcITEwn3gwvB8i1nAZgtOXKsXtuHxZnZyBT32VIITsDqbWSYxrzV3LyL4wP27cPSrM8HrqcQLwNFmdqWZpYc/J8ecX3Ag6oT7f8lPLYLXzmZgazjatMeHAnfPBWYTvE7+6e47wlX73XYpuv8kLYV2inH3z9w9p4zVvyQYrp0RDnm9TtC7hqA3V4+gRz6DYOh8v4XheDbwgLuvifmZE97n4LAnei5B73MN8CnBCVAQBNRU4FWCN40n2P3VtasJgncdcBzwv3LK+T1wErAJ+A8xow/hm+P3CYa+VxCcMHZ5zPpcYC5Br+CdfT1IzHB6S4JjzSU6ErTxVuB94BF3fytsp5csPGO+PO6+meAY7SExy54j6JVMCf+WHxF8HarE/rbVGII34LcJRmt2AjckUh/BORQLLfjO9IMEx7pLHZlIFuFr5GqCk+02ELwuhoTrSvaNrgRt8TXBh8nGFfTYLxG83t4IH/eNuE2uJxjSXkNwjHpszG23AN8F+hGMjqwh2A8O5iuVWwk+sJf8fIfgRLkBwBaCDzhPl3K7pwg+mJcMjR9o26Xc/pPMzH1/TnYVqT7MbAzBd4LviLoWkWRjZmcQDJO3D3vJkgT0xXepkSy4qMklQLdoKxFJPmaWDvyU4Ex1BXYS0fC41DhmdifBcPO97r486npEkkl4/HwjwffIH4i4HImj4XEREZEUoZ62iIhIiki6Y9rNmjXz9u3bR12GiIhIlZkzZ87X7t68vO2SLrTbt29PTk5Z32gSERGpfszsi/K30vC4iIhIylBoi4iIpAiFtoiISIpQaIuIiKQIhbaIiEiKUGiLiIikCIW2iIhIiki672mLiIhErbComE07Cti4o4CN2wvYtCOfDduC+U3b8+nTrRVHNm9Q5XUptEVEpNoqKCr+JnQ3bg8COAjicD5cvmlHwe75bQVs2VVY5n2awXGtGiu0RURESpNfWMzGHflsCkN3w7b8sNe7O3g3xk1v2lHA1n2Eby2DrMwMsuql0zgznWYNMjjq0AY0rpdOk8wMsjLTycpMp3G9dLIyM2iSmU5WvQwa1q1NrVpWhc9+N4W2iIhUmZ0FRbt7tdt3B++GcHqvXnG4fHt+UZn3mVbLvgneJpkZHNaoLscc1pCseruDtyScs8LgbZyZTsM60YXvgVJoi4jIfttZUMTGkrCNDdrYQC5l+HlHQdnhW7uWBeGamU5WvXRaZtWlc8tG34Rt45jgbZKZEfaA02lQpzZmqRW+B0qhLSJSQ7k7O8LwLQnYoNe7ezp+yLlkeldhcZn3m5FWK+z1Br3aNodkckK9mB5vuHz30HOwvH5GWo0J3wOl0BYRSXHuzvb8opheb3zY7u4Fxx8Dzi/aR/jWrvVN8DbOTKd9s0yy6mWFvd70PYefY6brpSt8K4tCW0QkSbg7W3cVfhO8G/YI29jh5z2HojftyKegyMu837rptfYYTj6iWYO4Xm/JCVcZ3ww9Z2WmUzc9rQqfvSRCoS0iUsHcnc07C/fs1YbBuyFuKDo+hIuKyw7fzIy08ISr4Ezmo1s0+CZos+rFn3C1e/hZ4Vt9KLRFRMpQXOxs2VnIxh0lYZsfc+Zz3PDzN0PPQS95X+HboE7tmGO56Rx7WKM9hpkbfxPCQTg3DsO3Tm2Fb02n0BaRaq+o2Nm8I6ZXG9u7/eYYcLB8w/ZwKDoMXy87e2lYpzZZ9Xcfz22VVS/u+G7M14xihp/T03QFaTkwCm0RSRmFRcVs3lm499eMShlm3hTzvd/NO/cdvo3q1t5jOLndIZnfDDmXfM2oSf30PYaiG9VT+ErVU2iLSJUrKLmuc9w1nfcYfi7lUpNbdu770pKN68UEbWYG7ZvV3+P4bvzwc5PMDBrVSyctxS6wITVXQqFtZr2BB4E04HF3/1Pc+nbAGKA5sB4Y6O654boi4MNw0xXuflEF1S4iEYu/tGRpF9WI/ZrRhm2JXVqycUzQxl5asqSX26R+xu5twuUN6yp8pforN7TNLA14GDgXyAVmm9k0d/84ZrP7gHHu/pSZfQe4G7gyXLfD3btWcN0iUoF2FRbt+5rOZRwD3pbgpSWz6qVzaMO6HN1iz0tLxl/TOVUvLSlSVRLpafcAlrr7MgAzmwL0AWJDuzNwUzj9JvCviixSRBJTcmnJjaVcuzn+GPCGmKHo8i8tubtX2zKrLp0Ob7T7a0b197ymc0kg16RLS4pUlURCuxWwMmY+F+gZt818oC/BEPrFQEMza+ru64C6ZpYDFAJ/cncFukgF+u+SPH79r4/4avPOfV5aMj3N9hhOjr+0ZMnwc+xFOHRpSZHkkkhol/ZqjT8P8xbgITMbArwNrCIIaYC27r7azI4A3jCzD939sz0ewGwEMAKgbdu2+1G+SM02fdFX/HjCXNo3y2Twqe337O3G/NcjXVpSpHpIJLRzgTYx862B1bEbuPtq4BIAM2sA9HX3TTHrcPdlZvYW0A34LO72o4BRANnZ2fv4YoaIlHhl4RqunzSXzoc3YtzQnjTOTI+6JBGpZIl8yXA20NHMOphZBtAPmBa7gZk1M7OS+7qN4ExyzKyJmdUp2QY4jT2PhYvIAXjxwy+5buJcjm/VmPHDFdgiNUW5oe3uhcD1wCvAImCquy80s5FmVvL1rbOAxWa2BGgB3BUu7wTkmNl8ghPU/hR31rmI7Kdp81dzw+R5dG2TxbihPWhUV4EtUlOY7+syQRHIzs72nJycqMsQSUrPzcvl5qnzObn9IYwZcjL16+j6SCLVgZnNcffs8rbTNfhEUsTUnJX8bOp8eh3RlLFXKbBFaiK96kVSwORZK7jt2Q/5VsdmjB6UrX+1KFJDqactkuTGv/85tz37Id8+prkCW6SGU09bJImNfW85v3/+Y87t3IKHBnTT/1MWqeEU2iJJavTby7jrxUX0Pu4w/ta/Gxm1NTAmUtMptEWS0CNvLeWelxdzYZfDeeDyrvq/zSICKLRFks6Dr3/K/a8voU/XlvzlhydSW4EtIiGFtkiScHfuf20Jf3tjKX1Pas09l3bR/4cWkT0otEWSgLtzzyuLefStz+h3chv+ePEJ+p/SIrIXhbZIxNydP764iNHvLGdgr7aMvOh4BbaIlEqhLRIhd2fkCx8z9r3PGXJqe377/c7695kiUiaFtkhEioud30z7iAkzVjD89A786sJOCmwR2SeFtkgEioud25/7kCmzV/KjM4/kl72PUWCLSLkU2iJVrKjY+eU/F/DMnFxu+M5R/OzcoxXYIpIQhbZIFSosKubnzyzguXmruOmco/npOR2jLklEUohCW6SKFBYVc9PU+Tw/fzU/P+8Yrvv2UVGXJCIpRqEtUgUKior5yeR5vPTRGm6/4FhGnHFk1CWJSApSaItUsvzCYq6fNJdXP/6KX3+vM8NO7xB1SSKSohTaIpVoZ0ER106cyxufrGVkn+MYdEr7qEsSkRSW0H8iMLPeZrbYzJaa2a2lrG9nZtPNbIGZvWVmrePWNzKzVWb2UEUVLpLsdhYUMWL8HN74ZC1/vPgEBbaIHLRyQ9vM0oCHgfOBzkB/M+sct9l9wDh37wKMBO6OW38n8N+DL1ckNezIL2L4Uzm882ke9/TtwoCebaMuSUSqgUR62j2Ape6+zN3zgSlAn7htOgPTw+k3Y9ebWXegBfDqwZcrkvy27Srkqidn8b/Pvua+S0/kspPbRF2SiFQTiYR2K2BlzHxuuCzWfKBvOH0x0NDMmppZLeAvwM8PtlCRVLB1VyFDxs5i9ucbuP/yrvTt3rr8G4mIJCiR0C7tUk0eN38LcKaZzQPOBFYBhcC1wIvuvpJ9MLMRZpZjZjl5eXkJlCSSfDbvLGDQEzOZt2Ijf+vXjT5d4z/biogcnETOHs8FYsf3WgOrYzdw99XAJQBm1gDo6+6bzOwU4Ftmdi3QAMgws63ufmvc7UcBowCys7PjPxCIJL1N2wsYNHYWH6/exEMDTqL38YdFXZKIVEOJhPZsoKOZdSDoQfcDBsRuYGbNgPXuXgzcBowBcPcrYrYZAmTHB7ZIqtuwLZ8rx8xkyZqtPHpFd87p3CLqkkSkmip3eNzdC4HrgVeARcBUd19oZiPN7KJws7OAxWa2hOCks7sqqV6RpLJu6y4GPD6TJV9t5bFBCmwRqVzmnlyj0dnZ2Z6TkxN1GSLl+nrrLq4YPZPP121j9KBszji6edQliUiKMrM57p5d3na6IprIAVi7eScDHp/Jqg07GDvkZE49qlnUJYlIDaDQFtlPazbtZMDoGazZvJMnrzqZnkc0jbokEakhFNoi+2H1xh30Hz2DdVvzGT+sB93bHRJ1SSJSgyi0RRK0cv12Bjw+g43bCxg/rAfd2jaJuiQRqWEU2iIJ+GLdNgaMnsnWXYVMHN6TLq2zoi5JRGoghbZIOZZ/vY3+o2awq7CIicN7cnyrxlGXJCI1lEJbZB+Wrt3KgNEzKCx2Jl3di06HN4q6JBGpwRTaImVY8tUWBoyeCcCUEb04ukXDiCsSkZoukX8YIlLjLPpyM/1HzaCWKbBFJHmopy0S56NVm7jyiZnUTU9j0tW96NCsftQliYgA6mmL7GFB7kYGjJ5BZkZtnh5xigJbRJKKetoioXkrNjBozCyyMtOZNLwXbQ7JjLokEZE9KLRFgJzP1zNk7GyaNshg8tW9aJlVL+qSRET2ouFxqfFmLlvHoDGzOLRhHZ4ecYoCW0SSlnraUqP9b+nXDHsqh1ZN6jFpeE8ObVQ36pJERMqknrbUWO98msdVT86m7SGZTBnRS4EtIklPPW2pkd5cvJZrxs/hyOYNmDi8J4fUz4i6JBGRcim0pcZ5/eOvuHbiXI4+rAEThvUkK1OBLSKpQcPjUqO8/NEafjRhDp0Ob8jEYb0U2CKSUhIKbTPrbWaLzWypmd1ayvp2ZjbdzBaY2Vtm1jpm+Rwz+8DMFprZjyr6CYgk6j8LvuS6SXPp0rox44f3pHFmetQliYjsl3JD28zSgIeB84HOQH8z6xy32X3AOHfvAowE7g6Xfwmc6u5dgZ7ArWbWsqKKF0nUvz9YxU+mzOOktlmMG9aTRnUV2CKSehLpafcAlrr7MnfPB6YAfeK26QxMD6ffLFnv7vnuvitcXifBxxOpUM/OzeWmpz/g5PZNePKqHjSoo1M5RCQ1JRKirYCVMfO54bJY84G+4fTFQEMzawpgZm3MbEF4H39299UHV7JI4qbOXsnN/5jPqUc2Y+yQHtRXYItICksktK2UZR43fwtwppnNA84EVgGFAO6+Mhw2PwoYbGYt9noAsxFmlmNmOXl5efv1BETKMnHmF/zinws4o2NzHh+cTb2MtKhLEhE5KImEdi7QJma+NbBHb9ndV7v7Je7eDfhVuGxT/DbAQuBb8Q/g7qPcPdvds5s3b76fT0Fkb+Pe/5xfPfcR3zn2UB67sjt10xXYIpL6Egnt2UBHM+tgZhlAP2Ba7AZm1szMSu7rNmBMuLy1mdULp5sApwGLK6p4kdI88e5yfvPvhZzbuQV/H6jAFpHqo9zQdvdC4HrgFWARMNXdF5rZSDO7KNzsLGCxmS0BWgB3hcs7ATPNbD7wX+A+d/+wgp+DyDce++9n3PnCx5x//GE8csVJZNTWuY8iUn2Ye/zh6WhlZ2d7Tk5O1GVICnr4zaXc+8pivn9iS+6/7ERqpymwRSQ1mNkcd88ubzudSivVwoOvf8r9ry/h4m6tuPfSLgpsEamWFNqS0tydv762hP97YymXdm/Nn/t2Ia1WaV94EBFJfQptSVnuzp9e/oTH/ruM/j3acNcPTqCWAltEqjGFtqQkd+cP/1nEE+8uZ2Cvtoy86HgFtohUewptSTnuzu+f/5gn//c5Q05tz2+/3xkzBbaIVH8KbUkpxcXOr//9ERNnruDqb3Xg9gs6KbBFpMZQaEvKKC52bn/uQ6bMXsmPzzqSX5x3jAJbRGoUhbakhKJi5xfPLOCfc3P5ydkduemcjgpsEalxFNqS9AqLirn5H/P59wer+dm5R/OTsztGXZKISCQU2pLUCoqKufHpD/jPgi/5Re9juPaso6IuSUQkMgptSVr5hcX8ZPI8Xl64hl9d0Imrzzgi6pJERCKl0JaktKuwiOsmzuP1RV/xm+91ZujpHaIuSUQkcgptSTo7C4r48YQ5vLk4jzv7HMeVp7SPuiQRkaSg0JaksrOgiKvH5fDu0q+5+5IT6N+jbdQliYgkDYW2JI3t+YUMfyqH95et456+XfhhdpuoSxIRSSoKbUkK23YVctWTs8n5fD1/vexELu7WOuqSRESSjkJbIrdlZwFXjZ3NvJUbeaBfNy46sWXUJYmIJCWFtkRq884CBo+ZxYe5m/i//t244ITDoy5JRCRpKbQlMpu2F3DlmJks+nIzD19xEucdd1jUJYmIJLVaiWxkZr3NbLGZLTWzW0tZ387MppvZAjN7y8xah8u7mtn7ZrYwXHd5RT8BSU0btuUz4PEZfPLlFv4+sLsCW0QkAeWGtpmlAQ8D5wOdgf5m1jlus/uAce7eBRgJ3B0u3w4McvfjgN7AA2aWVVHFS2pat3UX/UfP4NO1Wxk1qDtnd2oRdUkiIikhkZ52D2Cpuy9z93xgCtAnbpvOwPRw+s2S9e6+xN0/DadXA2uB5hVRuKSmvC1BYH++bhtjBp/MWcccGnVJIiIpI5HQbgWsjJnPDZfFmg/0DacvBhqaWdPYDcysB5ABfHZgpUqqW7t5J/1Gvc/K9TsYO6QHp3dsFnVJIiIpJZHQLu2fFnvc/C3AmWY2DzgTWAUUfnMHZocD44Gr3L14rwcwG2FmOWaWk5eXl3Dxkjq+3LSDy0fNYM2mnTw1tAenHNm0/BuJiMgeEjl7PBeIvTRVa2B17Abh0PclAGbWAOjr7pvC+UbAf4A73H1GaQ/g7qOAUQDZ2dnxHwgkxa3auIP+o2awfls+44b1oHu7Q6IuSUQkJSXS054NdDSzDmaWAfQDpsVuYGbNzKzkvm4DxoTLM4DnCE5S+0fFlS2pYuX67Vz+2Pts2J7PhOE9FdgiIgeh3NB290LgeuAVYBEw1d0XmtlIM7so3OwsYLGZLQFaAHeFyy8DzgCGmNkH4U/Xin4Skpy+WLeNyx97ny07C5k0vBdd2+iLAyIiB8Pck2s0Ojs723NycqIuQw7Ssryt9B89g/zCYiYO70Xnlo2iLklEJGmZ2Rx3zy5vO10RTSrc0rVb6D96JsXFzuQRvTj2MAW2iEhFUGhLhVq8ZgtXPD4DMKaM6EXHFg2jLklEpNpQaEuF+Xj1ZgY+MZP0NGPS1b04snmDqEsSEalWFNpSIT5atYmBT8ykXnoak6/uRftm9aMuSUSk2knoH4aI7Mv8lRsZMHoG9TNq8/SIUxTYIiKVRD1tOShzvtjAkDGzyKqfzuSre9G6SWbUJYmIVFsKbTlgsz9fz5Axs2jesA6TR/Ti8Mb1oi5JRKRaU2jLAZmxbB1Dn5zNYY3rMvnqXrRoVDfqkkREqj0d05b99t7SrxkydhatsuoxZYQCW0SkqqinLfvlv0vyGDEuhw7N6jNheE+aNagTdUkiIjWGQlsS9uYna7lm/ByOOrQBE4b35JD6GVGXJCJSoyi0JSGvLlzDdZPmcuxhjRg/rAdZmQpsEZGqptCWcr304ZfcMHkex7dqzFNDe9C4XnrUJYmI1EgKbdmn5+ev5sanP6BrmyyevOpkGtZVYIuIREVnj0uZ/jVvFT+dMo/ubZvw1NAeCmwRkYippy2lemZOLj9/Zj69OjTliSHZZGZoVxERiZreiWUvT89ewa3PfsjpRzVj1JXZ1MtIi7okERFBw+MSZ8KML/jlPz/kjI7NGT1IgS0ikkzU05ZvPPnecn73/MecfeyhPDLwJOrUVmCLiCSThHraZtbbzBab2VIzu7WU9e3MbLqZLTCzt8ysdcy6l81so5m9UJGFS8UFhfBfAAAQkklEQVR6/J1l/O75jznvuBY8OrC7AltEJAmVG9pmlgY8DJwPdAb6m1nnuM3uA8a5exdgJHB3zLp7gSsrplypDI++9Rl/+M8iLjzhcB4acBIZtXXUREQkGSXy7twDWOruy9w9H5gC9InbpjMwPZx+M3a9u08HtlRArVIJ/m/6p/z55U+46MSWPNivK+lpCmwRkWSVyDt0K2BlzHxuuCzWfKBvOH0x0NDMmiZahJmNMLMcM8vJy8tL9GZyENyd+19bwl9eW8Il3Vpx/+Vdqa3AFhFJaom8S1spyzxu/hbgTDObB5wJrAIKEy3C3Ue5e7a7Zzdv3jzRm8kBcnfue3UxD07/lB92b829PzyRtFql/ZlFRCSZJHL2eC7QJma+NbA6dgN3Xw1cAmBmDYC+7r6pooqUiuPu/OmlT3js7WX079GWu35wPLUU2CIiKSGRnvZsoKOZdTCzDKAfMC12AzNrZmYl93UbMKZiy5SK4O6MfOFjHnt7GYNOaafAFhFJMeWGtrsXAtcDrwCLgKnuvtDMRprZReFmZwGLzWwJ0AK4q+T2ZvYO8A/gbDPLNbPzKvg5SAKKi53fTlvI2Pc+Z+hpHfj9RccpsEVEUoy5xx+ejlZ2drbn5OREXUa1Ulzs/OpfHzF51gquOeMIbj3/WMwU2CIiycLM5rh7dnnb6Ypo1VxRsXPbswuYmpPLdd8+klu+e4wCW0QkRSm0q7GiYufn/5jPs/NW8dOzO3LjOR0V2CIiKUyhXU0VFhXzs6nzmTZ/NTefezQ3nN0x6pJEROQgKbSroYKiYm6c8gH/+fBLftn7WH581pFRlyQiIhVAoV3N5BcWc8Pkubyy8CvuuLATw791RNQliYhIBVFoVyO7Cou4buJcXl+0lt99vzNDTusQdUkiIlKBFNrVxM6CIn40YQ5vLc7jDz84noG92kVdkoiIVDCFdjWwI7+IEeNzeHfp1/zpkhPo16Nt1CWJiEglUGinuO35hQx7MocZy9dx76Uncmn31lGXJCIilUShncK27ipk6NjZ5Hyxnvsv68oPusX/x1QREalOFNopasvOAoaMnc0HKzfyYL9ufP/EllGXJCIilUyhnYI27Shg0JhZLFy1iYf6d+P8Ew6PuiQREakCCu0Us3F7Plc+MYtP1mzmkStO4rvHHRZ1SSIiUkUU2ilk/bZ8Bj4+k6Vrt/LYld35zrEtoi5JRESqkEI7RXy9dRcDH5/J8q+3MXpwNmce3TzqkkREpIoptFPA2i07uWL0TFZu2M6YISdz2lHNoi5JREQioNBOcl9t3kn/0TNYs2knY4f04JQjm0ZdkoiIREShncRWb9zBgNEzyNuyi6eG9uDk9odEXZKIiERIoZ2kcjdsp//oGWzcVsC4YT3p3q5J1CWJiEjEaiWykZn1NrPFZrbUzG4tZX07M5tuZgvM7C0zax2zbrCZfRr+DK7I4qurFeu2c/ljM9i0vYAJwxXYIiISKDe0zSwNeBg4H+gM9DezznGb3QeMc/cuwEjg7vC2hwC/BXoCPYDfmpkSaB8+/3obl496n235hUy6uhcntsmKuiQREUkSifS0ewBL3X2Zu+cDU4A+cdt0BqaH02/GrD8PeM3d17v7BuA1oPfBl109fZa3lctHvc+uwmImDe/F8a0aR12SiIgkkURCuxWwMmY+N1wWaz7QN5y+GGhoZk0TvC1mNsLMcswsJy8vL9Haq5VPv9pCv1EzKCp2Jl/di84tG0VdkoiIJJlEQttKWeZx87cAZ5rZPOBMYBVQmOBtcfdR7p7t7tnNm9e8i4Z8smYz/UbNAGDKiF4cc1jDiCsSEZFklMjZ47lAm5j51sDq2A3cfTVwCYCZNQD6uvsmM8sFzoq77VsHUW+1s3D1JgY+PpM6tdOYdHVPjmjeIOqSREQkSSXS054NdDSzDmaWAfQDpsVuYGbNzKzkvm4DxoTTrwDfNbMm4Qlo3w2XCfBh7iYGjJ5JvfQ0nr6mlwJbRET2qdzQdvdC4HqCsF0ETHX3hWY20swuCjc7C1hsZkuAFsBd4W3XA3cSBP9sYGS4rMb7YOVGBjw+g4Z1a/P0NafQrmn9qEsSEZEkZ+57HWKOVHZ2tufk5ERdRqWa88UGhoyZRZP6GUwe0YtWWfWiLklERCJkZnPcPbu87RK6uIpUnFnL1zPoiZk0a1iHp69RYIuISOIU2lXo/c/WMXjMLA5rXJcpI3pxeGMFtoiIJE7XHq8i7376NcPHzaZNk0wmXd2L5g3rRF2SiIikGPW0q8Bbi9cy9KnZtG9anykjFNgiInJg1NOuZNMXfcWPJ8ylY4sGTBjWkyb1M6IuSUREUpRCuxK9snAN10+aS6fDGzF+aE8aZ6ZHXZKIiKQwDY9Xkhc//JLrJs7luJaNGT9MgS0iIgdPoV0Jps1fzQ2T59G1TRbjh/WgcT0FtoiIHDwNj1ew5+blcvPU+WS3P4SxQ06mfh01sYiIVAz1tCvQ1JyV/GzqfHod0ZQnr1Jgi4hIxVJoV5DJs1bwi2cWcPpRzXhi8MlkZiiwRUSkYilZKsD49z/n1/9eyLePac6jA7tTNz0t6pJERKQaUmgfpLHvLef3z3/MOZ1a8PAV3ahTW4EtIiKVQ6F9EEa/vYy7XlxE7+MO42/9u5FRW0cbRESk8ii0D9Ajby3lnpcXc+EJh/NAv66kpymwRUSkcim0D8CDr3/K/a8voU/XlvzlhydSW4EtIiJVQKG9H9yd+19bwt/eWMolJ7Xi3ktPJK2WRV2WiIjUEArtBLk797yymEff+ozLs9tw9yUnUEuBLSIiVUihnQB3548vLmL0O8u5omdb7uxzvAJbRESqXEIHY82st5ktNrOlZnZrKevbmtmbZjbPzBaY2QXh8gwzG2tmH5rZfDM7q4Lrr3TuzsgXPmb0O8sZfEo7/vADBbaIiESj3J62maUBDwPnArnAbDOb5u4fx2x2BzDV3R81s87Ai0B74GoAdz/BzA4FXjKzk929uIKfR6UoLnZ+M+0jJsxYwbDTO3DHhZ0wU2CLiEg0Eulp9wCWuvsyd88HpgB94rZxoFE43RhYHU53BqYDuPtaYCOQfbBFV4XiYuf25z5kwowVXHPmEQpsERGJXCKh3QpYGTOfGy6L9TtgoJnlEvSybwiXzwf6mFltM+sAdAfaxD+AmY0wsxwzy8nLy9vPp1DxioqdX/xzAVNmr+T6bx/Frb2PVWCLiEjkEgnt0tLK4+b7A0+6e2vgAmC8mdUCxhCEfA7wAPA/oHCvO3Mf5e7Z7p7dvHnz/am/whUWFXPLP+bzzJxcbjynIzd/92gFtoiIJIVEzh7PZc/ecWt2D3+XGAb0BnD3982sLtAsHBK/qWQjM/sf8OlBVVyJCouKuWnqfJ6fv5qfn3cM1337qKhLEhER+UYiPe3ZQEcz62BmGUA/YFrcNiuAswHMrBNQF8gzs0wzqx8uPxcojDuBLWkUFBVzw+R5PD9/Nbedf6wCW0REkk65PW13LzSz64FXgDRgjLsvNLORQI67TwNuBkab2U0EQ+dD3N3DM8ZfMbNiYBVwZaU9k4OQX1jM9ZPm8urHX3HHhZ0Y/q0joi5JRERkL+Yef3g6WtnZ2Z6Tk1Nlj7ezoIhrJ87ljU/W8vuLjmPwqe2r7LFFREQAzGyOu5f77aoafUW0nQVFjBg/h7eX5HHXxcdzRc92UZckIiJSphob2jvyi7h6XA7vffY1f+57Apef3DbqkkRERPapRob2tl2FDHtqNrOWr+e+S0+kb/fWUZckIiJSrhoX2lt3FXLV2FnM+WID91/elT5d468TIyIikpxqVGhv3lnAkDGzmJ+7if/rfxIXdjk86pJEREQSVmNCe9P2AgaNncXCVZt4eEA3eh+vwBYRkdRSI0J7w7Z8rhwzk8VrtvDowO6c27lF1CWJiIjst2of2uu27mLgE7P4LG8ro67M5tvHHhp1SSIiIgekWof2+m35DBg9k8/XbePxQdmccXS0/4xERETkYFTr0K5fJ40jmtfnN9/vzGlHNYu6HBERkYNSrUO7Tu00Hh3YPeoyREREKkQi/+VLREREkoBCW0REJEUotEVERFKEQltERCRFKLRFRERShEJbREQkRSi0RUREUoRCW0REJEWYu0ddwx7MLA/4Iuo6Kkkz4Ouoi0gyapO9qU32pPbYm9pkb6neJu3cvdxrbSddaFdnZpbj7tlR15FM1CZ7U5vsSe2xN7XJ3mpKm2h4XEREJEUotEVERFKEQrtqjYq6gCSkNtmb2mRPao+9qU32ViPaRMe0RUREUoR62iIiIilCoS0iIpIiFNqVxMzamNmbZrbIzBaa2U/D5YeY2Wtm9mn4u0nUtVYlM0szs3lm9kI438HMZobt8bSZZURdY1Uysywze8bMPgn3lVO0j9hN4WvmIzObbGZ1a9p+YmZjzGytmX0Us6zU/cICfzOzpWa2wMxOiq7yylNGm9wbvnYWmNlzZpYVs+62sE0Wm9l50VRd8RTalacQuNndOwG9gOvMrDNwKzDd3TsC08P5muSnwKKY+T8D94ftsQEYFklV0XkQeNndjwVOJGibGruPmFkr4CdAtrsfD6QB/ah5+8mTQO+4ZWXtF+cDHcOfEcCjVVRjVXuSvdvkNeB4d+8CLAFuAwjfa/sBx4W3ecTM0qqu1Mqj0K4k7v6lu88Np7cQvBm3AvoAT4WbPQX8IJoKq56ZtQYuBB4P5w34DvBMuElNa49GwBnAEwDunu/uG6nB+0ioNlDPzGoDmcCX1LD9xN3fBtbHLS5rv+gDjPPADCDLzA6vmkqrTmlt4u6vunthODsDaB1O9wGmuPsud18OLAV6VFmxlUihXQXMrD3QDZgJtHD3LyEIduDQ6Cqrcg8AvwCKw/mmwMaYF10uwQebmuIIIA8YGx4yeNzM6lOD9xF3XwXcB6wgCOtNwBxq9n5Soqz9ohWwMma7mto+Q4GXwulq2yYK7UpmZg2AfwI3uvvmqOuJipl9D1jr7nNiF5eyaU36DmJt4CTgUXfvBmyjBg2FlyY8TtsH6AC0BOoTDP/Gq0n7SXlq+usIM/sVwSHJiSWLStmsWrSJQrsSmVk6QWBPdPdnw8VflQxdhb/XRlVfFTsNuMjMPgemEAx3PkAwlFc73KY1sDqa8iKRC+S6+8xw/hmCEK+p+wjAOcByd89z9wLgWeBUavZ+UqKs/SIXaBOzXY1qHzMbDHwPuMJ3X3ik2raJQruShMdrnwAWuftfY1ZNAwaH04OBf1d1bVFw99vcvbW7tyc4QeQNd78CeBO4NNysxrQHgLuvAVaa2THhorOBj6mh+0hoBdDLzDLD11BJm9TY/SRGWfvFNGBQeBZ5L2BTyTB6dWdmvYFfAhe5+/aYVdOAfmZWx8w6EJykNyuKGiuarohWSczsdOAd4EN2H8O9neC49lSgLcEb1A/dPf6Ek2rNzM4CbnH375nZEQQ970OAecBAd98VZX1Vycy6EpyYlwEsA64i+DBdY/cRM/s9cDnBcOc8YDjB8cgas5+Y2WTgLIJ/N/kV8FvgX5SyX4Qfbh4iOEt6O3CVu+dEUXdlKqNNbgPqAOvCzWa4+4/C7X9FcJy7kODw5Evx95mKFNoiIiIpQsPjIiIiKUKhLSIikiIU2iIiIilCoS0iIpIiFNoiIiIpQqEtIiKSIhTaIiIiKeL/AZISAeoJIWjhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nn_three_neurons = [\n",
    "    (10, 10, 10),\n",
    "    (64, 64, 64),\n",
    "    (128, 128, 128)\n",
    "]\n",
    "\n",
    "nn_three_accuracies = []\n",
    "\n",
    "for n in nn_three_neurons:\n",
    "    nn_accuracies = cross_validate_six(n)\n",
    "    nn_mean_accuracy = np.mean(nn_accuracies)\n",
    "    nn_three_accuracies.append(nn_mean_accuracy)\n",
    "\n",
    "plt.figure(figsize=(8,4))\n",
    "plt.title(\"Mean Accuracy vs. Neurons In Three Hidden Layers\")\n",
    "\n",
    "x = [i[0] for i in nn_three_neurons]\n",
    "plt.plot(x, nn_three_accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary**\n",
    "\n",
    "Using 3 hidden layers improved our simple accuracy to 96%, even with 6-fold cross validation. This seems to be in line with the research literature out there about deep neural networks for computer vision. Having more layers and more neurons tends to improve the network's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Here are some next steps to try:**\n",
    "\n",
    "* Use the existing pipeline you've built to try other machine learning models:\n",
    "  * Decision Tree Models\n",
    "  * Random Forest Models\n",
    "* Write up a summary of the effectiveness of the different models you tried, with data to back up your conclusions.\n",
    "* Experiment with different activation functions to see if the accuracies improve.\n",
    "* Research some techniques for visualizing weights in neural network models and add a visualization step to your pipeline.\n",
    "   * [Visualization of MLP Weights on MNIST](https://scikit-learn.org/stable/auto_examples/neural_networks/plot_mnist_filters.html)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}