Predicting Seasonal Vaccine Status.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-21T13:09:47.889196Z",
     "start_time": "2022-04-21T13:09:47.726471Z"
    }
   },
   "source": [
    "![img](./images/Mayo_Clinic.Header.webp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:41:36.617018Z",
     "start_time": "2022-04-22T19:41:35.594907Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.dummy import DummyClassifier \n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier,ExtraTreesClassifier,\\\n",
    "                                VotingClassifier\n",
    "from sklearn.metrics import plot_confusion_matrix, plot_roc_curve, confusion_matrix,\\\n",
    "                    roc_curve, roc_auc_score, accuracy_score, recall_score, precision_score, f1_score\n",
    "\n",
    "import pickle\n",
    "\n",
    "from imblearn.pipeline import Pipeline as ImPipeline\n",
    "\n",
    "from numpy import sqrt, argmax\n",
    "\n",
    "import model2\n",
    "from model2 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pickle has been imported because models were ran in another notebook to save computing time in this notebook. \n",
    "\n",
    "The 'import model2' line is for a class ModelWithCV that has built in functions to run a cross validation and print accuracy, recall, precision, and f1 scores. \n",
    "\n",
    "All models were entered into the class in the source notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:41:54.889066Z",
     "start_time": "2022-04-22T19:41:54.883927Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ignoring Future Warnings. The version of python used in these notebooks is Python 3.8.5, the version of Sklearn is 1.0.2. Running this notebook with newer versions of either will lead to problems in regards to some methods and functions used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Mayo Clinic has tasked us with predicting how likely individuals are to receive the seasonal flu vaccine. \n",
    "We will concentrate on:\n",
    "- Predicting who will get the seasonal vaccine.\n",
    "- The most significant features in determining whether or not an individual will get vaccinated?.\n",
    "- How can we get more people to get vaccinated?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-19T16:02:53.075735Z",
     "start_time": "2022-04-19T16:02:53.069621Z"
    }
   },
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data was provided by DrivenData as a part of their competition. Details about the competition and getting the data can be found https://www.drivendata.org/competitions/66/flu-shot-learning/\n",
    "\n",
    "The data was provided in two separate files, which were combined for cleaning, splitting again later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:41:58.105161Z",
     "start_time": "2022-04-22T19:41:57.964131Z"
    }
   },
   "outputs": [],
   "source": [
    "training_set_features = pd.read_csv('./Data/training_set_features.csv')\n",
    "training_set_labels = pd.read_csv('./Data/training_set_labels.csv')\n",
    "complete_data = pd.merge(training_set_features, training_set_labels, on='respondent_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:41:59.021490Z",
     "start_time": "2022-04-22T19:41:58.981263Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "employment_occupation    13470\n",
       "employment_industry      13330\n",
       "health_insurance         12274\n",
       "income_poverty            4423\n",
       "doctor_recc_h1n1          2160\n",
       "doctor_recc_seasonal      2160\n",
       "rent_or_own               2042\n",
       "employment_status         1463\n",
       "marital_status            1408\n",
       "education                 1407\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "complete_data.isna().sum().sort_values(ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a considerable number of NA values in the dataset.\n",
    "\n",
    "If the employment_status column has a value of Not in Labor Force, then columns employment_industry and employment_occupation will have NA's. \n",
    "\n",
    "Values in columns employment_industry and employment_occupation are represented as short random character strings. \n",
    "\n",
    "We will be filling NA's in employment_occupation and employment_industry associated with \"Not in Labor Force\" with because they would otherwise be imputed as the most frequent occupation or industry.\n",
    "\n",
    "Filling Unemployed with \"Not in Labor Force\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:02.680504Z",
     "start_time": "2022-04-22T19:42:02.641409Z"
    }
   },
   "outputs": [],
   "source": [
    "complete_data['employment_status'].replace(\"Unemployed\", \"Not in Labor Force\", inplace=True)\n",
    "complete_data.loc[complete_data['employment_status'] == \"Not in Labor Force\", 'employment_industry'] = \"Not in Labor Force\"\n",
    "complete_data.loc[complete_data['employment_status'] == \"Not in Labor Force\", 'employment_occupation'] = \"Not in Labor Force\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:03.032228Z",
     "start_time": "2022-04-22T19:42:02.998533Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "health_insurance         12274\n",
       "income_poverty            4423\n",
       "doctor_recc_h1n1          2160\n",
       "doctor_recc_seasonal      2160\n",
       "rent_or_own               2042\n",
       "employment_occupation     1786\n",
       "employment_industry       1646\n",
       "employment_status         1463\n",
       "marital_status            1408\n",
       "education                 1407\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "complete_data.isna().sum().sort_values(ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:03.679831Z",
     "start_time": "2022-04-22T19:42:03.671706Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12274"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "complete_data['health_insurance'].isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:04.118072Z",
     "start_time": "2022-04-22T19:42:04.090579Z"
    }
   },
   "outputs": [],
   "source": [
    "complete_data['health_insurance'].fillna('Unknown', inplace=True)\n",
    "complete_data['health_insurance'].replace(0.0,'0', inplace=True)\n",
    "complete_data['health_insurance'].replace(1.0,'1', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the substantial number of NA's in Health Insurance we have decided to create a new category of \"Unknowns\". Replaced float values of 1.0 and 0.0 with string values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target Distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:05.684531Z",
     "start_time": "2022-04-22T19:42:05.673102Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.534392\n",
       "1    0.465608\n",
       "Name: seasonal_vaccine, dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "complete_data['seasonal_vaccine'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:05.900105Z",
     "start_time": "2022-04-22T19:42:05.889464Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Vax_dist = complete_data['seasonal_vaccine'].value_counts(normalize=True).mul(100).round(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:06.315229Z",
     "start_time": "2022-04-22T19:42:06.126284Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW8UlEQVR4nO3de7RdZX3u8e9jgkAlQZBAg6BRjBdsFY/BG9VDBTxYRdChIt7i7TDsRcR6Qw9DcfT0HIZV9Hhpa4pKqIiAVq5VibHRKook3NSCxVIUSkwicvdG8Hf+mO+Wxc5O9toha++E+f2Mscea652X97fWnvtZc71zrrVTVUiS+uMBM12AJGl6GfyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr+2SknuSPLIEW37S0kWj2LbQ/T9iiQXbsHt/SDJgW36hCSf2YLbfneSk7fU9rT1iNfxazJJvgJcXFXvGdd+OPAJYK+qWj8jxU0iyQnAo6rqldPQ1ynAy4Fft6YfA+cBJ1bVrZuxrRuq6vgprHMCm/lY24vHZ6pqr6muq22PR/waxinAq5JkXPurgNO21tCfIe+vqjnAPOC1wNOAbyV50JbsJMnsLbk99YvBr2GcDewKPHOsIckuwPOBU5M8Jcm3k9ySZHWSjyV54MCyj0+yLMnPk6xJ8u7WPqsNJ/xHktuTrEqyd5tXSR7Vpk9J8vEkF7TlLk6yz8D2/1+S65Pc1rbxzNZ+KPBu4Mg2dHRFa1+R5A1t+gFJjk/y4yRrk5yaZOc2b0GrY3GSnyT5WZL/NcwTVlW/qqpLgBcAD6F7ESDJa5J8s00nyYdav7cmuTLJHyQ5GngF8I5W93lt+euSvDPJlcCdSWa3toMHut4hyRntebo0yRMHnqffPacDz+v/bi9KXwL2bP3dkWTP8UNHSV7QhpZuac/h4wbmXZfkbe0x3Npq2GGY50rTz+DXpKrql8CZwKsHml8KXF1VVwB3A28BdgOeDhwE/BlAkjnAV4EvA3sCjwKWt238JXAU8CfAXOB1wC82UsZRwPuAXYAfAX89MO8SYD+6F6fPAmcl2aGqvgz8H+CMqtqpqp7Ihl7Tfv4YeCSwE/Cxccv8EfCY9rjeMxh4k6mq24FlDLxoDngO8Czg0cCDgSOBm6pqCXAa3buHnarqsIF1jgKeBzx4I++0DgfO4p7n4uwk201S453Ac4EbW387VdWNg8skeTRwOnAs3buZfwbOG3yBp9snDgUeATyB7nnVVsjg17CWAi9JsmO7/+rWRlWtqqrvVNX6qrqObtz/v7flng/8tKo+2I6Cb6+qi9u8NwDHV9UPq3NFVd20kf7/qaq+28LuNLqgp/X/maq6qfX/QWB7uqAexiuAk6rq2qq6A3gX8LJxQynvq6pfthe5K4CJXkA25Ua6IB7vLmAO8Fi6821XVdXqSbb1kaq6vr0YT2RVVX2+qu4CTgJ2oBtuuq+OBC6oqmVt2x8AdgSeMa62G6vq53TnNvbbAv1qBAx+DaWqvgmsAw5vV9vsT3dESZJHJzk/yU+T3EZ3lL1bW3Vv4D82stlNzRvvpwPTv6A7Mqf1/9YkV7UhhluAnQf6n8yedCdhx/wYmA3sMUzfQ3oo8PPxjVX1Nbp3Fx8H1iRZkmTuJNu6ftj5VfVb4Aa6x3hf3et5atu+nu6xjbmvz5OmicGvqTiV7kj/VcCFVbWmtf8dcDWwsKrm0o2rj50Ivh7YZ/yGhpg3lDae/066YYZdqurBwK0D/U922dqNwMMH7j8MWA+smXjxKde3E3Aw8K8Tza+qj1TVk4HH0w35vH1s1kY2Odnj2Xug7wcAe9E9RujC+PcGlv39KWz3Xs9TO9G/N/Bfk6ynrZDBr6k4lS7E/idtmKeZA9wG3JHkscCfDsw7H/j9JMcm2T7JnCRPbfNOBv4qycJ2ovMJSR4yxZrm0AX1OmB2kvfQnS8YswZY0EJwIqcDb0nyiBbSY+cE7tOVSu2xPpnuxPjNwKcnWGb/JE9tY/B3Ar+iO18yVvfmfI7hyUle1IaqjqW7tPQ7bd7lwMvTnVQ/lHuG48b6e8jYie0JnAk8L8lBrd63tm1ftBk1aoYZ/BpaG7+/CHgQcO7ArLfRXb9+O/APwBkD69wOHAIcRjcUcA3diVToxqDPBC6ke+H4JN248VR8he6KlH+nG4r4FfceDjmr3d6U5NIJ1v8U8I/AN4D/bOu/aYo1DHpHktvphnZOBVYBz2gnUMebS/d83dxqv4lu7By652LfdgXN2VPo/xy68fib6d6ZvaiNyQO8me73cAvduY3fbbeqrqZ7Eby29Xmv4aGq+iHwSuCjwM/adg6rqt9MoTZtJfwAlyT1jEf8ktQzBr8k9YzBL0k9Y/BLUs9sE1/0tNtuu9WCBQtmugxJ2qasWrXqZ1U1b3z7NhH8CxYsYOXKlTNdhiRtU5L8eKJ2h3okqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZ7aJT+5K92d77vmWmS5BW7Ebb/zQFt+mR/yS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMyP95G6S64DbgbuB9VW1KMmuwBnAAuA64KVVdfMo65Ak3WM6jvj/uKr2q6pF7f5xwPKqWggsb/clSdNkJoZ6DgeWtumlwBEzUIMk9daog7+AC5OsSnJ0a9ujqlYDtNvdJ1oxydFJViZZuW7duhGXKUn9Mepv5zygqm5MsjuwLMnVw65YVUuAJQCLFi2qURUoSX0z0iP+qrqx3a4Fvgg8BViTZD5Au107yhokSfc2suBP8qAkc8amgecA3wfOBRa3xRYD54yqBknShkY51LMH8MUkY/18tqq+nOQS4Mwkrwd+ArxkhDVIksYZWfBX1bXAEydovwk4aFT9SpI2zU/uSlLPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9cyov6tnxp142V0zXYK2Usc9abuZLkGaER7xS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMyMP/iSzklyW5Px2f9cky5Jc0253GXUNkqR7TMcR/5uBqwbuHwcsr6qFwPJ2X5I0TUYa/En2Ap4HnDzQfDiwtE0vBY4YZQ2SpHsb9RH/h4F3AL8daNujqlYDtNvdJ1oxydFJViZZuW7duhGXKUn9MbLgT/J8YG1Vrdqc9atqSVUtqqpF8+bN28LVSVJ/zR7htg8AXpDkT4AdgLlJPgOsSTK/qlYnmQ+sHWENkqRxRnbEX1Xvqqq9qmoB8DLga1X1SuBcYHFbbDFwzqhqkCRtaCau4z8ROCTJNcAh7b4kaZqMcqjnd6pqBbCiTd8EHDQd/UqSNuQndyWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4ZKviTHDBMmyRp6zfsEf9Hh2yTJG3lZm9qZpKnA88A5iX5y4FZc4FZoyxMkjQamwx+4IHATm25OQPttwEvHlVRkqTR2WTwV9XXga8nOaWqfjxNNUmSRmiyI/4x2ydZAiwYXKeqnr2xFZLsAHwD2L6t8/mqem+SXYEz2rauA15aVTdvTvGSpKkbNvjPAv4eOBm4e8h1fg08u6ruSLId8M0kXwJeBCyvqhOTHAccB7xzinVLkjbTsMG/vqr+biobrqoC7mh3t2s/BRwOHNjalwIrMPgladoMeznneUn+LMn8JLuO/Uy2UpJZSS4H1gLLqupiYI+qWg3QbnffyLpHJ1mZZOW6deuGLFOSNJlhj/gXt9u3D7QV8MhNrVRVdwP7JXkw8MUkfzBsYVW1BFgCsGjRohp2PUnSpg0V/FX1iPvSSVXdkmQFcCiwJsn8qlqdZD7duwFJ0jQZ9isbfi/J8e3KHpIsTPL8SdaZ1470SbIjcDBwNXAu97yDWAycs5m1S5I2w7BDPZ8GVtF9ihfgBrorfc7fxDrzgaVJZtG9wJxZVecn+TZwZpLXAz8BXrJZlUuSNsuwwb9PVR2Z5CiAqvplkmxqhaq6EnjSBO03AQdNuVJJ0hYx7FU9v2nDNQWQZB+66/QlSduYYY/43wt8Gdg7yWnAAcBrRlWUJGl0hr2qZ1mSS4GnAQHeXFU/G2llkqSRGPaqnhfSfXr3gqo6H1if5IiRViZJGolhx/jfW1W3jt2pqlvohn8kSduYYYN/ouWGPT8gSdqKDBv8K5OclGSfJI9M8iG66/olSduYYYP/TcBv6L5H/0zgl8Cfj6ooSdLoTDpc0z55e05VHTwN9UiSRmzSI/72DZu/SLLzNNQjSRqxYU/Q/gr4XpJlwJ1jjVV1zEiqkiSNzLDBf0H7kSRt44b95O7S9l09D6uqH464JknSCA37yd3DgMvpvq+HJPslOXeEdUmSRmTYyzlPAJ4C3AJQVZcD9+m/ckmSZsawwb9+8CsbGv8PriRtg4Y9ufv9JC8HZiVZCBwDXDS6siRJozKVT+4+nu6fr3wWuBU4dkQ1SZJGaJNH/El2AN4IPAr4HvD0qlo/HYVJkkZjsiP+pcAiutB/LvCBkVckSRqpycb4962qPwRI8kngu6MvSZI0SpMd8d81NuEQjyTdP0x2xP/EJLe16QA7tvsBqqrmjrQ6SdIWt8ngr6pZ01WIJGl6DHs5pyTpfsLgl6SeMfglqWcMfknqGYNfknrG4JeknhlZ8CfZO8m/JLkqyQ+SvLm175pkWZJr2u0uo6pBkrShUR7xrwfeWlWPA54G/HmSfYHjgOVVtRBY3u5LkqbJyIK/qlZX1aVt+nbgKuChwOF0X/5Guz1iVDVIkjY0LWP8SRYATwIuBvaoqtXQvTgAu29knaOTrEyyct26ddNRpiT1wsiDP8lOwBeAY6vqtsmWH1NVS6pqUVUtmjdv3ugKlKSeGWnwJ9mOLvRPq6p/as1rksxv8+cDa0dZgyTp3kZ5VU+ATwJXVdVJA7POBRa36cXAOaOqQZK0oWH/2frmOAB4FfC9JJe3tncDJwJnJnk98BPgJSOsQZI0zsiCv6q+Sfe9/RM5aFT9SpI2zU/uSlLPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzIwv+JJ9KsjbJ9wfadk2yLMk17XaXUfUvSZrYKI/4TwEOHdd2HLC8qhYCy9t9SdI0GlnwV9U3gJ+Paz4cWNqmlwJHjKp/SdLEpnuMf4+qWg3Qbnff2IJJjk6yMsnKdevWTVuBknR/t9We3K2qJVW1qKoWzZs3b6bLkaT7jekO/jVJ5gO027XT3L8k9d50B/+5wOI2vRg4Z5r7l6TeG+XlnKcD3wYek+SGJK8HTgQOSXINcEi7L0maRrNHteGqOmojsw4aVZ+SpMlttSd3JUmjYfBLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPzEjwJzk0yQ+T/CjJcTNRgyT11bQHf5JZwMeB5wL7Akcl2Xe665CkvpqJI/6nAD+qqmur6jfA54DDZ6AOSeql2TPQ50OB6wfu3wA8dfxCSY4Gjm5370jyw2morQ92A34200VsDd410wVoY9xHByQfvi+rP3yixpkI/kzQVhs0VC0Bloy+nH5JsrKqFs10HdLGuI+O3kwM9dwA7D1wfy/gxhmoQ5J6aSaC/xJgYZJHJHkg8DLg3BmoQ5J6adqHeqpqfZK/AL4CzAI+VVU/mO46eszhM23t3EdHLFUbDK9Lku7H/OSuJPWMwS9JPWPwz7AkK5L8j3Ftxyb52y2w7ZM391PRSQ5M8ozNWO+6JLttTp+afkkqyQcH7r8tyQmTrHPERPtV22e+Pa5tdpI1SeZvgVoXJfnIfVj/3ZuxzmuSfGxz+9xaGfwz73S6K5sGvay13ydV9Yaq+rfNXP1AYMrBr23Or4EXTfHF+gi6r1sZ7xvAXkkWDLQdDHy/qlZvdoVNVa2sqmPuwyamHPz3Vwb/zPs88Pwk2wO0P5o9gZcnWZnkB0neN7Zwkv2TXJTkiiTfTTInyawkH0jyvSRXJnlTW3ZFkkVt+o4kf93W+06SPVr7YUkuTnJZkq8m2aPV8EbgLUkuT/LMJPOSfCHJJe3ngLb+Q5Jc2Nb/BBN/QE9br/V0V9G8ZfyMJA9PsrztU8uTPKy9C3wB8Ddt39hnbPmq+i1wFnDkwGZeBpye5Cltv72s3T6m9bGxfXei/fzAJOe3+Sck+VTbx69NcsxA3WcnWdX+do5ubScCO7aaT2ttr2zbvjzJJ9r3iJHktUn+PcnXgQO23FO9Fakqf2b4B7gAOLxNHwf8DbBruz8LWAE8AXggcC2wf5s3l+6S3D8FvgDMbu1j664AFrXpAg5r0+8Hjm/Tu3DP1V1vAD7Ypk8A3jZQ42eBP2rTDwOuatMfAd7Tpp/X+tltpp9Tf4be9+5o+9F1wM7A24AT2rzzgMVt+nXA2W36FODFG9ne/sBlbXp7YG3bx+YO7J8HA19o0xvsu5vYzw8Ezh/YPy9qfewG3ARsN7aNdrsj8H3gIWOPdaDOx7XHN7bO3wKvBuYDPwHmtTq+BXxspn9PW/pnJr6yQRsaG+45p92+DnhpO1qZTbcz7ksXqqur6hKAqroNIMnBwN9X1frW/vMJ+vgNcH6bXgUc0qb3As5oY7APBP5zIzUeDOyb/O6Afm6SOcCzgBe1fi9IcvOUH71mVFXdluRU4BjglwOznk773QL/SHfAMNm2LkmyUzuifxzwnaq6OcnewNIkC+n24+3aKhvsu0n+kIn38/HdXVBVvwZ+nWQtsAfdNwMck+SFbZm9gYV0LwyDDgKeDFzStrsj3YvUU4EVVbWu9XkG8OjJHve2xuDfOpwNnJTkv9HtgDfTHXnt3/5oTgF2oBtGmeiDFxtrH3RXtUMb4G7u+d1/FDipqs5NciDdkdREHgA8vaoGg2Hsj9EPg2z7PgxcCnx6E8sM+3v+HN0BzOO451zVXwH/UlUvbEOJK1r7RPvuMPszdOcnxtwNzG778MF0++ovkqyg+9sZL8DSqrrXd/UlOWLIvrdpjvFvBarqDro/hE/R/aHMBe4Ebm1j8c9ti14N7Jlkf4A27jkbuBB4Y5smya5T6H5n4L/a9OKB9tuBOQP3LwT+YuxOkv3a5DeAV7S259K9rdc2pr1LPBN4/UDzRdxz4cErgG+26fH7xninA68Ens09X8cyuJ+9ZmDZifbdje3nw9gZuLmF/mOBpw3MuyvJ2DuN5cCLk+w+1m+ShwMXAwe2c1fbAS8Zst9tisG/9TgdeCLwuaq6ArgM+AHdi8G3AKr7/wVHAh9NcgWwjO5o5mS6cckrW/vLp9DvCcBZSf6Ve38V7nnAC8dO7tINAyxqJ+D+je7kL8D7gGcluRR4TqtD26YP0o2XjzkGeG2SK4FXAW9u7Z8D3t5O1O4zbhtUdyXZL4CvVdWdrfn9wP9N8i2681ZjNth3N7GfD+PLdEf+V9K9y/jOwLwlrZ/TWo3HAxe2ZZcB86u7+ugE4NvAV+neBd3v+JUNktQzHvFLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1zP8HAN1qhbnRGwsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "bar1 = plt.bar(0, Vax_dist[1], color='lightskyblue', label='Unvaxinated')\n",
    "bar2 = plt.bar(1, Vax_dist[0], color='midnightblue', label='Unvaxinated')\n",
    "ax.set_xticks([0,1])\n",
    "ax.set_xticklabels(['Vaccinated', 'Not Vaccinated'])\n",
    "ax.set_ylabel('Percent')\n",
    "ax.set_title('Vaccination Distribution')\n",
    "plt.savefig('./images/Vaccination_Distribution.jpeg', bbox_inches = 'tight')\n",
    "plt.savefig('./images/Vaccination_Distribution.png', bbox_inches = 'tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our target variable is distributed quite evenly. 47:53."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we separated out our target variable 'seasonal_vaccine' and dropped 'respondent_id' and 'h1n1_vaccine'.\n",
    "\n",
    "The 'values.ravel()' method reshapes y_train into a 1-D array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:07.749257Z",
     "start_time": "2022-04-22T19:42:07.711491Z"
    }
   },
   "outputs": [],
   "source": [
    "X = complete_data.drop(['h1n1_vaccine', 'seasonal_vaccine', 'respondent_id'], axis=1)\n",
    "y = complete_data.filter(['seasonal_vaccine'])\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "y_train = y_train.values.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipelines for Column Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All columns are in a categorical format, however, some are scales and may better be represented as numeric columns.\n",
    "\n",
    "Numeric columns include h1n1_concern, h1n1_knowledge, opinion_h1n1_vacc_effective, opinion_h1n1_risk, opinion_h1n1_sick_from_vacc, opinion_seas_vacc_effective, opinion_seas_risk, and opinion_seas_sick_from_vacc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:08.544759Z",
     "start_time": "2022-04-22T19:42:08.538441Z"
    }
   },
   "outputs": [],
   "source": [
    "num_cols = ['h1n1_concern', 'h1n1_knowledge', 'opinion_h1n1_vacc_effective',\\\n",
    "            'opinion_h1n1_risk', 'opinion_h1n1_sick_from_vacc',\\\n",
    "            'opinion_seas_vacc_effective', 'opinion_seas_risk', 'opinion_seas_sick_from_vacc']\n",
    "cat_cols = X.columns.drop(num_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:08.977357Z",
     "start_time": "2022-04-22T19:42:08.922452Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>h1n1_concern</th>\n",
       "      <th>h1n1_knowledge</th>\n",
       "      <th>opinion_h1n1_vacc_effective</th>\n",
       "      <th>opinion_h1n1_risk</th>\n",
       "      <th>opinion_h1n1_sick_from_vacc</th>\n",
       "      <th>opinion_seas_vacc_effective</th>\n",
       "      <th>opinion_seas_risk</th>\n",
       "      <th>opinion_seas_sick_from_vacc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>26615.000000</td>\n",
       "      <td>26591.000000</td>\n",
       "      <td>26316.000000</td>\n",
       "      <td>26319.000000</td>\n",
       "      <td>26312.000000</td>\n",
       "      <td>26245.000000</td>\n",
       "      <td>26193.000000</td>\n",
       "      <td>26170.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.618486</td>\n",
       "      <td>1.262532</td>\n",
       "      <td>3.850623</td>\n",
       "      <td>2.342566</td>\n",
       "      <td>2.357670</td>\n",
       "      <td>4.025986</td>\n",
       "      <td>2.719162</td>\n",
       "      <td>2.118112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.910311</td>\n",
       "      <td>0.618149</td>\n",
       "      <td>1.007436</td>\n",
       "      <td>1.285539</td>\n",
       "      <td>1.362766</td>\n",
       "      <td>1.086565</td>\n",
       "      <td>1.385055</td>\n",
       "      <td>1.332950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       h1n1_concern  h1n1_knowledge  opinion_h1n1_vacc_effective  \\\n",
       "count  26615.000000    26591.000000                 26316.000000   \n",
       "mean       1.618486        1.262532                     3.850623   \n",
       "std        0.910311        0.618149                     1.007436   \n",
       "min        0.000000        0.000000                     1.000000   \n",
       "25%        1.000000        1.000000                     3.000000   \n",
       "50%        2.000000        1.000000                     4.000000   \n",
       "75%        2.000000        2.000000                     5.000000   \n",
       "max        3.000000        2.000000                     5.000000   \n",
       "\n",
       "       opinion_h1n1_risk  opinion_h1n1_sick_from_vacc  \\\n",
       "count       26319.000000                 26312.000000   \n",
       "mean            2.342566                     2.357670   \n",
       "std             1.285539                     1.362766   \n",
       "min             1.000000                     1.000000   \n",
       "25%             1.000000                     1.000000   \n",
       "50%             2.000000                     2.000000   \n",
       "75%             4.000000                     4.000000   \n",
       "max             5.000000                     5.000000   \n",
       "\n",
       "       opinion_seas_vacc_effective  opinion_seas_risk  \\\n",
       "count                 26245.000000       26193.000000   \n",
       "mean                      4.025986           2.719162   \n",
       "std                       1.086565           1.385055   \n",
       "min                       1.000000           1.000000   \n",
       "25%                       4.000000           2.000000   \n",
       "50%                       4.000000           2.000000   \n",
       "75%                       5.000000           4.000000   \n",
       "max                       5.000000           5.000000   \n",
       "\n",
       "       opinion_seas_sick_from_vacc  \n",
       "count                 26170.000000  \n",
       "mean                      2.118112  \n",
       "std                       1.332950  \n",
       "min                       1.000000  \n",
       "25%                       1.000000  \n",
       "50%                       2.000000  \n",
       "75%                       4.000000  \n",
       "max                       5.000000  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[num_cols].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:09.195824Z",
     "start_time": "2022-04-22T19:42:09.170947Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "h1n1_concern                   2.0\n",
       "h1n1_knowledge                 1.0\n",
       "opinion_h1n1_vacc_effective    4.0\n",
       "opinion_h1n1_risk              2.0\n",
       "opinion_h1n1_sick_from_vacc    2.0\n",
       "opinion_seas_vacc_effective    4.0\n",
       "opinion_seas_risk              2.0\n",
       "opinion_seas_sick_from_vacc    2.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[num_cols].median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there are some missing values in the numeric columns, it is important determine the best measure of central tendency to replace these values. Because the values are scaler in nature we have decided to use the 'median' of each column to fill in these values.  \n",
    "\n",
    "For categorical variables, the most frequent value for each column will be used to fill in the missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:10.195946Z",
     "start_time": "2022-04-22T19:42:10.188459Z"
    }
   },
   "outputs": [],
   "source": [
    "subpipe_cat = Pipeline(steps=[('cat_impute', SimpleImputer(strategy='most_frequent')),\n",
    "                              ('ohe', OneHotEncoder(drop='first', sparse=False, handle_unknown='ignore')) ])\n",
    "\n",
    "subpipe_num = Pipeline(steps=[('num_impute', SimpleImputer(strategy='median')),\n",
    "                              ('ss', StandardScaler()) ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:10.518044Z",
     "start_time": "2022-04-22T19:42:10.513961Z"
    }
   },
   "outputs": [],
   "source": [
    "ct = ColumnTransformer(transformers = [\n",
    "    ('subpipe_cat', subpipe_cat, cat_cols),\n",
    "    ('subpipe_num', subpipe_num, num_cols)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-20T13:55:25.299557Z",
     "start_time": "2022-04-20T13:55:25.278401Z"
    }
   },
   "source": [
    "Now that our Column Transformer pipeline has been created we can start running models. \n",
    "- First Models were executed with their default parameters. \n",
    "- Then they were run through a GridSearchCV to find the best parameter combos. To save time, no more than a few parameters were chosen to be tuned for each model. Because of our business problem, only models with the feature_importances_ attribute were picked.\n",
    "- After tuning these models they were entered into ensembles as VotingClassifiers. In the end a VotingClassifier with just two estimators, Random Forest and Extra Trees, performed just as well as a VotingClassifier with five estimators. Due to the difference in computation time the simpler model was selected as the final model. \n",
    "\n",
    "Below you will see the process and results for the estimators and the final model selected. You can find the process and results for all other models in the source notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dummy Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All models were executed in the following format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:12.530997Z",
     "start_time": "2022-04-22T19:42:12.523494Z"
    }
   },
   "outputs": [],
   "source": [
    "dummy_model_pipe = Pipeline(steps=[\n",
    "    ('ct', ct),\n",
    "    ('dummy', DummyClassifier(random_state=42))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:13.400527Z",
     "start_time": "2022-04-22T19:42:13.127820Z"
    }
   },
   "outputs": [],
   "source": [
    "dummy_model_pipe.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:22.716457Z",
     "start_time": "2022-04-22T19:42:14.145325Z"
    }
   },
   "outputs": [],
   "source": [
    "dummy_pipe = ModelWithCV(dummy_model_pipe, model_name='dummy_pipe', X=X_train, y=y_train) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:22.719092Z",
     "start_time": "2022-04-22T19:42:22.717514Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `dummy_pipe` model:\n",
      "            0.53105 ± 0.00015 accuracy\n",
      "            recall_score: 0.0\n",
      "            precision_score: 0.46895434857961904\n",
      "            f1_score: 0.0    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "dummy_pipe.print_cv_summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'print_cv_summary' function comes from the ModelWithCV class where the model runs a 10 fold cross validation to return accuracy, recall, precision, and f1 scores.\n",
    "\n",
    "Our dummy classifier predicts the most common value for the target value. It's accuracy is at about 53%, which is to be expected given the target distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first Estimator to be run was SkLearn's Random Forest Classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:25.323050Z",
     "start_time": "2022-04-22T19:42:25.296751Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('./Data/forest.sav', 'rb') as f:\n",
    "    forest = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T19:42:27.456734Z",
     "start_time": "2022-04-22T19:42:27.451099Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `forest_pipe` model:\n",
      "            0.77153 ± 0.01219 accuracy\n",
      "            recall_score: 0.7391263692604574\n",
      "            precision_score: 0.8263930425720363\n",
      "            f1_score: 0.7521140408449998    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "forest.print_cv_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T14:10:46.772888Z",
     "start_time": "2022-04-22T14:10:46.760396Z"
    }
   },
   "source": [
    "The Random Forest Classifier performed considerably better than the Dummy Classifier Model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Random Forest  Tuned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our Grid Search determined the optimal parameters for Random Forest was max_features= 'sqrt' and n_estimators= 1250."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.570409Z",
     "start_time": "2022-04-22T17:24:28.559910Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('./Data/tuned_forest.sav', 'rb') as f:\n",
    "    tforest = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.574782Z",
     "start_time": "2022-04-22T17:24:28.571327Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `tforest_pipe` model:\n",
      "            0.77706 ± 0.01164 accuracy\n",
      "            recall_score: 0.7499721891140246\n",
      "            precision_score: 0.8336055852918713\n",
      "            f1_score: 0.759324132932402    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "tforest.print_cv_summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Tuned Random Forest Classifier does perform slightly better on each scoring metric. However, we are concentrated primarily on accuracy scores. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extra Trees Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next Estimator is SkLearn's Extra Trees Classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.588252Z",
     "start_time": "2022-04-22T17:24:28.577608Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('./Data/etc.sav', 'rb') as f:\n",
    "    etc = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.593979Z",
     "start_time": "2022-04-22T17:24:28.589201Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `etc_pipe` model:\n",
      "            0.77058 ± 0.01184 accuracy\n",
      "            recall_score: 0.733285941313355\n",
      "            precision_score: 0.8226913437164194\n",
      "            f1_score: 0.7498575863366292    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "etc.print_cv_summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again this classifier perform substantially better than the Dummy Classifier and just about as well as the Random Forest Classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  Extra Trees Tuned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our Grid Search determined the optimal parameters for Gradient Booster was max_depth = 50 and n_estimators = 500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.610166Z",
     "start_time": "2022-04-22T17:24:28.595556Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('./Data/tuned_etc.sav', 'rb') as f:\n",
    "    tuned_etc = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.614048Z",
     "start_time": "2022-04-22T17:24:28.611274Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `tetc_pipe` model:\n",
      "            0.77399 ± 0.01175 accuracy\n",
      "            recall_score: 0.740079743458766\n",
      "            precision_score: 0.8302410092727717\n",
      "            f1_score: 0.7543782442015724    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "tuned_etc.print_cv_summary() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Tuned Extra Trees Classifier does perform slightly better on each scoring metric. But again we are concentrated primarily on accuracy scores. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensembled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tuned Random Forest model and tuned Extra Trees model were entered into a Voting Classifier. Below the results of that Voting Classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.627223Z",
     "start_time": "2022-04-22T17:24:28.615844Z"
    }
   },
   "outputs": [],
   "source": [
    "with open('./Data/avg2_model.sav', 'rb') as f:\n",
    "    avg2_model = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:28.630344Z",
     "start_time": "2022-04-22T17:24:28.628493Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV Results for `avg2_pipe` model:\n",
      "            0.77718 ± 0.01159 accuracy\n",
      "            recall_score: 0.7463972132357114\n",
      "            precision_score: 0.8337415272006232\n",
      "            f1_score: 0.7585331782501775    \n",
      "        \n"
     ]
    }
   ],
   "source": [
    "avg2_model.print_cv_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Voting Classifier does slightly better than both of it's estimators did individually. It may be beneficial to tune again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our business problem it was necessary to extract the most important features. \n",
    "- First we extract the One Hot Encoded column names (ohe_cat_cols)\n",
    "- The because the numeric columns were after the One Hot Encoding we add the list of numeric column names (num_cols) to the list of One Hot Encoded columns (ohe_cat_cols) to get the column names after the Column Transformer has been executed (post_ct_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:50.887400Z",
     "start_time": "2022-04-22T17:24:28.632367Z"
    }
   },
   "outputs": [],
   "source": [
    "avg2_fit = avg2_model.model.fit(X_train, y_train); #22s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:50.890204Z",
     "start_time": "2022-04-22T17:24:50.888415Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ohe_col_names = avg2_model.model.named_steps['ct'].named_transformers_['subpipe_cat']['ohe'].get_feature_names_out()\n",
    "\n",
    "ohe_cat_cols = ohe_col_names.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:50.893157Z",
     "start_time": "2022-04-22T17:24:50.891096Z"
    }
   },
   "outputs": [],
   "source": [
    "post_ct_cols = ohe_cat_cols + num_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we make a Data Frame using the array that is created after the Column Transformer and name the columns from the list created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:51.123430Z",
     "start_time": "2022-04-22T17:24:50.894559Z"
    }
   },
   "outputs": [],
   "source": [
    "X_post = pd.DataFrame(avg2_model.model.named_steps['ct'].fit_transform(X_train, y_train), columns=post_ct_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:24:51.140901Z",
     "start_time": "2022-04-22T17:24:51.124553Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x0_1.0</th>\n",
       "      <th>x1_1.0</th>\n",
       "      <th>x2_1.0</th>\n",
       "      <th>x3_1.0</th>\n",
       "      <th>x4_1.0</th>\n",
       "      <th>x5_1.0</th>\n",
       "      <th>x6_1.0</th>\n",
       "      <th>x7_1.0</th>\n",
       "      <th>x8_1.0</th>\n",
       "      <th>x9_1.0</th>\n",
       "      <th>...</th>\n",
       "      <th>x26_xtkaffoo</th>\n",
       "      <th>x26_xzmlyyjv</th>\n",
       "      <th>h1n1_concern</th>\n",
       "      <th>h1n1_knowledge</th>\n",
       "      <th>opinion_h1n1_vacc_effective</th>\n",
       "      <th>opinion_h1n1_risk</th>\n",
       "      <th>opinion_h1n1_sick_from_vacc</th>\n",
       "      <th>opinion_seas_vacc_effective</th>\n",
       "      <th>opinion_seas_risk</th>\n",
       "      <th>opinion_seas_sick_from_vacc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.686593</td>\n",
       "      <td>1.197462</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-0.270422</td>\n",
       "      <td>-0.265923</td>\n",
       "      <td>-0.030392</td>\n",
       "      <td>0.934322</td>\n",
       "      <td>-0.089462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.516834</td>\n",
       "      <td>1.197462</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>1.291698</td>\n",
       "      <td>-1.002707</td>\n",
       "      <td>0.903974</td>\n",
       "      <td>1.660931</td>\n",
       "      <td>-0.845239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.516834</td>\n",
       "      <td>1.197462</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-1.051483</td>\n",
       "      <td>-1.002707</td>\n",
       "      <td>0.903974</td>\n",
       "      <td>0.934322</td>\n",
       "      <td>-0.845239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.686593</td>\n",
       "      <td>1.197462</td>\n",
       "      <td>1.147458</td>\n",
       "      <td>1.291698</td>\n",
       "      <td>-0.265923</td>\n",
       "      <td>0.903974</td>\n",
       "      <td>-0.518896</td>\n",
       "      <td>-0.089462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.686593</td>\n",
       "      <td>1.197462</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>1.291698</td>\n",
       "      <td>-0.265923</td>\n",
       "      <td>-0.030392</td>\n",
       "      <td>0.934322</td>\n",
       "      <td>-0.089462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17888</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.415121</td>\n",
       "      <td>-0.425980</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-0.270422</td>\n",
       "      <td>-0.265923</td>\n",
       "      <td>-0.030392</td>\n",
       "      <td>-0.518896</td>\n",
       "      <td>-0.845239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17889</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.686593</td>\n",
       "      <td>-0.425980</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-0.270422</td>\n",
       "      <td>-1.002707</td>\n",
       "      <td>-0.030392</td>\n",
       "      <td>-0.518896</td>\n",
       "      <td>1.422094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17890</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.415121</td>\n",
       "      <td>-0.425980</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-0.270422</td>\n",
       "      <td>-0.265923</td>\n",
       "      <td>-0.030392</td>\n",
       "      <td>-0.518896</td>\n",
       "      <td>-0.089462</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17891</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.415121</td>\n",
       "      <td>-0.425980</td>\n",
       "      <td>-2.853217</td>\n",
       "      <td>-1.051483</td>\n",
       "      <td>1.207646</td>\n",
       "      <td>-2.833489</td>\n",
       "      <td>-1.245505</td>\n",
       "      <td>1.422094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17892</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.516834</td>\n",
       "      <td>-0.425980</td>\n",
       "      <td>0.147289</td>\n",
       "      <td>-0.270422</td>\n",
       "      <td>1.207646</td>\n",
       "      <td>0.903974</td>\n",
       "      <td>-0.518896</td>\n",
       "      <td>-0.089462</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17893 rows × 99 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       x0_1.0  x1_1.0  x2_1.0  x3_1.0  x4_1.0  x5_1.0  x6_1.0  x7_1.0  x8_1.0  \\\n",
       "0         0.0     1.0     0.0     1.0     0.0     0.0     1.0     1.0     1.0   \n",
       "1         0.0     0.0     0.0     1.0     0.0     0.0     1.0     0.0     0.0   \n",
       "2         0.0     1.0     0.0     1.0     0.0     0.0     1.0     0.0     0.0   \n",
       "3         0.0     1.0     0.0     1.0     0.0     0.0     1.0     0.0     0.0   \n",
       "4         0.0     1.0     0.0     1.0     0.0     0.0     1.0     0.0     1.0   \n",
       "...       ...     ...     ...     ...     ...     ...     ...     ...     ...   \n",
       "17888     0.0     0.0     0.0     1.0     0.0     0.0     1.0     0.0     0.0   \n",
       "17889     0.0     0.0     0.0     1.0     0.0     0.0     1.0     0.0     0.0   \n",
       "17890     0.0     1.0     0.0     0.0     0.0     0.0     1.0     0.0     0.0   \n",
       "17891     0.0     1.0     0.0     1.0     0.0     0.0     1.0     1.0     1.0   \n",
       "17892     0.0     0.0     0.0     1.0     0.0     1.0     1.0     0.0     0.0   \n",
       "\n",
       "       x9_1.0  ...  x26_xtkaffoo  x26_xzmlyyjv  h1n1_concern  h1n1_knowledge  \\\n",
       "0         1.0  ...           0.0           0.0     -0.686593        1.197462   \n",
       "1         1.0  ...           0.0           0.0      1.516834        1.197462   \n",
       "2         0.0  ...           0.0           0.0      1.516834        1.197462   \n",
       "3         0.0  ...           1.0           0.0     -0.686593        1.197462   \n",
       "4         0.0  ...           0.0           0.0     -0.686593        1.197462   \n",
       "...       ...  ...           ...           ...           ...             ...   \n",
       "17888     0.0  ...           0.0           0.0      0.415121       -0.425980   \n",
       "17889     0.0  ...           0.0           0.0     -0.686593       -0.425980   \n",
       "17890     0.0  ...           0.0           0.0      0.415121       -0.425980   \n",
       "17891     1.0  ...           0.0           0.0      0.415121       -0.425980   \n",
       "17892     0.0  ...           0.0           0.0      1.516834       -0.425980   \n",
       "\n",
       "       opinion_h1n1_vacc_effective  opinion_h1n1_risk  \\\n",
       "0                         0.147289          -0.270422   \n",
       "1                         0.147289           1.291698   \n",
       "2                         0.147289          -1.051483   \n",
       "3                         1.147458           1.291698   \n",
       "4                         0.147289           1.291698   \n",
       "...                            ...                ...   \n",
       "17888                     0.147289          -0.270422   \n",
       "17889                     0.147289          -0.270422   \n",
       "17890                     0.147289          -0.270422   \n",
       "17891                    -2.853217          -1.051483   \n",
       "17892                     0.147289          -0.270422   \n",
       "\n",
       "       opinion_h1n1_sick_from_vacc  opinion_seas_vacc_effective  \\\n",
       "0                        -0.265923                    -0.030392   \n",
       "1                        -1.002707                     0.903974   \n",
       "2                        -1.002707                     0.903974   \n",
       "3                        -0.265923                     0.903974   \n",
       "4                        -0.265923                    -0.030392   \n",
       "...                            ...                          ...   \n",
       "17888                    -0.265923                    -0.030392   \n",
       "17889                    -1.002707                    -0.030392   \n",
       "17890                    -0.265923                    -0.030392   \n",
       "17891                     1.207646                    -2.833489   \n",
       "17892                     1.207646                     0.903974   \n",
       "\n",
       "       opinion_seas_risk  opinion_seas_sick_from_vacc  \n",
       "0               0.934322                    -0.089462  \n",
       "1               1.660931                    -0.845239  \n",
       "2               0.934322                    -0.845239  \n",
       "3              -0.518896                    -0.089462  \n",
       "4               0.934322                    -0.089462  \n",
       "...                  ...                          ...  \n",
       "17888          -0.518896                    -0.845239  \n",
       "17889          -0.518896                     1.422094  \n",
       "17890          -0.518896                    -0.089462  \n",
       "17891          -1.245505                     1.422094  \n",
       "17892          -0.518896                    -0.089462  \n",
       "\n",
       "[17893 rows x 99 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_post"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next this for loop creates a dictionary with the an array of feature importance for each estimator. \n",
    "\n",
    "We then get the average feature importance by adding the arrays together and dividing them by two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:13.901510Z",
     "start_time": "2022-04-22T17:24:51.142635Z"
    }
   },
   "outputs": [],
   "source": [
    "feat_imp = {}\n",
    "for est in avg2_model.model.named_steps['avg2'].named_estimators:\n",
    "    etc = avg2_model.model.named_steps['avg2'].named_estimators[est]\n",
    "    feat_imp[etc] = etc.fit(X_post, y_train).feature_importances_ #22s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:13.905601Z",
     "start_time": "2022-04-22T17:25:13.902995Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "avg_feat_imp = (list(feat_imp.values())[0] + list(feat_imp.values())[1])/2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we create a Data Frame with the columns representing the Features and their Importances. We then sort values by Feature Importance and create a new Data Frame with just the top ten feature importances, clean the value names up a bit and make a bar chart for visual purposes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:13.928478Z",
     "start_time": "2022-04-22T17:25:13.907046Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame()\n",
    "df['Features'] = post_ct_cols\n",
    "df['Feature_importance'] = avg_feat_imp\n",
    "top_ten_feat = df.sort_values('Feature_importance', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:13.934066Z",
     "start_time": "2022-04-22T17:25:13.929691Z"
    }
   },
   "outputs": [],
   "source": [
    "top_ten_feat.Features = [row.strip().lower().replace('_',' ') for row in top_ten_feat.Features]\n",
    "top_ten_feat['Features'].replace('x8 1.0', 'Doctor Recommended', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:13.943487Z",
     "start_time": "2022-04-22T17:25:13.935393Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Features</th>\n",
       "      <th>Feature_importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>opinion seas risk</td>\n",
       "      <td>0.077815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>opinion seas vacc effective</td>\n",
       "      <td>0.075493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Doctor Recommended</td>\n",
       "      <td>0.068744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>x13 65+ years</td>\n",
       "      <td>0.036875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>opinion h1n1 risk</td>\n",
       "      <td>0.032015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>opinion h1n1 vacc effective</td>\n",
       "      <td>0.029749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>opinion seas sick from vacc</td>\n",
       "      <td>0.026720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>h1n1 concern</td>\n",
       "      <td>0.025773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>opinion h1n1 sick from vacc</td>\n",
       "      <td>0.024122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>h1n1 knowledge</td>\n",
       "      <td>0.021408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Features  Feature_importance\n",
       "97            opinion seas risk            0.077815\n",
       "96  opinion seas vacc effective            0.075493\n",
       "8            Doctor Recommended            0.068744\n",
       "17                x13 65+ years            0.036875\n",
       "94            opinion h1n1 risk            0.032015\n",
       "93  opinion h1n1 vacc effective            0.029749\n",
       "98  opinion seas sick from vacc            0.026720\n",
       "91                 h1n1 concern            0.025773\n",
       "95  opinion h1n1 sick from vacc            0.024122\n",
       "92               h1n1 knowledge            0.021408"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_ten_feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:14.139141Z",
     "start_time": "2022-04-22T17:25:13.944751Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAEFCAYAAAACOoX/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2JElEQVR4nO3de7ync73//8dzZhQGg8hOYZJTZMww45BDyNZBtSklOeewpzaSnw57J6T8Cntnl4rGlHO2aJRGO2OLGWTMLOboWKEohTCMw2Dm+f3jen/4+PRZa33WmrXWLPN53m+3dXN93ufrmm69rut9Hd6yTURERLSHIct6ABERETFwEvgjIiLaSAJ/REREG0ngj4iIaCMJ/BEREW0kgT8iIqKNJPBHRES0kQT+iOhzkh6U9LykhXV/6/ZBm3v01Rhb6O8USZcMVH9dkXSopJuX9Thi+ZDAHxH95cO2V6n7+8uyHIykYcuy/956vY47Bq8E/ogYMJJGSPqRpEck/VnSNyQNLXnvkPQbSX+X9LikSyWtXvIuBtYHfllmD74oaVdJDze0/8qsQLliv1LSJZKeBg7tqv8Wxm5Jn5X0O0nPSPp6GfOtkp6W9FNJbyhld5X0sKT/KPvyoKQDGo7DRZIek/RHSSdKGlLyDpV0i6SzJD0BXA6cC+xQ9v2pUm4vSbNK3w9JOqWu/ZFlvIdI+lMZw1fq8oeWsf2h7MvtktYreZtJuk7SE5LulfSJHv0jx6CXwB8RA+lC4GVgI2AMsCdwRMkT8E1gXeCdwHrAKQC2DwL+xKuzCGe02N+/AFcCqwOXdtN/K94PbANsD3wRmAAcUMb6LmD/urL/BKwFvBU4BJggadOSdzYwAtgQeA9wMHBYXd3tgPuBNwMHAuOBW8u+r17KPFvqrQ7sBXxG0t4N490J2BR4L3CSpHeW9OPLWD8IrAZ8GnhO0nDgOuAnpe/9gR9I2qL1QxSDXQJ/RPSXn0t6qvz9XNI6wAeA42w/a/tR4CzgkwC2f2/7OtuLbD8GfJsqKC6NW23/3PYSqgDXaf8tOt3207bvBOYDU2zfb3sB8L9UJxP1vlr2ZypwDfCJMsOwH/Dvtp+x/SDwX8BBdfX+Yvts2y/bfr7ZQGzfaHue7SW25wKX8Y/H62u2n7c9B5gDbFXSjwBOtH2vK3Ns/x34EPCg7fNL33cAPwP27cExikEu944ior/sbfv/aj8kbQusADwiqZY8BHio5L8Z+C6wM7BqyXtyKcfwUN32Bl3136K/1W0/3+T3P9X9ftL2s3W//0g1m7EW8Ibyuz7vrZ2MuylJ2wHfopppeAPwRuCKhmJ/rdt+DlilbK8H/KFJsxsA29VuJxTDgIu7G0+8fuSKPyIGykPAImAt26uXv9Vs16aRvwkYGGV7NaopbtXVb1xK9Flg5dqPciW9dkOZ+jrd9d/X1ihT5zXrA38BHgdeogqy9Xl/7mTczX5DNR1/NbCe7RFUzwGoSblmHgLe0Un61Lrjs3q5vfCZFtuN14EE/ogYELYfAaYA/yVpNUlDysNxtenpVYGFwFOS3gp8oaGJv1HdE6+5D1ixPOS2AnAi1VVvb/vvD1+T9AZJO1NNo19hezHwU+A0SatK2oDqnntXrw7+DXhb7eHBYlXgCdsvlNmUT/VgXBOBr0vaWJVRkt4ETAY2kXSQpBXK37i6ZwNiOZDAHxED6WCqaem7qKbxrwTeUvK+BmwNLKC6Hz6poe43gRPLMwMnlPvqn6UKYn+mmgF4mK511X9f+2vp4y9UDxaOt31PyTuGarz3AzdTXb3/uIu2fgPcCfxV0uMl7bPAqZKeAU6iOplo1bdL+SnA08CPgJVsP0P1wOMny7j/CpxOFydU8foju9kMUkRE9JakXYFLbL9tGQ8l4h/kij8iIqKNJPBHRES0kUz1R0REtJFc8UdERLSRfMAnBrW11lrLI0eOXNbDiIh4Xbn99tsft934XQsggT8GuZEjR9LR0bGshxER8boi6Y+d5WWqPyIioo0k8EdERLSRBP6IiIg2ksAfERHRRhL4IyIi2kgCf0RERBtJ4I+IiGgjCfwRERFtJN/qj0HtDW9Yx2ut9allPYyIiAH1l7+ctVT1Jd1ue2yzvFzxR0REtJEE/oiIiDaSwB8REdFGEvgjIiLaSAL/AJI0UdLm3ZQZL+nggRpTX5O0rqQru8gfKWn+QI4pIiJelWV5B5DtI1ooc+5AjKU/SBpm+y/Avst6LBER0Vyu+JeCpOMlzS9/x5W0kZLukXShpLmSrpS0csm7UdLYsr1Q0mmS5kiaLmmdkn6KpBPK9uiSN1fSVZLWqGvndEkzJN0naecmY3uLpGmSZpfx7VzS95R0q6Q7JF0haZWSfpKkmaXsBEkq6cdKuquM4X+a9HNoaeeXwJT6K3pJW5Qxzi71N26ou6GkWZLG9c2/SEREdCeBv5ckbQMcBmwHbA8cKWlMyd4UmGB7FPA08NkmTQwHptveCpgGHNmkzEXAl0o784CT6/KG2d4WOK4hveZTwLW2RwNbAbMlrQWcCOxhe2ugAzi+lP+e7XG23wWsBHyopH8ZGFPGML6Tw7EDcIjt3RvSxwPfKWMYCzxcy5C0KfAz4DDbM+srSTpKUoekjiVLnu+ky4iI6I0E/t7bCbjK9rO2FwKTgNqV90O2bynbl5SyjV4EJpft24GR9ZmSRgCr255aki4EdqkrMqmzusVM4DBJpwBb2n6G6gRlc+AWSbOBQ4ANSvndJN0maR6wO7BFSZ8LXCrpQODlJv0AXGf7iSbptwL/IelLwAa2a1F8beAXwIG2ZzdWsj3B9ljbY4cMWamTLiMiojcS+HtPXeQ1fg6x2ecRX/Krn01cTM+ft1jUVV3b06hOFP4MXFweGBRVkB5d/ja3fbikFYEfAPva3hI4D1ixNLUX8H1gG+B2Sc3G+WyzAdr+CfAR4HngWkm1GYEFwEPAjj3c54iIWEoJ/L03Ddhb0sqShgP7ADeVvPUl7VC29wdu7mnjthcAT9bdvz8ImNpFldeQtAHwqO3zgB8BWwPTgR0lbVTKrCxpE14N8o+Xe/77lvwhwHq2bwC+CKwOrNKDMWwI3G/7u8DVwKiS9SKwN3CwpHyPNyJiAOWp/l6yfYekC4AZJWmi7VmSRgJ3A4dI+iHwO+CcXnZzCHBueTjwfqpnClq1K/AFSS8BC4GDbT8m6VDgMklvLOVOtH2fpPOoniN4kOo2AcBQ4JJy20HAWbaf6sEY9gMOLGP4K3AqsBqA7WclfQi4TtKztn/Rg3YjIqKXskhPHyuBf3J5SC6WUhbpiYh2lEV6IiIiok9kqr+P2X4QyNV+REQMSgn8MaiNGrUeHR1LN+UVERGvylR/REREG0ngj4iIaCMJ/BEREW0kr/PFoJbX+SJiebO0r+q1Iq/zRUREBJDAHxER0VYS+CMiItpIAn9EREQbGdSBX9JESZt3U2Z8WXI2WiTpWEl3S7pU0hsl/Z+k2ZL262E7I+tX15M0VtJ3+37EERHRVwb1l/tsH9FCmXMHYizLmc8CH7D9gKTtgRVsj+5FOyOBTwE/AbDdAXT01SAjIqLvDegVv6TjJc0vf8eVtJGS7pF0oaS5kq4sy9Ai6UZJY8v2QkmnSZojabqkdUr6KZJOKNujS95cSVdJWqOundMlzZB0X90a9/Vje4ukaeXKd36tjKQ9Jd0q6Q5JV5T16pF0kqSZpewESSrpx0q6q4zhf5r0c5ukLep+3yhpG0nbSvqtpFnlv5uW/KGS/lPSvNLmMSV9XCk3p+zXqk36+kIZ41xJXytp5wIbAldL+hJwCTC67Pc7ylimSrpd0rWS3lLqbVRmBuaUY/EO4FvAzqXu5yXtKmmypCGSHpS0et1Yfi9pHUlrS/pZGddMSTu2/r+giIhYWgMW+CVtQ7We/HbA9sCRksaU7E2BCbZHAU9TXZE2Gg5Mt70VMA04skmZi4AvlXbmASfX5Q2zvS1wXEN6zaeAa8uV71bAbElrAScCe9jemupq9vhS/nu2x5Xld1cCPlTSvwyMKWMY36Sf/wE+UY7JW4B1bd8O3APsYnsMcBLw/5fyRwFvr2vzUklvAC4HPleOxx7A8/WdSNoT2BjYFhgNbCNpF9vjgb8Au9k+HTgCuKns95+As4F9bW8D/Bg4rTR5KfD90t+7gUfKvt5ke7TtV15Mtb0E+AWwTxnLdsCDtv8GfAc4y/Y44GPAxMYDJOkoSR2SOpYseb4xOyIilsJAXvHvBFxl+1nbC4FJQO3K+yHbt5TtS0rZRi8Ck8v27VTTzK+QNAJY3fbUknQhsEtdkUmd1S1mAodJOgXY0vYzVCcomwO3SJoNHAJsUMrvVq7e5wG7A7Wr+LlUwflA4OUm/fwU+HjZ/gRwRdkeAVwhaT5wVl17ewDn2n4ZwPYTVCdKj9ieWdKeruXX2bP8zQLuADajOhHoyqZUKwteV/b3ROBtZTbhrbavKv29YPu5btq6HKg9M/DJ8ru2P98r7V8NrNY4W2F7gu2xtscOGbJSN91ERERPDOQ9fnWR1/j5wGafE3zJr35mcDE9H/uiruraniZpF2Av4GJJZwJPAtfZ3r++rKQVgR8AY20/VE4WVizZe1GdcHwE+KqkLeqDsu0/S/q7pFFUgfFfS9bXgRts7yNpJHBjrTv+8Xg0S2sk4Ju2f9hNucY6d9re4TWJ0mo9aKPmVmAjSWsDewPfKOlDgB1s51I+ImIZGMgr/mnA3pJWljScahr4ppK3vqRasNkfuLmnjdteADxZd//+IGBqF1VeQ9IGwKO2zwN+BGwNTAd2lLRRKbOypE14Ncg/Xu7571vyhwDr2b4B+CKwOrBKk+7+p+SPsD2vpI0A/ly2D60rOwUYL2lY6WNNqtsC60oaV9JWreXXuRb4dN0zCW+V9OZuDsO9wNq1fwtJK5QTl6eBhyXtXdLfqOo5jGeAf3i2AKCcpF0FfBu42/bf6/bn6Fo5SaO7GVNERPShAQv8tu8ALgBmALcBE23PKtl3A4dImgusCZzTy24OAc4s7YwGTu1B3V2p7uvPorr3/B3bj1EF4ctKm9OBzWw/BZxH9RzBz6luEwAMBS4p0/+zqO5lP9Wkryuppr9/Wpd2BvBNSbeUdmomUt17nytpDvAp2y9SzRacXdKu49WTEQBsT6F62v7WMp4r6SRI19V5keok5vTS7myq+/lQnUgdW47Db4F/orqt8XJ54O/zTZq8HDiQV6f5AY4FxpYHDu+i+XMQERHRT5b5Ij1lWntyeUgu4jWySE9ELG+ySE9EREQMmGX+AR/bD1I9SR4RERH9LFf8ERERbWSZX/FHdGXUqPXo6Oj/+2EREe0iV/wRERFtJIE/IiKijSTwR0REtJFl/h5/RFfyHn9E9LeBeK9+oOU9/oiIiAAS+CMiItpKAn9EREQbWa4Cv6TFkmZLurMsHHN8WTGvN20dV1agW5rxXCDpgTKmOZLeuzTtDWaSDpX0vR7WeVDSWv01poiI+EfLVeAHnrc92vYWwD8DHwRO7mVbxwE9CvyShjZJ/oLt0aW9c3s5loiIiD6xvAX+V9h+FDgKOFqVFSWdL2mepFmSdoMqWEv6z5I+V9Ixko4F1gVukHRDKbd/KTNf0um1fiQtlHSqpNuAHboY0q3AW+v6PFPSzNLnv9a198XSzxxJ3yppoyVNL2WvkrRGSb9R0lmSpkm6W9I4SZMk/U7SN0qZkZLukTSxjP1SSXtIuqWU27aUGy7px2VMsyT9S0k/tLT561L+jLqxHibpPklTgR3r0teW9LPS1kxJO5b0N0maUtr/IaBe/vNGREQvLdef7LV9f5nqfzPVuvDY3lLSZsAUSZsAhwFvB8bYflnSmrafkHQ8sJvtxyWtC5wObAM8WerubfvnwHBgvu2TuhnO+4Gfl+3DgQW2x0l6I3CLpCnAZsDewHa2n5O0Zil/EXCM7amSTqWaxTiu5L1oexdJnwN+Ucb4BPAHSbV3VDYCPk51IjQT+BSwE/AR4D9Kn18BfmP705JWB2ZI+r9SfzQwBlgE3CvpbOBl4GulvwXADcCsUv47wFm2b5a0PnAt8M4y7pttnypprzKeiIgYQMt14C9qV5U7AWcD2L5H0h+BTYA9gHNtv1zynmjSxjjgRtuPAUi6FNiFKpAvBn7WRf9nlqvkNwPbl7Q9gVGS9i2/RwAbl7Gcb/u52lgkjQBWtz21lL0QuKKu/avLf+cBd9p+pIzxfmA94CngAdvzSvqdwPW2LWkeMLJuTB+RdEL5vSKwftm+3vaCUv8uYANgrYZjcjnV8aTsx+bSKxf0q0latRyzj5Z9u0bSk80OmKSjKCcFQ4eu2qxIRET00nId+CVtSBWYH6XzaWUB3X3FqKsp6RdsL+4i/wvAJOBYqqC9TWnvGNvXNoz3/S2MpdGi8t8lddu138MayjSWqy8j4GO2720Y03YN9RfX1elsrEOAHWw/39BWV3VeYXsCMAGqD/h0Vz4iIlq33N7jl7Q21cN033P1ecJpwAElbxOqq9l7gSnAeEnDSl5tev0ZoHa5eRvwHklrlQf49gdqV+Ddsr2Eavp7iKT3UU19f0bSCrXxSBpexvJplbcJym2HBcCTknYuzR3Uk7574FrgGJXoLGlMN+VvA3Yt9+1XoLqVUDMFOLr2Q9Losln/b/ABYI2+GXpERLRqebviX0nSbGAFqnvQFwPfLnk/AM4t09svA4faXiRpItUU9VxJLwHnAd+juuL8X0mP2N5N0r9T3ccW8Cvbv+jJwMrU+jeAL1K9cTASuKME2seAvW3/ugTJDkkvAr+iugd/SBn7ysD9VM8l9LWvA/9NdRwEPAh8qIv9eUTSKVQPLT4C3AHU3mo4Fvi+pLlU/xubBoyneibgMkl3UJ28/Kkf9iMiIrqQb/XHoJZv9UdEf8u3+iMiImK5lcAfERHRRhL4IyIi2sjy9nBfLGdGjVqPjo7l7/5bRMSykiv+iIiINpLAHxER0UYS+CMiItpI3uOPQS3v8Ud/Wh7f346AvMcfERERRQJ/REREG0ngj4iIaCMJ/BEREW0kgX8ZkvRrSU9JmtyQ/iNJcyTNlXSlpFU6qT9K0q2S7pQ0T9KKJf1GSfdKml3+3jwQ+xMREYNfAv+ydSZwUJP0z9veyvYoqqVrj24sIGkYcAkw3vYWwK7AS3VFDrA9uvw92qT+jZJG9sE+9Iikod2XioiI/pLA388kjStX7itKGl6uzt8FYPt64JnGOrafLnUFrAQ0e+dyT2Cu7Tmlzt9tL+7DcX9d0ufqfp8m6diy/QVJM8t+fa2uzM8l3V728ai69IWSTpV0G7CDpG9JuqvU/8++GnNERHQvgb+f2Z4JXA18AzgDuMT2/O7qSTof+CuwGXB2kyKbAJZ0raQ7JH2xIf/8Ms3/1XIC0VM/Ag4pYxkCfBK4VNKewMbAtsBoYBtJu5Q6n7a9DTAWOFbSm0r6cGC+7e2Au4B9gC3KjMY3muz7UZI6JHUsWfJ8L4YeERGdSeAfGKcC/0wVEM9opYLtw4B1gbuB/ZoUGQbsBBxQ/ruPpPeWvANsbwnsXP4OApB0WO2+fxnLr8rvq5r0/yDwd0ljqGYXZtn+e9neE5gF3EF1YrJxqXaspDnAdGC9uvTFwM/K9tPAC8BESR8FnmvS9wTbY22PHTJkpW6PVUREtC6Bf2CsCawCrAqs2GqlMnV/OfCxJtkPA1NtP277OeBXwNal3p/Lf58BfkJ1dY7t82v3/YEO4IPl9z6dDGEicChwGPDjkibgm3XPD2xk+0eSdgX2AHawvRXViUFtX1+o3Yaw/XIZz8+AvYFft3o8IiJi6SXwD4wJwFeBS4HTuyqoyka1beDDwD1Nil4LjJK0cnnQ7z3AXZKGSVqr1F8B+BDQ7a2FTlwFvB8YV/qr9fvp2psGkt5a3hoYATxp+zlJmwHbd7J/qwAjbP8KOI7qdkFERAyQYct6AMs7SQcDL9v+SXmi/beSdrf9G0k3UU2VryLpYeBw4DrgQkmrUV1dzwE+09iu7SclfRuYSfXw369sXyNpOHBtCfpDgf8DzuvN2G2/KOkG4Km6K/Ypkt4J3FoeHVgIHEh15T5e0lzgXqrp/mZWBX5RXj0U8PnejC0iInoni/REp8pDfXcAH7f9u2UxhizSE/0pi/TE8iqL9ESPSdoc+D1w/bIK+hER0fcy1R9N2b4L2HBZjyMiIvpWAn8MaqNGrUdHR6ZjIyL6Sqb6IyIi2kgCf0RERBtpKfBLeoekN5btXSUdK2n1fh1ZRERE9LmWXuer+8TrSKoPuFwNbGr7g/05uIi8zhf18vpdRGv64nW+JeVTq/sA/23788Bb+mqAERERMTBaDfwvSdqfarW2ySVthf4ZUkRERPSXVgP/YcAOwGm2H5D0duCS/htWRERE9IeW3uO3fZekLwHrl98PAN/qz4FFRERE32v1qf4PA7MpS6hKGi3p6n4c13JD0sTy+duuyowvi/ksbV+7SprcSd7Rkn4vybXV+1po78eSHpXU8up+kj4i6ctd5B8q6XutthcREX2r1an+U6jWUH8KwPZs4O39MqLljO0jyudvuypzru2L+nkotwB7AH/sQZ0LqJblbYmkYbavtp3ZoIiIQarVwP+y7QUNaW25rJ+k4yXNL3/HlbSRku6RdKGkuZKulLRyybtR0tiyvVDSaZLmSJouaZ2SfoqkE8r26JI3V9JVktaoa+d0STMk3Sdp506GuErp/x5Jl6qsnWt7lu0Hm+zPKeXK/kZJ90s6tpZnexrwRDfH4wJJ3y7L955ef0Uv6ePlOM2RNK1J3b0k3drqDERERCy9VgP/fEmfAoZK2ljS2cBv+3Fcg5KkbagedNwO2B44UtKYkr0pMMH2KOBp4LNNmhgOTLe9FTANOLJJmYuAL5V25gEn1+UNs70tcFxDer0xJX9zqkV2dmxh1zYD3kc1q3OypJ6+sbEJsIft/68h/STgfWV/P1KfIWkf4MvAB20/3pB3lKQOSR1Lljzfw6FERERXWg38xwBbAIuAnwALqIJLu9kJuMr2s7YXApOA2pX3Q7ZvKduXlLKNXuTV1yFvp/og0iskjQBWtz21JF0I7FJXZFJndevMsP2w7SVUz2V0Vq7eNbYXlQD8KLBOC3XqXWF7cZP0W4ALJB0JDK1L3w34ErCX7ScbK9meYHus7bFDhqzUw6FERERXun2qX9JQ4GrbewBf6f8hDWrqIq/x1kezWyEv+dVPJS6m56sjLmqh7qK67Vb76E2des82S7Q9XtJ2wF7AbEmjS9b9VLMRmwAdPewrIiKWQrdX/OVK7rlyNdrupgF7S1pZ0nCqLxneVPLWl7RD2d4fuLmnjZfnKJ6su39/EDC1iyqDmqR32L7N9knA48B6JeuPwEeBiyRtscwGGBHRhlqd6n8BmCfpR5K+W/vrz4ENRrbvoHrSfQZwGzDR9qySfTdwiKS5wJrAOb3s5hDgzNLOaODUpRlzTVlY6WHgbcBcSRNbqHMZcCuwqaSHJR3ew27PlDSvvA44DZhTy7B9L3AAcIWkd/Sw3YiI6KVWF+k5pFm67Qv7fESvQ5JGApNtv2tZj2V5k0V6ol4W6YloTVeL9LT65b4E+IiIiOVAS4Ff0gM0eVjN9oZ9PqLXofJ+fK72IyJi0Gv16e366YIVgY9T3ceO6FejRq1HR0emdyMi+kpLD/fZ/nvd359t/zewe/8OLSIiIvpaq1P9W9f9HEI1A7Bqv4woIiIi+k2rU/3/Vbf9MvAA8Im+H05ERET0p1Zf59vQ9v0NaW+3/UC/jSyCvM7XLvKaXkTf6up1vlY/4HNli2kRERExiHU51S9pM6rFeUZI+mhd1mpUT/dHRETE60h39/g3BT4ErA58uC79GZovKRsRERGDWJeB3/YvgF9I2sH2rQM0poiIiOgnrd7jnyXp3yT9QNKPa3/9MSBJEyVt3k2Z8ZIO7oO+dpU0uZO8oyX9XpIlrdViez+W9GhZlGbQknSZpLmSPi9pM0mzJc3q6WI55fi9u+53n/y7RERE/2k18F8M/BPwPqplYt9GNd3f52wfYfuubsqca/ui/ui/zi3AHlRLyLbqAuD9/TKaPiLpn4B32x5l+yxgb+AXtsfY/kMPm9sVeCXwD9C/S0RELIVWA/9Gtr8KPFsW7NkL2LKVipKOlzS//B1X0kZKukfSheXK80pJK5e8GyWNLdsLJZ0maY6k6ZLWKemnSDqhbI8ueXMlXSVpjbp2Tpc0Q9J9dWvcN1ql9H+PpEslCcD2rPIN/sb9OaVc2d8o6X5Jx9bybE8DnujiWIyQ9KCkIeX3ypIekrSCpCMlzSz7+rO647FO2a855e/dJf3gss9zJF3cpK/hZZwzy9X8v5SsKcCby1X+ycBxwBGSbij1DizHbLakH0oaWtLfL+mO0t/1ZUXC8cDnS9mda/8ukt4paUbdWEaqWmYYSdtImirpdknXSnpLZ8crIiL6XquB/6Xy36ckvQsYAYzsrpKkbYDDgO2A7YEjJY0p2ZsCE2yPAp4GPtukieHAdNtbUa3n3uyBwouAL5V25gEn1+UNs70tVXA7uUldgDElf3NgQ2DH7vYL2Ixq9mNb4GRJK7RQB9sLqNakf09J+jBwre2XgEm2x5V9vRs4vJT5LjC1pG8N3ClpC+ArwO4l/XNNuvsK8Bvb44DdgDMlDQc+AvzB9mjbXwPOBc6yvZukdwL7ATvaHg0sBg6QtDZwHvCx0t/Hy0lRre5o2zfV7efdwBsk1RZx2g/4aTlOZwP72t4G+DFwWuPAJR0lqUNSx5Ilz7dyaCMiokWtBv4J5Ur6q8DVwF3AGS3U2wm4yvazthcCk4DalfdDtm8p25eUso1eBGr34G+n4WRD0ghgddtTS9KFwC51RSZ1VrfODNsP214CzO6iXL1rbC+y/TjwKLBOC3VqLqcKhACfLL8B3iXpJknzgAOoXqOEak2EcwBsLy4nD7sDV5b+sd1slmFP4MuSZgM3Ur1+uX43Y3svsA0ws9R7L9XJ0PbAtNoHmzrpr9FPefXrjvuV/dyUahXD60r7J1LdNnoN2xNsj7U9dsiQlVroKiIiWtXSJ3ttTyybU6kCQavUVbPd/AZ4ya9+WnAxrX9iuGZRC3UX1W232kdv6tRcDXxT0ppUQfY3Jf0CYG/bcyQdSnX/vDOi+fFqLPMx2/e+JrGaou+qzoW2/72hzkda6K/R5cAVkiYBtv07SVsCd9reoYdtRUREH2npir/cZ/6RpP8tvzeXdHh39aim5/cu97KHA/sAtSnh9SXVAsD+wM09HHtt6vzJuvv3B1GdnAxaZeZjBvAdYLLtxSVrVeCRMh1+QF2V64HPAEgaKmm1kvYJSW8q6c2WSL4WOKb2zELdLZauXA/sK+nNtXYlbQDcCrxH0tsb+nuGThZrKg8KLqaaJarNatwLrF37dy/PNmzRrH5ERPSPVqf6L6AKJOuW3/dR3Rfvku07St0ZwG3ARNuzSvbdwCHloa81KdPZvXAI1f3rucBo4NRetvMako6V9DDVVPRcSRNbqHMZVZDcVNLDXZwcXQ4cyKsBEaoAeRtwHXBPXfrngN3KLYDbgS1s30l1b3yqpDnAt5v08XVghTL2+eV3l8rbFCcCU8rxvA54i+3HgKOASaW/2rh/CexTe7ivi/38aWn/RWBf4PTSzmzq3gqIiIj+1+oiPTNtj5M0y/aYkja7PADW806r6ebJtt/Vm/rRPrJIT3vIIj0RfUt9sEjPs2Va2aXB7YEFfTS+iIiIGCCtPpR2PNVDae+QdAuwNtWUba+UV8FytR8RETHAupzql7S+7T+V7WFUr2MJuLe8ex7Rr8aOHeuOjo5lPYyIiNeVpZnq/3nd9uW277Q9P0E/IiLi9am7wF//Hn5P3t+PiIiIQai7wO9OtiMiIuJ1qLt7/IuBZ6mu/FcCnqtlUX2NbbV+H2G0tbzO9/qU1/Milq2u7vF3+VS/7aH9M6SIiIhYFlp9jz8iIiKWAwn8ERERbSSBPyIioo0MysAvaaKkzbspM17SwQM1pv4i6bfd5N8oqekDGnVldpZ0Z1ksJwvYR0REp3q6vv2AsH1EC2XOHYix9DfbfbE63QHAf9o+vz5R0tC6ZX8jIiIG5opf0vGS5pe/40raSEn3SLpQ0lxJV0paueS9cpUraaGk0yTNkTRd0jol/RRJJ5Tt0SVvrqSrJK1R187pkmZIuq/Z0rGS3iJpWrlanl8rI2lPSbdKukPSFZJWKeknSZpZyk6oW+/+WEl3lTH8T5N+tijjmF3KbFzbv7oyX5Q0r+zrtxrqDynH6hsN6UcAnwBOknSppF0l3SDpJ8A8SStKOr+0O0vSbqXeoZJ+LumXkh6QdHT5d5pVjuWaDf2MkPSgpCHl98qSHpK0gqQjyzGZI+lndf+O65R/jznl790l/eByDOZIurj7/wVFRERf6ffAL2kb4DBgO2B74EhJY0r2psAE26OAp4HPNmliODDd9lbANODIJmUuAr5U2pkHnFyXN8z2tsBxDek1nwKuLUsMbwXMlrQW1br0e9jeGuigWqgI4Hu2x5UlhVcCPlTSvwyMKWMY36Sf8cB3Sj9jgYfrMyV9ANgb2K7s6xn1+wBcCtxn+8T6erYnUi2g9AXbB5TkbYGv2N4c+LdSbktgf+BCSSuWcu8q+78tcBrwXFl2+VbgNbdRbC8A5gDvKUkfLsftJWBSOSZbAXcDh5cy3wWmlvStgTslbQF8Bdi9pH+u8UBJOkpSh6SOJUueb3IoIyKitwbiin8n4Crbz9peCEwCalfeD9m+pWxfUso2ehGYXLZvB0bWZ0oaAaxue2pJuhDYpa7IpM7qFjOBwySdAmxp+xmqE5TNgVskzQYOATYo5XeTdJukecDuwBYlfS5wqaQDgZeb9HMr8B+SvgRsYLsxou0BnG/7OQDbT9Tl/RCYb/u0Ju02M8P2A2V7J+Di0uY9wB+BTUreDbafsf0Y1TLLvyzp82h+rC4H9ivbnyy/Ad4l6aZyTA7g1WOyO3BO6XtxOXnYHbjS9uNN9pOSNsH2WNtjhwzJIwsREX1pIAK/ushr/Gxgs88IvuRXPy+4mJ4/l7Coq7q2p1GdKPwZuFjVA4MCrrM9uvxtbvvwcqX8A2DfcgV9HlC7et4L+D6wDXC7qtUM6/v5CfAR4HngWkm7NwxFdP5Z5N9SnXCs2El+o2cb2u3MorrtJXW/l9D8OF8NfKDcBtgG+E1JvwA4uhyTr/HqMWmmq/2MiIh+NhCBfxqwd7knPBzYB7ip5K0vaYeyvT9wc08bL1eRT9bdvz8ImNpFldeQtAHwqO3zgB9RTUlPB3aUtFEps7KkTXg1oD1e7vnvW/KHAOvZvgH4IrA6sEpDPxsC99v+LlUAHdUwlCnAp+vuj9ffY/8R8CvgisYTihZMo7oKp+zD+sC9PWwDgDJjMwP4DjC57sHBVYFHJK1Q66u4HvhM6XuopNVK2ickvamkv+ZZgoiI6F/9/lS/7TskXUAVMAAm2p4laSTV/eBDJP0Q+B1lWrgXDgHOLUHzfqpnClq1K/AFSS8BC4GDbT8m6VDgMklvLOVOtH2fpPOopsIfpLpNADAUuKTcdhBwlu2nGvrZDziw9PNX4NT6TNu/ljQa6JD0IlWg/4+6/G+X9i+WdIDtJS3u3w+ojs08qlsQh9peVJ5J7I3LgSuojlvNV4HbqG4jzKM6EYDq/v0ESYdTzbh8xvatkk4DpqpaC2IWcGhvBxMRET3T5SI9/dpxFfgnl4fkIprKIj2vT1mkJ2LZUheL9AzKD/hERERE/1hmH/Cx/SDV62QRERExQAbll/siakaNWo+OjkwbR0T0lUz1R0REtJEE/oiIiDaSwB8REdFGltnrfBGtyOt8rw95fS9icMnrfBEREQEk8EdERLSVBP6IiIg2ksAfERHRRhL4lwFJIyXN7yTv45LulLREUtMHM5rUOU3SQ5IW9u1IIyJieZPAP/jMBz5KtZxuq34JbNs/w+k7koYu6zFERLS7BP5lZ6ik88rV/RRJKwHYvtv2vY2FJR0qaZKkX0v6naQzanm2p9t+pKvOJK0i6XxJ8yTNlfSxkr5/SZsv6fS68gvLTMIcSdMlrVPS15F0VUmfI+ndJf1ASTMkzZb0w1qQL+2cKuk2YIfO2o2IiIGRwL/sbAx83/YWwFPAx1qoMxrYD9gS2E/Sej3o76vAAttb2h4F/EbSusDpwO6l7XGS9i7lhwPTbW9FNftwZEn/LjC1pG8N3CnpnWVcO9oeDSwGDqhrZ77t7Wzf3EW7ERExABL4l50HbM8u27cDI1uoc73tBbZfAO4CNuhBf3sA36/9sP0kMA640fZjtl8GLgV2KUVeBCY3Gd/uwDmljcW2FwDvBbYBZkqaXX5vWMovBn5WN47O2n2FpKMkdUjqWLLk+R7sYkREdCer8y07i+q2FwMr9aJOT/79BDR+plFdlH/Jr37Wsbu+BFxo+9+b5L1ge3FP2rU9AZgA1Zf7uug3IiJ6KFf87WMKcHTth6Q1gNuA90haq9yT3x+Y2k071wOfKW0MlbRaSdtX0ptL+pqSejIbERERAySBf5CRtI+kh4EdgGskXdtCnTNKnZUlPSzplCbFvgGsUR7imwPsVh4I/HfgBmAOcIftX3TT3eeA3STNo5qq38L2XcCJwBRJc4HrgLe0tMMRETGgskhPDGpZpOf1IYv0RAwuWaQnIiIigAT+iIiItpLAHxER0UbyOl8MaqNGrUdHR+4fR0T0lVzxR0REtJEE/oiIiDaSwB8REdFG8h5/DGp5j3/wyLv6Ea8feY8/IiIigAT+iIiItpLAHxER0UYS+CMiItrIoAn8kiZK2rybMuMlHdwHfe0qaXIneUdL+r0kS1qrxfZ+LOlRSfN7OI6xkr7bTZmFLbRzrKS7JV3ak/4jIqL9DJov99k+ooUy5w7AUG4BJgM39qDOBcD3gIt60pHtDqCjJ3U68VngA7YfqE+UNMz2y33QfkRELCf67Ypf0vFl7ff5ko4raSMl3SPpQklzJV0paeWSd6OksWV7oaTTJM2RNF3SOiX9FEknlO3RJW+upKskrVHXzumSZki6T9LOnQxxldL/PZIulSQA27NsP9hkf04pV/Y3Srpf0rG1PNvTgCe6OR4fL8dijqRpJe2VmQdJq0g6X9K8sk8fa6i/lqRbJe3VkH4usCFwtaTPl3FOkDQFuEjSBpKuL21eL2n9Uu8CSedIuqHsz3vK/t0t6YIm4/+ApJ/W/d5V0i/L9jmSOiTdKelrdWXGSfpt2ecZklaVNFTSf9bt5zFdHbeIiOhb/RL4JW0DHAZsB2wPHClpTMneFJhgexTwNNXVaqPhwHTbWwHTgCOblLkI+FJpZx5wcl3eMNvbAsc1pNcbU/I3pwqcO7awa5sB7wO2BU6WtEILdWpOAt5X9ukjTfK/CiywvWXZp9/UMsqJzzXASbavqa9kezzwF2A327UXrbcB/sX2pygzEaXNS4H6WwtrALsDnwd+CZwFbAFsKWl0w/iuA7aXNLz83g+4vGx/pbwvOgp4j6RRkt5Q8j9X9nkP4HngKODtwJi6Mb2GpKPKiUTHkiXPNzlUERHRW/11xb8TcJXtZ20vBCYBtSvvh2zfUrYvKWUbvUg13Q5wOzCyPlPSCGB121NL0oXALnVFJnVWt84M2w/bXgLM7qJcvWtsL7L9OPAosE4LdWpuAS6QdCQwtEn+HsD3az9sP1k2VwCuB75o+7oW+7radi1i7gD8pGxfzGuP9y9dfcFpHvA32/PK8biThuNRbhn8GviwpGHAXsAvSvYnJN0BzKI6cdic6gTvEdszS/2nSxt7AOfWbkHY/oeZEtsTbI+1PXbIkJVa3OWIiGhFfwV+dZHX+KnAZp8OfMmvflJwMT1/FmFRC3UX1W232kdv6gCvXJmfCKwHzJb0poYiovmxeJnqBOZ9rfYFPNvVUOq2a/uzhNfu2xKa79vlwCeoZglm2n5G0tuBE4D3liv4a4AV6Xx/OkuPiIgB0F+Bfxqwt6SVy9TwPsBNJW99STuU7f2Bm3vauO0FwJN19+8PAqZ2UWWZk/QO27fZPgl4nOoEoN4U4Oi68muUTQOfBjaT9OVedP1b4JNl+wB6cbzr3AhsTXXrpTbNvxrVicaCckviAyX9HmBdSeMAyv39YVT7Ob5sI2nNpRhPRET0UL8Eftt3UD3pPgO4DZhoe1bJvhs4RNJcYE3gnF52cwhwZmlnNHDq0oy5RtWrcQ8DbwPmSprYQp3LgFuBTSU9LOnwJsXOLA+0zac6MZrTkP8NYI3aA4DAbrUM24upgvdukpo9E9GVY4HDynE6CPhcD+u/ooxjMlVwn1zS5lBN8d8J/Jjqlga2X6R6DuDssj/XUc0ETAT+RHVs5wD5EH9ExAAa0EV6JI0EJtt+14B1Gq9rWaRn8MgiPRGvH8oiPREREQED/AGf8n58rvYjIiKWkUHz5b6IZkaNWo+OjkwxR0T0lUz1R0REtJEE/oiIiDaSwB8REdFGBvR1voieyut8/Suv6EUsn/I6X0RERAAJ/BEREW0lgT8iIqKNJPBHRES0kQT+pSBpZFl0p1nexyXdKWmJpKYPWDSpc5qkhyQt7KLMKZJO6O2YW9XVvnVR5wJJ+/bXmCIiYukl8Pef+cBHqVbia9UvgW37ZzgREREJ/H1hqKTzytX9FEkrAdi+2/a9jYUlHSppkqRfS/qdpDNqeban236k1Y4lHSnpfyWtJOlGSadLmiHpPkk7lzIrSjq/LAk8S9JuJf1XkkaV7VmSTirbX5d0REM/QyWdKWmmpLmS/rWkS9L3JN0l6RrgzXV1PijpHkk3S/qupMklfbikH5e2Zkn6l5aPdERELLUE/qW3MfB921sATwEfa6HOaKq16rcE9pO0Xk87lXQ08GFgb9vPl+RhtrcFjgNOLmn/BmB7S2B/4EJJK1LNROwsaTXgZWDHUn4n4KaG7g4HFtgeB4wDjpT0dmAfYNOyH0cC7y5jWxH4IfAB2zsBa9e19RXgN6Wt3YAzJQ1v2LejJHVI6liy5HkiIqLvJPAvvQdszy7btwMjW6hzve0Ftl8A7gI26GGfBwEfAD5me1Fd+qQm49gJuBjA9j3AH4FNqIL7LiX/GmAVSSsDI5vMVOwJHCxpNnAb8CaqE55dgMtsL7b9F+A3pfxmwP22Hyi/L2to68ulrRuBFYH16zuzPcH2WNtjhwxZqcVDEhERrcjqfEuvPvAuBlqJVI11evrvMJ9q1uBtwAN16bV269tUJ23MBMYC9wPXAWtRXbXf3qSsgGNsX/uaROmDQLNPP3bWZy3vY81ug0RERP/LFf/r0yzgX4GrJa3bTdlpwAEAkjahurq+1/aLwEPAJ4DpVDMAJ/CP0/wA1wKfkbRCrZ0yPT8N+GR5BuAtVFP3APcAG0oaWX7v19DWMZJU2hrT8l5HRMRSS+DvJ5L2kfQwsANwjaRrW6hzRqmzsqSHJZ3SWVnbN1MF6mskrdVFsz+gegBxHnA5cGjd7YGbgL/Zfq5sv43mgX8i1S2JO8orfj+kmlG4CvgdMA84B5haxvY88Fng15JuBv4GLChtfR1YAZhb2vp6F2OPiIg+lkV6ol9IWsX2wnJl/33gd7Z7vCJMFunpX1mkJ2L5lEV6Ylk4sjzAdycwgmqWICIilrE83Bf9olzd53IyImKQSeCPQW3UqPXo6Mj5Q0REX8lUf0RERBtJ4I+IiGgjCfwRERFtJIE/IiKijSTwR0REtJEE/oiIiDaSwB8REdFGEvgjIiLaSAJ/REREG8kiPTGoSXoM+OOyHkdExOvMBrbXbpaRwB8REdFGMtUfERHRRhL4IyIi2kgCf0RERBtJ4I+I5YakhQPc30hJnxrIPiOWVgJ/REQvSBoGjAQS+ON1ZdiyHkBERF+TtCvwNeBvwGhgEjAP+BywErC37T9IugB4AdgCWAc43vZkSSsC5wBjgZdL+g2SDgX2AlYEhgMrA++UNBu4ELgKuLjkARxt+7dlPKcAjwPvAm4HDrRtSeOA75Q6i4D3As8B3wJ2Bd4IfN/2D/vyGEX7SuCPiOXVVsA7gSeA+4GJtreV9DngGOC4Um4k8B7gHcANkjYC/g3A9paSNgOmSNqklN8BGGX7iRLQT7D9IQBJKwP/bPsFSRsDl1GdPACMoTrB+AtwC7CjpBnA5cB+tmdKWg14HjgcWGB7nKQ3ArdImmL7gT4/StF2EvgjYnk10/YjAJL+AEwp6fOA3erK/dT2EuB3ku4HNgN2As4GsH2PpD8CtcB/ne0nOulzBeB7kkYDi+vqAMyw/XAZz2yqE44FwCO2Z5a+ni75ewKjJO1b6o4ANgYS+GOpJfBHxPJqUd32krrfS3jt//c1fsXMgLpo99ku8j5PdXthK6pnqF7oZDyLyxjUpH9K+jG2r+2ir4heycN9EdHuPi5piKR3ABsC9wLTgAMAyhT/+iW90TPAqnW/R1BdwS8BDgKGdtP3PcC65T4/klYtDw1eC3xG0gq1MUga3kU7ES3LFX9EtLt7galUD/eNL/fnfwCcK2ke1cN9h9peJP3DRMBc4GVJc4ALgB8AP5P0ceAGup4dwPaLkvYDzpa0EtX9/T2AiVS3Au5Q1eljwN59sK8R+VZ/RLSv8lT/ZNtXLuuxRAyUTPVHRES0kVzxR0REtJFc8UdERLSRBP6IiIg2ksAfERHRRhL4IyIi2kgCf0RERBv5fxp6mZfv01w2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.barh(y= top_ten_feat.Features, width= top_ten_feat['Feature_importance'], color='midnightblue')\n",
    "ax.set_title('Feature Importance')\n",
    "ax.set_ylabel('Features')\n",
    "ax.set_xlabel('Importance')\n",
    "plt.xticks([])\n",
    "ax.invert_yaxis()\n",
    "plt.savefig('./images/Feature_Importance.jpeg', bbox_inches = 'tight');\n",
    "plt.savefig('./images/Feature_Importance.png', bbox_inches = 'tight');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top 3 most important features in determining whether a person was to get vaccinated are \n",
    "- A person's opinion on the risk of the seasonal flu \n",
    "- If it was Doctor Recommended\n",
    "- An opinion on seasonal vaccine effectiveness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will take a look at our final model's performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:14.141789Z",
     "start_time": "2022-04-22T17:25:14.140086Z"
    }
   },
   "outputs": [],
   "source": [
    "final_model = avg2_fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:14.144426Z",
     "start_time": "2022-04-22T17:25:14.142874Z"
    }
   },
   "outputs": [],
   "source": [
    "#final_model.fit(X_train, y_train); "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:17.136065Z",
     "start_time": "2022-04-22T17:25:14.145527Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEGCAYAAACJnEVTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnrElEQVR4nO3deZwU1bn/8c93ZtiURTYRwYgLLqAGBQ3GaHC5ilvUaBRjXGISo1dNNHqNZtX8olcTExP3PS4Xt7hEg7soKCoiKCAgKgoqSiTgBsjO8/ujzkhDZilgYHq6v29f9eqap+tUne7Gp0+fOnVKEYGZmZWHisaugJmZrTtO+mZmZcRJ38ysjDjpm5mVESd9M7MyUtXYFTBQVatQ8zaNXQ1bBTtu+5XGroKtgnffncasWbO0JvuobLtpxJL5ubaN+f9+PCIGrsnx1hYn/SKg5m1osfWRjV0NWwXPv3RlY1fBVsFuX+u3xvuIJfNz/3+6YOxVndb4gGuJk76ZWS4CNf0ecSd9M7M8BFRUNnYt1piTvplZXlqj0wJFwUnfzCwXd++YmZUXt/TNzMqEcEvfzKx8yC19M7Oy4tE7ZmblwidyzczKh3D3jplZWXFL38ysXLh7x8ysfAio9IlcM7Py4T59M7Ny4e4dM7Py4pa+mVkZcUvfzKxMyNMwmJmVF0/DYGZWLnwi18ysvLh7x8ysTHg+fTOzcuLuHTOz8uITuWZmZaQE+vSb/m8VM7N1Qal7J89S767UUtIoSeMkTZR0QYqfL+kDSWPTckBBmfMkTZH0hqT9CuJ9Jb2WnrtcqvubyS19M7O8Gq6lvxDYKyLmSmoGjJD0aHrusoi4dMXDqhcwCOgNbAw8JWmriFgKXAOcBIwEHgEGAo9SC7f0zcxykpRrqU9k5qY/m6Ul6ihyCHBXRCyMiKnAFGAXSV2BthHxYkQEcBtwaF3HdtI3M8shu1ti7qTfSdLoguWk/9ifVClpLDATeDIiXkpPnSZpvKSbJbVPsW7A+wXFp6dYt7S+crxWTvpmZnlIqCLfAsyKiH4Fy/Ur7y4ilkZEH6A7Wat9O7Kumi2APsAM4E/VR6+hRlFHvFZO+mZmOTVU906hiPgUGAYMjIiP0pfBMuAGYJe02XRgk4Ji3YEPU7x7DfFaOembmeXUUElfUmdJG6T1VsA+wOTUR1/tMGBCWn8IGCSphaTNgJ7AqIiYAcyR1D+N2jkOeLCuY3v0jplZTqvaiq9DV+BWSZVkje97ImKIpNsl9SHropkG/BggIiZKugeYBCwBTk0jdwBOAW4BWpGN2ql15A446ZuZ5SNq7kFfDRExHtixhvixdZS5ELiwhvhoYLu8x3bSNzPLQax6f30xctI3M8upoqLpnwZ10jczy8ktfTOzctGAffqNyUnfzCwnt/TNzMqET+SamZWZNMVCk+akb2aWh9y9Y2ZWVpz0zczKiJO+mVmZ8IlcM7Ny0/RzvpO+mVku8jQMZmZlxd07ZmblpOnnfCd9y69F8yoevv4MWjSrorKqkoeGvsrF1z/Cdj278adzB9F6vRa8N2M2J/36VubMW0D7dutz68U/YMdem3LnkJGc88e/f7mvw/5rJ876/n5UVFbw5IgJ/PaKOm/2Yw3k6jue5vZ/vAASvbbcmKt+8z3+cusT3PaPF+i4QWsAfn3qt9h3t958/Olcjj/3Jl6d9C5HH9SfP55zZCPXvvG5pV8HST2AIRGxXUHsfGBuRFxaw/brA+8Bm0fEZwXxfwB3RMQ9DVCnFyLi66tZ9gTgiYio8/6TK5XpwUrvQVO2cNESDjnlcubNX0RVZQWP3vgznnphEpf8z3f49V8f4IVXpnDMwf05/di9uejah1m4cDEXXTuEbbfYmG23WH4XuPbt1ud3PzmUAcf+gdmfzuXq3x7LHjtvxbMvv9mIr670fTjzU667ezgj7/4lrVo25/vn3cT9T4wB4JSj9+T0Y/dZYfsWLZrxi5MP4vW3P+T1t2c0RpWLyurc/7YYFc1ZiYiYBzwBHFodk9QO+AYwpIGOsVoJPzkB2Lgh6tGUzZu/CIBmVZU0q6okItjyKxvywitTABg2ajIH79kHgC8WLGLkuHdYsGjxCvvo0a0jU96byexP5wIwfNRkvrVXn3X2GsrZkiVLWbBwMUuWLOWLBYvYqHO7Wrddv1ULdu2zBS2bN1uHNSxua+PG6OtaoyR9ScMkXSJplKQ3Je2enroTGFSw6WHAY0CFpKGSXpH0mqRDCvZ1nKTxksZJuj3Fukh6IMXGSfp6is9NjwNSHe6VNFnS4HRTYST9RtLLkiZIul6ZI4B+wGBJYyW1ktRX0nBJYyQ9Xn1D4xQfJ+lF4NS1+06uexUV4tnB5/LmExcz7KXJjJn4LpPfmcH+e2wPwCF770S3Lu3r3Mc77/+bnpt2YZOuHaisrOCAAV+tt4ytuY033IDTv7c32x/8a7bZ/5e0Xb8Ve/XfFoAb/v4sux19Eaf97v/49PMvGrmmxUsVyrUUs8Zs6VdFxC7AGcBvU+wxoK+kjunvQWRfBAuAwyJiJ2BP4E8pGfcGfgnsFRFfBX6ayl0ODE+xnYCJNRx/x3TsXsDmwG4pfmVE7Jy6ZFoBB0XEvcBo4JiI6EN2Y+IrgCMioi9wM8vvXfk34CcRsWtdL17SSZJGSxodS+bX81YVj2XLgj2OuZjeB/6KnXpvyrZbdOW03w3mh9/Zg2duO4fW67Vg8eKlde7jsznzOfuSu7n5ohN55PozeW/GbJYsWbaOXkH5+vTzL3jk2dcY++AFvP7ohXyxYBF3PzKKEw/fnVcfOJ/nBp9Ll05t+dVf7m/sqhatUmjpr80TuVFPvPpf1higB0BELJL0EHCEpPuAPmRdPgIukrQHsAzoBnQB9gLujYhZqfzHaZ97Acel2FLgy3MEBUZFxHQASWNTHUYAe0o6B1gP6ED2hfHPlcpuTXYj4ifTB1wJzEjdURtExPC03e3A/jW+CRHXA9cDVKy3YW3vVdH6fO58Rox5i7137cWV/zeUw0+/CoAtvrIh+36jd73lH3tuAo89NwGA4w/bjWVLnfTXtmGjJrPpxh3p1L4NAAfv+VVGjZ/KUQfs8uU2xx+6G0edeW1jVbG4lciEa2uzpT8bWPk3ewdgVlpfmB6XsuKXT3UXzxHAgxGxGDgG6Az0TS3tj4CWZF8Gq5swFxasLwWqJLUEriZrwW8P3JCOszIBEyOiT1q2j4h917A+Ra/jBq1p27oVAC1bNGPALlvz1rSP6NQ+G/UhibNP3I+/3Tei3n1Vl2nXphU/OGJ3bnvwxbVXcQOg+0YdGP3aVL5YsIiIYPjLb7D1Zl3416zlbaIhw8atcNLdlhMg5VuK2Vpr6UfEXEkzJO0dEUMldQAGAn8Fvl9H0WeAW8n6w09PsXbAzIhYLGlPYNMUHwo8IOmyiJgtqUNq7Q8FTgH+IqkSWD8iPs9R7eoEP0tSa7IvnntTbA7QJq2/AXSWtGtEvCipGbBVREyU9Jmkb0TECLIvq5KxUae2XH3+sVRWVFBRIR546hUeHzGBHw8awA+P2AOAIcPGMvifI78sM+7BC2izfkuaNavigG/uwOGnX8UbU//FxWcdQe+e3QD4442P8fZ7MxvlNZWTftv14Ft778iA711CZWUFO2zdneMP242f/P4OXntzOpL4StcOXPaLo78ss8O3fsOceQtYvHgJjwwfz31XnMo2m5frl0Lxd93koYi11zCV1Au4iuUt/j9GxGBJw4CzI2K0pE7A6IjoUVDur8B3gO4RsSxt80+gGTCWrP99/4iYJul44H/IWuuvRsQJkrqQdZ1snuKnpOQ8NyJaSxqQjn9QOt6VqQ63SPo92S+NacD7wLsRcb6kw4GLgPnArmRdPJeTfSFVAX+JiBskVffxfwE8Tvaroc4hmxXrbRgttvYY6Kbkk5evbOwq2CrY7Wv9GDNm9Bpl7JYbbRWbHn9Frm3f/MPAMRHRb02Ot7as1aRv+TjpNz1O+k1LgyT9rltFj5xJ/41Lijfp+4pcM7McRDZkuakrmouzzMyKXUOdyJXUMl2nNE7SREkXpHgHSU9Keis9ti8oc56kKZLekLRfQbxvun5piqTLq685qo2TvplZTg04Tn8hy68v6gMMlNQfOBcYGhE9yQaknJuO24vsXGNvsgExV6dBKgDXACcBPdMysK4DO+mbmeWRs5WfJ+dHZm76s1laAjiEbPQi6fHQtH4IcFdELIyIqcAUYJc0E0DbiHgxshO0txWUqZGTvplZDkJUVFTkWoBO1Vfcp+Wk/9ifVJkuDJ0JPBkRLwFdImIGQHrcMG3ejWw0YbXpKdYtra8cr5VP5JqZ5bQKw/Rn1Td6J80W0EfSBmTXG9U1tLumI0cd8Vq5pW9mltPamHsnIj4FhpH1xX9UMHljV7JfAZC14DcpKNYd+DDFu9cQr5WTvplZHg3Ypy+pc2rhI6kVsA8wGXgIOD5tdjxQfXehh4BBklpI2ozshO2o1AU0R1L/NGrnuIIyNXL3jplZDtncOw02Tr8rcGsagVMB3BMRQ9KU7PdI+gHZTaW+A5CmeLkHmEQ2y++pqXsIsilnbiGbFfjRtNTKSd/MLKeGyvkRMZ5seveV47OBvWspcyHLp3AvjI8mm/U3Fyd9M7OcSuGKXCd9M7M8SmQ+fSd9M7McqufTb+qc9M3McimN+fSd9M3MciqBnO+kb2aWi3wi18ysbDTwOP1G46RvZpaTk76ZWRkpgZzvpG9mlpdb+mZm5SLnZGrFzknfzCyH7CYqTT/rO+mbmeVUUQJNfSd9M7OcSiDnO+mbmeUhT7hmZlZeSqBL30nfzCwvn8g1MysTIhvB09Q56ZuZ5VQCDX0nfTOzXOT59M3MykoJ5HwnfTOzPIQvzjIzKysevWNmVibkCdfMzMqLu3fMzMpI00/5dSR9SVcAUdvzEfGTtVIjM7MiVQpDNivqeG40MKaOxcysbGSjd/It9e5L2kTSM5JelzRR0k9T/HxJH0gam5YDCsqcJ2mKpDck7VcQ7yvptfTc5arnm6nWln5E3JrjfTAzKw9q0JuoLAHOiohXJLUBxkh6Mj13WURcuuKh1QsYBPQGNgaekrRVRCwFrgFOAkYCjwADgUdrO3C9ffqSOgM/B3oBLavjEbFX/tdnZtb0NVT3TkTMAGak9TmSXge61VHkEOCuiFgITJU0BdhF0jSgbUS8mOp3G3AodST9urp3qg0GXgc2Ay4ApgEv5yhnZlYyVrF7p5Ok0QXLSbXuV+oB7Ai8lEKnSRov6WZJ7VOsG/B+QbHpKdYtra8cr1WepN8xIm4CFkfE8Ig4Eeifo5yZWUlRmn+nvgWYFRH9Cpbra9lfa+A+4IyI+Jysq2YLoA/ZL4E/VW9aQ/GoI16rPEM2F6fHGZIOBD4EuucoZ2ZWUhpy7I6kZmQJf3BE3A8QER8VPH8DMCT9OR3YpKB4d7JcPJ0V83F1vFZ5Wvq/l9QOOAs4G7gRODNHOTOzkiFBZYVyLfXvSwJuAl6PiD8XxLsWbHYYMCGtPwQMktRC0mZAT2BUOjcwR1L/tM/jgAfrOna9Lf2IqP6m+QzYs95XY2ZWohpwnP5uwLHAa5LGptgvgKMl9SHropkG/BggIiZKugeYRDby59Q0cgfgFOAWoBXZCdxaT+JCvtE7f6OGPqLUt29mVjYaKudHxAhq7i16pI4yFwIX1hAfDWyX99h5+vSHFKy3JPvJUWefkZlZqREqj7l3IuK+wr8l3Qk8tdZqZGZWjMp4ls2ewFcauiLlbIdtNuHp5/7S2NWwVbDTb55o7CrYKpj64ecNsp9SmHsnT5/+HFbs0/8X2RW6ZmZlQ0BlOST9iGizLipiZlbsSuDGWfWP05c0NE/MzKzUNdQsm42prvn0WwLrkc0h0Z7lw4vaks3yZmZWNrLbJRZ5Rs+hru6dHwNnkCX4MSxP+p8DV63dapmZFZ9ib8XnUdd8+n8F/irp9Ii4Yh3WycysKJVAQz/X3DvLJG1Q/Yek9pL+e+1Vycys+AioknItxSxP0v9RRHxa/UdEfAL8aK3VyMysSEn5lmKW5+KsCkmKiACQVAk0X7vVMjMrLlKZTMMAPA7cI+lasou0TqaeWdzMzEpRCeT8XEn/52Q33T2FrFvrVaBrnSXMzEpQSY/eqRYRyySNBDYHjgI6kN3txcysbAhy3SCl2NV1cdZWwCDgaGA2cDdARPhGKmZWfprA1bZ51NXSnww8BxwcEVMAJPk2iWZWttSgd8ltHHUN2TycbEbNZyTdIGlvGva+wGZmTYYojbl3ak36EfFARBwFbAMMI7sZehdJ10jadx3Vz8ysaJR00q8WEfMiYnBEHAR0B8YC567tipmZFRtJuZZitkp3zoqIj4Hr0mJmVjYkqMwzh0GRW53bJZqZlaVyuSLXzKzsVZ/Ibeqc9M3MciqBhr6TvplZPqKiBEatO+mbmeUgSqOlXwLnos3M1gFBVYVyLfXuStpE0jOSXpc0UdJPU7yDpCclvZUe2xeUOU/SFElvSNqvIN5X0mvpuctVz5hRJ30zsxyqW/oNdBOVJcBZEbEt0B84VVIvsmughkZET2Bo+pv03CCgNzAQuDrd2wTgGrKZkHumZWBdB3bSNzPLqSLdSKW+pT4RMSMiXknrc4DXgW7AIcCtabNbgUPT+iHAXRGxMCKmAlOAXSR1BdpGxIvpRle3FZSpkfv0zcxyWoU+/U6SRhf8fX1EXF/zPtUD2BF4CegSETMg+2KQtGHarBswsqDY9BRbnNZXjtfKSd/MLAexSl0jsyKiX737lFqT3Z/kjIj4vI7u+JqeiDritXLSNzPLQw17Ra6kZmQJf3BE3J/CH0nqmlr5XYGZKT4d2KSgeHfgwxTvXkO8Vu7TNzPLIbsit2H69NMIm5uA1yPizwVPPQQcn9aPBx4siA+S1ELSZmQnbEelrqA5kvqnfR5XUKZGbumbmeXUgMP0dwOOBV6TNDbFfgFcDNwj6QfAe8B3ACJioqR7gElkI39OjYilqdwpwC1AK+DRtNTKSd/MLKeG6t2JiBHU/h2ydy1lLgQurCE+Gtgu77Gd9M3Mcin+ufLzcNI3M8thFUfvFC0nfTOznDyfvplZuRDu3jEzKxfu3jEzKzNu6ZuZlZGmn/Kd9M3MchFQ6Za+mVn5KIGc76RvZpaPUAl08Djpm5nl5Ja+mVmZyIZsNv2s76RvZpZH/vvfFjUnfTOznDwNg5lZmchuotLYtVhzTvpmZjl59I6ZWRkpgd4dJ31bfTfeM5w7/vkiEfDdb/XnR0cOYOJbH3DupffwxfxFdN+oA1f+9ljarN+SxUuWcvbFdzHhzeksWbqUIwbuzOnH/ldjv4SS17yqghtP3JnmVRVUVoihEz/i2mfepm2rKi4+8qtsvEFLPvx0AT+/exxzFiz5stxG7Vpy72lf57phb3P78+8CUFUpzj1wW/r2aM+ygKuGvsXTk2bWduiS5JZ+A5M0DPjfiHi8IHYGsFVE/Pca7vtG4M8RMWk1yg4AFkXEC6tYbhrQLyJmreoxi93kd2Zwxz9f5OEbfkazqkqOOes69t61N/9zyV38+tRD2HXHLblryEiuueNpzvnRAQx5eiyLFi9h6G0/Z/6CRQz43v9y6D47sUnXjo39UkraoiXL+PEto5m/aClVFeKmH+7C82/NYq9eGzLqndnc8tw0Tti9B9/ffTMuf/KtL8udNXBrnn9rxX+2P9xjcz6et4jDLn8eCdq1arauX06jKpU+/WKbKfROYNBKsUEpvkYi4oerk/CTAcDX17QOpeStaR+xU+8etGrZnKqqSvrvuAWPPTuet9+bSf8+WwCw+85b88jwcUD2s/iL+YtYsmQp8xcupllVFa3Xb9mYL6FszF+U3T+7qlJUVYgAvrnNhgx59UMAhrz6IQO23fDL7Qds05kPPpnPO/+et8J+vrVTN25+dioAEfDpF4vXzQsoFhIVOZdiVmxJ/17gIEktACT1ADYGvitptKSJki6o3ljSzpJekDRO0ihJbSRVSrpU0muSxks6PW07TFK/tD5X0oWp3EhJXVL8YEkvSXpV0lOSuqQ6nAycKWmspN0ldZZ0n6SX07JbKt9R0hOp/HWUxqR8Ndpm840YOfZtPv5sHvMXLOLpFyfx4cxP2XrzrjwxYgIAQ54Zy4cffQrAgXv2Yb1Wzdnx0N+wy+EXcPLRe9K+7fqN+ArKR4XgzlP689Q5A3jp7dlMmP4ZHddvzqy5iwCYNXcRHdZvDkDLZpWcsPtmXDfs7RX20bpl1inw33tvweCT+3PJkTt8WaacKOdSzIoq6UfEbGAUMDCFBgF3A7+MiH7ADsA3Je0gqXl67qcR8VVgH2A+cBKwGbBjROwADK7hUOsDI1O5Z4EfpfgIoH9E7AjcBZwTEdOAa4HLIqJPRDwH/DX9vTNwOHBjKv9bYEQq/xDwldpeq6ST0hfZ6Nmzml7vT88eG3Hq9/bm6DOv4ZizrqXXlt2orKzgz+cdzS33j2DgiZcy74uFNGtWCcDYSe9SWVHBK//4HSP//muuu+sZ3v2g6b3upmhZwNHXjGTgn56ld/d2bLFh61q3PXmvLRj8wrtf/jqoVlUhNmrXkrHvfcox145k/PTPOHO/rdZ21YtK1r3T9Fv6RdWnn1R38TyYHk8EjpR0Ell9uwK9gABmRMTLABHxOYCkfYBrI2JJin9cwzEWAUPS+hig+oxid+BuSV2B5sDUWuq4D9Cr4IYKbSW1AfYAvp2O+7CkT2p7kRFxPXA9QJ+d+kZt2xWzow/qz9EH9Qfgf68bQtfOG7Dlpl2487JTAHj7vZkMfTHrUXvgyVcY8LVtaFZVSaf2bdh5+80YN/l9Nu3WqdHqX27mLljCmKkf8/WeHZk9bxGdWmet/U6tm/PxvKzVv333duzTqws/3Xcr2rSsYlnAosXLuHvU+8xftJRnXs9O3D414V8culO3xnw5jaK403k+RdXST/4B7C1pJ6AV8AlwNrB3ark/DLQke/9rSpa1xQstjojqbZay/MvvCuDKiNge+HE6Tk0qgF1Ty79PRHSLiDnpuSaZwFfHrE+yl/zBvz7h0eHjOXSfnb6MLVu2jL/e+gTHHpKdCunWZQOef+UtIoIv5i/klUnvsuWmXRqt7uVig/Wafdk106Kqgq9t0ZFp/57Hs5P/zUE7bgzAQTtuzPDJWTL/wU0vc9Blz3HQZc9xx8j3uPm5d7h71PsAPPvGTPr16ADALpt35J2ZcxvhFTWyEujfKbqWfkTMTaN4biZr9bcF5gGfpb73/YFhwGRgY0k7R8TLqaU9H3gCOFnSsIhYIqlDLa39mrQDPkjrxxfE56R6VHsCOA34I4CkPhExlqyr6Bjg95L2B9qv0otvYn70y7/xyefzqKqs5MKfHcEGbdfjxnuGc8v9IwA44Js7cNSBXwPghG/vzpkX3cFex15CEBx1wNfoteXGjVn9stC5TQsu+PZ2VEpI4smJ/+K5N2cx/v3PuOSoHTh0p27867MFnHP3uHr3dfkTb/H/Dt+es/ffmk++WMT5D0xcB6+guBR7100eWt7gLR6SDgPuB7aNiMmSbgG+BrwDLAQeiohbJO1M1jpvRZbw9wEWAH8gOy+wGLghIq5MXyRnR8RoSXMjonU61hHAQRFxgqRDgMvIEv9IYOeIGCBpK7KTzMuA04HXgauAbcm+OJ+NiJMldST7ouoEDCfr6ulb35DNPjv1jaefe2nN3zhbZ75x4dONXQVbBVP/dhrzZ7y5Rhl72+13jNseHJZr21222GBMOg9ZdIqupQ8QEQ9Q8CMpIk6oZbuXgf41PPWztBRuO6BgvXXB+r1kCZ2IeJDsXMLKx3mT7CRyoaNq2G42sG9B6Mya6m1mTVTTb+gXZZ++mVnRybrr8/1X776kmyXNlDShIHa+pA/S0PCxkg4oeO48SVMkvSFpv4J43zQ8fYqky6X6+5+c9M3M8kjz6edZcriF5UPTC11WMEDkEQBJvchGMvZOZa6WVJm2v4ZsmHrPtNS0zxU46ZuZ5dRQg3ci4lkg7wCTQ4C7ImJhREwFpgC7pKHlbSPixTQa8Tbg0Pp25qRvZpZLNgIqzwJ0qr74Mi0n5TzIaWkmgZslVY/+6wa8X7DN9BTrltZXjtfJSd/MLKdV6N6ZFRH9Cpbrc+z+GmALoA8wA/hT9WFr2DbqiNfJSd/MLIe8XTurO8AnIj6KiKURsQy4AdglPTUd2KRg0+7AhynevYZ4nZz0zczyWotZP/XRVzsMqB7Z8xAwSFILSZuRnbAdFREzgDmS+qdRO8dRw5DzlRXlOH0zs2LUUDdRkXQn2ZTtnSRNJ5uscYCkPmRdNNPIpoIhIiZKugeYBCwBTo2I6hnxTiEbCdQKeDQtdXLSNzPLqaFmYYiIo2sI31TH9hcCF9YQHw1styrHdtI3M8sj/xj8ouakb2aWk++Ra2ZWJoRb+mZmZaUEcr6TvplZbiWQ9Z30zcxyKoWbqDjpm5nl1PRTvpO+mVl+JZD1nfTNzHKovolKU+ekb2aWhy/OMjMrLyWQ8530zczy+fIGKU2ak76ZWU4lkPOd9M3M8liTG6QUEyd9M7O8SiDrO+mbmeXkIZtmZmXEffpmZuVCUOGkb2ZWTpp+1nfSNzPLwTdRMTMrMyWQ8530zczyckvfzKyMeBoGM7My0vRTvpO+mVku8tTKZmblpRSuyK1o7AqYmTUZyrnUtxvpZkkzJU0oiHWQ9KSkt9Jj+4LnzpM0RdIbkvYriPeV9Fp67nLlOOngpG9mllMD5XyAW4CBK8XOBYZGRE9gaPobSb2AQUDvVOZqSZWpzDXASUDPtKy8z//gpG9mlouoUL6lPhHxLPDxSuFDgFvT+q3AoQXxuyJiYURMBaYAu0jqCrSNiBcjIoDbCsrUyn36ZmY5rOIVuZ0kjS74+/qIuL6eMl0iYgZARMyQtGGKdwNGFmw3PcUWp/WV43Vy0jcza3izIqJfA+2rpq+aqCNeJ3fvmJnlVD1ss75lNX2UumxIjzNTfDqwScF23YEPU7x7DfE6OembmeWknP+tpoeA49P68cCDBfFBklpI2ozshO2o1BU0R1L/NGrnuIIytXL3jplZHg14cZakO4EBZH3/04HfAhcD90j6AfAe8B2AiJgo6R5gErAEODUilqZdnUI2EqgV8Gha6uSkb2aWQ0NOrRwRR9fy1N61bH8hcGEN8dHAdqtybCd9M7OcSuGKXCd9M7OcPPeOmVkZKYGc76RvZpZbCWR9J30zsxwEuaZYKHbKpmywxiTp38C7jV2PtaATMKuxK2GrpFQ/s00jovOa7EDSY2TvTx6zIqLeyc8ag5O+rTWSRjfgpei2DvgzK32+ItfMrIw46ZuZlREnfVub6ptK1oqPP7MS5z59M7My4pa+mVkZcdI3MysjTvolSlIPSRNWip0v6exatl9f0mxJ7VaK/0PSkQ1UpxfWoOwJkjZexTL/8R40RZKGSdpvpdgZkq5ugH3fmG68vTplB0j6+mqUmyYp73h3a2BO+gZARMwDnqDgxsrpC+AbwJAGOsYqJ4gCJwCrlPRLyJ3AoJVig1J8jUTEDyNi0moWHwCsyWdqjcBJvwylluMlkkZJelPS7umplZPLYcBjQIWkoZJekfSapEMK9nWcpPGSxkm6PcW6SHogxcZVtwYlzU2PA1Id7pU0WdLgdOcfJP1G0suSJki6XpkjgH7AYEljJbWS1FfScEljJD1ecJu5vumYLwKnrt13cp25FzhIUgvIfsGQfQF+V9JoSRMlXVC9saSdJb2Q3odRktpIqpR0afr8xks6PW07TFK/tD5X0oWp3EhJXVL8YEkvSXpV0lPp8+0BnAycmT6T3SV1lnRf+vxelrRbKt9R0hOp/HWUxAw2TVhEeCnBBegBTFgpdj5wNjAM+FOKHQA8ldabk92Xs2P6+zHgQLI5mtqmWCdgCtn/uL2BN4BO6bkO6fFu4Iy0Xgm0S+tz0+MA4DOye3pWAC8C3yjcR1q/HTg4rQ8D+qX1ZsALQOf091HAzWl9PPDNtP7Hld+DproADwOHpPVz02urfr8r0/uzQ/oM3wF2Ts+1TZ/fKcB9QNVKn1Xh+xoF7/cfgF+l9fYsH+n3w4J/O+cDZxfU8Y6Cz/ErwOtp/XLgN2n9wHScTo39npbr4gnXSldtY3Gr4/enxzFkXxBExCJJDwFHSLoP6EPW5SPgIkl7AMuAbkAXYC/g3oiYlcp/nPa5F9n9Oonstm6f1VCPURExHUDS2FSHEcCeks4B1gM6ABOBf65UdmuyuwU9mX4gVAIzUnfUBhExPG13O7B/Le9DU1P9K+zB9HgicKSkk8iSelegF9nnOyMiXgaIiM8BJO0DXBsRS1L84/84AixieVfeGOC/0np34O70a6o5MLWWOu4D9NLyScnaSmoD7AF8Ox33YUmfrPKrtwbjpF+6ZpO10Ap1YPn/sAvT41JW/HdwJ/ArskT/YEQslnQC0Bnom/6eBrRM26zuhR4LC9aXAlWSWgJXk7U835d0fjrOygRMjIhdVwhKG6xBfYrdP4A/S9qJ7H6on5D9ats5Ij6RdAt1fyZ5PqvFkZrjrPjv4grgzxHxkKQBZC38mlQAu0bE/BUOnH0JlOrn0uS4T79ERcRcstbv3gCSOgADyVrTdXkG6EnWH159orAdMDMl/D2BTVN8KFlrs2PBMarjp6RYpaS2OatdneBnSWoNHFHw3BygTVp/A+gsadd0jGaSekfEp8Bnkr6Rtjsm53GLXvo8hwE3k30ubYF5ZK+3C8t/0UwGNpa0M0Dqz68i+8V2clov/KzyaAd8kNaPL4gXfiakY5xW/YekPmn1WdJnIWl//rMxYuuQk35pOw74Veo+eRq4ICLerqtARCwj6/vtSPY/K8BgoJ+k0WT/805O204ku1nzcEnjgD+n7X9K1k3zGlk3Qe88lU1J+wbgNbKW7csFT98CXJteSyXZF8Il6bhjWT6K5PvAVelE7gotzhJwJ/BV4K6IGAe8Stb9dTPwPGRddGTnOK5I782TZF+mNwLvAeNT/LurcNzzgb9Leo4Vp13+J3BY9Ylc4Cdk/07GS5pEdqIX4AJgD0mvAPumelgj8TQMZmZlxC19M7My4qRvZlZGnPTNzMqIk76ZWRlx0jczKyNO+layJC1NwwknSPq7pPXWYF+3pDmA6p2ZUp590oqYk76VsvkR0ScitiObYuDkwiclVa7OTqP+mSkH4NknrUg56Vu5eA7YMrXCn5F0B/BaumL4j2lWyPGSfgygzJWSJkl6GNiwekcrzUw5UNnso+OUzUTaA88+aUXMc+9YyUtTD+xPNmsowC7AdhExNU1Y9llE7Kxs6uLnJT0B7Eg2sdv2ZJPLTSK78rVwv53JriDeI+2rQ0R8LOlashlFL03b3QFcFhEjJH0FeBzYFvgtMCIififpQOCktfpGmOGkb6WtVZq2AbKW/k1k3S6jIqJ64rl9gR2q++vJ5pnpSTYz5J1pltAPJT1dw/77A89W76uWmSvBs09aEXHSt1I2PyL6FAZS4p1XGAJOj4jHV9ruAOqfGTLvLKOefdKKhvv0rdw9DpwiqRmApK0krU822dyg1OffFdizhrIvAt+UtFkqWz1zpWeftKLlpG/l7kay/vpXlN1E/TqyX8APAG+Rzfh5DTB85YIR8W+yfvj708yVd6enPPukFS3PsmlmVkbc0jczKyNO+mZmZcRJ38ysjDjpm5mVESd9M7My4qRvZlZGnPTNzMrI/wdKFT1/2brzmAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fm_cm = plot_confusion_matrix(final_model, X_test, y_test, cmap=\"Blues\",\\\n",
    "                              display_labels=['UnVaccinated','Vaccinated'])\n",
    "fm_cm.ax_.set(xlabel='Predicted', ylabel='Actual')\n",
    "\n",
    "plt.savefig('./images/Confusion.jpeg', bbox_inches = 'tight');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix above shows the counts of our predicted target variable against the actual target variable counts. Our true positive counts and true negative counts are much more dense then our false positives and false negatives. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:19.512892Z",
     "start_time": "2022-04-22T17:25:17.137038Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = final_model.predict(X_test) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:19.518066Z",
     "start_time": "2022-04-22T17:25:19.514314Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<function sklearn.metrics._classification.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)>,\n",
       " <function sklearn.metrics._classification.recall_score(y_true, y_pred, *, labels=None, pos_label=1, average='binary', sample_weight=None, zero_division='warn')>,\n",
       " <function sklearn.metrics._classification.precision_score(y_true, y_pred, *, labels=None, pos_label=1, average='binary', sample_weight=None, zero_division='warn')>,\n",
       " <function sklearn.metrics._classification.f1_score(y_true, y_pred, *, labels=None, pos_label=1, average='binary', sample_weight=None, zero_division='warn')>)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score, recall_score, precision_score, f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:19.547829Z",
     "start_time": "2022-04-22T17:25:19.519574Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Our final model's accuracy on the test set is 0.79. \n",
      "\n",
      "Our final model's recall on the test set is 0.75 \n",
      "\n",
      "Our final model's precision on the test set is 0.78 \n",
      "\n",
      "Our final model's f1-score on the test is 0.77.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"\"\"\n",
    "Our final model's accuracy on the test set is {round(accuracy_score(y_test, y_hat), 2)}. \\n\n",
    "Our final model's recall on the test set is {round(recall_score(y_test, y_hat), 2)} \\n\n",
    "Our final model's precision on the test set is {round(precision_score(y_test, y_hat), 2)} \\n\n",
    "Our final model's f1-score on the test is {round(f1_score(y_test, y_hat), 2)}.\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:22.029454Z",
     "start_time": "2022-04-22T17:25:19.548898Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2SUlEQVR4nO3dd3xV9f348dc7exJGwgyBsKegRnArjrq1Vmud/Tmo+nXXtlb7raPqr9qq/bnaIrWKrRS01kFdWLW4LUv2BoEECGRA9ro3798f5yS9hCT3BHJzk9z38/HII/fs97mB8z6fz+ecz0dUFWOMMZErKtwBGGOMCS9LBMYYE+EsERhjTISzRGCMMRHOEoExxkS4mHAH0Fbp6ek6dOjQcIdhjDFdypIlSwpVNaO5ZV0uEQwdOpTFixeHOwxjjOlSRGRbS8usasgYYyKcJQJjjIlwlgiMMSbCWSIwxpgIZ4nAGGMiXMgSgYi8ICJ7RGRVC8tFRJ4WkU0iskJEjghVLMYYY1oWyhLBLODMVpafBYx0f64H/hjCWIwxxrQgZO8RqOqnIjK0lVUuAP6iTj/YX4tITxEZoKq7QhWTMcZ0tOo6PzW+emp99RRX1OKvV+pV8dcrvoDPe8pqiBbBV++sm19STVSUAODzK35Vcob04sRRzb4TdkjC+ULZICA3YDrPnXdAIhCR63FKDWRlZXVIcMaYyKKqVNX5Kav2UVZdR3VdPVV1fjbvKWdrUSU+fz2++v9ewP319ZRU1VFe48Pndy7ovnqlvnG58m1hBTW++naL8caThne7RCDNzGt2lBxVnQnMBMjJybGRdIwxjUoq6yitrqPOX0+dX6nz11NR46Okqo49ZTWUVNVRU+enqs5PcUUdZdV1FFXUUlxRS62vnlp/PXX+esqqffjrm7+8xEYLcdFRREcJMQ2/o4TEuGh6J8URFSXERkeRECuNy6JEGN0/lbjoKMYM6EFcTBTVtX4yeyW6+3HWiY5yfgBS4mNIiY8hNjqK2OgokuKjSYiJJjpKiBIQae6yeejCmQjygMEB05nAzjDFYowJk4aqkzr3glxTV09ptXOnvWNvFXvKaiitqmN3aTXFlXUUltVQUeujqtbPnrIaT8cQgbjoKPokx5GaEEvv5DjGD+xBfEw0cTHORTw5PoaeibEkx8eQlhhLfEwU8bHRDOmdRFbvpMZqmu4onIlgHnCLiMwFpgIl1j5gTPdRX68UVdSyu7SazQXlrN1VRlF5Dfuq6qio8VFW7WN7cSUlVXVB9xUXHUVGajzpKXH0SYljVHIKCbHR9EiMJUqE7PQkEmKj/3snHRdNWqJzwU9PiSc2WkJ2N90dhCwRiMgc4GQgXUTygPuBWABVnQG8C5wNbAIqgWtCFYsxpn1V1fpZs6uU/JJqiitqqKj1821BBYqSW1zFtqIK8kuraa6mZWTfFNISY+mVHMekwWn0SY6nR2IscdHSeCHvkRhLcnw0/XokMKhnIvExUXYhD6FQPjV0WZDlCtwcquMbYw5eVa2fdfmlbNhdxo69VRSU11JaXUdpVR25xZVsLapscdvJg3ty9LA+DOyZSJ+UOAakJTCkTzKZvRJJjovp1lUsXVWX64baGHPoSqvr2Li7jB37qiksq2HHviry9laSt7eKHfuq2Ff53+qa6CihV1IcPRJiSEuKZWh6Mhcensno/qkMTU+id3IcqfFOnbpd5LsmSwTGdDMNz6tX1voorfaRX1JNQVk1G/eUs2jrXmp8fr4trEADqm0SY6PJ7JVIZq9EJg/uyYC0BDJ7JTFhUBqZvRJJiI0O3wmZkLNEYEwXVuurZ497kd+QX8ZH6/aweGtxs3XzSXHRDEhLID4mmjtOHcWEQT0Y3DuJ9JR4eiXFWh18BLNEYEwn5q9XduytYuWOEooqaiiv8VFa5ePLzYXsKa1hd1n1fnf2o/qlcMNJwxvr4xPjohmYlki/HvGkp8Rb1Y1pliUCYzqBOn89q3aUsHZXGRt2l7GlsIKd+6rYtKf8gHVjo4X4mGh6JMRw2ykj6dcjgZH9UujfI4HBvZPCEL3p6iwRGNNBqmr9FJY7L0NtLaxkW1EFu0qqWZ63j2W5+xrv7ONiohjdL5URGSmcNCqDnomxTMnuTb8eCfRPS7BHKU2785QIRCQKmAQMBKqA1aq6O5SBGdMVVdX6yd1byY59VVTV+vl8UyHfbN/H5j3l1PoP7HMmNT6GIelJ3DJtBCP7pXLkkF6kp8QRH2ONs6bjtJoIRGQ48HPgNGAjUAAkAKNEpBJ4DnhJVduvVyVjupCKGh/vrcrn43W7WZdfxpaCigPWGdw7kWuOH0qPhNjGLg4yeyWSnZFMj4TYMERtzP6ClQgexhkn4Ab3BbBGItIXuBy4CngpNOEZ07n465Wd+6p4b9Uu3vhmJ1sLK6iq8zMwLYExA3pw7mEDGZiWwLCMFFITYuiVFEf/tIRwh21Mq1pNBK29Hayqe4An2zsgYzqT4opaPlyzm7X5pSzP3ceKvBJ8Ac9mnjm+P9NPyObIIb2s3t50WQfdWCwip6vqv9ozGGPCQVVZl1/Gmp2lrN1VSklVHXsra8nbW8W6/DIAEmKjGNgzkR8cNZjR/VPJTk9m8uCepFrVjukGDuWpoT8DNkqM6XI2F5SzdNteKmp8bNxTzoL1BezYVwU4F/zeSXEkxccwpHcSJ43O4Mzx/Zk8uKfd8ZtuK1hj8byWFgF92j8cYw6dqlJQVsO24ko+21CAr94ZeWrXvmo+WrebOv9/q3aS4qI5bkQ6N00bztTs3gzunWRP7JiIE6xEcAJwJdD0rRYBpoQkImMOQt7eSt5atpO5i7aTW1x1wPLUhBh6JsVyRFYvjhjSi+OGpzOqfwp9kuMbR4cyJlIFSwRfA5Wq+knTBSKyPjQhGeNNra+e++etZvXOElbklQCQnZ7MyaMzODKrFxMz0xjaJ5khfZKsWseYVgR7auisVpad2P7hGNM6f73y0drdzFm4nX+vLwAgPSWem04ezlkTBjAxMy3MERrT9VgXE6bTW59fxsfr9rA8dx+bC8rZ6Pa/M6Z/KjdNG8H5kwaGOUJjujZLBKZTKiyv4YkP1jNnYW7jvPiYKBLjovnlOWM5f/JA+qbai1rGtAdLBKbTqPPX89evtvHZxgK+2FxEra+eaaMzGNw7iVtPGUlGany4QzSmW7JEYMKqvl7ZWlTB0u37eHz+evJLq0mOi+a8wwZyw0nDGNUvNdwhGtPteU4EIvKAqj7Q0rQxXvnrlcVbi3lvVT6vL82jtNoHQEZqPL+7ZBLfOyIzzBEaE1naUiJYEmTamBb5/PV8vG4P/1iax9Lt+ygoqwHgnIkDOHFUOpMG92RERgox0VFhjtSYyOM5EajqP1ubNqY5u0ureeqjjby/Kp/iiloAjh7Wm/vOHceJIzNIS7K+eowJt2BdTDwDNDMMtkNVb2v3iEy3MG/5Tl76citLtu0F4PRx/bj4yExOGpVBQqx14WBMZxKsRLC4Q6Iw3cIXmwr5ZEMBL325lRqfM1bRj08bxQWTBzI0PTnM0RljWhLszeL9BpwRkWRVPXAIJhORan31vLNyJ++uzOdfa/47cml6ShxDk+N55Yaj6ZkUF8YIjTFeeB2z+BicbqdTgCwRmYQzatlNoQzOdD5l1XX89ettzF+9m7U7S6n119O/RwKnjOlLvx4J3HP2GBt+0Zguxmtj8ZPAGcA8AFVdLiLW11CEKKms419rd/P5xgLeXLazcf6lRw3mtLH9OHl0hj3tY0wX1panhnKb9ODob/9wTGfir1dm/2cb9721GoA+yXFcMHkg5x42kFPG9LXum43pJrwmglwRORZQEYkDbgPWhi4sE27zV+fz0NtryNtbRVx0FDOuOoKTR/Ulyi7+xnQ7XhPBjcBTwCBgBzAfuDlUQZnwUFXeXZnPnz7bwrLcfcRGC1cfO5SfnjGalHjrjcSY7srT/25VLQSuaOvOReRMnAQSDTyvqo82WZ4GvIwz9nEM8LiqvtjW45hDo6r8c8Uu/vz5tyzP3UdqfAx3nDaS647PtsHZjYkAXp8aGoZzQT8a5wWzr4Afq+qWVraJBn4PnA7kAYtEZJ6qrglY7WZgjaqeJyIZwHoRma2qtQd3OsYrVeXFL7by8bo9fL6psHH+/5w8nNtPHWkvfRkTQbyW9/+Gc1G/0J2+FJgDTG1lmynApoZkISJzgQuAwESgQKo4rdApQDHg8xy9abPKWh/PfryJuYtyG7t8mDS4JyeNyuDmacNt4HZjIpDXRCCq+teA6ZdF5JYg2wwCcgOm8zgwcTyL80jqTiAV+IGq1h9wcJHrgesBsrKyPIZsmnpnxS4eeW8teXurGJCWwK8vnMj3jhhkd//GRLhgfQ31dj/+W0TuBubi3MX/AHgnyL6be7ykab9FZwDLgFOA4cC/ROQzVS3dbyPVmcBMgJycnBb7PjLNW/htMTfNXkphudPj5wtX5zBtdF8b0N0YAwQvESzBuXg3XDFuCFimwEOtbJsHDA6YzsS58w90DfCoqiqwSUS+BcYAC4PEZTyas3A797y+EoAfnZDNT74z2koAxpj9BOtrKPsQ9r0IGCki2TiPnF4KXN5kne3AqcBnItIPGA202ABtvFNVXlmUyy/ecJLAmzcfx+TBPcMblDGmU2rLCGUTgHFA44jhqvqXltZXVZ/bjjAf5/HRF1R1tYjc6C6fgVOimCUiK3FKHT93H1U1h6Cixsf0lxbz1ZYixg7owas3HG2PgRpjWuT18dH7gZNxEsG7wFnA50CLiQBAVd911w+cNyPg807gO22K2LSquKKWH7+yjK+2FJHVO4k3bjrWqoKMMa3yWiK4GJgEfKOq17jVOM+HLixzMGb/ZxsPzFtNnV+58/RR3HrKCGsQNsYE5TURVKlqvYj4RKQHsAcYFsK4TBv4/PU8++9NPPnhRqZk9+bO00cxNbu3JQFjjCdeE8FiEekJ/AnnSaJy7MmeTqHG5yfnoQ8pq/GR2SuRpy89nP5pCcE3NMYYl9e+hhoGoJkhIu8DPVR1RejCMl7U+uqZ/tJiymp8XHxkJr+96DDrHdQY02bBXig7orVlqrq0/UMyXry1bAd3vrocf71y3fHZ3HvuuHCHZIzpooKVCJ5oZZnivBFsOtiv313LzE+d1y1+fNoobj9tZJgjMsZ0ZcFeKJvWUYGY4Pz1ym/nr2Pmp1vo3yOB+T8+kbREez/AGHNobLSRLuSht9cw68utHD8inWcuO9ySgDGmXVgi6AJqffXc99Yq5i7K5fxJA3nq0sn2aKgxpt1YIujk/PXKdS8t4rONhVySk8lD351gScAY066ivKwkjitF5D53OktEpoQ2NANw9YsL+WxjIdcdn81vL55kA8cYY9qdp0QA/AE4BrjMnS7DGbHMhIjPX8/j89fz2cZCjhzSi1+eMzbcIRljuimvVUNTVfUIEfkGQFX3ikhcCOOKaD5/Pbf87RveX53PoJ6JPGzVQcaYEPKaCOrcwegVwB1o/oAhJU37+OOCzby/Op/LpmTxyPcmhjscY0w357Vq6GngDaCviPxfnC6ofx2yqCLYirx9PPGvDeQM6cWvL5wQ7nCMMRHAa19Ds0VkCc5oYgJ8V1XXhjSyCFRV6+f8Z78gSrCng4wxHcbrwDRPAa+oqjUQh9CTH24A4M7TRzF2QI8wR2OMiRReq4aWAr8UkU0i8piI5IQyqEi0q6SK5z7dQnxMFD860YZ6MMZ0HE+JQFVfUtWzgSnABuA3IrIxpJFFmOc+cTqRe/qyw+1dAWNMh/JaImgwAhgDDAXWtXs0EWrD7jJmfbmVCYN6cMb4/uEOxxgTYby+WdxQAngQWA0cqarnhTSyCKGq3PWaM8bPo987LMzRGGMikdf3CL4FjlHVwlAGE2lUlatfXMSy3H1Mze7NhEFp4Q7JGBOBgo1QNkZV1+GMT5wlIlmBy22EskMz45MtfLKhgKOG9uLl6VPDHY4xJkIFKxHcCVxP8yOV2Qhlh+g37zvNLHN+dDQx0W1trjHGmPYRbISy692PZ6lqdeAyEUkIWVQRYNWOEgCuPDrLkoAxJqy8XoG+9DjPeLC5oJxLZ35NXEwU04+3dwaMMeEVrI2gPzAISBSRw3G6lwDoASSFOLZu6/a53yDAe7efwND05HCHY4yJcMHaCM4ArgYygd8FzC8DfhGimLq1JduKWbWjlO9OHsjwjJRwh2OMMUHbCF4CXhKRi1T1Hx0UU7e1u7Sa6S8tpmdSLD8/a0y4wzHGGCB41dCVqvoyMFRE7my6XFV/18xmpgWPz1/P3so6nr38cAakJYY7HGOMAYI3FjdUYKcAqc38tEpEzhSR9W5ndXe3sM7JIrJMRFaLyCdtiL1Lqa9XFmwo4ISR6Zx72MBwh2OMMY2CVQ095/7+VVt37I5o9nvgdCAPWCQi81R1TcA6PXHGQz5TVbeLSN+2HqerePyD9RSU1XDTycPDHYoxxuzHa19DvxWRHiISKyIfiUihiFwZZLMpwCZV3aKqtcBc4IIm61wOvK6q2wFUdU9bT6Cr2F1aA8DVxw4NbyDGGNOE1/cIvqOqpcC5OHf3o4CfBdlmEJAbMJ3nzgs0CuglIgtEZImI/LC5HYnI9SKyWEQWFxQUeAy586jx+flycyF9U+Nt1DFjTKfjNRHEur/PBuaoarGHbZq74mmT6RjgSOAcnEdV7xWRUQdspDpTVXNUNScjI8NjyJ3HC59vZVdJNY99f1K4QzHGmAN47X30nyKyDqgCbhKRDKA6yDZ5wOCA6UxgZzPrFKpqBVAhIp8Ck3AGv+k2Ptng1HidNKrrJTFjTPfndYSyu4FjgBxVrQMqOLC+v6lFwEgRyRaROOBSYF6Tdd4CThCRGBFJAqYCa9tyAp1dZa2Pr7cUc8XUrOArG2NMGHgdvD4WuAo40a3j/gSY0do2quoTkVuA+UA08IKqrhaRG93lM1R1rYi8D6wA6oHnVXXVQZ9NJ/TIu04Po2dNGBDmSIwxpnleq4b+iNNO8Ad3+ip33vTWNlLVd4F3m8yb0WT6MeAxj3F0KV9vKeKvX2/j2OF9OH5kerjDMcaYZnlNBEepamBL58cisjwUAXUX24oq+OELC0lPiefJH0wOdzjGGNMir08N+UWk8U0oERkG+EMTUtdXVF7DBb//giiBv/1oKn172NANxpjOy2uJ4GfAv0VkC85joUOAa0IWVRemqlwzaxH7Kut45HsTGdUvaE8cxhgTVkETgfuoaAnOm8J9cRLBOlWtCXFsXdK9b61iRV4Jl03J4rIp9qSQMabza7VqSESmA6uBZ4BlwFBVXW5JoHlLtu3l5a+3M7pfKg9dMD7c4RhjjCfBSgR3AONVtcBtF5jNge8CGNdrS3KJi4nitf85xsYhNsZ0GcGuVrWqWgCgqluA+NCH1DUtWL+HOQtzOWtCf1ITYoNvYIwxnUSwEkGmiDzd0rSq3haasLqWlXklXDtrET2TYrll2ohwh2OMMW0SLBE07WF0SagC6aq2FVVw3UuLSE2I5Z+3HM/g3knhDskYY9rEy5jFpgUllXVc9eeF7CmrYfb0qZYEjDFdUrCnhmaKyIQWliWLyLUickVoQuv87njlG7YXV3La2L4cN8K6kDDGdE3Bqob+ANwnIhOBVUABkACMBHoAL+A8SRRx6uuVNbtKOWN8P567Kifc4RhjzEELVjW0DLhERFKAHGAAzpgEa1V1fejD67zmLspld2kN00Z322GWjTERwlMXE6paDiwIbShdR3Wdn0feXUtW7yQuyRkcfANjjOnE7K2ngzB34XbKanz84uwxREXZGMTGmK7NEkEb7S6t5o+fbCY9JZ7vjOsf7nCMMeaQtSkRiEhyqALpKq55cRGF5bX8+f/kWGnAGNMteEoEInKsiKzBHU9YRCaJyB+CbNbt+Pz1rNlVymGZaUwa3DPc4RhjTLvwWiL4f8AZQBGAqi4HTgxVUJ3VuvwyAE4alRHmSIwxpv14rhpS1dwmsyJuhLK3lu0AYEp27zBHYowx7cfrCGW5InIsoCISB9yGW00UKfz1yt+X5NE7OY6jhloiMMZ0H15LBDcCNwODgDxgMnBTiGLqlN5ZuYt9lXVcf+IwYm2sAWNMN+K1RDBaVffrU0hEjgO+aP+QOqeH314DwPeOGBTmSIwxpn15vbV9xuO8bqmixkdRRS1j+qfSNzUh3OEYY0y7arVEICLHAMcCGSJyZ8CiHkB0KAPrTHaVVOOvV64/cVi4QzHGmHYXrGooDkhx10sNmF8KXByqoDqbTXvKARjUMzHMkRhjTPsL1vvoJ8AnIjJLVbd1UEydTt7eSgCy+tjAM8aY7sdrY3GliDwGjMcZjwAAVT0lJFF1MqVVdQD0s/YBY0w35LWxeDawDsgGfgVsBRaFKKZO59uiSgb1TLS+hYwx3ZLXRNBHVf8M1KnqJ6p6LXB0COPqVP6zpYiM1Phwh2GMMSHhNRHUub93icg5InI4kBmimDqVxVuL2VNWw/iBPcIdijHGhITXRPCwiKQBPwF+CjwP3BFsIxE5U0TWi8gmEbm7lfWOEhG/iHS6J5Huem0FvZPj+PHpo8IdijHGhITXoSrfdj+WANOg8c3iFolINPB74HScbikWicg8VV3TzHq/Aea3LfTQ27Gvii2FFUw/Ppv0FKsaMsZ0T62WCEQkWkQuE5GfisgEd965IvIl8GyQfU8BNqnqFlWtBeYCFzSz3q3AP4A9bQ8/tFbtKAHglLE2QL0xpvsKViL4MzAYWAg8LSLbgGOAu1X1zSDbDgICu67OA6YGriAig4ALgVOAo1rakYhcD1wPkJWVFeSw7eff6/YQJTB+QFqHHdMYYzpasESQAxymqvUikgAUAiNUNd/Dvpt71lKbTD8J/FxV/SItP5qpqjOBmQA5OTlN9xESpdV1vLo4l1PG9CUtKbYjDmmMMWERLBHUqmo9gKpWi8gGj0kAnBLA4IDpTGBnk3VygLluEkgHzhYRn4fSRsh9uamIeoUfnWD9CxljurdgiWCMiKxwPwsw3J0WQFX1sFa2XQSMFJFsYAdwKXB54Aqqmt3wWURmAW93hiQA8G1hBQDZ6clhjsQYY0IrWCIYe7A7VlWfiNyC8zRQNPCCqq4WkRvd5TMOdt8d4b1VuxjdL9VeJDPGdHvBOp07pI7mVPVd4N0m85pNAKp69aEcqz3lFleyIq+E6cdn01rbhTHGdAc25mIz7nl9JQAXTLbRyIwx3Z8lgiZUlc83FXL2xP5MzLTHRo0x3Z/nRCAiiSIyOpTBdAabC5xBaEb2TQ2ypjHGdA+eEoGInAcsA953pyeLyLwQxhU2ucVVABwzvE+YIzHGmI7htUTwAE6XEfsAVHUZMDQUAYXboq3FREcJh1m1kDEmQnhNBD5VLQlpJJ2AqvLmNzs4fHBPkuK8Dt5mjDFdm9dEsEpELgeiRWSkiDwDfBnCuMLiX2t2s7OkmvMnDwx3KMYY02G8JoJbccYrrgH+htMd9R0hiilsNuwuA+DUsf3CHIkxxnQcr/Ufo1X1f4H/DWUw4fa3/2xnQFoCg3omhjsUY4zpMF5LBL8TkXUi8pCIjA9pRGFSXednV2k1h2f1DHcoxhjToTwlAlWdBpwMFAAzRWSliPwylIF1tKKKWlThmOHp4Q7FGGM6lOcXylQ1X1WfBm7EeafgvlAFFQ7fbN8LQGYvqxYyxkQWry+UjRWRB0RkFc4QlV/ijC/QbTz/2bfEx0RxrL1IZoyJMF4bi18E5gDfUdWmg8t0eZW1PtbuKiU7PZn4mOhwh2OMMR3KUyJQ1aNDHUg4bS+upMZXzzXHDQ13KMYY0+FaTQQi8qqqXiIiK9l/vGEvI5R1GTv3Of0LDe1jo5EZYyJPsBLB7e7vc0MdSDitz3d6HB3RNyXMkRhjTMdrtbFYVXe5H29S1W2BP8BNoQ+vY3y0djf9eyTQOzku3KEYY0yH8/r46OnNzDurPQMJl883FrJ4214unTLYhqU0xkSkYG0E/4Nz5z9MRFYELEoFvghlYB3l7RXOQ1DnTBwQ5kiMMSY8grUR/A14D3gEuDtgfpmqFocsqg60raiS3slxjOxnI5IZYyJTsESgqrpVRG5uukBEeneHZOBXpdZXH+4wjDEmbLyUCM4FluA8PhpYia7AsBDF1WEqanzW0ZwxJqK1mghU9Vz3d3bHhNOxfP56thRUMCW7d7hDMcaYsPHa19BxIpLsfr5SRH4nIlmhDS305q/eTVWdn6OGWiIwxkQur4+P/hGoFJFJwF3ANuCvIYuqg6zPLwXgxFEZYY7EGGPCpy2D1ytwAfCUqj6F8whpl/bmsp1kpyeTEm8D1RtjIpfXRFAmIvcAVwHviEg0EBu6sEKvsLyG7cWVpCV26dMwxphD5jUR/ABn4PprVTUfGAQ8FrKoOsDqnU610AWTB4Y5EmOMCS+vQ1XmA7OBNBE5F6hW1b+ENLIQq6zxAVhDsTEm4nl9augSYCHwfeAS4D8icrGH7c4UkfUisklE7m5m+RUissL9+dJtjO4QH6/bQ5TAkD5JHXVIY4zplLy2kv4vcJSq7gEQkQzgQ+C1ljZw2xF+j9NhXR6wSETmqeqagNW+BU5S1b0ichYwE5ja9tNou29y99EnJZ7UBGsjMMZENq9tBFENScBV5GHbKcAmVd2iqrXAXJynjhqp6pequted/JoOGgd5b0Utm/aUc8b4fh1xOGOM6dS8lgjeF5H5OOMWg9N4/G6QbQYBuQHTebR+t38dTgd3BxCR64HrAbKyDv09th3uiGTDM2wgGmOM8Tpm8c9E5HvA8Tj9Dc1U1TeCbNZc5/7azDxEZBpOIji+hePPxKk2Iicnp9l9tEXe3koARvbt8q9CGGPMIQs2HsFI4HFgOLAS+Kmq7vC47zxgcMB0JrCzmWMcBjwPnKWqRR73fUgqavwA9g6BMcYQvJ7/BeBt4CKcHkifacO+FwEjRSRbROKAS4F5gSu4/RW9DlylqhvasO9DsreyFoCs3vbEkDHGBKsaSlXVP7mf14vIUq87VlWfiNwCzAeigRdUdbWI3OgunwHcB/QB/uAOE+lT1Zy2nkRbVdc5JYLEuOhQH8oYYzq9YIkgQUQO57/1/YmB06raamJQ1Xdp0qjsJoCGz9OB6W0N+lD56p1mhpgoG6PYGGOCJYJdwO8CpvMDphU4JRRBhZq/XokSiLJEYIwxQQemmdZRgXSkvZW1JMdZj6PGGAPeXyjrVnbsrWJAz4Rwh2GMMZ1CRCaCiho/vZLiwh2GMcZ0ChGZCBZuLaaHvUNgjDGA995HxR2r+D53OktEpoQ2tNAoLK8BIC46InOgMcYcwOvV8A/AMcBl7nQZTs+iXU7DOwQn2TjFxhgDeO90bqqqHiEi3wC43UZ3yUp2n999hyDaHh01xhjwXiKoc8cXUGgcj6A+ZFGFULXPKRHEWNWQMcYA3hPB08AbQF8R+b/A58CvQxZVCK3MKwEgPaVLFmiMMabdee2GeraILAFOxele4ruqujakkYVIQx/W1uGcMcY4PCUCt5fQSuCfgfNUdXuoAguVOr9ToxUXY1VDxhgD3huL38G5mRYgAcgG1gPjQxRXyGwrqiQuJooeNlaxMcYA3quGJgZOi8gRwA0hiSjE9pRW079HAgmx1gW1McbAQb5Z7HY/fVQ7x9IhdpfW0CvZGoqNMaaB1zaCOwMmo4AjgIKQRBRi24oqOHJo73CHYYwxnYbXNoLAUd59OG0G/2j/cEKvpKqOPlYiMMaYRkETgfsiWYqq/qwD4gm5Wn89STZEpTHGNGq1jUBEYlTVj1MV1OXt3FdFnV+tC2pjjAkQrESwECcJLBORecDfgYqGhar6eghja3cFZU7Po2nWBbUxxjTy2kbQGyjCGaO44X0CBbpUIthVUgVgo5MZY0yAYImgr/vE0Cr+mwAaaPObdF7uS8XWRmA6vbq6OvLy8qiurg53KKaLSUhIIDMzk9hY7zUfwRJBNJDC/gmgQZdLBOU1dQD0To4PcyTGtC4vL4/U1FSGDh2KiHWZbrxRVYqKisjLyyM7O9vzdsESwS5VffDQQus8Pt9UBEDfVEsEpnOrrq62JGDaTETo06cPBQVte80r2JvF3epfYcPJJMd7bRoxJnwsCZiDcTD/boIlglMPLpTOqbzGx5j+qcFXNMaYCNJqIlDV4o4KpCMsz91HzyR7dNQYL6Kjo5k8eTITJkzg+9//PpWVlSxevJjbbrvtoPeZkpICwM6dO7n44ovbK1TuuOMOPv3008bpgoICYmNjee6555o9foNZs2Zxyy23NE7/5S9/YcKECYwfP55x48bx+OOPH3Js77//PqNHj2bEiBE8+uijza5TUlLCeeedx6RJkxg/fjwvvvhi47J9+/Zx8cUXM2bMGMaOHctXX30FwE9/+lM+/vjjQ44PDrLTua5KBGp8XXKETWM6XGJiIsuWLWPVqlXExcUxY8YMcnJyePrppw953wMHDuS1115rhyihuLiYr7/+mhNPPLFx3t///neOPvpo5syZ43k/7733Hk8++SQffPABq1evZunSpaSlpR1SbH6/n5tvvpn33nuPNWvWMGfOHNasWXPAer///e8ZN24cy5cvZ8GCBfzkJz+htrYWgNtvv50zzzyTdevWsXz5csaOHQvArbfe2mJiaauIqiyvV5gw8ND+sMZ0tF/9czVrdpa26z7HDezB/ed5H07khBNOYMWKFSxYsIDHH3+ct99+mwceeIDNmzezY8cOcnNzueuuu/jRj34EwGOPPcarr75KTU0NF154Ib/61a/229/WrVs599xzWbVqFbNmzWLevHlUVlayefNmLrzwQn77298C8MEHH3D//fdTU1PD8OHDefHFFw+4q3/ttdc488wz95s3Z84cnnjiCS6//HJ27NjBoEGDgp7jI488wuOPP87AgQMB5zHMhvM5WAsXLmTEiBEMGzYMgEsvvZS33nqLcePG7beeiFBWVoaqUl5eTu/evYmJiaG0tJRPP/2UWbNmARAXF0dcnNMzwpAhQygqKiI/P5/+/fsfUpwRUyKor1eKK2qJibYGOGPawufz8d577zFx4sQDlq1YsYJ33nmHr776igcffJCdO3fywQcfsHHjRhYuXMiyZctYsmTJftU2zVm2bBmvvPIKK1eu5JVXXiE3N5fCwkIefvhhPvzwQ5YuXUpOTg6/+93vDtj2iy++4Mgjj2yczs3NJT8/nylTpnDJJZfwyiuveDrPVatW7beflsyePZvJkycf8NNcVdeOHTsYPHhw43RmZiY7duw4YL1bbrmFtWvXMnDgQCZOnMhTTz1FVFQUW7ZsISMjg2uuuYbDDz+c6dOnU1HR2LkDRxxxBF988YWn82tNxJQICiuc7iVqrWrIdDFtuXNvT1VVVUyePBlwSgTXXXcdX3755X7rXHDBBSQmJpKYmMi0adNYuHAhn3/+OR988AGHH344AOXl5WzcuHG/qpumTj311MZqmHHjxrFt2zb27dvHmjVrOO644wCora3lmGOOOWDbXbt2kZGR0Tg9d+5cLrnkEsC5A7/uuuu48847D9iuQVufsrniiiu44oorPK2reuDrVs0db/78+UyePJmPP/6YzZs3c/rpp3PCCSfg8/lYunQpzzzzDFOnTuX222/n0Ucf5aGHHgKgb9++7Ny5s03xNyekiUBEzgSewnkx7XlVfbTJcnGXn40zJvLV7qA37a60ygfAUTYWgTGeNLQRtKbpRU1EUFXuuecebrjB+yCG8fH/fbcnOjoan8+HqnL66acHredPTEzc7w3sOXPmsHv3bmbPng04DdMbN25k5MiRJCYmUltb21i9UlxcTHp6OgDjx49nyZIlnHLKKa0eb/bs2Tz22GMHzB8xYsQB7R6ZmZnk5uY2Tufl5TVWPQV68cUXufvuuxERRowYQXZ2NuvWrSMrK4vMzEymTp0KwMUXX7xfu0B1dTWJiYmtxutFyKqG3O6rfw+cBYwDLhORcU1WOwsY6f5cD/wxVPEUljslgkTrXsKYdvPWW29RXV1NUVERCxYs4KijjuKMM87ghRdeoLy8HHCqR/bs2dPmfR999NF88cUXbNq0CYDKyko2bNhwwHpjx45tXGf9+vVUVFSwY8cOtm7dytatW7nnnnuYO3cuACeddBIvv/wy4JR4Xn31VaZNmwbAPffcw1133UV+fj4ANTU1zTaMX3HFFSxbtuyAn+Yav4866ig2btzIt99+S21tLXPnzuX8888/YL2srCw++ugjAHbv3s369esZNmwY/fv3Z/Dgwaxfvx6Ajz76aL/2hQ0bNjBhwgSP32jLQlkimAJsUtUtACIyF7gACGwyvwD4izrlp69FpKeIDFDVXe0dTGmV071Eoo1VbEy7mTJlCueccw7bt2/n3nvvZeDAgQwcOJC1a9c2VuOkpKTw8ssv07dv3zbtOyMjg1mzZnHZZZdRU+PcyD388MOMGjVqv/XOOeccnnvuOaZPn86cOXO48MIL91t+0UUXcemll3Lvvffy1FNPccMNN/D000+jqvzwhz9srLI6++yz2b17N6eddhqqiohw7bXXHuxXA0BMTAzPPvssZ5xxBn6/n2uvvZbx452qvhkzZgBw4403cu+993L11VczceJEVJXf/OY3jSWVZ555hiuuuILa2lqGDRvW+GhpXV0dmzZtIicn55BiBJw6rFD8ABfjVAc1TF8FPNtknbeB4wOmPwJymtnX9cBiYHFWVpYejMVbi/TGvy7W/JKqg9remI60Zs2acIcQ1P3336+PPfZYuMNQVdXjjjtO9+7dG+4wOtTrr7+uv/zlL5td1ty/H2CxtnC9DmWJwEtHdZ46s1PVmcBMgJycnIPq7O7IIb05coi1DxjTHT3xxBNs376dnj17hjuUDuPz+fjJT37SLvsKZSLIAwYHTGcCTZu3vaxjjOmEHnjggXCH0KihMTWSfP/732+3fYXyPYJFwEgRyRaROOBSYF6TdeYBPxTH0UCJhqB9wJiuSJt59NCYYA7m303ISgSq6hORW4D5OI+PvqCqq0XkRnf5DOBdnEdHN+E8PnpNqOIxpitJSEigqKiIPn36WC+kxjN1xyNISGjbKIzS1e46cnJydPHixeEOw5iQshHKzMFqaYQyEVmiqs0+YhQxbxYb05XExsa2aYQpYw5FxPQ1ZIwxpnmWCIwxJsJZIjDGmAjX5RqLRaQA2HaQm6cDhe0YTldg5xwZ7Jwjw6Gc8xBVzWhuQZdLBIdCRBa31GreXdk5RwY758gQqnO2qiFjjIlwlgiMMSbCRVoimBnuAMLAzjky2DlHhpCcc0S1ERhjjDlQpJUIjDHGNGGJwBhjIly3TAQicqaIrBeRTSJydzPLRUSedpevEJEjwhFne/Jwzle457pCRL4UkUnhiLM9BTvngPWOEhG/iFzckfGFgpdzFpGTRWSZiKwWkU86Osb25uHfdpqI/FNElrvn3KV7MRaRF0Rkj4isamF5+1+/Whq6rKv+4HR5vRkYBsQBy4FxTdY5G3gPZ4S0o4H/hDvuDjjnY4Fe7uezIuGcA9b7GKfL84vDHXcH/J174owLnuVO9w133B1wzr8AfuN+zgCKgbhwx34I53wicASwqoXl7X796o4lginAJlXdoqq1wFzggibrXAD8RR1fAz1FZEBHB9qOgp6zqn6pqnvdya9xRoPryrz8nQFuBf4B7OnI4ELEyzlfDryuqtsBVLWrn7eXc1YgVZyBG1JwEoGvY8NsP6r6Kc45tKTdr1/dMREMAnIDpvPceW1dpytp6/lch3NH0ZUFPWcRGQRcCMzowLhCycvfeRTQS0QWiMgSEflhh0UXGl7O+VlgLM4wtyuB21W1vmPCC4t2v351x/EImhvOqekzsl7W6Uo8n4+ITMNJBMeHNKLQ83LOTwI/V1V/Nxnly8s5xwBHAqcCicBXIvK1qm4IdXAh4uWczwCWAacAw4F/ichnqloa4tjCpd2vX90xEeQBgwOmM3HuFNq6Tlfi6XxE5DDgeeAsVS3qoNhCxcs55wBz3SSQDpwtIj5VfbNDImx/Xv9tF6pqBVAhIp8Ck4Cumgi8nPM1wKPqVKBvEpFvgTHAwo4JscO1+/WrO1YNLQJGiki2iMQBlwLzmqwzD/ih2/p+NFCiqrs6OtB2FPScRSQLeB24qgvfHQYKes6qmq2qQ1V1KPAacFMXTgLg7d/2W8AJIhIjIknAVGBtB8fZnryc83acEhAi0g8YDWzp0Cg7Vrtfv7pdiUBVfSJyCzAf54mDF1R1tYjc6C6fgfMEydnAJqAS546iy/J4zvcBfYA/uHfIPu3CPTd6POduxcs5q+paEXkfWAHUA8+rarOPIXYFHv/ODwGzRGQlTrXJz1W1y3ZPLSJzgJOBdBHJA+4HYiF01y/rYsIYYyJcd6waMsYY0waWCIwxJsJZIjDGmAhnicAYYyKcJQJjjIlwlggigNvz5rKAn6GtrFveDsebJSLfusdaKiLHHMQ+nheRce7nXzRZ9uWhxujup+F7WeX2XtkzyPqTReTsgzjOABF52/18soiUiMg3IrJWRO4/iP2d39ALp4h8t+F7cqcfFJHT2rrPZo4xS4L01up2Y+H5EWT33N/2sF6zvW+KyOMicorX4xnvLBFEhipVnRzws7UDjvkzVZ0M3A0819aNVXW6qq5xJ3/RZNmxhx4e8N/vZQJOJ183B1l/Ms7z2211J/CngOnPVPVwnDefrxSRI9uyM1Wdp6qPupPfBcYFLLtPVT88iBg7k1nAmc3Mfwbn35NpZ5YIIpCIpIjIR+7d+koROaDXTvcu9tOAO+YT3PnfEZGv3G3/LiIpQQ73KTDC3fZOd1+rROQOd16yiLwjTl/yq0TkB+78BSKSIyKPAoluHLPdZeXu71cC79Ddu9iLRCRaRB4TkUXi9Nd+g4ev5SvcjrtEZIo4YzZ84/4e7b7V+iDwAzeWH7ixv+Ae55vmvkfXRcD7TWe63UAsAYa7pY2v3XjfEJFebiy3icgad/5cd97VIvKsiBwLnA885sY0vOFOXkTOEpFXA76bk0Xkn+7nNv0NReQ+9xxXichMkf06brrS/Y5WicgUd32v30uzWup9U1W3AX1EpH9b9mc86Kg+tu0nfD+AH6dTrmXAGzhvlPdwl6XjvKHY8HJhufv7J8D/up+jgVR33U+BZHf+z4H7mjneLNy+/4HvA//B6QhtJZCM01XwauBwnIvknwK2TXN/LwByAmMKWKchxguBl9zPcTg9MiYC1wO/dOfHA4uB7GbiLA84v78DZ7rTPYAY9/NpwD/cz1cDzwZs/2vgSvdzT5z+fJKbHCMbWBIwfTLwtvu5D7AVGI/zJvBJ7vwHgSfdzzuB+IZjNI0j8LsOnHb/xtsD/lZ/BK48yL9h74D5fwXOC/gb/cn9fCJu//ktfS9Nzj0H563nlv7NDqWZ/vhxSlYXhfv/VHf76XZdTJhmValTTQOAiMQCvxaRE3G6IRgE9APyA7ZZBLzgrvumqi4TkZNwqiG+cG8K43DupJvzmIj8EijA6e30VOANde6CEZHXgRNw7pQfF5Hf4FwkPmvDeb0HPC0i8ThVCZ+qapWIfAc4LKCOOw0YCXzbZPtEEVmGc9FZAvwrYP2XRGQkTq+OsS0c/zvA+SLyU3c6Achi/759BrjfQaATROQbnO/+UZxOxHqqasNoYi/hJCZwEsRsEXkTeLOFOA6gTtcM7wPnichrwDnAXUBb/oYNponIXUAS0Bsnif/TXTbHPd6nItJDnHaWlr6XwPgWA9O9nk+APcDAg9jOtMISQWS6AmckpyNVtU5EtuL8Z23k/sc+EecC8lcReQzYC/xLVS/zcIyfqeprDRPSQgOmqm5w68jPBh4RkQ9U9UEvJ6Gq1SKyAKcb4h/gXpRw+pu5VVXnB9lFlapOFpE04G2cNoKncfqu+beqXihOw/qCFrYXnLvT9a0dgybfLU4bwbmNO3GO35JzcO62zwfuFZHxrazb1Cs451QMLFLVMrdax+vfEBFJAP6AUzrLFZEH2P98mvZRo7TwvYjTIdyhSsD5Tk07sjaCyJQG7HGTwDRgSNMVRGSIu86fgD/jDJ33NXCciDTU+SeJyCiPx/wU+K67TTJOtc5nIjIQqFTVl4HH3eM0VeeWTJozF6fTrRNwOibD/f0/DduIyCj3mM1S1RLgNuCn7jZpwA538dUBq5bhVJE1mA/c2lBnLiKHN7P7DTgljha5x98rbjsMcBXwiYhEAYNV9d84d/M9carVAjWNKdACnO/zRzhJAdr+N2y46Be6bQlNnyRqaNM5HqcXzBK8fS8HaxTQZTvR66wsEUSm2UCOiCzGKR2sa2adk4FlbhXGRcBTqlqAc2GcIyIrcC4qY7wcUFWX4tQ7L8RpM3heVb8BJgIL3Sqa/wUebmbzmcAKcRuLm/gA5475Q3WGMgRnzIU1wFJxHkF8jiClXzeW5TjdHP8Wp3TyBU77QYN/A+MaGotxSg6xbmyr3Omm+60ANjdceFvxf3Cq01bgPJ30oHvsl8XpVfMb4P+p6r4m280FfuY2yg5vcmw/TknnLPc3bf0busf7E077zps4VYaB9orzOO8MnCpA8PC9iPMgwPPNHVOc3je/AkaLSJ6IXOfOj8V58GBxS/Gag2O9jxoTYiJyIU413C/DHUtX5n6PR6jqveGOpbuxNgJjQkxV3xCRPuGOoxuIAZ4IdxDdkZUIjDEmwlkbgTHGRDhLBMYYE+EsERhjTISzRGCMMRHOEoExxkS4/w/oY9qH/Dkg9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fm1_roc_curve = plot_roc_curve(final_model, X_test, y_test)\n",
    "plt.savefig('./images/ROC_Curve.jpeg', bbox_inches = 'tight');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take it one step further and choose a specific threshold to optimize our scores. The code to calculate on optimal threshold was provided by https://machinelearningmastery.com/threshold-moving-for-imbalanced-classification/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:24.444087Z",
     "start_time": "2022-04-22T17:25:22.030440Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Threshold=0.471400, G-Mean=0.791\n"
     ]
    }
   ],
   "source": [
    "y_hat = final_model.predict_proba(X_test)[:, 1]\n",
    "# calculate roc curves\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_hat)\n",
    "# calculate the g-mean for each threshold\n",
    "gmeans = sqrt(tpr * (1-fpr))\n",
    "# locate the index of the largest g-mean\n",
    "ix = argmax(gmeans)\n",
    "print('Best Threshold=%f, G-Mean=%.3f' % (thresholds[ix], gmeans[ix]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.350343Z",
     "start_time": "2022-04-22T17:25:24.445202Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = (final_model.predict_proba(X_test)[:,1] >= thresholds[ix]).astype(bool) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.369114Z",
     "start_time": "2022-04-22T17:25:27.351747Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Our final model's accuracy on the test set is 0.79. \n",
      "\n",
      "Our final model's recall on the test set is 0.79 \n",
      "\n",
      "Our final model's precision on the test set is 0.76 \n",
      "\n",
      "Our final model's f1-score on the test is 0.78.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"\"\"\n",
    "Our final model's accuracy on the test set is {round(accuracy_score(y_test, y_hat), 2)}. \\n\n",
    "Our final model's recall on the test set is {round(recall_score(y_test, y_hat), 2)} \\n\n",
    "Our final model's precision on the test set is {round(precision_score(y_test, y_hat), 2)} \\n\n",
    "Our final model's f1-score on the test is {round(f1_score(y_test, y_hat), 2)}.\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.373780Z",
     "start_time": "2022-04-22T17:25:27.370075Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test, y_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.545282Z",
     "start_time": "2022-04-22T17:25:27.376198Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEGCAYAAACaSwWnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAngElEQVR4nO3dd5wV1f3/8dd7d6kRVFSQYmIJFmwoaogttlgSjRobJlFTicYa49f29WfQb0j8RpN8o4kajEY0imJJ7FHA3gUrYAFjQylGEUERBT6/P+YsXnH37t1lZ/cyvJ8+5nFnzpQzl3v97LmfOXNGEYGZmRVDTXufgJmZtR4HdTOzAnFQNzMrEAd1M7MCcVA3MyuQuvY+gcZ02eIYd8uxz3nroT+29ylYFVq1a62W9RjNiTnzn/rTMteXF7fUzcwKpGpb6mZmbUrFaOM6qJuZAdTUtvcZtAoHdTMzAFVtmrxZHNTNzMDpFzOzQnFL3cysQNxSNzMrELfUzcwKxL1fzMwKxOkXM7MCcfrFzKxA3FI3MysQB3UzswKp9YVSM7PicE7dzKxAnH4xMyuQgrTUi/GnycxsWamm8qncYaTOkh6X9IykSZLOSuXDJL0p6ek0faNkn9MkTZX0oqQ9SsoHSXourTtfavovj1vqZmbQmi31BcAuETFPUgfgQUl3pHV/iIjzPlutBgBDgI2BPsBYSetHxCLgImAo8ChwO7AncAdluKVuZgbZMAGVTmVEZl5a7JCmcs8/3Re4JiIWRMQrwFRgG0m9ge4R8UhEBHAFsF+Tb6Ppd2pmtgJoRvpF0lBJ40umoZ85lFQr6WlgFjAmIh5Lq46R9KykyyStmsr6Am+U7D4tlfVN80uXl+WgbmYGWfqlwikiRkTEViXTiNJDRcSiiBgI9CNrdW9ClkpZDxgITAd+V19zA2cTZcrLclA3M4NWu1BaKiLeA+4F9oyImSnYLwYuAbZJm00D1irZrR/wVirv10B5WQ7qZmbQmr1f1pC0SprvAuwGvJBy5PX2Byam+ZuBIZI6SVoH6A88HhHTgbmSBqdeL4cDNzX1Ntz7xcwMWnM89d7ASEm1ZA3n0RFxq6QrJQ0kS6G8CvwUICImSRoNTAYWAkenni8ARwGXA13Ier2U7fkCDupmZplW6tIYEc8CWzRQfliZfYYDwxsoHw9s0pz6HdTNzMDDBJiZFUpBhglwUDczAyq4A3+54KBuZoaDuplZoajGQd3MrDDcUjczKxAHdTOzAnFQNzMrkmLEdAd1MzNwS93MrFBqanxHqZlZYbilbmZWJMWI6Q7qZmbglrqZWaE4qJuZFYiHCTAzKxC31M3MCsRB3cysQBzUzcwKxEHdzKxIihHTHdTNzMDDBJiZFYrTL2ZmRVKMmO6g3h46daxj7KUn0LFjHXW1tfxj7FP86uLbufKcH9B/7V4ArNKtC+/Nnc/gIedQV1fDRWd+l4EbrkVdbQ1X3fY45112FwBbbLQWI846jC6dOnDnQ5P4xW+vb8+3Zq3o2quv5KYbryMi2PfbBzHku4cz5cUX+N/hZzF//oes2acvZw//LV9YaSUARl46gltuuoGamlpOPPl0Bm+7fTu/g+WLW+rWYgs+XsieQ8/ng/kfU1dXw92XnchdD03msFP/tmSbc07cnznz5gNwwG5b0qljHVsf/Gu6dO7AUzecweg7xvP69Hc5//RDOOZXo3js2Vf455+OYvftBnDXQ5Pb661ZK3l56hRuuvE6LrvyWuo6dOCEo4ey7fY78uuzz+TYn/8XW261Nbf88wb+PvIyfnr0cbzy8lTG3HkHV19/C/95exbHHvkjRv/zdmpra9v7rSw3ihLUi3FlYDn0wfyPAehQV0tdXS0R8Zn1B3x9S0b/awIAQdC1c0dqa2vo0qkjH3+yiLkffMSaq3en2xc689izrwBw9a2Ps89Om7XtG7FcvPrKy2y86eZ07tKFuro6thy0NffdM47XXnuFLQZtBcA2g7flnnHZL7b7772br++xFx07dqRP3370W+uLTJ74XHu+heWOpIqnapZLS13SBUA0tj4ijsuj3uVJTY14+OpTWG+tNfjLtffzxMTXlqzbbsv1mPnuXF5+/W0Abhz7FHvvtBmvjBlO184dOfm8G5n9/odsOeCLvDnrvSX7vTnzPfr0XKWN34nlYd31+nPxn/7InPfeo1OnTjz84P1sOGBj1luvPw/cezc77rwr48bcyayZMwB4++1ZbLzpp3/Qe/bsxduzZrbX6S+XijL2S14t9fHABKAzsCUwJU0DgUWN7SRpqKTxksYv/M+knE6tOixeHAwecg5f3uMMttrkSwxYr/eSdQfvuRXX/Wv8kuWtN16bRYsWs+7u/81G3/wlxx+2C2v3Xa3B6zpLt/ht+bTOuutx2Pd/zLFH/YgTjh5K//U3oK6ujv8e9iuuHz2KI75zIB9++AF1HToAjXzuVd6irDat1VKX1FnS45KekTRJ0lmpvIekMZKmpNdVS/Y5TdJUSS9K2qOkfJCk59K681XBz4RcWuoRMTKd0PeBnSPik7R8MXBXmf1GACMAumxxzAoRnebMm8/946ew+7YDmPzydGpra9h3l83Z7ju/XbLNwXttxV0PT2bhwsW8PXsejzz9bwYN+CIPPfUyfUta5n17rcL0t+e0w7uwPHxr/wP41v4HAHDRBX9gjV5rsvY663L+RX8F4PXXXuXhB+4Hspb5rBkzluw7a9ZM1lijZ9uf9HKsFdMqC4BdImKepA7Ag5LuAL4NjIuIcySdCpwKnCJpADAE2BjoA4yVtH5ELAIuAoYCjwK3A3sCd5SrPO+ceh+gW8nySqlshbb6qiux8kpdAOjcqQO7fGUDXnw1+6m8y1c24KVXZ34mrTJtxrvstPUGAHTt3JFtNlubF1+dyYz/vM+8DxewzaZrA/Cdvbfh1vuebdP3Yvl59913AJgx/S3uvXssu+/5jSVlixcv5m+XXMz+Bx4MwA477cyYO+/g448/5q03p/HG668xYJNN2+3cl0dS5VM5kZmXFjukKYB9gZGpfCSwX5rfF7gmIhZExCvAVGAbSb2B7hHxSGQ/xa4o2adRefd+OQd4StI9aflrwLCc66x6a67enUvOPozamhpqasQNY57kjgcmAnDQHoOWXCCtd/G19zPirO8x4fr/RoIrb3qUiVPeAuC4X1/LiLO+R5dOHbjrocnc+aB7vhTFaScdz5z33qOurgMnnXoG3buvzLVXX8n1114NwE67fJ299/02kOXgd919Dw49YB9qa2s56dQz3POlmZrTUpc0lKwFXW9EyjTUr68lS0F/GfhzRDwmqVdETAeIiOmS6n9K9SVrideblso+SfNLl5c/t7xzsJLWBL6SFh+LiBnltq+3oqRfrHneeuiP7X0KVoVW7Vq7zLmTDU65s+KY8+L/7lFRfZJWAf4BHAs8GBGrlKybHRGrSvoz8EhE/D2VX0qWankd+E1E7JbKdwBOjoh9ytWZa/olJfV3AzaPiJuAjpK2ybNOM7OWaK30S6mIeA+4lywXPjOlVEivs9Jm04C1SnbrB7yVyvs1UF5W3jn1C4GvAoem5bnAn3Ou08ys2WpqVPFUjqQ1UgsdSV3IGrYvADcDR6TNjgBuSvM3A0MkdZK0DtAfeDylauZKGpwayIeX7NOovHPqX4mILSU9BRARsyV1zLlOM7Nma8UeoL2BkSmvXgOMjohbJT0CjJb0I7LUykEAETFJ0mhgMrAQODr1fAE4Crgc6ELW66VszxfIP6h/kt5YQPYXDFicc51mZs3WWl0aI+JZYIsGyt8Bdm1kn+HA8AbKxwObNKf+vNMv55NdJOgpaTjwIPCbnOs0M2u2PHLq7SHXlnpEXCVpAtlfJwH7RcTzedZpZtYSfkhGBSRdGRGHkV0kWLrMzKxqVHsLvFJ559Q3Ll1I+fVBOddpZtZs1T76YqVy+b2RBqeZC2wm6X1Jc9PyLCrokmNm1taKklPPJahHxG8iohtwbkR0j4huaVotIk7Lo04zs2Xh8dQrEBGnpeEl+5MNw1tffn+e9ZqZNVeVx+qK5X2h9MfA8WS3tz4NDAYeAXbJs14zs+Zq6k7R5UXefXiOB7YGXouInck65L+dc51mZs3m9EtlPoqIj9I/RKeIeEHSBjnXaWbWbFUeqyuWd1Cflga2+ScwRtJsKhhlzMysrVV7C7xSeV8o3T/NDksPylgZ+FeedZqZtURBYnruLfX6G456Aa+kojXJRigzM6saRblQmnfvl2OBXwIz+XR0xgA2y7NeM7PmcvqlMscDG6QhJ83MqpaDemXeAObkXIeZ2TIrSEzPPaj/G7hX0m3AgvrCiPh9zvWamTWLW+qVeT1NHdNkZlaVChLTc+/SeFaexzczay3u/VKGpP+LiBMk3UJ6PmmpiPhWHvWambVUTUGa6nm11K9Mr+fldHwzs1ZVkJieT1CPiAlpdjwwPyIWw5IbkTrlUaeZ2bIoyoXSvEdpHAd0LVnuAozNuU4zs2arUeVTNcu790vniJhXvxAR8yR1LbeDmVl7KMqF0rxb6h9I2rJ+QdIgYH7OdZqZNZua8V81y7ulfgJwnaT64XZ7A4fkXKeZWbMVpKGeez/1JyRtCGwACHghIj7Js04zs5bwhdIKSDoa+EJETIyI54CVJP0szzrNzFpCqnyqZnnn1H8SEe/VL0TEbOAnOddpZtZsNVLFUzmS1pJ0j6TnJU2SdHwqHybpTUlPp+kbJfucJmmqpBcl7VFSPkjSc2nd+arg50TeOfUaSYqISCdYi8eAMbMq1Iq9XxYCv4iIJyV1AyZIGpPW/SEiPnNTpqQBwBBgY6APMFbS+hGxCLgIGAo8CtwO7AncUfZ9tNa7aMSdwGhJu0raBRiFH2dnZlWotdIvETE9Ip5M83OB54G+ZXbZF7gmIhZExCvAVGAbSb2B7hHxSGoYXwHs19T7yDuonwLcDRwFHE12M9LJOddpZtZszUm/SBoqaXzJNLShY0paG9gCeCwVHSPpWUmXSVo1lfUle/ZEvWmprG+aX7q8rLx7vywm+/lwUZ71mJktq+YkXyJiBDCi7PGklYAbgBMi4n1JFwH/QzbI4f8AvwN+2EjVUaa8rEaDuqQLyh0gIo5r6uCS+gO/AQYAnUv2Xbepfc3M2lJrdmmU1IEsoF8VETcCRMTMkvWXALemxWnAWiW79wPeSuX9Gigvq1xLfXwlJ9+Ev5E9ePoPwM7AD2jeH0QzszbRWtdJUw+VS4HnS5/yJql3RExPi/sDE9P8zcDVkn5PdqG0P/B4RCySNFfSYLL0zeHABU3V32hQj4iRLXlDS+kSEeNSD5jXgGGSHiAL9GZmVaMVe79sBxwGPCfp6VR2OnCopIFkGZBXgZ8CRMQkSaOByWQ9Z45OPV8gux55OdlgiHfQRM8XqCCnLmkNsgueS6dQdmlqX+AjSTXAFEnHAG8CPSvYz8ysTbVW+iUiHqThjMTtZfYZDgxvoHw8sElz6q+k98tVZF1y1gHOIvsL80SFxz+BbOjd44BBwPeAI5pzgmZmbWFFGnp3tYi4VNLxEXEfcJ+k+yo8/sI09O48sny6mVlVKsrYL5UE9foBuKZL+ibZ1dd+ZbYv9fvUgf46ss71k1pwjmZmuStGSK8sqP9K0srAL8iuvHYHfl7JwSNiZ0lrAgcDIyR1B66NiF+19ITNzPJQW+15lQo1GdQjor4v5RyybonNEhEzgPMl3UN2N+mZgIO6mVWVFSb9IulvNHATUkT8sIJ9NyJ7KMaBwDvANWQtfjOzqlKQmF5R+uXWkvnOZJ3mm7yrKfkb2SBeu0dEpfuYmbW5pobUXV5Ukn65oXRZ0ihgbCUHj4jBLTwvM7M2VZCY3qIBvfoDX6xkQ0nbAcOAL6W6BEQlY7/MfuJPLTg1K7q1j7q+vU/BqtCMSw5c5mOsSDn1uXw2pz6D7A7TSlxK1lNmArCoiW3NzNpN7YoS1COi2zIcf05ENDlWgZlZeytIj8amhwmQNK6SskbcI+lcSV+VtGX91OyzNDPLWeGHCZDUmWzcltXTEzrq30p3suEhK/GV9Dqo/rBkqZxKBgMzM2szK0JO/adkA3L1IcuJ17/j94E/lzuopBPTbH13yADeBh5Mz+AzM6sq1d4Cr1Sj6ZeI+GNErAOcFBHrRsQ6ado8IprqmtItTSulqRuwFXCHpCGtdfJmZq2ltR483d4q6dK4WNIqEfEeQErFHBoRFza2Q0Sc1VC5pB5kfdyvacG5mpnlpq7ao3WFKhlP/Sf1AR0gImYDP2lJZRHxLsUZDM3MCmRFaqnXpMfRBYCkWqBjSyqTtAswuyX7mpnlaYUZJgC4Exgt6WKyC55H0sRz8iQ9x+cHAetBNmbM4S04TzOzXBUkplcU1E8BhpI9AFXAU0DvJvbZe6nlAN6JiA+afYZmZm2gKL1fKrmjdLGkR4F1yYbR7QHc0MQ+r7XO6ZmZtY3CPyRD0vrAEOBQsrHQr4XsaUZtc2pmZm2nIDG9bEv9BeABYJ+ImAogqaLH2JmZLW9UkI555bo0HkA2IuM9ki6RtCvujmhmBVWUsV/K3VH6j4g4BNgQuJdsCN1eki6StHsbnZ+ZWZsofFCvFxEfRMRVEbE30A94Gjg17xMzM2tLkiqeqlmznnyU7gj9S5rMzAqjtpL765cDLXmcnZlZ4RTljtKC/G0yM1s2rZVTl7SWpHskPS9pkqTjU3kPSWMkTUmvq5bsc5qkqZJelLRHSfkgSc+ldeergtyPg7qZGa06oNdC4BcRsREwGDha0gCya5HjIqI/MC4tk9YNATYG9gQuTGNsAVxEdkd//zTt2VTlDupmZkANqngqJyKmR8STaX4u8DzQF9gXGJk2Gwnsl+b3Ba6JiAXpIUJTgW0k9Qa6R8QjaUDFK0r2aZRz6mZm5DOgl6S1gS2Ax4BeETEdssAvqWfarC/waMlu01LZJ2l+6fKyHNTNzIC6ZnRAlzSULC1Sb0REjFhqm5XIxsk6ISLeL5MOb2hFlCkvy0HdzIzmtdRTAB/R2HpJHcgC+lURcWMqnimpd2ql9wZmpfJpwFolu/cjG6Z8Wppfurws59TNzMi6NFY6lZN6qFwKPB8Rvy9ZdTNwRJo/ArippHyIpE6S1iG7IPp4StXMlTQ4HfPwkn0a5Za6mRmtmlPfDjgMeE7S06nsdOAcsgcO/Qh4HTgIICImSRoNTCbrOXN0RCxK+x0FXA50IXs4UdkHFIGDupkZ0Hppi4h4kMYHP9y1kX2GA8MbKB8PbNKc+h3Uzcwozh2lDupmZjiom5kVSjFCuoO6mRmQz81H7cFB3cwMqn6c9Eo5qJuZUZybdhzUzczwhVIzs0Jx+sXMrECcfjEzKxC31M3MCqQYId1B3cwMgFq31M3MiqMgMd1B3cwMQAVJwDiom5nhlrqZWaHUuKVuZlYcbqmbmRWIhwkwMyuQmmLEdAd1MzNw7xczs0IpSPbFQb0aXDnycm684Tok0b//+pw9/Ddc9tcR3HD9aHqs2gOAY084kR12/BrvvTebX5xwHJMmTuRb++3P6Wec2c5nb62lU10N/zx5JzrW1VBXK26d8Cbn3jyZfQb15aRvDaD/mt3Z69d388xrswHoUCvOPWwQm39pVRZH8P+ueYaHX3obgKuP355eK3emrlY8OuU/nHbVUyyO9nx31c8tdWsVM2fO5OqrruAfN99O586d+a8Tj+dft98GwGGHf58jfvCjz2zfsWMnjj72eKZOncLUKVPa45QtJwsWLuaA393HhwsWUVcrbj55Z8ZNnMELb77PDy98hHMPG/SZ7b+3w7oA7HzWGFbv1omrjt+ePYePIwKG/uVR5n20EIC/HjmYfbbqx01PTGvz97Q8KUpOvSijTS7XFi1axIKPPmLhwoXM/+gj1ujZs9Ftu3btypaDtqJTx05teIbWVj5csAiADrVZaz0CpsyYy8sz531u2/X7dOOB52cB8J+5C3j/w08Y+KVVAZYE9Lpa0bGuBtxKb1KNVPFUzVo9qEvqUW5q7fqWd7169eKI7/+QPXbbmd122p5uK63EttttD8A1V1/Fgfvvw5lnnMb7c+a085laW6gRjD1zNyb+bh/uf34WT73ybqPbTnpjDnsO7ENtjfji6l3Z7Eur0KdH1yXrR52wPRN/tw/zPlrILRPcSm+KmjFVszxa6hOA8en1beAlYEqan1BuR0lDJY2XNP7SS0bkcGrV5/05c7jn7nHcftc4xtzzAPPnz+fWW27i4EMO5dZ/jWH0DTexxho9Oe/cc9r7VK0NLA7Y7eyxbHHybWyx9qps2Kd7o9uOeuhV3po9nzvP2JWzDxnI+JffYeGixUvWH/p/D7L5SbfSsa6G7Tds/NefZYrSUm/1nHpErAMg6WLg5oi4PS3vBezWxL4jgBEAHy1cMX4wPvrow/Tt148ePbIfMbvutjvPPPUUe++z75Jtvn3gQRz7syPb6xStHbw//xMefultdt5kTV546/0Gt1m0OPjl6GeWLN9yys68MuuzaZoFCxdz1zPT2XNgH+5PqRprWHWH6srlmVPfuj6gA0TEHcDXcqxvubRm7z48+8wzzJ8/n4jgsUcfYZ311uPttz/9H/DusWP5cv/+7XiW1hZWW6kj3bt0AKBzhxp22KgXU2fMbXT7Lh1r6dqxFoAdN+rJwsWLeWn6XLp2qqXnyp0BqK0Ru266ZtnjWFKQ/EuevV/+I+kM4O9kl2m+B7yTY33Lpc0225yv774HQw7an9raOjbcaCMOPOgQhp3537z4wgtI0KdPX/7fsLOX7LPX13dh3rx5fPLJJ9xz91guHnEZ6335y+34Lqw19Fy5C+f/cCtqa7Kf+DePn8aYZ6ez1xZ9GH7oQFZbqRN/P247Jr7xHof+34Os3q0To07YgcURzJg9n2MvfQKArh3ruOKYbelYV0NtjXjwhbcZed+/2/ndVb/WTKtIugzYG5gVEZuksmHAT8hS0QCnl2QyTgN+BCwCjouIO1P5IOByoAtwO3B8RJTNYqiJ9S2WLor+EtiRLKjfD5wdEY1f+SmxoqRfrHnWPur69j4Fq0IzLjlwmSPyE/+eU3HM2XrdlcvWJ2lHYB5wxVJBfV5EnLfUtgOAUcA2QB9gLLB+RCyS9DhwPPAoWVA/P2U9GpVbSz0F7+MlrRQRn++PZWZWTVoxrRIR90tau8LN9wWuiYgFwCuSpgLbSHoV6B4RjwBIugLYDygb1HPLqUvaVtJkYHJa3lzShXnVZ2a2LNSM/5bBMZKelXSZpFVTWV/gjZJtpqWyvml+6fKy8rxQ+gdgD1IePSKeIUvFmJlVHak506fdr9M0tIIqLgLWAwYC04Hf1VfdwLZRprysXIcJiIg39NmLD4vyrM/MrKWa0/4u7X7djH1mLqlLugS4NS1OA9Yq2bQf8FYq79dAeVl5ttTfkLQtEJI6SjoJeD7H+szMWkxSxVMLj9+7ZHF/YGKavxkYIqmTpHWA/sDjETEdmCtpsLJKDwduaqqePFvqRwJ/5NO80F3Az3Ksz8ysxVrzRlFJo4CdgNUlTSPrCbiTpIFkKZRXgZ8CRMQkSaPJrj8uBI6OiPqsxlF82qXxDpq4SAr5BvUNIuK7pQWStgMeyrFOM7MWac17iiLi0AaKLy2z/XBgeAPl44FNmlN3numXCyosMzNrf76jtGGSvgpsC6wh6cSSVd2B2tauz8ysNfghGY3rCKyUjt2tpPx94MAc6jMzW2ZVPvhixfIYpfE+4D5Jl0fEa619fDOzPDioN+1DSecCGwOd6wsjYpcc6zQza5GipF/yvFB6FfACsA5wFlkXnidyrM/MrMWac0dpNcszqK8WEZcCn0TEfRHxQ2BwjvWZmbVYQTq/5Jp++SS9Tpf0TbLbW/uV2d7MrP1Ue7SuUJ5B/VeSVgZ+QdY/vTvw8xzrMzNrsWp/9mil8hxPvX6wmjnAznnVY2bWGooR0nMM6pLWIHt009ql9aTcuplZdSlIVM8z/XIT8ADZo5k85K6ZVbWidGnMM6h3jYhTcjy+mVmrKUhKPdcujbdK+kaOxzczazXu0ti044HTJS0g694oICKie451mpm1SEsfflFt8uz90q3prczMqkNBYnouQ+9uGBEvSNqyofUR8WRr12lmtqwKEtNzaamfCAzl0ydllwrAA3qZWfUpSFTPY+jdoenVNxyZ2XKjKF0ac+v9IuloSauULK8qyQ+eNrOq5FEam/aTiHivfiEiZpPdYWpmVnVqVPlUzfLs0lgjSRERAJJqyR51Z2ZWhao8Wlcoz6B+JzBa0sVkF0iPBP6VY31mZi1W7WmVSuUZ1E8h6wVzFNmfwLuAv+ZYn5lZixUkpuca1LsAl0TExbAk/dIJ+DDHOs3MWqQoLfU8L5SOIwvs9bqQjdhoZlZ1JFU8VbM8W+qdI2Je/UJEzJPUNcf6zMxarLpDdeXybKl/UDpUgKRBwPwc6zMza7Gi9FPPs6V+AnCdpLfScm/gkBzrMzNrMd9R2oSIeALYkKz3y8+AjSJiQl71mZktk1YcUF3SZZJmSZpYUtZD0hhJU9LrqiXrTpM0VdKLkvYoKR8k6bm07nxVkNDPM/0CsAEwANgCOFTS4TnXZ2bWIq38kIzLgT2XKjsVGBcR/ck6kpwKIGkAMATYOO1zYeotCHARWdfw/mla+pifk+fYL78ELkjTzsBvgW/lVZ+Z2bKokSqemhIR9wPvLlW8LzAyzY8E9ispvyYiFkTEK8BUYBtJvYHuEfFIujP/ipJ9Gn8fFbzXljoQ2BWYERE/ADYn66duZlZ1mnOhVNJQSeNLpqEVVNErIqYDpNeeqbwv8EbJdtNSWd80v3R5WXleKP0oIhZLWiipOzALWDfH+szM2kREjABGtNLhGmr6R5nysvJ48tGfgFHA42no3UuACcA84PHWrs/MrDW0QVfFmZJ6R8T0lFqZlcqnAWuVbNcPeCuV92ugvKw80i9TgPOAvYHTgEeBrwNHpDSMmVnVUTP+a6GbgSPS/BHATSXlQyR1krQO2QXRx1OKZq6kwanXy+El+zQqjycf/RH4o6QvkV3R/RvQGRglaX5ETGntOs3MllVrttQljQJ2AlaXNA34JXAO2ci1PwJeBw4CiIhJkkYDk4GFwNERsSgd6iiynjRdgDvSVL7uNNx5riRtAVwGbBYRtU1tD/DRwqZzR7biWfuo69v7FKwKzbjkwGUOyXMXLK445nTrVL2PysizS2MHSftIuorsr8tLwAF51WdmtizaIP3SJvK4UPp14FDgm2QXRq8BhkbEB61dl5lZa6n2MV0qlUeXxtOBq4GTImLpzvdmZlWpIDE9lwulO7f2Mc3McleQqJ7nzUdmZsuNSm7/Xx60Se8XWzaShqY72MyW8PfCGpL3KI3WOioZV8JWPP5e2Oc4qJuZFYiDuplZgTioLx+cN7WG+Hthn+MLpWZmBeKWuplZgTiom5kViIN6EyStXfpE8FQ2TNJJ7XQ+W0k6fxn2P70F+3w/PfzEmiDp3tKnwaeyEyRd2ArH/mt6SHFL9t1J0rYt2O9VSau3pE5rHw7qy5mIGB8Rxy3DIZod1K1ZRpE9R6DUkFS+TCLixxExuYW77wQ0O6jb8sdBfRmkVtn/Snpc0kuSdkjlj0naeKntBknaRtLDkp5Krxuk9bWSzpP0nKRnJR2byrdO2z2T6uiWWly3pvXDJF2Wjv9vSceV1PlPSRMkTap/KK6kc4Aukp5OQyIj6Xvp2E9L+ouk2lT+g/Se7gO2a5t/0UK4HthbUifIfukBfYDvpAcUT5J0Vv3GjXzGjX0f7pW0VZqfJ2l42u9RSb1S+T7p+/eUpLGSeqVzOBL4efqcd5C0hqQbJD2Rpu3S/qtJuivt/xcKMyLKCiQiPJWZgLWBiUuVDQNOAu4FfpfKvgGMTfM/B85K872Bl9J8d6Auze8G3JDmjwJuKFnXA+gI/BvYunRfshbXrSXn8TDQCVgdeAfoUH+M9NoFmAislpbnlbyPjYBbSva5kOyRWb3JnsyyRjqPh4A/tfdnsbxMwG3Avmn+VODcks+jNn1vNivzGX/u+5Be7wW2SvMB7JPmfwuckeZX5dNebT8u+X4OIxs5tf4crwa2T/NfBJ5P8+cDZ6b5b6Z6Vm/vf1NPlU8e0KtpjfX5rC+/Mb1OIPsDADAaGEP2CKuDgetS+crASEn90/4dUvluwMURsRAgIt6VtCkwPSKeSGXvA+jzgw7dFhELgAWSZgG9yB5Ye5yk/dM2a5E99/CdpfbdFRgEPJGO24XsYbhfAe6NiLdTndcC6zfy72CfV5+CuSm9/hA4OP1iqiP7ozmA7DvQ0Gf8ue9DA3V8DNya5ieQPQcYsocTX6vswcYdgVcaOcfdgAEl36fukroBOwLfTvXeJml2s9+9tSunX5r2Dlnrp1QP4D9pfkF6XUQa9TIi3gTekbQZcAjZg0IA/ge4JyI2AfYhe3YrZD9xl/7j0VBZQxaUzC8C6iTtRPY/7VcjYnPgqZK6lq5jZEQMTNMGETEsrfMNDC33T2BXSVuS/aGcTfbLbteI2IysJd+Zxj/jSj77TyKifpsl3z3gArJfVZsCP6Xhzx2y//e/WvLZ942IuWmdP/vlmIN6EyJiHjBd0q4AknoAewIPNrHrNcDJwMoR8VwqWxl4M81/v2Tbu4AjJdWV1PEC0EfS1qmsW/36CqwMzI6IDyVtCAwuWfeJpPpfCOOAAyX1rK9X2QPDHwN2SvnVDqQH5Fpl0nfmXrLn8o4iS6t8AMxJue+90qaNfcYNfR8qVfodO6KkfC7QrWT5LuCY+gVJA9Ps/cB3U9lefL5BY1XOQb0yhwNnSHoauJssX/5yE/tcT/bTe3RJ2W+B30h6iCy3Wu+vZDnsZyU9A3wnIj4ma+VfkMrG0Hira2n/ImuxP0v26+DRknUjUj1XRdaT4gzgrrTtGKB3REwny8E+AowFnqywXvvUKGBz4JqIeIbs19IkskD/EECZz/hz34dm1DsMuE7SA3z6axKyayf7118oBY4DtkoXYieTXUgFOAvYUdKTwO7pPGw54mECzMwKxC11M7MCcVA3MysQB3UzswJxUDczKxAHdTOzAnFQt1xIWpS6z02UdJ2krstwrMslHZjmy45UKI9GaCs4B3XLy/x0p+ImZLe0H1m6sn7gsOaKpkcq3AmPRmgrMAd1awsPAF9Oreh7JF0NPJdGIzw3jRL4rKSfAijzJ0mTJd0G9Kw/0FIjFe4p6ck0UuE4j0Zohgf0snylW933IrvLFWAbYJOIeCUNcDUnIrZWNlTtQ5LuArYANgA2JRugbDLZnZilx10DuATYMR2rRxoI7WKykSjPS9tdDfwhIh6U9EXgTrLRKX8JPBgRZ0v6JjA0138IszbioG556ZKGVYCspX4pWVrk8YioHzlwd2Cz+nw52bgl/clGChwVEYuAtyTd3cDxBwP31x+rkZEMwaMR2grGQd3yMj8iBpYWpMD6QWkRcGxE3LnUdt+g6ZECKx3Fsn40wvkNnIvHyLDCcU7d2tOdwFH1o0ZKWl/SF8hGChyScu69gZ0b2PcR4GuS1kn71o9k6NEIbYXmoG7t6a9k+fInlT3c+y9kvx7/AUwBngMuAu5besf0AI+hwI1pJMNr0yqPRmgrNI/SaGZWIG6pm5kViIO6mVmBOKibmRWIg7qZWYE4qJuZFYiDuplZgTiom5kVyP8H+wL5JrvTUboAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(cm, annot=True, cmap='Blues', fmt='d',\n",
    "            xticklabels= ['Unvaccinated', 'Vaccinated'],\n",
    "            yticklabels= ['Unvaccinated', 'Vaccinated'])\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "plt.savefig('./images/Confusion2.jpeg', bbox_inches = 'tight');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By choosing a specific threshold we were able to reduce the number of false negatives at the expense of false positives. This isn't really useful in our case, but it may be useful if we receive new data. In addition this may be helpful if we want to concentrate specifically on identifying Vaccinated individuals over UnVaccinated individuals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.548727Z",
     "start_time": "2022-04-22T17:25:27.546406Z"
    }
   },
   "outputs": [],
   "source": [
    "TN = cm[0][0]\n",
    "FN = cm[1][0]\n",
    "TP = cm[1][1]\n",
    "FP = cm[0][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.552815Z",
     "start_time": "2022-04-22T17:25:27.550148Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "76.33277551996174"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100*(TP/ (TP+FP))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.556748Z",
     "start_time": "2022-04-22T17:25:27.553791Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "81.62383934355431"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "100*(TN/ (TN+FN))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:27.843648Z",
     "start_time": "2022-04-22T17:25:27.559034Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEICAYAAAAKrmAuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd30lEQVR4nO3dd5hV1b3G8e/LDAOINEGUIiVqYkFEQBSNJZbYGwKKEDUmlmsSo8aCJgYxMbFjSYwoV0NURAGvNUaxxYY4oGJAATGKoqiIDaQPv/vH2YPDOEOd4Sxm3s/zzMPZa7ffOsJ5XXuvOVsRgZmZWWrq5LsAMzOzijigzMwsSQ4oMzNLkgPKzMyS5IAyM7MkOaDMzCxJDiizaiLp75L+mL3eS9K0dTzOLZIuqdrqzNLngLJaTdJ7khZKmi/pE0l3SNq0qs8TEc9HxA/WoJ6TJb1Qbt8zIuIPVV1TuXOGpL7VdQ6zdeGAMoMjImJToCuwK/C78htIKtzgVW04JwGfZ39uMDX8PbUq4IAyy0TEh8BjQCeAbFTxC0lvA29nbYdLel3Sl5JektS5dH9Ju0h6VdI8SfcC9cus21fSrDLLW0m6X9IcSXMl/UXS9sAtQM9sRPdltu2KS4XZ8qmSZkj6XNJDklqXWReSzpD0tqQvJP1Vkirrs6T2wD7AacBBkrYos65A0sWS3sn6NFHSVtm6HSWNzWr4RNLFldRavt/vSbpQ0hvAN5IKJQ0sc443JR1TrsZTJb1VZn1XSedLGlNuu5skXV9ZX23j44Ayy2QfvocCr5VpPhrYDdhBUlfgduB0oDkwFHhIUj1JRcADwJ3AZsAo4NhKzlMAPALMBDoAbYCREfEWcAYwLiI2jYimFey7H/BnoC/QKjvGyHKbHU5uJLhztt1Bq+j2icCEiBgDvAX0L7PuXKBf9p40Bk4BFkhqBDwJ/AtoDWwDPLWKc5TXDzgMaBoRy4B3gL2AJsBg4C5JrbL+9gEuzepsDBwJzAXuAg6W1DTbrhA4jtz7bzWEA8oMHshGKy8A/wb+VGbdnyPi84hYCJwKDI2I8RFREhHDgcXA7tlPXeD6iFgaEaOB4krO14PcB/v5EfFNRCyKiBcq2ba8/sDtEfFqRCwGLiI34upQZpsrIuLLiHgfeAbosorjnQiMyF6PYOXLfD8HfhcR0yJnUkTMJReAH0fEtVnt8yJi/BrWD3BjRHyQvadExKiI+CgilkfEveRGqz3K1HBVRBRnNcyIiJkRMRt4DuiTbXcw8FlETFyLOixxDigzODoimkZE+4g4s/SDM/NBmdftgd9kl/e+zEJtK3Jh0xr4MFb+9uWZlZxvK2BmNnpYW63LHjci5pMbUbQps83HZV4vACqc9CFpT6Aj347ARgA7SepSps53Kqm/ovY1VfY9RdKJZS6bfknuEmuLNTjXcGBA9noAHj3VOA4os1UrGzgfAJdnYVb6s0lE3APMBtqUu9/TrpJjfgC0q2SSwOoeL/ARuaAEQFJDcpcbP1xdRypwEiDgdUkfA6WjoBPL1Ll1BftV1g7wDbBJmeUtK9hmRR+ze2C3Ab8EmmeXNSdnda3uXA8AnSV1Ijequ7uS7Wwj5YAyW3O3AWdI2k05DSUdlt2TGQcsA87Kbvz34tvLVOW9Qi7QrsiOUT8bzQB8ArTN7mlVZATwU0ldJNUjdzlyfES8tzYdkVSf3P2p08hdAiz9+RXQPwvPYcAfJG2b9bezpObk7p9tKens7P5bI0m7ZYd+HThU0maStgTOXk0pDckF1pysrp+STVLJDAPOk9Qtq2GbLNSIiEXA6Ow9eSW7pGk1iAPKbA1FxARy96H+AnwBzABOztYtAXply1+Qu2F/fyXHKQGOIDe54H1gVrY9wNPAFOBjSZ9VsO9TwCXAGHIhtzVw/Dp052hgIfCPiPi49Af4X6CA3D2d64D7gCeAr7N1DSJiHnBg1oePyd0z+lF23DuBScB72X73rqqIiHgTuJZcwH8C7AS8WGb9KOByciE0j9yoabMyhxie7ePLezWQ/MBCM9tYSWoHTAW2jIiv812PVS2PoMxsoySpDrmp8CMdTjWTf5PbzDY62eSQT8jNaDw4z+VYNfElPjMzS5Iv8ZmZWZJ8ia8KtWjRIjp06JDvMszMNioTJ078LCI2L9/ugKpCHTp0YMKECfkuw8xsoyKpwm9d8SU+MzNLkgPKzMyS5IAyM7Mk+R5UNVu6dCmzZs1i0aJF+S5lo1O/fn3atm1L3bp1812KmeWBA6qazZo1i0aNGtGhQwdW8WBTKycimDt3LrNmzaJjx475LsfM8sCX+KrZokWLaN68ucNpLUmiefPmHnma1WIOqA3A4bRu/L6Z1W4OKDMzS5LvQVWhN974gNatz1mpbfjw3ixZ8kEle1Ssa9cObLPNdpSULKNjx234wx+G0KBBg3Wq6ZJLzmXvvffnwAMPY/DgCxgw4OdsvfX3K9y2uHgcdevWpUuX7gCMGnUn9es34Igjeq/TuavCp59+zv77n7P6Dc0Sc9ajV+W7hA1m4C7VM5HJI6gE1atXn/vu+xdjxjxJ3bpFjB5910rrS0pK1um4gwZdVWk4AUyYMI5JkyauWO7T5yd5DSczq908gkpc1667Mn36VIqLxzF06PW0aNGS6dOnMGrUWG644QomTBjH0qVLOO64E+ndewARwRVX/J7i4hdp3bodZb+t/mc/68u55/6WHXfcmRdffJabbrqK5ctLaNq0GYMGXc3o0XdTp04dHn30/xg4cDDjx7/IJps05KSTTmfq1ClcfvnFLFq0kLZt2zN48NU0btyUn/2sLzvt1IXi4nHMm/c1l156FV277saMGdMYNOg8li5dSsRyrrlmKO3bezaema05B1TCli1bxosvPssee+wLwOTJrzNmzFjatGnH6NF306hRI0aMeIQlSxZz8sm96Nlzb6ZOncJ7773DqFFjmTt3Dr16HcDRR/dd6biffz6Xyy67kNtvH0WbNu346qsvadKkKb17918RSADjx6948jaXXHIOF154Gd27787NN1/LLbdczwUXXArkRnR33/0wzz//NEOHXs/QofcwevTdnHDCKRx22DEsXbqEkpLlG+Q9M7OawwGVoMWLF9G3b+4ZbF279uCYY47j9dcn0qlTF9q0aQfAyy8/x/TpUxk79p8AzJ8/j5kz32PixPEccshRFBQU0LLllvToscd3jv+f/7xK1667rThWkyZNV1nPvHlfM2/e13TvvjsARxxxLOeff+aK9fvtl6t1hx124qOPZgHQuXNXhg27iU8/nc1++x3i0ZOZrTUHVIJK70GVV3aiRAQMHHgZe+yxz0rbvPDC08Cqp2dHRJVO4S4qKgKgTp0Cli3L3R879NCj2WmnLjz//NOceeYABg26ih499qyyc5pZzedJEhupnj335r777mTp0qUAzJz5XxYuXEC3brvx+OMPUVJSwpw5n1Bc/NJ39u3cuRsTJ77Mhx++D8BXX30JQMOGm/LNN/O/s32jRo1p3LgJr746HoBHHrmfbt12W2V9s2bNpG3b9pxwwinss8+BTJ/+1vp018xqIY+gNlK9evXjo49m0a/foUQEzZo1Z8iQ29hvv4N55ZWX6NPnQNq1+x7duu3+nX0326w5l1xyBeeeezoRy2nWrDlDh45gn30O4LzzzuDZZ8cycODglfa57LLrVkySaNOmHZddds0q63v88Yd59NH/o7CwLi1abM7pp59dld03s1pAZWd52fopKtoiWrQ4YaW24cN707JluzxVtPH79NP3Oemk0fkuw2yt+feg1pykiRHRvXy7L/GZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliT/HtQG9tjyLav0eIfU+Xi12/TsuR3jxk2t0vNWpbvuGsaxx/Zf50eKmFnN5BGUrZdly5atcnlN3H337SxatLCqSjKzGsIjqFqkuHgct9wyhGbNmjFjxnS2334n/vSnG5DE5MmTuOqqS1m0aAF16xZx6633UFhYyOWX/5Y333yDgoJCzjvvEnbddQ8efHAUzz//FEuWLGbhwoUcfvixKy3feOPtXHHF73n77amUlJRwxhnn8KMf/ZiSkhJuuOHPvPTSv5FEr179iAjmzPmEU089jqZNN2PYsHvz/TaZWSIcULXMtGlTGDPmSTbffAtOPrkXr79eTKdOXbjwwjO58sqb6dRpZ+bPn0e9evUZMeJ2AEaPHsu7787gf/5nAA8++CwAb7zxKqNGPUGTJk158MFRKy3feOOV9OixB4MHX8PXX3/FgAFHsvvuP+Thh8fw4YcfMHLkYxQWFq54zMeddw7jttvupVmzzfL4zphZahxQtcyOO+7MFlu0AuAHP9iBjz6axaabNqZFi5Z06rQzAJtu2giA114rpl+/kwHo2HEbWrVqw8yZ7wKw++57rfSYjrLLL7/8HP/+91iGD78VgCVLFjN79oeMH/8CvXv3p7Aw99dudY/5MLPazQFVy5Q+GgO+fTxGZY/fWNX3NDZosEmlyxHBtdcOpUOHrb9zvKp8zIeZ1WyeJGF07Lg1c+Z8wuTJkwD45pv5LFu2jG7dduOf/3wAyD3O4+OPP6JDh++t9ng9e+7DPff8fUXATZ06OWvfi9Gj71oxkeLbx3w0ZMGC7z7mw8xqN4+gNrA1mRa+odWtW8SVV97MlVf+nsWLF1GvXn2GDh1B374/4Y9/vJjevQ+koKCQyy67lqKieqs93mmn/Zqrr76UPn1+TETQunVbbrrp7xxzTD9mznyXvn1/TGFhXXr16sfxx5/MsceewC9+cRItWrT0JAkzW8GP26hCftxG1fPjNmxj5cdtrDk/bsPMzDYqDigzM0uSA8rMzJLkgDIzsyQ5oMzMLEkOKDMzS5J/D2oDO+SQ66r0eI89du5qt/nss0+5+urBTJkyiaKiIlq12ooLLhhE+/ar/6Xb9fXgg6Po2XMvWras2seMmFnN54Cq4SKCc889jSOOOJYrr/wrAFOnTmHu3DmrDaiSkhIKCgoqXV4TDz00im22+b4DyszWmgOqhisufonCwkL69PnJirbtttuRiOC66y7nxRefQRKnnvorDjroSIqLxzF06PW0aNGS6dOncNFFl6+0PGrUWG644QomTBjH0qVLOO64E+ndewAAd9zxNx599H7q1KnDnnvuyw47dObNN9/g4ot/Tb169fnHPx6gfv36+XorzGwj44Cq4WbMmMb22+/0nfannnqMadOmcN99j/Pll5/Tv/8RdO26GwCTJ7/OmDFjadOmHcXF41ZaHj36bho1asSIEY+wZMliTj65Fz177s27777DM888wZ13PkSDBg1WPEpj5MjhnHvub9lxx503dNfNbCPngKqlXnutmEMOOYqCggKaN9+cbt12Y8qUSTRs2IhOnbrQps23X89Udvnll59j+vSpjB37TwDmz5/HzJnvMX78Cxx1VJ8Vj233ozTMbH05oGq4rbf+Pk8++c/vtK/6URoNKl2OgIEDL2OPPfZZaZuXXnrWj9IwsyrlaeY1XI8ee7JkyRLGjBmxom3y5Ek0btyExx9/mJKSEj7/fC6vvvoKnTp1We3xevbcm/vuu5OlS5cCucdwLFy4gJ499+aBB+5j4cKFQPlHaXxT5f0ys5rPI6gNbE2mhVclSVx33a1cffVg7rjjZoqK6tG69Vacf/4gFixYQN++ByGJs8++iBYtWvLuu++s8ni9evXjo49m0a/foUQEzZo1Z8iQ29hzz32ZNm0K/fsfRmFhET/84Y8466wLOfLIPvzxjxd7koSZrTU/bqMK+XEbVc+P27CNlR+3seb8uA0zM9uoOKDMzCxJqw0oSR0kTS7Xdqmk86qvrFXW013Sjeux/8XrsM/Jkv6yLudbvnzVM+aschHB8uX5rsLM8mWjG0FFxISIOGs9DrHWAbU+Zs78gmXLFjik1lJEsGzZAmbO/CLfpZhZnqzXLD5JzwLjgR8BTYGfRcTzksYDp0TElDLb/QYoAK4HGgALgZ9GxDRJBcCVwEFAALdFxE2SdgVuABoCi4H9gW7AeRFxuKRLgXbA97I/r4+IG7NzPgBsBdQHboiIWyVdATSQ9DowJSL6SxoAnAUUZX05MyJKJP0UuAiYDUzPzr/Whgx5gXPOgfbtm1Fno/vfgfxZvjwX7kOGvJDvUswsT6pimnlhRPSQdCgwCDgAGAn0BQZJagW0joiJkhoDe0fEMkkHAH8CjgVOAzoCu2TrNpNUBNwLHBcRxdm+Cys4/3bkArIRME3S3yJiKbmA/FxSA6BY0piIGCjplxHRBUDS9sBxwJ4RsVTSzUB/SWOBweTC8CvgGeC1ijov6bSsfgoKGn1n/VdfLebSS59au3fUzMzWKKAquzZV2n5/9udEoEP2+j5gLLnA6guMytqbAMMlbZvtXzo38QDglohYBpAFy07A7Igoztq+Bir6toJHI2IxsFjSp8AWwCzgLEnHZNtsBWwLzC23b+mIrDg7bgPgU2A34NmImJOd817g+xW+CRG3ArdCbpp5RduYmdnaW5OAmgs0K9e2GfBu9rr00ldJ6fEi4kNJcyV1JjdCOT3b5g/AMxFxjKQOwLNZu/huEFbUVpGyl95KgEJJ+5ILvZ4RsSC7xFjRb4gKGB4RF63UKB29huc2M7Nqstq7IhExH5gtaX8ASZsBBwOruzkwErgAaBIR/8namgAfZq9PLrPtE8AZkgrLnGMq0Dq7D4WkRqXr10AT4IssnLYDdi+zbqmk0pHbU0BvSS1LzyupPbl7UftKap5t22cNz2tmZlVkTW/bnwj8Lptc8DQwOCJW/Z04MBo4ntzlvlJXAX+W9CK5CROlhgHvA29ImgScEBFLyI2+bsraxlLxKKgi/yI3knqD3Kjt5TLrbs3Oc3dEvAn8Dngi23Ys0CoiZgOXAuOAJ4FX1/C8ZmZWRfxVR1Wooq86MrPayV91tOb8VUdmZrZRcUCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSCvNdQE3SufNWTJgwJN9lmJnVCB5BmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIUEfmuocYoKtoiWrQ4Id9lmNl6OuvRq/JdwkZl4C5112t/SRMjonv5do+gzMwsSQ4oMzNLkgPKzMyS5IAyM7MkOaDMzCxJDigzM0uSA8rMzJLkgDIzsyQ5oMzMLEkOKDMzS5IDyszMkuSAMjOzJDmgzMwsSQ4oMzNLkgPKzMyS5IAyM7MkOaDMzCxJDigzM0uSA8rMzJLkgDIzsyQ5oMzMLEkOKDMzS5IDyszMkuSAMjOzJDmgzMwsSQ4oMzNLkgPKzMyS5IAyM7MkOaDMzCxJDigzM0uSA8rMzJLkgDIzsyQ5oMzMLEkOKDMzS5IDyszMkuSAMjOzJDmgzMwsSQ4oMzNLkgPKzMyS5IAyM7MkVUtASXpW0kHl2s6WdHMVHHuYpB3Wcd99Je2xDvu9J6nFupzTzMzWTXWNoO4Bji/XdnzWvl4i4ucR8eY67r4vsNYBZWZmG151BdRo4HBJ9QAkdQBaAydImiBpiqTBpRtL2lXSS5ImSXpFUiNJBZKukfQfSW9I+lW27bOSumev50u6PNvvZUlbZO1HSBov6TVJT0raIqvhDOAcSa9L2kvS5pLGSCrOfvbM9m8u6Yls/6GAqul9MjOzSlRLQEXEXOAV4OCs6XjgXuC3EdEd6AzsI6mzpKJs3a8jYmfgAGAhcBrQEdglIjoDd1dwqobAy9l+zwGnZu0vALtHxC7ASOCCiHgPuAUYEhFdIuJ54IZseVfgWGBYtv8g4IVs/4eAdpX1VdJpWehOWL584dq9UWZmVqnCajx26WW+B7M/TwH6SjotO28rYAcggNkRUQwQEV8DSDoAuCUilmXtn1dwjiXAI9nricCB2eu2wL2SWgFFwLuV1HgAsIO0YoDUWFIjYG+gV3beRyV9UVknI+JW4FaAoqItorLtzMxs7VTnLL4HgP0ldQUaAF8A5wH7ZyOiR4H65C6fVfTBXll7WUsjonSbEr4N3JuAv0TETsDp2XkqUgfomY2oukREm4iYl61z2JiZ5VG1BVREzAeeBW4nN5pqDHwDfJXdKzok23Qq0FrSrgDZ/adC4AngjOw1kjZbi9M3AT7MXp9Upn0e0KjM8hPAL0sXJHXJXj4H9M/aDgGarcW5zcysClT370HdA+wMjIyIScBrwBRyofUiQEQsAY4DbpI0CRhLbsQzDHgfeCNrP2EtznspMErS88BnZdofBo4pnSQBnAV0zyZhvEluEgXAYGBvSa8CP87qMDOzDUjfXiGz9VVUtEW0aLE2OWpmKTrr0avyXcJGZeAudddrf0kTswl0K/E3SZiZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJAeUmZklyQFlZmZJckCZmVmSHFBmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmlqTCfBdQk3TuvBUTJgzJdxlmZjWCR1BmZpYkB5SZmSXJAWVmZklyQJmZWZIcUGZmliQHlJmZJckBZWZmSXJAmZlZkhxQZmaWJEVEvmuoMSTNA6blu448aQF8lu8i8sj9d//d/3XXPiI2L9/orzqqWtMionu+i8gHSRNqa9/B/Xf/3f/q6L8v8ZmZWZIcUGZmliQHVNW6Nd8F5FFt7ju4/+5/7VYt/fckCTMzS5JHUGZmliQHlJmZJckBVQUkHSxpmqQZkgbmu57qJmkrSc9IekvSFEm/zto3kzRW0tvZn83yXWt1kVQg6TVJj2TLtanvTSWNljQ1+zvQs5b1/5zs7/1kSfdIql+T+y/pdkmfSppcpq3S/kq6KPssnCbpoPU5twNqPUkqAP4KHALsAPSTtEN+q6p2y4DfRMT2wO7AL7I+DwSeiohtgaey5Zrq18BbZZZrU99vAP4VEdsBO5N7H2pF/yW1Ac4CukdEJ6AAOJ6a3f+/AweXa6uwv9nnwPHAjtk+N2efkevEAbX+egAzIuK/EbEEGAkcleeaqlVEzI6IV7PX88h9QLUh1+/h2WbDgaPzUmA1k9QWOAwYVqa5tvS9MbA38L8AEbEkIr6klvQ/Uwg0kFQIbAJ8RA3uf0Q8B3xerrmy/h4FjIyIxRHxLjCD3GfkOnFArb82wAdllmdlbbWCpA7ALsB4YIuImA25EANa5rG06nQ9cAGwvExbben794A5wB3ZJc5hkhpSS/ofER8C1wDvA7OBryLiCWpJ/8uorL9V+nnogFp/qqCtVszdl7QpMAY4OyK+znc9G4Kkw4FPI2JivmvJk0KgK/C3iNgF+IaadTlrlbJ7LUcBHYHWQENJA/JbVVKq9PPQAbX+ZgFblVluS27IX6NJqksunO6OiPuz5k8ktcrWtwI+zVd91WhP4EhJ75G7nLufpLuoHX2H3N/3WRExPlseTS6wakv/DwDejYg5EbEUuB/Yg9rT/1KV9bdKPw8dUOuvGNhWUkdJReRuED6U55qqlSSRuwfxVkRcV2bVQ8BJ2euTgAc3dG3VLSIuioi2EdGB3H/rpyNiALWg7wAR8THwgaQfZE37A29SS/pP7tLe7pI2yf4d7E/uHmxt6X+pyvr7EHC8pHqSOgLbAq+s60n8TRJVQNKh5O5LFAC3R8Tl+a2oekn6IfA88B++vQ9zMbn7UPcB7cj9Q+4TEeVvrtYYkvYFzouIwyU1p5b0XVIXchNEioD/Aj8l9z+7taX/g4HjyM1mfQ34ObApNbT/ku4B9iX3SI1PgEHAA1TSX0m/BU4h9/6cHRGPrfO5HVBmZpYiX+IzM7MkOaDMzCxJDigzM0uSA8rMzJLkgDIzsyQ5oMzMLEkOKDMzS9L/A2WRfv40ML3DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "b2 = plt.barh(['Vaccinated', 'Unvaccinated'], \n",
    "              [100*(TP/ (TP+FP)), 100*(TN/ (TN+FN))],\n",
    "              color=\"midnightblue\")\n",
    "\n",
    "b1 = plt.barh(['Vaccinated', 'Unvaccinated'], \n",
    "              [100*(FP/ (TP+FP)),100*(FN/ (TN+FN))], \n",
    "              left=[100*(TP/ (TP+FP)), 100*(TN/ (TN+FN))], \n",
    "              color=\"lightskyblue\")\n",
    "\n",
    "plt.legend([b1, b2], [\"Incorrect\", \"Correct\"], \n",
    "           title=\"Predictions\", loc=\"best\")\n",
    "plt.title('Prediction Accuracy')\n",
    "\n",
    "plt.savefig('./images/Prediction_Accuracy.jpeg', bbox_inches = 'tight')\n",
    "plt.savefig('./images/Prediction_Accuracy.png', bbox_inches = 'tight')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we are using an older version of SkLearn we have to plot our ROC Curve by hand. The following code block was also found on https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/. Newer versions of SkLearn take predicted y and actual y values as arguments rather than the model and y values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-04-22T17:25:58.314144Z",
     "start_time": "2022-04-22T17:25:27.846177Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvHElEQVR4nO3deXxV9Zn48c+Tm30nJKwhEgEVZNWgoKJSi/v8qEtVbLW2zlhtbWde09J1VOzYjlMYO9qNodairUU7ai3jgmiVoqgsAiKbgGwJAUL29Sa59z6/P85NGkOWC+Tm5t7zvF+v+8o953zPOc8h4TznfL/nfL+iqhhjjHGvuEgHYIwxJrIsERhjjMtZIjDGGJezRGCMMS5nicAYY1wuPtIBnKjc3FwdPXp0pMMwxpio8sEHH5Sral5Xy6IuEYwePZoNGzZEOgxjjIkqInKgu2VWNWSMMS5nicAYY1zOEoExxricJQJjjHE5SwTGGONyYUsEIvKEiJSJyNZulouIPCYie0Rki4icE65YjDHGdC+cdwRLgSt7WH4VMC74uQv4dRhjMcYY042wvUegqqtFZHQPReYCT6nTD/b7IpItIsNV9XC4YjLGmP7W4vPT4gvQ6gtQ09iCP6AE1Pn4A0ogOF1Z10xcnODzB/AFApTXeokTQaG93NmjBnHumC7fCTslkXyhbCRQ3GG6JDjvuEQgInfh3DVQUFDQL8EZY9xFVWlu9dPQ7KPB20qzL0Bzq5/iinpKKxvxBQIEAs7J2/kEqG9qpaHF1z4/EFD8qu3ThyobaPEF+izGmy4YE3OJQLqY1+UoOaq6BFgCUFRUZCPpGGPa1TW10tDc6lxJ+5VWfwBvi486byuV9c3UNbXS4vPT3OqnttEpW93QQk1jC63+AD5/gFZ/gAavj0A3A3XFxwkJ8XF44gRPnPMzLk5ITvCQlZpInAgJnjg8Cc58jzg/Rw/JIMETR+HQTBI8cTS3+hmanRLcjhAXLOeJc06HqYnxpCbFE++JIz4ujpRED4kJHqecgEhXp81TF8lEUAKM6jCdD5RGKBZjTIS0VZ20nZBbfAEavK00tvgoq2misq6Zem8rFXVeappaqa5vprHFR3Orn8r65pD2IUBCfBxZqYmkJSWQlZbI2GGZJMZ7iPcI8Z44UhPjyUhJICUpnvTkBBLj40iM9zB8UCrDB6USF6aT8EAQyUSwHLhXRJ4BzgdqrH3AmNgRUKWmoYWKOi8Hy+vZV1ZHdYNzhd7U4qOh2cfhqkbqva29bivBE0dOehJZaYlkpyVyWl46iQke0pMTiBNhZE4qSQke4uPiiPfEkZzoLMtKTWRQehLxcRK2q+lYELZEICLLgEuBXBEpAR4AEgBUdTHwCnA1sAdoBL4crliMMX3L2+pn79Faymu91DQ209Ti51BFA4pypLqJ0soGKuq8BLqoaSnITScjJYHM1ETOHJFFdloSackJJASvzOPj4khPTiAlycPg9GSGZqeQ4ImzE3kYhfOpoXm9LFfg6+HavzHm5Hlb/ewvq2V/WR1Ha5qobmih3ttKvbeVI9WNlFY2drvuWSOzmXzaYIZkpZCdlkhuRjIjctIYmp1CSmJ8TFexRKuo64baGHPqGryt7D9Wx7EaL1UNzZTVNHG0upGjNU0crWmirunv1TVxIsG6dacOfWROGpdNyqdwSAYjBqWSlZZIapJTp24n+ehkicCYGNPqD1DT0IK31Ue910dFnZfKei8HjtWzrbiKFl9bNc7fJSV4GJqVwrDsFM4cmU1uRjLDslMZOzyLoVkpJCV4InY8JvwsERgTxVr9ASqDjbH7y+pYu7uMbcWVXdbNJyd4yMtMJjHewxcvOYOxwzIZlp3KoPQkMlMSrA7exSwRGDOA+QNKWU0Tuw/XUN3QTFOLc5W/eV85lfXNVNR5P3Vlf1peOp+fOaa9Pj45wUNeVgo56UkMSk+yqhvTJUsExgwAPn+APUdq2Hu0jgPH6iiuaOBYTRMHy+uPKxsfJyTGe0hLjucLF49jcEYyBbnpTnXOoNQIRG+inSUCY/qJt9XvXNU3+yitaqS0soFjtV52lVaz81B1+5V9gieO0UMyGJWbTtGYPDJSEphUkENORjK5GckkxtujlKZvWSIwpg95W/0cqWrkWG0T3hY/G/eVs/NQNcXl9bT6j+9zJjUpnhGDUpl30VhOy8tgwqhBZKclkhhvjbOm/1giMOYUNLX4eGfHEdbuPsq+sjpKKhqOKzMsO4XPnTeatOQEstOcLg6GZqeQn5NGWnJCBKI25tMsERhzAvwB5VhNE2/vPMybH5VyqLKB5lY/eZnJFA7N5JIJI8jLSiZ/cDppSfFkpiSSm5kc6bCN6ZElAmN6UNPYwvu7jrL3aC0fl1azq7QGf4dnMy88axg3zChkQv4gq7c3UcsSgXE9VWVfWR2fHKllb1kt9U2t1Da2cLSmiX1ldQAkxceRl5XCldNGMTovg5GD0zhrZDZpSVa1Y6KfJQLjOsXl9ew4VEVTs48D5fVs2HOMozVNgHPCz0xNJCUxnuGDUikak8dF44dx5ohsu+I3McsSgYk5qkplfTOHqxr5YO8x/AFn5KljtV7W7jqKr0PVTnKCh2mFudx84RgmFeQwbFCqPbFjXMcSgYkJR6sbeXNrKSs2HeRIddNxy9s6TJswahDjRw5iamEuo4ekk5Wa1D46lDFuZYnARK1Wf4BfrdjGnsM17DpcA8DInDSmj81jQv4gxg3PYkROGiMGpVq1jjE9sERgooo/oKzdfZRXNx5k3Z5jAAxKS+LmC8cwa/xwxg3PinCExkQfSwRmwGvrVfPjUucN3bb+dwqHZHDLhWO5dOKICEdoTHSzRGAGpOqGZpa+9TGvbipun5cYH0dSgoe75oxn9sQR5KTbi1rG9AVLBGbA8PkD/N+GA2zce4xN+ypo9Qc4b2weQ7NT+cKscQxKT4p0iMbEJEsEJqICqpRWNrCjpJqlb31MeZ2XlEQPl549ghtnns7oIRmRDtGYmGeJwPQ7f0DZXlzJOzuP8MaWEuq9PgBy0pOYP3cKn52cH+EIjXEXSwSmX/gDAdbuLuOND0vYcaiayvpmAGaNH07RmFzOHJFNQV46nri4CEdqjPtYIjBhVVHn5Q+rd7Nm5xFqGlsAmHxaDl+9fALnnu4MumKMiSxLBCYsVm0t5S/r97O9pAqAmWcM5fIp+Zw7Jo+kBOvCwZiBxBKB6TOb9pWz4ZNjLF+/nxafMxrXbZecweyJIxiZkxbh6Iwx3bFEYE5aqz/A6m2lvL3jCO/tOto+PzstkRGDklj4pRlkpiRGMEJjTCgsEZgT0tDcyksbDrBmpzNYS6s/QG5GMueNG0JuRjL/eNlZNvyiMVHGEoHpVV1TK+/vOsrGvcd4c2tp+/wrp41i5hlDmT42z572MSaKWSIw3fIHlJc/OMAvV2wDICs1kdkTR3DJhBGcN26Idd9sTIywRGC69O7OIyx+fTtHq5tI8MRx3+fPYfrYIcRZd87GxBxLBKadqvL2jiM8//5edh6qJj5OmDt9NHfMPpPUJPtTMSZWhfV/t4hcCTwKeIDHVfXhTsuzgD8ABcFYFqnq78IZkzmeqvK3bYd5Ye0+Pi6tJjUpntsuHsd1MwptcHZjXCBsiUBEPMAvgTlACbBeRJar6vYOxb4ObFfVfxCRPOBjEXlaVVvCFZdxqCovrtvP2t1lbNpX3j7/5gvG8MVLxtm4vca4SDjvCM4D9qjqXgAReQaYC3RMBApkiDOOYDpQCfjCGJPreVt8/PGdPazYVNze5cOZI7IpGpPHLReNsQRgjAuFMxGMBIo7TJcA53cq8wtgOVAKZAA3q2qg84ZE5C7gLoCCgoKwBOsGq7cf5vG/7uBodRO5mcn88zWT+OzkkXbyN8blwpkIunq8RDtNXwFsBj4DjAFeF5G3VbX2UyupLgGWABQVFXXehunFRwcr+fFzG6lqcHr8/NEtRZw3dogN6G6MAcKbCEqAUR2m83Gu/Dv6MvCwqiqwR0T2AWcB68IYl6u8svEgj778EQA3zCjkjtln2h2AMeZTwpkI1gPjRKQQOATcAtzaqcxB4DLgbREZCpwJ7A1jTK6hqqzYXMxjwSTw6Fcu5KyR2ZENyhgzIIUtEaiqT0TuBV7DeXz0CVXdJiJ3B5cvBv4dWCoiH+FUJX1XVcu73agJSVOLjwee3cCH+ys4fWgmi740wx4DNcZ0K6zvEajqK8ArneYt7vC9FLg8nDG4TU1jCz99cTMf7q9g+KBUHv3KBVYVZIzpkb0uGkNe/uAAv1qxDV9Auf2SM7h11lhrEDbG9MoSQQzwBwIse3sPv1+9m0kFOdx+6RlMKsixJGCMCYklgijX4vNz8yNv0NjsY2h2Ct+7bhq5mcmRDssYE0UsEUSxVn+AB57dQGOzjzlT8vnXf5hsvYMaY06YJYIo9dbWQ/z0xQ8JqHL9+YV89fIJkQ7JGBOlLBFEod+8sYPn3nNet7jtkjP44sXjIhyRMSaaWSKIIv6A8rs3d/Lce3vJzUjmf+6+mHQbH9gYc4osEUSRJa9v58V1+5lWmMsPrp9mScAY0ycsEUSBVn+AX7y6lRWbirn07BF877qp9mioMabPWCIY4PwB5f5n1rNxbzlXTM3n3qsmWhIwxvSpkBOBiKSpakM4gzHHu2/ZOjbuLbcng4wxYRPXWwERuUBEtgM7gtNTRORXYY/M5fyBAEvf+pgP9pYzIX8Qd80ZH+mQjDExqtdEAPwMZwCZCgBV/RC4OJxBuZ0/EODHz29i2Tt7GJqVwjeutuogY0z4hFQ1pKrFnU5E/vCEYwCeXfMJa3Ye4epzCvjnayZFOhxjTIwLJREUi8gFgIpIIvBNgtVEpu/tKq3myVW7OHvUIL559cRIh2OMcYFQqobuBr6OMxh9CTAV+FoYY3Itb6ufb/x2DXGCPR1kjOk3odwRnKmqX+g4Q0QuBNaEJyT3+sPfdgFOtxGnD82McDTGGLcI5Y7g5yHOM6fgWG0T//veXhLj47hx5umRDscY4yLd3hGIyEzgAiBPRP61w6JMnDGITR9q60Tu+9dNs6EljTH9qqeqoUQgPVgmo8P8WuDGcAblNvvL6nhx3X7GDsvkgrOGRTocY4zLdJsIVPVvwN9EZKmqHujHmFxFVfnZS1sA+JdrJ0c4GmOMG4XSWNwoIguBs4H2MRBV9TNhi8olVJV/W7aenYeqmXxaDuOGZ0U6JGOMC4XSWPw0sBMoBB4E9gPrwxiTa/zp3b1s+OQYZ48axH984fxIh2OMcalQEsFgVf0t0Kqqf1PVrwAzwhyXKzzx5k4AFt4+g3hPKL8KY4zpe6FUDbUGfx4WkWuAUiA/fCG5w+7DNQBce24BnjhLAsaYyAklETwkIlnAt3DeH8gE/iWcQcW64vJ6vvPU+yR44rh+hr0zYIyJrF4Tgaq+FPxaA8yG9jeLzUl6+M+bQODXd81iZE5apMMxxrhct3USIuIRkXki8m0RmRicd62IvAv8ot8ijDHbiivZc6SWGeOGMCo3PdLhGGNMj3cEvwVGAeuAx0TkADAT+J6qvtgPscWcijovC57dQEZKAl+57KxIh2OMMUDPiaAImKyqARFJBsqBsap6pH9Ciz1L3/qY2qZWfnD9NPIyUyIdjjHGAD0/PtqiqgEAVfUCu040CYjIlSLysYjsEZHvdVPmUhHZLCLbRORvJ7L9aBJQZcMnxzj39FwuOXtEpMMxxph2Pd0RnCUiW4LfBRgTnBZAVbXH/hBExAP8EpiDM47BehFZrqrbO5TJBn4FXKmqB0VkyMkfysC29K2Pqaxv5uYLx0Q6FGOM+ZSeEsGpjpZ+HrBHVfcCiMgzwFxge4cytwIvqOpBAFUtO8V9DliVdc0AzJ0+OrKBGGNMJz11OneqHc2NBIo7TJcAnftROANIEJFVOD2cPqqqT3XekIjcBdwFUFBQcIph9b8Wn59N+8vJSU+yUceMMQNOOF9p7eqMp52m44FzgWuAK4D7ROSM41ZSXaKqRapalJeX1/eRhtmf1+6nvNbLt/7flEiHYowxxwnlzeKTVYLz+GmbfJzuKTqXKVfVBqBBRFYDU4BdYYyr3234xKnxKhoTfUnMGBP7QrojEJEUETnzBLe9HhgnIoUikgjcAizvVOYvwCwRiReRVJyqox0nuJ8BzdviY8uBSq45N/qqtIwx7tBrIhCRfwA2AyuC01NFpPMJ/Tiq6gPuBV7DObn/SVW3icjdInJ3sMyO4Ha34Ly49riqbj3JYxmQHv+r08PoRWcNj3AkxhjTtVCqhhbgPAG0CkBVN4vI6FA2rqqvAK90mre40/RCYGEo24s2Ww5U8H8bDjB19GDOOT030uEYY0yXQqka8qlqTdgjiTGllQ384Ol1DEpL4jufmxrpcIwxpluh3BFsFZFbAY+IjAO+Cbwb3rCiW3VDM998Yg1xAg9/8XwGZyT3vpIxxkRIKHcE38AZr7gZ+CNOd9T/EsaYopqqct+y9dQ1tXL3FWczekhGpEMyxpgehXJHcKaq/hD4YbiDiQW/eHUruw7XcPU5BVx9jj0pZIwZ+EK5I3hERHaKyL+LyNlhjyiKbS+p4qUPDjI6L4N7r7J/KmNMdOg1EajqbOBS4BiwREQ+EpF/C3dg0Wjl5mISPHE88uWZNg6xMSZqhHS2UtUjqvoYcDfOOwX3hzOoaLR+TxmvbirmovHDSEtKiHQ4xhgTslBeKBsvIgtEZCvOEJXv4nQXYYJ2H67h/mfWk5GSwLyLxkY6HGOMOSGhNBb/DlgGXK6qnfsKcr3Sygbuf2Y9qUkJ/OLOixg2KDXSIRljzAnpNRGo6oz+CCQa1TW18oM/rqOyvpmHv3i+JQFjTFTqNhGIyJ9U9SYR+YhPdx8d0ghlbvDTFzdxuKqRGeOGMK3QupAwxkSnnu4I/jn489r+CCTaBFT55GgtF545lPtvKop0OMYYc9K6bSxW1cPBr19T1QMdP8DX+ie8gWvFpmIq6pqZPi5mh1k2xrhEKI+Pzuli3lV9HUg0aW7185s3djB8UCpXTB3V+wrGGDOA9dRGcA/Olf/pIrKlw6IMYE24AxvIVmw6SGOzj2/9w2TibAxiY0yU66mN4I/Aq8B/AN/rML9OVSvDGtUAVlHn5dl3P2FQWhIXnDUs0uEYY8wp6ykRqKruF5Gvd14gIjluTQb3LVtPVX0L//2VC+xuwBgTE3q7I7gW+ADn8dGOZz0FTg9jXAOSPxDgk6O1jB+ZzZkjsiMdjjHG9IluE4GqXhv8Wdh/4Qxse4/WAVA0Ji/CkRhjTN8Jpa+hC0UkLfj9iyLyiIi4sqP9t7YeAmDiaTkRjsQYY/pOKI+P/hpoFJEpwHeAA8DvwxrVAOQPKCs/LCErNZGJoywRGGNiR6iD1yswF3hUVR/FeYTUVd7efpi6plZunHk68R4ba8AYEztC6X20TkS+D9wGzBIRD+C6Dvf/5/XtAHx28sgIR2KMMX0rlEvbm3EGrv+Kqh4BRgILwxrVANPU4qO6oYXCIRnkpCdHOhxjjOlToQxVeQR4GsgSkWsBr6o+FfbIBpBjtV4Cqtw403VPzBpjXCCUp4ZuAtYBnwduAtaKyI3hDmwgKS6vB2BoVkqEIzHGmL4XShvBD4HpqloGICJ5wBvAc+EMbCA5Ut0IYAPPGGNiUihtBHFtSSCoIsT1Yka9txWAwRnWPmCMiT2h3BGsEJHXcMYtBqfx+JXwhTTwlFY2MiQrxfoWMsbEpFDGLJ4vItcDF+H0N7REVf8c9sgGkC0HKhhi7QPGmBjV03gE44BFwBjgI+DbqnqovwIbKLYVV1JZ38zMM4dGOhRjjAmLnur6nwBeAm7A6YH05ye6cRG5UkQ+FpE9IvK9HspNFxH/QHwa6ZHlW8hKTeT2S86IdCjGGBMWPVUNZajqb4LfPxaRjSey4eAbyL/EGeqyBFgvIstVdXsX5f4TeO1Ett8fymqaKKls4PoZhWSnJUU6HGOMCYueEkGyiEzj7+MQpHScVtXeEsN5wB5V3QsgIs/g9Fe0vVO5bwDPA9NPMPaw23O4BoDzbYB6Y0wM6ykRHAYe6TB9pMO0Ap/pZdsjgeIO0yXA+R0LiMhI4LrgtrpNBCJyF3AXQEFB//WAvXZPGXECY4Zm9ds+jTGmv/U0MM3sU9x2V89aaqfp/wa+q6p+6eHRTFVdAiwBKCoq6ryNsGjwtrJyczHnjR1CRorr+tgzxrhIKO8RnKwSYFSH6XygtFOZIuCZYBLIBa4WEZ+qvhjGuEKyeX8FAcX6FzLGxLxwJoL1wDgRKQQOAbcAt3Ys0HEYTBFZCrw0EJIAQElFAwAjctIiHIkxxoRX2BKBqvpE5F6cp4E8wBOquk1E7g4uXxyuffeFd3YcZnReBjnp9rSQMSa29ZoIxKm3+QJwuqr+KDhe8TBVXdfbuqr6Cp26o+guAajqHSFF3A+OVDWy63AN188opKe2C2OMiQWhdB73K2AmMC84XYfzfkDM+u+XPwLgMxNtNDJjTOwLpWrofFU9R0Q2AahqlYgkhjmuiFFVNu0rZ9b4YYwbbo+NGmNiXyh3BK3Bt38V2scjCIQ1qghqG4TmtLyMCEdijDH9I5RE8BjwZ2CIiPwYeAf4SVijiqAj1U0ATBk9OMKRGGNM/wilG+qnReQD4DKcl8Q+p6o7wh5ZhGwtriROhDOsWsgY4xKhPDVUADQC/9dxnqoeDGdgkaCqvPnRIcbnZ5OcGM5XLIwxZuAI5Wz3Mk77gADJQCHwMXB2GOOKiPd2HeVYrZebLhgT6VCMMabfhFI1NKnjtIicA3w1bBFF0P6yOgBmnGGD0Bhj3OOEB6EPdj894LqM7guvbDxIbmayDUtpjHGVUNoI/rXDZBxwDnAsbBFFSHOrn/JaLxeNHxbpUIwxpl+F0kbQ8YF6H06bwfPhCSdyahpbUGDK6NxIh2KMMf2qx0QQfJEsXVXn91M8EbOjpAqAYdlWLWSMcZdu2whEJF5V/ThVQTHvhbX7SIyPsxfJjDGu09MdwTqcJLBZRJYD/ws0tC1U1RfCHFu/8bb42Hu0lpE5aSTGeyIdjjHG9KtQ2ghygAqccYXb3idQIGYSweGqRlp8AT533uhIh2KMMf2up0QwJPjE0Fb+ngDa9Mu4wf2lrNbpX2ikjUZmjHGhnhKBB0gntEHoo9r+MqfH0VG56RGOxBhj+l9PieCwqv6o3yKJoLW7j5KbkUxWaswOs2CMMd3q6c1iV4zRuHFvOduKq7hq2igbltIY40o9JYLL+i2KCPrb9lIAZk0YHuFIjDEmMrpNBKpa2Z+BRMrhqkayUhNtRDJjjGudcKdzsSYQUFp9MTvypjHG9Mr1iaCpxcf4/OxIh2GMMRHj6kTgDwQormiwx0aNMa7m6kSwZudRmlv9TByVE+lQjDEmYlydCPaV1QJw7pi8CEdijDGR4+pE8NbWUkbmpJGaZAPVG2Pcy7WJoLqhmcNVjWSkJEQ6FGOMiSjXJoI9R5xqodkTR0Q4EmOMiSzXJoKmFh8AZ1tDsTHG5cKaCETkShH5WET2iMj3ulj+BRHZEvy8KyJTwhlPR+t2lxEnMCIntb92aYwxA1LYEkFwvONfAlcBE4B5IjKhU7F9wCWqOhn4d2BJuOLpbOeharLTkkhLsjYCY4y7hfOO4Dxgj6ruVdUW4BlgbscCqvquqlYFJ98H8sMYT7vaxhYOltdzwZlD+2N3xhgzoIUzEYwEijtMlwTndedO4NWuFojIXSKyQUQ2HDt27JQDK6txRiSzN4qNMSa8iSDkkc1EZDZOIvhuV8tVdYmqFqlqUV7eqb/8daS6EYCCXOtx1BhjwvkmVQkwqsN0PlDauZCITAYeB65S1YowxtOuqcUPYO8QGGMM4b0jWA+ME5FCEUkEbgGWdywgIgXAC8BtqrorjLF8Sm1TCwDDsu2JIWOMCdsdgar6RORe4DXAAzyhqttE5O7g8sXA/cBg4FfBYSJ9qloUrpjaNLc6dwTJiZ5w78oYYwa8sHayo6qvAK90mre4w/d/BP4xnDF0JRBwmio8cTZGsTHGuPLNYn9AiROIs8HqjTHGnYmgtqmF5ETrcdQYY8CliaCspom8zORIh2GMMQOCKxNBY4ufrNTESIdhjDEDgisTwdaDldbHkDHGBLkuEVQ3NAMQ73HdoRtjTJdcdzZse4dg+lgbp9gYY8CFicBn7xAYY8ynuC4RtATvCOLjXHfoxhjTJdedDXcfqQEgO92eGjLGGHBhItBgR9jDrcM5Y4wBXJgIfP4AAAnxrjt0Y4zpkuvOhqVVjSR44khPtvcIjDEGXJgIKuu85GYmkxhvXVAbYwy4MBFU1DeTmWINxcYY08Z1iaC0soHhg6yh2Bhj2rguEdR7W63DOWOM6cB1iaDVF7AhKo0xpgNXJYKymiZ8AbU2AmOM6cBViaAq2PNoRoo9OmqMMW1clQiO1TQBkGujkxljTDtXDdwb7HiU5ARrIzDRo7W1lZKSErxeb6RDMVEgOTmZ/Px8EhJCr/lwVSJobG4FIDs1KcKRGBO6kpISMjIyGD16NCLWfbrpnqpSUVFBSUkJhYWFIa/nqqqhTfsqAMjJsERgoofX62Xw4MGWBEyvRITBgwef8N2jqxJBm5REV90ImRhgScCE6mT+VlyVCBpbfBQOyYh0GMYYM6C4KhF8fKjaHh015iR4PB6mTp3KxIkT+fznP09jYyMbNmzgm9/85klvMz09HYDS0lJuvPHGU4rvxhtvZO/eve3TmzZtQkR47bXX2uft37+fiRMnfmq9BQsWsGjRovbpRYsWcdZZZzFx4kSmTJnCU089dUpxATz55JOMGzeOcePG8eSTT3ZZ5uDBg8yePZtp06YxefJkXnnllU8tu/zyyxk/fjwTJkxg//79ANxyyy3s3r37lOMDlyUCEefNYmPMiUlJSWHz5s1s3bqVxMREFi9eTFFREY899tgpb3vEiBE899xzJ73+tm3b8Pv9nH766e3zli1bxkUXXcSyZctC3s7ixYt5/fXXWbduHVu3bmX16tVo20hWJ6myspIHH3yQtWvXsm7dOh588EGqqqqOK/fQQw9x0003sWnTJp555hm+9rWvtS+7/fbbmT9/Pjt27GDdunUMGTIEgHvuuYef/vSnpxRfG1dVlqvC2OFZkQ7DmJP269e2sfdobZ9u8/Shmdxzxdkhl581axZbtmxh1apVLFq0iJdeeokFCxbwySefcOjQIYqLi/nOd77DP/3TPwGwcOFC/vSnP9Hc3Mx1113Hgw8++Knt7d+/n2uvvZatW7eydOlSli9fTmNjI5988gnXXXdd+8lu5cqVPPDAAzQ3NzNmzBh+97vfkZ6eztNPP83cuXPbt6eqPPfcc7z++uvMmjULr9dLcnLv7w795Cc/4a233iIzMxOArKwsvvSlL4X879KV1157jTlz5pCTkwPAnDlzWLFiBfPmzftUORGhttb5vdbU1DBixAgAtm/fjs/nY86cOcDf76LA+T3ccccd+Hw+4uNP7VTumjuCgCo1jS144qzRzZiT5fP5ePXVV5k0adJxy7Zs2cLLL7/Me++9x49+9CNKS0tZuXIlu3fvZt26dWzevJkPPviA1atX97iPzZs38+yzz/LRRx/x7LPPUlxcTHl5OQ899BBvvPEGGzdupKioiEceeQSANWvWcO6557avv2bNGgoLCxkzZgyXXnrpp6pZulNXV0ddXR1jxozptezChQuZOnXqcZ+uqskOHTrEqFGj2qfz8/M5dOjQceUWLFjAH/7wB/Lz87n66qv5+c9/DsCuXbvIzs7m+uuvZ9q0acyfPx+/3w9AXFwcY8eO5cMPP+w15t645o6gOti9RKvfqoZM9DqRK/e+1NTUxNSpUwHnSvTOO+/k3Xff/VSZuXPnkpKSQkpKCrNnz2bdunW88847rFy5kmnTpgFQX1/P7t27ufjii7vd12WXXUZWlnPnPmHCBA4cOEB1dTXbt2/nwgsvBKClpYWZM2cCcPjwYfLy8trXX7ZsGbfccgvg1KP//ve/5/rrr+/2aRoRQVVDftpm/vz5zJ8/P6SyXVUtdbWfZcuWcccdd/Ctb32L9957j9tuu42tW7fi8/l4++232bRpEwUFBdx8880sXbqUO++8E4AhQ4ZQWlr6qUR4MsKaCETkSuBRwAM8rqoPd1ouweVXA43AHaq6MRyx1Ht9AEwclROOzRsT09raCHrS+QTXdoL9/ve/z1e/+tWQ95WU9Pf3fDweDz6fD1Vlzpw5Xdb5p6SktD837/f7ef7551m+fDk//vGP21+wqqurY/DgwcfVz1dWVlJYWEhmZiZpaWns3bv3U20NXVm4cCFPP/30cfMvvvji49pM8vPzWbVqVft0SUkJl1566XHr/va3v2XFihUAzJw5E6/XS3l5Ofn5+UybNq09ps997nO8//777YnA6/WSkpLSY7yhCFvVkIh4gF8CVwETgHkiMqFTsauAccHPXcCvwxVP2x2BdS9hTHj85S9/wev1UlFRwapVq5g+fTpXXHEFTzzxBPX19YBTVVJWVnbC254xYwZr1qxhz549ADQ2NrJr1y4Axo8f3z7/jTfeYMqUKRQXF7N//34OHDjADTfcwIsvvkh6ejrDhw/nr3/9K+AkgRUrVnDRRRcB8P3vf5+vf/3r7XX1tbW1LFmy5LhY5s+fz+bNm4/7dNVwfsUVV7By5Uqqqqqoqqpi5cqVXHHFFceVKygoaI9rx44deL1e8vLymD59OlVVVRw7dgyAN998kwkT/n4a3bVrF2effep3ieFsIzgP2KOqe1W1BXgGmNupzFzgKXW8D2SLyPBwBFPvdbqXSLJEYExYnHfeeVxzzTXMmDGD++67jxEjRnD55Zdz6623MnPmTCZNmsSNN95IXV3dCW87Ly+PpUuXMm/ePCZPnsyMGTPYuXMnANdcc037VfeyZcu47rrrPrXuDTfcwB//+EcAnnrqKR566CGmTp3KZz7zGR544IH2doF77rmH2bNnM336dCZOnMgll1xCauqpjWaYk5PDfffdx/Tp05k+fTr3339/e8Px/fffz/LlywH4r//6L37zm98wZcoU5s2bx9KlSxERPB4PixYt4rLLLmPSpEmoansj/NGjR0lJSWH48FM/ZcqpPh7V7YZFbgSuVNV/DE7fBpyvqvd2KPMS8LCqvhOc/ivwXVXd0Glbd+HcMVBQUHDugQMHTjiebcWVvPD+Pr525dkMzrDeR0302LFjB+PHj490GD1asGAB6enpfPvb3+73fTc1NTF79mzWrFmDx+OeC72f/exnZGZmtlcTddTV34yIfKCqRV1tK5xtBF21vHTOOqGUQVWXAEsAioqKTipznT0qh7OtfcCYmJOSksKDDz7IoUOHKCgoiHQ4/SY7O5vbbrutT7YVzkRQAozqMJ0PlJ5EGWPMALdgwYKI7r+revdY9+Uvf7nPthXONoL1wDgRKRSRROAWYHmnMsuB28UxA6hR1cNhjMmYqBSuKlwTe07mbyVsdwSq6hORe4HXcB4ffUJVt4nI3cHli4FXcB4d3YPz+GjfpThjYkRycjIVFRXWFbXpVdvjsqG8Sd1R2BqLw6WoqEg3bNjQe0FjYoSNUGZORHcjlEWqsdgY0wcSEhJOaLQpY06Ua/oaMsYY0zVLBMYY43KWCIwxxuWirrFYRI4BJ/5qsSMXKO/DcKKBHbM72DG7w6kc82mqmtfVgqhLBKdCRDZ012oeq+yY3cGO2R3CdcxWNWSMMS5nicAYY1zObYng+M7FY58dszvYMbtDWI7ZVW0Exhhjjue2OwJjjDGdWCIwxhiXi8lEICJXisjHIrJHRL7XxXIRkceCy7eIyDmRiLMvhXDMXwge6xYReVdEpkQizr7U2zF3KDddRPzBUfOiWijHLCKXishmEdkmIn/r7xj7Wgh/21ki8n8i8mHwmKO6F2MReUJEykRkazfL+/78paox9cHp8voT4HQgEfgQmNCpzNXAqzgjpM0A1kY67n445guAQcHvV7nhmDuUexOny/MbIx13P/yes4HtQEFwekik4+6HY/4B8J/B73lAJZAY6dhP4ZgvBs4BtnazvM/PX7F4R3AesEdV96pqC/AMMLdTmbnAU+p4H8gWkVMfATpyej1mVX1XVauCk+/jjAYXzUL5PQN8A3geKOvP4MIklGO+FXhBVQ8CqGq0H3cox6xAhjiDNaTjJAJf/4bZd1R1Nc4xdKfPz1+xmAhGAsUdpkuC8060TDQ50eO5E+eKIpr1eswiMhK4Dljcj3GFUyi/5zOAQSKySkQ+EJHb+y268AjlmH8BjMcZ5vYj4J9VNdA/4UVEn5+/YnE8gq6GcOr8jGwoZaJJyMcjIrNxEsFFYY0o/EI55v8Gvquq/hgZ2SuUY44HzgUuA1KA90TkfVXdFe7gwiSUY74C2Ax8BhgDvC4ib6tqbZhji5Q+P3/FYiIoAUZ1mM7HuVI40TLRJKTjEZHJwOPAVapa0U+xhUsox1wEPBNMArnA1SLiU9UX+yXCvhfq33a5qjYADSKyGpgCRGsiCOWYvww8rE4F+h4R2QecBazrnxD7XZ+fv2Kxamg9ME5ECkUkEbgFWN6pzHLg9mDr+wygRlUP93egfajXYxaRAuAF4LYovjrsqNdjVtVCVR2tqqOB54CvRXESgND+tv8CzBKReBFJBc4HdvRznH0plGM+iHMHhIgMBc4E9vZrlP2rz89fMXdHoKo+EbkXeA3niYMnVHWbiNwdXL4Y5wmSq4E9QCPOFUXUCvGY7wcGA78KXiH7NIp7bgzxmGNKKMesqjtEZAWwBQgAj6tql48hRoMQf8//DiwVkY9wqk2+q6pR2z21iCwDLgVyRaQEeABIgPCdv6yLCWOMcblYrBoyxhhzAiwRGGOMy1kiMMYYl7NEYIwxLmeJwBhjXM4SgRmQgr2Fbu7wGd1D2fo+2N9SEdkX3NdGEZl5Ett4XEQmBL//oNOyd081xuB22v5dtgZ73MzupfxUEbm6L/ZtYpc9PmoGJBGpV9X0vi7bwzaWAi+p6nMicjmwSFUnn8L2Tjmm3rYrIk8Cu1T1xz2UvwMoUtV7+zoWEzvsjsBEBRFJF5G/Bq/WPxKR43oaFZHhIrK6wxXzrOD8y0XkveC6/ysivZ2gVwNjg+v+a3BbW0XkX4Lz0kTk5WD/91tF5Obg/FUiUiQiDwMpwTieDi6rD/58tuMVevBO5AYR8YjIQhFZL04f818N4Z/lPYKdjYnIeeKMM7Ep+PPM4Ju4PwJuDsZyczD2J4L72dTVv6NxoUj3vW0f+3T1Afw4HYltBv6M8xZ8ZnBZLs5blW13tPXBn98Cfhj87gEygmVXA2nB+d8F7u9if0sJjlcAfB5Yi9N520dAGk73xtuAacANwG86rJsV/LkK5+q7PaYOZdpivA54Mvg9EacXyRTgLuDfgvOTgA1AYRdx1nc4vv8FrgxOZwLxwe+fBZ4Pfr8D+EWH9X8CfDH4PRunD6K0SP++7RPZT8x1MWFiRpOqTm2bEJEE4CcicjFO1wkjgaHAkQ7rrAeeCJZ9UVU3i8glwARgTbBrjUScK+muLBSRfwOO4fTQehnwZ3U6cENEXgBmASuARSLynzjVSW+fwHG9CjwmIknAlcBqVW0KVkdNlr+PopYFjAP2dVo/RUQ2A6OBD4DXO5R/UkTG4fREmdDN/i8H/p+IfDs4nQwUEN39EZlTZInARIsv4Iw+da6qtorIfpyTWDtVXR1MFNcAvxeRhUAV8LqqzgthH/NV9bm2CRH5bFeFVHWXiJyL09/Lf4jISlX9USgHoapeEVmF03XyzcCytt0B31DV13rZRJOqThWRLOAl4OvAYzj97bylqtcFG9ZXdbO+ADeo6sehxGvcwdoITLTIAsqCSWA2cFrnAiJyWrDMb4Df4gz39z5woYi01fmnisgZIe5zNfC54DppONU6b4vICKBRVf8ALArup7PW4J1JV57B6ShsFk5nagR/3tO2joicEdxnl1S1Bvgm8O3gOlnAoeDiOzoUrcOpImvzGvANCd4eici07vZh3MMSgYkWTwNFIrIB5+5gZxdlLgU2i8gmnHr8R1X1GM6JcZmIbMFJDGeFskNV3YjTdrAOp83gcVXdBEwC1gWraH4IPNTF6kuALW2NxZ2sxBmX9g11hl8EZ5yI7cBGcQYt/x96uWMPxvIhTtfMP8W5O1mD037Q5i1gQltjMc6dQ0Iwtq3BaeNy9vioMca4nN0RGGOMy1kiMMYYl7NEYIwxLmeJwBhjXM4SgTHGuJwlAmOMcTlLBMYY43L/H81Nkb2dLyGNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# generate a no skill prediction (majority class)\n",
    "ns_probs = [0 for _ in range(len(y_test))]\n",
    "# fit a model\n",
    "final_model.fit_transform(X_train, y_train)\n",
    "# predict probabilities\n",
    "lr_probs = final_model.predict_proba(X_test)\n",
    "# keep probabilities for the positive outcome only\n",
    "lr_probs = lr_probs[:, 1]\n",
    "# calculate scores\n",
    "ns_auc = roc_auc_score(y_test, ns_probs)\n",
    "lr_auc = round((roc_auc_score(y_test, lr_probs)), 2)\n",
    "# calculate roc curves\n",
    "ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)\n",
    "lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)\n",
    "# plot the roc curve for the model\n",
    "plt.plot(lr_fpr, lr_tpr, label=f'Pipeline(AUC = {lr_auc})', color='steelblue')\n",
    "# axis labels\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "# show the legend\n",
    "plt.legend(loc='lower right')\n",
    "# show the plot\n",
    "plt.show()\n",
    "plt.savefig('./images/ROC_Curve2.jpeg', bbox_inches = 'tight');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choosing our classification threshold here doesn't improve our model's metrics significantly, however, it could prove essential in a data set where the target variable is more imbalanced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Possible Room for Improvement:\n",
    "- With more time we could run and tune more models. More and better models could improve our model's performance. \n",
    "- Tuning again after combining the estimators may also prove beneficial. \n",
    "- Getting another data source may add to the robustness of our models.\n",
    "- We may be able to predict other target variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (learn-env)",
   "language": "python",
   "name": "learn-env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "216px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}