week6/Lecture_6_Decision_Trees_Ensemble_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e9d9cf29",
   "metadata": {},
   "source": [
    "# Decision Trees and Ensemble Learning\n",
    "\n",
    "## Credit Risk Scoring\n",
    "\n",
    "* Build a model, that the bank can use to take a decision on whether they give a credit or not\n",
    "* The model gives the risk, that a customer won't pay back the credit (\"Risk of Defaulting\")\n",
    "![model](Screenshot_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b63894d1",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7db7c64f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee4633ea",
   "metadata": {},
   "source": [
    "## Data Cleaning and Preparation\n",
    "* Download the data\n",
    "* Re-encoding categorical variables\n",
    "* Doing the train / validation / test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a83950ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !wget -P ../data/ https://github.com/gastonstat/CreditScoring/raw/master/CreditScoring.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "891b66d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Status\",\"Seniority\",\"Home\",\"Time\",\"Age\",\"Marital\",\"Records\",\"Job\",\"Expenses\",\"Income\",\"Assets\",\"Debt\",\"Amount\",\"Price\"\r\n",
      "1,9,1,60,30,2,1,3,73,129,0,0,800,846\r\n",
      "1,17,1,60,58,3,1,1,48,131,0,0,1000,1658\r\n",
      "2,10,2,36,46,2,2,3,90,200,3000,0,2000,2985\r\n",
      "1,0,1,60,24,1,1,1,63,182,2500,0,900,1325\r\n",
      "1,0,1,36,26,1,1,1,46,107,0,0,310,910\r\n",
      "1,1,2,60,36,2,1,1,75,214,3500,0,650,1645\r\n",
      "1,29,2,60,44,2,1,1,75,125,10000,0,1600,1800\r\n",
      "1,9,5,12,27,1,1,1,35,80,0,0,200,1093\r\n",
      "1,0,2,60,32,2,1,3,90,107,15000,0,1200,1957\r\n"
     ]
    }
   ],
   "source": [
    "!head ../data/CreditScoring.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "003a5e1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Status</th>\n",
       "      <th>Seniority</th>\n",
       "      <th>Home</th>\n",
       "      <th>Time</th>\n",
       "      <th>Age</th>\n",
       "      <th>Marital</th>\n",
       "      <th>Records</th>\n",
       "      <th>Job</th>\n",
       "      <th>Expenses</th>\n",
       "      <th>Income</th>\n",
       "      <th>Assets</th>\n",
       "      <th>Debt</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>73</td>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>58</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>131</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>36</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>90</td>\n",
       "      <td>200</td>\n",
       "      <td>3000</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>2985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>182</td>\n",
       "      <td>2500</td>\n",
       "      <td>0</td>\n",
       "      <td>900</td>\n",
       "      <td>1325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>36</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>310</td>\n",
       "      <td>910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Status  Seniority  Home  Time  Age  Marital  Records  Job  Expenses  \\\n",
       "0       1          9     1    60   30        2        1    3        73   \n",
       "1       1         17     1    60   58        3        1    1        48   \n",
       "2       2         10     2    36   46        2        2    3        90   \n",
       "3       1          0     1    60   24        1        1    1        63   \n",
       "4       1          0     1    36   26        1        1    1        46   \n",
       "\n",
       "   Income  Assets  Debt  Amount  Price  \n",
       "0     129       0     0     800    846  \n",
       "1     131       0     0    1000   1658  \n",
       "2     200    3000     0    2000   2985  \n",
       "3     182    2500     0     900   1325  \n",
       "4     107       0     0     310    910  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read the data\n",
    "df = pd.read_csv(\"../data/CreditScoring.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9181665e",
   "metadata": {},
   "source": [
    "* lower the capitals of the column names\n",
    "* replace numbers with strings in categorical values\n",
    "* replace missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "37a51cf7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>status</th>\n",
       "      <th>seniority</th>\n",
       "      <th>home</th>\n",
       "      <th>time</th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>records</th>\n",
       "      <th>job</th>\n",
       "      <th>expenses</th>\n",
       "      <th>income</th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>amount</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>73</td>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>58</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "      <td>131</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>36</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>90</td>\n",
       "      <td>200</td>\n",
       "      <td>3000</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>2985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>24</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>63</td>\n",
       "      <td>182</td>\n",
       "      <td>2500</td>\n",
       "      <td>0</td>\n",
       "      <td>900</td>\n",
       "      <td>1325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>36</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>310</td>\n",
       "      <td>910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   status  seniority  home  time  age  marital  records  job  expenses  \\\n",
       "0       1          9     1    60   30        2        1    3        73   \n",
       "1       1         17     1    60   58        3        1    1        48   \n",
       "2       2         10     2    36   46        2        2    3        90   \n",
       "3       1          0     1    60   24        1        1    1        63   \n",
       "4       1          0     1    36   26        1        1    1        46   \n",
       "\n",
       "   income  assets  debt  amount  price  \n",
       "0     129       0     0     800    846  \n",
       "1     131       0     0    1000   1658  \n",
       "2     200    3000     0    2000   2985  \n",
       "3     182    2500     0     900   1325  \n",
       "4     107       0     0     310    910  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lower the capitals of the column names\n",
    "df.columns = df.columns.str.lower()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "777ee623",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    3200\n",
       "2    1254\n",
       "0       1\n",
       "Name: status, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.status.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ada25e36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace numbers with strings in categorical values\n",
    "# define the map dictionaries\n",
    "status_values = {\n",
    "    1: \"ok\", \n",
    "    2: \"default\", \n",
    "    0: \"unk\"\n",
    "}\n",
    "\n",
    "home_values = {\n",
    "    1: 'rent',\n",
    "    2: 'owner',\n",
    "    3: 'private',\n",
    "    4: 'ignore',\n",
    "    5: 'parents',\n",
    "    6: 'other',\n",
    "    0: 'unk'\n",
    "}\n",
    "\n",
    "marital_values = {\n",
    "    1: 'single',\n",
    "    2: 'married',\n",
    "    3: 'widow',\n",
    "    4: 'separated',\n",
    "    5: 'divorced',\n",
    "    0: 'unk'\n",
    "}\n",
    "\n",
    "records_values = {\n",
    "    1: 'no',\n",
    "    2: 'yes',\n",
    "    0: 'unk'\n",
    "}\n",
    "\n",
    "job_values = {\n",
    "    1: 'fixed',\n",
    "    2: 'partime',\n",
    "    3: 'freelance',\n",
    "    4: 'others',\n",
    "    0: 'unk'\n",
    "}\n",
    "\n",
    "\n",
    "# map the values of the dictionaries\n",
    "df.status = df.status.map(status_values)\n",
    "df.home = df.home.map(home_values)\n",
    "df.marital = df.marital.map(marital_values)\n",
    "df.records = df.records.map(records_values)\n",
    "df.job = df.job.map(job_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c351eafc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>status</th>\n",
       "      <th>seniority</th>\n",
       "      <th>home</th>\n",
       "      <th>time</th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>records</th>\n",
       "      <th>job</th>\n",
       "      <th>expenses</th>\n",
       "      <th>income</th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>amount</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ok</td>\n",
       "      <td>9</td>\n",
       "      <td>rent</td>\n",
       "      <td>60</td>\n",
       "      <td>30</td>\n",
       "      <td>married</td>\n",
       "      <td>no</td>\n",
       "      <td>freelance</td>\n",
       "      <td>73</td>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>800</td>\n",
       "      <td>846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ok</td>\n",
       "      <td>17</td>\n",
       "      <td>rent</td>\n",
       "      <td>60</td>\n",
       "      <td>58</td>\n",
       "      <td>widow</td>\n",
       "      <td>no</td>\n",
       "      <td>fixed</td>\n",
       "      <td>48</td>\n",
       "      <td>131</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>default</td>\n",
       "      <td>10</td>\n",
       "      <td>owner</td>\n",
       "      <td>36</td>\n",
       "      <td>46</td>\n",
       "      <td>married</td>\n",
       "      <td>yes</td>\n",
       "      <td>freelance</td>\n",
       "      <td>90</td>\n",
       "      <td>200</td>\n",
       "      <td>3000</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>2985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ok</td>\n",
       "      <td>0</td>\n",
       "      <td>rent</td>\n",
       "      <td>60</td>\n",
       "      <td>24</td>\n",
       "      <td>single</td>\n",
       "      <td>no</td>\n",
       "      <td>fixed</td>\n",
       "      <td>63</td>\n",
       "      <td>182</td>\n",
       "      <td>2500</td>\n",
       "      <td>0</td>\n",
       "      <td>900</td>\n",
       "      <td>1325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ok</td>\n",
       "      <td>0</td>\n",
       "      <td>rent</td>\n",
       "      <td>36</td>\n",
       "      <td>26</td>\n",
       "      <td>single</td>\n",
       "      <td>no</td>\n",
       "      <td>fixed</td>\n",
       "      <td>46</td>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>310</td>\n",
       "      <td>910</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    status  seniority   home  time  age  marital records        job  expenses  \\\n",
       "0       ok          9   rent    60   30  married      no  freelance        73   \n",
       "1       ok         17   rent    60   58    widow      no      fixed        48   \n",
       "2  default         10  owner    36   46  married     yes  freelance        90   \n",
       "3       ok          0   rent    60   24   single      no      fixed        63   \n",
       "4       ok          0   rent    36   26   single      no      fixed        46   \n",
       "\n",
       "   income  assets  debt  amount  price  \n",
       "0     129       0     0     800    846  \n",
       "1     131       0     0    1000   1658  \n",
       "2     200    3000     0    2000   2985  \n",
       "3     182    2500     0     900   1325  \n",
       "4     107       0     0     310    910  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "321ca8f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seniority</th>\n",
       "      <th>time</th>\n",
       "      <th>age</th>\n",
       "      <th>expenses</th>\n",
       "      <th>income</th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>amount</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>8.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>763317.0</td>\n",
       "      <td>1060341.0</td>\n",
       "      <td>404382.0</td>\n",
       "      <td>1039.0</td>\n",
       "      <td>1463.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>8703625.0</td>\n",
       "      <td>10217569.0</td>\n",
       "      <td>6344253.0</td>\n",
       "      <td>475.0</td>\n",
       "      <td>628.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>105.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>1118.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>1400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>12.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>6000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1300.0</td>\n",
       "      <td>1692.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>48.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>99999999.0</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>11140.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       seniority    time     age  expenses      income      assets  \\\n",
       "count     4455.0  4455.0  4455.0    4455.0      4455.0      4455.0   \n",
       "mean         8.0    46.0    37.0      56.0    763317.0   1060341.0   \n",
       "std          8.0    15.0    11.0      20.0   8703625.0  10217569.0   \n",
       "min          0.0     6.0    18.0      35.0         0.0         0.0   \n",
       "25%          2.0    36.0    28.0      35.0        80.0         0.0   \n",
       "50%          5.0    48.0    36.0      51.0       120.0      3500.0   \n",
       "75%         12.0    60.0    45.0      72.0       166.0      6000.0   \n",
       "max         48.0    72.0    68.0     180.0  99999999.0  99999999.0   \n",
       "\n",
       "             debt  amount    price  \n",
       "count      4455.0  4455.0   4455.0  \n",
       "mean     404382.0  1039.0   1463.0  \n",
       "std     6344253.0   475.0    628.0  \n",
       "min           0.0   100.0    105.0  \n",
       "25%           0.0   700.0   1118.0  \n",
       "50%           0.0  1000.0   1400.0  \n",
       "75%           0.0  1300.0   1692.0  \n",
       "max    99999999.0  5000.0  11140.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace missing values\n",
    "# have a look at some statistical values of the numerical values\n",
    "df.describe().round()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de01293f",
   "metadata": {},
   "source": [
    "* We see that ```Income```, ```assets``` and ```debt``` have max value ```99999999.0```\n",
    "* Replace this number with  ```nan```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3ccac244",
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in [\"income\", \"assets\", \"debt\"]:\n",
    "    df[c] = df[c].replace(to_replace=99999999, value= np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c26d57be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seniority</th>\n",
       "      <th>time</th>\n",
       "      <th>age</th>\n",
       "      <th>expenses</th>\n",
       "      <th>income</th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>amount</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4421.0</td>\n",
       "      <td>4408.0</td>\n",
       "      <td>4437.0</td>\n",
       "      <td>4455.0</td>\n",
       "      <td>4455.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>8.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>131.0</td>\n",
       "      <td>5403.0</td>\n",
       "      <td>343.0</td>\n",
       "      <td>1039.0</td>\n",
       "      <td>1463.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>11573.0</td>\n",
       "      <td>1246.0</td>\n",
       "      <td>475.0</td>\n",
       "      <td>628.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>105.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>28.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>1118.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>3000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>1400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>12.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>6000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1300.0</td>\n",
       "      <td>1692.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>48.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>959.0</td>\n",
       "      <td>300000.0</td>\n",
       "      <td>30000.0</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>11140.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       seniority    time     age  expenses  income    assets     debt  amount  \\\n",
       "count     4455.0  4455.0  4455.0    4455.0  4421.0    4408.0   4437.0  4455.0   \n",
       "mean         8.0    46.0    37.0      56.0   131.0    5403.0    343.0  1039.0   \n",
       "std          8.0    15.0    11.0      20.0    86.0   11573.0   1246.0   475.0   \n",
       "min          0.0     6.0    18.0      35.0     0.0       0.0      0.0   100.0   \n",
       "25%          2.0    36.0    28.0      35.0    80.0       0.0      0.0   700.0   \n",
       "50%          5.0    48.0    36.0      51.0   120.0    3000.0      0.0  1000.0   \n",
       "75%         12.0    60.0    45.0      72.0   165.0    6000.0      0.0  1300.0   \n",
       "max         48.0    72.0    68.0     180.0   959.0  300000.0  30000.0  5000.0   \n",
       "\n",
       "         price  \n",
       "count   4455.0  \n",
       "mean    1463.0  \n",
       "std      628.0  \n",
       "min      105.0  \n",
       "25%     1118.0  \n",
       "50%     1400.0  \n",
       "75%     1692.0  \n",
       "max    11140.0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().round()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d34d65d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ok         3200\n",
       "default    1254\n",
       "unk           1\n",
       "Name: status, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the status varianle again\n",
    "df.status.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8597c2e",
   "metadata": {},
   "source": [
    "* The row with status = 1 (unknown) is useless for us\n",
    "* We will remove it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "436f9381",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df.status != \"unk\"].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3196e02a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ok         3200\n",
       "default    1254\n",
       "Name: status, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.status.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "306d3460",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do train / validation / test split\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=11)\n",
    "df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d2ca546d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.reset_index(drop=True)\n",
    "df_val = df_val.reset_index(drop=True)\n",
    "df_test = df_test.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0265d061",
   "metadata": {},
   "source": [
    "* define y as status\n",
    " * set the default values to 1 and the ok values to 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7f9e8293",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = (df_train.status == \"default\").astype(int)\n",
    "y_val = (df_val.status == \"default\").astype(int)\n",
    "y_test = (df_test.status == \"default\").astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "44fbf2d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove status column from dataframe\n",
    "del df_train[\"status\"]\n",
    "del df_val[\"status\"]\n",
    "del df_test[\"status\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1d8757a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>seniority</th>\n",
       "      <th>home</th>\n",
       "      <th>time</th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>records</th>\n",
       "      <th>job</th>\n",
       "      <th>expenses</th>\n",
       "      <th>income</th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>amount</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>owner</td>\n",
       "      <td>36</td>\n",
       "      <td>36</td>\n",
       "      <td>married</td>\n",
       "      <td>no</td>\n",
       "      <td>freelance</td>\n",
       "      <td>75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>parents</td>\n",
       "      <td>48</td>\n",
       "      <td>32</td>\n",
       "      <td>single</td>\n",
       "      <td>yes</td>\n",
       "      <td>fixed</td>\n",
       "      <td>35</td>\n",
       "      <td>85.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1100</td>\n",
       "      <td>1330</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>parents</td>\n",
       "      <td>48</td>\n",
       "      <td>40</td>\n",
       "      <td>married</td>\n",
       "      <td>no</td>\n",
       "      <td>fixed</td>\n",
       "      <td>75</td>\n",
       "      <td>121.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1320</td>\n",
       "      <td>1600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>parents</td>\n",
       "      <td>48</td>\n",
       "      <td>23</td>\n",
       "      <td>single</td>\n",
       "      <td>no</td>\n",
       "      <td>partime</td>\n",
       "      <td>35</td>\n",
       "      <td>72.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1078</td>\n",
       "      <td>1079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>owner</td>\n",
       "      <td>36</td>\n",
       "      <td>46</td>\n",
       "      <td>married</td>\n",
       "      <td>no</td>\n",
       "      <td>freelance</td>\n",
       "      <td>60</td>\n",
       "      <td>100.0</td>\n",
       "      <td>4000.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1100</td>\n",
       "      <td>1897</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   seniority     home  time  age  marital records        job  expenses  \\\n",
       "0         10    owner    36   36  married      no  freelance        75   \n",
       "1          6  parents    48   32   single     yes      fixed        35   \n",
       "2          1  parents    48   40  married      no      fixed        75   \n",
       "3          1  parents    48   23   single      no    partime        35   \n",
       "4          5    owner    36   46  married      no  freelance        60   \n",
       "\n",
       "   income   assets  debt  amount  price  \n",
       "0     0.0  10000.0   0.0    1000   1400  \n",
       "1    85.0      0.0   0.0    1100   1330  \n",
       "2   121.0      0.0   0.0    1320   1600  \n",
       "3    72.0      0.0   0.0    1078   1079  \n",
       "4   100.0   4000.0   0.0    1100   1897  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d4f80a",
   "metadata": {},
   "source": [
    "## Decision Trees\n",
    "\n",
    "* How a decision tree looks like\n",
    "* Training a decision tree\n",
    "* Overfitting\n",
    "* Controlling the size of a tree\n",
    "\n",
    "A Decision Tree is a data structure, that looks like this\n",
    "![decisoin tree](Screenshot_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12d16882",
   "metadata": {},
   "source": [
    "* A decision tree is a sequence of if-else-then \n",
    "* A simple example\n",
    "![example](Screenshot_3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2343051c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def asses_risk(client):\n",
    "    if client[\"records\"] == \"yes\":\n",
    "        if client[\"job\"] == \"parttime\":\n",
    "            return \"default\"\n",
    "        else:\n",
    "            return \"ok\"\n",
    "    else:\n",
    "        if client[\"assets\"] > 6000:\n",
    "            return \"ok\"\n",
    "        else:\n",
    "            return \"default\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "eef57d56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'seniority': 10,\n",
       " 'home': 'owner',\n",
       " 'time': 36,\n",
       " 'age': 36,\n",
       " 'marital': 'married',\n",
       " 'records': 'no',\n",
       " 'job': 'freelance',\n",
       " 'expenses': 75,\n",
       " 'income': 0.0,\n",
       " 'assets': 10000.0,\n",
       " 'debt': 0.0,\n",
       " 'amount': 1000,\n",
       " 'price': 1400}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take first entry as example\n",
    "xi = df_train.iloc[0].to_dict()\n",
    "xi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "be9fb790",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ok'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asses_risk(xi)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96122e10",
   "metadata": {},
   "source": [
    "* This if-then-eñse rules can be learned from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1535ddfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "350bb1aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'seniority': 10,\n",
       "  'home': 'owner',\n",
       "  'time': 36,\n",
       "  'age': 36,\n",
       "  'marital': 'married',\n",
       "  'records': 'no',\n",
       "  'job': 'freelance',\n",
       "  'expenses': 75,\n",
       "  'income': 0.0,\n",
       "  'assets': 10000.0,\n",
       "  'debt': 0.0,\n",
       "  'amount': 1000,\n",
       "  'price': 1400},\n",
       " {'seniority': 6,\n",
       "  'home': 'parents',\n",
       "  'time': 48,\n",
       "  'age': 32,\n",
       "  'marital': 'single',\n",
       "  'records': 'yes',\n",
       "  'job': 'fixed',\n",
       "  'expenses': 35,\n",
       "  'income': 85.0,\n",
       "  'assets': 0.0,\n",
       "  'debt': 0.0,\n",
       "  'amount': 1100,\n",
       "  'price': 1330}]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dicts = df_train.fillna(0).to_dict(orient=\"records\")\n",
    "val_dicts = df_val.fillna(0).to_dict(orient=\"records\")\n",
    "train_dicts[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5fc38f06",
   "metadata": {},
   "outputs": [],
   "source": [
    "dv = DictVectorizer(sparse=False)\n",
    "X_train = dv.fit_transform(train_dicts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4ed33971",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier()"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train the decision tree\n",
    "dt = DecisionTreeClassifier()\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ccc8fdfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# test the decision tree\n",
    "X_val = dv.transform(val_dicts)\n",
    "y_pred = dt.predict_proba(X_val)[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f0951b97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6456431183550896"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695a1c44",
   "metadata": {},
   "source": [
    "* The score is not very high, look at the roc_auc_score for the train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "89f43b30",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_train = dt.predict_proba(X_train)[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "5485fd9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train, y_pred_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f044c32",
   "metadata": {},
   "source": [
    "* Our model is overfitting\n",
    "* It memorizes the training data, but does not generalize\n",
    "* This happens, because we did not restrict the tree, but it can learn very specific conditions\n",
    "* One possibility to avoid this is to restrict the depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "753e88db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "roc_auc_score on validation data: 0.7389079944782155\n",
      "roc_auc_score on training data: 0.7761016984958594\n"
     ]
    }
   ],
   "source": [
    "# retrain the model, with max depth restriced to 3\n",
    "dt = DecisionTreeClassifier(max_depth=3)\n",
    "dt.fit(X_train, y_train)\n",
    "y_pred = dt.predict_proba(X_val)[:,1]\n",
    "ras = roc_auc_score(y_val, y_pred)\n",
    "print(f\"roc_auc_score on validation data: {ras}\")\n",
    "\n",
    "y_pred_train = dt.predict_proba(X_train)[:,1]\n",
    "ras_train = roc_auc_score(y_train, y_pred_train)\n",
    "print(f\"roc_auc_score on training data: {ras_train}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb102c58",
   "metadata": {},
   "source": [
    "* The performance of our model on validation data is much better\n",
    "* Visualize the tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "de7c37c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- feature_25 <= 0.50\n",
      "|   |--- feature_27 <= 6.50\n",
      "|   |   |--- feature_1 <= 862.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |   |--- feature_1 >  862.50\n",
      "|   |   |   |--- class: 1\n",
      "|   |--- feature_27 >  6.50\n",
      "|   |   |--- feature_12 <= 103.50\n",
      "|   |   |   |--- class: 1\n",
      "|   |   |--- feature_12 >  103.50\n",
      "|   |   |   |--- class: 0\n",
      "|--- feature_25 >  0.50\n",
      "|   |--- feature_16 <= 0.50\n",
      "|   |   |--- feature_12 <= 74.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |   |--- feature_12 >  74.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |--- feature_16 >  0.50\n",
      "|   |   |--- feature_2 <= 8750.00\n",
      "|   |   |   |--- class: 1\n",
      "|   |   |--- feature_2 >  8750.00\n",
      "|   |   |   |--- class: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import export_text\n",
    "\n",
    "print(export_text(dt))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb27f869",
   "metadata": {},
   "source": [
    "* To know what the features are use ```dv.get_feature_names()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "82db8f54",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['age',\n",
       " 'amount',\n",
       " 'assets',\n",
       " 'debt',\n",
       " 'expenses',\n",
       " 'home=ignore',\n",
       " 'home=other',\n",
       " 'home=owner',\n",
       " 'home=parents',\n",
       " 'home=private',\n",
       " 'home=rent',\n",
       " 'home=unk',\n",
       " 'income',\n",
       " 'job=fixed',\n",
       " 'job=freelance',\n",
       " 'job=others',\n",
       " 'job=partime',\n",
       " 'job=unk',\n",
       " 'marital=divorced',\n",
       " 'marital=married',\n",
       " 'marital=separated',\n",
       " 'marital=single',\n",
       " 'marital=unk',\n",
       " 'marital=widow',\n",
       " 'price',\n",
       " 'records=no',\n",
       " 'records=yes',\n",
       " 'seniority',\n",
       " 'time']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dv.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3133148e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- records=no <= 0.50\n",
      "|   |--- seniority <= 6.50\n",
      "|   |   |--- amount <= 862.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |   |--- amount >  862.50\n",
      "|   |   |   |--- class: 1\n",
      "|   |--- seniority >  6.50\n",
      "|   |   |--- income <= 103.50\n",
      "|   |   |   |--- class: 1\n",
      "|   |   |--- income >  103.50\n",
      "|   |   |   |--- class: 0\n",
      "|--- records=no >  0.50\n",
      "|   |--- job=partime <= 0.50\n",
      "|   |   |--- income <= 74.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |   |--- income >  74.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |--- job=partime >  0.50\n",
      "|   |   |--- assets <= 8750.00\n",
      "|   |   |   |--- class: 1\n",
      "|   |   |--- assets >  8750.00\n",
      "|   |   |   |--- class: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(export_text(dt, feature_names=dv.get_feature_names()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bccd0e98",
   "metadata": {},
   "source": [
    "## Decision Tree Algorithm Learning\n",
    "* Finding the best plit for one column\n",
    "* Finding the best split for the entire dataset\n",
    "* Stopping criteria\n",
    "* Decision tree learning algorithm\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f8b9d3a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "1    2000  default\n",
       "2       0  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "6    9000       ok\n",
       "7    3000  default"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# simple dataset\n",
    "data = [\n",
    "    [8000, 'default'],\n",
    "    [2000, 'default'],\n",
    "    [   0, 'default'],\n",
    "    [5000, 'ok'],\n",
    "    [5000, 'ok'],\n",
    "    [4000, 'ok'],\n",
    "    [9000, 'ok'],\n",
    "    [3000, 'default'],\n",
    "]\n",
    "\n",
    "# transform to dataframe\n",
    "df_example = pd.DataFrame(data, columns=[\"assets\", \"status\"])\n",
    "df_example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4bdb5b2",
   "metadata": {},
   "source": [
    " * Train a decision tree using the \"assets\" column\n",
    " * We then have a model that decides the status only on the value of \"assets\"\n",
    " * When assets > threshold, the default status will be ok, else it will be default\n",
    " * We look for the best threshold\n",
    " * I.e. we split the dataset in two sets \n",
    "     * 1: assets > threshold\n",
    "     * 2: assets >= threshold\n",
    " * What are possible values for the threshold?\n",
    "     * Look at the dataset, sorted for values of asset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f0b84400",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "2       0  default\n",
       "1    2000  default\n",
       "7    3000  default\n",
       "5    4000       ok\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "0    8000  default\n",
       "6    9000       ok"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_example.sort_values(\"assets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c47253",
   "metadata": {},
   "source": [
    "* We could then split the data at ```asset = 2000```, ```asset = 3000```, ```assets = 4000```, ```assets = 5000``` or , ```assets = 8000```.\n",
    "* Other values would either lead to the same results or to an empty set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "50ccb684",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7d9276f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "threshold: 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "2       0  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "1    2000  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "6    9000       ok\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.571429\n",
      "default    0.428571\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 2000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "1    2000  default\n",
       "2       0  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "6    9000       ok\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.666667\n",
      "default    0.333333\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 3000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "1    2000  default\n",
       "2       0  default\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "6    9000       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.8\n",
      "default    0.2\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 4000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "1    2000  default\n",
       "2       0  default\n",
       "5    4000       ok\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.75\n",
      "ok         0.25\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "6    9000       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.75\n",
      "default    0.25\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 5000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "1    2000  default\n",
       "2       0  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.5\n",
      "ok         0.5\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "6    9000       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.5\n",
      "ok         0.5\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 8000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets   status\n",
       "0    8000  default\n",
       "1    2000  default\n",
       "2       0  default\n",
       "3    5000       ok\n",
       "4    5000       ok\n",
       "5    4000       ok\n",
       "7    3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.571429\n",
      "ok         0.428571\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets status\n",
       "6    9000     ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok    1.0\n",
      "Name: status, dtype: float64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Ts = [0, 2000, 3000, 4000, 5000, 8000]\n",
    "\n",
    "for T in Ts:\n",
    "    \n",
    "    print(f\"threshold: {T}\")\n",
    "    df_left = df_example[df_example.assets <= T]\n",
    "    df_right = df_example[df_example.assets > T]\n",
    "    \n",
    "    display(df_left)\n",
    "    print(df_left.status.value_counts(normalize=True))\n",
    "    display(df_right)\n",
    "    print(df_right.status.value_counts(normalize=True))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e188805",
   "metadata": {},
   "source": [
    "* Now we want to find the best split\n",
    "    * To evaluate the split, we can look at the \"Misclassification Rate = Impurity\":\n",
    "        * For df_left predict every sample as \"default\"\n",
    "        * For df_right predict every sample as \"ok\"\n",
    "        * Count how many errors are made on each side\n",
    "        * This is the misclassification rate (e.g. T=4, misclassification rate left: 1/4 = 25%, misclassification rate right: 1/4 = 24%) \n",
    "        * Take the (weighted) average of both misclassification rate (For sake of simplicity in this example we will take the normal average)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38d5442",
   "metadata": {},
   "source": [
    "T   |  Decision Left | Impurity Left | decision Right | Impurity Right | Average Impurity\n",
    ":--:|:--------------:|:-------------:|:--------------:|:--------------:|:----------------:\n",
    "0   | Default        | 0%            | OK             | 93%            | 21%   \n",
    "2000| Default        | 0%            | OK             | 33%            | 16% \n",
    "3000| Default        | 0%            | OK             | 20%            | 10%\n",
    "4000| Default        | 25%           | OK             | 25%            | 25%\n",
    "5000| Default        | 50%           | OK             | 50%            | 50%\n",
    "8000| Default        | 42%           | OK             | 0%             | 21%  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5873fbd8",
   "metadata": {},
   "source": [
    "* Threshold = 3000 gives the lowest impurity\n",
    "* I.e., this is the best split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1b81e114",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok\n",
       "7    3000  2000  default"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add another feature to the data\n",
    "# simple dataset\n",
    "data = [\n",
    "    [8000, 3000, 'default'],\n",
    "    [2000, 1000, 'default'],\n",
    "    [   0, 1000, 'default'],\n",
    "    [5000, 1000, 'ok'],\n",
    "    [5000, 1000, 'ok'],\n",
    "    [4000, 1000, 'ok'],\n",
    "    [9000, 500, 'ok'],\n",
    "    [3000, 2000, 'default'],\n",
    "]\n",
    "\n",
    "# transform to dataframe\n",
    "df_example = pd.DataFrame(data, columns=[\"assets\", \"debt\", \"status\"])\n",
    "df_example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1aa3db8",
   "metadata": {},
   "source": [
    "* Split the \"debt\" feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "fcb3cc7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "6    9000   500       ok\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "7    3000  2000  default\n",
       "0    8000  3000  default"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_example.sort_values(\"debt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12252b2b",
   "metadata": {},
   "source": [
    "* Possible thresholds: ```debt = 500```, ```debt = 1000```, ```debt = 2000```\n",
    "* Do the same as before, but generalize a bit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "85886806",
   "metadata": {},
   "outputs": [],
   "source": [
    "Ts = [500, 1000, 2000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "03d6cbac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "####################\n",
      "assets\n",
      "threshold: 0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "2       0  1000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "1    2000  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.571429\n",
      "default    0.428571\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 2000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.666667\n",
      "default    0.333333\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 3000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.8\n",
      "default    0.2\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 4000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "5    4000  1000       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.75\n",
      "ok         0.25\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "6    9000   500       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.75\n",
      "default    0.25\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 5000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.5\n",
      "ok         0.5\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "6    9000   500       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.5\n",
      "ok         0.5\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 8000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.571429\n",
      "ok         0.428571\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt status\n",
       "6    9000   500     ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok    1.0\n",
      "Name: status, dtype: float64\n",
      "\n",
      "####################\n",
      "debt\n",
      "threshold: 500\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt status\n",
       "6    9000   500     ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok    1.0\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    0.571429\n",
      "ok         0.428571\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 1000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.666667\n",
      "default    0.333333\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n",
      "\n",
      "threshold: 2000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4000</td>\n",
       "      <td>1000</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9000</td>\n",
       "      <td>500</td>\n",
       "      <td>ok</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3000</td>\n",
       "      <td>2000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "1    2000  1000  default\n",
       "2       0  1000  default\n",
       "3    5000  1000       ok\n",
       "4    5000  1000       ok\n",
       "5    4000  1000       ok\n",
       "6    9000   500       ok\n",
       "7    3000  2000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ok         0.571429\n",
      "default    0.428571\n",
      "Name: status, dtype: float64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>debt</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>3000</td>\n",
       "      <td>default</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   assets  debt   status\n",
       "0    8000  3000  default"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default    1.0\n",
      "Name: status, dtype: float64\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# more general, put all thresholds in a dictionary\n",
    "thresholds = {\n",
    "    \"assets\": [0, 2000, 3000, 4000, 5000, 8000],\n",
    "    \"debt\": [500, 1000, 2000]\n",
    "} \n",
    "\n",
    "for feature, Ts in thresholds.items():\n",
    "    print(20*'#')\n",
    "    print(feature)\n",
    "    for T in Ts:\n",
    "    \n",
    "        print(f\"threshold: {T}\")\n",
    "        df_left = df_example[df_example[feature] <= T]\n",
    "        df_right = df_example[df_example[feature] > T]\n",
    "    \n",
    "        display(df_left)\n",
    "        print(df_left.status.value_counts(normalize=True))\n",
    "        display(df_right)\n",
    "        print(df_right.status.value_counts(normalize=True))\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00483d1d",
   "metadata": {},
   "source": [
    "T   |  Decision Left | Impurity Left | decision Right | Impurity Right | Average Impurity\n",
    ":--:|:--------------:|:-------------:|:--------------:|:--------------:|:----------------:\n",
    "500 | OK             | 0%            | Default        | 43%            | 21%   \n",
    "1000| OK             | 33%           | Default        | 0%             | 16% \n",
    "2000| OK             | 43%           | Default        | 0%             | 21%"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae7dac3",
   "metadata": {},
   "source": [
    "* We see that \"debt\" is not as good to split the data as \"assets\", assets = 3000 still is the best split "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "176d46e7",
   "metadata": {},
   "source": [
    "![split algorthm](Screenshot_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c31d82c",
   "metadata": {},
   "source": [
    "* Repeat this process for each split with the remaining features\n",
    "* We need some stopping criteria, when we stop splitting:\n",
    "    * Split is already pure\n",
    "    * Tree reached depth limit\n",
    "    * Group to small to split\n",
    "    \n",
    "* Documentation: https://scikit-learn.org/stable/modules/tree.html\n",
    "* There are other criteria to find the best split, e.g. Giny, Entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c73f8af",
   "metadata": {},
   "source": [
    "![algorithm summary](Screenshot_5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cddd241",
   "metadata": {},
   "source": [
    "## Decision Trees Parameter Tuning\n",
    "\n",
    "* selecting ```max_depth```: The maximal depth of the tree\n",
    "* selecting ```min_samples_leaf```: This is the minimum number of samples in the leaf node to continue splitting\n",
    "* Parameter tuning means, select paramters, such that the chosen metric on the validation data is maximized, in this case the metric is auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "092ad31d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "depth: 1   , auc: 0.606\n",
      "depth: 2   , auc: 0.669\n",
      "depth: 3   , auc: 0.739\n",
      "depth: 4   , auc: 0.761\n",
      "depth: 5   , auc: 0.766\n",
      "depth: 6   , auc: 0.748\n",
      "depth: 10  , auc: 0.698\n",
      "depth: 15  , auc: 0.654\n",
      "depth: 20  , auc: 0.641\n",
      "depth: None, auc: 0.652\n"
     ]
    }
   ],
   "source": [
    "# first max_depth\n",
    "for d in [1, 2, 3, 4, 5, 6, 10, 15, 20, None]:\n",
    "    dt = DecisionTreeClassifier(max_depth=d)\n",
    "    dt.fit(X_train, y_train)\n",
    "    y_pred = dt.predict_proba(X_val)[:,1]\n",
    "    auc = roc_auc_score(y_val, y_pred)\n",
    "    print(f\"depth: {str(d):4s}, auc: {auc:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29f67525",
   "metadata": {},
   "source": [
    "* Best depths values are between 4, 5 and 6\n",
    "* Look at min_samples_leaf for these depths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a7c30969",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "depth: 4   , min_sample_leaf:    1, auc: 0.761\n",
      "depth: 4   , min_sample_leaf:    2, auc: 0.761\n",
      "depth: 4   , min_sample_leaf:    5, auc: 0.761\n",
      "depth: 4   , min_sample_leaf:   10, auc: 0.761\n",
      "depth: 4   , min_sample_leaf:   15, auc: 0.764\n",
      "depth: 4   , min_sample_leaf:   20, auc: 0.761\n",
      "depth: 4   , min_sample_leaf:  100, auc: 0.756\n",
      "depth: 4   , min_sample_leaf:  200, auc: 0.747\n",
      "depth: 4   , min_sample_leaf:  500, auc: 0.680\n",
      "depth: 5   , min_sample_leaf:    1, auc: 0.767\n",
      "depth: 5   , min_sample_leaf:    2, auc: 0.766\n",
      "depth: 5   , min_sample_leaf:    5, auc: 0.768\n",
      "depth: 5   , min_sample_leaf:   10, auc: 0.762\n",
      "depth: 5   , min_sample_leaf:   15, auc: 0.772\n",
      "depth: 5   , min_sample_leaf:   20, auc: 0.774\n",
      "depth: 5   , min_sample_leaf:  100, auc: 0.763\n",
      "depth: 5   , min_sample_leaf:  200, auc: 0.759\n",
      "depth: 5   , min_sample_leaf:  500, auc: 0.680\n",
      "depth: 6   , min_sample_leaf:    1, auc: 0.749\n",
      "depth: 6   , min_sample_leaf:    2, auc: 0.763\n",
      "depth: 6   , min_sample_leaf:    5, auc: 0.762\n",
      "depth: 6   , min_sample_leaf:   10, auc: 0.778\n",
      "depth: 6   , min_sample_leaf:   15, auc: 0.785\n",
      "depth: 6   , min_sample_leaf:   20, auc: 0.774\n",
      "depth: 6   , min_sample_leaf:  100, auc: 0.776\n",
      "depth: 6   , min_sample_leaf:  200, auc: 0.768\n",
      "depth: 6   , min_sample_leaf:  500, auc: 0.680\n"
     ]
    }
   ],
   "source": [
    "for d in [4, 5, 6]:\n",
    "    for s in [1, 2, 5, 10, 15, 20, 100, 200, 500]:\n",
    "        dt = DecisionTreeClassifier(max_depth=d, min_samples_leaf=s)\n",
    "        dt.fit(X_train, y_train)\n",
    "        y_pred = dt.predict_proba(X_val)[:,1]\n",
    "        auc = roc_auc_score(y_val, y_pred)\n",
    "        print(f\"depth: {str(d):4s}, min_sample_leaf: {s:4}, auc: {auc:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a8819e",
   "metadata": {},
   "source": [
    "* For easer readability put the results in a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "ac3100a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "\n",
    "for d in [4, 5, 6]:\n",
    "    for s in [1, 2, 5, 10, 15, 20, 100, 200, 500]:\n",
    "        dt = DecisionTreeClassifier(max_depth=d, min_samples_leaf=s)\n",
    "        dt.fit(X_train, y_train)\n",
    "        y_pred = dt.predict_proba(X_val)[:,1]\n",
    "        auc = roc_auc_score(y_val, y_pred)\n",
    "        \n",
    "        scores.append((d, s, auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "19401aff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_samples_leaf</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.761283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>0.761283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.761283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.761283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>0.763726</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  min_samples_leaf       auc\n",
       "0          4                 1  0.761283\n",
       "1          4                 2  0.761283\n",
       "2          4                 5  0.761283\n",
       "3          4                10  0.761283\n",
       "4          4                15  0.763726"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [\"max_depth\", \"min_samples_leaf\", \"auc\"]\n",
    "df_scores = pd.DataFrame(scores, columns=columns)\n",
    "df_scores.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "4a9ae2a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_samples_leaf</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>0.785250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>0.778299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>6</td>\n",
       "      <td>100</td>\n",
       "      <td>0.776434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5</td>\n",
       "      <td>20</td>\n",
       "      <td>0.774028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "      <td>0.773337</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    max_depth  min_samples_leaf       auc\n",
       "22          6                15  0.785250\n",
       "21          6                10  0.778299\n",
       "24          6               100  0.776434\n",
       "14          5                20  0.774028\n",
       "23          6                20  0.773337"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scores.sort_values(\"auc\", ascending=False).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96795237",
   "metadata": {},
   "source": [
    "* Look at this table in a different way: columns: max_depth, rows=min_samples_leaf, values=auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7655ccbb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">auc</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min_samples_leaf</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.761</td>\n",
       "      <td>0.766</td>\n",
       "      <td>0.750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.761</td>\n",
       "      <td>0.767</td>\n",
       "      <td>0.758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.761</td>\n",
       "      <td>0.768</td>\n",
       "      <td>0.759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.761</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.764</td>\n",
       "      <td>0.773</td>\n",
       "      <td>0.785</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.761</td>\n",
       "      <td>0.774</td>\n",
       "      <td>0.773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>0.756</td>\n",
       "      <td>0.763</td>\n",
       "      <td>0.776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>0.747</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500</th>\n",
       "      <td>0.680</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    auc              \n",
       "max_depth             4      5      6\n",
       "min_samples_leaf                     \n",
       "1                 0.761  0.766  0.750\n",
       "2                 0.761  0.767  0.758\n",
       "5                 0.761  0.768  0.759\n",
       "10                0.761  0.762  0.778\n",
       "15                0.764  0.773  0.785\n",
       "20                0.761  0.774  0.773\n",
       "100               0.756  0.763  0.776\n",
       "200               0.747  0.759  0.768\n",
       "500               0.680  0.680  0.680"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_scores_pivot = df_scores.pivot(index=\"min_samples_leaf\", columns=[\"max_depth\"], values=[\"auc\"])\n",
    "df_scores_pivot.round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d14f1d31",
   "metadata": {},
   "source": [
    "* Transform this table into a heat map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "42c0e4f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEHCAYAAABcCaZFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABXC0lEQVR4nO3dd3wcxfn48c9zdzpLsopl2ZJ7t3FvuIKJeyOAbQKmB2MSQguk0EMgkF+oaZCQLzhgOhgDobuCCy644957lWRbsi3Zanf7/P7Y80mydJYEutOdNO+89pXbndnd2eU8N5qdnUdUFcMwDCOyOaq7AIZhGMaPZypzwzCMGsBU5oZhGDWAqcwNwzBqAFOZG4Zh1ACu6i7A+Zx54XYz1CbIJCW1uotQKxTMWVLdRajxEl//Wn7sMQqP7a5wnRPVoM2PPl9VCuvK3DAMI6Qsb3WX4AczlblhGMZZalV3CX4wU5kbhmGcZZnK3DAMI+KpaZkbhmHUAKZlbhiGUQN4C6u7BD+YqcwNwzDOMt0sNZOjZWfcgyeCOPBsWoJn1ewS6a7eI3F17GeviAOp35jcKfdB/hlwx+AecROO5CaAUjD3Lay0PTjb9SZqwGVI/UbkT3sGK2N/6C8sjCzZcYTnZq3BspQJvdsw+ZLOJdLfWLKFGev3AeC1lD3HTjH//vEkxtbhVG4BT36+gp0ZJxER/jSuHz2aNwDg/eXbmbZiB06HcEn7Jvx2VM9QX1pYcXXtS/T1d4LDQeG3M8mfMa1EunvMRNwDh9krDieOJi3Ivucq9HQ28c+/g+blguVFvV5OP3kXAFI3npg7HsXRIBXrWDpn/vNnOJMT6kurWlXYzSIiY4AXACfwqqo+c076/cANvlUX0AloqKqZIvJb4BeAAhuAW1Q173znM5V5ICK4h1xH/icvoDlZRF/7MN7d69HMI/4snjVz8ayZC4CzdTdcvYbbFTngHjwR775NFMyYAg4nuNwAWMcPk//lK7iH31D6nLWM17J4esYqXr5pKKkJMdzw37kMvqApbVMS/XkmXdyJSRd3AmDhtkO88902EmPrAPDcrDVc1K4xf71mEIUeL7mF9hjhlXvSWbD1EB/eMQa3y0lmznn/DdR84iD6pl9z+q8PoplHiXvsJQrXLsU6XNSQKJg1nYJZ0wFw9RhAndE/Q09n+9NPP/t7NOdUicPWufRavJu/58yMadS59Fqif3oteR++GpprCpKqegAqIk7gJWAkcBBYKSKfq+rmonPp88DzvvyXA7/1VeRNgXuAzqqaKyLTgWuBN853TvM6fwCO1FboyQz01DGwvHi2r8TZpnvA/M4L+uLZtspecUfjaNoe7ybfW3+WFwpyAdCsNPREerCLHxE2Hsqkef14mtWPI8rlZHTXFizYdihg/pkb9jGmW0sAcvIKWbPvKBN6twEgyuUkIcb+wZy+cie3DOqE2+UEoH5cdJCvJLw521yAlXEYPXoEvB4KVywgqtfFAfNHDRhGwbL55R7X1esiCpbMAaBgyRxc5zlmxLCsii/n1w/Yqaq7VbUAmAaMO0/+64D3i627gBgRcQGxwOHyTmgq8wAkLgnNzvKva84JJC6p7MyuKJwtu+DducbeN6EBmpuDe+TNRF/3CO7hN/pb5kaRjFO5NEqI9a+nJsSQcSq3zLy5BR6W7kxjRKdmABzMyiEptg6Pfbqca16exROfrSC3wAPAvuPZrNl/lBv/O4dbX/+GjYeOB/9iwpgkNUAzM/zrVuZRJCm57MzuOri69sGzelHRNlXq3vcscY//h6jBP/VvdiQmoScz7SwnM3Ek1AtG8UPLW1jhRURuE5FVxZbbih2pKXCg2PpB37ZSRCQWGAN8DKCqh4C/AvuBI8BJVZ1TXtGrpTIXkVuq47w/WoCoTM7W3bEO7/J3sYjDgSOlOZ71C8l7/ym0sICoPqNDWdKIoJS+nxJgtotvtx+mZ4sG/i4Wr6VsPZLFxL7t+eD2MUS7XUxdvNmflp1bwNu/GMlvRvbkgQ+XUrsjapVxUwPcjqieA/Hu3FSiiyXnqd+Q86c7OP33R6gz7AqcHboFqZxhQK0KL6o6RVX7FFumFDtSWd/kQF/Cy4ElqpoJICJJ2K341kAToK6I3Fhe0aurZf5EoITiv3ZTl24OlC3oNCcLiS9qiUtcPfT0iTLzOjv0xbN9pX/dyjmB5pzASt8LgHfnGhwpLYJZ3IiUmhBL2qkz/vX0U7k0jI8pM++sjfsY07XoHqYmxJCSEEO3ZnYLc2TnZmw5kuVPG9apGSJCt2bJOASyzuQH8UrCm2YdReqn+Ncd9RuiJ8r+ayWq3xAKl5fsYjmbV7NPULhmCc42HQGwTmYhifUBkMT6WKdOBKH0IVZ13SwHgebF1psRuKvkWkp2sYwA9qjqUVUtBP4HXFTeCYNWmYvI+gDLBiDgVH3Ff+0mX9Q5ULags9L3IfVSkIRkcDhxdeiLd/f60hnd0Tibtce7a13RtjOn0OxMpJ59mc7mHbGKPTg1bF2a1Gf/8WwOZeVQ6PEye+N+Bl9Q+i/R7LwCVu89ytCOzfzbGsTH0Cgxlr3H7Idyy3en06ah/eB0aMdmrNxjdyvsO3aKQq9Fkq9FXxt592zDmdIUadAInC67wv5+aemMMXVxXtCdwjXF0tzREB3j/+zqeiHWwb0AeNZ+h/viUXbSxaPwlHXMSFOJlnk5VgLtRaS1iLixK+zPz80kIonAYOCzYpv3AwNEJFZEBBgObCnvhMEczZIKjAayztkuQPj/V1eLggUfUGf8PfbQxM1L0cwjuLpdAoBng92n6GzbC+++zeApKLF7wYIPcI+ZjDidWCePUTD3LV/+nkQNvgaJiaPOuLuxjh4g/9N/hfbawoTL6eChSy/kjrcXYqnFuF5taJeSyIcrdwJwdd92AMzbcpCBbRsR4y75dX1w7IU88vF3FHotmibF8eT4/gCM79Waxz9bwc9emkmU08Gfxw9AAvXf1AaWRe67/6Lu75+xhyYumoV1eB/uIZcBULDgSwCiel+MZ9NqKCga/SOJSdS9+0/2itNJ4bJ5eDbaf4XmfzWN2DsfJeonY9DjGfbQxEhXRUMTVdUjIncDs7GHJk5V1U0icrsv/WVf1gnAHFU9XWzf5SLyEbAG8ADfA1MohwSrL1FEXgNeV9XFZaS9p6rXl3cMM5958Jn5zEPDzGcefFUxn3neuhkVrnOie1waVi2EoLXMVfXW86SVW5EbhmGEnNdT3SX4wcxLQ4ZhGGeZ1/kNwzBqABNpyDAMowYwLXPDMIwawMxnbhiGUQOYlrlhGEYN4DGjWQzDMCKeqnkAakQqh5k4MxRc7ZpUdxGMijB95oZhGDWA6TM3DMOoAUzL3DAMowYwr/MbhmHUAKabxTAMowYw3SyGYRg1gKnMayZHy864B0+0g1NsWoJn1ewS6a7eI3F17GeviAOp35jcKffZsUDdMbhH3IQjuQmgFMx9CyttD852vYkacBlSvxH5057Bytgf+gsLI0t2HOa5GauxVJnQuy2Tf9KlRPobizczY/1ewI7tuefoKeY/eCWJsXU4lVvAk58tZ2fGSQT40/j+9GjRkAemL/ZHIMrOKyQ+Oorpd14a4isLL8423XCPutH+Lq9dSOF3X5ZIjxpwKc6uAwEQcSINmnDmH3dB3mli7vobFOShaodLy5v6OACO1Ba4x04CVxRYFgWz3sQ6vDvUl1a1TDdLDSSCe8h15H/yApqTRfS1D+PdvR4tFv7Ns2YunjVzAXC27oar13B/UGf34Il4922iYMYUcDjB5QbAOn6Y/C9fwT38htBfU5jxWhZPf7mKl28eRmpCDDe8MpvBHZvRNiXRn2fSoM5MGmSHD1y49SDvfLfNH9T5uZmruah9Y/567SUUerzkFtovfDw3cZB//7/NWkNcnagQXlUYEsE95ufkvfcceiqT6MlP4NmxBj1WFJKycNkMCpfNAMDZvidR/cZAnj/4DbnvPA25OSUO6x52DYWLPsW7az3Ott1xD7uGvHeeDs01BUsEt8zNGyMBOFJboScz0FPHwPLi2b4SZ5vuAfM7L+iLZ9sqe8UdjaNpe7ybfNFlLC8U5AKgWWnoifRgFz8ibDx4nOb142hWP44ol5PR3VqyYOvBgPlnbtjHmG4tAcjJK2TN3gwm9G4LQJTLSUKMu0R+VWXOxv2M6d4yeBcRARxN2mJlZqAnjoLlxbt5Ga4OvQPmd3UeiGfTsnKPqwq4ffFB68Si2SeqpsDVyeup+BJmTMs8AIlLQrOLwpdqzgkcjVqXndkVhbNlFwrmT7P3TWiA5ubgHnkzjgZNsTL2U7Bweqk4obVdRnYujRLr+tdTE2LZcPBYmXlzCzws3XmEh3/aB4CDWTkk1a3DY58sY3vaCTo3qc8Dl15YIk7omn1HSY6LpmVyQnAvJMxJfBKafdy/rqcycTRtW3Zmlxtn227kz36rxObo6x8AVTzfz8fz/QIACua+S/R198OIa0GEvDdqQAzQCO5mCWrLXEQ6ishwEYk7Z/uYYJ43aALES3W27o51eJe/i0UcDhwpzfGsX0je+0+hhQVE9RkdypJGhLJuZ6DAy99uO0TP5g38XSxey2LrkSwm9m3PB3eOJdrtZOqiTSX2mbVhr78lb5wjQKRLZ/teeA/uKNHFkvfmn8l77THypv0V14UjcDS/AICoC4dRMPddcv/1Wwrmvkedy34RipIHl2VVfAkzQavMReQe4DPg18BGERlXLPmp8+x3m4isEpFVU5duDlbxyqU5WUh8UlG54uqhp0+UmdfZoS+e7Sv961bOCTTnBFb6XgC8O9fgSGkRzOJGpNSEGNJOFlUa6afO0DA+psy8szbuY0z3VsX2jSUlIZZuzRsAMLJzC7YcLvpLyuO1+GbzQUZ3NZW5Zmch8cn+dUmoj+ZklZnX1aV/qS4WzTlhfziTjXfbahxN2th5uw3C6+ta9G5Z4d8e0UxlXqZfAheq6nhgCPBHEbnXlxYwqrWqTlHVPqraZ/JFnYNYvPOz0vch9VKQhGRwOHF16It39/rSGd3ROJu1x7trXdG2M6fQ7EykXioAzuYdsYo9ODVsXZomsz8zm0NZORR6vMzesI/BHZuWypedV8DqvRkM7djMv61BfAyNEmL9o1aW706jTbEHp8t3p9G6QQKpibHBv5AwZx3ejaN+KpLYABxOnJ0H4Nn+femMdWJwtuiId/vqom1RbnBH+z8723RFj9rPNTTnBI4WHQFwtOqMlZkW7EsJPtWKL2EmmH3mTlXNAVDVvSIyBPhIRFpynso8bKhFwYIPqDP+Hns41+alaOYRXN0uAcCzYREAzra98O7bXKo/vGDBB7jHTEacTqyTxyiY+5Yvf0+iBl+DxMRRZ9zdWEcPkP/pv0J7bWHC5XTw0E/7cMdb87EsZVzvNrRLqceHK3cAcHXf9gDM23KQgW0blegPB3jwp3145KOlFHotmibF8eSEAf60WRv21foHn35qUTD7LaKvewAcgmfdt+ixQ7h6DwXAs2Y+AK4LLsS7eyMUFn2XpW4ida6y22DicODZ9B3e3RsAyP9qKu5RN9ijtTyFFMx4PcQXFgRh2OKuKNEg/cKIyDzgd6q6ttg2FzAVuEFVneUd48wLt4ffz18NI40aV3cRagVr567qLkKNV/cPb/3oRmLuO3+ocJ0Tc+NfwqpRGsyW+c+BEuN3VNUD/FxEXgnieQ3DMH6YCG6ZB60yV9WAA4ZVdUmwzmsYhvGDhWFfeEWZceaGYRhnmZa5YRhGDWAqc8MwjMinXhPQ2TAMI/KZlrlhGEYNEMFzs5jK3DAM4yzLjGYxIpWz3He3jCogbvNPLSJUYTeLb0LBFwAn8KqqPnNO+v3A2cAGLqAT0FBVM0WkHvAq0BV7WrTJqvrd+c5nvmGGYRhnVVFlLiJO4CVgJHAQWCkin6uqf/ZAVX0eeN6X/3Lgt6qa6Ut+AZilqleJiBsod5IhU5kbhmGcVXWjWfoBO1V1N4CITAPGAYGmgr0OeN+XNwH4CTAJQFULgHKDIZhIQ4ZhGGdZWvHl/JoCB4qtH/RtK0VEYoExwMe+TW2Ao8DrIvK9iLwqInXL2rc4U5kbhmGcpVaFl+KxF3zLbcWOVNYkXIF+AS4HlhTrYnEBvYH/U9VewGngofKKbrpZDMMwzqrEaBZVnQJMCZB8EGhebL0ZcDhA3mvxdbEU2/egqi73rX9EBSpz0zI3DMPwUcuq8FKOlUB7EWnte4B5LfD5uZlEJBEYjB2VzS6DahpwQEQu8G0aTuC+dj/TMjcMwzirisaZq6pHRO4GZmMPTZyqqptE5HZf+su+rBOAOap6+pxD/Bp41/dDsBu4pbxzmsr8PBwtO+MePNGONLRpCZ5Vs0uku3qPxNWxn70iDqR+Y3Kn3GcHdnbH4B5xE47kJoBSMPctrLQ9ONv1JmrAZUj9RuRPewYrY3/oLyyMLNl+iOe+WoVlKRP6tGPy4K4l0t9YtIkZa/cAdhDnPUdPMf+Rq0mMrcOp3AKe/OQ7dqafQAT+dOVF9GjRkK2HM/nL58vJL/TicggPX9HfHyu0tnK06op7+PUggmf9IjwrZpRId/Udg6uzL1KTOJDkJuS+dC/knSb6tuegIA/UbpHmv/2kna1hc9yjbkKiotGTx8j/aoqdL5JV4dwsqjoDmHHOtpfPWX8DeKOMfdcCfSpzPlOZByKCe8h15H/yApqTRfS1D+PdvR4tFsvTs2YunjVzAXC27oar13C7Igfcgyfi3beJghlT7LBaLjcA1vHD5H/5Cu7hN5Q+Zy3jtSye/mIFL98ygtSEWG74v5kM7tSMtin1/HkmXdKFSZd0AWDhlgO8s3QLibF1AHjuq5Vc1L4Jf71+MIUeL7mF9j/Ef85ew6+GdmfQBU1ZtO0Q/5y9htd+MSrk1xc2RHCPvJH86X9DszOJvukxvLvWoseLunA9K2fhWTkLAGfbHrguHAV5RY3FvA+eg9ycEod1j55E4YIPsA5ux9l1EFF9x1K45JPQXFOwRPDcLKbPPABHaiv0ZAZ66hhYXjzbV+Js0z1gfucFffH4IpXjjsbRtD3eTb4YHJYXCnIB0Kw09ER6sIsfETYePE7z+vE0qx9PlMvJ6O4tWbDlQMD8M9fvZUz31gDk5BWwZm86E/q0AyDK5SQhxv7BFOB0fqE/X8P4mOBeSJhzNG6DZmWgJ4/a3+Wty3G26xkwv7NjfzxblwdM9x+3fiOsg9sBsPZtwtnhwqoqcvWpuqGJIReylrmIDMIeSL9RVeeE6rw/lMQlodlZ/nXNOYGjUeuyM7uicLbsQsH8afa+CQ3Q3BzcI2/G0aApVsZ+ChZOLxX0ubbLOHWGRolFw2dTE+qy4cCxMvPmFnhYuuMwD19ud2sdzMwhKTaaxz5eyva0LDo3SeaBy/oQ447i/p/25c43vubvs1ZjWcqbvxoTkusJVxJXD83O9K9rdhaOxm3Kzuxy42zdlYJv3i3apkr01b8HVQrXLcS7fiEA1rFDONv1xLtzLc4L+iIJ9YN5GaERwRNtBa1lLiIrin3+JfBvIB54XEQCDrMpPnZz6tJyH+CGVoCQUs7W3bEO7/J3sYjDgSOlOZ71C8l7/ym0sICoPqNDWdKIUNbtlAAhcr/depCeLRr6u1i8lrL1SCYT+3fgg7svI9rtYurCTQB8uGIb913ah9kP/Iz7ftqHJz4575QWtUDFhzw72/bAOrSzRBdL/ntPk/fWE+R9/A+ieg3D0awDAAWzpuLqNYzomx4DdzR4PWUeM6JEcMs8mN0sUcU+3waMVNUngFEUTS5TiqpOUdU+qtpn8kWdg1i889OcLCQ+yb8ucfXQ0yfKzOvs0BfP9pX+dSvnBJpzAit9LwDenWtwpLQIZnEjUmpiLGkniyqN9FOnaZhQdpfIrPV7GdOj6C+j1MRYUhJi6da8IQAju7Zgy2G79fnFmt0M72Lf71FdW7Lx4PFgXUJEsL/LRa1miU9Cc06UmdfZqXQXi/97fyYb7441OBrb/x00M438D/9O3ttP4t2yHOtERjCKH1Lq8VZ4CTfBrMwdIpIkIsmAqOpRAN8QnLD/CbfS9yH1UpCEZHA4cXXoi3f3+tIZ3dE4m7XHu2td0bYzp9DsTKReKgDO5h2xij04NWxdmiaz/3g2hzKzKfR4mb1+H4M7Ni+VLzuvgNV70xnaqZl/W4P4GBol1mXv0ZMALN+VRpuURAAaJsSwao/9XGLF7jRaJMeH4GrCl3VkD5KUiiQ2sL/LHfvj3bm2dEZ3DM5mHfDu/L5oW5QboqL9nx2tumAdPWSvx569r0LUwMvxrF0QxKsIkQhumQezzzwRWI39N56KSCNVTROROMr+uy+8qEXBgg+oM/4ee2ji5qVo5hFc3S4BwLNhEQDOtr3w7ttcqj+8YMEHuMdMRpxOrJPHKJj7li9/T6IGX4PExFFn3N1YRw+Q/+m/QnttYcLldPDQ5f24441vsFQZ17sd7VLr8eFy+6Ha1f3tP+fnbT7AwHaNiXFHldj/wcv68sj0xRR6LZrWj+PJn10EwGPjB/LcVyvxWorb5eCP4weE9sLCjVoUfP0Oda76HTgceDYsRo8fxtVjCACedQsAcLbvjXfvJigs+i5LbCJ1xt9trzgceLYsx9q7EQBXx/64eg0DwLtjDd6Ni0N2SUETwX3mogH6gYN2QntSmVRV3VNe3jMv3B5+P381jDRtVn4m40fTPburuwg1Xuz9U390IzHnd1dUuM6J+/vnYdUoLbebRUQu9v1/nao4oaqeqUhFbhiGEWpqaYWXcFORPvMXff9f24cEGIZR09XwPvNCEXkdaCoiL56bqKr3VH2xDMMwqkEYjlKpqIpU5pcBI4Bh2A80DcMwaqYwbHFXVLmVuaoeA6aJyBZVXVdefsMwjEgV6gEhVakyQxO3ichdQBcg+uxGVZ1c5aUyDMOoDhHcMq/MS0NvA42A0cBC7MgZ2cEolGEYRrWo4Q9Az2qnqleLyDhVfVNE3sOeeN2IYHrUzOAYCs6x11V3EYwKUE/kvjRUmcq80Pf/J0SkK5AGtKryEhmGYVSXyK3LK1WZTxGRJOCP2LHs4oDHglIqwzCMahCOLwNVVIUrc1V91fdxIRBgMmTDMIwIFsGVeYUfgIpIqoi8JiIzfeudReTW4BXNMAwjxKxKLGGmMqNZ3sB+4NnEt74d+E0Vl8cwDKPa1PS5Wc5qoKrT8f0mqaoHiNx3Xw3DMM6hHq3wEm4q8wD0tC/QhAKIyADgZFBKZRiGUR3CsPukoipTmf8OexRLWxFZAjQErgpKqcKEo2Vn3IMn2sEpNi3Bs6rksHpX75G4OtoBhhEHUr8xuVPus2OBumNwj7gJR3ITQCmY+xZW2h6c7XoTNeAypH4j8qc9g5WxP/QXFkaW7D3G8wu3YlnK+K7NmNy3ZNDsN1ftYcbWNAC8arEn8zTzfjWUxOgosvMKeeLrTew6noMgPD6yCz2a1OMfi7bx7e6jRDkdNEuM5YmRXYiPjirr9LXG4u838+zUj7AsiyuHX8StV44qkf76p18zY5Ed+tDjtdhzKI2FU58h81QOD/x9qj/fwfTj3HntT7npsqFs3XOQP78yjYLCQpxOB3/45TV0a98qlJdV5SI4NkWlRrOsEZHBwAXYkYK2qWphObtFLhHcQ64j/5MX0Jwsoq99GO/u9Wix8G+eNXPxrJkLgLN1N1y9hvuDOrsHT8S7bxMFM6aAwwkuNwDW8cPkf/kK7uEBw6DWGl5LeWb+Fv7vygtJjYvmhveXMbhNQ9omx/nz3NynNTf3sSv4hbszeHfNPhJ9FfNzC7dyUasG/PWynhR6LfIK7V6/AS2S+fXF7XE5HLywaDtTV+7h3ks6hP4Cw4TXa/HUf6cz5bG7SU2ux3UPPs+Qvt1o27yxP88t40dwy/gRACxYuYG3v5xPYnxdEuPr8uHfHvYfZ8Rtf2B4vx4A/OPtT7l94lgu6d2FRas38Y+3P2Xqk78J+fVVqQiuzCsSnOLKswtwBXZl3gG43Lct0H6JIvKMiGwVkeO+ZYtvW70qu4IgcaS2Qk9moKeOgeXFs30lzjbdA+Z3XtAXz7ZV9oo7GkfT9ng3LbHXLS8U5AKgWWnoCfPWJcDGtJM0T4ylWWIsUU4Hozs0YsGuwEGBZ21LY8wFdgWUk+9hzaEsJnRpCkCU0+FvfQ9s2QCXw/5qd2ucSHpOXpCvJLxt3LmXFo0a0KxRA6KiXIwZ1Jv5K8uIZ+szc/Eqxg66sNT25Ru20Ty1IU1S7ODQApzOte9t9plcGiYlBqX8oaRWxZdwU5GW+eXnSVPgfwHSpgPzgCGqmgYgIo2Am4EPgZGVKGfISVwSmp3lX9ecEzgatS47sysKZ8suFMyfZu+b0ADNzcE98mYcDZpiZeynYOH0UnFCa7uM03mkxvvnbCM1PpqNaWU/hskt9LJ07zEeGtoJgEMnz5AU4+bxOZvYfiybTikJPDDkAmKiSn6lP9t0iFEdGgXvIiJAeuZJUhsk+ddT6yexYcfeMvPm5hewZO0WHvnFxFJps5asLlHJPzD5Km7/80v87c1PUFXe+svvq7zsIReGlXRFldsyV9VbzrP4Z0wUkZvP2bWVqj57tiL3HStNVZ8FWgQ6n4jcJiKrRGTV1KWbf8g1BU+A6TGdrbtjHd7l72IRhwNHSnM86xeS9/5TaGEBUX1Gh7KkkaESAwK+3X2Unk3q+btYPKpszcjm6u7NmHbDQGKinExdubfEPq+u2I3T4eDSjo3LOGItUsb3VgJEr1y4agM9L2hDYnzdEtsLCz0sWLmBURf18m+bPnsR90+6krlT/h/3T/oZj//n3SotdnWwPBVfwk1lhiaW595z1veJyAMiknp2g+/FoweBA4EOoqpTVLWPqvaZfFHnKixe5WhOFhJf1JqRuHro6RNl5nV26Itn+0r/upVzAs05gZW+FwDvzjU4UgL+ftVaKXHRpGcXdYGkZ+fRsG7ZoWZnby/qYgFIjYsmJa4O3RrXA2BE+1S2Zpzyp3+++RDf7j7KX8Z0QwLVXLVEanI90o8V/ZWZnplFw/pld4nMWryasZeU7mJZ/P1mOrVpTnK9BP+2zxcsZ8SAngCMuqgXG3fuq9qCV4NI7mapysr83H8x1wDJwEIRyRSRTGABUB+4ugrPGxRW+j6kXgqSkAwOJ64OffHuLqOf0R2Ns1l7vLuKxe04cwrNzkTq2b9jzuYdsYo9ODVsXRolsP/EGQ6dPEOh12L29jSGtE0plS87v5DVBzMZ0rahf1uDunVoFB/N3szTAKzYf5w2yXZrcsneY7yxai//vKIXMVHO0FxMGOvSriX7jhzlYPoxCgs9zFq8hiF9Sj//yT6dy6rNOxnat3RaWf3oDZMSWbVpBwDLN2ynReOGpfaLOCoVX8JMZYYmlqfE33KqmgU86FtKEJFbgNer8NxVTy0KFnxAnfH32EMTNy9FM4/g6nYJAJ4NiwBwtu2Fd9/mUv3hBQs+wD1mMuJ0Yp08RsHct3z5exI1+BokJo464+7GOnqA/E//FdprCxMuh4MHh3bkzk/WYKkyrktT2ibH8eF6+w+3q7s3B2D+zgwGtGxQqj/8wSEdeWTWBjyWRdOEGJ4Y1RWAZ+dvocBrccf/7CiH3Ron8ujw6vsrr7q5nE4e+cVE7vjzS3gtZfywAbRr0Zjps+3v8MTR9nd63vJ1XNSjI7HRJf86ys0v4Lt1W/njr0pO4/v4Hdfz7NSP8Hot3G4Xj98e+dP8hmOLu6KkqsIkicj3qtqr/JwgIvtVtdx+hzMv3B5+r1nVNO7aPf46VJyXXFHdRajx6nQd+aOby0cGDa1wndN48fywap5XZct8SfEVEQk09kmA1ABphmEY1cbyhlX9XCkVrsxF5F7srpFs4FWgF/CQqs4BUNW7z9klFTvEXNY52wVY+kMLbBiGESxV2c0iImOAFwAn8KqqPnNO+v3A2bcHXUAnoKGqZvrSncAq4JCqXlbe+SrzAHSyqp4CRmG/yn8L8Mx58n8JxKnqvnOWvdgPQg3DMMKKWlLh5Xx8FfFLwFigM3CdiJR4cKOqz6tqT1XtCTwMLDxbkfvcC2ypaNkrU5mfLf2lwOuquo7SI1iKF/RWVV0cIO36SpzXMAwjJFQrvpSjH7BTVXeragEwDRh3nvzXAe+fXRGRZsBPsXtBKqQylflqEZmDXZnPFpF4Ivp9KcMwjJIq0zIv/oKjb7mt2KGaUvJ9moO+baWISCwwBvi42OZ/Ag9QiTq2Mg9AbwV6ArtV9YxvOtxbKrG/YRhGWCuv+6REXtUpwJQAyWUdKFB7/nJgSbG+8suADFVdLSJDKlqeyrTMFbvv5x7fel0gOnB2wzCMyGJ5pcJLOQ4CzYutNwMOB8h7LcW6WICLgStEZC9298wwEXmnvBNWpmX+H+wm/zDgSexRLR8DfStxjEqRurHBOrRxVtva+zJNKDkata3uIhgVoFX3ZudKoL2ItAYOYVfYpZ4VikgiMBi4sagM+jD2A1F8LfP7VPXGc/c9V2Uq8/6q2ltEvvedMEtE3JXY3zAMI6xV1dBEVfWIyN3YcZOdwFRV3SQit/vSX/ZlnQDMUdXTP/aclanMC33Dbc6GjWuIeQBqGEYNYlXhnCuqOgOYcc62l89ZfwN44zzHWEAFh3JXpjJ/EfgESBGRv2CHjHu0EvsbhmGEtSrsZgm5yoSNe1dEVgPDsZ/UjlfVCg9oNwzDCHeVGc0SbsqtzEWkfrHVDEoObK9/zhtLhmEYEaumz82yGrufPNC4yTZVWiLDMIxqUpV95qFWbmWuqgECXxqGYdQstaLPHEBErgQGYbfIF6nqp8EoVLhYsieD577ZjKXKhO7Nmdy/XYn0N1bsYsZm+z0Ar1rsOZ7D/LtGkhjj5lReIU/OXs/OY9kI8KcxPejRtCgM3ZsrdvGPhVuZf9dIkmJr7wjPJZv28Nz0b7AsZcLF3Zk8pn+J9DfmrGDGCjsWrNdS9hw5zvy/3kVWdi4PvPq5P9+hYye54/KLuXF4H176fDEL1u1ARKgfH8uTN19KSr24kF5XuFm8bBXP/PNlvJbFzy4fwy9uKhmweeq7H/HVnPkAeL1edu87wKKvppGYEM9b0z7h4y9mISK0b9uK//fI76hTx81Lr73Dx5/PIqmeHYLu3l/dzE8u6hfya6tKVRTeoVpUZgrc/wDtKOozv11ERqrqXUEpWTXzWsrTczfx8sT+pMZHc8PbixncNpW2DeL9eSb1a8ukfvbLIAt3pvPO6j0kxtgV83PzNnFR64b8ddyFFHotcgu9/v3STuWybN8xGifEhPaiwozXsnj6/bm8fO9EUpPiueHptxncvS1tmzTw55k0qh+TRtkVxML1O3nnm9Uk1o0hsW4M0x+d5D/OqIf+j2E92wNw88i+3HXFIADem7eaKV8t5dEbRoX24sKI1+vl//3tJf77z6dolNKAa35xL0MH9adt65b+PJNvuIrJN1wFwILFy3jrg09JTIgn/egx3v3oMz579xWi69Th9398iplfL2T8T0cCcNM147nl+quq5bqCIZK7WSrzOv9gYLSqvq6qr2NPuDUkUGbfXL5nPyeKyGsisl5E3ise5DlcbTxyguZJsTSrF0uU08Hojk1YsDM9YP6ZWw8zpmMTAHLyC1lzMJMJ3ey3eaOcDhKiiyL6/HX+Zn4zuFNwLyACbNx7hOYpSTRrWI8ol5PRfTuyYP3OgPlnrtzKmD4dS21fvnUfzRrUo0my3UKMiykKe5ZbUBgwEn1tsWHLdlo0a0Lzpo2Jiopi7PDBzFu0LGD+GV8v5NKRg/3rHq+X/PwCPB4vuXn5NGxQP+C+kc6ypMJLuKlMZb4NKB7qrTkQKJoQwFPFPv8NOII9ocxK4JVKnLdaZOTk0Si+qOWcGh9NRk5emXlzC70s3XOUER0aAXDwxBmSYtw8NnM917y5iCdmrSe3wAPAgp3pNIyL5oKUhDKPVZtkZOXQKKnoL53UevFkZOWUmTe3oJClm/YwoneHUmmzV21lbN+SP47/+nQRox9+mRkrtnDH5YOqtuARJuPoMRqlFAVbTk1pQMbR42Xmzc3LY/GyVYwcYt+z1IYNmHTdzxhx5c8ZOu564uvGcnH/osDO73/8BRN+fgePPvV3Tp7KDu6FhIClUuEl3FSmMk8GtojIAhFZAGwGGorI5yLy+fl3pY+qPuoLTvEPoFWgjMWnlXzt2/P9VgRXWV1ngf7zfbsrnZ5Nk/xdLF5VtqafYmLPFnxw8yVERzmZumIXuYVeXl22kzsHla6QaqMy73GAm/zt+l30bNuUxLolu6YKPV4WrtvFyAsvKLH91+MvYfbTt3Npv05MW7CmikocmcrqBw50nxcsXk6v7p1JTLB/ZE+eymb+omXM/vB15n32Lrl5+Xwxex4A10z4KTOnT+XjN16iYXJ9nv/3f4N1CSGjKhVewk1lKvPHsKNmPO5bLgX+jN3q/lsZ+VNE5Hci8nsgQaTE1yfgeVV1iqr2UdU+t/6keyWKV7VS46JJy871r6dn59EwruxJImdtKepiObtvSnw03ZrYDzxHXtCYLeknOXjiNIdOnmHiG4sY+8o8MrLzuO6tRRwL0OKv6VKT4kjLKmrNpZ/IpmGAB5WzVm5hTN/SXSyLN+6mY4sUkhPqlrnf2L6d+Ob7HVVT4AiVmtKAtIyj/vX0jGM0bJBcZt6Z3yzk0hFD/OvLVq2laZNU6ifVI8rlYvjgi1i7wX4g3aB+Ek6nE4fDwVVXjGXj5u1BvY5QqBUtc1VdqKoLge+BDWeXYtvP9V8gHogD3gQaAIhII2Dtjyx30HVpnMj+rNMcOnGGQq/F7K2HGdyudFd/dn4hqw9mMrRYWoO4aBrFR7M30+4yWL7vGG2S42nfMIH5d41k5q+GMfNXw0iJj+b9n19CgwA/EjVdl5aN2Z+RxaFjJyj0eJm9ciuDu7crlS87N5/VOw4ytEfptFmrtjLmnC6WfelFYWcXrt9F69Sa28dbEV07dmD/wcMcPJxGYWEhM79ZyNBBA0rly845zarvNzD0koH+bY1TG7J+41Zy8/JQVZavWkublvazoKPHit4X/GbhUtq1aVnqmJFGK7GEm8qMZrkNuyWeiz3BlnCel4ZU9YkA29NEZH7lixpaLoeDh0Z05Y6PVmBZyrhuzWjXIJ4P1+4D4Oqe9hd33o40BrZqQIy75K18cHgXHvlyLYVei6b1YnlybI+QX0O4czkdPHTNCO548SMsy2LcRd1o16QBH367FoCrf9ITgHnf72Bg51bE1Ck5hDO3oJBlW/aWGqny4qcL2ZuehUOgcf1E/nD9yFBcTthyuZw88ts7+NXvHsXr9TLhslG0a9OSDz75CrC7S8CukC/q15vYmKLGRfcuHRk5dBATb/k1TqeTjh3acvW4sQD87T+vsW3HbhBo2iiVxx+4p/TJI0w4trgrSrSCAytFZAcwUFWP/eiTiuxX1Rbl5ct99Xfh+ANYs5j5zEPC1W1YdRehxotq0OZH18SLGl1V4TrnkrSPwqrmr8xLQ7uAMxXNLCKBnl4KEPZDEw3DqH00cIz6sFeZyvxhYKmILAfyz25U1UB/W6UCo4Gsc7YLsLQyhTQMwwgFK4L7AipTmb8CzMN+8FmRoBRfAnGquvbcBN/QRsMwjLBi1ZKWuUdVf1fRzKp663nSSsXCMwzDqG61pZtlvm9EyxeU7GYx85kbhlEjRHIczMpU5mdb0w8X22bmMzcMo8bw1oaWuZnX3DCMmq62tMwRka5AZ8D/VoGqvlXVhfIfO+d0sA5t+DhcdcrPZPxons3fVncRaryon/z4ToJa0WcuIo9jT3nbGZiBPU/LYiBolblhGEYoheHMthVWmYm2rgKGA2mqegvQAzDNOsMwagwLqfASbirTzZKrqpaIeEQkAcjAPPw0DKMG8ZafJWxVpjJfJSL1sGdDXA3kACuCUSjDMIzqYEVwWKrKjGa50/fxZRGZBSSoavVFjzAMw6hiEfw2f8X7zEXkYhE5GwFgEDBJRCJ/AmPDMAwfqxJLuKnMA9D/A86ISA/gAWAfZiSLYRg1iCUVX8JNZedmUREZB7ygqq+JyM3BKlg4cLTsgnvwRHA48GxcjGfV7BLprgtH4erYz14RB1K/Mbmv/B7yz0CdGNwjbsKR3BRQCua+hXVkN872vYkacDlSvxH57z+DlbEv9BcWRpZs3MWz78/GspQJl/Tk1ksvLpH+xqzvmLF8IwAer8WeI8dY8I/fkZVzhgde+Z8/38GjWdw5bjA3juzv3/bm7O/4+4ffsOAfvyMpPjY0FxSmlmzcxXPTvsayLCZc0pPJYweWSH9j9jJmLNsEgNey2HPkOPP/cS9Z2Wd44JVP/fkOHTvBHeMu4cYR/Xjp04UsWLsDEaF+QixP3nIZKfXiiWThOEqloipTmWeLyMPAjcBPRMQJRAXKLCKJ2K/+jwfOhgbPAD4DnlHVEz+kwCEjgnvodeT/759oThbR1z2Md/d6NPOIP4tn9Rw8q+cA4GzdHVfv4XZFDrgHX4N37yYKvpoCDidE2VFyrGOHyf/yZdzDbwj9NYUZr2Xx1LszeeV3N5CalMD1/+81hvTsQNsmRZHkJ40ZyKQxdsWzYO123vl6OYlxMSTGxTD98V/6jzPyvhcY1rsoqHNa5km+27yHxvUTQntRYchrWTz93hxe/u21pCYlcMNf3mBwj/a0bdLAn2fS6AFMGm2Hklu4bgfvzF1JYt0YEuvGMP3xW/3HGXX/vxnWy77PN48ewF3jBwPw3jcrmfLFEh69aUyIr65qeSO3Lq9UN8s12BNs3aqqaUBT4Pnz5J+OPZf5EFVNVtVkYKhv24c/sLwh42jUGj2ZgZ46BpYXz/ZVONsGDv3mvKAvnm0r7RV3NI6m7fFuWmKvW17It4NDa1YampUe7OJHhI17DtM8pT7NGiYR5XIypl8XFqwNHBR41opNjO3XpdT25Vv20LxhEk2S6/m3Pf/BXH571XAkgkcnVJWNew7TvGGS/z6P7tvpvPd55orNjOlXOgLV8i17adawHk2SEwGIiyl6zSQ3v5CacKtrRZ+5qqap6t9VdZFvfX/xV/lF5Ltzdmmlqs/6Kv7ix3gWKDdkXHWTuvXQ7KK4GpqdhdStV3ZmVxTOVl3w7lhj75vYAM3Nxj3qZqKv/wPuETeBy132vrVYRlY2jZKKWs4pSfGkZ2WXmTc3v5AlG3cxonenUmmzVmxmTP+iSn7B2u2k1IvnguYmoBVAxokcGhX7CyU1KZ6ME4Hv89KNuxlx4QWl0mav3MLYcyr5f32ykNEP/JsZyzdxx7ifVG3Bq0FVBnQWkTEisk1EdorIQ2Wk3y8ia33LRhHxikh9EWkuIvNFZIuIbBKReytS9sq0zMtzboj5fSLygIj4/0WJSKqIPAgcCHQQEblNRFaJyKqpS7dUYfEqqRKtDGebHliHd/m7WEScOFJa4Fm/kLz3/oIW5hPVN7L//AwGLeOfRKDW3cJ12+nZrjmJcTElthd6vCxct51RF9qVfG5+If/9ajF3jhtc5eWNVGXF+ZUAX/Bv1++gZ7tmJNYt6z7vYGSfkj+mv54wmNnP3c2l/bswbd6qqit0NamqB6C+buiXsKc96QxcJyIlfglV9XlV7amqPbG7pBf6phT3AL9X1U7AAOCuc/ctS1VW5ud+Y64BkoGFIpIlIpnAAqA+MDHgQVSnqGofVe0z+aLSrbBQ0ZwTSHySf13ik9DTJ8rM6+zQB8+2ovenrJwsNCcLK20vAN4da3CkhP0fIyGXmpRAWtYp/3pGVnbAB2izVm5mbP/SXSyLN+ykY4tGJCfGAfaD0EPHTjDxif8y9sF/kZ51imv//CrHTuYE5yIiQGpSPGmZRfc5PSubhvXiysw7a8WWMrtYFm/cRccWqSQn1C1jLxjbvwvfrNlWNQWuRlXYzdIP2Kmqu1W1AJgGjDtP/uuA9wFU9YiqrvF9zga2YHdrn1dVVuYlqGoW8DpwN9BcVeuraidVfRD7QsOalbYXqZeCJCSDw4mrQx+8u9aVzuiOxtmsQ8m0M6fsbpkk+48SZ4uOWMePlN63luvSqgn70zM5eDSLQo+XWSs2MbhHh1L5ss/ksXrbPob0LJ0285x+9PbNUljwj98x89lfM/PZX5OalMC0P/6CBollV161QZdWTdifkcWhoyco9HiZvXILg3u0L5Uv+0weq7fvZ2jP0mmzVmxmzDnPK/alF8WlWbh2B60bJVd94UOsMpV58V4E33JbsUM1pWQPxEECVMgiEguMAT4uI60V0AtYXl7ZKzUFbjlK/OEhIvcAd2H/qrwqIveq6me+5KeAWVV47qqnFgXzp1Fnwr0gDjyblqCZR3B1s/sFPRvsKU2d7Xrh3bcZPAUldi9YMA33mFsRhxPr1DEK5rxp52/bk6gh1yIxcdQZdzfWsQPkf/JiaK8tTLicDh6+fgx3/PN9LMti/MU9ade0IdMXrAZg4pALAZj3/TYGdmlDbJ2Szx1y8wtZtnkPf7zp0pCXPZK4nA4eun4kd/xzGpYq4y7uTrumDflwgf2M5+ohvQGY9/12BnZpTUyA+/zojSW7Cl/83wL2ph3HIULj5ET+cGPkdyVWZjSLqk4BpgRILutIgbraLweWnBu1TUTisCv436jqqTL3LJ6/rP60H0JEuqrqxmLrG4CBqprj+3X5CHhbVV8Qke9VtVd5xzzzz19F8tu1EcFx4aDqLkKtoBrJUzhFhpifTPrR42n+3fzGCtc5dx94J+D5RGQg8CdVHe1bfxhAVZ8uI+8nwIeq+l6xbVHAl8BsVf17RcpTmdf5rxSRHSJyUkROiUi2iPh/LYpX5D5OVc3xpe3Fngt9rIj8nUo9XjQMwwiNKhzNshJoLyKtRcQNXAt8fm4m3/s4g7Hfvzm7TYDXgC0Vrcihcn3mzwFXqGqiqiaoaryqnu+NjDQR6Xl2xVexXwY0ALpV4ryGYRghUVWjWVTVg/28cDZ2V/N0Vd0kIreLyO3Fsk4A5qhq8bBqFwM3AcOKDV0sty+xMn3m6apambGCP8ceYuPnu8Cfi8grlTiOYRhGSFTly0CqOgM7KlvxbS+fs/4G8MY52xbzA3ovKjuf+QfAp9hvgp498f/KyqyqBwMdSFWXVOK8hmEYIRHJTzYqU5knAGeAUcW2KVBmZW4YhhFpwnE2xIqqTHCKW4JZEMMwjOoWjnOuVFS5lbmIPKCqz4nIvyjjIa6q3hOUkhmGYYRYJI+FrkjL/OxDz1WE+Fr1aGb5mYwfxdqyprqLUCs4LjRzxUQCK4Kr83Irc1X9wvdxM/AI0KrYfoqJNmQYRg1Ro7tZinkHuB/YQGRfs2EYRplqy2iWo6pa6g0mwzCMmqJWjGYBHheRV4FvqMA4c8MwjEhTo/vMi7kF6Igd9/NsN4sZZ24YRo0RuVV55SrzHqpq5lQxDKPGiuSHgZWpzJeJSGdV3VzRHXyzf/XDnpRdgcPACq2qeXcNwzCqkDeC2+aVqcwHATeLyB7sPnMBVFW7l5VZREYB/wF2AId8m5sB7UTkTlWd88OLHRrO9j1x//QWcDjwrPqGwm8/LZEeNegKnD0vAUAcDqRhM848dSvk5hBz30uQn4eqBZaXvP8UxXN1DRhD1ICxYHnxbFtD4ex3QnlZYWXJnqM8P38zlirjuzZncv+2JdLfXLmbGVvsr4/XUvZk5jDvjhEkxrjJzivkiTkb2HUsGxF4fHR3ejRJ4qUl21m4Mx0RqB9bhyfGdCcl7twQtbXLknXbePatz7EsZcLQvtx6xdAS6W98sZAZS78HwOO12HMogwWvPEbWqdM88K93/fkOZmRy51UjuXGs/b1/b/YSps1ZitPh4Ce9OvHb6yM7UEhtaZlXNozIC8AI31zmfiLSGnsmseoL8FkR4sB9+a3kvf5n9FQm0Xc8jWfLKvRo0fxhhYs/p3CxPcDH2fFCoi66DHKLYk3mvvYnOFMyCrqjdRdcnfqS+6/fg9cDdc83i3DN5rWUZ77ZxP9d1Y/U+GhueHcJg9ul0Da5KA7ozX3bcHPfNgAs3JXOu6v3khhjR8J5bv5mLmrVkL9e0ZtCr0VeoT2w7OY+rbnrYjvE3Htr9jLlux08OrL29hB6LYunXv+UVx7+BanJiVz/6L8Z0rszbZv5Y60z6fLBTLrcfrFpwerNvDNzMYlxsSTGxTL96d/4jzPyrr8wrE9XAFZs2sWCVZv56Jnf4o5ycbwGxFmN5AegFZ7PXFX3lbWcZxcXdty7cx3Cfoga1hzN2mFlpqFZGeD14F2/BFenPgHzu7oPwrN+cbnHjeo/ioJvP7UrcoDT5UaDqrE2pp2geb1YmtWLJcrpYPQFjVmwMz1g/llbDzOmY2MAcvILWXMwkwndmgEQ5XQQH21/reLqFH29cgs92L19tdfGnQdonppMs9RkolwuxgzswYLVgXtLZ323jrEX9Si1ffnGnTRPTaZJQzvQ+Ydff8fkK4bgjrLbhMk1IM5qFQanCLmqjAF6rqnAShGZRlFg0+bYETdeC+J5q4Qk1EdPHvev66lMHM1LB7oFIMqNs31P8r8odlkK0bc8CgqelXPxrPzaPm6DJjhbdcI98jrwFFIw8y2sQ7uCeSlhKyMnj9T4ou6P1PgYNh45UWbe3EIvS/ce46FhdlDhQydzSYp18/js9WzPyKZTagIPDOtMjK9i+ffibXy56RBxdVxMmdg/6NcSzjKyTtIouZ5/PaV+Iht27i8zb25+AUvWbePhSaUDyc/6bh1jBvb0r+9LO8aabXv41/TZ1Ily8bsbfkrXts2ruvghFcndLJWJNFQpvlh3N2D3rQ8ELvJ9vqGsOHhnFY94PfX73cEqXvnKDMda9u+xs2MfvPu3luhiyZvyKHkvPUjem3/B1X80jlZ2r5I4HEh0XfJefoSCWW9T59rfBaP0kaESzZtvd6XTs0mSv4vFY1lsTT/F1T1aMu3ng4iJcjF1RdH35e5BFzDrV8MY26kJH3x/vj8ga76yvraB/lpZuGYLPTu0IjEutsT2Qo+Hhas3M2pAUXeVx2tx6nQu7zx5F7+9/qfc/+K7RPrYBq3E/8JN0CpzAFXdrKrPqOqvVfVu3+fzjoZR1Smq2kdV+0zu1SaYxTsvPZmJJCb71yWhPnqq7Im/XN0vxrOuZLwNzc6yP5w+hXfzChzN2gFgnczEs3m5/fngTlALYmtnv3lKfDTp2Xn+9fTsXBrG1Skz7+xtR/xdLGC34lPio+nWuB4AIzo0Ymv6yVL7je3UlG92pFVtwSNMav1E0o6f8K9nZJ4kJans71ygLpbFa7fRsXVTkhOLnmek1k9keN+uiAjd2jXHIUJW9ulS+0YSD1rhJdwErTIXkUQReUZEtorIcd+yxbetXrDOW1WsQztxJDdGklLA6cLZ/WI8W1eVzlgnFmerzni3rCzaFlUH3NH+z852PdB0u6fJu2UFzjZ260aSG4PTBWdqZ795l0aJ7D9xmkMnz1DotZi97QhD2qaWypedX8jqg5kMaVeU1qBuHRrFR7M30/5raMX+Y7RJtvts92UVVSgLd6bTqn7k9+X+GF3aNmN/2nEOZmRS6PEw67t1DL6w9PiD7DO5rN6ymyEXdimVNnPpWsYOLFnJD+3ThRWb7C7CvUeOUujxkhRfNzgXESKmz7xs04F5wBBVTQMQkUbAJOBDYGQQz/3jWRYFX7xG9KQ/gDjwrJmPZhzE1c8utmfFXABcnfvh3bkOCv0zHCBxidS54X77s8OJZ/1ivDvW2vutnk+dK+8g5p6/oV4P+R+/FNrrCiMuh4MHh3Xhzo9XYFkwrmsz2jaI58N1drfI1T1aAjB/RzoDWjbw94ef9eCwLjwyYy0er9I0MZYnxtijZF9ctJV9madxiNA4IYY/jOga2gsLMy6nk4cnjeOOZ17DsizGD+lLu2aNmP71MgAmjhgAwLyVmxjYrT2x0e4S++fmF7Bs407++IsrS2yfMKQPj73yEVc+8HeiXE7+fMfEiH/YHMmjWSRYfVwisk1VL6hsWnGn/3B15N7ZCCEtm1V3EWoFM5958EVfOP5H/5L8slXF65z/7v0wrH65gtlnvk9EHhAR/9/GIpIqIg9SNLrFMAwjbJgHoGW7BkgGFopIpohkAguA+sDVQTyvYRjGD2JVYgk3QeszV9Us4EHfUoKI3AK8HqxzG4Zh/BCRPDdLUIcmnscT1XRewzCMgCzVCi/hJmgtcxFZHygJKD3+zDAMo5qFXxVdccEcmpgKjAayztkuwNIgntcwDOMHieShicGszL8E4lR17bkJIrIgiOc1DMP4QcJxlEpFBfMB6K3nSbs+WOc1DMP4ocLxNf2KCmbL/Ecr2Hq0uotQ49VJqN2vuoeK7qtwgC7jh7pw/I8+hGmZG4Zh1ADhOH68okxlbhiG4RPJU/hW1zhzwzCMsGOhFV7KIyJjRGSbiOwUkYfKSL9fRNb6lo0i4hWR+hXZtyymMjcMw/Cpqtf5RcQJvASMBToD14lI5+J5VPV5Ve2pqj2Bh4GFqppZkX3LEtRuFrHnw+wHNMUej38YWKGR/LeMYRg1lrfqes37ATtVdTeAL3zmOCDQk/DrgPd/4L5AcN8AHQX8B9iBHcQZoBnQTkTuVNU5wTq3YRjGD1GZdqaI3AbcVmzTFFWd4vvclJKzwx4EygxGKyKxwBjg7sruW1wwW+YvACNUdW/xjSLSGpgBlA51EmZcPfsRO/lucDjJ/+Yr8j95r0R6nXHX4L7EDlYhTieOpi04OXk8mpNtZ3A4iH/2FazMY5x++mEA6v7uMRxNWtj71I1DT+eQfd8vQndRYcbRuivu4deDw4Fn3bd4ls8oke7qNwZX54G+zA4kuQm5/7oH8k4TffvzUJAHloVaXvLfehIAadgc9+ifI+5o9OQx8r94xc5Xiy3ZdojnvlyBZSkT+rZn8pBuJdLf+HYjM9baMVS9lrIn4yTzH72GxNg6nMot4Mn/LWVnehaC8KefXUSPlilsPZzJXz79jnyPF5fDwcPj+tOtecPquLwqU5l2ua/inhIgucwowgHyXg4sUdWzcSkrs69fMCtzF/YvyrkOAVFBPG/VcDiI/eW95Dx5H9bxo8Q/+zKFK5dgHSwKDpz/2Qfkf/YBAFF9BlLnsquLKnKgzk9/hnVoH8QUhdI6/fcn/Z9jbr4DPRPZMRN/FBHcI28i/4O/otmZRN/8GN6da9Hjh/1ZPCtm4VkxCwBn2x64+o6GvKJ7lvf+syUCaQO4x95C4fwPsA5sw9ntEqL6j6Vw0SehuaYw5LUsnv58GS/fOorUhFhueOkrBndqTtvUev48k37SlUk/sSMyLdxygHcWbyYx1o7H+twXK7ioQxP+esMQCj1ecgu9APxz5ip+NbwHgy5oxqKtB/nnzNW8dtuYkF9fVarCceYHgebF1pthdzOX5VqKulgqu69fMB+ATgVWisiDInK9b3kQWA68FsTzVglnu45YaYew0o+Ax0Ph4nm4+14cML970HAKFn/jX5f6DYnqPYD8r78KvM9FQ0vsU9s4GrdBT2SgJ4+C5cWzZQXO9r0C5nd2HoBny7Lyj1u/EdaBbQBYezfh7HBhlZU5Em08cIzmyQk0qx9PlMvJ6B6tWbAlcHyYmev2MKZHawBy8gpYszedCX3aAxDlcpIQY4eVExFO5xf68zVMiA3ylQRfFY5mWQm0F5HWIuLGrrA/PzeTiCQCg4HPKrvvuYL5Ov/TIvIZcAUwEPtPh4PADaoa9q/DOeo3xDpW9AaqlXkUZ/sAD5TddXD17MeZV1/wb4qdfDe5b7+CxJT9BXd17o51IgvryKEy02sDiU9CT2X61zU7E0fjtmVndrlxtu5Kwdx3irapEj3xPkApXLsA77qFAFjHDuFs1wvvzu9xduyDxNcP4lWEv4xTZ2iUWPTXYWpCLBsOlP12dW6Bh6XbD/HwFXYX7cHMHJLq1uGxj5aw/UgWnZsm88DlfYlxR3H/ZX25c+rX/H3GKixV3rz90pBcTzBV1dgMVfWIyN3AbMAJTFXVTSJyuy/9ZV/WCcAcVT1d3r7lnTOoo1l8lXbYV9xlKrPXquz/0FF9LsKzbaO/iyXqwoFYJ7Pw7t6Oq0vPMvc5tyVvnFX2PXa264l1aGeJLpb8d59Cc05AbDzR19yHHj+CdXA7BTNewz3iBqIuvgLPzrVgeUNT9DBV1h0NFHj5260H6Nkyxd/F4rUsth7O5KHL+9OtRUOe/WIFUxds5K5Rvfhw2Tbuu6wvI7q2ZPb6vTzx8VJe+cWoIF5J8FXhaBZUdQb288Hi214+Z/0N4I2K7FueoHWziEiiiDwjIltF5Lhv2eLbVu88+90mIqtEZNUbe8rtJgoa6/hRHA2KHuY46jdEM4+Vmdc9aBgFi4oqZmfHrrj7XkzC/02j7m8fI6pbL2Lv+UPRDg4nUf0voWDJ/KCVPxJodhaSUNRqlvj6duVcBmenfng2Ly+5/9m8Z7Lxbl+Do0kbe3tmGvnT/0bem0/g3bwMKysjGMWPGKkJsaSdLPoRTD91JmCXyKxiXSwAqYl1SUmIpVsL+9/CyK4t2XL4OABfrNnF8C72w/xR3Vqy8WDZ/z4iSSQHpwhmn/l07LnMh6hqsqomA0OBE8CHgXZS1Smq2kdV+0xq3SSIxTs/785tOBo3w5HSCFwuogYNo2BVGdOwx9bF1bkHhSuX+DflvftfTt52NafuuJbT/3iSwg3fc+bFv/jTXd0vxHtoP5pZuycSs47sQZJSkMQG4HDi6tQP787vS2d0x+BsfgHenWuKtkW5wR3t/+xo3RXrqO95e2y8L5MQddHleNYuCOZlhL0uzRqw/9gpDmVmU+jxMnvdHgZ3alYqX3ZeAav3pDO0c9GztwbxMTSqV5e9R08CsHzXEdqk1AOgYUIsq/akA7BiVxotkuNLHTPSaCWWcBPMbpZWqvps8Q2qmgY844sBGt4sL2defYG4Pz4PDgcF82ZiHdiLe9QVABTMsZ9HuPtfgmfdKsiv+NA396BhFCyeF5RiRxS1KJj7LnUm/h7EgWfDIvTYYVw9hwD4K2Fnh954926CwgL/rhKbSJ0rfcNyHU48m5dh7dkIgKvTAFy9hwHg3b4a74ZFIbukcORyOnjoiv7cMfVrLLUY16c97VKT+HC5/ZD46v4XADBv034Gtm9CjLvkYLMHL+/PIx8sotBr0bR+HE9eZQ8EeOzKgTz3xQq8luJ2OfnjlReF9sKCIJKDU0iwXsYUkTnA18Cbqpru25YKTAJGquqI8o6R9bMhkXtnI0Sdfq3Lz2T8aNK+fXUXocaLufKRsh8EVMLApkMrXOd8d2j+jz5fVQpmN8s1QDKwUESyRCQTWADUByYG8byGYRg/iFetCi/hJphDE7NE5HVgLrBMVf1vdojIGGBWsM5tGIbxQ0RycIpgjma5B3sg/N3ARhEZVyz5qWCd1zAM44dS1Qov4SaYD0B/CVyoqjki0gr4SERaqeoLlD2K2zAMo1pF8gPQYFbmzrNdK6q6V0SGYFfoLTGVuWEYYSgcW9wVFcwHoGki0vPsiq9ivwxoAHQLtJNhGEZ1qcpIQ6EWzJb5zwFP8Q2q6gF+LiKvBPG8hmEYP0g4jlKpqGCOZilr+tuzaUsCpRmGYVSXSB7NErSXhqqCy900fAtnGEZY8RQc+tHP4jql9KtwnbMlY0VYPfsL6qyJhmEYkSSSW+amMjcMw/AJx9kQK8pU5oZhGD6mZW4YhlEDmNEsAYgdzqQf0BR7CuDDwAoN56euhmHUWmoq89JEZBTwH2AHcDbQZTOgnYjcqapzgnVuwzCMHyIcXwaqqGC2zF8ARqjq3uIbRaQ1dmy7TkE8t2EYRqVFcqdBMF/ndwFlvTh0CIgqY3vYGT1qCJs2fsvWzYt54P67yswz+CcDWbVyDuvWzmPe1x/5t997zy9Zt3Yea7//hnfefok6dewAuUlJ9Zg14322bFrMrBnvU69eYkiuJVyZexwa5j5XTCS/zh/MynwqsFJEHhSR633Lg8By4LUgnrdKOBwOXnzhL1x2+Y106zGUa64ZT6dOJaPFJCYm8K9/PcWEKyfRo+cwrrnuVwA0adKIu++aTP8Bl9Kz13CcTifXTLRnAH7wgbuYN38xnboMYt78xTz4QNn/sGoDc49Dw9znivNaVoWXcBO0ylxVnwZuwJ4hcSBwke/zDb60sNavby927drLnj37KSwsZPr0z7ji8tEl8lx37QQ+/XQmBw4cBuDo0eP+NJfLRUxMNE6nk9iYGI4cSQPg8stH89bbdjzrt97+kCuuGBOiKwo/5h6HhrnPFaeV+F+4CWbLHFXdrKrPqOqvVfVu3+fNwTxnVWnStBEHDh72rx88dIQmTRqVyNO+fRvq1Uvkm7kfsnzZTG688SoADh9O4+//eJk9u1ZwcP/3nDx1irlffwtAakoD0tIyAEhLyyClYXKIrij8mHscGuY+V1wkB6cIZqShRBF5RkS2ishx37LFt63eefa7TURWicgqyzodrOKVyx5VWdK5/wFdLicX9u7O5eN+zqU/vZ4/PPwb/z+KKy4fTbsOA2jesjd168Zy/fVXhqroEcPc49Aw97niIrnPPJijWaYD84AhqpoGICKNgEnAh8DIsnZS1SnAFKjeibYOHTxC82ZN/OvNmjbmyJH0knkOHeH48UzOnMnlzJlcFi1eRvfunQHYs3c/x45lAvDJpzMZOKAP7733P9IzjtGoUQppaRk0apRCRrE/Z2sbc49Dw9znigvHFndFBbObpZWqPnu2IgdQ1TRVfQZoEcTzVomVq9bSrl1rWrVqTlRUFBMnjuOLL0sOjf/8i9kMurg/TqeTmJho+vXrxdatOziw/xD9+/cmJiYagGFDB7F16w4AvvxiDj+/6WoAfn7T1XzxxezQXlgYMfc4NMx9rjhLtcJLuAlmy3yfiDwAvKmq6QAikordMj8QxPNWCa/Xy72/eZQZX72H0+HgjTc/YPPm7dz2y5sAmPLft9m6dSez58zn+zVfY1kWU6e+z6ZN2wD43/++YuWK2Xg8Htau3cR/X30XgGeff4lp773MLZOu48CBQ/5RA7WRucehYe5zxUXy6/xBm89cRJKAh4BxQCr26/zpwOfAs6qaWd4xzHzmhmFUVFXMZ55Qt02F65xTp3eH1XzmIQtOISKXYM/TsqGir/KbytwwjIqqiso8LrZ1heucnDN7wqoyD+ZolhXFPv8CeBGIAx4XkYeCdV7DMIwfKpLHmQezz7z4K/u/Akap6lER+SuwDHgmiOc2DMOotHB8sFlRwRzN4hCRJBFJxu7OOQqgqqcBTxDPaxiG8YNYalV4KY+IjBGRbSKyM1BvhIgMEZG1IrJJRBYW2/5b37aNIvK+iESXd75gVuaJwGpgFVDfN8YcEYnDfq3fMAwjrFTVG6Ai4gReAsYCnYHrRKTzOXnqYU8TfoWqdgGu9m1vCtwD9FHVroATuLa8sgetm0VVWwVIsoAJwTqvYRjGD1WFA0L6ATtVdTeAiEzDHtlXfDqT64H/qep+37kziqW5gBgRKQRisQP7nFdQ52Ypi6qeUdU9oT6vYRhGebQSSzmaUvJ9moO+bcV1AJJEZIGIrBaRnwOo6iHgr8B+4AhwsiIjAMM6BmhVDDUKNRG5zTclgREk5h4HX229x5Wpc0TkNuC2YpumFLtnZR3n3N8AF3AhMByIAb4TkWXAUexWfGvgBPChiNyoqu+crzwhb5nXAreVn8X4kcw9Dj5zj8uhqlNUtU+xpfiP30GgebH1ZpTuKjkIzFLV06p6DPgW6AGMAPao6lFVLQT+hz2F+HmZytwwDKPqrQTai0hrEXFjP8D8/Jw8nwGXiIhLRGKB/sAW7O6VASISK/aUl8N9288rrLtZDMMwIpGqekTkbmA29miUqaq6SURu96W/rKpbRGQWsB57YMirqroRQEQ+AtZgD+P+Ht9MsucTstf5a4va2tcYSuYeB5+5x5HHVOaGYRg1gOkzNwzDqAFMZW4YhlEDmMq8GohIXxHxishV1V2Wmsg338VJ35wXa0XkseouU00UaF4Ro3qY0Swh5puz4Vnsp9xG8CxS1cuquxA1VbF5Rcao6n4RSanmItV6pmV+HiLyqe81202+t70QkZxi6VeJyBu+z6ki8omIrPMtgQb5/xr4GMgIkF7rBOk+G8UE4R6fb14RoxqYlvn5TVbVTBGJAVaKyMfnyfsisFBVJ/ha33HnZvDNhjYBGAb0DUqJI1OV3mefgSKyDvutu/tUdVMVlznSVPU97gBEicgCIB54QVXfqvJSGxVmKvPzu0dEzs7w2Bxof568w4CzE+V4gZNl5Pkn8KCqeu0Xuwyfqr7Pa4CWqpojIpcCn5ZzzNqgqu9xmfOKqOr2qiuyURmmmyUAERmCPUfCQFXtgf0WVjQlJ8s574TxInJXsYdwTYA+wDQR2QtcBfxHRMZXfekjRzDus6qeUtUcAFWdgd2CbBCUC4gAQfouB5pXxKgmpjIPLBHIUtUzItIRGODbni4inUTEQcl52b8B7gD7IaeIJKjqS6ra07ccVtXWqtrKN9f7R8Cdqvpp6C4pLFX5fRaRRr45LRCRftjf8+Ohu6SwU+X3mMDzihjVxFTmgc0CXCKyHvgzdtxSgIeAL4F52HMNn3UvMFRENmBHWOoSwrJGsmDc56uAjb4+8xeBa7V2v+pc5fdYVbf4jrseWEGxeUWM6mFe5zcMw6gBTMvcMAyjBjCVuWEYRg1gKnPDMIwawFTmhmEYNYCpzA3DMGoAU5kbhmHUAKYyN/xEREXkb8XW7xORP1VjkYLGN33rlz9w33oicmdVHMswqoqpzI3i8oEra/Or7xVUD7izvEyGEUqmMjeK82BHAf/tuQki0lJEvhGR9b7/b+Hb/oaIvCgiS0VktxQLuCEi94vISt8+T5R1QhH5k4i8KSJzRGSviFwpIs+JyAYRmSUiUb58j/mOtVFEpojN5ds2xJfnaRH5S6CLE5ExIrJVRBYDVxbbXldEpvqO9b2IjPNtnyQin/nKsU1EHvft8gzQ1jdPyfO+bXEi8pHv+O+enU7AMEJGVc1iFlQVIAdIAPZiz+dxH/AnX9oXwM2+z5OBT32f3wA+xG4YdAZ2+raPwv5hEF/al8BPyjjnn4DFQBT2RE1ngLG+tE+A8b7P9Yvt8zZwue9zF+w5QUZiTyDlDnBt0cAB7NkCBZgOfOlLewq40fe5HrAdqAtMwn7NPRl7ZsCN2JOltQI2Fjv2EOyZBZv5rvU7YFB1//c0S+1aTMvcKEFVTwFvAfeckzQQeM/3+W1gULG0T1XVUtXNQKpv2yjf8j32lLQdCTzt6kxVLQQ2AE7sOT/wrbfyfR4qIst984UMwzdfiNrzlL+N/WMzWVULApyjI7BHVXeoqgLvFEsbBTwkImuBBdgVfwtf2lxVPa6qucD/zrnu4lao6kFVtYC1xcptGCFh5jM3yvJP7Ar49fPkKT6pT36xz1Ls/59W1VeK7yQidwG/9K1eWnx/VbVEpNBX2QJY2BNERWOHKOujqgd8D2WLT9naDThB0Q9JRcpcoljAz1R12zll7V/GPoGOUfweeDH/towQMy1zoxRVzcTuhri12OalwLW+zzdgd42cz2xgsojEgR1lSURStPRUqhVxtuI+5jte8X75K7G7QX4CvCh2bMqybAVai0hb3/p155T118Wmze1VLG2kiNQXO0LPeGAJkI0dXccwwoapzI1A/gYUH9VyD3CLbxrVm7CnSQ1IVedgd8t85+sa+YgfWAGq6gngv9jdLp8CKwF8o26eAW5VO8LNv4EXAhwjD7gN+Mr3AHRfseQ/Y/fZrxeRjb71sxZjd+OsBT5W1VWqehxY4nsY+zyGEQbMFLiGEYCITMLu2rm7ustiGOUxLXPDMIwawLTMjRpHRD4BWp+z+UFVnV0d5TGMUDCVuWEYRg1gulkMwzBqAFOZG4Zh1ACmMjcMw6gBTGVuGIZRA/x/XHbtxEFuZU4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(df_scores_pivot, annot=True, fmt=\".3f\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a7e8e9",
   "metadata": {},
   "source": [
    "* Train the final Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "ff503cf0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6eeac79",
   "metadata": {},
   "source": [
    "## Ensembles and Random Forests\n",
    "* Board of experts\n",
    "    * The idea of a Random Foret is to have several Decision Trees and let each decide by its own, we then take the average of there decisions if we have a regression problem or the majority class if we have a classification problem\n",
    "    * Why Random? If we use the same parameters for each tree, all would be the same, and then the result of each would be the same\n",
    "    * In a Random Forest each tree gets a random subset of features, e.g. we have a total of 10 features, but each tree gets only 7 of them and these are different for each tree.\n",
    "* Ensembling models\n",
    "* Random forest - ensembling decision trees\n",
    "![Random Forest](Screenshot_6.png)\n",
    "* Tuning a random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "17b2053f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use sklearn to train a random forest\n",
    "from sklearn.ensemble import RandomForestClassifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "30d98ada",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(n_estimators=10)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(n_estimators=10)\n",
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "590aca57",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = rf.predict_proba(X_val)[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "d65b59cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7859824174759633"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6dc31fe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.9, 0.1]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply the model to an example\n",
    "rf.predict_proba(X_val[[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9956d78",
   "metadata": {},
   "source": [
    "* The Random Forest gives different results each time we train again, because of the random features in it\n",
    "* To avoit this we can set a random seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "320323d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.9, 0.1]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf = RandomForestClassifier(n_estimators=10, random_state=1)\n",
    "rf.fit(X_train, y_train)\n",
    "\n",
    "y_pred = rf.predict_proba(X_val)[:,1]\n",
    "\n",
    "roc_auc_score(y_val, y_pred)\n",
    "rf.predict_proba(X_val[[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a9983e",
   "metadata": {},
   "source": [
    "* Try different number of estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "194c828a",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "\n",
    "for n in range(10, 201, 10):\n",
    "    rf = RandomForestClassifier(n_estimators=n, random_state=1)\n",
    "    rf.fit(X_train, y_train)\n",
    "\n",
    "    y_pred = rf.predict_proba(X_val)[:,1]\n",
    "\n",
    "    auc = roc_auc_score(y_val, y_pred)\n",
    "    scores.append((n, auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6c096b8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd+ElEQVR4nO3daYxc5b3n8e+/931fsNuN2zi2sUPAIQ2TFUhIWHKTyyQjXUAzo4i5CkICDZklCplI914przKZ5BXRRYyCuDPKhGQG7g3R5Q5NQkJmMlncgB27bQzeu9t27/tay39e1Ol2uel2l3txdZ/z+0ilqnrqnPK/T5V/deo5z3nK3B0REQmvnGwXICIi60tBLyIScgp6EZGQU9CLiIScgl5EJOTysl3AYurq6rylpSXbZYiIbBpvvvlmv7vXL/bYhgz6lpYW2tvbs12GiMimYWZnl3pMXTciIiGXUdCb2X1mdtzMTpjZU4s8XmlmPzezQ2bWYWaPBO3NZvYrMzsWtD+51n+AiIhc2bJBb2a5wA+A+4F9wMNmtm/BYo8DR939FuAu4HtmVgDEgf/g7nuBjwKPL7KuiIiso0z26G8HTrj7KXefBV4AHliwjAPlZmZAGTAIxN39gru/BeDuY8AxoGnNqhcRkWVlEvRNQGfa/S7eH9ZPA3uB88Bh4El3T6YvYGYtwIeBPyz2j5jZo2bWbmbtfX19mVUvIiLLyiTobZG2hTOh3QscBLYC+4Gnzaxi/gnMyoAXga+5++hi/4i7P+vure7eWl+/6AghERFZgUyCvgtoTru/jdSee7pHgJc85QRwGrgRwMzySYX8j9z9pdWXLCIiVyOToD8A7DKzHcEB1oeAlxcscw64G8DMGoE9wKmgz/6HwDF3//7alS0iEh4jkzF+drCbv/31yXV5/mVPmHL3uJk9AbwK5ALPuXuHmT0WPP4M8G3geTM7TKqr5xvu3m9mnwT+NXDYzA4GT/mf3P2VdfhbREQ2BXfnRO84v3ynl9ff6eXNs0Mkks6WyiK++qkd5OWu7SlOthF/eKS1tdV1ZqyIhMl0LMEfTg/y+rEeXj/eS+fgFAD7tlTwmRsb+MzeBm7ZVkVuzmKHRZdnZm+6e+tij23IKRBERMKgZ3SaX73Tyy/f6eW3J/qZnE1QlJ/DJ3bW8didO/n0nga2VhWvex0KepENzN0ZnY4zMD5D//hs6npilv6xGQYmZhgYn6V//NL1bCJJSUEexfm5FBfkUlKQu+B2Xuo6aL/8dh5bqor44NYKCvNys/2nb0rJpHO4eyTokunhSHdqkOHWyiK+fGsTd9/YyMd21lKUf223r4JeJAvcneHJGOdHprgwPM2FkSnOj0zTOzqTCu4gxAfGZ5lNJBd9juqSfGrLCqkrK2Dv1grqSgsoyMthKpZgcjbB1GxwHUswMD5L52yc6ViSydk4k7MJZuKLP29BXg63bKvkI9traN1ezUe2V1NdWrAu22FgfIYj50c50j3Cke4RxmfiNJQXcV1lIY0VRfOX6yqKqCsrWPO+65VKJp0Lo9Oc7Z/g9MAEZ/onODMwydvnhukfn8EMbr2+mq/fu4e79zawp7Gc1NiU7FDQiwRm40mmZhPk5Rp5uUZ+Tg45K+wvnZiJc2Fkiu7haS4Mp0L8wvAUF0am58N9Kpa4bJ38XKO+rJC68kLqywrZe13FfJDXlRVSW1ZAbWkhdeUF1JSsPvSSSZ//UJgOrk/3j9N+Zoj2s0P88P+e4pk3UsfwdtaX0rq9ho+0VNO6vZoddaVXHVy9Y9NBoI9yuHuEju4Rzo9Mzz/eUltCVUkBJ3v76R2bIZ68/PhhjkFdWSHXVRZd+jAoL6Kx8tKHQU1pwfw3lJX2dadvn56xaU73T3Cmf5Iz84E+wdmBycs+KAvyctheU8LHdtby6T313LWngZp1+nBcCR2MlUhyd84NTnKwc5i3zw1zsHOYo+dH37f3nGOQl5tDQW5O6gMgJ4f8tA+C+ba8HPJzjPGZOOeHpxidjl/2PGbQUF7IlspitlYVBdfFbK0sYktwXVdWuOIPlvUwHUtwqHOY9rNDvBlcRqZiANSWFnDr9lTot7ZUc1NT5Xx3j7tzcXSaw10jHDk/Skf3CIe7R+gdmwFS22JHXSkfaqrkpq2V3NRUyb6tFVQW58//28mkMzAxS8/oND2j01wcnaZndIaekbnbqcvQZGzJ+gtycyjKz0l1ZRXkUpSfS3F+zvwHQer+pa6r4oJcpmKJVJj3T3J2cILp2PvDfHttKTvq5q5LaakrZUtFUdZfuysdjFXQy6qMTMV46+wQfzg9yFtnhygryuOOXXXcuaeBltqSrH5dTTc8OcvBzuH5y6HO4fmQKM7P5UPbKtnfXMV1FUXEk0liCSeWSBJPOLFk6jqeSBJLOrF4knjy0uPpy5cU5KUF+aXrxooi8jdIt8NKJZPOyb5x2s8O0X5miDfPDnJmYBJIheDNTZWUFuZxpHuEgYlZIPVB+YGGMm5aEOplhWvTmTAdS9A3NjMf/kMTs0zHkkzFUl1WU0EX1tz96diC+8Htua6sgtwcrq8toaW2lJbaElrqUmG+vbaELZXFq/6WsJ4U9CEXSyR5/Z1e/uHtbhJJZ8915exuLGfPdeXsqCtd04DpHZ3mj2cGOXB6kD+eGeKdi6O4Q16OcVNTJSNTMU73TwBwfU0Jd+6u587d9XxsZy2la/Sfezmz8STHLoxeFuxzNZnBroYy9jdXsb+5mv3NVexuLNswfb+bTd/YTLC3P0j72SGmY0lu2lrBh7ZV8sGtlezbUkFxweY4sJsMuoqyvWe+Ugr6kDrZN85PD3Ty4lvd9I/PUF9eSGVxPqf7J0gEb9r8XGNnfdl88O9uLGdPYznbqouXfUO7O2cGJoNQH+TAmUHOBntwxfm5fGR7Nbe11HDbjmo+3Fw9/x/67MAEv3m3jzfe7eP/nRxgcjZBfq5xW0sNdwTBf+N1qz84NRNP0Dk4Nd9verp/go7zo5d1wTSUF6ZC/foq9m+r4kPbKikvyl/mmUU2HwV9iEzMxPnHwxf46YFO2s8OkZtjfObGBh5sbeauPfXk5eYwE09wsneCd3vGON4zxrsXU9ddQ1Pzz1NSkMuuxnL2NF7+IdA/PsOB04McODPEH88M0hf0q1aX5NPaUsPtLTXctqOGD26tyOibwkw8wZtnhngjCP53Lo4B0FhRyB276rlzTz2f/EAdVSWLH7iajSc5NzjJ2SDI5w6Ene6f4PzwFOnH6yqL89nTWJ4K9ebUZUtl0YbpPhJZTwr6Tc7debtzmJ8e6OTnh84zMZvghrpS/uK2Zr58axMN5UUZPc/YdIz3esfng//dnjGOXxynf3zmfcturSzith013NZSwz/bUcPO+rI1+Up7cWSa37yXCv3/824fo9Nxcgxuaa7izt31lBflz++hnxmYoHvo8jCvKMoL+kxLg/7T4KBYbem6DQEU2QwU9JtU//gM//B2Nz850Ml7veMU5+fyZzdv4cHbmmndXr1me6oD4zMc7xnjvZ5xKorzuK2lhm3VJWvy3FcSTyQ51DUyv7f/p65h3KE8Lcx3BAfEWupKaaktpbokX3voIotQ0G8iiaTzm3f7+MmBTn5xrId40tnfXMWDtzXzhZu3hLp/eXhylkTSqSktUJiLXCXNdbOBTczEU90pPWO8c2GMVw5f4OLoNDWlBXzl4y08eFszuxvLs13mNbFUP72IrI6C/hqZjiU4EQT6uz1z15cfIC3Iy+FjN9TyV1/cx2f3NlKQpyF/IrJ6Cvo1NhNPcKovNeLlvZ7xoO97jHODk/MHFfNzjRvqUmO5H2xtZldjObsby9heW7qhT8gQkc1JQb+Gvvrf2nn9nd75Mey5OUZLbQn7tlbwwP6mYBhjKtA3+1mSIrJ5KOjXyInecV472sOf3byFe/Y1zp+VquleRSTbFPRr5LWjPQB86/N7r8kPCYiIZEr9B2uk7ehFbt5WqZAXkQ1HQb8GekenefvcMPfsa8x2KSIi76OgXwNtQbfNPR+8LsuViIi8n4J+DbQd7WFHXSm7GsqyXYqIyPso6FdpdDrG7072c8++Rp22LyIbkoJ+lX59vI9Ywrnng+qfF5GNSUG/Sm0dF6krK2R/c3W2SxERWZSCfhVm4gl+fbyPz+1r0NQFIrJhKehX4XcnBxifiXPPPo22EZGNS0G/Cm1HeygtyOVjO2uzXYqIyJIU9CuUTDqvHe3hrj0NFOVrPhsR2bgU9Cv0ducwfWMzGm0jIhuegn6F2o5eJD/X+PSNDdkuRUTkihT0K+DutHX08NEbaqkI8W+4ikg4KOhX4GTfOKf7JzS3jYhsCgr6FXi1IzWJ2ef2qn9eRDY+Bf0KtB3t4ZbmKq6rLMp2KSIiy8oo6M3sPjM7bmYnzOypRR6vNLOfm9khM+sws0fSHnvOzHrN7MhaFp4tF0emOdSpuedFZPNYNujNLBf4AXA/sA942Mz2LVjsceCou98C3AV8z8wKgseeB+5bq4Kz7bVjqW6bezWsUkQ2iUz26G8HTrj7KXefBV4AHliwjAPllpqntwwYBOIA7v6b4H4otHVc5Ib6Uj7QUJ7tUkREMpJJ0DcBnWn3u4K2dE8De4HzwGHgSXdPXk0hZvaombWbWXtfX9/VrHrNjEzF+N3JAc1tIyKbSiZBv9i0jL7g/r3AQWArsB942swqrqYQd3/W3VvdvbW+vv5qVr1mfn28l3hSc8+LyOaSSdB3Ac1p97eR2nNP9wjwkqecAE4DN65NiRtHW0cP9eWF7N9Wle1SREQylknQHwB2mdmO4ADrQ8DLC5Y5B9wNYGaNwB7g1FoWmm3TsQS/Pt7L5/Y1kqO550VkE1k26N09DjwBvAocA37q7h1m9piZPRYs9m3g42Z2GPgl8A137wcwsx8DvwP2mFmXmf3levwh6+13JweYmE1oWKWIbDp5mSzk7q8Aryxoeybt9nngniXWfXg1BW4UbUcvUlaYp7nnRWTT0ZmxGUjMzz1fT2Ge5p4Xkc1FQZ+Bg51D9I/Pcq8mMRORTUhBn4FXO3rIzzXu2rMxh32KiFyJgn4Z7s6rHRf5+M46yjX3vIhsQgr6ZbzXO87ZgUmdJCUim5aCfhltHRcBzT0vIpuXgn4ZbUd7+PD1VTRUaO55EdmcFPRXcH54ij91jWgSMxHZ1BT0V/CLYO559c+LyGamoL+Cto4ePtBQxs76smyXIiKyYgr6JYxMxvj9qQHNbSMim56CfgmvH+8J5p5X/7yIbG4K+iW0dfTQWFHIzU2V2S5FRGRVFPSLmI4leOPdPs09LyKhoKBfxG9P9DM5m9CwShEJBQX9Ito6eigvzOOjN2jueRHZ/BT0CySSzi+O9fDpGxsoyNPmEZHNT0m2wFvnhhiY0NzzIhIeCvoF2jouUpCbw52ae15EQkJBnyY193wPn/hALWWFGf2crojIhqegT3O8Z4xzg5M6SUpEQkVBn6atowczuHtvQ7ZLERFZMwr6NG1HL3Lr9dU0lGvueREJDwV9IJZIcvT8KB+9oSbbpYiIrCkFfeDiyDRJh+trSrJdiojImlLQB7qGpgDYVq2gF5FwUdAHuoYmAdhWXZzlSkRE1paCPtA1NIUZbKlU0ItIuCjoA11DU1xXUaT5bUQkdJRqga6hSZqqtDcvIuGjoA90D0+pf15EQklBD8QTSS6MTGvEjYiEkoIeuDg6TSLp2qMXkVBS0KMx9CISbgp60oNee/QiEj4KeqA7CPotVZrMTETCJ6OgN7P7zOy4mZ0ws6cWebzSzH5uZofMrMPMHsl03Y2ga2iSxopCCvNys12KiMiaWzbozSwX+AFwP7APeNjM9i1Y7HHgqLvfAtwFfM/MCjJcN+u6hqbUPy8ioZXJHv3twAl3P+Xus8ALwAMLlnGg3MwMKAMGgXiG62Zd1/Ck+udFJLQyCfomoDPtflfQlu5pYC9wHjgMPOnuyQzXBcDMHjWzdjNr7+vry7D81YsnklwYnlbQi0hoZRL0tkibL7h/L3AQ2ArsB542s4oM1001uj/r7q3u3lpfX59BWWujZ2yGeNJpqlLXjYiEUyZB3wU0p93fRmrPPd0jwEuecgI4DdyY4bpZ1a2hlSIScpkE/QFgl5ntMLMC4CHg5QXLnAPuBjCzRmAPcCrDdbNK89CLSNjlLbeAu8fN7AngVSAXeM7dO8zsseDxZ4BvA8+b2WFS3TXfcPd+gMXWXZ8/ZWXmTpbaqpkrRSSklg16AHd/BXhlQdszabfPA/dkuu5G0jU0SUN5IUX5GkMvIuEU+TNjU2PotTcvIuEV+aDvHp6iSSdLiUiIRTroE0nnvH5wRERCLtJB3zs2TSyheehFJNwiHfSah15EoiDiQa8x9CISftEO+sHUHn2TxtCLSIhFOui7h6eoK9MYehEJt0gHvcbQi0gURDzoNQ+9iIRfZIM+mXS6h/XLUiISfpEN+t6xGY2hF5FIiGzQzw2tbFLQi0jIRTbou4dTQyubFfQiEnKRDfq5s2L1E4IiEnYRDvpJ6soKKC7QGHoRCbcIB72mJxaRaIh00G/T1AciEgGRDPpLY+gV9CISfpEM+v7xGWbjSQW9iERCJIO+U/PQi0iERDLoNQ+9iERJRIM+GEOvoBeRCIhs0NeUFlBSkJftUkRE1l0kg14jbkQkSiIZ9JqHXkSiJHJB7+50D2keehGJjsgFfd/4DDMaQy8iERK5oO+aH0OvoBeRaIhc0HdremIRiZjIBb3G0ItI1EQw6CepLsmnrFBj6EUkGiIY9BpxIyLREsGg1xh6EYmWSAW9e2oe+ib94IiIREhGQW9m95nZcTM7YWZPLfL4183sYHA5YmYJM6sJHnsyaOsws6+tcf1XZWBilumYxtCLSLQsG/Rmlgv8ALgf2Ac8bGb70pdx9++6+3533w98E3jD3QfN7Cbgq8DtwC3AF8xs1xr/DRnr0jz0IhJBmezR3w6ccPdT7j4LvAA8cIXlHwZ+HNzeC/ze3SfdPQ68AXxpNQWvxvw89DXaoxeR6Mgk6JuAzrT7XUHb+5hZCXAf8GLQdAS4w8xqg8c+DzQvse6jZtZuZu19fX2Z1n9V5sfQq49eRCIkk6C3Rdp8iWW/CPzW3QcB3P0Y8B3gNeB/A4eA+GIruvuz7t7q7q319fUZlHX1uoYmqSrJp7wof12eX0RkI8ok6Lu4fC98G3B+iWUf4lK3DQDu/kN3v9Xd7wAGgfdWUuha6B7SiBsRiZ5Mgv4AsMvMdphZAakwf3nhQmZWCdwJ/GxBe0NwfT3wZRZ8EFxLqZOlFPQiEi3LzgPg7nEzewJ4FcgFnnP3DjN7LHj8mWDRLwFt7j6x4CleNLNaIAY87u5Da1d+5tydrqEp7ti9Pt1CIiIbVUYTvrj7K8ArC9qeWXD/eeD5Rdb91MrLWzuDE7NMxRLaoxeRyInMmbEaQy8iURW5oNfBWBGJmsgEffdw6mQpzUMvIlETmaDvGpqioiiPymKNoReRaIlU0Kt/XkSiKEJBr3noRSSaIhH0c2PotUcvIlEUiaAfmowxOZvQgVgRiaRIBH33/Bh6Bb2IRE8kgn5+HnoFvYhEUESCXmfFikh0RSToJynXGHoRiaiIBL3moReR6IpE0HcPa2iliERX6IP+0hh67dGLSDSFPuhHpmKMz8QV9CISWaEPeo24EZGoi0DQawy9iERbBIJeZ8WKSLRFIujLCjWGXkSiKxJBv626GDPLdikiIlkRgaDXPPQiEm2hDnp3p1vz0ItIxIU66Een4oxpDL2IRFyog75rODW0UvPciEiUhTvodbKUiEhUgl579CISXSEP+klKC3KpKtEYehGJrpAHfWrEjcbQi0iUhT7om9RtIyIRF+qg79bJUiIi4Q36kakYo9MaQy8iEtqg79bQShERIMRBr3noRURSQhz02qMXEYEMg97M7jOz42Z2wsyeWuTxr5vZweByxMwSZlYTPPbvzKwjaP+xmRWt9R+xmO7hKYrzc6nWGHoRibhlg97McoEfAPcD+4CHzWxf+jLu/l133+/u+4FvAm+4+6CZNQH/Fmh195uAXOChNf4bFjU3PbHG0ItI1GWyR387cMLdT7n7LPAC8MAVln8Y+HHa/Tyg2MzygBLg/EqLvRpzPzgiIhJ1mQR9E9CZdr8raHsfMysB7gNeBHD3buC/AOeAC8CIu7ctse6jZtZuZu19fX2Z/wVL6NI89CIiQGZBv1jfhy+x7BeB37r7IICZVZPa+98BbAVKzexfLbaiuz/r7q3u3lpfX59BWUsbnY4xMhXTHr2ICJkFfRfQnHZ/G0t3vzzE5d02nwVOu3ufu8eAl4CPr6TQqzE3hl7TH4iIZBb0B4BdZrbDzApIhfnLCxcys0rgTuBnac3ngI+aWYmljoreDRxbfdlXppOlREQuyVtuAXePm9kTwKukRs085+4dZvZY8PgzwaJfAtrcfSJt3T+Y2f8C3gLiwNvAs2v8N7yPTpYSEblk2aAHcPdXgFcWtD2z4P7zwPOLrPvXwF+vuMIV6Bqaoig/h9rSgmv5z4qIbEihPDNW89CLiFwSzqAf1vTEIiJzwhn0Q1M0VSnoRUQghEE/PhNneDKmETciIoHQBf2loZXaoxcRgRAGvYZWiohcLoRBr5OlRETShTDoJynMy6GuTGPoRUQghEHfPTxFk+ahFxGZF7qg1/TEIiKXC2nQ60CsiMicUAX9xEycwYlZBb2ISJpQBX33sEbciIgsFKqgnxtDr+kPREQuCVXQz50V26yuGxGReaEK+q6hKQrycqgrK8x2KSIiG0bogn5bVTE5ORpDLyIyJ2RBP6kfBBcRWSBkQa+TpUREFgpN0CeTzp2767l9R3W2SxER2VAy+nHwzSAnx/j+g/uzXYaIyIYTmj16ERFZnIJeRCTkFPQiIiGnoBcRCTkFvYhIyCnoRURCTkEvIhJyCnoRkZAzd892De9jZn3A2WzXsYQ6oD/bRVyB6lsd1bc6qm91VlPfdnevX+yBDRn0G5mZtbt7a7brWIrqWx3Vtzqqb3XWqz513YiIhJyCXkQk5BT0V+/ZbBewDNW3OqpvdVTf6qxLfeqjFxEJOe3Ri4iEnIJeRCTkFPRLMLNmM/uVmR0zsw4zezJo/xsz6zazg8Hl81ms8YyZHQ7qaA/aaszsNTN7L7jOyk9umdmetG100MxGzexr2d5+ZvacmfWa2ZG0tiW3mZl908xOmNlxM7s3S/V918zeMbM/mdnfm1lV0N5iZlNp2/KZLNW35Gu6QbbfT9JqO2NmB4P2a7r9rpAp6//+c3ddFrkAW4Bbg9vlwLvAPuBvgP+Y7fqCus4AdQva/jPwVHD7KeA7G6DOXOAisD3b2w+4A7gVOLLcNgte70NAIbADOAnkZqG+e4C84PZ30uprSV8ui9tv0dd0o2y/BY9/D/irbGy/K2TKur//tEe/BHe/4O5vBbfHgGNAU3arysgDwN8Ft/8O+OfZK2Xe3cBJd8/62c7u/htgcEHzUtvsAeAFd59x99PACeD2a12fu7e5ezy4+3tg23rWcCVLbL+lbIjtN8fMDPgL4MfrWcNSrpAp6/7+U9BnwMxagA8Dfwiangi+Rj+Xra6RgANtZvammT0atDW6+wVIvbGAhqxVd8lDXP6fa6NsvzlLbbMmoDNtuS6y/2H/b4B/Sru/w8zeNrM3zOxT2SqKxV/Tjbb9PgX0uPt7aW1Z2X4LMmXd338K+mWYWRnwIvA1dx8F/hbYCewHLpD6Kpgtn3D3W4H7gcfN7I4s1rIoMysA/hz4n0HTRtp+y7FF2rI2HtnMvgXEgR8FTReA6939w8C/B/6HmVVkobSlXtMNtf2Ah7l8hyMr22+RTFly0UXaVrT9FPRXYGb5pF6QH7n7SwDu3uPuCXdPAv+Vdf4qeiXufj647gX+Pqilx8y2AATXvdmqL3A/8Ja798DG2n5pltpmXUBz2nLbgPPXuDYAzOwrwBeAf+lBB27wlX4guP0mqT7c3de6tiu8phtp++UBXwZ+MteWje23WKZwDd5/CvolBP15PwSOufv309q3pC32JeDIwnWvBTMrNbPyudukDtgdAV4GvhIs9hXgZ9moL81le1EbZfstsNQ2exl4yMwKzWwHsAv447UuzszuA74B/Lm7T6a115tZbnD7hqC+U1mob6nXdENsv8BngXfcvWuu4Vpvv6UyhWvx/rtWR5w32wX4JKmvSX8CDgaXzwP/HTgctL8MbMlSfTeQOiJ/COgAvhW01wK/BN4LrmuyuA1LgAGgMq0tq9uP1IfOBSBGao/pL6+0zYBvkdrTOw7cn6X6TpDqq517Hz4TLPsvgtf+EPAW8MUs1bfka7oRtl/Q/jzw2IJlr+n2u0KmrPv7T1MgiIiEnLpuRERCTkEvIhJyCnoRkZBT0IuIhJyCXkQk5BT0IiIhp6AXEQm5/w8UwNXmctdZogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_scores = pd.DataFrame(scores, columns=[\"n_estimators\", \"auc\"])\n",
    "plt.plot(df_scores.n_estimators, df_scores.auc);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e287048",
   "metadata": {},
   "source": [
    "* Performance is improving util ~50 trees\n",
    "* We can now tune the Random Forest, the parameters we tune are the same as in Decision Trees as the Random Forest constits of Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "2af22eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "\n",
    "for d in [5, 10, 15]:\n",
    "    for n in range(10, 201, 10):\n",
    "        rf = RandomForestClassifier(n_estimators=n, \n",
    "                                    max_depth=d,\n",
    "                                    random_state=1)\n",
    "        rf.fit(X_train, y_train)\n",
    "\n",
    "        y_pred = rf.predict_proba(X_val)[:,1]\n",
    "\n",
    "        auc = roc_auc_score(y_val, y_pred)\n",
    "        scores.append((d, n, auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "e08176dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>0.787699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>20</td>\n",
       "      <td>0.797731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>30</td>\n",
       "      <td>0.800305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>40</td>\n",
       "      <td>0.799708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "      <td>0.799878</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max_depth  n_estimators       auc\n",
       "0          5            10  0.787699\n",
       "1          5            20  0.797731\n",
       "2          5            30  0.800305\n",
       "3          5            40  0.799708\n",
       "4          5            50  0.799878"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns=[\"max_depth\", \"n_estimators\", \"auc\"]\n",
    "df_scores = pd.DataFrame(scores, columns=columns)\n",
    "df_scores.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "afb45c1e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABFeUlEQVR4nO3deXxU5b348c93su8LWSCBAELYwxqxrC4UxBW1ti5tpbR1abXV3rZKe69XvfbX0tba1VuXq2Krrd3cq1VEK7iyKCYQtrBnITtJyD6Z7++PMwkhJGSAhEmY7/v1Oq+ZeeacM985ged75jnPeR5RVYwxxgQel78DMMYY4x+WAIwxJkBZAjDGmABlCcAYYwKUJQBjjAlQwf4O4EQkJSXpiBEj/B2GMcYMKBs3bixX1eTO5QMqAYwYMYINGzb4OwxjjBlQRGRfV+XWBGSMMQHKEoAxxgQoSwDGGBOgLAEYY0yAsgRgjDEByhKAMcYEKJ8SgIgsFpHtIpIvIsu7eD9ORF4WkU9FZIuILPOWDxORt0Vkq7f89g7b3CsihSKyybtc3HtfyxhjTE96vA9ARIKAh4CFQAGwXkReUtW8DqvdCuSp6mUikgxsF5FnADfwXVX9WERigI0isqrDtr9U1Qd69RsZY84cqlCwAfa9B4NGQ9pUiE0HEX9Hdkbw5UawmUC+qu4GEJFngSVAxwSgQIyICBANVAJuVS0GigFUtVZEtgLpnbY1xpijle2A3L9C7t+gau/R70Ulw5CpkDbNSQhDpkJsmiWFk+BLAkgHDnR4XQCc02md3wEvAUVADHCNqno6riAiI4BpwEcdim8TkRuADTi/FKpOKHpjzJmjpgg2/8Op9Is/BXHByPkw//uQuQiq9kHxJij6BIo2wa7V0FbNRKU4ySBt2pHkEDvk1OJpbYGmWmiqgZYGiEiAyCQI6sMBFBqrne95aD8c8j5W7XOeX/5bGJrdqx/nyzfpKq12nkbsQmATcAEwClglImtVtQZARKKBfwB3tJUBvwfu9+7rfuAXwFeP+XCRm4CbADIyMnwI1xgzYDQcgq0vOZX+nrWAOpX3hT+BSVdBzOAj60anwLCzj7xuroeSzUcSQtEnkP/mkaQQnXokIaSMB4/bqcybaqHR+9hWwTd1eN32nruh65gjEp1YopIhKslJPlHJEJ3sLeuwhEUfvW1znbdy71CxH9p3pNJvPHT0+qHRED8cEkY4CbGX+ZIACoBhHV4PxTnT72gZsEKd+SXzRWQPMA5YJyIhOJX/M6r6XNsGqlrS9lxEHgNe6erDVfVR4FGA7Oxsm7/SmBPR0ghbnnfOlqNTncokPgMSvI+hUf6JaefrTqW/4w1obYKEkXDunZD1eUjK9G0/oZEwbKaztGmug4PepND2a2HnG0eSQhtxQVgMhMV6lxjn7D5hJITHdngvxlmCw53K+XAZ1JVBXSnUlcPBXKesqbrrGEMinSQRFge1xVBffvT7weHO3yF+OAw92/t3afsbjXB+dfRh05YvCWA9kCkiI4FC4Frg+k7r7AcWAGtFJBUYC+z2XhN4HNiqqg923EBEhnivEQBcCWw++a9hjDlKdQFseAI2roT6CucstakG3I1HrxeZdCQZxA/v8HwExA+D4LDeicfTCnvfddr18152KsyoZMheBllfgPTpvVPRhUZBxjnO0qbpMFTuguCII5V7SGTvVqzuJichtCWGujI4XOpNFmXOL530aUfO5tuOd3SKX69d9JgAVNUtIrcBrwNBwBOqukVEbvG+/zBOE85KEcnFaTK6S1XLRWQu8GUgV0Q2eXf5Q1V9FfiZiEzFaQLaC9zcq9/MmECj6lSy6x6Fbf8EFMZcBDNvhLPOc9Y5XNqhfXnfkWaI4k9h6yvgaTl6nzFDnIoqdgggoK1OZa4ep0nF09qprOPrVvB4nMe2ijA0GsZf5pzpjzy3b9vT24RFw5ApffsZwWEQl+4sA4g4rTYDQ3Z2ttpw0MZ00lwHOX+BdY9BaZ7TbDD9Bsj+mnNG7ytPK9QePPbi46H9TvMFgASBK+jIY8fnEgQu17HriMup+MdcCGMvgpCIvjkOplsislFVj7mCPKDmAzDGdFC5G9b9H3zytNOkMjgLLv8dZF19cpWsK+jIWezw2b0fr+l3LAEYM5B4PLDrLVj3COxc5VTa4y+Hc26GYedYX3hzQiwBGDMQNFbDpj85zTyVu5yLuufeCTOWnXp/dxOwLAEY09+0uqFiJxTnwMEc5wJt4UZoqYehM+G8H8CEJRAc6u9IzQBnCcAYf2ppgJI8OPjpkQq/ZMuR7prB4ZAyAaZeD9O+5NzYZEwvsQRgzOnSUNXhrD7HuYmofIfTTRIgPA4GT4azv+5c0B08GZLGnJ6ukiYg2b8sY/pSfSV89Ah8+ienO2WbmDQYMtnpEz9kslPZx2fYRVxzWlkCMKYvHC6DDx9yumk218LohZD9VaeiHzLFGR7AGD+zBGBMb6ophvd/AxuedNrxJ10F874LqRP9HZkxx7AEYExvOLQf3v0VfPJH547aydfAvP/wfWAzY/zAEoAxp6JiF6x9EHKeBcTpqTP3DmfAL2P6OUsAxpyM0q2w9hfOBCZBoU7PndnfHnCDgZnAZgnAmBNR/Cms+TlsfRlComDWbTD7W86wvsYMMJYAjPHFgfVOxb/zdWdyj/l3wme+AZGJ/o7MmJNmCcCY7rS6Yfs/4cOHYf/7zlSAF/wXnH0jRMT7OzpjTpklAGM6azjk9Ob56FGo3u/coHXhj2H60mPneDVmALMEYEyb8nz46GFn1M2WOhg+Fxb/GMZe7Ay7bMwZxhKA6f/czc54OX0xk5SqM77+Rw87k4cHhTrTFZ5zizNEgzFnMJ8SgIgsBn6NMyfw/6nqik7vxwFPAxnefT6gqk+KyDDgD8BgwAM8qqq/9m6TCPwFGIEzJ/AXVLWqF76T6W9UofqA07TSVOOMbd9Y0+F5dRflNUfK20bGTBh5ZJC0wVnOEpt2cuPnNNc70yh+9DCUbXPG1z/vB85wDdajxwSIHucEFpEgYAewECgA1gPXqWpeh3V+CMSp6l0ikgxsx6n0BwFDVPVjEYkBNgJXqGqeiPwMqFTVFSKyHEhQ1buOF4vNCTzAqDqzVr39I6f7ZHeCI5yRMMNjISzWeQyPO/q5xwOlW5wRNCt3H9k2IvFIMmhbksZAUEjXn1VdCOsfg40rndE5B0+Gz3zTGbIhOKxXv74xp6qupY78Q/nsOrSL+UPnkxRxcmNIncqcwDOBfFXd7d3Rs8ASIK/DOgrEiIgA0UAl4FbVYqAYQFVrRWQrkO7ddglwnnf7p4B/A8dNAGYA2bMG3voRHPjIuYi6eAXEDT1SqYfFQni887y7yro7TbXOmPkHc48s6//vyC+FoFBIGe8kg1RvUhBxZtPKexFQGHeJU/FnzLIROM1xqSql9aXEhMYQGRLZJ5/R6G5kT/Ue8g/lH1mq8imqK2pf51fn/YoFwxf06uf6kgDSgQMdXhcA53Ra53fAS0AREANco6qejiuIyAhgGvCRtyjVmyBQ1WIR6fJ3t4jcBNwEkJGR4UO4xq8OrIe37oc970DMELjkQZj25d6dvSosBjI+4yxtWt1Qke9NCN6x9rf/y5kwvX27OKfv/sybIGF478Vjzihuj5ttldvYWLKRj0s+5pPST6hqclqnI4IjSIpIal8GhQ9yHiMGHVMe0sWJTYunhf01+9l5aCe7Du0iv8qp7PfX7sfjrTKDXcGMjBvJlJQpXB1/NaPiR5EZn0l6TO/fZe5LAujq9Khzu9GFwCbgAmAUsEpE1qpqDYCIRAP/AO5oK/OVqj4KPApOE9CJbGtOo+IcePv/wY5/QWQSXPgTyF7WNxduuxIUDCnjnGXy550yVThc4iSDhkMw9iLrxmmO0ehuJLc8t73C/7TsU+rd9QAMixnG/KHzmZg0kUZ3I+UN5ZQ3lFPRUMHuQ7tZ17iO6qbqLvcbFxZHUriTEKJDo9lfu5891Xtwe9wAuMRFRkwGo+NHs3jk4vaKPiM2gxDXCf4qPkm+JIACYFiH10NxzvQ7WgasUOeCQr6I7AHGAetEJASn8n9GVZ/rsE2JiAzxnv0PAUpP+lsY/ynbDm//GPJecNrqF/w3zLy5f1S0IhAz2FmM8apprmFT6SY2lGzg45KP2VKxBbfHjSCMSRjDktFLmJ46nekp00mJ7LlDQHNrM5WNle3JoeNS0VBBeUM5ZdVlDI0Zyrz0eYyOH83o+NGMjBtJeHD4afjG3fMlAawHMkVkJFAIXAtc32md/cACYK2IpAJjgd3eawKPA1tV9cFO27wELAVWeB9fPOlvYU6/yj3w7xWQ+1cIiXSGRph1q90ha3pFU2sTOWU5rDu4jnXF68iryCPYFUxEcAThweFEBEe0L22vI4Mju31fVckpy+Hj0o/ZWbUTRQl2BTNp0CRumHADM1JnMDVlKrGhsScca2hQKIOjBjM4auCdaPSYAFTVLSK3Aa/jdAN9QlW3iMgt3vcfBu4HVopILk6T0V2qWi4ic4EvA7kissm7yx+q6qs4Ff9fReRrOAnk87383UxfqC6ENT9z2tZdwU6lP+c7EDXI35GZAazF08KW8i1OhX9wHZtKN9HU2oRLXExInMDnxnwOQWhwN1DvrqfR3UiDu4EGdwNVjVXtz9vK3eo+5jMigiOYmjyVRVMXMT11OllJWX4/A/e3HruB9ifWDdSPDpfCu7+E9Y+DemDGV2D+96x5xZyUVk8r26u2s654HR8d/IiPSz5ub3cfmzCWswefzTlDzmFG6gxiQmNOeP8trS00tDbQ0OIkBo96yIjNINgVmPe+nko3UBPIGqrg/d/Ch78HdxNMvR7OvdPp2mmMj1SVXYd28dHBj1hXvI4NJRuoaXb6g4yMG8lloy5j5uCZnD34bBLCE07580KCQggJCjmpJp1AYgnAdK25zrlL9r1fO3flTvqcc6ds0mh/R2b8wKMe6lvqOdxymLqWOuexue6o112VtT2W15e3d6VMj05nQcYCZg6ZyczBM3260Gr6hiUAczR3s3OX7JqfQ10pjFkMF9wNgyf5OzLThzzqoay+jMLDhRQcLqCw1vt4uJDCw4WU1pe291M/nojgCKJCoogOiW5/HBY9jKykLKYmT+XswWczNGboafhGxheWAIzD0+qMjfPvnzgTnA+fC9c8DRmd7/kzA5GqUtNcQ0FtwZGKvbawvYIvOlxEs6f5qG1SIlJIj0knOzWbIVFDiAuLO7pyD40+6nVUSFTAtrEPVPbXGggqdsH212DYOZA2zbnpqbeoOtMbvvUjKN8OQ6bCpb+CURfYEAkDSKO7kZL6Eg7WHTyy1B886vXhlsNHbRMXFkd6dDqZCZmcP+x80qPTSY9JJz06nbToNMKCbGykM50lgIHg5dth71rneWgMDJ8NI+fByPnOWDcu14nvUxV2vw2r/weKPnEGUPvCH2D85Vbx9zOqysG6gxTVFR1TwZfUOZV+W/t6R4nhiaRGpjIsZpjT9BI9lPSYdOcxOp3o0H5ws57xK0sA/d3+D53K/9zlzjAHe9Y6A63tfN15PyIBhs+Bkec6CSF5bM8V+IF1TsW/dy3EDYMl/wuTr+ndXxbmlJU3lPPP3f/khfwXyD+Uf9R7MaExzs1HkYOZlDSp/UakwZHOY2pUqp3Bmx7Z//j+bs3PIXIQzPk2hEbBxCud8pqiI8lgzxrY9opTHpVy5NfBiHmQeNaRhFCyBVbfDzteg6hkuOhnTn9+Gwa532hpbWFNwRpeyH+BtYVradVWJidNZvnM5YyMHdleuUeFRPk7VHMGsATQnxVuhPw3YcE9TuXfUWwaTLnGWQCq9nqTgTcpbP6Hd72hTkJobYbNzznDMF9wtzPjVX8Yr8cAsK1yGy/mv8g/d/+TqqYqkiOSuWHiDVwx6grOij/L3+GZM5QlgP5szS+cMfPP/nrP6yaMcJbpNzjt++U7nSGZ966FHa9DSwPMvQNmfxsiE/s2buOTqsYq/rn7n7y460W2VW4jxBXC+cPOZ8noJcxOm209akyfs39h/dXBzbD9n87NV+EneDejCCSPcZaZNzqzaXlarKmnH2jxtPBe4Xu8kP8C7xS8g9vjZsKgCfzwnB9y0YiLiA+P93eIJoBYAuiv1j7g9Pg55+ZT35fLBS6r/P1FVdl5aCcv5b/EK7tfoaKxgsTwRK4fdz1LRi9hTMIYf4doApQlgP6obDtseQHmfsfp5WP6PVWlqqmK/TX72Vezj301+9hfu7/9db27nmAJZv7Q+Vwx+grmDp172ib9MKY7lgD6o7W/cGbSmnWrvyMxnVQ3VR9Vwe+r2cf+Gqeir22pbV8vSIJIi04jIzaDaSnTGBU/is8O/yyJ4Xb9xfQflgD6m8rdkPs3Z8LyqCR/RxOwaptr2VG1g+2V29letZ38qnz21e47avo/l7gYEjWE4bHDufisixkeO5zhscPJiMkgPSbdzvBNv2cJoL9Z+yC4QmD2t/wdSUBQVQoPF7ZX9G2PhYcL29dJCEsgMyGTRcMXHankYzMYGj2U0KBenOzemNPMEkB/cugAfPpnmLHMJlrpA43uRnYd2sW2ym3tlf2Oqh3tY+QIwvDY4UxKmsTVY65mTMIYxiaMJSUyBbHhMcwZyKcEICKLgV/jTAn5f6q6otP7ccDTQIZ3nw+o6pPe954ALgVKVXVSh23uBW4EyrxFbVNFBq73fgUIzLnd35EMaB71UHi4kPyqfHYe2snOqp3sqNrB3pq97UMaRwZHMiZhDJecdQljE8cyNmEso+NHExkS6efojTl9ekwAIhIEPAQsBAqA9SLykqrmdVjtViBPVS8TkWRgu4g8o6rNwErgd8Afutj9L1X1gVP9EmeEmmL4+I8w9TqIH+bvaAaMysZKdlbtJP9QPjurdrY/b5teEJwJSDLjM1k4fGF7ZT80ZiguOYlB9Iw5g/jyC2AmkK+quwFE5FlgCdAxASgQI87v5GigEnADqOoaERnRm0Gfkd7/LXjcMPc//B1Jv9TgbmDXoV1OJe89q99ZtZOKxor2deLD4slMyOSK0VeQmZBJZkImo+NH27g5xnTDlwSQDhzo8LoA6DxLyO+Al4AiIAa4RtWH6YPgNhG5AdgAfFdVjxnTVkRuAm4CyMg4Q+ehrSuHDU9A1uchcaS/o+kXWlpb2FS2iQ+KPuD9ovfZWrm1vfkmLCiMUfGjmJs+t72iH5MwhkHhg6yt3pgT4EsC6Op/lHZ6fSGwCbgAGAWsEpG1qlpznP3+Hrjfu6/7gV8AXz3mg1QfBR4FyM7O7vy5Z4YPfgfuRpj3XX9H4jeqyp7qPbxf9D7vF73PhpINNLgbCJIgJidP5sasGxmXOI7R8aMZFjOMIFeQv0M2ZsDzJQEUAB0bpYfinOl3tAxYoaoK5IvIHmAcsK67napqSdtzEXkMeMXXoM8o9ZWw7jGYeIUzdk8AqWys5MOiD/mg2DnLL60vBWB47HCWjFrCrLRZnD34bGJCY/wcqTFnJl8SwHogU0RGAoXAtcD1ndbZDywA1opIKjAW2H28nYrIEFUt9r68Eth8IoGfMT56BJoPw7zv+TuSPtfU2sQnpZ/wQdEHfFD0AVsrtwIQGxrLZ4Z8hllps5iVNov06HQ/R2pMYOgxAaiqW0RuA17H6Qb6hKpuEZFbvO8/jNOEs1JEcnGajO5S1XIAEfkzcB6QJCIFwD2q+jjwMxGZitMEtBfohVHPBpjGGvjo9zD2Ehg8qef1B6CDdQd5a/9brClYw8aSjTS2NhLsCmZq8lS+Ne1bzE6bzfjE8dakY4wf+HQfgLd//qudyh7u8LwIWNTNttd1U/5l38M8Q61/DBqrYf6Zdfa/+9BuVu9fzer9q9lSsQWAEbEj+NyYzzE7bTbZqdnW396YfsDuBPaX5jr44CEY/VlIn+7vaE6JRz1sLt/MW/vfYvX+1eyt2QtAVlIWt0+/nQsyLuCsOJvVypj+xhKAv2x4EuorYP6d/o7kpLR4Wlh/cD1v7X+Lt/e/TWlDKcESTPbgbL44/oucP+x8UqNS/R2mMeY4LAH4Q0sDvP8bZ9L2jM63VBzrhfwX+GPeH3F73HjUg6J41HPUoqp4OLasVVtRlCAJIiE8gcTwxPZlUMQg5zF80JHyiETiw+K7vEu2vqWe94veZ/X+1bxT8A61zbVEBEcwJ20OF2RcwPyh84kLi+uLI2aM6QOWAPzhk6fhcAl87v96XPXlXS9z93t3My5xHKPiR+ESFy5ciAhBEoSI4BLXkee4nHW8S9t6LZ4WKhsrqWyspOBwATllOVQ1VbXfXNWRS1wkhCWQGHEkWTS0NPBB8Qc0tTYRFxbH+cPOZ0HGAmalzSIiOKIvjpIxpo9ZAjjd3M3w7q9g2DnOL4DjWL1vNXe/dzczB8/koQUPER4c3quhtHpaqW6uprKhsj05VDRWUNFQ0f66srGSzeVOD93PZX6OBRkLmJ463SYsN+YMYP+LT7dP/ww1BXDZr53J27vxXuF7fG/N95iYNJHfXvDbXq/8AYJcQe1n+MaYwGPDIZ5OrW5490FImwajF3S72oaDG7jj7TsYFTeK/13wv9Zl0hjTJywBnE6b/w5Ve2H+97s9+99SvoXb3rqNwVGDeWThI3ZR1RjTZywBnC6eVljzAKROgjEXdbnKzqqd3PzmzcSHxfPYoscYFDHoNAdpjAkklgBOl7wXoWKnM+Kn69jDvq9mHzetuolQVyiPLXyMwVE2JaQxpm/ZReDTweNxzv6TxsCEJce8fbDuIDe+cSNuj5uVi1cyLNZmBDPG9D37BXA6bH0JSrd4z/6PHvSsvKGcG9+4kdrmWh5Z+Aij4kf5KUhjTKCxXwB9bd/78MI3IWUiTLr6qLeqm6q5adVNlNSX8MjCR5gwaIKfgjTGBCJLAH1p/0fwzOchdgh8+TkIOnK461rq+Mab32Bv9V4eWvAQ01Km+TFQY0wgsgTQVwo2wNOfg+hUWPoKxBy5qNvobuS21beRV5HHg+c9yKy0WX4M1BgTqCwB9IXCjfDHKyFqECx92fkF4NXS2sJ3/v0dNpZs5CfzfsIFGRf4MVBjTCCzi8C9rWiTU/lHxDtn/nFHpjd0e9zctfYu3i18l7tn3c0lZ13itzCNMcanBCAii0Vku4jki8jyLt6PE5GXReRTEdkiIss6vPeEiJSKyOZO2ySKyCoR2el9TDj1r+NnxTnwhyUQFutU/vFHunN61MO979/Lqn2r+F729/j8mM/7MVBjjPEhAYhIEPAQcBEwAbhORDp3V7kVyFPVKTjz//5CREK9760EFnex6+XAalXNBFZ7Xw9cBzc7lX9otNPskzC8/S1VZcW6Fby460W+OeWbLJ241I+BGmOMw5drADOBfFXdDSAizwJLgLwO6ygQIyICRAOVgBtAVdeIyIgu9rsEJ1kAPAX8G7jrhL9Bf1CSB3+4HILDYelLkDgScM76Pyz6kD9u/SPvFr7LDRNu4JYpt/g5WGOMcfiSANKBAx1eFwCdp7H6HfASUATEANeodjHTyNFSVbUYQFWLRSTFt5D7mbLtTuXvCoGvvAKDRlHbXMtLu17i2W3PsrdmL4nhidw+/Xa+NulryHGGgDbGmNPJlwTQVY2lnV5fCGwCLgBGAatEZK2q1pxaeCAiNwE3AWRkZJzq7npX+U546jJA4CuvsNPl4dkP7ufl3S/T4G5gSvIUfjLvJywavojQoNAed2eMMaeTLwmgAOg4OM1QnDP9jpYBK1RVgXwR2QOMA9YdZ78lIjLEe/Y/BCjtaiVVfRR4FCA7O7tz4vGfil2w8lJa1MPbi/6TP2/4MRtKNhDqCuXisy7m2nHXMnHQRH9HaYwx3fIlAawHMkVkJFAIXAtc32md/cACYK2IpAJjgd097PclYCmwwvv44gnE7V+Vuyl/6lL+HgF/G5RO6aYHSYtK4zszvsNVo68iPjze3xEaY0yPekwAquoWkduA14Eg4AlV3SIit3jffxi4H1gpIrk4TUZ3qWo5gIj8Gedib5KIFAD3qOrjOBX/X0XkazgJpN/3i1RVPt39On9e/T3eSAzCLcHMHjSO/xp7LfOHzieo00BvxhjTn4nTajMwZGdn64YNG07757a0tvDK7lf485an2Fq9i2iPsmTEYq6Zfisj40ae9niMMeZEiMhGVc3uXG5DQfjg95/+nsdyH2N0K9x9uIFLr3iayIzP+DssY4w5JZYAfPBhwRqmueGp0kPIDS9A+gx/h2SMMafMxgLqQXNrM9uqdjC1vg750nNW+RtjzhiWAHqwrXIbLShZUcNg2Nn+DscYY3qNNQH1ILcsB4CspCw/R2LMiTtQWc8vV+1g9bZSZp01iEunDGHBuFQiQq3HmrEE0KOcog9JcbsZnDbd36EY47PSmkZ+93Y+f163H5cInx2fyrq9lfxry0EiQoJYMD6FSyencd7YZMJDLBkEKksAPcgt30xWU7Mzp68x/Vx1fQuPrNnFE+/twd2qXHP2ML69IJPU2HBaPcpHeyp4JaeYf20+yCs5xUSHBbNoQiqXThnC3NHJhAZbq3AgsQRwHFWNVRxoquBzTU2QMt7f4RjTrfpmNyvf38vD/95FTaOby6ek8R8LxzAiKap9nSCXMHtUErNHJfE/l0/k/V0VvJJTxL82H+S5TwqJiwjhwompXDo5jdmjBhEcZMngdFFVmtweGltaaWhppaG5lfrm1qNeTx+eQFJ0WK9+riWA48gtzwVgckgChMf6ORpjjtXs9vDs+v389q18ymqbuGBcCt9bNJYJacf/9xoc5GL+mGTmj0nmR1dksXZnGa/kFPNq7kH+uqGAxKhQLpo0mEsnpzFzZCJBru5HsW12e6huaGlfatoeG1uornee1zW7CXa5CAt2EdpxCepUFhTU5TrhIS6iw0KIDg8mMiQI13HiOVGqSm2Tm9KaJkprGymrbWp/Xup9XlXfDICIEOQCl4h3cRKriBAkgqvDe0Eu532XCCLQ2OKhocVbqTe3tj+v9z7v6Z7cJ5edzflje3fQZEsAx5FTloNLYWKinf2b/qXVo7y4qZBfvrmDA5UNzByRyO+/OJ3sEYknvK/QYBcLxqeyYHwqjS2t/Ht7Ga/kFPHcx4U889F+kmPCOG9MMq2q7ZV7x6Wx5fgjv0eEBBEVFkyrx0OT20Oz24Pbc/IjEIhAdFgwMWHBRIcHO8/DneQQExZMTHhwe7Joex0W4qLicDOltU1OBV/b6K3kneddfYewYBcpsWEkR4cxLDESl0Crx0kYrap4FDwexaNKq8dZmlud106583fyqKIK4SEuIkKDGBQVSkRCEOEhQUR4l8jQIMJDj7yOCD36MTwkiOGDIk/6mHXHEsBx5JZ+yqiWFiKHWQ8g0z+oKqvySvjFGzvYXlLLhCGxPLlsEueNSe6VuSbCQ4JYPGkwiycNpr7ZzVvbSnnl02JWbyslIiSI2IgQ4iKCGZkURVxECHERIcSGhxAX6X3uLWsvjwjp8rqCx6M0tx5JCM2tHppaWmlu9b72Lk3e140trdQ1tXK4qYXDjW5qGt0cbnJz2Pt4qL6ZA1X17a/rm1u7/Y4x4cGkxISRHBPG1GHxpMSEkRIbRkpMePvz5JhwYsODz/j5OywBdMOjHnLLc1jU1ASpdgHYnLyiQw38M6eY0GAXMeHO2WqM98w1tu15eDAhPbS5v7+rnJ+/vp1P9h9iZFIUv71uGpdkDenV5pCOIkODuXRyGpdOTuv1fbtcQrgrqM96ILlbPdQ1tVLb1MLhJjcNza0MinIqfesCe4QlgG7sq9lHrbuerEZLAObk/TOnmB88l0NNo7vHdcNDXO3JISbsSKKICQ+moKqB93dVMDg2nJ9clcXVM4b2mDACWXCQi7hIF3GRIf4OpV+zBNCN9gvAboXEUX6Oxgw0dU1u7n1pC3/bWMCUYfH84vOTSYgMpbbR7V1a2psxahtb2stqG93UNh1Z52BNI7WNLQS7XPznxeP58qzh1m/f9BpLAN3IKcshEuGs+FEQZIfJ+G7TgUPc8ewn7Kus57bzR3P7ZzPbz9YH9XI3PmNOhdVs3cgtz2VScytBqXYB2Pim1aM8/M4ufrlqBykxYTx742c456xB/g7LmG5ZAuhCo7uRHZXbWVpfC6kT/B2OGQCKDjXwnb9s4qM9lVwyeQg/viLL2p9Nv2cJoAvbKrfh1lZnCAi7AGx60Hah1+1Rfn71ZK6eMfSM7z5ozgw+dSMQkcUisl1E8kVkeRfvx4nIyyLyqYhsEZFlPW0rIveKSKGIbPIuF/fOVzp1Od4RQCc3NdkYQKZbdU1uvv+3T7n1Tx8zMjmaV789j89nD7PK3wwYPf4CEJEg4CFgIVAArBeRl1Q1r8NqtwJ5qnqZiCQD20XkGaC1h21/qaoP9OL36RW55bkMljCSwxIhundvvTZnhk8PHOL2bi70GjNQ+NIENBPIV9XdACLyLLAE6JgAFIgR59QnGqgE3MA5Pmzb7+SW55LlVqf9387mTAd2odecSXw5ZUkHDnR4XeAt6+h3wHigCMgFbldVjw/b3iYiOSLyhIgkdPXhInKTiGwQkQ1lZWU+hHtqKhoqKDxcyOTaKmv+MUcpOtTA9Y99yM9f386Fkwbz2u3zrfI3A5ovCaCrU+DOIzldCGwC0oCpwO9EJLaHbX8PjPKuXwz8oqsPV9VHVTVbVbOTk5N9CPfUtN0AllV/2C4Am3b/zClm8a/WkFtYzc+vnszvrptmvXzMgOdLE1ABMKzD66E4Z/odLQNWqKoC+SKyBxh3vG1VtaStUEQeA1454ej7QE5ZDkG4mNDcbF1AA1h9s5uPdleyZmcZa3aUsausjinD4vn1NVOPGmPfmIHMlwSwHsgUkZFAIXAtcH2ndfYDC4C1IpIKjAV2A4e621ZEhqhqsXf7K4HNp/ZVekdOeQ6ZIXFEKJBsw0AHCo9HySuuYe3OctbsKGPjviqaWz2Eh7g4Z+Qgls4ewXUzM+xCrzmj9JgAVNUtIrcBrwNBwBOqukVEbvG+/zBwP7BSRHJxmn3uUtVygK629e76ZyIyFadJaC9wc29+sZPhUQ9byrdwsScMEs+C0N4ff9v0H6U1jU6Fv7OMd3eWU1HnTPoxbnAMy+aMYF5mMtkjEmzsHXPG8ulGMFV9FXi1U9nDHZ4XAYt83dZb/uUTivQ02FO9h8Mth8mqr4fUyf4Ox/SyxpZW1u+tbD/L33awFoCk6FDmZSYxf0wyc0cnkRIb7udIjTk97E7gDtpvAKsshPHX+Tkac7JqGlsoqGyg8FADhVX1FFQ1sL2klnV7KmlyewgNcpE9IoHlF41jXmYS4wfH9tmY+sb0Z5YAOsgtzyUmOIIRLS3WA6ifUlWq6lsorGqgoKqewkMNFFQ5i/O8ntpOY++HBbsYMSiK68/JYP6YZM4ZmUhkqP3TN8b+F3SQW57LxPAUXGyHFOsB1B/sr6jntc3FfLi7or2ib2g5erq/6LBghiZEkB4fwdkjEkiPj2BoQiTp3rKk6FAbnsGYLlgC8GpwN7CzaidfjRgBIZGQMNLfIQWsXWWHeS23mNc2H2RLUQ0AY1NjGJUczfwxyaTHR5CeEMHQhAiGxkcSG3Hmz91qTF+wBOCVV5FHq7Yyua4GUsaDy7r7nS6qyo6Sw7yaW8xrm4vZUXIYgBnDE/ivS8Zz4cTBDEu0HlnG9DZLAF65Zd47gMv2wthL/BtMAFBVthTV8NrmYl7LPcju8jpEYOaIRO67fCIXThzM4DjrjWNMX7IE4JVTnkN65GAG1a2zMYD6iMejbCo4xL82H+S1zcUcqGwgyCXMOmsQX5s3kkUTBpMcY1MmGnO6WALwyi3PZWrkEOeF9QDqNarKx/sP8UpOEf/afJDi6kZCgoS5o5P41vmZLJyQSkJUqL/DNCYgWQIAyurLOFh3kKyIEU6BJYBTll96mBc3FfLipiL2V9YTGuzi3DHJ3Ll4LBeMSyUuwgZSM8bfLAHgNP8ATK4/DNGDITLRzxENTKU1jbz0aREvbioit7Aal8Cc0Ul8e0EmF05MJSbcKn1j+hNLADgXgIMlmHEV++3s/wTVNrbwr80HeXFTEe/vKsejkJUex39dMp7Lp6TZsArG9GOWAHDa/8cmjCF831tw1vn+Dqffa3Z7eGdHGS9sKuTNvBKa3B4yEiO57fzRXD41ndEp0f4O0Rjjg4BPAK2eVjaXb+bytHnQ2gypk/wdUr/k8Sgb91fx/CeFvJpbzKH6FhKjQrnm7GEsmZrO9Ix4uxnLmAEm4BPArupd1Lvrmezy3mhkQ0AcpbS2kWfXHeAv6w9QeKiBiJAgFk1M5Yqp6czNTLLx8Y0ZwAI+AbTfANbQABIEyWP9HJH/qSqfHDjEH97fyz9zi2lpVeZlJvG9C8ewaMJgosIC/p+NMWeEgP+fnFueS2xoLMMr9kNSJgQH7o1IjS2tvJJTzB8+2EtOQTUxYcF86TPD+fJnhnNWsrXrG3OmCfgEkFOeQ1ZSFrL5PRh2tr/D8YuiQw08/eE+nl1/gMq6ZjJTorn/iklcOS2daDvbN+aM5dP/bhFZDPwaZ1rH/1PVFZ3ejwOeBjK8+3xAVZ883rYikgj8BRiBMyXkF1S16tS/ku/qW+rZdWgXC9LmQvWfYcbSPvmcllYPxYcaSU+IIKifTDyiqny4u5I/fLCXN/JKUFU+Oz6Vr8wewaxRg+yCrjEBoMcEICJBwEPAQqAAWC8iL6lqXofVbgXyVPUyEUkGtovIM0DrcbZdDqxW1RUistz7+q7e/HI92VKxBY96yAqKcQr6oAdQSU0jX3tqPZsLa4gICWJCWixZ6XFMTItlUnoco1OiT+uF1PpmN89/Usgf3t/H9pJa4iNDuHHeWXzxnAwbcdOYAOPLL4CZQL6q7gYQkWeBJUDHBKBAjDinjdFAJeAGzjnOtkuA87zbPwX8m9OcANqmgMxqbHIKUnu3B9Dmwmq+/tQGahtb+MFF4zhY08iWwhr+tuEAK5udSU1Cg12MHxzDxPQ4JqXFkZUex5jB0YQF9+5E5Psq6vjDB/v464YD1Da6mZgWy8+unszlU9Js0nNjApQvCSAdONDhdQFOxd7R74CXgCIgBrhGVT0icrxtU1W1GEBVi0UkpasPF5GbgJsAMjIyfAjXd7nluQyLGUZCxW4Ii4W4Yb2279e3HOSOZzeRGBXKP745m3GDY9vf83iUPRV1bC6sZktRDZsLq3nl0yL+9NF+AIJdwpjUGCalO78SJqbFkZEYSX2zm9pGZznc5OZwUwuHG93UNnnLvOXOOi3edZzyirpmgl3CRVlD+Mrs4UzPSLBmHmMCnC8JoKtaQju9vhDYBFwAjAJWichaH7c9LlV9FHgUIDs7+4S27UluWS7Zg7Mh/1On/38vVIiqyqNrdrPiX9uYMjSeR2+YQUrM0cMhuFzCqORoRiVHs2Rqevt2Byob2FxUzebCajYX1fDm1lL+uqHAp88Ndgkx4cHEhIcQHRZMdHgwQ+LC25+nx0fyuenpNjSDMaadLwmgAOh4ajwU50y/o2XAClVVIF9E9gDjeti2RESGeM/+hwClJ/MFTtbBuoOUNpQyOSkL3vszZH3ulPfZ7PbwXy/k8tcNBVw6eQgPfH6Kz80rIkLGoEgyBkVycZYzLLWqcrCmkc2FNRQdamivzGPCg4kJC2l/Hh0WTFiwy87ojTEnxJcEsB7IFJGRQCFwLXB9p3X2AwuAtSKSCowFdgOHjrPtS8BSYIX38cVT+iYnKLfcuQFscsRgaKo+5UHgDtU3c8vTG/lwdyXfXpDJHQsycZ1ijx8RYUhcBEPiIk5pP8YY05UeE4CqukXkNuB1nK6cT6jqFhG5xfv+w8D9wEoRycVp9rlLVcsButrWu+sVwF9F5Gs4CeTzvfvVji+3LJcQVwhjGxudglOYBWx32WG+9tQGCqsa+NU1U7liWnovRWmMMX3Hp/sAVPVV4NVOZQ93eF4ELPJ1W295Bc6vBr/IKc9hfOJ4Qsu3OwUn2QPog10V3PL0RoJdwp9uPIfsETaXgDFmYAjIkbzcHjd5FXlkJWdByRan90943Anv56/rD/Dlxz8iJSaMF26dY5W/MWZACcj7/PMP5dPgbiArKQvWv3DC7f8ej/LTf23jkTW7mZeZxENfnE6szXZljBlgAjIBtN0ANjlhHFTshLEX+bxtfbObO57dxBt5JXz5M8O557IJBNuQyMaYASggE0BueS4JYQkMbawDj9vnXwAHq51hHbYW13DvZRP4ypyRfRypMcb0ncBMAGW5TEqahJRudQp8SAC5BdV8/Q/rqWtq5fGlZ3P+uC5vXDbGmAEj4NouDjcfZnf1bu8F4M0QFAqDRh93m/d3lfOFRz4g2OXi79+YZZW/MeaMEHAJYHPFZhRlctJkKM2DpLEQdPwLuL//9y4So0J54dY5R43pY4wxA1nAJYC2KSAnJU1yuoD20PxT29jCh7sruGTyEJJjAne2MGPMmSfgEkBOeQ4jYkcQ19oKtcU93gC2dmc5La3OZCnGGHMmCagEoKrkluUyOXmyc/YPPf4CeDOvhPjIEKZnxPd9gMYYcxoFVAIoriumorHCuQGs1DufzXHGAHK3enh7eykXjE2xvv7GmDNOQNVqOeXeGcDaegBFJEDM4G7X/3j/IarqW/jsBGv+McaceQIqAeSW5RIWFMaYhDFQkufMAXycMfTf3FpCSJAwLzPpNEZpjDGnR0DdCJZbnsv4xPGEEASlW2Hal467/ptbS/jMWYOIsXF+TIBoaWmhoKCAxrZh0s2AEh4eztChQwkJ8a3OCpgE0OJpIa8ijy+M/QIc2gstdce9ALyr7DC7y+r4yuwRpy1GY/ytoKCAmJgYRowYYTPMDTCqSkVFBQUFBYwc6dswNQHTBLSzaidNrU3ODWA+9ABavbUEgAvsrl8TQBobGxk0aJBV/gOQiDBo0KAT+vUWMAmgbQRQ5wJwHiCQPK7b9d/cWsr4IbEMTYg8TREa0z9Y5T9wnejfzqcEICKLRWS7iOSLyPIu3v++iGzyLptFpFVEEr3v3e4t2yIid3TY5l4RKeyw3cUnFPkJyi3PJTE8kbSoNKcHUMIICIvuct2qumY27K3ks+Pt7N8Yc+bqMQGISBDwEHARMAG4TkSOun1WVX+uqlNVdSrwA+AdVa0UkUnAjcBMYApwqYhkdtj0l23beaeO7DM5ZTlMTprsZMjSvOM2//x7Rykexe7+Ncac0Xz5BTATyFfV3araDDwLLDnO+tcBf/Y+Hw98qKr1quoG3gGuPJWAT0Z1UzV7a/Y6zT/N9VC5+7gJ4M28UlJiwshKP/FpIo0xA9OIESMoLy8/qW1XrlxJUVFRr+xrxIgRZGVlMXXqVLKzs09qH77yJQGkAwc6vC7wlh1DRCKBxcA/vEWbgfkiMsj73sXAsA6b3CYiOSLyhIgkdLPPm0Rkg4hsKCsr8yHcY20pdy76ZiVlQdk2UE+3CaDZ7eGdHWUsGJ+Cy2VtocaYnnVOAKfq7bffZtOmTWzYsKHX9tkVX7qBdlULajfrXga8p6qVAKq6VUR+CqwCDgOfAm7vur8H7vfu637gF8BXj/kg1UeBRwGys7O7+9zjyinPQRBnBNAtLzqF3QwB8dGeCg43ua35xwS8+17eQl5RTa/uc0JaLPdcdvzxt/bu3cvixYuZO3cuH374IVOmTGHZsmXcc889lJaW8swzzwBwxx130NDQQEREBE8++SRjx47lwQcfZPPmzTzxxBPk5uZy3XXXsW7dOiIjj+3MUVFRwXXXXUdZWRkzZ85E9Uj18vTTT/Ob3/yG5uZmzjnnHP73f/+XoKAgoqOjufnmm3n77bdJSEjg2Wef5Z133mHDhg188YtfJCIigg8++ACA3/72t7z88su0tLTwt7/9jXHjuu904i++/AIo4Oiz9qFAd6nuWo40/wCgqo+r6nRVnQ9UAju95SWq2qqqHuAxnKamPhEWFMbstNnEhMY4XUCDIyCx636yb+aVEB7iYs5ou/vXGH/Jz8/n9ttvJycnh23btvGnP/2Jd999lwceeIAf//jHjBs3jjVr1vDJJ5/wP//zP/zwhz8EnKSQn5/P888/z7Jly3jkkUe6rPwB7rvvPubOncsnn3zC5Zdfzv79+wHYunUrf/nLX3jvvffYtGkTQUFB7Umnrq6O6dOn8/HHH3Puuedy3333cfXVV5Odnc0zzzzDpk2biIiIACApKYmPP/6Yb3zjGzzwwAOAc2Y/derUY5bZs2e3xyUiLFq0iBkzZvDoo4/22TEG334BrAcyRWQkUIhTyV/feSURiQPOBb7UqTxFVUtFJAO4CpjlLR+iqsXe1a7EaS7qE8smLWPZpGXOi5ItkDIOXEHHrKeqvLm1lLmjkwkPOfZ9YwJJT2fqfWnkyJFkZWUBMHHiRBYsWICIkJWVxd69e6murmbp0qXs3LkTEaGlpQUAl8vFypUrmTx5MjfffDNz5szp9jPWrFnDc889B8All1xCQoLTCr169Wo2btzI2WefDUBDQwMpKSnt+7/mmmsA+NKXvsRVV13V7f7b3psxY0b755x//vls2rTpuN/9vffeIy0tjdLSUhYuXMi4ceOYP3/+cbc5WT0mAFV1i8htwOtAEPCEqm4RkVu87z/sXfVK4A1Vreu0i3+IyCCgBbhVVau85T8Tkak4TUB7gZtP9cv4pGQLjF3c5VvbDtZSeKiBby84/hSRxpi+FRZ2ZPIll8vV/trlcuF2u7n77rs5//zzef7559m7dy/nnXde+/o7d+4kOjrapzb5rvrNqypLly7lJz/5yUlt3/k7BAUF4XY7Ld9vv/023/nOd45ZNzIykvfffx+AtLQ0AFJSUrjyyitZt25dnyUAn+4DUNVXVXWMqo5S1f/nLXu4Q+WPqq5U1Wu72Haeqk5Q1SmqurpD+ZdVNUtVJ6vq5R1+DfSdw6VQX95t+/+beSWIwAXjrP3fmP6surqa9HSnL8rKlSuPKr/99ttZs2YNFRUV/P3vf+92H/Pnz29v2nnttdeoqnLOTRcsWMDf//53SktLAaisrGTfvn0AeDye9n3+6U9/Yu7cuQDExMRQW1vbY9xtvwA6L22Vf11dXft+6urqeOONN5g0aZLPx+VEBcxYQIBzAxh0OwvYm9tKmTI03qZ+NKafu/POO1m6dCkPPvggF1xwQXv5d77zHb75zW8yZswYHn/8cc4//3zmz5/f3oTT0T333MN1113H9OnTOffcc8nIyABgwoQJ/OhHP2LRokV4PB5CQkJ46KGHGD58OFFRUWzZsoUZM2YQFxfHX/7yFwC+8pWvcMsttxx1EfhklJSUcOWVTk95t9vN9ddfz+LFXbdY9AbpeOW7v8vOztZT6hb1/u/gjf+E7++CqKMv8pbWNDLzx6v5/oVjufV8awIygWnr1q2MHz/e32H0W9HR0Rw+fNjfYRxXV39DEdmoqsfcVBAwYwEBTvt/VMoxlT/A6m3Oz70FNvyDMSZABFYTUOmWbm8AW721hKEJEYxNjTnNQRlj+tKTTz7Jr3/966PK5syZw0MPPXTC++rvZ/8nKnASQKsbSrfBzBuPeauhuZW1O8u5bmaGjYRozBlm2bJlLFu2zN9h9EuB0wRUuRtam7r8BfBufjlNbo/d/WuMCSiBkwDaegClHNsDaPXWEmLCgpk5MvE0B2WMMf4TOAmgNA/EdcwkMB6Pc/fvuWOTCQ0OnMNhjDGBU+OVbIFBoyEk/KjinMJqyg83WfOPMSbgBFYC6KL9/828EoJcwnljk/0QlDGmv+gv8wF89atfJSUl5Zg7gCsrK1m4cCGZmZksXLiw/c7lUxEYvYCaauHQPpj25WPeenNrCdnDE4iPDPVDYMb0Y68th4O5vbvPwVlw0Yre3Wc/sHLlSiZNmtQ+js+p+MpXvsJtt93GDTfccFT5ihUrWLBgAcuXL2fFihWsWLGCn/70p6f0WYHxC6B0q/PY6RfAgcp6th2sZeEEa/4xpr/Yu3cv48aN4+tf/zqTJk3ii1/8Im+++SZz5swhMzOTdevWsW7dOmbPns20adOYPXs227dvB+DBBx/kq191phXJzc1l0qRJ1NfXd/k5FRUVLFq0iGnTpnHzzTcfMx/AzJkzmTp1KjfffDOtra2Acyfwd7/7XaZPn86CBQsoKyvj73//e/t8AFOnTqWhoQFw5gOYPn06WVlZbNu2zefvP3/+fBITj+2Q8uKLL7J06VIAli5dygsvvODzPrulqgNmmTFjhp6U9U+o3hOrWrnnqOIn392tw+96RXeXHT65/RpzhsnLy/N3CLpnzx4NCgrSnJwcbW1t1enTp+uyZcvU4/HoCy+8oEuWLNHq6mptaWlRVdVVq1bpVVddpaqqra2tOm/ePH3uued0xowZ+u6773b7Od/61rf0vvvuU1XVV155RQEtKyvTvLw8vfTSS7W5uVlVVb/xjW/oU089paqqgD799NOqqnrffffprbfeqqqq5557rq5fv75938OHD9ff/OY3qqr60EMP6de+9jVVVX3rrbd0ypQpxyyzZs065hhMnDjxqLK4uLijXsfHx3f5vbr6GwIbtIs6NTCagEq2QGg0xGUcVfzm1lJGJUcxMinKT4EZY7oSyPMBnE6BkQCmfQkyPgOuIy1eNY0tfLSngq/O7XpmMGOM/wTyfADdSU1Npbi4mCFDhlBcXNzlCKcnKjCuAaRNhayrjypas6OMllZloXX/NGbAOVPnAzieyy+/nKeeegqAp556iiVLlvS4TU8CIwF0YfXWUhKjQpmWkeDvUIwxJ+jOO+/kBz/4AXPmzGm/QAvHzgewfPny9oq8s3vuuYc1a9Ywffp03njjjS7nA5g8eTILFy6kuNiZr6rjfABvvfUW//3f/w0cmQ+g40Xgk3Xdddcxa9Ystm/fztChQ3n88ccBWL58OatWrSIzM5NVq1axfPnyU/ocCLT5ALzcrR5m/OhNPjs+lV98YUovRGbMmcHmAzi+gJwPQEQWi8h2EckXkWPSjoh8X0Q2eZfNItIqIone9273lm0RkTs6bJMoIqtEZKf38bSdim/YV0V1QwsLJ9jY/8aYwNVjAhCRIOAh4CJgAnCdiBw1opqq/lxVp6rqVOAHwDuqWikik4AbgZnAFOBSEcn0brYcWK2qmcBq7+vTYvXWEkKDXMzLtLt/jTnTPfnkk0ydOvWo5dZbbz2pffX3s/8T5UsvoJlAvqruBhCRZ4ElQF43618H/Nn7fDzwoarWe7d9B7gS+Jl3H+d513sK+Ddw1wl/gxOkqqzKK2HWqEFEhQVGJyhjApnNB9A9X5qA0oEDHV4XeMuOISKRwGLgH96izcB8ERnkfe9iYJj3vVRVLQbwPnbZHiMiN4nIBhHZUFZW5kO4x7errI69FfV81qZ+NMYEOF8SQFcdXbu7cnwZ8J6qVgKo6lbgp8Aq4F/Ap4D7RAJU1UdVNVtVs5OTT73JZvXWEgAWWPdPY0yA8yUBFHDkrB1gKNDdHRbXcqT5BwBVfVxVp6vqfKAS2Ol9q0REhgB4H7vuq9XL3txawsS0WNLiI07HxxljTL/lSwJYD2SKyEgRCcWp5F/qvJKIxAHnAi92Kk/xPmYAV3EkQbwELPU+X9p5u75QWdfMxn1VdvZvjDH4kABU1Q3cBrwObAX+qqpbROQWEbmlw6pXAm+oal2nXfxDRPKAl4FbVbVtEOsVwEIR2Qks9L7uU29vK8Wj2N2/xphj9Pf5AO69917S09PbezK9+uqrJ7X/jnzqBqOqrwKvdip7uNPrlcDKLrad180+K4AFPsbZK97cWkJqbBiT0mNP58caMyD9dN1P2Vbp+zDGvhiXOI67ZvZ5Z7/T7nTMBwDOnc7f+973Tvkz2gTMUBBN7lbW7ChjwfjU4w7gZIzxL5sPoOv5APpEV2NE99flpOcDUNV/by/V4Xe9om9tLTnpfRhzprP5APrvfAD33HOPDh8+XLOysnTZsmVaWVnZ5fey+QC68GZeCREhQcwaNcjfoRhjemDzARzrG9/4BnfffTciwt133813v/tdnnjiiZPaV5uASACqyuqtJczLTCI8JMjf4RhjemDzARwrNfVI55Ubb7yRSy+9tMf4ehIQ1wDyimsoqm7kszb3rzFnhECcD6BtSGqA559//pheQicjIBLA6q2liMAF42z4B2POBIE4H8Cdd95JVlYWkydP5u233+aXv/zlKX0OBMh8AH9df4AN+yr52dU29r8xx2PzARzfmTYfQEBcA/jC2cP4wtnDel7RGGMCSEAkAGNM4HryySf59a9/fVTZnDlzeOihh054X/397P9EWQIwxhxFVc+omyUDaT6AE23SD4iLwMYY34SHh1NRUXHCFYnxP1WloqKC8PBwn7exXwDGmHZDhw6loKCA3ph8yZx+4eHhDB061Of1LQEYY9qFhIQwcuRIf4dhThNrAjLGmABlCcAYYwKUJQBjjAlQA+pOYBEpA/b5O45uJAEnNwXQ6WHxnRqL79RYfKfuVGIcrqrJnQsHVALoz0RkQ1e3WvcXFt+psfhOjcV36voiRmsCMsaYAGUJwBhjApQlgN7zqL8D6IHFd2osvlNj8Z26Xo/RrgEYY0yAsl8AxhgToCwBGGNMgLIEcBJEZJiIvC0iW0Vki4jc7i2/V0QKRWSTd7nYjzHuFZFcbxwbvGWJIrJKRHZ6HxP8FNvYDsdok4jUiMgd/jx+IvKEiJSKyOYOZd0eLxH5gYjki8h2EbnQT/H9XES2iUiOiDwvIvHe8hEi0tDhOD7sp/i6/Xv2k+P3lw6x7RWRTd5yfxy/7uqUvv03qKq2nOACDAGme5/HADuACcC9wPf8HZ83rr1AUqeynwHLvc+XAz/tB3EGAQeB4f48fsB8YDqwuafj5f1bfwqEASOBXUCQH+JbBAR7n/+0Q3wjOq7nx+PX5d+zvxy/Tu//AvhvPx6/7uqUPv03aL8AToKqFqvqx97ntcBWIN2/UflkCfCU9/lTwBX+C6XdAmCXqvr1Dm9VXQNUdiru7ngtAZ5V1SZV3QPkAzNPd3yq+oaqur0vPwR8Hwe4l3Vz/LrTL45fG3Fmv/kC8Oe+jOF4jlOn9Om/QUsAp0hERgDTgI+8Rbd5f5I/4a8mFi8F3hCRjSJyk7csVVWLwfkHB6T4LbojruXo/3j95fhB98crHTjQYb0C/H8C8FXgtQ6vR4rIJyLyjojM81dQdP337G/Hbx5Qoqo7O5T57fh1qlP69N+gJYBTICLRwD+AO1S1Bvg9MAqYChTj/Kz0lzmqOh24CLhVROb7MZYuiUgocDnwN29Rfzp+x9PVfIl+608tIv8JuIFnvEXFQIaqTgP+A/iTiMT6IbTu/p796vgB13H0SYjfjl8XdUq3q3ZRdsLH0BLASRKREJw/1DOq+hyAqpaoaquqeoDH6OOftcejqkXex1LgeW8sJSIyBMD7WOqv+LwuAj5W1RLoX8fPq7vjVQAM67DeUKDoNMcGgIgsBS4FvqjexmFvs0CF9/lGnPbhMac7tuP8PfvT8QsGrgL+0lbmr+PXVZ1CH/8btARwErxtho8DW1X1wQ7lQzqsdiWwufO2p4OIRIlITNtznIuFm4GXgKXe1ZYCL/ojvg6OOvPqL8evg+6O10vAtSISJiIjgUxg3ekOTkQWA3cBl6tqfYfyZBEJ8j4/yxvfbj/E193fs18cP6/PAttUtaCtwB/Hr7s6hb7+N3g6r3SfKQswF+fnVg6wybtcDPwRyPWWvwQM8VN8Z+H0EPgU2AL8p7d8ELAa2Ol9TPTjMYwEKoC4DmV+O344iagYaME5u/ra8Y4X8J84Z4bbgYv8FF8+Tjtw27/Bh73rfs77d/8U+Bi4zE/xdfv37A/Hz1u+Eril07r+OH7d1Sl9+m/QhoIwxpgAZU1AxhgToCwBGGNMgLIEYIwxAcoSgDHGBChLAMYYE6AsARhjTICyBGCMMQHq/wOAV+I72yP5+wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for d in [5, 10, 15]:\n",
    "    df_subset = df_scores[df_scores.max_depth==d]\n",
    "    plt.plot(df_subset.n_estimators, df_subset.auc, label=f\"max_depth={d}\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c617f33f",
   "metadata": {},
   "source": [
    "* We can see that max_depth=10 seems to be the best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "5fa1af99",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "max_depth=10\n",
    "\n",
    "for s in [1, 3, 5, 10, 50]:\n",
    "    for n in range(10, 201, 10):\n",
    "        rf = RandomForestClassifier(n_estimators=n, \n",
    "                                    max_depth=max_depth,\n",
    "                                    min_samples_leaf=s,\n",
    "                                    random_state=1)\n",
    "        rf.fit(X_train, y_train)\n",
    "\n",
    "        y_pred = rf.predict_proba(X_val)[:,1]\n",
    "\n",
    "        auc = roc_auc_score(y_val, y_pred)\n",
    "        scores.append((s, n, auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "0f92a399",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>min_samples_leaf</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>0.791365</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>20</td>\n",
       "      <td>0.808496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>0.811584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>0.817839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0.817058</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   min_samples_leaf  n_estimators       auc\n",
       "0                 1            10  0.791365\n",
       "1                 1            20  0.808496\n",
       "2                 1            30  0.811584\n",
       "3                 1            40  0.817839\n",
       "4                 1            50  0.817058"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns=[\"min_samples_leaf\", \"n_estimators\", \"auc\"]\n",
    "df_scores = pd.DataFrame(scores, columns=columns)\n",
    "df_scores.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "6f2554b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABVIElEQVR4nO2dd3xVRfbAv5NGIAVCCBBKCASkBiIEEKS5iCDFhgq4KoIN164/hRV3dVfXRrEsKouisIKKdYOCgFKkCJKAoSXU0BJCgJBCQkLy3ju/P+YlvISUlx7IfD+f+dz77p0797ybl3Nmzp05R4kIBoPBYKh7uNS0AAaDwWCoGYwBMBgMhjqKMQAGg8FQRzEGwGAwGOooxgAYDAZDHcWtpgUoC02aNJHg4OCaFsNgMBguK7Zt23ZGRAIKH7+sDEBwcDBRUVE1LYbBYDBcViiljhZ13LiADAaDoY5iDIDBYDDUUYwBMBgMhjqKMQAGg8FQRzEGwGAwGOooxgAYDAZDHcUYAIPBYKijXFbrAAwGg+GywnoBTm+Cs5Hg3R78w6FBEChV05IBxgAYDAZD5XLuIJxYAYkr4dRasGQWPF+vCTQO18Xfvq3fokaMgjEABoOhVpKVBb/+CsuXwx9/QI8eMGQIDBoETZvWtHQO5J6DpLWQaFf6GXH6uHc7aDsRAodDk/6QeRjORkFylN7GvA5i1XU9mxc0CI3DoX6zKhddXU4ZwcLDw8WEgjAYrlzi4rTC/+knWLtWG4H69aF7d9i9GzLtnekuXbQxyDMIzapeV15EbJCyQyv7xBVw5jew5YKbFzT7k1b4gcPBp33J7VjO63bORl0sabGAXSc3aHXRGDQOh4B+4O5bLpGVUttEJPyS484YAKXUCOBdwBX4WETeKHS+IbAICEKPKmaKyKdKqdbAf4HmgA2YJyLv2q95GXgQOG1v5gURWV6SHMYAGAxOkn0a0mIgPVYrlQunoOkgaDEKvIJqWrp8LlyA9esvKv19+/Tx9u2FW0afZ9QNaQT672BX9HqatgnH4jacrZHe/PorbNwIGRm6fufOMHiwNgiDB0Pz5uWX6dw5iI+H48f1Nj4eMs+eIrTJz3RrvJIOPivxcj0FQAo9SPYYwTnv4Vj8+tPAux5eXuSX+vXL6NnJzYCUPwqOFM7t1+cG/wAtR5frO5XbACilXIH9wDAgHogEJohIjEOdF4CGIjJVKRUA7EMrfX8gUES2K6V8gG3ALSISYzcAGSIy09kvYQyAweCACGQlaAWfr+xjID0GLiRfrOfmBe6NdF2ARqHaELQYBU2uAZeSPcEicOoUNGgA3t7lcFVnn4G0PZBzFnLTOHsylUN70zhxNJW002l410vFzyuNVs1SCWiUhk+9VFysaSixXNJUjgWSznmQqZrg4tWWHPeeHD8byobtbVmxPojYY63Jzq1Px44XRwiDB0NgoP4e6ekFFXt8vJCalMyFlHjISqCeJR7/BvG09EuglX88rRrr/YYN0gE4nd6En3cPY8WOEazadQNJaSVbGqX0c2vYUBupHj0gLEyXTp3A3d2J55eTBinbwa8neDQs48PPk6P8BqAf8LKIDLd//iuAiLzuUOevQGvgUSAY+Bm4SkRshdqKAOaIyM/GABgMTmK9AOfjHRS8fZsWC5ZzF+t5+EHDLuDbBRp2vrht0FqfT98HJ5bpcmoDiEVfEzhC9ywDR0C9xiQlwdatBUtqqm5CKfD11aVhw0v3m/ql0jFgG219o2hZP4qmblF4yZEiv1bGBW9yaYRb/YY0aNQIF8+GJKflsmvfcbbvOsSp1Fw8fZrS65qh9OrTn5TEGFLjd5CbHkd922la+Vlp0QhcCk1mz7AEcCKtDbFH23DoZBBHz7TB4taUBi6nCfC2K/fGF5W7p8eFAtfbxIVsFYjFoyWu3q2o59cKN98gaDoY8etJ9gUXMjMpsWRkFPyckqJdWLt3Q3a2/c/lAV27XjQKPXro0qiRkJyczKFDhzh06BAHDx7k0KFDTJs2jc6dO5frJ1QRA3A7MEJEHrB/vgfoKyKPOdTxAZYCnQAfYJyILCvUTjCwHugmIul2A3AfkA5EAc+KSEoR938IeAggKCio19GjRUY1NRguP2y5kJ0E509A1gnITry471gunCl4Xf1A8O1sV/YOW8+mznfPc9Lg5Cpyjy5DEpbjYTuNTVzYdrQf3/0+imV/jCLmRCjduin69oVu3SAnB9LSdC86LQ1yz6fTzP0PWntH0aFxFJ2bRdEu4GD+LQ4ltSPqcDhRceHsONaD1KwAOoU24ppBDRk6wperOrqhFCQlJfHZZ5/xySefEBsbi5eXF3feeSf3338//fv3RxXxnWw2G/v27eP3LRvZH72GpLhILGlxtPYXgvyhc1AD2jV3J6BBFh4uOfnXWWweZNgCyVJNyHZpRLbyIjWnPqcyPDiZ5kb8WeH4mVzOpqSTlpZGamoqaWlpZGZm4u/vT/PmzQkMDCywddz39/fHpbBFcsBigf374Y8/bGzalElUVC779nmSnt4gv45SxxHZDkQDO4BoWra0sHDhpwwdOtS5v28hKmIA7gCGFzIAfUTkcYc6twPXAs8AIegRQA8RSbef9wZ+Bf4lIt/ZjzUDzqDfeLyCdhVNLkkWMwIwXFZYs3WvOy1G+3HPJxRU7NmnyH/hl4dy0TNC6reA+i3IcWvBH3tb8PvOlpzJ7UyGS2fq+TSiUSNKLPXqXWwyIyOD33//naZNW5CV1Y4dO+rl9+x37wYRG+Fto/jzdcsY0/NH2jXaDoCtfmtcWo6ClqPA/xo4t++iX/pslP5u+S8sg/JnsFgbhpPu1pO0bP98Y2GxQO/e2oUEYLFY+Omnn5g/fz7Lli3DYrHQv39/7r//fu644w58fHzK/LgzMjKIiopiy5Yt+SUpKQl/b2jWEE6lw5lzxV/v4uJCw4YNadiwIY0aNSqwbdCgAcnJySQmJnLy5ElOnjxJRt4LCAfc3Nxo1qzZJYbBz8+P+Pj4/N58XFwc2XlDAcDVtSXNmt2At/cARLqTnt6W06cbY7Np4+fjA19/DcOHl/mxAFXvAloGvCEiG+yf1wDTRGSrUsod+BFYKSKzi7lHMPCjiHQrSRZjAAy1knxFv8deYvQ245CeMQKA0j30+oH5yr1AaWDf1msKLq7s2wfvvw8LFuiXkh066M59aqp2J+TmliySpyf4+lqxWs+SmnoEqzUX6AF4AeDmlkbz5se46qo0+vQRhg5tSO/ebWjYsKEehST+BAnL4OSqS+ex129RaB57L/3dnGDfvn18+umnLFy4kJMnT9K0aVMmTpzI5MmT6dSpUxkeeumICMeOHWPLli3s3r2b+vXrX6LYHbfe3t5FjjaKIyMjI98YOBoGx/2TJ0+SlJSEzWajQYMGhISEEBISQvv27Qvst27dGje3gu9isrK0gd6xA6Kj4emnISSkfM+iIgbADf0SeCiQgH4JfJeI7HGo8yGQJCIv23v229G/tmRgIXBWRJ4q1G6giCTa959Gu5XGlySLMQCGGsWaDel7IXWPftGap+wdFb1yBZ8O0LCrds3kbX2uAtd6JTdv1TNh/v1vWLVKvyAcNw4eewz69r1YT0Qrh9TUosuhQ8msXr2N3buPY7X6EBBwFc2ataZly9N4e8dgsfzGqVObiYs7RFJSUgEZmjRpUkA5XdW+DWEt0mnZ4DTWBh3Iqt+FHLcmWK1WLBYLVqvVqf2EhAQWLlzIpk2bcHV1ZdSoUUyePJmRI0fi7tSb0MsXq9VKeno6jRo1KpOBqUwqOg10JPAOehroJyLyL6XUFAARmauUagEsAAIBhR4NLFJKDQA2ALvQ00DBPt1TKfUZEIYeQx4BHs4zCMVhDICh2rHmwP734OC8QorerQRF71GmW5w9C598Ah98AIcPQ4sW8Mgj8OCDZZvfvnXrVt566y2+++473N3dmThxIs8++ywdO3Ys9ppz584RFxdX4GVj3vbYsWNU5jqhq666ivvvv5977rmHwMDASmvXUDoVMgC1BWMADNWGCCT8CNufgYyD0Ow6CBh0Udn7dCizoi/Mjh0wZw4sXqx79IMG6d7+Lbc4OT0Q7eb46aefeOutt/j1119p1KgRjzzyCE888QTNKzIZHrhw4QJHjhzh0KFDxMfHo5TC1dUVV1dX3Nzcitwv7py3tzddunSpsR5wXac4A2BCQRgMhUndA9ufhpM/g28nGPITtBhRKU3n5sL332vFv2GDXih0993w6KN6CqCz5OTk8MUXXzBjxgz27NlDq1atmD17Ng888EC5XqAWRb169ejYsWOJIwjD5Y0xAAZDHheSYedLcHAuuPlAr3ehwyPgUnEfdVISzJsHc+fCiRPQti3MnAmTJkHjxs63k56ezkcffcTbb79NQkICoaGh/Pe//2X8+PFXvC/dUPkYA2C44jl9GmJidNiB3NxLiyU3l47qQ8LrvYyHSuOPjEdYe+YfpO/yL1AvJ6f8JSVFv+QdPhz+8x+48UZwdXX+OyQmJvLee+/x4YcfkpaWxnXXXcfHH3/M8OHDjVvFUG6MATBcUYjAwYM6TsymTXqbF1+mKIZ3X8Hsu5+hS8tYft51PU8veps98RdnI7u7XyweHrrUq3dx37H4+hZ93MND9/Lvvhuc8abYbDYOHDhAZGRkfomKisJqtXL77bfz3HPPER5+iTvXYCgz5iVwHeDkSa0EGzeGgABo0gTcrhDTn5urQwU7KvxTOk4Xfn5w7bUwYABcfbWOyZKnzL1t+2hx8hm805eT69mejA6zsAWOwd1D5ddxda36EO0iQnx8/CXKPi0tDYD69evTs2dP+vfvz8MPP0xIeSeCG+o05iVwHUJEK8Uff9QlMvLSOn5+2hg4W+qVPIW9ar5E+l6dTen0Rh1LvUEQF9xD2JcYwpbdIazcFMJPa5uSlaW1dLt22sUyYIAunTpdGieGnFTY9U848G9wawBXz8D9qsfxK2WOfmVx5syZAso+MjIyfy6+m5sb3bt3Z/z48fTu3ZvevXvTpUuXSxYIGQyVhRkBXCGcPw9r1sAPP2ilf+KE7r327Qtjxuhl+Kmp2h9eXDlzRvupC+PhdoFruh3m+jGtmPSQN61aVcEXsF6As9suKvwzm/IjWua4NOF4akcayHGa+RzHxeXib/aC1Yss13bUaxJC/SYh4BMC3iE6GYdXm4svcG0WOPQx7PybbjfkAejxqtMrWCvC6tWrmTdvHlu3buXIkSMAKKXo2LEjvXv3pk+fPvTu3ZsePXrg6elZ5fIY6h5mBFBDiMC0abBoEQQH615px44XS0iI83O+CxMfD8uWaaW/erWOMujtDTfeYGXMjRZGDM0lwM+iA7HUq6cDipSAzVbQSKScSicg7T90dX0bHze9Ru/k0mbstbajUct2NGsfgvJup5WtdwjUb65j2ThDTgqc/u2iwk/eCjZ7VEafDpz3u4k1uwfw9qIBrInsgJeXol8/GDTgAkP7HiGs/SEaWA9RL+MQ9TLiIGM/HFihV+vmoVx1jBqfEMhK1Ct3mw6GXu+AX1iZn3dZ2bp1Ky+88AKrV6+madOmDBo0iEceeYTevXvTq1cvfH3Ll9zDYKgszAigCrHZ4C9/gYP/+YUZjV9HWS1knRcsuYILNhSCC4Knp1DfoXjW08XNVVAi2orYbIjFwoVzuWSds5BzPhfJteBOLh7Kgoer3ldWi76mKDp10kOCa67R29DQol8GZCXBvnfhwAeQmwbNhkLwBM4mnuLgH3FcSD5Ea784Wvsfx9XFIeK3qyd4tb3YA88rPiHgUg/ObL6o8NN262uUGzTuCQEDsDQewM9/9OeDT5qxfLl+fkOGwAMPwG236TnzJSI2regzDum0fOcO2fcP6cibXadD69uq3LEfExPDiy++yPfff0+TJk2YPn06U6ZMMb17Q41hVgJXM1arXsr/zafpJHhdhXdDV5Q9oleuVXE+S3H+vCLzvCLjvAsZ5xWZmQqrKARd3NwVXt4Kb2+FDRfiT7mTecENC+74N3OjZbA7QSFu+AW4o9zd9FDCzU2XvP28bWqqDv+4ZYvu3oPWqOHh2hj07QvdmkP6Yoj7FGw50HosdJmqA345kJ0NX30F//kwh1OHj9IlKI7xo+O4/po4AjztyjfjEFgujZaIuy806QcBA3Tx78P+uAZ88okOfJaUpJN33HcfTJ4M7UvJqlebOHLkCC+99BKLFi3Cy8uL//u//+Ppp5+utIVZBkN5Kc4AICKXTenVq5dcDuTmitx1l+66b+r3rNiUEtm6tdTrLBaRQ4dEli8XefttkSlTRIYMEQkMFPHzExk/XmTxYpHk5AoIZ7OJxMWJfPGFyJNPilxzjYiHe944Q8QPkcHBIi8/K7Junci5cyU2FxkpMmmSiKenvnzgQJEvvxS5kG0TyTolcnqLyOHPRQ78R+RstIjVIiIimZkiCxeKDBqkr3N1FbnpJpGlS/Xzu5w4efKkPPbYY+Lu7i716tWTZ599Vk6fPl3TYhkM+QBRUoROrXGlXpZyORiAnByR22/XT3buU7Eibm4i999f02Jdis0mkrhaZPUNIgsR+VcDkeeHioy/TaR9+4sGwcVFpHt3keeeEzl/vtjmzpwRmTFDpF07fVnz5iJ/+5tIfHzBW0ZGasPm66vrtW8v8sYbIidOVMN3rmRSUlLkhRdekAYNGoirq6s8+OCDcvz48ZoWy2C4BGMAqoHsbN2LBZHZs2wiN9wg0rChSFJSTYt2EatF5Ni3Ij/1FlmMyLfNRHa/LnIhpWC906dFli0T+fvfRYYNE1FKjxZK+S5Wqx7BjBqlL3F1FRk7VuStt0R69NDPxtNT5J57RH79VRuFy43MzEx54403xM/PTwAZP3687Nu3r6bFMhiKxRiAKub8eZEbb9RPdM4cEfnf//SHd96padE0lmyRAx+JLL1KK/6IEJH9c0UsWc5d/913IvXriwQHi+zZ49Qlhw6JPP+8iL+/fhS9eol88IFISkr5v0ZNkpOTIx988IEEBgYKICNHjpQ//vijpsUyGErFGIAqJCND5PrrdY933jwRycoSadtWpEsX7ROqKaw5IifXiWx/TuS7QK34l/cUObIk3xdfJiIjtW/H11dk1SqnLzt/Xr92uFyxWCyyaNEiadeunQAyYMAAWb9+fU2LZTA4TXEGwKwDqCDnzsHo0ToEwYIFcO+9wKszdWaPX34p/yT/8pKVpNP5nVgOiav0NE4Xd2h2PfT7r57SWd5pkOHheibR6NE6mtkHH8BDD5V6Wf36Ovrl5UZOTg6LFi3izTffZP/+/fTo0YNly5Zx4403mgBshisCYwAqQFqa1oNbt+qkHuPHA8ePw2uvwdixMHRo1QshNr2CNmEZnFimk3WDztsadAe0GAnNrwf3SpqK2Lq1tnbjxsHDD8P+/fDmm2ULbVnLyczM5KOPPmLWrFnEx8dz9dVX89VXXzF27FhcLoktYTBcvhgDUE7OntVxZ3bs0HPib7vNfuK55/T8mZkzq+7mOam6d39iue7tZ58CFDS5Brq/qpW+X1jVLXjy8YGlS3WW6lmz4NAhvdTZy6tq7ldNpKSkMGfOHN59912Sk5MZNGgQH3/8MTfccIPp8RuuSIwBKAdnzsCwYTrG/HffaY8IAOvWwZIl8PLLOu5DZSGik4+fWKaV/umNIFbw8IPAEdBiFAQOB88mlXfP0nBz09nLr7oKnnoKBg/WRqFFi+qToZJITExk9uzZzJ07l4yMDMaMGcO0adPo379/TYtWcVJTISoKDhwAf3+9yq55c7319q5p6Qw1jLNJ4UcA76KTwn8sIm8UOt8QWAQEoY3KTBH5VCnVGvgv0BydFH6eiLxrv6YxsAQIRieFv1NEUkqSozasBE5K0p6dQ4fgf//TowBAx9vp2RPS0yE21om4BU5iy4Utk+HIIv25UXet8FuOAv++4FILbPiyZdr/1aiRjkRXSm7D3NxcUlJSSE1NLbAt6ZjNZiMsLIzw8HDCw8Pp3r176aEVrFY9RNu4UUfHu/VW6NMnf2R06NAh3nrrLRYsWIDFYmH8+PFMmzaN0NDQSnow1Ux2NkRH6/CvW7fqsn9/8fW9vC4ag+bNi98PCCgYMsRq1ffKytJbZ/eLysZTUrFYLu4rBR066PAloaHQrVupsa0ue6xW2LNHr97fsgVeeKHcS+PLHQpCKeUK7AeGAfFAJDBBRGIc6rwANBSRqUqpAGAfWun7A4Eisl0p5QNsA24RkRil1FvAWRF5Qyk1DfATkaklyVLTBiAhQSv/48d1ALY//cnh5Jw58Pjj8M032v9fGVizYeM4SFgKXV/Q6QkbVEUozkpgxw49FEpNhS+/hFGjAMjKymL27Nl89dVXJCcnk5qaSmZmZolN1atXDz8/P/z8/GjUqBF+fn5YLBb++OMPTtvDWLi5uREaGppvEMLDw+kWEoLHjh062e6GDfDbb/otPeh3FFYrdOpE4g038MqRI/znxx9xd3dn0qRJPPfcc7Rr164qn1DlYrXC3r0XFX1kpP4bWCz6fPPmOrxH797a6HXurP82iYk6QcTJk0Xvp6Zeei8XF23cc3O1Qs+7R0VQqmC2ndJKbq5OapHhEF4kOFgbgjyjEBqqIyxerqkxk5Lg998vKvzIyIvf198fvvhCux7KQUUMQD/gZREZbv/8VwARed2hzl+B1sCj6B79z8BVImIr1FYEMEdEflZK7QOGiEiiUioQWCciJeZLqkkDcOyYVvhJSbB8OQwc6HDyzBndO+nVC37+uXJ877kZsP5mSFoD4e/DVX+peJtVzYkTOvZ0dDS2t9/m88aNeeGFFzh+/DjXXXcdbdu2zVfoeVvH/bxtcT17EeH48eNERUURFRVF7ObNeERFcXVGBgOBcCAvqv/Zli2xXXstfqNH4zpkCDRsyIHXXuPCvHl0S0nBChxs25amzz2H3333Vd6IrbKxWLTSPXNGu3Lyevfbtl1UDr6+eoZWnz669O4NLVuW73eYnX3RKDgahuRkndrM01M/K0/PkveL+uyo0MszacBmg6NHYdcuXXbv1tt9+y4aJXd3bQQcjUJoKAQFVX12n7Jw4YIereUp+y1bwB4qHDc3CAu7GLjxmmt02OAKyF8RA3A7MEJEHrB/vgfoKyKPOdTxAZYCnQAfYJyILCvUTjCwHugmIulKqVQRaeRwPkVE/Iq4/0PAQwBBQUG9jh496tw3rkSOHIHrrtN5XVes0H+PAkyZAh9/rHtgXbtW/IY5KbB2JJzdCn0/hXb3VrzN6iIzkzMjRtBk40b+DSzq2ZMZb7/NoEGDKt52QsLF3v3GjfqfXwRxcyO5bVt2N2rEz9nZLD5yhKP2nr+npydXX301IsKWLVto0qQJ/7jnHia5ulL/q6+0ZW/YUM9quu8+/cetLEWRlaWH8Dt3ah/8+fP6mGPJc5MUd7xwb9vDQyuHPEXfp49+D1OXZydduKCNQJ5ByCvHjl2s4+2tjUDLltCqlS55+3lbf//KNxI2mzbUp09rA56n7Ldv18miQc+sc1T2PXtWeoekIgbgDmB4IQPQR0Qed6hzO3At8AwQgh4B9BCRdPt5b+BX4F8i8p39mFMGwJGaGgHccw9ERMDatbqTX4Dt23Xv68kn4e23K36z7FOw5gZIj4Vrv4TWt1a8zWri0KFDTJ06le+//Zb3vb2ZkpGB3HgjaskS5/y12dk6ycHx47ocO3Zxf+9evbYCtO+6f389DBs4UCvBBg3ym7HZbBw8eDB/pBAVFcXZs2d56KGHeOCBB2iQV9dmg19/1Qs4vvlGK+gOHbQhuOce/Y/pDCJa7p07dScgb7t/v74H6F5dgwb6H7u0ktd7diwNG2rF3717DaRnu0xJS9MGeNcuPWMjPl6XhAQ9srHZCtavV694A9GsmTbI6em63fR05/bPndO/jzzyIvDmKfu+fXX7VUxVu4CWAW+IyAb75zXANBHZqpRyB34EVorIbIdrLgsXUG4uNG0Kt9wCn35a6KSIzj144ID+Z2/UqGI3yzwOa66H88dh0P8g8IaKtVdNpKSk8K9//Yv33nsPDw8Ppk2bxjPPPEODzz6DRx/Vo6KICK0Ei1Lueft5yXwdCQjQirhdu4tKPyys8pManzsH336rjcGvv+qe4NCh2hjceutFA5PXq3dU9Dt36uFhHsHB+kV49+669Oih5b+C1kpc9lgs2rWVkFDQMBTez+ulF0eDBto4+/pe3Dru5239/HRi6m7dauQdRUUygkUCHZRSbYEEYDxwV6E6x4ChwAalVDOgIxCn9OTp+UCso/K3sxSYCLxh30aU4ftUG+vX6/diN99cxMnPP9cvGj/+uOLKP/2AVv65qXDdKmg6oGLtVQO5ubnMnTuXl19+mZSUFCZPnswrr7xCYGCgrvDww3oJ8B13FL0UOG9Y3rq1Hva2bq1L3rFWrarPN+/jo5X9ffdBXBx89hksXAh3363PDRyojzv26r28tH/5jjsuKvzQUP1Pb6jduLld7On37Vt0HRH97iUhQb/8c1T2eeUyz9fs7DTQkcA76Gmgn4jIv5RSUwBEZK5SqgWwAAgEFHo0sEgpNQDYAOxCTwMFeEFEliul/IGv0FNHjwF3iMjZkuSoiRHAE09o/X7mTAEvg+4xduyoh2+//14xH2zqLlgzTM/tv26lzpBVixERfvzxR/7v//6P/fv3M3ToUGbNmkWP4qZ/7t2rV8s1bXpRubdurf+ZatOLucLYbPp9w8KF2m/bsePFHn337rpXX5d974bLBpMRrByI6NF8WJj2YBRg2jQdAmHLluJ7EM5wZiusGwGu9eFPv0DDzhWQuOqJjo7m2WefZc2aNXTq1ImZM2cycuRIs1LWYKjFFGcATPelBKKjtXv6EvfP/v0we7Z2F1RE+SetgzVDwb0RDNtYq5X/iRMnmDx5Mj179mTHjh3MmTOHnTt3MmrUKKP8DYbLlMvbgVXFREToEf6YMYVOPP20nqnx+utFXucUCcth41idRP1PP0ODqp8JUB7Onz/PrFmzeOONN7BYLDz77LNMnz6dRhV952EwGGocYwBKICJCTzwJCHA4uGyZXgk2c6ZebVkejn4Fv/0Z/HrAkBXVG8PHSUSEL7/8kqlTp3L8+HHGjh3Lm2++SUhISE2LZjAYKgnjAiqGo0e1C6iA++fCBR34rFMnHfahPByaD79N0JE7/7S6Vir/33//nf79+3PXXXcREBDAr7/+yjfffGOUv8FwhWEMQDHkvfQtYADefhsOHoR339UrMsvK3nfg9weg+TA928ejdk0XPH78OHfffTfXXHMNR44c4dNPPyUyMrJyVvEaDIZah3EBFUNEBHTpoheGAnou8Kuv6hVhN5RxgZYI7H4Fdr0ErcdC/8Xg6vxqTqvVyrlz56rM756ZmcmMGTN46623sNlsTJ8+nalTp+JzpUdbNBjqOGYEUAQpKXoxaIHe//PP69WDs2aVvcGdf9PKv+1EHd6hDMof4G9/+xt+fn6EhYUxdepU1q5dS05pKxSdwGazsWjRIjp27Mg//vEPbrrpJvbu3curr75qlL/BUBcoKlFwbS3VlRT+s89EQGTLFvuBDRv0gRdfLHtjsbN1MvYtD4jYrGW+PC0tTXx9faV3794yZMgQcXd3F0C8vLxk9OjR8u9//1sOHDhQ5nZ/++036dOnjwASHh4uGzZsKHMbBoPh8oBiksLXuFIvS6kuA3D77SKBgSLWPH19000izZuLZGSUraFDC7XyXz9WxGoplyyzZs0SQKKiokREJD09XSIiIuQvf/mLhISECCCAtGvXTh555BGJiIiQ9PT0Yts7evSoTJgwQQAJDAyUBQsWiNVadsNkMBguH4wBcJKsLBFvb5GHH7YfSE4WcXcXefbZsjV0fKnI564ivwwVsWSXS5bc3FwJCgqSwYMHF1vnwIEDMmfOHBkzZox4eXkJIO7u7jJ48GB5/fXXZfv27WK1WiUjI0P+9re/iaenp3h6esqLL74o586dK5dcBoPh8sIYACdZtkw/leXL7QfmztUHtm93vpGk9SJfeor8FC6SU3xvvDS+/PJLASQiIsKp+hcuXJA1a9bI1KlTJSwsLH900LRpU2nevLkAMn78eDly5Ei5ZTIYDJcfxRkAEwuoEA8/rIN8njljD7s+aJD+sGePc4HLUnbAL4OhfnO4fgN4BpR+TRGICH379iUtLY3Y2FhcyhF07OTJk6xatYqVK1eSkpLCiy++eGUkOjcYDGWiIuGg6ww2GyxdCjfeaFf+R4/qaJCvvuqc8j93CNYOB3cfHdK5nMofYOPGjURGRvLhhx+WS/kDNG/enHvvvZd7772MMooZDIZqwxgAB7Zu1Tki8qd/fvGF3t5VOP1BEWQlwtobQCxw3VrwCqqQLLNnz8bf398ob4PBUGWYdQAORETopE0jR6IXby1apIMBFZXMxJGcVFg7ArKTYPDyCkf1PHDgABERETzyyCMX0xcaDAZDJWMMgAMRETBkiM7exs6d2u//5z+XfJHlPPw6RufwHfg9NOlTYTneeecd3N3defTRRyvclsFgMBSHMQB29u+H2FgH98/ixTrd2513Fn+RLRc2joPTm3R4h8BhFZYjOTmZTz/9lLvvvpvm5Y02ajAYDE5g3gHYyQv+dtNN6LfBX3wBw4dDk2KidYoNttwPJ36E3h9C0B2VIsd//vMfsrKyePrppyulPYPBYCgOMwKwExGhUz+2aYPOBB8fX7z7RwS2PwtHPoPur0CHKZUiw4ULF/j3v//N8OHD6datW6W0aTAYDMVhDABw6hT89psO9Alo94+Xl304UAQxr8O+d+CqJ6Dr9EqT44svvuDkyZM8++yzldamwWAwFIdTBkApNUIptU8pdVApNa2I8w2VUj8opXYopfYopSY5nPtEKXVKKbW70DUvK6USlFLR9jKy4l+nfPzwg+7U33wzOunLN9/ArbdqI1CYg/Ngx3QI/jP0etu59QFOICLMnj2b0NBQrr/++kpp02AwGEqiVAOglHIF3gduBLoAE5RSXQpVexSIEZEewBBgllIqL2PKAmBEMc2/LSJh9rK8HPJXChER2vXTowc63WNqatHun2PfQOQj0GIkXPMpqMobQP3yyy/s2rWLZ555xiRZNxgM1YIzGqwPcFBE4kQkB/gSuLlQHQF8lNZc3sBZwAIgIuvtn2slmZnw88+6968U2v3TtCkU7oWfXK3z+DbpBwO+Bhf3SpVj1qxZNG/enAkTJlRquwaDwVAczhiAlsBxh8/x9mOOzAE6AyeAXcCTImJzou3HlFI77W4iv6IqKKUeUkpFKaWiTp8+7USTZWPVKsjOtrt/0tLgxx9h3Dg9BTSPU+th/S3g2xEG/wBulbs4a/fu3axcuZLHH3+cevXKlizGYDAYyoszBqAof0ThCHLDgWigBRAGzFFK+ZbS7odAiL1+IlBkqi0RmSci4SISHhBQ/tg6xRERAY0awcCBwLff6ncAee6f3AyIehJ+GQKezWDICvAo0k5ViNmzZ1O/fn0efvjhSm/bYDAYisMZAxAPtHb43Ard03dkEvCdPfLoQeAw0KmkRkUkSUSs9pHCR2hXU7VisegO/+jR4O6Odv+EhECfPnBiJSzvBvvfgw5/gRv/gAYtKl2GkydPsnjxYiZNmoS/v3+lt28wGAzF4YwBiAQ6KKXa2l/sjgeWFqpzDBgKoJRqBnQE4kpqVCkV6PDxVmB3cXWrik2bIDnZ7v5JSIC1a2HcrbB5IqwbAa71YdhG6D1HR/isAt5//31yc3PNwi+DwVDtlLoSWEQsSqnHgJWAK/CJiOxRSk2xn58LvAIsUErtQruMporIGQCl1BfomUFNlFLxwEsiMh94SykVhnYnHQGq3f8REQEeHnrBL//5Qs8FbT4fjp7T8/u7vQiunlV2//Pnz/Phhx9y88030759+yq7j8FgMBSFU6Eg7FM0lxc6Ntdh/wRwQzHXFjmtRUTucV7MykdEG4ChQ8HHNR7mvQLtgA4h0Hc++HWvchkWLlxIcnIyzzzzTJXfy2AwGApTZ1cC79kDhw/b+OvtH8LcjnAgHe68CW7YXC3K32az8fbbb9O7d28GDBhQ5fczGAyGwtTZYHDrl+1j3YsPMtBzA2xrAy7H4cn/gEv1PJIff/yRAwcO8OWXX5qFXwaDoUaoeyMAWy7seY37A3sQFrwL+nwMm120L6gawy/PmjWLoKAgxo4dW233NBgMBkfqlgE4uw1W9IYd0/lh+2gWnomBU53h8OHSE79UIlFRUaxfv54nn3wSN7c6OwgzGAw1TN3QPpbzsOtl2DsLPJvxU+Z33PHurcTEAHNeBU9PHfytmpg9eza+vr488MAD1XZPg8FgKEzdMABbH4IjiyHkAbh6Bu+MaUSHDtApJBe++kqHffYtbeFy5XDs2DG++uornnrqKXyr6Z4Gg8FQFHXDAHR9EULuh2bXkZam13s9+SSon1fBmTPV6v557733AHjiiSeq7Z4Gg8FQFHXDADTspAvw00+Qm2tP/vL+YmjcGEYUF626cklPT+ejjz7ijjvuICgoqFruaTAYDMVRt14Coxd/BQTANd0y9Ic77tDLgauB+fPnk56ebjJ+GQyGWkGdMgA5OTrfy5gx4PrD/+D8+Wpz/1gsFt59910GDhxIeHh4tdzTYDAYSqJOGYB16yA93e7+WbwYgoLg2mur5d7ffvstR48eNb1/g8FQa6hTBiAiAho0gOu7n9JpwO66C1yq/hGICLNmzaJDhw6MGTOmyu9nMBgMzlA3XgKjg78tXQo33AD1ly4Bq7Xa3D+bNm0iMjKSDz74AJdqMDgGg8HgDHVGG23fDvHx9tj/ixdD9+7QrVuV3/fUqVNMmjSJgIAA7r333iq/n8FgMDhLnTEA//uf9vbc1OUg/P57tfT+z507x8iRI0lISCAiIgIvL68qv6fBYDA4S51xAUVEwIAB0HjF56AUTCgyTUGlceHCBW699Vaio6OJiIigX79+VXo/g8FgKCt1YgQQFwe7dsHNN4l2/wwaBK1bl35hObHZbEycOJHVq1czf/58Ro0aVWX3MhgMhvJSJwxARITe3hmyDfbvr1L3j4jw5JNPsmTJEt566y0mTpxYZfcyGAyGilAnDEBmJvTvD61+XaxX/d5+e5Xd6/XXX2fOnDk888wz/N///V+V3cdgMBgqilMGQCk1Qim1Tyl1UCk1rYjzDZVSPyildiil9iilJjmc+0QpdUoptbvQNY2VUj8rpQ7Yt34V/zpF8+KLsPFXK3z5JYwcCX5Vc6uPP/6Y6dOnc/fddzNjxgyT6ctgMNRqSjUASilX4H3gRqALMEEp1aVQtUeBGBHpAQwBZiml8gLsLACKirY2DVgtIh2A1fbPVYZauwZOnqwy909ERAQPP/wwI0aM4JNPPjHz/Q0GQ63HGS3VBzgoInEikgN8CdxcqI4APkp3eb2Bs4AFQETW2z8X5mZgoX1/IXBLmaUvC4sX65j/o0dXetMbNmxg/PjxhIeH8/XXX+Pu7l7p9zAYDIbKxhkD0BI47vA53n7MkTlAZ+AEsAt4UkRspbTbTEQSAezbpkVVUko9pJSKUkpFnT592glxiyArC777Tvv+PT3L10Yx7Nq1izFjxtCmTRuWLVuGt7d3pbZvMBgMVYUzBqAoR7YU+jwciAZaAGHAHKVUpaS7EpF5IhIuIuEBAQHla+SHH+DcuUp3/xw5coThw4fj7e3NypUradKkSaW2bzAYDFWJMwYgHnCcNN8K3dN3ZBLwnWgOAoeBTqW0m6SUCgSwb085J3I5WLECWrSAwYMrrckzZ84wfPhwsrKyWLFiBW3atKm0tg0Gg6E6cMYARAIdlFJt7S92xwNLC9U5BgwFUEo1AzoCcaW0uxTImyQ/EYhwVugy8/HHsGkTuLpWSnMZGRmMHDmSY8eO8cMPP9CtGmIKGQwGQ2VTqgEQEQvwGLASiAW+EpE9SqkpSqkp9mqvAP2VUrvQM3qmisgZAKXUF8BmoKNSKl4pdb/9mjeAYUqpA8Aw++eqwcUFgoMrpamcnBzGjh3L9u3bWbJkCQMGDKiUdg0Gg6G6USKF3fm1l/DwcImKiqqx+9tsNu655x4+//xz5s+fz+TJk2tMFoPBYHAWpdQ2EbkkFaGZrO4kIsKzzz7L559/zmuvvWaUv8FguOwxBsBJZsyYwTvvvMOTTz7JtGlVumbNYDAYqgVjAJzgu+++Y+rUqUyYMIHZs2ebEA8Gg+GKwBgAJ/jqq69o1aoVCxYsMCEeDAbDFYPRZk4QExNDWFgYHh4epVc2GAyGywRjAErBYrGwb98+unQpHP/OYDAYLm+MASiFuLg4cnJyjAEwGAxXHMYAlEJMTAwAnTt3rmFJDAaDoXIxBqAUjAEwGAxXKsYAlEJMTAytW7fGx8enpkUxGAyGSsUYgFKIiYkx/n+DwXBFYgxACdhsNvbu3WsMgMFguCIxBqAEjh49SlZWljEABoPhisQYgBLIewFsDIDBYLgSMQagBMwMIIPBcCXjVtMC1GZiYmJo3rw5fn5+NS2KoZyICNHR0ezevRtvb28aNmx4SantIT5EhNzc3ALFYrGUuu/42Waz4eHhQb169S4pnp6elxwzMa/qBsYAlICZAXR5k5KSwg8//MDhw4dp3LgxycnJpKenUzgJUv369S8xCo0aNcrf9/LyKjECrIhgtVqx2WzFbi0WCzk5OVy4cKHIUtQ5x2PlwcXFBXd3d9zd3XFxcclvz5kkUO7u7gUMhIeHB25ubri6uuYXFxeXS46VVFxcXHBxcSmw71iKOp53LE8eQ+ViDEAxiAgxMTHcd999NS2KoYzYbDZ+//131q5di1KKUaNG0atXL5RS2Gw2zp07R1paWn5JTU0lPT2ds2fPcvjwYXJycgq05+rqipeXFyJSpHKvaFa9onrkeSOTvGMeHh75yjyvuLm5lbhfVC8+bzSRZ1iys7OLNEiFj+cZD6vVisViwWq1FlmqMsNg06ZNadeuHSEhIbRp0wZ3d/cqu1ddwRiAYoiPjycjI8OMAC4zTp06xdKlS0lISKBDhw6MHj0aX1/f/PMuLi75PfuiEBEuXLhAampqASORmZmJUiq/R1qebVHuF3d392rNL6GUwsPDAw8PjypZ3Giz2fJHPIWNQ945R+Pp7LGsrCyOHDlCZGQkW7ZswdXVldatW+cbhMDAwFqZp0NEyMrKIjMzs8jnUJZn0blzZxo1alSp8hkDUAyxsbHAlTcDKM9dkecbtlgsRZbC59zc3AgODsbPz69W/qNZrVY2bNjAhg0b8PT05LbbbqNbt25lllUphaenJ82bN6d58+ZVJO2VS57rxs2t8lXLoEGDyM3N5ejRo8TFxREXF8eaNWtYs2YN9evXp127dvmlshVlYWw2GxkZGWRkZHDu3LkC28LHbTZbpdwzICCgZgyAUmoE8C7gCnwsIm8UOt8QWAQE2ducKSKflnStUupl4EHgtL2ZF0RkeUW/UGVRm6aA5vUiyus/zvucp8wrQqNGjfJ7XW3btqV+/fqV9C3LT0JCAkuXLuXUqVOEhoYyfPhwvLy8alosQxXg7u5O+/btad++PQAZGRn5xuDQoUPs2bMHgMaNG+f/ToODg/H09LykLavVWqb/KUelnpmZWaR8DRo0wNvbG29vb5o0aZK/7+Xllf++pKT3HiV9rgqXlyrNZ6eUcgX2A8OAeCASmCAiMQ51XgAaishUpVQAsA9oDliLu9ZuADJEZKazwoaHh0tUVFQZvl75eeihh/j+++85ffo0hw4dIjY2lgEDBlR5z6IwSUlJrFixgiNHjpRatygXQ57/OO8lXp6PuLhS3PmsrKz8fzRHP3mLFi3y/9Fat26Nq6trFT+Ri+Tk5LB27Vp+//13vL29GT16NFdddVW13d9QuxARTp8+nf87PXLkCLm5uSilaNasGUABhW61Wp1qN+//KE+Z5xUfH58CW29v72r9/ZcFpdQ2EQkvfNyZEUAf4KCIxNkb+hK4GYhxqCOAj9LjbW/gLGAB+jpxba0kbwaQiPDTTz+RnJxMdHQ0/fr1Y+DAgVU+dTA7O5t169axdetWPD09ue666/D19S1WyVe1L9nLy4smTZrQp08frFYrCQkJ+f9omzZtYuPGjbi7uxMcHJw/DA8ICKgymQ4fPswPP/xASkoKvXr1YtiwYWaWSB1HKUXTpk1p2rQp11xzDVarlePHjxMXF0dCQgJubm4FOkXF/S8V7jjVRpdnZeGMAWgJHHf4HI9W7I7MAZYCJwAfYJyI2JRSpV37mFLqXiAKeFZEUgrfXCn1EPAQQFBQkBPiVpy8GUDjxo3jwIEDJCcnM2zYMJKSkti4cSPR0dEMHTqUHj16VPqPQ0TYsWMHv/zyC5mZmYSHh3PdddfRoEGDSr1PRXB1dSUoKIigoCCGDBlCdnY2R44cyTcIBw4cAMDHxyffGLRp0wYfH58Kzy/Pzs5m1apV/PHHHzRu3JiJEycSHBxcCd+q5snNzSU+Pp7s7OyaFuWKIjAwkMDAwFLrVWTKbW3B09OTVq1aOe0ucsYAFKXhCvuNhgPRwJ+AEOBnpdSGUq79EHjF/vkVYBYw+ZLKIvOAeaBdQE7IW2GSkpJISUmhc+fObN68GV9fX/r27Yurqyu9e/dmxYoVREREEBkZyfDhwyvNMJ04cYKffvqJ+Ph4WrVqxZ///Genfrg1jaenJ506daJTp04ApKam5huD/fv3s2PHDkD30Hx8fPD19cXX17fAft5nHx+fYl8g7t27l2XLlpGZmcm1117L4MGDr6ipgPHx8fj4+BAcHHxF9zoNVYOIkJycTHx8PG3btnXqGmcMQDzQ2uFzK3RP35FJwBuiXygcVEodBjqVdK2IJOUdVEp9BPzolMTVQN4L4KCgIHbs2MGwYcPyfXutWrXi/vvvZ9euXfzyyy98+umndOvWjeuvv77YqYWlcf78edasWcO2bdvw8vLilltuoXv37petEmjUqBE9e/akZ8+eiAiJiYkkJCSQnp7OuXPnSE9PJykpiQMHDpCbm3vJ9V5eXpcYiqSkJGJiYmjWrBkTJkygRYsWNfDNqpbs7Gyj/A3lRimFv78/p0+fLr2yHWcMQCTQQSnVFkgAxgN3FapzDBgKbFBKNQM6AnFAanHXKqUCRSTRfv2twG6npa5i8gxAZmYmHh4e9OzZs8B5pRTdu3enU6dObNq0id9++429e/fSv39/rr32WqffD9hsNrZv386aNWvIzs7mmmuuYfDgwUXOWLhcUUrRokWLIhV23px7R8OQV/IWax0/fpysrCxcXV3505/+RP/+/Wvti7bKwCh/Q0Uo6++nVAMgIhal1GPASvRUzk9EZI9Saor9/Fy0C2eBUmoX2u0zVUTO2AW65Fp7028ppcLQLqAjwMNlkrwKiY2NpUWLFsTFxdG7d+9iFbKHhwfXXXcdPXv25JdffmH9+vX88ccfXH/99YSGhpb4xzh+/DjLly/n5MmTBAcHc+ONN9K0adOq+kq1krw5956eniV+99zcXESk1sfsMRguN0qdBlqbqK5poNdddx1BQUGEhITw+OOPOx0M7tixY6xYsYLExERatmzJiBEjaNWqVYE6GRkZ/PLLL+zYsQMfHx+GDx9Oly5dTM/PQGxsrIk8a6gwRf2OipsGakL+FcH+/fsJDg6mc+fOZYoEGhQUxIMPPsjNN99MWloa8+fP5/vvvyc9PR2r1cqWLVuYM2cOu3btYsCAATz22GN07drVKH/DZcPSpUt54403Sq9YwwQHB3PmzJlKaeu+++7jm2++Kde1p0+fpm/fvlx99dVs2LCh1Prr16+nZ8+euLm5lfueZcGEgijEmTNnaNWqFS4uLlxzzTVlvl4pRVhYGJ07d2bjxo1s3ryZ2NhYfH19SU5Opn379owYMQJ/f/8qkN5gqFpuuukmbrrpppoW47Jh9erVdOrUiYULFzpVPygoiAULFjBzptPrYyuEMQCFiImJ4ZprrsHb25vWrVuXfkEx1KtXj6FDh9KzZ09Wr15NcnIy48aNo2PHjqbHbyiVp556iujo6EptMywsjHfeeafY80eOHGHEiBEMGDCALVu20KNHDyZNmsRLL73EqVOnWLx4MTExMURFRTFnzhzuu+8+fH19iYqK4uTJk7z11lvcfvvtRbadmJjIuHHjSE9Px2Kx8OGHHzJw4EAeeeQRIiMjycrK4vbbb+cf//gHoHvwd911F2vXriU3N5d58+bx17/+lYMHD/Lcc88xZcoU1q1bx9///nf8/f3Zt28fgwYN4oMPPrhkrcmiRYt47733yMnJoW/fvnzwwQcA3H///URFRaGUYvLkyTz99NOlPsNt27bxzDPPkJGRQZMmTViwYAGBgYF89NFHzJs3j5ycHNq3b89nn33G/v37ef7558nKyiIsLIzNmzeXGjolb01LdeVjMC6gQuQtMAoPv8RdVi78/Py4/fbbefjhh+nUqZNR/oZazcGDB3nyySfZuXMne/fu5fPPP2fjxo3MnDmT11577ZL6iYmJbNy4kR9//JFp06YV2+7nn3/O8OHDiY6OZseOHYSFhQHwr3/9i6ioKHbu3Mmvv/7Kzp07869p3bo1mzdvZuDAgflumC1btvD3v/89v87WrVuZNWsWu3bt4tChQ3z33XcF7hsbG8uSJUvYtGkT0dHRuLq6snjxYqKjo0lISGD37t3s2rWLSZMmlfpscnNzefzxx/nmm2/Ytm0bkydPZvr06QDcdtttREZGsmPHDjp37sz8+fMJCwvjn//8J+PGjSM6Opr69eszbtw4wsLCLin//e9/S71/VWBGAIU4ffp0/kIjg6GmKKmnXpW0bduW0NBQALp27crQoUNRShEaGlpkPKpbbrkFFxcXunTpQlJS0iXn8+jduzeTJ08mNzeXW265Jd8AfPXVV8ybNw+LxUJiYiIxMTF0794dIN/VFBoaSkZGRv5CQU9PT1JTUwHo06cP7dq1A2DChAls3LixwChk9erVbNu2jd69ewOQlZVF06ZNGTNmDHFxcTz++OOMGjWKG264odRns2/fPnbv3s2wYcMAHUwub6Hm7t27efHFF0lNTSUjI4Phw4cX2caSJUtKvU91YgyAAwkJCbi7u+fHDTEY6hqO8ZRcXFzyP7u4uBQZSdaxfkkzCgcNGsT69etZtmwZ99xzD8899xwDBw5k5syZREZG4ufnx3333VcgDIbjvQvLlSdL4RF14c8iwsSJE3n99dcvkWnHjh2sXLmS999/n6+++opPPvmkWPnz2uratSubN2++5Nx9993H//73P3r06MGCBQtYt25dkW2MGzeOffv2XXL8mWee4d577y3x/lWB0XIObNmyhZycnFoVd8dguBI4evQoLVu25MEHHyQzM5Pt27fTo0cPvLy8aNiwIUlJSfz0008MGTKkTO1u3bqVw4cP06ZNG5YsWcJDDz1U4PzQoUO5+eabefrpp2natClnz57l3LlzeHl54eHhwdixYwkJCXEq81/Hjh05ffo0mzdvpl+/fuTm5rJ//366du3KuXPnCAwMJDc3l8WLF9OyZcsi2zAjgFpKWloae/bsITIykqFDh9a0OAbDFcW6deuYMWMG7u7ueHt789///pe2bdty9dVX07VrV9q1a1cut2u/fv2YNm0au3btYtCgQdx6660Fznfp0oVXX32VG264AZvNhru7O++//z7169dn0qRJ+claihohFMbDw4NvvvmGJ554grS0NCwWC0899RRdu3bllVdeoW/fvrRp04bQ0FDOnTtX5u8CEBkZya233pqfz/qll17Kz3FQFZiFYHZWrVrFli1bmD17Np999hljxoypkvsYDMVhFoKVjXXr1jFz5kx+/LHWhBGrFZiFYGXkwoULbN++nfr165OWllYrsoAZDAZDVWNcQOipnxcuXCAzMxNPT88rJr68wVDd7Nq1i3vuuafAsXr16vH7779X+r2GDBlS5ncGJfHoo4+yadOmAseefPJJp6aIXq7UeQNgs9nYsmULQUFBfP3113Ts2PGKjjZpMFQloaGhlb6Arbp4//33a1qEaqfOu4BiY2NJS0ujX79++WkgDQaDoS5Q5w3Ali1baNy4MS1atODo0aPGABgMhjpDnTYAx48fJz4+nr59++bnsTUGwGAw1BXqtAHYvHkznp6ehIWF5WcBMwbAYDDUFeqsAUhJSWHv3r306tULDw8PYmJicHd3JyQkpKZFMxhqLSYfQNkoaz6AuXPnEhoaSlhYGAMGDMjvmFYVdXYW0JYtW1BK0bdvX0CHgb7qqqtwd3evYckMhtqLyQdQNsqaD+Cuu+5iypQpgDa2zzzzDCtWrKgy+eqkAcjOzuaPP/6gW7du+Pj4ANoAXH311TUsmcGgeeopqOzZlGFhUFKQUZMPoObzAfj6+ubvZ2ZmVnn4eKdcQEqpEUqpfUqpg0qpS4J+K6UaKqV+UErtUErtUUpNKu1apVRjpdTPSqkD9q3zuRcryLZt28jNzaVfv36ADhEbFxdnluEb6jwmH0DxVFc+gPfff5+QkBCef/553nvvvVLlqgiljgCUUq7A+8AwIB6IVEotFRFH59SjQIyIjFFKBQD7lFKLAWsJ104DVovIG3bDMA2YWplfriisVitbt26lbdu2NG/eHNA5gG02m3kBbKg11FA6AJMPoASqKx/Ao48+yqOPPsrnn3/Oq6++6rT7qDw44wLqAxwUkTgApdSXwM2AowEQwEfp8Yo3cBawAH1LuPZmYIj9+oXAOqrBAMTExJCens6oUaMKHAMzA8hgMPkAiqe68wGMHz+eRx55pESZKoozBqAlcNzhczxasTsyB1gKnAB8gHEiYlNKlXRtMxFJBBCRRKVU03LIXyZEhM2bN+Pv70+HDh3yj8fGxuLi4sJVV11V1SIYDHUSkw9AU9oI4MCBA/m6admyZQX0VFXgjAEo6i1EYVM/HIgG/gSEAD8rpTY4eW3JN1fqIeAhgKCgoLJcegnHjh0jMTGR0aNHF+gpxMTE0L59+wK9DIPBUHmYfADOMWfOHH755Rfc3d3x8/OrUvcPoHvFJRWgH7DS4fNfgb8WqrMMGOjweQ3adVTstcA+INC+HwjsK02WXr16SUX44osv5M0335ScnJwCxzt37iy33HJLhdo2GCpKTExMTYtwWbF27VoZNWpUTYtR6yjqdwRESRE61ZlZQJFAB6VUW6WUBzAe7e5x5BgwFEAp1QzoCMSVcu1SYKJ9fyIQ4YzBKi/Jycns27eP3r17F5jrn5OTw4EDB4z/32Aw1DlKdQGJiEUp9RiwEnAFPhGRPUqpKfbzc4FXgAVKqV1ot89UETkDUNS19qbfAL5SSt2PNiB3VO5XK8iWLVtwdXXNnw2Qx8GDB7FYLMYAGAyVgMkHcHnh1EIwEVkOLC90bK7D/gmgyHlURV1rP56MfdRQ1Zw/f57o6GhCQ0Px9vYucC5vBpBZA2AwVByTD+Dyok7EAtq2bRsWiyV/4ZcjMTExKKXo1KlTDUhmMBgMNUedMAC+vr707NmTpk0vnWkaGxtLcHAwDRo0qAHJDAaDoeaoE7GAevToQY8ePYo8Z7KAGQyGukqdGAEUh8ViYd++fcYAGAyGOkmdNgCHDx/mwoULxgAYDE5i8gGUjbLmA1iwYAEBAQH5QeI+/vjjct3XWeqEC6g4TAwgg6FsmHwAZaOs+QBAxwuaM2dOFUp1EWMAMFNADbWQbU9BSnTltukXBr3eKfa0yQdQ8/kAqps67QKKiYmhVatW+UlhDIa6jskHUDzVlQ/g22+/pXv37tx+++0cP368OHEqhTo/AjDuH0OtpISeelVi8gEUT3XkAxgzZgwTJkygXr16zJ07l4kTJ7JmzZpSZSsvddYA2Gw29u7de0n4WIOhLmPyARSPVEM+AH9///xjDz74IFOnVm2KlDprAI4dO8b58+fNCMBgqAZMPgBNaSOAxMTE/FHF0qVLq/z9ZJ01AGYGkMFQfZh8AM7x3nvvsXTpUtzc3GjcuDELFiwoVzvOokoattU2wsPDJSoqqlLamjlzJs899xzJyck0bty4Uto0GCpCbGysmZFWBtatW8fMmTP58ccfa1qUWkVRvyOl1DYRCS9ct87OAoqJiaF58+ZG+RsMhjpLnXYBGfePwVC5mHwAlxd10gCICDExMdx77701LYrBcEVh8gFcXtRJF1BCQgLnzp0zIwCDwVCnqZMGIDY2FjAzgAwGQ92mThoAMwXUYDAY6rAB8Pf3JyAgoKZFMRgMhhrDKQOglBqhlNqnlDqolLok4pNS6jmlVLS97FZKWZVSje3nnrQf26OUesrhmpeVUgkO142stG9VCnkzgAovGzcYDCVj8gGUjbLmA1i/fj09e/bEzc3tknsuXLiQDh060KFDhzKFly6JUmcBKaVcgfeBYUA8EKmUWioiMXl1RGQGMMNefwzwtIicVUp1Ax4E+gA5wAql1DIROWC/9G0RmVkp38RJRIQ9e/Zw5513VudtDYay8dRTUNmzacLC4J13KtSEyQdQNsqaDyAoKIgFCxYwc2ZBtXj27Fn+8Y9/5Iev7tWrFzfddBN+fn4Vks+ZEUAf4KCIxIlIDvAlcHMJ9ScAX9j3OwNbROS8iFiAX4Fbi72yGjh16hQpKSnG/28wFOLIkSN06tSJBx54gG7duvHnP/+ZX375hWuvvZYOHTqwdetWFixYwGOPPQbonvETTzxB//79adeuXYm95MTERAYNGkRYWBjdunXL7w0/8sgjhIeH07VrV1566aX8+sHBwbzwwgv069eP8PBwtm/fzvDhwwkJCWHu3LmAXgmcF/6hS5cuTJkyJT+0gyOLFi2iT58+hIWF8fDDD2O1WrFardx3331069aN0NBQ3n77baee0bZt2xg8eDC9evVi+PDhJCYmAvDRRx/Ru3dvevTowdixYzl//jzR0dE8//zzLF++nLCwMLKyskptPzg4mO7du1+S02DlypUMGzaMxo0b4+fnx7Bhw1ixYoVTMpeEM+sAWgKOQanjgb5FVVRKNQBGAI/ZD+0G/qWU8geygJGAYyyHx5RS99qPPSsiKUW0+RDwEGjrWFFMEhjDZUEFe+rl5eDBg3z99dfMmzeP3r175+cDWLp0Ka+99hq33HJLgfp5+QD27t3LTTfdVGxCmLx8ANOnT8dqtXL+/HlA5wNo3LgxVquVoUOHsnPnzvxw0Hn5AJ5++mnuu+8+Nm3aRHZ2Nl27dmXKlCmADgYXExNDmzZtGDFiBN99910BGRzzAbi7u/OXv/yFxYsX07Vr1/x8AEB+eOmSyMsHEBERQUBAAEuWLGH69Ol88skn3HbbbTz44IMAvPjii8yfP5/HH3+cf/7zn/kJdKD0aKDFkZCQQOvWrfM/t2rVioSEhFJlLg1nDEBRjvLiAgiNATaJyFkAEYlVSr0J/AxkADuAvJiyHwKv2Nt6BZgFTL7kRiLzgHmgYwE5IW+JmBlABkPxmHwAxVMd+QCKo6iYbZXxDtMZAxAPtHb43Ao4UUzd8Vx0/wAgIvOB+QBKqdfs7SEi+b8WpdRHQLVEdIqNjcXX15cWLVpUx+0MhssKkw+geKojH0BxtGrVqkCb8fHxlRIGw5l3AJFAB6VUW6WUB1rJLy1cSSnVEBgMRBQ63tS+DQJuw24glFKBDtVuRbuLqhwzA8hgqH6OHj1K06ZNefDBB7n//vvZvn076enpl+QDKCt5+QBsNhtLlixhwIABBc4PHTqUb775hlOnTgH6ZerRo0c5c+YMNpuNsWPH8sorr7B9+/ZS7+WYDwC0S2jPnj0Al+QDKI4lS5YQHR19SSktLM3w4cNZtWoVKSkppKSksGrVqmJHGWWh1BGAiFiUUo8BKwFX4BMR2aOUmmI/P9de9VZglYhkFmriW/s7gFzgUQc//1tKqTC0C+gI8HBFv4wzxMTEMGrUqOq4lcFgsGPyAThHZGQkt956KykpKfzwww+89NJL7Nmzh8aNG/O3v/0t35X197//vVIiGdepfADJyck0adKEmTNn8uyzz1aiZAZDxTH5AMqGyQdQNCYfQDGYGEAGg8FwkToVDtrMADIYqhaTD+Dyos4ZgAYNGhSYT2swGCoPkw/g8qJOuYBiYmLo3LnzJavsDAaDoS5SpzRhbGyscf8YDAaDnTpjANLT04mPjzcGwGAwGOzUGQNgZgAZDAZDQeqMATAzgAyGimPyAZSNsuYDWLBgAQEBAYSFhREWFsbHH3+cf65G8gFcKcTExFCvXj3atm1b06IYDKWyYsUKTp48WaltNm/enBEjRlSoDZMPoGyUNR8A6HhBedFD86jJfABXBDExMXTq1AlXV9eaFsVgqJWYfAClU9X5AIqjqvIBICKXTenVq5eUl+DgYBk/fny5rzcYqpqYmJgavf/hw4fF1dVVdu7cKVarVXr27CmTJk0Sm80m//vf/+Tmm2+WTz/9VB599FEREZk4caLcfvvtYrVaZc+ePRISElJs2zNnzpRXX31VREQsFoukp6eLiEhycnL+scGDB8uOHTtERKRNmzbywQcfiIjIU089JaGhoZKeni6nTp2SgIAAERFZu3at1KtXTw4dOiQWi0Wuv/56+frrr/OvP336tMTExMjo0aMlJydHREQeeeQRWbhwoURFRcn111+fL19KSkqxsk+cOFG+/vprycnJkX79+smpU6dEROTLL7+USZMmiYjImTNn8utPnz5d3nvvPRGRAs9LROTOO++UHj16XFIWLlyYX7958+YSGhoqY8eOlWPHjomIyIwZM+SVV17Jb+ef//ynzJgxo0h5i/odAVFShE6tEy6gzMxMjhw5wuTJl6QbMBgMDph8AMVTHfkAxowZw4QJE6hXrx5z585l4sSJrFmzpkbzAVz25MXfNi+ADYaSMfkAikeqIR+Av79//rEHH3yQqVOnAjWbD+Cyx8wAMhhqFpMPQFNaPoC8dwqgZ1zlRfWssXwAVwIxMTG4ubnRvn37mhbFYKiTmHwAzvHee++xdOlS3NzcaNy4MQsWLAAw+QCg/PkA5s+fz2+//cb8+fOrQCqDoXIw+QDKhskHUDQmH0Ah7r//fqP8DQaDoRB1wgVkMBiqB5MP4PLCGACDoRYhIpUyva+mMPkAapayuvTrhAvIYLgc8PT0JDk5ucz/xAYDaOWfnJyMp6en09c4NQJQSo0A3gVcgY9F5I1C558D/uzQZmcgQETOKqWeBB4EFPCRiLxjv6YxsAQIBo4Ad4pIitOSGwxXGK1atSI+Pp7Tp0/XtCiGyxRPT09atWrldP1SZwEppVyB/cAwIB6IBCaISEwx9ccAT4vIn5RS3YAvgT5ADrACeEREDiil3gLOisgbSqlpgJ+ITC1JlvLOAjIYDIa6TEVmAfUBDopInIjkoBX6zSXUnwB8Yd/vDGwRkfMiYgF+BfIm6t4M5IXIWwjc4oQsBoPBYKgknDEALYHjDp/j7ccuQSnVABgBfGs/tBsYpJTyt58bCeRlZG8mIokA9m3TYtp8SCkVpZSKMkNjg8FgqDycMQBFTUkozm80BtgkImcBRCQWeBP4Ge3+2QFcGlCkBERknoiEi0h4QEBAWS41GAwGQwk48xI4nou9doBWwIli6o7novsHABGZD8wHUEq9Zm8PIEkpFSgiiUqpQOBUaYJs27btjFLqqBMy1wRNgMpJQVQ1GPkqhpGvYhj5Kk5FZGxT1EFnXgK7oV8CDwUS0C+B7xKRPYXqNQQOA61FJNPheFMROaWUCgJWAf1EJEUpNQNIdngJ3FhEni/nl6txlFJRRb1kqS0Y+SqGka9iGPkqTlXIWOoIQEQsSqnHgJXoaaCfiMgepdQU+/m59qq3Aqsclb+db5VS/kAu8KjDVM83gK+UUvcDx4A7Kv51DAaDweAsTq0DEJHlwPJCx+YW+rwAWFDEtQOLaTMZPaowGAwGQw1gVgJXHvNqWoBSMPJVDCNfxTDyVZxKl/GyCgdtMBgMhsrDjAAMBoOhjmIMgMFgMNRRjAEoB0qp1kqptUqpWKXUHnvAO5RSLyulEpRS0fYysgZlPKKU2mWXI8p+rLFS6mel1AH71q+GZOvo8IyilVLpSqmnavL5KaU+UUqdUkrtdjhW7PNSSv1VKXVQKbVPKVXx5Kzlk2+GUmqvUmqnUup7pVQj+/FgpVSWw3OcW2zDVStfsX/PWvL8ljjIdkQpFW0/XhPPrzidUrW/QRExpYwFCAR62vd90OskugAvA/9X0/LZ5ToCNCl07C1gmn1/GvBmLZDTFTiJXqhSY88PGAT0BHaX9rzsf+sdQD2gLXAIcK0B+W4A3Oz7bzrIF+xYrwafX5F/z9ry/AqdnwX8vQafX3E6pUp/g2YEUA5EJFFEttv3zwGxFBMfqZZRGwPwDQUOiUiNrvAWkfXA2UKHi3teNwNfisgFETkMHEQHTaxW+URkleggiwBb0Kv0a4Rinl9x1Irnl4dSSgF3UiiKQXVSgk6p0t+gMQAVRCkVDFwN5OW8e8w+JP+kplwsdgRYpZTappR6yH7MqQB81Uzh8CG15flB8c/L6QCJ1chk4CeHz22VUn8opX5VShW5FqeaKOrvWdue30AgSUQOOByrsedXSKdU6W/QGIAKoJTyRkc+fUpE0oEPgRAgDEhEDytrimtFpCdwI/CoUmpQDcpSJEopD+Am4Gv7odr0/EqiLAESqxyl1HR0kMXF9kOJQJCIXA08A3yulPKtAdGK+3vWqudHwRD2UIPPrwidUmzVIo6V+RkaA1BOlFLu6D/UYhH5DkBEkkTEKiI24COqeFhbEiJywr49BXxvlyVJ6cB7KCcD8FUxNwLbRSQJatfzs1Pc8ypLgMQqRSk1ERgN/FnszmG7WyDZvr8N7R++qrplK+HvWZuenxtwGzo7IVBzz68onUIV/waNASgHdp/hfCBWRGY7HA90qHYrOh9CtaOU8lJK+eTto18W7gaWAhPt1SYCETUhnwMFel615fk5UNzzWgqMV0rVU0q1BToAW6tbOKVTtU4FbhKR8w7HA5TO5IdSqp1dvrgakK+4v2eteH52rgf2ikhelOIaeX7F6RSq+jdYnW+6r5QCDEAPt3YC0fYyEvgM2GU/vhQIrCH52qFnCOwA9gDT7cf9gdXAAfu2cQ0+wwZAMtDQ4ViNPT+0IUpEBy2MB+4v6XkB09E9w33AjTUk30G0HzjvNzjXXnes/e++A9gOjKkh+Yr9e9aG52c/vgCYUqhuTTy/4nRKlf4GTSgIg8FgqKMYF5DBYDDUUYwBMBgMhjqKMQAGg8FQRzEGwGAwGOooxgAYDAZDHcUYAIPBYKijGANgMBgMdZT/B6cQk+cPqSwuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "colors = [\"black\", \"blue\", \"orange\", \"red\", \"grey\"]\n",
    "min_samples_leaf_values = [1, 3, 5, 10, 50]\n",
    "\n",
    "for s, col in zip(min_samples_leaf_values, colors):\n",
    "    df_subset = df_scores[df_scores.min_samples_leaf==s]\n",
    "    plt.plot(df_subset.n_estimators, df_subset.auc, label=f\"min_samples_leaf={s}\", color=col)\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9d325a",
   "metadata": {},
   "source": [
    "* The best values for min_samples_leaf are 1, 3, 5, in this example we set min_samples_leaf=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "59568724",
   "metadata": {},
   "outputs": [],
   "source": [
    "min_samples_leaf = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "e2fd2cfb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=10, min_samples_leaf=3, random_state=1)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train Random Forest with these values\n",
    "rf = RandomForestClassifier(n_estimators=100, \n",
    "                                    max_depth=max_depth,\n",
    "                                    min_samples_leaf=min_samples_leaf,\n",
    "                                    random_state=1)\n",
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eb25920",
   "metadata": {},
   "source": [
    "* Other useful parameters:\n",
    "    * ```max_features```: Sets how many features each tree gets\n",
    "    * ```bootstrap```: Another way of randomization on the row level\n",
    "    * ```n_jobs```: To parallize the training of the Desicion Trees\n",
    "    * Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e417898",
   "metadata": {},
   "source": [
    "## Gradient Boosing and XGBoost\n",
    "* Gradient boosint vs. random forest\n",
    "    * Gradient Boosing: A different way of combining several Decision Trees\n",
    "    * First make a prediction with one model, then train a second model correcting the errors of the irst one, etc.\n",
    "    * We will use XGBoost in this course\n",
    "    \n",
    "![boosting](Screenshot_6.png)\n",
    "* Installing XGBoost\n",
    "* Training the first model\n",
    "* Performance monitoring\n",
    "* Parsing XGBoost's monitoring output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "8a9f7e92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: xgboost in /home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages (1.5.2)\n",
      "Requirement already satisfied: scipy in /home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages (from xgboost) (1.7.3)\n",
      "Requirement already satisfied: numpy in /home/jens/miniconda3/envs/ml-zoomcamp/lib/python3.9/site-packages (from xgboost) (1.21.2)\n"
     ]
    }
   ],
   "source": [
    "# install XGBoost\n",
    "!pip install xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "379189fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae311bc5",
   "metadata": {},
   "source": [
    "* Wrap the training data in a special data structure\n",
    "* For training we need to define the parameters: https://xgboost.readthedocs.io/en/stable/parameter.html\n",
    "    * Important parameters: ```eta```: learning rate, ̣```max_depth```: size of the trees, ```min_child_weight```: equivalent to min_samples_weight\n",
    "    * ```objective```: defines the task, in our case we have a binary classification problem\n",
    "    * ```nthread```: To parallize training\n",
    "    * Set the random seed, with ```seed```\n",
    "    * Set ```verbosity``` to 1, to see warnings, while training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "3cb11e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "features = dv.get_feature_names_out()\n",
    "dtrain = xgb.DMatrix(X_train, label=y_train, feature_names=features)\n",
    "dval = xgb.DMatrix(X_val, label=y_val, feature_names=features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "dc91894d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[07:34:59] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n"
     ]
    }
   ],
   "source": [
    "# num_boost_round=200 is the number of trees we let grow\n",
    "xgb_params = {\n",
    "    \"eta\": 0.3,\n",
    "    \"max_depth\": 6,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, num_boost_round=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "16691220",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(dval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "f2a62fb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8003166549611296"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f4884b",
   "metadata": {},
   "source": [
    "* The performance of the model is depending on ```num_boost_round```\n",
    "* We can evaluate the model after each round by defining a \"watchlist\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "9af60355",
   "metadata": {},
   "outputs": [],
   "source": [
    "watchlist = [(dtrain, \"train\"), (dval, \"val\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "dd02972b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-auc:0.86300\tval-auc:0.76818\n",
      "[1]\ttrain-auc:0.88485\tval-auc:0.79410\n",
      "[2]\ttrain-auc:0.89923\tval-auc:0.80017\n",
      "[3]\ttrain-auc:0.90951\tval-auc:0.80886\n",
      "[4]\ttrain-auc:0.91967\tval-auc:0.80414\n",
      "[5]\ttrain-auc:0.92863\tval-auc:0.80606\n",
      "[6]\ttrain-auc:0.93385\tval-auc:0.80949\n",
      "[7]\ttrain-auc:0.93880\tval-auc:0.81101\n",
      "[8]\ttrain-auc:0.94150\tval-auc:0.81445\n",
      "[9]\ttrain-auc:0.94581\tval-auc:0.81527\n",
      "[10]\ttrain-auc:0.95002\tval-auc:0.81558\n",
      "[11]\ttrain-auc:0.95342\tval-auc:0.81678\n",
      "[12]\ttrain-auc:0.95554\tval-auc:0.81821\n",
      "[13]\ttrain-auc:0.95985\tval-auc:0.81677\n",
      "[14]\ttrain-auc:0.96176\tval-auc:0.81801\n",
      "[15]\ttrain-auc:0.96558\tval-auc:0.81680\n",
      "[16]\ttrain-auc:0.96741\tval-auc:0.81807\n",
      "[17]\ttrain-auc:0.96981\tval-auc:0.81808\n",
      "[18]\ttrain-auc:0.97083\tval-auc:0.81901\n",
      "[19]\ttrain-auc:0.97252\tval-auc:0.81732\n",
      "[20]\ttrain-auc:0.97316\tval-auc:0.81775\n",
      "[21]\ttrain-auc:0.97469\tval-auc:0.81765\n",
      "[22]\ttrain-auc:0.97678\tval-auc:0.81865\n",
      "[23]\ttrain-auc:0.97925\tval-auc:0.81786\n",
      "[24]\ttrain-auc:0.98104\tval-auc:0.81937\n",
      "[25]\ttrain-auc:0.98174\tval-auc:0.81878\n",
      "[26]\ttrain-auc:0.98321\tval-auc:0.81925\n",
      "[27]\ttrain-auc:0.98491\tval-auc:0.81840\n",
      "[28]\ttrain-auc:0.98608\tval-auc:0.81904\n",
      "[29]\ttrain-auc:0.98688\tval-auc:0.81902\n",
      "[30]\ttrain-auc:0.98772\tval-auc:0.81788\n",
      "[31]\ttrain-auc:0.98895\tval-auc:0.81820\n",
      "[32]\ttrain-auc:0.99012\tval-auc:0.81687\n",
      "[33]\ttrain-auc:0.99063\tval-auc:0.81491\n",
      "[34]\ttrain-auc:0.99136\tval-auc:0.81431\n",
      "[35]\ttrain-auc:0.99180\tval-auc:0.81510\n",
      "[36]\ttrain-auc:0.99260\tval-auc:0.81346\n",
      "[37]\ttrain-auc:0.99286\tval-auc:0.81388\n",
      "[38]\ttrain-auc:0.99309\tval-auc:0.81355\n",
      "[39]\ttrain-auc:0.99388\tval-auc:0.81235\n",
      "[40]\ttrain-auc:0.99456\tval-auc:0.81387\n",
      "[41]\ttrain-auc:0.99472\tval-auc:0.81363\n",
      "[42]\ttrain-auc:0.99489\tval-auc:0.81390\n",
      "[43]\ttrain-auc:0.99540\tval-auc:0.81442\n",
      "[44]\ttrain-auc:0.99569\tval-auc:0.81465\n",
      "[45]\ttrain-auc:0.99596\tval-auc:0.81462\n",
      "[46]\ttrain-auc:0.99627\tval-auc:0.81429\n",
      "[47]\ttrain-auc:0.99654\tval-auc:0.81258\n",
      "[48]\ttrain-auc:0.99664\tval-auc:0.81237\n",
      "[49]\ttrain-auc:0.99674\tval-auc:0.81146\n",
      "[50]\ttrain-auc:0.99688\tval-auc:0.81128\n",
      "[51]\ttrain-auc:0.99725\tval-auc:0.81069\n",
      "[52]\ttrain-auc:0.99727\tval-auc:0.81007\n",
      "[53]\ttrain-auc:0.99745\tval-auc:0.80924\n",
      "[54]\ttrain-auc:0.99760\tval-auc:0.80945\n",
      "[55]\ttrain-auc:0.99787\tval-auc:0.80874\n",
      "[56]\ttrain-auc:0.99824\tval-auc:0.80833\n",
      "[57]\ttrain-auc:0.99846\tval-auc:0.80755\n",
      "[58]\ttrain-auc:0.99856\tval-auc:0.80770\n",
      "[59]\ttrain-auc:0.99874\tval-auc:0.80864\n",
      "[60]\ttrain-auc:0.99889\tval-auc:0.80801\n",
      "[61]\ttrain-auc:0.99891\tval-auc:0.80840\n",
      "[62]\ttrain-auc:0.99897\tval-auc:0.80801\n",
      "[63]\ttrain-auc:0.99903\tval-auc:0.80738\n",
      "[64]\ttrain-auc:0.99905\tval-auc:0.80770\n",
      "[65]\ttrain-auc:0.99919\tval-auc:0.80751\n",
      "[66]\ttrain-auc:0.99923\tval-auc:0.80731\n",
      "[67]\ttrain-auc:0.99932\tval-auc:0.80705\n",
      "[68]\ttrain-auc:0.99937\tval-auc:0.80746\n",
      "[69]\ttrain-auc:0.99940\tval-auc:0.80644\n",
      "[70]\ttrain-auc:0.99944\tval-auc:0.80732\n",
      "[71]\ttrain-auc:0.99946\tval-auc:0.80770\n",
      "[72]\ttrain-auc:0.99949\tval-auc:0.80799\n",
      "[73]\ttrain-auc:0.99951\tval-auc:0.80822\n",
      "[74]\ttrain-auc:0.99959\tval-auc:0.80728\n",
      "[75]\ttrain-auc:0.99964\tval-auc:0.80692\n",
      "[76]\ttrain-auc:0.99965\tval-auc:0.80787\n",
      "[77]\ttrain-auc:0.99977\tval-auc:0.80702\n",
      "[78]\ttrain-auc:0.99980\tval-auc:0.80695\n",
      "[79]\ttrain-auc:0.99983\tval-auc:0.80738\n",
      "[80]\ttrain-auc:0.99985\tval-auc:0.80677\n",
      "[81]\ttrain-auc:0.99985\tval-auc:0.80647\n",
      "[82]\ttrain-auc:0.99985\tval-auc:0.80680\n",
      "[83]\ttrain-auc:0.99985\tval-auc:0.80682\n",
      "[84]\ttrain-auc:0.99986\tval-auc:0.80661\n",
      "[85]\ttrain-auc:0.99986\tval-auc:0.80671\n",
      "[86]\ttrain-auc:0.99986\tval-auc:0.80662\n",
      "[87]\ttrain-auc:0.99990\tval-auc:0.80652\n",
      "[88]\ttrain-auc:0.99990\tval-auc:0.80654\n",
      "[89]\ttrain-auc:0.99991\tval-auc:0.80652\n",
      "[90]\ttrain-auc:0.99991\tval-auc:0.80637\n",
      "[91]\ttrain-auc:0.99992\tval-auc:0.80649\n",
      "[92]\ttrain-auc:0.99993\tval-auc:0.80546\n",
      "[93]\ttrain-auc:0.99995\tval-auc:0.80564\n",
      "[94]\ttrain-auc:0.99995\tval-auc:0.80593\n",
      "[95]\ttrain-auc:0.99996\tval-auc:0.80580\n",
      "[96]\ttrain-auc:0.99996\tval-auc:0.80538\n",
      "[97]\ttrain-auc:0.99996\tval-auc:0.80557\n",
      "[98]\ttrain-auc:0.99996\tval-auc:0.80542\n",
      "[99]\ttrain-auc:0.99997\tval-auc:0.80546\n",
      "[100]\ttrain-auc:0.99999\tval-auc:0.80508\n",
      "[101]\ttrain-auc:0.99999\tval-auc:0.80526\n",
      "[102]\ttrain-auc:0.99999\tval-auc:0.80510\n",
      "[103]\ttrain-auc:0.99999\tval-auc:0.80594\n",
      "[104]\ttrain-auc:0.99999\tval-auc:0.80637\n",
      "[105]\ttrain-auc:0.99999\tval-auc:0.80603\n",
      "[106]\ttrain-auc:0.99999\tval-auc:0.80551\n",
      "[107]\ttrain-auc:1.00000\tval-auc:0.80517\n",
      "[108]\ttrain-auc:1.00000\tval-auc:0.80491\n",
      "[109]\ttrain-auc:1.00000\tval-auc:0.80550\n",
      "[110]\ttrain-auc:1.00000\tval-auc:0.80527\n",
      "[111]\ttrain-auc:1.00000\tval-auc:0.80486\n",
      "[112]\ttrain-auc:1.00000\tval-auc:0.80519\n",
      "[113]\ttrain-auc:1.00000\tval-auc:0.80488\n",
      "[114]\ttrain-auc:1.00000\tval-auc:0.80408\n",
      "[115]\ttrain-auc:1.00000\tval-auc:0.80382\n",
      "[116]\ttrain-auc:1.00000\tval-auc:0.80448\n",
      "[117]\ttrain-auc:1.00000\tval-auc:0.80456\n",
      "[118]\ttrain-auc:1.00000\tval-auc:0.80441\n",
      "[119]\ttrain-auc:1.00000\tval-auc:0.80435\n",
      "[120]\ttrain-auc:1.00000\tval-auc:0.80431\n",
      "[121]\ttrain-auc:1.00000\tval-auc:0.80423\n",
      "[122]\ttrain-auc:1.00000\tval-auc:0.80443\n",
      "[123]\ttrain-auc:1.00000\tval-auc:0.80428\n",
      "[124]\ttrain-auc:1.00000\tval-auc:0.80349\n",
      "[125]\ttrain-auc:1.00000\tval-auc:0.80391\n",
      "[126]\ttrain-auc:1.00000\tval-auc:0.80511\n",
      "[127]\ttrain-auc:1.00000\tval-auc:0.80502\n",
      "[128]\ttrain-auc:1.00000\tval-auc:0.80509\n",
      "[129]\ttrain-auc:1.00000\tval-auc:0.80418\n",
      "[130]\ttrain-auc:1.00000\tval-auc:0.80395\n",
      "[131]\ttrain-auc:1.00000\tval-auc:0.80390\n",
      "[132]\ttrain-auc:1.00000\tval-auc:0.80425\n",
      "[133]\ttrain-auc:1.00000\tval-auc:0.80435\n",
      "[134]\ttrain-auc:1.00000\tval-auc:0.80402\n",
      "[135]\ttrain-auc:1.00000\tval-auc:0.80404\n",
      "[136]\ttrain-auc:1.00000\tval-auc:0.80379\n",
      "[137]\ttrain-auc:1.00000\tval-auc:0.80348\n",
      "[138]\ttrain-auc:1.00000\tval-auc:0.80290\n",
      "[139]\ttrain-auc:1.00000\tval-auc:0.80299\n",
      "[140]\ttrain-auc:1.00000\tval-auc:0.80278\n",
      "[141]\ttrain-auc:1.00000\tval-auc:0.80265\n",
      "[142]\ttrain-auc:1.00000\tval-auc:0.80242\n",
      "[143]\ttrain-auc:1.00000\tval-auc:0.80224\n",
      "[144]\ttrain-auc:1.00000\tval-auc:0.80235\n",
      "[145]\ttrain-auc:1.00000\tval-auc:0.80236\n",
      "[146]\ttrain-auc:1.00000\tval-auc:0.80247\n",
      "[147]\ttrain-auc:1.00000\tval-auc:0.80236\n",
      "[148]\ttrain-auc:1.00000\tval-auc:0.80224\n",
      "[149]\ttrain-auc:1.00000\tval-auc:0.80168\n",
      "[150]\ttrain-auc:1.00000\tval-auc:0.80184\n",
      "[151]\ttrain-auc:1.00000\tval-auc:0.80179\n",
      "[152]\ttrain-auc:1.00000\tval-auc:0.80143\n",
      "[153]\ttrain-auc:1.00000\tval-auc:0.80130\n",
      "[154]\ttrain-auc:1.00000\tval-auc:0.80156\n",
      "[155]\ttrain-auc:1.00000\tval-auc:0.80162\n",
      "[156]\ttrain-auc:1.00000\tval-auc:0.80179\n",
      "[157]\ttrain-auc:1.00000\tval-auc:0.80212\n",
      "[158]\ttrain-auc:1.00000\tval-auc:0.80212\n",
      "[159]\ttrain-auc:1.00000\tval-auc:0.80166\n",
      "[160]\ttrain-auc:1.00000\tval-auc:0.80143\n",
      "[161]\ttrain-auc:1.00000\tval-auc:0.80109\n",
      "[162]\ttrain-auc:1.00000\tval-auc:0.80149\n",
      "[163]\ttrain-auc:1.00000\tval-auc:0.80167\n",
      "[164]\ttrain-auc:1.00000\tval-auc:0.80135\n",
      "[165]\ttrain-auc:1.00000\tval-auc:0.80120\n",
      "[166]\ttrain-auc:1.00000\tval-auc:0.80134\n",
      "[167]\ttrain-auc:1.00000\tval-auc:0.80129\n",
      "[168]\ttrain-auc:1.00000\tval-auc:0.80133\n",
      "[169]\ttrain-auc:1.00000\tval-auc:0.80136\n",
      "[170]\ttrain-auc:1.00000\tval-auc:0.80123\n",
      "[171]\ttrain-auc:1.00000\tval-auc:0.80090\n",
      "[172]\ttrain-auc:1.00000\tval-auc:0.80050\n",
      "[173]\ttrain-auc:1.00000\tval-auc:0.80045\n",
      "[174]\ttrain-auc:1.00000\tval-auc:0.80093\n",
      "[175]\ttrain-auc:1.00000\tval-auc:0.80046\n",
      "[176]\ttrain-auc:1.00000\tval-auc:0.80073\n",
      "[177]\ttrain-auc:1.00000\tval-auc:0.80150\n",
      "[178]\ttrain-auc:1.00000\tval-auc:0.80179\n",
      "[179]\ttrain-auc:1.00000\tval-auc:0.80179\n",
      "[180]\ttrain-auc:1.00000\tval-auc:0.80128\n",
      "[181]\ttrain-auc:1.00000\tval-auc:0.80117\n",
      "[182]\ttrain-auc:1.00000\tval-auc:0.80112\n",
      "[183]\ttrain-auc:1.00000\tval-auc:0.80078\n",
      "[184]\ttrain-auc:1.00000\tval-auc:0.80101\n",
      "[185]\ttrain-auc:1.00000\tval-auc:0.80089\n",
      "[186]\ttrain-auc:1.00000\tval-auc:0.80107\n",
      "[187]\ttrain-auc:1.00000\tval-auc:0.80087\n",
      "[188]\ttrain-auc:1.00000\tval-auc:0.80109\n",
      "[189]\ttrain-auc:1.00000\tval-auc:0.80096\n",
      "[190]\ttrain-auc:1.00000\tval-auc:0.80139\n",
      "[191]\ttrain-auc:1.00000\tval-auc:0.80153\n",
      "[192]\ttrain-auc:1.00000\tval-auc:0.80148\n",
      "[193]\ttrain-auc:1.00000\tval-auc:0.80148\n",
      "[194]\ttrain-auc:1.00000\tval-auc:0.80160\n",
      "[195]\ttrain-auc:1.00000\tval-auc:0.80118\n",
      "[196]\ttrain-auc:1.00000\tval-auc:0.80083\n",
      "[197]\ttrain-auc:1.00000\tval-auc:0.80070\n",
      "[198]\ttrain-auc:1.00000\tval-auc:0.80024\n",
      "[199]\ttrain-auc:1.00000\tval-auc:0.80032\n"
     ]
    }
   ],
   "source": [
    "# num_boost_round=200 is the number of trees we let grow\n",
    "xgb_params = {\n",
    "    \"eta\": 0.3,\n",
    "    \"max_depth\": 6,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22fb2ef2",
   "metadata": {},
   "source": [
    "* This trains and evaluates 200 models\n",
    "* The model is overfitting\n",
    "* To not evaluate every step, we can set ```verbose_eval```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "acd1dfbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-auc:0.86300\tval-auc:0.76818\n",
      "[5]\ttrain-auc:0.92863\tval-auc:0.80606\n",
      "[10]\ttrain-auc:0.95002\tval-auc:0.81558\n",
      "[15]\ttrain-auc:0.96558\tval-auc:0.81680\n",
      "[20]\ttrain-auc:0.97316\tval-auc:0.81775\n",
      "[25]\ttrain-auc:0.98174\tval-auc:0.81878\n",
      "[30]\ttrain-auc:0.98772\tval-auc:0.81788\n",
      "[35]\ttrain-auc:0.99180\tval-auc:0.81510\n",
      "[40]\ttrain-auc:0.99456\tval-auc:0.81387\n",
      "[45]\ttrain-auc:0.99596\tval-auc:0.81462\n",
      "[50]\ttrain-auc:0.99688\tval-auc:0.81128\n",
      "[55]\ttrain-auc:0.99787\tval-auc:0.80874\n",
      "[60]\ttrain-auc:0.99889\tval-auc:0.80801\n",
      "[65]\ttrain-auc:0.99919\tval-auc:0.80751\n",
      "[70]\ttrain-auc:0.99944\tval-auc:0.80732\n",
      "[75]\ttrain-auc:0.99964\tval-auc:0.80692\n",
      "[80]\ttrain-auc:0.99985\tval-auc:0.80677\n",
      "[85]\ttrain-auc:0.99986\tval-auc:0.80671\n",
      "[90]\ttrain-auc:0.99991\tval-auc:0.80637\n",
      "[95]\ttrain-auc:0.99996\tval-auc:0.80580\n",
      "[100]\ttrain-auc:0.99999\tval-auc:0.80508\n",
      "[105]\ttrain-auc:0.99999\tval-auc:0.80603\n",
      "[110]\ttrain-auc:1.00000\tval-auc:0.80527\n",
      "[115]\ttrain-auc:1.00000\tval-auc:0.80382\n",
      "[120]\ttrain-auc:1.00000\tval-auc:0.80431\n",
      "[125]\ttrain-auc:1.00000\tval-auc:0.80391\n",
      "[130]\ttrain-auc:1.00000\tval-auc:0.80395\n",
      "[135]\ttrain-auc:1.00000\tval-auc:0.80404\n",
      "[140]\ttrain-auc:1.00000\tval-auc:0.80278\n",
      "[145]\ttrain-auc:1.00000\tval-auc:0.80236\n",
      "[150]\ttrain-auc:1.00000\tval-auc:0.80184\n",
      "[155]\ttrain-auc:1.00000\tval-auc:0.80162\n",
      "[160]\ttrain-auc:1.00000\tval-auc:0.80143\n",
      "[165]\ttrain-auc:1.00000\tval-auc:0.80120\n",
      "[170]\ttrain-auc:1.00000\tval-auc:0.80123\n",
      "[175]\ttrain-auc:1.00000\tval-auc:0.80046\n",
      "[180]\ttrain-auc:1.00000\tval-auc:0.80128\n",
      "[185]\ttrain-auc:1.00000\tval-auc:0.80089\n",
      "[190]\ttrain-auc:1.00000\tval-auc:0.80139\n",
      "[195]\ttrain-auc:1.00000\tval-auc:0.80118\n",
      "[199]\ttrain-auc:1.00000\tval-auc:0.80032\n"
     ]
    }
   ],
   "source": [
    "xgb_params = {\n",
    "    \"eta\": 0.3,\n",
    "    \"max_depth\": 6,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  verbose_eval=5,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "775768d1",
   "metadata": {},
   "source": [
    "* It is quite difficult to extract this information from xgboost and plot it\n",
    "* A possibility is to use the jupyter-notebook magic command \"capture\" to capture the output of a cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "2b6964e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "xgb_params = {\n",
    "    \"eta\": 0.3,\n",
    "    \"max_depth\": 6,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  verbose_eval=5,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "e6b47ad4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-auc:0.86300\tval-auc:0.76818\n",
      "[5]\ttrain-auc:0.92863\tval-auc:0.80606\n",
      "[10]\ttrain-auc:0.95002\tval-auc:0.81558\n",
      "[15]\ttrain-auc:0.96558\tval-auc:0.81680\n",
      "[20]\ttrain-auc:0.97316\tval-auc:0.81775\n",
      "[25]\ttrain-auc:0.98174\tval-auc:0.81878\n",
      "[30]\ttrain-auc:0.98772\tval-auc:0.81788\n",
      "[35]\ttrain-auc:0.99180\tval-auc:0.81510\n",
      "[40]\ttrain-auc:0.99456\tval-auc:0.81387\n",
      "[45]\ttrain-auc:0.99596\tval-auc:0.81462\n",
      "[50]\ttrain-auc:0.99688\tval-auc:0.81128\n",
      "[55]\ttrain-auc:0.99787\tval-auc:0.80874\n",
      "[60]\ttrain-auc:0.99889\tval-auc:0.80801\n",
      "[65]\ttrain-auc:0.99919\tval-auc:0.80751\n",
      "[70]\ttrain-auc:0.99944\tval-auc:0.80732\n",
      "[75]\ttrain-auc:0.99964\tval-auc:0.80692\n",
      "[80]\ttrain-auc:0.99985\tval-auc:0.80677\n",
      "[85]\ttrain-auc:0.99986\tval-auc:0.80671\n",
      "[90]\ttrain-auc:0.99991\tval-auc:0.80637\n",
      "[95]\ttrain-auc:0.99996\tval-auc:0.80580\n",
      "[100]\ttrain-auc:0.99999\tval-auc:0.80508\n",
      "[105]\ttrain-auc:0.99999\tval-auc:0.80603\n",
      "[110]\ttrain-auc:1.00000\tval-auc:0.80527\n",
      "[115]\ttrain-auc:1.00000\tval-auc:0.80382\n",
      "[120]\ttrain-auc:1.00000\tval-auc:0.80431\n",
      "[125]\ttrain-auc:1.00000\tval-auc:0.80391\n",
      "[130]\ttrain-auc:1.00000\tval-auc:0.80395\n",
      "[135]\ttrain-auc:1.00000\tval-auc:0.80404\n",
      "[140]\ttrain-auc:1.00000\tval-auc:0.80278\n",
      "[145]\ttrain-auc:1.00000\tval-auc:0.80236\n",
      "[150]\ttrain-auc:1.00000\tval-auc:0.80184\n",
      "[155]\ttrain-auc:1.00000\tval-auc:0.80162\n",
      "[160]\ttrain-auc:1.00000\tval-auc:0.80143\n",
      "[165]\ttrain-auc:1.00000\tval-auc:0.80120\n",
      "[170]\ttrain-auc:1.00000\tval-auc:0.80123\n",
      "[175]\ttrain-auc:1.00000\tval-auc:0.80046\n",
      "[180]\ttrain-auc:1.00000\tval-auc:0.80128\n",
      "[185]\ttrain-auc:1.00000\tval-auc:0.80089\n",
      "[190]\ttrain-auc:1.00000\tval-auc:0.80139\n",
      "[195]\ttrain-auc:1.00000\tval-auc:0.80118\n",
      "[199]\ttrain-auc:1.00000\tval-auc:0.80032\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(output.stdout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "53d57343",
   "metadata": {},
   "outputs": [],
   "source": [
    "s = output.stdout"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6bd51a",
   "metadata": {},
   "source": [
    "* Now we can use this string and read out the numbers we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "af157e2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[0]\\ttrain-auc:0.86300\\tval-auc:0.76818'"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line = s.split(\"\\n\")[0]\n",
    "line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "14fce7c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[0]', 'train-auc:0.86300', 'val-auc:0.76818']"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line.split(\"\\t\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "01efbf81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('[0]', 'train-auc:0.86300', 'val-auc:0.76818')"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_iter, train_auc, val_auc = line.split(\"\\t\")\n",
    "num_iter, train_auc, val_auc "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "58e20d3f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int(num_iter.strip(\"[]\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "92659d46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.863"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "float(train_auc.split(\":\")[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "d2af50da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.76818"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "float(val_auc.split(\":\")[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf5b3c2e",
   "metadata": {},
   "source": [
    "* Now generalize this process to every line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "dc10c9ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_xgb_output(output):\n",
    "    results = []\n",
    "\n",
    "    for line in output.stdout.strip().split('\\n'):\n",
    "        it_line, train_line, val_line = line.split('\\t')\n",
    "\n",
    "        it = int(it_line.strip('[]'))\n",
    "        train = float(train_line.split(':')[1])\n",
    "        val = float(val_line.split(':')[1])\n",
    "\n",
    "        results.append((it, train, val))\n",
    "    \n",
    "    columns = ['num_iter', 'train_auc', 'val_auc']\n",
    "    df_results = pd.DataFrame(results, columns=columns)\n",
    "    return df_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "ec710ef1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>num_iter</th>\n",
       "      <th>train_auc</th>\n",
       "      <th>val_auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.86300</td>\n",
       "      <td>0.76818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>0.92863</td>\n",
       "      <td>0.80606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>0.95002</td>\n",
       "      <td>0.81558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>0.96558</td>\n",
       "      <td>0.81680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20</td>\n",
       "      <td>0.97316</td>\n",
       "      <td>0.81775</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   num_iter  train_auc  val_auc\n",
       "0         0    0.86300  0.76818\n",
       "1         5    0.92863  0.80606\n",
       "2        10    0.95002  0.81558\n",
       "3        15    0.96558  0.81680\n",
       "4        20    0.97316  0.81775"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_score = parse_xgb_output(output)\n",
    "df_score.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "0b1c6085",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAi9klEQVR4nO3de3Qc9X338fdXq/vFsnWxLUu2JRtjx4FgQBgHKCEQiA0Fh5CmJiGkueCHp5CT0JNT/DSnSdqkDW2f9pymSesDqZNAuYSG+IQkhgRSiPsUgy/BYGNsfJUs3yRLtiXrvtrf88eMpLUsWWtbuyPPfl7n7JnZuex+d7T67MxvfjtrzjlERCS8MoIuQEREkktBLyIScgp6EZGQU9CLiIScgl5EJOQygy5gOGVlZa66ujroMkRELhibNm066pwrH27euAz66upqNm7cGHQZIiIXDDOrG2memm5EREJOQS8iEnIKehGRkFPQi4iEnIJeRCTkRg16M1tlZo1mtnWE+WZm3zWzXWb2tpldETdvsZnt8OetGMvCRUQkMYns0f8IWHyG+UuAOf5tOfBvAGYWAb7vz58P3G1m88+nWBEROXuj9qN3zq01s+ozLLIUeNx51zt+3cwmmlkFUA3scs7tATCzZ/xlt5131XJBc84RjTl6+2L0Rh09fTF6+2JE+wbH+289UXfq/T5HbzRGX2zw8toON+TxwfnDmHP+uPOmO0dsYL4bsow3Tvzypzzm6Jf07r/qtxtuosgo8nMyuf9Ds8f8ccfiC1OVwP64+w3+tOGmXz3Sg5jZcrwjAmbMmDEGZclY6os5mtq6OXiik0PHuzh4vJMjrV109PbR1dtHd2+M7mgfXUOGPdGYd+uL0R03nm7ZZxZ0BXIhKCvMGbdBP9xb2J1h+rCcc48CjwLU1tamWQwEKxZzHD3ZzaETXRw60cmhE10cPtE1cP/g8S6OtHYRjZ36Z8nLilCQk0lOZga5WRnkZEYGhmWFmeRkRsjOzBi8RTLIiRvPzswgK5JBVsT8YQZZmRlkR4zMDG88K2Jk98+LZJCdaWRHImRkgMWl59A3mxlkmGF4y5l5y2T0j/vDwWW88f51jcF1+p+nf7nRmFJdxpmxCPoGYHrc/SrgIJA9wnRJoZPdUQ6f6KKxtYvDrV0cae3mSGsXjW1emPffHxri2ZkZVBTnUlGcy9U1JVRMzKWiOI9pA8M8JuRmKtRELgBjEfTPAw/6bfBXAyecc4fMrAmYY2Y1wAFgGfCpMXg+GSIWcxxq7WJ340l2N/m3xnZ2N52ksa37tOWLcjKZUpzLlAk5AyE+tTiPigm5A4E+KT9LIS4SEqMGvZk9DdwAlJlZA/ANIAvAObcSWAPcCuwCOoDP+fOiZvYg8GsgAqxyzr2ThNeQlvY0neTld4/w8rZGthw4QWdv38C8CbmZzJ5cyPUXlzOrvIDKiXlMLsplanEuk4tyKMgZl9eyE5EkSaTXzd2jzHfAAyPMW4P3QSDnKRZzvLn/OC9tO8JL2w6zu6kdgPkVE7h74QwumlzI7PICZpUXUlaYrb1xERmgXbtx7s36Y/xkw35efreRoye7ycwwrp5VwmcWzeQj86dQNSk/6BJFZJxT0I9DsZjjt9sbeWztHtbva6EwJ5Mb5pZz8/wp3DB3MsV5WUGXKCIXEAX9ONLV28fqNw/w2H/vYU9TO5UT8/jG7fP5ZO10tauLyDlTeowDx9p7eOL1Oh5ft4+jJ3u4tLKYf7n7cpZcMpXMiK47JyLnR0EfIOccT75Rz3fWvEt7Tx8fnlvO8utns2hWiU6misiYUdAHZH9LBw8/9zav7W7muovK+Prt87l4SlHQZYlICCnoUyx+Lx7gb++8lLsXTtcevIgkjYI+hRqOeXvx/7PL24t/5K5L1T1SRJJOQZ8CzjmeWl/P3/7K24v/mzsv4VMLZ2gvXkRSQkGfZF29fXzp6Td5adsR7cWLSCAU9EnU0RPlvsc38j+7mvnLP5zP56+t1l68iKScgj5JWrt6+fwPN/D7+mP84x9dxl1XVgVdkoikKQV9Ehxr7+HeVet591Ar3/vUFdx6aUXQJYlIGlPQj7HGti4+84P17G1u59F7r+TGeVOCLklE0pyCfgwdPN7JPT94g0Mnuvjhn1zFtReVBV2SiIiCfqzUNbfzqcfeoLWzlye+sJDa6pKgSxIRART0Y2LH4TbuXfUG3dEYT923iEurioMuSURkgC6NeJ5e3dHIXf/2GjEHzyxXyIvI+KM9+vPw49f28Ve/eId5Uyfwg8/WMm1iXtAliYicRkF/DqJ9Mf76l9t4fF0dH3nfFP552QL9MIiIjFtKp7PU2tXLg0+9ydr3mrjvD2pYseR9RDL0bVcRGb8U9Gdhf0sHn//RBvYebec7H7+UuxfOCLokEZFRKegTtKmuheWPb6K3L8bjn1/INeojLyIXCAV9Ag4c7+SeH6xnyoQc/v1PrmJ2eWHQJYmIJExBn4DvrHmXmHP8xxev1iWGReSCo370o1i/t4Vfvn2I+z80WyEvIhckBf0ZxGKOv/7lO1QU53L/h2YHXY6IyDlR0J/BTzc1sPVAKyuWzCMvOxJ0OSIi50RBP4K2rl7+/tfbuXLmJO64bFrQ5YiInDMF/Qi+91+7OHqyh2/cPl8//yciFzQF/TD2Hm1n1f/s5RNXVvGBqolBlyMicl4U9MP4m1+9S3Ykgz//6NygSxEROW8K+iH+e2cTL797hAduvIjJE3KDLkdE5Lwp6ONE+2J865fbmFGSz+evrQm6HBGRMZFQ0JvZYjPbYWa7zGzFMPMnmdlqM3vbzNab2SVx8/aZ2RYz22xmG8ey+LH21Pp63jtykq/d9j5ys9SdUkTCYdRLIJhZBPg+cDPQAGwws+edc9viFvsLYLNz7k4zm+cvf1Pc/A87546OYd1j7nhHD//00ntcM7uUW+ZPCbocEZExk8ge/UJgl3Nuj3OuB3gGWDpkmfnAbwGcc9uBajO7oNLyX/5rF62dvXxd3SlFJGQSCfpKYH/c/QZ/Wry3gI8DmNlCYCZQ5c9zwG/MbJOZLR/pScxsuZltNLONTU1NidY/Jtq6enlmfT1LF1Qyb+qElD63iEiyJRL0w+3euiH3HwEmmdlm4EvAm0DUn3etc+4KYAnwgJldP9yTOOcedc7VOudqy8vLEyp+rKx+8wDtPX3c+8GZKX1eEZFUSOQyxQ3A9Lj7VcDB+AWcc63A5wDMa/fY699wzh30h41mthqvKWjteVc+RpxzPL6ujksri1kwfWLQ5YiIjLlE9ug3AHPMrMbMsoFlwPPxC5jZRH8ewBeBtc65VjMrMLMif5kC4BZg69iVf/7W7WlmV+NJ7v3gTLXNi0gojbpH75yLmtmDwK+BCLDKOfeOmd3vz18JvA943Mz6gG3AF/zVpwCr/QDNBJ5yzr049i/j3D3+Wh0T87O4XRcuE5GQSugXppxza4A1Q6atjBtfB8wZZr09wGXnWWPSHDrRyUvvHuGL19Wo37yIhFZafzP2qTfqiTnHPYt0ElZEwittg7472sfT6+u5ce5kppfoJwJFJLzSNuhf3HqYoyd7+Iy6VIpIyKVt0D++ro7q0nyun5PaPvsiIqmWlkG/9cAJNtUd455FM8nIUJdKEQm3tAz6J9bVkZuVwR9dOX30hUVELnBpF/QnOnr5+VsHuPPySorzs4IuR0Qk6dIu6P9z0366emN8ZlF10KWIiKREWgV9LOZ44vU6amdOYv40XaVSRNJDWgX973Y2Udfcwb3XVAddiohIyqRV0D+xro6ywhwWv39q0KWIiKRM2gT9/pYOXtnRyKcWTic7M21etohI+gT9G3tbcA7uWKCrVIpIekmboK9vbifDYEZJQdCliIikVNoEfV1LB9Mm5qnZRkTSTtqkXl1zBzNLdZVKEUk/aRP09S0darYRkbSUFkHf1tVLS3uP9uhFJC2lRdDXNXcAMFM/MCIiaSgtgr6+xQv6GdqjF5E0lBZBP7BHX6o2ehFJP2kR9PUt7ZQWZFOYkxl0KSIiKZcWQV/X3KFmGxFJW2kT9DoRKyLpKvRB3xONcehEJzPUPi8iaSr0Qd9wrIOYU9dKEUlfoQ/6upb+HjcKehFJT6EP+vpm9aEXkfQW+qCva+4gPztCeWFO0KWIiAQi9EFf39LOjJJ8zCzoUkREAhH6oK9r7mCGTsSKSBoLddDHYo76Fl2HXkTSW6iDvrGtm+5oTH3oRSStJRT0ZrbYzHaY2S4zWzHM/ElmttrM3jaz9WZ2SaLrJlNdczugPvQikt5GvcqXmUWA7wM3Aw3ABjN73jm3LW6xvwA2O+fuNLN5/vI3Jbhu0qgPvUj66O3tpaGhga6urqBLSarc3FyqqqrIyspKeJ1ELue4ENjlnNsDYGbPAEuB+LCeD3wHwDm33cyqzWwKMCuBdZOmvrmDSIYxbWJeKp5ORALU0NBAUVER1dXVoe1l55yjubmZhoYGampqEl4vkaabSmB/3P0Gf1q8t4CPA5jZQmAmUJXguklT19JB5cQ8siKhPhUhIkBXVxelpaWhDXkAM6O0tPSsj1oSScDhtpobcv8RYJKZbQa+BLwJRBNc13sSs+VmttHMNjY1NSVQ1ujqm9vVbCOSRsIc8v3O5TUmEvQNwPS4+1XAwfgFnHOtzrnPOecWAPcC5cDeRNaNe4xHnXO1zrna8vLyxF/BGdS1qA+9iKTG8ePH+dd//dezXu/WW2/l+PHjY19QnESCfgMwx8xqzCwbWAY8H7+AmU305wF8EVjrnGtNZN1kOdHZy/GOXu3Ri0hKjBT0fX19Z1xvzZo1TJw4MUlVeUY9Geuci5rZg8CvgQiwyjn3jpnd789fCbwPeNzM+vBOtH7hTOsm56WcauBiZiXqQy8iybdixQp2797NggULyMrKorCwkIqKCjZv3sy2bdv42Mc+xv79++nq6uLLX/4yy5cvB6C6upqNGzdy8uRJlixZwnXXXcdrr71GZWUlP//5z8nLO//OJAn9iKpzbg2wZsi0lXHj64A5ia6bCnUtfh967dGLpJ2/+sU7bDvYOqaPOX/aBL5x+/tHnP/II4+wdetWNm/ezKuvvsptt93G1q1bB3rHrFq1ipKSEjo7O7nqqqu46667KC0tPeUxdu7cydNPP81jjz3GJz/5SZ577jnuueee8649tL+WXTewR6+gF5HUW7hw4SldIL/73e+yevVqAPbv38/OnTtPC/qamhoWLFgAwJVXXsm+ffvGpJbQBn19cwdlhTkU5IT2JYrICM60550qBQWDzcavvvoqL7/8MuvWrSM/P58bbrhh2C6SOTmDl1OPRCJ0dnaOSS2h7WBe16KulSKSOkVFRbS1tQ0778SJE0yaNIn8/Hy2b9/O66+/ntLaQru7W9/cwaJZpaMvKCIyBkpLS7n22mu55JJLyMvLY8qUKQPzFi9ezMqVK/nABz7A3LlzWbRoUUprC2XQd0f7ONTapZ8PFJGUeuqpp4adnpOTwwsvvDDsvP52+LKyMrZu3Tow/atf/eqY1RXKppv9LZ04px43IiIQ0qCv97tWqg+9iEhIg76/a6X26EVEQhz0BdkRSguyR19YRCTkQhn09S0dzCgtSIsr2YmIjCaUQV/X3K6fDxQR8YUu6GMxx/5jnWqfF5FxrbCwMGXPFbqgP9zaRU80pj70IiK+0H1haqDHjbpWikgKPfzww8ycOZM//dM/BeCb3/wmZsbatWs5duwYvb29fPvb32bp0qUpry10QV+vyxOLyAsr4PCWsX3MqZfCkkdGnL1s2TK+8pWvDAT9s88+y4svvshDDz3EhAkTOHr0KIsWLeKOO+5IeUeR0AV9XXMHmRlGRXFu0KWISBq5/PLLaWxs5ODBgzQ1NTFp0iQqKip46KGHWLt2LRkZGRw4cIAjR44wderUlNYWvqBv6aBqUh6ZkdCdfhCRRJ1hzzuZPvGJT/DTn/6Uw4cPs2zZMp588kmamprYtGkTWVlZVFdXD3t54mQLXdDXN3t96EVEUm3ZsmXcd999HD16lN/97nc8++yzTJ48maysLF555RXq6uoCqSt0QV/X3M6C6RODLkNE0tD73/9+2traqKyspKKigk9/+tPcfvvt1NbWsmDBAubNmxdIXaEK+uMdPbR2RXUiVkQCs2XL4EngsrIy1q1bN+xyJ0+eTFVJ4epHr9+JFRE5XbiCvqX/qpVqoxcR6ReqoK9v7r8OvfboRUT6hSro65o7mFyUQ152JOhSRCQAzrmgS0i6c3mN4Qr6lg6diBVJU7m5uTQ3N4c67J1zNDc3k5t7dl8IDVWvm/rmDq69qCzoMkQkAFVVVTQ0NNDU1BR0KUmVm5tLVVXVWa0TmqDvizkqJ+Uxf9qEoEsRkQBkZWVRU1MTdBnjUmiCPpJhPPe/rwm6DBGRcSdUbfQiInI6Bb2ISMgp6EVEQk5BLyIScgp6EZGQU9CLiIRcQkFvZovNbIeZ7TKzFcPMLzazX5jZW2b2jpl9Lm7ePjPbYmabzWzjWBYvIiKjG7UfvZlFgO8DNwMNwAYze945ty1usQeAbc65282sHNhhZk8653r8+R92zh0d6+JFRGR0iezRLwR2Oef2+MH9DLB0yDIOKDLvp80LgRYgOqaViojIOUkk6CuB/XH3G/xp8b4HvA84CGwBvuyci/nzHPAbM9tkZstHehIzW25mG81sY9ivVSEikkqJBL0NM23o5eE+CmwGpgELgO+ZWf9FZ651zl0BLAEeMLPrh3sS59yjzrla51xteXl5IrWLiEgCEgn6BmB63P0qvD33eJ8DfuY8u4C9wDwA59xBf9gIrMZrChIRkRRJJOg3AHPMrMbMsoFlwPNDlqkHbgIwsynAXGCPmRWYWZE/vQC4Bdg6VsWLiMjoRu1145yLmtmDwK+BCLDKOfeOmd3vz18JfAv4kZltwWvqedg5d9TMZgGrvXO0ZAJPOedeTNJrERGRYdh4/DWW2tpat3GjutyLiCTKzDY552qHm6dvxoqIhJyCXkQk5BT0IiIhp6AXEQk5Bb2ISMgp6EVEQk5BLyIScgp6EZGQU9CLiIScgl5EJOQU9CIiIaegFxEJOQW9iEjIKehFREJOQS8iEnIKehGRkFPQi4iEnIJeRCTkFPQiIiGnoBcRCbnMoAsIhWgPtB2C1gNwosG7DYwfgBP7wcWgaKp3K/SHRRVQNAWKpkHFZZCdH/QrEZEQUtCPpi/qBfWxfV54t/qB3nYIWg96w/am09fLnQjF06G4EqYvhIxMOHkY2g7D/te9YV/P4PKZeTD7Rph3G1y8GApKU/UKRSTkFPT9Oo/D3rVeoB/bCy17veHx/eD6Tl02v9TbC59QAdMuhwnTvL3z4irvNqEScgrP/HzOQecxL/CP18Gu38KONbDjV2AZMH2RF/rzboWSWcl61SKSBsw5F3QNp6mtrXUbN25M3RO2HYEfLoaWPd793IlQUgOTagaHk6q9EC+qgKzc5NThHBzaDNvXwPZfQeM73vTSi6Ds4lPrKamBiTMgkjW4fl8UOpqh46h3lNF+FLpOQPV1UD43OTWLyLhgZpucc7XDzdMefUcLPHGnF/Z3/wRmXA15k4Kpxcw7Qph2Odz4Ne+oYscLsO//eR9Cu1+BaGfc8hHvwyeS7YV757GRH7vySrjsbrjkLsgvSf5rEZFxI7336Lvb4PGlcHgLfOpZmP3h5D/n+XDOa+qJb1pq2QuxKBSUQ0GZ16w0MF4GmTne0cHmp7wjhEg2zF0CCz4Ns2+CiD7rRcLgTHv06Rv0vV3w5Ceg7jX44ye89vAwcw4Ovw2bn4Ytz3pNPAWT4dJPwJxbYOY13oeCiFyQFPRD9fXCTz4D770Adz4Kl/1x8p5rPIr2wK6XvL38nb/xev9k5UPN9XDRR7xbSU3QVYrIWVAbfbxYH6y+3wv52/4x/UIeIDPb79FzG/S0e+cAdr7khf97L3rLlMyGOTfD1Eu9JqCBZqEyyC70zieIyAUhvYLeOfjlQ7D1p/CRb8JVXwy6ouBlF8DFH/VuAM27YdfL3m3Tj089+dsvkuOFfn4p5BZDThHkTvCGOUWQ449nF3q9giJZkJHlnR+IZHrDjCyv91J2AWQVeF8WyyqADH1ZW2SspU/QOwcv/SX8/sdw3Z/BdQ8FXdH4VDrbu139vyDa7X8hrL/L5tG4YbN362r1vgHc2Oqd3O5u9U4On6vMPD/88/0PiEzvFskcHM/IhIyI930DzBtahneU0T8eyfYeIyvXH+ZBZtx4brF3y5vodaftvx/fXfVsRbu97qy5E72jJpFxIn2Cfvd/wWv/AlfdBzd9PehqLgyZOd73ByZVJ76OcxDt8kK/56TXtz/W650H6It6w1ivd54g2uU1HfW2e8OejlPHY9HTb329XvNbtAdw3qUlXP8w5k2LxaCv2zvh3tsBvZ3ekATOR2UVeN1r80sGm6vy+5utSiGvxPswaz0EbQf9b0of9MY7mgcfJ7fYb/Lye0AV+OMTKr3vNJTPUzdXSZn0Cfo9r3h7ebd8W+3LyWTm7TFn5QGTg65mkHPeh0xvh/ch0t3qfRu660Tczb/f0TJ4xNKyx7vf3Xr6Y+aXed+OnjANqmq9Yd4k7/sM7f1fWmvymsP2v+E9noudun75PCi/2BuWXuRtv+62ITf/aKm3y9uu2QWDRz3949kF3pGOc3gfgHHD/g/A/vFYn//B2Dd4v/9DvWSWdy0mNaGFSvoEfd06mHZF8r7VKuObmRdmmTn+F+Iqz279aLcf/i3e+YeiqWffHTXW5zVzHX0PmrZD0w5vfOtz3gfMSLLyveeM5HjnTHra/SOUJMnMGwz9kho//Kf4HyiFcR8u/nhmjnaexrmEgt7MFgP/DESAHzjnHhkyvxj4D2CG/5j/1zn3w0TWTYmeDu/SAtd8KeVPLSGRmePtsU+Ydu6PkRGBSTO925ybB6c7BycboXmXt8zASe0iyC4a/kttsdhg6PffYlE/cC1umHHqeEbEP5cRibuf4TVvHdvnHcG07PG+iNeyB3b/1mtiOxOLxJ2QL/aHEwanZRd6R9OZ2f4J+Wz/JH1O3Hj/CfvhTt7neNMyc04fjz/ycEOOZPq3+bl8CPX1eh/uWfmJH93EYl7TY3eblzn9HQ8GXqM/nhE5+3rO06hBb2YR4PvAzUADsMHMnnfObYtb7AFgm3PudjMrB3aY2ZNAXwLrJt+Bjd4/wYwPpvRpRRJi5l+uekri62RkDO5Zj5WSmtO/HR6LDZ5/GPhQOTl4HqXnpHfrbvNOzHe3esPWhsH7Pe2nXql1TBlnPPeSmRd3nqT81PHcCd4R2slGaG/0hv3j8edbsvL9ZrL8uB5i+d4HwdAmtkTOA1mGdw6nuAqKZ8DE6f6Vbqu861dVXnG+G+X0zZDAMguBXc65PQBm9gywFIgPawcUmZkBhUALEAWuTmDd5Kt/HTDvcsEikriMjMGrsp4P5/yT6T1eQPb1n6DvGRyP9Z56wn7oMtHuwfv947Eopx7BxA/xz7k0e+dKTh6GI1u98fgPnqx8KJzsfVO8dDbM/KA3npXrHen0N5X1f7D1n+DPzvc+nHPijmAGjsQKBl9v/OuI9ngdBTqPec14LXtg7++8xwXvvM2f7z6/bT2MRIK+Etgfd78BL8DjfQ94HjgIFAF/7JyLmVki6wJgZsuB5QAzZsxIqPiE1b0Gk+cHd7EykXRnNtgsM5ZHIefCOf/I44TXi2q0S4qnop6u494l0c90ruY8JBL0wzVwDT0++SiwGbgRmA28ZGb/neC63kTnHgUeBe8SCAnUlZi+KDRsgMuWjdlDisgFzGzwexPjgZm3E5rEHdFEzjI0ANPj7lfh7bnH+xzwM+fZBewF5iW4bnId2eIdFql9XkTSVCJBvwGYY2Y1ZpYNLMNrpolXD9wEYGZTgLnAngTXTa66dd5QQS8iaWrUphvnXNTMHgR+jddFcpVz7h0zu9+fvxL4FvAjM9uC11zzsHPuKMBw6ybnpYygfp13Jrv4LPtNi4iEREL96J1za4A1Q6atjBs/CNyS6Lop45wX9LNvDOTpRUTGg3B/z7l5t9eVSs02IpLGwh309X77/Mxrgq1DRCRA4Q/6vBIouzjoSkREAhP+oJ/xQV1wSUTSWniDvu2I9/XimWqfF5H0Ft6gr3/NG+pErIikuRAH/evelesqLgu6EhGRQIU36Ote837153x+A1REJATCGfRdrd7lSNWtUkQkpEHfsN77LUy1z4uIhDTo69Z5P29WdVXQlYiIBC6cQV//OlR8IPgfFBARGQfCF/TRbu83YtVsIyIChDHoD272frVeQS8iAoQx6Ov1QyMiIvHCGfSlF0FhedCViIiMC+EK+ljMOxGrvXkRkQHhCvqm7dB1XEEvIhInXEHffyEzXbFSRGRAyIL+dSicCpNqgq5ERGTcCFfQ162DGYv0QyMiInEygy5gzES7YdYNMOtDQVciIjKuhCfoM3PgY98PugoRkXEnXE03IiJyGgW9iEjIKehFREJOQS8iEnIKehGRkFPQi4iEnIJeRCTkFPQiIiFnzrmgaziNmTUBdee4ehlwdAzLGSuq6+yN19pU19lRXWfvXGqb6Zwb9oc4xmXQnw8z2+icqw26jqFU19kbr7WprrOjus7eWNemphsRkZBT0IuIhFwYg/7RoAsYgeo6e+O1NtV1dlTX2RvT2kLXRi8iIqcK4x69iIjEUdCLiIRcaILezBab2Q4z22VmKwKsY7qZvWJm75rZO2b2ZX/6N83sgJlt9m+3BlTfPjPb4tew0Z9WYmYvmdlOfzgpxTXNjdsum82s1cy+EsQ2M7NVZtZoZlvjpo24fczs//jvuR1m9tEAavsHM9tuZm+b2Wozm+hPrzazzrhttzLFdY34t0vVNhuhrp/E1bTPzDb701O5vUbKiOS9z5xzF/wNiAC7gVlANvAWMD+gWiqAK/zxIuA9YD7wTeCr42Bb7QPKhkz7e2CFP74C+LuA/5aHgZlBbDPgeuAKYOto28f/u74F5AA1/nswkuLabgEy/fG/i6utOn65ALbZsH+7VG6z4eoaMv8fga8HsL1Gyoikvc/Cske/ENjlnNvjnOsBngGWBlGIc+6Qc+73/ngb8C5QGUQtZ2Ep8GN//MfAx4IrhZuA3c65c/1m9Hlxzq0FWoZMHmn7LAWecc51O+f2Arvw3ospq8059xvnXNS/+zpQlaznP5u6ziBl2+xMdZmZAZ8Enk7Gc5/JGTIiae+zsAR9JbA/7n4D4yBczawauBx4w5/0oH+IvSrVzSNxHPAbM9tkZsv9aVOcc4fAexMCkwOqDWAZp/7zjYdtNtL2GW/vu88DL8TdrzGzN83sd2b2BwHUM9zfbrxssz8AjjjndsZNS/n2GpIRSXufhSXobZhpgfYbNbNC4DngK865VuDfgNnAAuAQ3mFjEK51zl0BLAEeMLPrA6rjNGaWDdwB/Kc/abxss5GMm/edmX0NiAJP+pMOATOcc5cDfwY8ZWYTUljSSH+78bLN7ubUHYqUb69hMmLERYeZdlbbLCxB3wBMj7tfBRwMqBbMLAvvD/ikc+5nAM65I865PudcDHiMJB7in4lz7qA/bARW+3UcMbMKv/YKoDGI2vA+fH7vnDvi1zguthkjb59x8b4zs88Cfwh82vmNuv5hfrM/vgmvXffiVNV0hr9d4NvMzDKBjwM/6Z+W6u01XEaQxPdZWIJ+AzDHzGr8vcJlwPNBFOK3/f078K5z7p/iplfELXYnsHXouimorcDMivrH8U7kbcXbVp/1F/ss8PNU1+Y7ZS9rPGwz30jb53lgmZnlmFkNMAdYn8rCzGwx8DBwh3OuI256uZlF/PFZfm17UljXSH+7wLcZ8BFgu3OuoX9CKrfXSBlBMt9nqTjLnKIz2bfinb3eDXwtwDquwzusehvY7N9uBZ4AtvjTnwcqAqhtFt7Z+7eAd/q3E1AK/BbY6Q9LAqgtH2gGiuOmpXyb4X3QHAJ68fakvnCm7QN8zX/P7QCWBFDbLrz22/732kp/2bv8v/FbwO+B21Nc14h/u1Rts+Hq8qf/CLh/yLKp3F4jZUTS3me6BIKISMiFpelGRERGoKAXEQk5Bb2ISMgp6EVEQk5BLyIScgp6EZGQU9CLiITc/wdE33Al1nRqzQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the results\n",
    "plt.plot(df_score.num_iter, df_score.train_auc, label=\"train\")\n",
    "plt.plot(df_score.num_iter, df_score.val_auc, label=\"val\")\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d53a218e",
   "metadata": {},
   "source": [
    "* We can see that we are overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f813c34",
   "metadata": {},
   "source": [
    "## XGBoost Parameter Tuning\n",
    "* Tuning the following parameters\n",
    "    * ```eta```: learning rate, stepsize\n",
    "        * Defines how much weight the next model has\n",
    "        * E.g. eta = 1: only the next model's predictions are used, eta 0 0.3; 30% of the next model's predictions are used\n",
    "    * ```max_depth```\n",
    "    * ```min_child_weight```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "f6dca0ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "id": "5c98193a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "xgb_params = {\n",
    "    \"eta\": 0.1,\n",
    "    \"max_depth\": 6,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  verbose_eval=5,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6744b98",
   "metadata": {},
   "source": [
    "* Try different paramters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "2892c15e",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = f\"eta={xgb_params['eta']}\"\n",
    "scores[key] = parse_xgb_output(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "61ac36b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['eta=0.3', 'eta=1', 'eta=0.1', 'eta=0.05', 'eta=0.01', 'eta=1.0'])"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "36c300db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABEXklEQVR4nO3dd5xU1fn48c/Z2dnee6OzLJ2VKvZAUDACdrF3xKix5GdJ1HwTY7rRxJIgiUYssSSCEEMsERQ1dFiku0vdXtmZbTM75fz+uMOysLswu8wWZp7363VfM3Prmcvy3DvnnvMcpbVGCCGE/wrq7QIIIYToXhLohRDCz0mgF0IIPyeBXggh/JwEeiGE8HPBvV2A9iQlJemBAwf2djGEEOK0sWnTpiqtdXJ7y/pkoB84cCAbN27s7WIIIcRpQyl1sKNlUnUjhBB+TgK9EEL4OQn0Qgjh5yTQCyGEn5NAL4QQfk4CvRBC+DkJ9EII4eck0Ise53A5+HDfh2wo29DbRREiIPTJDlOib3G4HFiaLdTaaqm112JptmCxW2hyNnFWxlkMih3k1X5cbhcf7vuQP2/9M8X1xQCck3kOD054kGHxw7rzKwgR0CTQizZqbbWs2L+CD/d9yN7avTQ6G0+4/viU8VyWfRkXDriQCHNEm+Vaaz479BkvbHmBfZZ9jEwcyY+n/Jh9tftYtG0RV/3rKuYOmcs9ufeQGpnaXV9LiICl+uIIUxMnTtSSAqFnOdwOvi7+mmUFy/i86HOcbicjEkYwIXUCcaFxxIXGERsaS2xobMt7gI8OfMTS/KUcsB4g0hzJrEGzuHzo5YxOGg3AmtI1PL/5eXZU72BQ7CDuO+M+vtv/uyilALDYLSz6ZhFv734bkzJx48gbuW30bUSFRPXauRDidKSU2qS1ntjuMgn0ge3bw9+yrGAZH+77kBpbDQlhCVwy+BLmDJlDTkKOV/vQWrO5YjNL8pfwyYFPsLlsZMdnExMSw6byTaRHpnP3uLuZPWQ2wUHt/4gsqivihS0vsGL/CuJD47lz7J2MTx1PZmQmsaGxLReGrtJac6juEFsrt3LAcoD+Mf0ZkTCCwXGDMQeZT7hdaUMpeRV5bKnYwvaq7djddkKCQggxhRASFILZZD762RRCeHA4EeYIIoIjjnkfERxBRlQG2fHZBCl5PCZ8SwK9aOPbw9/y7MZn+brka4KDgrkg6wLmDp3L2ZlnnzDwnUxdcx3/2f8fluYvpcpWxS2jbuGqYVcRYgrxavsdVTv4/abfH/OgNjw4nMyoTNIj08mIyiAjKoPUiFRiQ2OJCYlpeY0OiW65kDQ6GtlRvYOtlVvZWrGVrZVbOWw/DIBCoTH+7kOCQhgaP5QRCSMYnjCcEYkjMClTS2DPq8yjorGipRxjksYQZY6i2d2Mw+Wg2d1Ms6u55bPdZafJ2USjo5Fmd3O73zE+NJ5JaZOYkj6FKelT6B/d/5QvZEJIoBctKhsreSnvJZYWLCXKHMXtY27nsqGXER8W39tFa6G1pqC2gEN1hyipL2mZShtKKa4vxtps7XDbKHMUUSFRVDZW4tIuAAbGDCQ3JZdxyeMYlzyOgbEDKawrZHf1bnbX7GZnzU521+zGYrccs6+MyAzGpYzjjJQzyE3OJTs+u8NfJO1xup0tQb/J2USDs4F9tftYW7qWdaXrKG8sByAtMo0paUbQv6DfBUSHRLc5HyUWGwerG2h2unG4NA6X2zMdfa+UIiw4iDCziXCziTCziTBzkOfVRHRYMNFhwYSbTXJh8UMS6AWNjkYW71zM37b/DYfbwbyceSwYt6Clrv10Ut9cT0VTBVa7FWuzFYvdgrXZitVuxdJswWq3khaZRm5KLmOTxhIXFnfSfWqtKWsoY1fNLpxuJ2OTx5IWmdZt30FrzUHrQdaXrWdt6Vo2lG2g1l5LmCmMqakzGBI6g8O1KewstbK71IrV5vTZsYODlCfom4kOCybG8xoVFkx0qPEaFWo++jk0mLTYMAYnRxIRIu03+ioJ9AHM5Xbxr33/4oXNL1DRVMGMATN4YPwD9I/p39tFC2gOl5sDVQ18W17Pt+V1fFtuYUf1Dir4guCYraggB9rWn1T1HSYmfYfRGUkMSYokLMREiCmIYJPCbAoixBSE2fPZrTV2hxubw0WTw4XtmPcu6mxOz+SgzubE6nk98rne7mx5dbnbjwuZceEMTYk6ZspOiSIuwruqOdF9ThTo5fLsx+qa67jzkzvZUb2DMUlj+N35v2N86vjeLlZA0VpTarGxq9TKrlIru8vqyC+vZ19VPQ6XEUyVggEJEeSkjubStKkMTA6iyPklnxYt5YB1MV83f0Cy6VLOT7m6Ry7QWmtsDjd1dgf1notDcW0TBRX1LdO6/dXYHO6Wbc7NTuKmqQOZNjwFU5BUC/U1ckfvx57Z8Ayv73ydX5zzCy4ZfInUy3Yzt1uzs9TKzhIrO1sFdkuTo2WdrPhwclKjyU6NJictiuyUaIamRBFmNrXZn9aaDWUbeHfPu6w8tBI3bmYPns09ufeQHpXek1+tDbdbtwT/LYW1vLehkDKrjcy4cG44cwDXTOpHQqTc5fckqboJQAetB7l02aXMHjybp85+qreL49ccLjfL80p4efVevi2vByAixEROWjQj0mOMKS2anLRoosO61qKpsrGSxTsW8/butwGYN3wed4y5o888RHe43Px3ZzmL1xxg7b4aQoKDmD02g5vPGsDYrLjeLl5AkEAfgH6w8gesK13Hvy//N0nhSb1dHL/U2OzknfWFvPLVfoprmxieFs1tZw9i0qAEBiREENQNVRil9aX8aeufWL53ORHBEdwy6hZuHHljuz2Se8uesjreWHuAJZuLaWx2kZMazajMGIanRTMsNZrhaTGkxoTKL0wfk0AfYNaWruXOT+7k/vH3c8eYO3q7OH6npqGZ1/53gNfXHKC20cHkQQncff4QLshJ7rHgVXC4gOe3PM+qwlUkhiWyYNwC5g6dS5gprM8EUKvNwfubili5u4I9ZXVU1NlblsWGm8lJjWZYWhSjMmIZlxXHsNQogk3SkayrJNAHEKfbyVX/uoomZxPLLl1GqCm0t4t02rI5XJRZbJRabJRZmyiptbG/qoEPvynB5nAzY2QqC84fwoQBvVd9kleRx3ObnmNzxWYAgoOCiTJHEWmObHmNDokmKiSKtIg0sqKzyIrOIjMqk7TItFPqHNdZhxua+ba8jj3ldewp80zlddR5mo6GmYMYk2kE/XH94hiXFUe/hPA+c+Hq60450CulZgJ/BEzAX7XWvz5ueSzwJtAfoyXPM1rrvyml+gGvA2mAG1iktf7jyY4ngb7r3tvzHj9f+3OeveBZZgyY0dvFOa3sKavjzbUH2XjwMGWWJg43OtqsExtuZsbIVO46bzDZqdHt7KXnaa1ZU7KGnTU7qW+up95RT4Oj4ehrcz3WZivljeU43Ufb45uUibRIT/CPyiI1MpX0yHTSItNIi0gjLTKNsOCwbi/7wepGthbVsrXQwtaiWrYXW7A7jRY9MWHBxEaYCQs2tXQECzUf7RQWG25mYFIkg5IiGJQURVZ8OOYT/CpwuNxU1tkps9qwNDnIjAunf0JEuw/DTzenFOiVUibgW2AGUARsAK7VWu9stc6PgVit9aNKqWRgD0ZwTwTStdablVLRwCbg0tbbtkcCfddYm61csuQSBscN5m8X/U3uhLzQ7HTzyc4yXl9zkPX7jYeIZw1JJCs+nPTYcNJiwkiPDSPNM53OHYZcbhcVjRUU1RdRVFdEYV0hxfXFLZ9rbDVttokLjSM9Mp3UyFRSI1JJi0wjNcJ4nxqZSkpECuHB4T4tp8PlZk9ZHVuLatlVaqXRfrQvgM3hbnlvd7qpqre3/CIAMAUp+idEMDAxggGJkTjdbsosdsqtNsqsNqrq7Rwf8pSCjNhwBidHMjAxkoFJkQxOimREegxpsd5f6LTW5BXWsmRzMV/vrcLhcuN2g1tr3FrjchvruLUmIiSYrPhw+iVEkBUfTlb8kVfjfVecajv6yUCB1nqfZ2fvAHOB1sFaA9HKiCxRQA3g1FqXAqWek1CnlNoFZB63rfCRRVsXUWuv5dFJj0qQP4kyi42/rz/E2+sPUVlnJys+nMdmDefqif7bLNAUZCI9Kp30qHQmpU1qs7zZ1Ux5QzlljWWUNZRR2lBKWYPxvqS+hC0VW9qkiQAjd8+Y5DFMSp3EpLRJDE8Yjimo63fIZlMQozNjGZ158l7bWmtqGpo5UN3AvsoGDlQ3sL+qgf1VjazzXLjTYsJIjQljZHoMqbFhpMWEkRYbSkyYmeLaJs/6DRyoauCDvOJjLhzZKVGcNyyZ84clM3lQQrt3/oU1jXywpZilW4rZV9VAaHAQ52YnERNmRilFkDIuQEopTEEQpBR1NidFhxv5Kr+K8jpby8UnLsJM3k8u7PK564g3gT4TKGz1uQiYctw6LwLLgRIgGrhGa+1uvYJSaiBwBrCuvYMopeYD8wH695dem5110HqQt3a/xWXZlzEicURvF6fPOljdwK//s5tPdpbj1poLhiVz49QBnD9MOvqEmELoF9OPfjH9OlynydlERWMF5Q3llDcaU2FdIZvLN7O6aDVg5BuakDqBSWlG4M+JzzmlwH8iSikSo0JJjAplwoCETm9//O3vkQvH/qoGthyqZXV+JW+sPcgrX+0nNDiIKYMTOX9YMmcOTmB7sYX3Nxezfr/xS2jKoAQWnD+EmWPSiOlEM1q700VprY2iw03U29tWF/qCN1U3VwEXaa3v8Hy+EZistb6v1TpXAmcDDwFDgE+BcVprq2d5FPAF8Aut9ZKTFUqqbjrvvpX3sb50vTSn7IDWmqVbinnyg+0EKcV1U/pz/ZQB9E/sO80ST3cVjRVsLNvI+rL1bCzfyEHrQQBCTaGkR6YbU1R6y/uMqAzSItNIDEskPLjvPnRtanaxdn81X+ypZHV+JfsqG1qWDU6O5PIzMpmbm0m/hN79WzrVqpsioPUlPgvjzr21W4Ffa+OqUaCU2g8MB9YrpczA+8Bb3gR50XlrStbweeHn3D/+fgny7bDaHDyxdDvLt5YweVACz12TS2acb+uVBaREpHDx4Iu5ePDFAJQ3lLOxfCM7q3e2VAOtLlpNVVNVm21NytTSQig6JLolC2l8aDznZJ7DeVnndfuD4Y6Eh5j4Tk4K38lJAYyqmnX7axiaEsW4rFMfK6EneHNHH4zxMHY6UIzxMPY6rfWOVuv8GSjXWv9UKZUKbAbGAdXAYqBGa/2At4Xyqzt6ex1seQucTTDuOoj27VB50pzyxDYdrOH+d/Iotdh48LvZ3H3B0ICvoultdped8oZyShtKKakvodZeS11zHfWOeuqb66lz1Bmfm+spbyyn1l5LlDmK6f2nc/Hgi5mcNrlT6aIDxSnd0WutnUqpe4GPMZpXvqq13qGUWuBZvhD4OfCaUmoboIBHtdZVSqlzgBuBbUqpPM8uf6y1XnHK36qva6iGdQth/SKw1RrzVj4Nwy+BSXfAwHOMx/2naEn+EgpqC3j2gmclyLfidLl5cVUBz3+WT1Z8BP9YMJXx/ftGuoBAF2oKpX9Mf68StLncLtaXrWfF/hX89+B/WbZ3GYlhicwcNJOLB13MmKQxAC0DvticNpqcTcY4AM5GLHYLVU1VVNuqqW6qPuZ9XXMdqZGpZEVltTQxbd3PwJ/+P0mHKV+rLYQ1L8KmxcZd/PBL4OwHIDweNr4KeW8ZgT8pBybeBuPmQXhclw61rXIbt39yO6OTRvPKha+cFj8he0JhTSMPvpvHxoOHufyMTH42d1SXc8yIvsPusvNl0Zes2L+CLwq/oNndTKgplGZXc8uIYR1RKOLD4kkMTyQxLJHE8ESizFGUN5S3NC+1uWzHbDM0bijfG/w9Lhl8SbeOTeAr0jO2J1Tuga/+ANveMz6PuRrOvh9Shh+7nqMJti+Bja9A8SYwR8CYK2HakxCV4vXhDlgOcNN/biLSHMkbF78hdfMYPVlf+Wo/f1pVgFKKpy8dzaVnZPZ2sUQ3qGuu47NDn1FwuICw4DDCg8MJCw4jIjjimM9xoXEkhScRFxp3wuoerTXVtuqW/gVF9UWsKVnDlootKBRT0qcwZ8gcpvef3qfyCrUmgb67aA37VsHahZD/MQSHw4SbYeq9ENdxE7UWJXlGwN/6LqSNgVtXQPDJfy5WNVVxw4obaHQ08sbFbzAgZsCpf5fTmNaa5VtL+O1HeyiubeKiUak88b2Rvd4KQpz+Cq2FLN+3nH/t/RfF9cVEBEcwY8AM5gyZwxmpZ/gshYTWmhpbDVVNVeQk5HRpHxLofa25Eb55B9a9DJW7ITIZJt4Ok+dDZGLn97dzGbx3k1GVc8lzJ1y1vrme2z6+jQPWA7x60auMThrdxS/hHzYfOszPP9zJlkO1jMqI4YnvjWTqkC78GwhxAm7tZnP5ZpbvXc4nBz+hwdGAOcjMoNhBDI0bSnZ8Ntlx2QyNH0pGZEabalSX20WDs4GGZiM1RVlDGfss+9hv2c8+yz72WfZhsVtICk9i1dWrulRGCfS+YimC9X+BTa8Z9ezp42DK3TD6cq/uxE/o0/+Dr/8Ac1+CM25odxWHy8Hdn93NprJNvDD9Bc7JPOfUjnkaK65t4rcf7WZZXgnJ0aE8fFEOV4zPkhY1ots1OZv4ougLdlbvJP9wPgW1BZQ1lLUsjzRHkhWVhd1lb8k51ORsandf8aHxDIodxOC4wQyONaazMs7q0vM2GUrwVFXvhVW/gB0fABpGzDYCfP8zfdJyBjDq6Eu2wIcPQeooyDjjmMVu7ebxrx9nXek6nj776YAN8i63ZuEXe3n+s3wA7ps2lAXnDyEyVP6URc8IDw5n5sCZzBw4s2VeXXMdBbUF5B/OJ/9wPiUNJYQHh7fJJBoVYrwmhScxOHZwjw0cI3f0J9JYA1/8Fjb8BUyhMOl2mHwnxHVTioaGKlh0gfF+/hfHVAM9s+EZFu9cHNA55otrm3jwnTzWH6hh1ug0nrhkpHR8EsJD7ug7y2k32r+v/p3R4Wn8TXDBj33e2amNyCS4+nV4dSa8fxvcsASCTCzesZjFOxdz3fDruH307d1bhj7qX1tL+PHSbbjdmt9fNY7Lx2dKc1IhvCSBvjWtYcdS+O9PofYgDJ0BM56C1JE9VgR3Ri4Hv/tjdn/1G3YtvZrd0YmsKV3DhQMu5JFJjwRccKu3O/nJsu0s2VzMGf3j+MM1uQxIjOztYglxWpFAf0TlHlh2LxSth5RRcONSGDKtRw698tBK1pauZXfNbvbU7KHR2QgpSQTX7SHbncENI27ggQkPdFsGwL5q86HDPPBOHkWHG/nB9Gx+MG2oDDUnRBdIoD/io8eg6luY8wLkXg89FFQ/L/yc+1fdT0RwBMMThnPp0EsZnjCcEbFDGLLsfsz5O+CCF41nBAHC7da8uKqAP36WT1pMGO/dNZWJAzufglYIYZBAD+B2QeF6Ix3B+Jt67LCNjkZ+te5XDI0bynuXvIfZdFzni6vfhJfPg7fnweWLIKvd5yx+RWvNT5Zv5821h5ibm8HPLx3dqdzeQoi25HcwQPkOaK6Hfmf26GFf/uZlShpKePLMJ9sGeYDYTLjmTeOB8F+nw/t3GLl0OsPZ7JvC9pDffbyHN9ceYsH5Q/jjvDMkyAvhAxLoAQo9g171m9xjh8w/nM/rO17nsqGXMT51fMcrDpgKP9gM5z0Mu/4FL06E//4MbNaOt2moho1/g9cugV+kwubXff8FusHCL/byp8/3ct2U/jw6s2vdwIUQbUmgBzi0FqLTu699/HHc2s3Ta58mKiSKByc8ePINQqNh2hNw3yYYORe+ehZeGG8Ec5dnfEubBfL+Dm9eAc9kw4cPQF0ZJI+Aj37c+V8CPezv6w7x6//sZva4DH4+d3TAtS4SojtJHT0Yd/T9pviul+tJLCtYxuaKzTx11lOd6xkXm2XU1U+5Cz5+3Ajm6xdB/CAo+BRczcbF6uwfwKjLjURptQfhT2cZ617/zx77jp3xr60lPP7BNr6Tk8yzV4+TNAZC+JgEeksxWAph6j09crjDtsM8u+lZxqeMZ+7QuV3bSeYEuPU/sGu5UY1TstkYzGT0Fcay1sE8fiB896fwn4dh69uQe51Xh7A5XDy+dDtaa+44dzAjM2K6VtaTWLW7ggffzWPSwAT+dP0EzNJ8Ugifk0BfuNZ47TelRw733KbnqG+u54kznyBInUJQU8qoxhnpxcVi0h2wY4nRhHTINIg+8SAKNoeLBW9u4vM9lUSEmFiypZhzs5OYf95gzhma5LNqlXX7qlnw5iaGp0fzys0TCQ8JrH4CQvQUuX0qXG8M/pE2ptsPtal8E0sLlnLTqJvIjs/u9uO1CAqCOS8aqR3+/UOjB3AHbA4Xd71hBPlfXT6GNY9N55GZOewuq+PGV9Zz8fNf8cGWYhwud5eKorWmsdnJ+v013LF4I1nx4Sy+dbKMACVEN5KkZi+fbzzsvOXDbj2Mw+Xg6g+vptHRyNK5S3tnlJqv/wif/gSu/JuRWvk4R4L8F99W8uvLxzBv8tGH03ani2VbSlj05T4KKurJiA3jxqkDSYkOpdnlptnpmVxu7J73Tc1OahodHG5opqahmcONxqvdaVwkMuPC+efdU0mPlcRkQpwqSWrWEXs9lG2Dc7xo+XKKXt/5OgW1Bbw47cXeG4rszHuMVMsrHoZB5x+THdPmcDH/jU18mV/Jb64YwzWTjm2BFBps4upJ/bhyQhar9lSwaPU+fvPR7g4PFWIKIswcREJkCPGRIaTHhjEyI8b4HBFCQqSZ7+SkkBIT1l3fVgjhEdiBvngTaJeRV747D1NfzMKtC5nWbxrn9zu/W491QqZgY2CTl8+Djx6FK/4KGEH+ztc38lVBFb+5fCxXT+p4GMSgIMX0EalMH5FK0eFGnC5NqDmIEFMQIcGeyRQkzSOF6EMCO9AXrgMUZE3qtkO43C5+9r+foZTiscmPddtxvJY60uh89fkvYdTl2IZcdDTIXzGWqyd6MdatR1a8jMkqxOkgsB/GHloLKSMgPK7bDrHwm4WsKV3Dw5MeJj0qvduO0xn6nAdwJY/CufwB7n/t8y4FeSHE6SNw7+jdbijaYLQ97yari1azcOtC5gyZw5XZV3bbcTpSVW/n3Q2FFB1upLKumcp6O1V1dirr7QxzXcsHIT9het0f+e4VL3GVBHkh/FbgBvrKXWC3dlv7+aK6In705Y/Iic/hiTOf6NE665qGZl5evZfX/3eQJoeLpKhQkqJCSI4OZUhSJMnRoSRF5bC38BBX5y8C9RlwS4+VTwjRswI30B/ydJTq7/tAb3fZeejzh9Ba89wFzxEe3DPNB2sbm/nLl/t47esDNDpczB2XwX3TsxmSHNX+Bq5fwdv7jQHJY7Ng6Hd7pJxCiJ7lVaBXSs0E/giYgL9qrX993PJY4E2gv2efz2it/+ZZ9ipwCVChtR7tw7KfmsJ1EJli5InxsV+u+yW7anbxwrQX6BdzalUi9XYnq3ZXEB8RQkpMKMlRocRFmI/5hWBpdPDKV/t49esDNDQ7+d6YdO6fnk12avSJd24Khqv+ZoxR+94tcPvHkDrqlMorhOh7ThrolVIm4CVgBlAEbFBKLdda72y12j3ATq31bKVUMrBHKfWW1roZeA14EehbuXIPrTXu5n1cpbIkfwlL8pdw55g7uaDfBae8vyc/2M7SLcXHzDObFMlRoSTHhJEUGcL6AzXU2ZxcPCaN+6cPIyftJAG+tdBouO49I9/9W1fDHf+FmL7x0FgI4Rve3NFPBgq01vsAlFLvAHOB1oFeA9HKuM2MAmoAJ4DWerVSaqAvC33K6sqMrI6T5/t0tzurd/KLtb9gSvoU7sk99SRpXxdUsXRLMXecM4gZI1OpqLNTWWenos5ORZ2Nyjo7xbVNnDM0ifumZXc98VhsJlz3Lrw6C96+Bm5ZAaEdVPcIIU473gT6TKB1MvMi4PiK7ReB5UAJEA1co7XuVDIUpdR8YD5A//7dnBf+yEAjPuwoZbFbeOjzh4gPi+e35/32lAfytjlcPPHBdgYkRvD/LsohzNzNCb/SxxnVOG/PM0aymvdWj42bK4ToXt60o2+vbuP4BDkXAXlABpALvKiU6tTtpdZ6kdZ6otZ6YnJycmc27bxD6yA4DNLG+mR3bu3mR1/+iPLGcp694FkSwk59IOuFX+xlf1UDT186uvuD/BHDLoJZv4Vv/wMf/7hnjimE6HbeBPoioPUTxSyMO/fWbgWWaEMBsB8Y7psidoPCtZAxHoJDTnlXbu3ml+t+yZfFX/LopEcZm3zqF499lfX8adVe5ozL4Nzsbr7oHW/ynXDm92HdQli7sGePLYToFt4E+g1AtlJqkFIqBJiHUU3T2iFgOoBSKhXIAfb5sqA+09wIpVt90qzS5XbxszU/490973LrqFu5JueaU96n1ponPthOqDmIJy4Zccr765ILn4ac7xn569f8CRpreqccQgifOGmg11o7gXuBj4FdwHta6x1KqQVKqQWe1X4OnKWU2gZ8Bjyqta4CUEq9DawBcpRSRUqp27vji3itZDO4ndDv1OrnnW4nT379JEvyl3DX2Lt4cMKDPukUtSyvhP/treaRmcNJie6lzI5BJrjiL8YzjI9/ZIxB+8blxiDjEvSFOO0EXj761c/Ayp/DI/shomt16Q63gx9/+WM+OvAR9+bey13j7vJJ0SyNDqY/+zlZ8REsufssgnp77FStoTQPdiw10hvXHoSgYCPF8ahLYfglXT6HQgjfknz0rRWug6RhXQ/yLgcPr36Yzw59xkMTHuLW0bf6rGi//mg3hxsdLL5tdO8HeTD6GGScYUzf/Zkn6H9gBP7l98G/HjBG5up/JvSbbPxKis3s5UILIY4XWIHe7TaGDhwxu0ubH0ltsLpoNY9OepQbRt7gs6JtOljD2+sPccc5gxiVEeuz/frMMUH/p0bQ3/1vo+PZ5teNh7cAMVnG849+U4wmm1GpEJkMIZE+75wmhPBOYAX6qm/BVtul9vNNziYeWPUA/yv5H0+e+SRX51zts2I5XG4eX7qd9NgwHpwxzGf77Tatgz6Ay2GM1FW43mjRdHANbH//2G2Cw42AH5VsvEYmQcJgSB1jpF2IyZALgRDdJLACfaEnkVknH8Rqrbl/5f2sLV3LU2c9xWXZl/m0WK9+tZ/dZXW8fOMEIkNPw38SkxkyxxvTmZ7n87WFULELGipbTVXQUAHWYijJg/qyo/sIj4fU0caUNhpSRhoXhPB4+TUgxCk6DaPKKTi0DiKSIHFIpzbbVbOLNaVr+OGEH/o8yH+ZX8lz//2W745I5aJRaT7dd6+K62dMJ2KzQPkOYyrbBuXbYdNr4Gw6dr0gszE4THi8MYXFGTl6gkPBFHLca6jRPyImExKHQlK2caEQIoAFVqAvXGvUHXfy7nDloZUEqSDmDJ3j0+Is2VzEI//8hqEpUfzy8r6T2LPHhMXCgLOM6Qi3C2r2Q+VuaKyGpsNGdVvTYWjyvNaXQXU+OJvBZW/1aqdtp22M5wZJ2cZD+KRsSM6BzIkQIkMhisAQOIG+vgJq9sH4mzu96crClYxPGe+T1AZgVAX9+Yu9/PajPZw1JJGFN04gJszsk32f9oJMkDTUmDpLa6OPhKMJLEXGM5mqfOOiUPUt5L0FzfXGuqYQyJoMg8+HQedB5gSjCupUuV2SI0j0OYET6EvyjNd+kzu1WaG1kPzD+Twy6RGfFMPl1vzsXzt4fc1B5ozL4JmrxhESHNhD9/qMUkawNpkhbKQxEHprWhuZS8u3w/4vYP9qWPVLWPULCIkyflkMOg+GTDfGEvb2l5/LCXv+baSMKFxr7GP0lUbrrm4cj1gIbwVOoG+sNl6jUju12crClQBM6z/tlItgc7h44J08PtpRxvzzBvPYzOF9o718oFDKyLUfkw7ZM4x5jTVw4Esj6O/7AvI/AZ6AxGwYOdfoGJY6uv2g33QYNr8B6/8ClkMQ1x8m3gYFn8Hye+HfD8HQGTDmChg2S6qKRK8JnEBvtxqvYZ1ro77y0EqGJwwnM+rUOgLVNjZz5+sb2XjwMD+5ZCS3neP7ka1EF0QkGAF95Fzjs6UYvv0Idi6Dr56FL58xmoEeWSc916gOWrcQtr4NjkYYcA7M/CXkXGxU22htpNrY9j7sWGLc7ZsjYfjFxoWg9TMJIXpA4AR6myfQh3qfPbmqqYotFVu4e9zdp3To4tombn51PYeqG3nh2jO4ZGzGKe1PdKPYTJh0uzE1VMHuD42g//Xz8NVzxi/C+nKjdc+Yq2DKXZB+XMZSpYw6/8wJcOHP4eD/YPs/jf1s+wdMuBVmPAVhXRwoRohOCpxAb7cYOeg7kZr4i8Iv0OhTqrZZu6+ae/++GbvTzeu3T+bMwYld3pfoYZFJMOEWY2qsgT0rIP9To4PXhFuNzl8nE2SCQeca08xfG88D1rwEBf+F2X+EodO7+1sI4VWaYv9grzPaXnfCysKVZEZlMiy+871Vtda88tV+rv/rOmLCzSz9/lkS5E9nEQlwxg1w9WI4/xHvgvzxzOFGCujbPjHev3m5kTPIZjn5tlp7mo8K0XmBc0dvs3aq2qbB0cDakrVcM/yaTqcfbmx28qMl21iWV8KFI1P5/dXjiJbmk+KIfpPgri/h81/B/543Ht7Ofh6yv3t0HYfNGDehaL2RiK9wg9F/IH6Q8YsibczRXsRxA7rWc9jtMpqiOhqNX7tSleS3AifQ262d+kP+qvgrmt3NTOvXuWqbg9UN3PXGJvaU1/HwRTncff4QaVkj2jKHwYyfwYg5sOz78NYVMHaeUV1UuM4I8q5mY934gUaTzfiBRn+A8u1GQrkjncNCoo3gH5VsNPV0O8HtMAK5y2G8dznAaTMG3nE0GAHeaTtaHlMITL0Hzv1hp3/5ir4vcAJ9J+/oVx5aSXxoPLkpuV5vs2pPBfe/vQWlFK/dOpnzh/XwMIDi9JM1Ae5aDV/8Br76g9EHIOMMOPNuo0NXv8kQldJ2u+YGI5dQ+XYo2268Vu81ngkEefoSBAV70kJEGe/N4UbrH3O40dTTfGQKh6KNxsPmvL/D9P+DcddCUODU7Pq7wAn0ditEe9eG3uFy8GXRl0wfMJ3goJOfIqfLzZ8+38tz//2W4WkxvHzDBPonSptp4aXgUJj+EzjrPiMQe9NgICQSsiYaky9MvtOYPnrM+IWx4S/Gw+MuZHoVfU8ABfo6r+/oN5RtoM5Rx/T+HbeI0Fqzo8TK0i3FLN9aQmWdnUtzM/jV5WMJD5Eu8KILwuN79/hZE40Hxdv/CZ/+H7x6EYy+whh05mQJ6kSfFjiBvhNVNysLVxIeHM6Z6W3vZooON7Isr4QPthSTX1GP2aT4Tk4KV07IYsbIVJ+MGytErwkKgrFXw/DvGVVJ/3sedq8wegin5xp9BtLGSD3+aSYwAr3bBc11Xj2MdWs3qw6t4uyMswkLPjo497K8Yv6+7hDr9huDY08cEM8vLhvN98akExfhfdt8IU4LIZEw7XEYfyOs/IXR7n/r20eXJwzxBP2xxkhiWRM73etc9JzACPT2OuPVizv6HVU7qGiqOKaT1Ktf7eepD3cyKCmSh2YM49LcTKmDF4Ehrj9c/vLRhHBl30DpN1C2FYo3G+MHA6CM5p4DpkL/qUaah2g/Gl/hNBcggf5InpuTB/rPDn2GSZk4L+s8AJZvLeGpD3cyc1QaL10/HpM0lRSBqHVCuGEXHZ3fVGuMH3xoHRz6H2x5C9YvMpbFDzKCfuZ4Y6jI6DSITofIFDAFRujpKwLjbLfc0Z+8XnFl4Uompk0kNjSWrwuq+OF7eUwemMAf5uVKkBfieOFxMPgCYwLP+MHfGOMGH1oD+R/D1r8ft5EyhomMTjOmqBTPOMIpR8cTjvSMLRyRKBcFHwiMM+hlQrN9ln3st+xnXs48thdbuOuNTQxOiuIvN08kzCwtaYQ4KZP5aEK3s+49WuVTV2okg6srPfq5rhzqSoxhJBsqjY5ebSjjQhCd7vlV4PlVEZ1hvIbFgXZ7Oom5PK9OzzyX8eA4tpOZZ10O45mEtRiSRxjjGpysRZTTbnRyK1xnTFX5EJ7gKbvnYhaVZjTxjko1nnH0YNrqwAj0XqYoXnVoFQA50VO55W8biAkLZvFtk4kNl/QFQnRJ6yqfE9HaGDKyvvLYAeXrKzwXhVI4fND4ldB0uDMFgAFnw5grjTTTEScYJa5su9Fh7Jt3obHq2GUxWUbATx0FKaOM4SgthUZQP7QOSrYYw1mC0YM5ZZRnTOTtsHfl0Rh0RFgsnPl9I/tpDzSrDYxA7+Ud/cpDK8mJH8kP3z6A0+3mnflTSYsNO+E2QggfUOro4O/JJ0ki6Gg6+qvAZjF6/aog4zXI5PlsMu7q938B37wHHz4AKx42BpwZc+XRgWAaqo3U0XlvGVVOQWbImQW51xkPlyt3G8G6fKcxiP3elcf+8jCFGM1OJ99pdC7Lmtx+x8zmRuMXTX05WEtg2z+NXEdrXjKC/ZnfP/FF6BR5FeiVUjOBPwIm4K9a618ftzwWeBPo79nnM1rrv3mzbY+we7IDnqCOvqKxgm+qviHePocKq4237jiToSnSVliIPsccDgmDjOlk+k+B8x42qlW2/QO2v2+kmzZHGqkmCtcZuYDSc2HW74yLQOuAG9fv6GhkYAxEX51vXABiMo3tzF7cDIZEHFvm0ZcbVVarf2dMa/8Mk+6Aqfd2LTPqSZw00CulTMBLwAygCNiglFqutd7ZarV7gJ1a69lKqWRgj1LqLcDlxbbd78jD2BO0uvn0gDFkYEnJUF6+bjwTBvRyL0UhhG8oBRm5xjTjKWMgmG3/gKINxt107nVGlYw3gkOMdb1d/0TSxsDVrxs5i1Y/A1//ETa/Dg/tNC5mPuTNHf1koEBrvQ9AKfUOMBdoHaw1EK2MbqFRQA3gBKZ4sW33s1mNn3Lmtg8/tNYs37uc3298Fpctlacvnsb0EZ0bV1YIcZpoPRBMX5EyAq58Bc5/1Giq6uMgD94F+kygsNXnIowA3tqLwHKgBIgGrtFau5VS3mwLgFJqPjAfoH///l4V3mtHUhQfl56g1lbLU2uf4tODnxLhHkaG+2bmTR7g22MLIYQ3koed/PlEF3mTh7S9xuP6uM8XAXlABpALvKiUivFyW2Om1ou01hO11hOTk31cR9VOnpv/lfyPK5ZfwarCVTww/gEiqu9hQEyWb48rhBB9gDeBvghonbouC+POvbVbgSXaUADsB4Z7uW33sx8N9HaXnd+s/w13fXoXkSGRvHXxW9w2+jbKrc2kxkgLGyGE//Gm6mYDkK2UGgQUA/OA645b5xAwHfhSKZUK5AD7gFovtu1+diOh2Z6aPTz25WMU1BZw7fBreXDCg4QHh1Nnc9DY7CItNrTHiyaEEN3tpIFea+1USt0LfIzRRPJVrfUOpdQCz/KFwM+B15RS2zCqax7VWlcBtLdt93yVE7BZ2RuTxLX/vpaYkBj+NP1PnJt19GFMudUYUk3u6IUQ/sirdvRa6xXAiuPmLWz1vgS40Ntte5zdws7gZBw2By/PeJmchJxjFpdZjB5taRLohRB+KDAGhbRZqTUZuWrSItumTi3z3NFLL1ghhD/y/0CvNdjrsAQFoVBEmaParCJVN0IIf+b/gd7RCNpFrdLEhMZgCmqbhbLMYiMuwiwZKoUQfsn/A70noZkVN7Eh7WevLLPapH5eCOG3/D/Qe9KD1moncaFx7a5SbrVJtY0Qwm/5f6D33NFb3M3EdJCmuMwid/RCCP/l/4Hec0dvcTW1e0fvdLmpqreTKi1uhBB+KnACvbOR2NC2dfSV9XbcGlJjpFesEMI/+X+gt1lxAPXOpnYfxpZZPG3opepGCOGn/D/Q261Yg4yv2d4dvbShF0L4O/8P9DYrFlPHgb7ljl7q6IUQfsr/A729Dkuo0Ru2vYexZVY7ZpMiISKkhwsmhBA9IwACvRVLaCTQcdVNSnQYQUHtjZEihBCnP/8P9DYLtSHGGIwdVd1ItY0Qwp/5f6C3W7GYjaaTHd3RS4sbIYQ/8/9Ab7NiCTZjUiaizdHHLNJaUybpD4QQfs7/A729DovJRExIDEodWw9fZ3fKEIJCCL8XAIHeiiVItV9tY5E29EII/+f/gd5mpRbd/oNYq/SKFUL4P/8O9E47uOxYcUlnKSFEwPLvQO9JUVzrdrTbWUrSHwghAoF/B/ojmSvddmJC2uaiL7PKEIJCCP/n94HeATS6m9tPf2CxS/28EMLv+XegP0lCMxlCUAgRCPw70Nut1AYZ1TLtJzSTXrFCCP/n34HeZsXiyUV//HixDhlCUAgRILwK9EqpmUqpPUqpAqXUY+0sf1gpleeZtiulXEqpBM+y+z3zdiilHvBx+U/MXtdSdXP8HX1lnR2tpQ29EML/nTTQK6VMwEvALGAkcK1SamTrdbTWv9Na52qtc4EfAV9orWuUUqOBO4HJwDjgEqVUto+/Q8fsR+/oj6+jb+ksJekPhBB+zps7+slAgdZ6n9a6GXgHmHuC9a8F3va8HwGs1Vo3aq2dwBfAZadS4E6xWY5mrjxuvFhJfyCECBTeBPpMoLDV5yLPvDaUUhHATOB9z6ztwHlKqUTPsouBfh1sO18ptVEptbGystLb8p+Y3UptSBjBKphIc+QxiyT9gRAiUHgT6Nsbekl3sO5s4GutdQ2A1noX8BvgU+AjYCvgbG9DrfUirfVErfXE5ORkL4rlBZsVS3AIMaFtM1eWWW2EmIJIiJQhBIUQ/s2bQF/EsXfhWUBJB+vO42i1DQBa61e01uO11ucBNUB+VwraJfY6LKbg9tMfWGykxIS2uQAIIYS/8SbQbwCylVKDlFIhGMF8+fErKaVigfOBZcfNT/G89gcu57gLQbeyGx2mOspcKdU2QohAEHyyFbTWTqXUvcDHgAl4VWu9Qym1wLN8oWfVy4BPtNYNx+3ifaVUIuAA7tFaH/Zd8U/CZqU2BDLa7RVrZ2RG2/w3Qgjhb04a6AG01iuAFcfNW3jc59eA19rZ9tyuF+8U2a1YiGTEcS1utNaUWWxMG57SSwUTQoie4989Y+11WLWrTR291eakyeGSqhshREDw30DvdmF31NPUzqAjLXnoJf2BECIA+G+gt1uxeBKatekVa5E29EKIwOG/gd5mpfZk6Q8k0AshAoD/Bnp7x7noj6Q/SImRPDdCCP/nx4G+riWh2fEPY8usNuJlCEEhRIDw30DfKhd9m4RmMrKUECKA+G+gt1up7aDqpsxqI01a3AghAoT/BnqbBUuQCXOQmfDg8GMWyaDgQohA4r+B3m7FagoiLjT2mMRlDpeb6ga7VN0IIQKGHwf6OmpNpjbVNhVHhhCUqhshRIDw30DvyUUfe3yLG+ksJYQIMP4b6O1W446+nRY3IEMICiECh/8GepsVa5AiLizumNktd/RSdSOECBB+G+i13UItut07+hBTEPER5l4qmRBC9Cy/DfQ2ex3NCmJCjx1cpMwqQwgKIQKL3wZ6S3Md0Db9QbkMISiECDD+G+gdRqBvm4veLnnohRABxT8DvdbUuoyHrq3v6I8MISh39EKIQOKfgb65HounCj4m5GgdvQwhKIQIRP4Z6O11LbnoW9/RyxCCQohA5J+B3tb+MILSK1YIEYj8M9DbjVz0YUFmwoKPBnUZQlAIEYj8M9DbjFz0MeaoY2bLEIJCiEDkn4HebsESFESsOfqY2TKEoBAiEHkV6JVSM5VSe5RSBUqpx9pZ/rBSKs8zbVdKuZRSCZ5lDyqldnjmv62U6v56E8/D2OPz3MgQgkKIQHTSQK+UMgEvAbOAkcC1SqmRrdfRWv9Oa52rtc4FfgR8obWuUUplAj8AJmqtRwMmYJ6Pv0NbnvFiY8Pij5ktQwgKIQJRsBfrTAYKtNb7AJRS7wBzgZ0drH8t8PZxxwhXSjmACKCk68X1kt1KbZCJ2LCkY2aXWeyMzojtYCMhRHdzOBwUFRVhs9l6uyinrbCwMLKysjCbvU/M6E2gzwQKW30uAqa0t6JSKgKYCdwLoLUuVko9AxwCmoBPtNafeF26LtJNFiymIGLDjgZ1GUJQiN5XVFREdHQ0AwcOlMSCXaC1prq6mqKiIgYNGuT1dt7U0bf3r6E7WHc28LXWugZAKRWPcfc/CMgAIpVSN7R7EKXmK6U2KqU2VlZWelGsjjXZa3EqdUxnKRlCUIjeZ7PZSExMlCDfRUopEhMTO/2LyJtAXwT0a/U5i46rX+ZxbLXNd4H9WutKrbUDWAKc1d6GWutFWuuJWuuJycnJXhSrY7X2WkA6SwnRF0mQPzVdOX/eBPoNQLZSapBSKgQjmC9v5+CxwPnAslazDwFnKqUilFG66cCuTpeykyx2C3BsoC863AhARlx4dx9eCCH6lJMGeq21E6PO/WOMIP2e1nqHUmqBUmpBq1Uvw6iDb2i17Trgn8BmYJvneIt8WP521R5JUdxqdKntxRZCg4MYkhzZ3YcXQpxmfvnLX/pkP3a7nWuuuYahQ4cyZcoUDhw40O56mzZtYsyYMQwdOpQf/OAHaG3Uhr/22mskJyeTm5tLbm4uf/3rX31SLq/a0WutV2ith2mth2itf+GZt1BrvbDVOq9prds0ndRa/5/WerjWerTW+kattd0nJT8Bi8O41rSuo99WbGF4egzBJv/sIyaE6DpfBfpXXnmF+Ph4CgoKePDBB3n00UfbXe/uu+9m0aJF5Ofnk5+fz0cffdSy7JprriEvL4+8vDzuuOMOn5TLm1Y3px2LywaEtlTduN2aHcVW5p6R0bsFE0K0+Nm/drCzxOrTfY7MiOH/Zo864Tpvvvkmzz//PM3NzUyZMoWYmBiamprIzc1l1KhRvPXWW1x66aUUFhZis9m4//77mT9/vlfHX7ZsGT/96U8BuPLKK7n33nvRWh9Tr15aWorVamXq1KkA3HTTTXzwwQfMmjWra1/aC/4Z6N3NtA70B2saqbM7GZMpbeiFCGS7du3i3Xff5euvv8ZsNvP973+fMWPGEB4eTl5eXst6r776KgkJCTQ1NTFp0iSuuOIKEhMTueaaa9izZ0+b/T700EPcdNNNFBcX06+f0XYlODiY2NhYqqurSUo62qenuLiYrKysls9ZWVkUFxe3fH7//fdZvXo1w4YN47nnnmvZ36nwv0DvsFGrNOEqmBBTCGBU2wCMlkAvRJ9xsjvv7vDZZ5+xadMmJk2aBEBTUxMpKSlt1nv++edZunQpAIWFheTn55OYmMi77757wv0fqWtv7fhWMidaZ/bs2Vx77bWEhoaycOFCbr75ZlauXOndlzsB/wv0nhTFsaajzSi3F1sIMQUxLDX6BBsKIfyd1pqbb76ZX/3qV8fMf+aZZ1ref/755/z3v/9lzZo1REREcMEFF7S0Wz/ZHX1WVhaFhYVkZWXhdDqxWCwkJCQcs25WVhZFRUUtn4uKisjIMKqVExMTW+bfeeedHdbxd5b/BXqbFWtQEHHmo61rthdbGJ4ejVkexAoR0KZPn87cuXN58MEHSUlJoaamhrq6OsxmMw6HA7PZjMViIT4+noiICHbv3s3atWtbtj/ZHf2cOXNYvHgxU6dO5Z///CfTpk1rc0efnp5OdHQ0a9euZcqUKbz++uvcd999gFF/n56eDsDy5csZMWKET763/wV6u5Vak4lYTy56rTXbiy1cMk4exAoR6EaOHMnTTz/NhRdeiNvtxmw289JLLzF//nzGjh3L+PHjefXVV1m4cCFjx44lJyeHM8880+v933777dx4440MHTqUhIQE3nnnnZZlubm5Lc8B/vznP3PLLbfQ1NTErFmzWh7EPv/88yxfvpzg4GASEhJ47bXXfPK9VXv1Rb1t4sSJeuPGjV3beN/nzPlsAdmZU/j9xYs5WN3A+b/7nF9dPoZrJ/f3bUGFEJ2ya9cun92lBrL2zqNSapPWemJ76/tfXYbNaiQ087ShP/IgVlrcCCECld8Fem0zRpeKCzceamwrtmA2KXkQK4QIWH4X6Osbq3ApRWy4kRhtR7GVnLRoQoL97qsKIYRX/C76WZpqAIiNTEFrzbZii1TbCCECmv8Fersn0IclUHS4CUuTQzpKCSECmv8FetthwEhRLA9ihRDCHwN9s5GiOC40jm3FFoKDFDlp8iBWCNGxnk5TPHPmTMaNG8eoUaNYsGABLpfLJ8fviN8F+lpHPQAxoTFsL7YwLDWa0GBTL5dKCNGX9XSa4vfee4+tW7eyfft2Kisr+cc//uGT43fE73rGWpxNEAwxIUagv3BkWm8XSQjRnv88BmXbfLvPtDEw69cnXKW30xQDxMTEAOB0Omlubu724RX9L9C7bUQSRIXVyeFGB6OzpH5eCGHoC2mKj7joootYv349s2bN4sorr+y27wx+GeibiQuKZLs8iBWibzvJnXd36Atpio/4+OOPsdlsXH/99axcuZIZM2Z09ut4ze8Cfa12ERMUyrZiC6YgxXB5ECuE8OgLaYpbCwsLY86cOSxbtkwCvddcDixKExccwbZiK9kpUYSZ5UGsEMLQF9IU19fXU1dXR3p6Ok6nkxUrVnDuued2y/c9wr8Cvb0OiymIDHMEm4otTBve9ieZECJw9YU0xQ0NDcyZMwe73Y7L5WLatGksWLCgO75uC/9KU3z4AOcuvZgL4s/gjXXX8dTcUdw0daDPyyeE6BpJU+wbAZ2m2N1UizUoCLfLGF1KUh8IIYSfBfq6hnLcSmFrDsMUpBiZHtPbRRJCiF7nV4He2lABQE1DOEOT5UGsEEKAnwX62sZKAIprQ6XaRgghPLwK9EqpmUqpPUqpAqXUY+0sf1gpleeZtiulXEqpBKVUTqv5eUopq1LqAZ9/Cw+LrRqAqsYoxmRKtY0QQoAXzSuVUibgJWAGUARsUEot11rvPLKO1vp3wO88688GHtRa1wA1QG6r/RQDS338HVrUelIUN7jiGSOpD4QQAvDujn4yUKC13qe1bgbeAeaeYP1rgbfbmT8d2Ku1Ptj5YnrHYjfSHjjcsYyQB7FCCC/1dJrixx9/nH79+hEVFeWT456MN4E+Eyhs9bnIM68NpVQEMBN4v53F82j/AuAzR3LRD05IIiLEv/qCCSG6T0+nKZ49ezbr16/3yTG94U00bC8jT0e9rGYDX3uqbY7uQKkQYA7wow4PotR8YD5A//79vShWWxZnI1EuzdjMjnNLCCH6ht+s/w27a3b7dJ/DE4bz6OT2g+sRfSFNcWd62/qCN4G+COjX6nMWUNLBuh3dtc8CNmutyzs6iNZ6EbAIjJ6xXpSrjcPORiLcSlrcCCHa1ZfSFPckbwL9BiBbKTUI42HqPOC641dSSsUC5wM3tLOPjurtfarGaSPcFSQPYoU4DZzszrs79KU0xT3ppIFea+1USt0LfAyYgFe11juUUgs8yxd6Vr0M+ERr3dB6e0+9/QzgLp+WvB0W7STMZZYesUKIdvW1NMU9xasnllrrFcCK4+YtPO7za8Br7WzbCCR2uYSdYMVFKpFEhsqDWCFEW30hTXFv8KuesXVBEBMc0dvFEEL0Ua3TFI8dO5YZM2ZQWlrakqb4+uuvZ+bMmTidTsaOHcuTTz7Z6TTF1dXVDB06lGeffZZf//roKFq5ubkt7x955BGysrJobGwkKyur5QFud/GbNMV2h4v7/nYeo+Mn8YOr/tA9BRNCnBJJU+wbnU1T7Dd1HKFmE4vmf93bxRBCiD7Hr6puhBBCtCWBXgjRo/pidfHppCvnTwK9EKLHhIWFUV1dLcG+i7TWVFdXExYW1qnt/KaOXgjR92VlZVFUVERlZWVvF+W0FRYWRlZWVqe2kUAvhOgxZrOZQYMG9XYxAo5U3QghhJ+TQC+EEH5OAr0QQvi5PtkzVilVCXR1JKokoMqHxfEVKVfn9dWySbk6R8rVeV0p2wCtdXJ7C/pkoD8VSqmNHXUD7k1Srs7rq2WTcnWOlKvzfF02qboRQgg/J4FeCCH8nD8G+kW9XYAOSLk6r6+WTcrVOVKuzvNp2fyujl4IIcSx/PGOXgghRCsS6IUQws/5TaBXSs1USu1RShUopR7rxXL0U0qtUkrtUkrtUErd75n/U6VUsVIqzzNd3EvlO6CU2uYpw0bPvASl1KdKqXzPa3wPlymn1XnJU0pZlVIP9MY5U0q9qpSqUEptbzWvw/OjlPqR529uj1Lqol4o2++UUruVUt8opZYqpeI88wcqpZpanbuFHe64e8rV4b9dT52zDsr1bqsyHVBK5Xnm9+T56ihGdN/fmdb6tJ8AE7AXGAyEAFuBkb1UlnRgvOd9NPAtMBL4KfD/+sC5OgAkHTfvt8BjnvePAb/p5X/LMmBAb5wz4DxgPLD9ZOfH8++6FQgFBnn+Bk09XLYLgWDP+9+0KtvA1uv1wjlr99+uJ89Ze+U6bvnvgZ/0wvnqKEZ029+Zv9zRTwYKtNb7tNbNwDvA3N4oiNa6VGu92fO+DtgFZPZGWTphLrDY834xcGnvFYXpwF6tdVd7Rp8SrfVqoOa42R2dn7nAO1pru9Z6P1CA8bfYY2XTWn+itXZ6Pq4FOpe/tpvKdQI9ds5OVC6llAKuBt7ujmOfyAliRLf9nflLoM8EClt9LqIPBFel1EDgDGCdZ9a9np/Yr/Z09UgrGvhEKbVJKTXfMy9Va10Kxh8hkNJLZQOYx7H/+frCOevo/PS1v7vbgP+0+jxIKbVFKfWFUurcXihPe/92feWcnQuUa63zW83r8fN1XIzotr8zfwn0qp15vdpuVCkVBbwPPKC1tgJ/BoYAuUApxs/G3nC21no8MAu4Ryl1Xi+Vow2lVAgwB/iHZ1ZfOWcd6TN/d0qpxwEn8JZnVinQX2t9BvAQ8HelVEwPFqmjf7u+cs6u5dgbih4/X+3EiA5XbWdep86ZvwT6IqBfq89ZQEkvlQWllBnjH/AtrfUSAK11udbapbV2A3+hG3/in4jWusTzWgEs9ZSjXCmV7il7OlDRG2XDuPhs1lqXe8rYJ84ZHZ+fPvF3p5S6GbgEuF57KnU9P/OrPe83YdTrDuupMp3g367Xz5lSKhi4HHj3yLyePl/txQi68e/MXwL9BiBbKTXIc1c4D1jeGwXx1P29AuzSWj/ban56q9UuA7Yfv20PlC1SKRV95D3Gg7ztGOfqZs9qNwPLerpsHsfcZfWFc+bR0flZDsxTSoUqpQYB2cD6niyYUmom8CgwR2vd2Gp+slLK5Hk/2FO2fT1Yro7+7Xr9nAHfBXZrrYuOzOjJ89VRjKA7/8564ilzDz3Jvhjj6fVe4PFeLMc5GD+rvgHyPNPFwBvANs/85UB6L5RtMMbT+63AjiPnCUgEPgPyPa8JvVC2CKAaiG01r8fPGcaFphRwYNxJ3X6i8wM87vmb2wPM6oWyFWDU3x75W1voWfcKz7/xVmAzMLuHy9Xhv11PnbP2yuWZ/xqw4Lh1e/J8dRQjuu3vTFIgCCGEn/OXqhshhBAdkEAvhBB+TgK9EEL4OQn0Qgjh5yTQCyGEn5NAL4QQfk4CvRBC+Ln/D54pph0BaoqQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "etas = [\"eta=0.05\", \"eta=0.3\", \"eta=0.1\"]\n",
    "for eta in etas:\n",
    "    df_score = scores[eta] \n",
    "    plt.plot(df_score.num_iter, df_score.val_auc, label=eta)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e202eb79",
   "metadata": {},
   "source": [
    "* ```eta=0.1``` seems to be the best\n",
    "* Take this now as a baseline model to optimize on ```max_depth```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "id": "293c9d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "a8943fe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "xgb_params = {\n",
    "    \"eta\": 0.1,\n",
    "    \"max_depth\": 10,\n",
    "    \"min_child_weight\": 1,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  verbose_eval=5,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "030e8867",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = f\"max_depth={xgb_params['max_depth']}\"\n",
    "scores[key] = parse_xgb_output(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "8fa5c3a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['max_depth=6', 'max_depth=3', 'max_depth=4', 'max_depth=10'])"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "4514645a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f35c28c8dc0>"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABd3klEQVR4nO3dd3yURf7A8c/sZtN7DwmQ0BJ670hVAQWxK+IJqKee5aeed4renfX0UNE7PVAPFbGgoAKKIGBBBekdEiCQhJLee9s2vz+eJSRh0yA983699vXsPm3nebKZ7/PMzDMjpJQoiqIoHY+upROgKIqitAwVABRFUTooFQAURVE6KBUAFEVROigVABRFUTooFQAURVE6qHoFACHENCFErBAiTgixwM5yLyHEd0KIw0KIGCHE/GrL9UKIg0KI9ZXm+QohfhRCnLJNfS7/cBRFUZT6qjMACCH0wBJgOtAHmC2E6FNttYeAY1LKgcBE4A0hhGOl5Y8Cx6ttswD4WUrZE/jZ9llRFEVpJvW5AxgBxEkpE6SURmAlMKvaOhLwEEIIwB3IAcwAQogw4Frgg2rbzAI+tr3/GLj+Ug5AURRFuTQO9VgnFEis9DkJGFltncXAOiAF8ABuk1Jabcv+Azxpm19ZkJQyFUBKmSqECLT35UKI+4D7ANzc3IZGRUXVI8mKoijKefv378+SUgZUn1+fACDszKvef8RU4BAwGegO/CiE2AaMBzKklPuFEBMbkuCKL5JyKbAUYNiwYXLfvn2XshtFUZQOSwhx1t78+hQBJQGdK30OQ7vSr2w+sEZq4oDTQBQwFrhOCHEGrehoshDiM9s26UKIEFviQoCMeh6LoiiK0gjqEwD2Aj2FEBG2it3b0Yp7KjsHTAEQQgQBkUCClPJpKWWYlDLctt0WKeWdtm3WAXNt7+cC317WkSiKoigNUmcRkJTSLIR4GNgM6IFlUsoYIcQDtuXvAS8By4UQR9GKjJ6SUmbVseuFwJdCiHvQAsgtl3EciqIoSgOJttQdtL06AJPJRFJSEmVlZS2UKuVyODs7ExYWhsFgaOmkKEq7JYTYL6UcVn1+fSqBW7WkpCQ8PDwIDw9Ha4WqtBVSSrKzs0lKSiIiIqKlk6MoHU6b7wqirKwMPz8/lfm3QUII/Pz81N2borSQNh8AAJX5t2Hqb6coLaddBABFURSl4VQAUBRF6aBUAGjDwsPDycqqq7WtfcuXLycl5cLzfJezr7y8PG6++WaioqLo3bs3O3fuvKT9KIrSvFQA6KCqB4DL8eijjzJt2jROnDjB4cOH6d27d6PsV1GUptXmm4FW9sJ3MRxLKWjUffbp5MlzM/vWus6ZM2eYNm0a48aNY9euXQwcOJD58+fz3HPPkZGRwYoVKwB47LHHKC0txcXFhY8++ojIyEjefPNNoqOjWbZsGUePHmX27Nns2bMHV1fXi74nOzub2bNnk5mZyYgRI6j8DMdnn33G22+/jdFoZOTIkbzzzjvo9Xrc3d25//77+eWXX/Dx8WHlypX89ttv7Nu3jzlz5uDi4lJxxf7f//6X7777DpPJxFdffUV9Ot4rKChg69atLF++HABHR0ccHR1r30hRlFZB3QE0kri4OB599FGOHDnCiRMn+Pzzz/n9999ZtGgRr7zyClFRUWzdupWDBw/y4osv8swzzwBaUIiLi2Pt2rXMnz+f//3vf3Yzf4AXXniBcePGcfDgQa677jrOnTsHwPHjx1m1ahXbt2/n0KFD6PX6iqBTXFzMkCFDOHDgABMmTOCFF17g5ptvZtiwYaxYsYJDhw7h4uICgL+/PwcOHOBPf/oTixYtAuCXX35h0KBBF73GjBkDQEJCAgEBAcyfP5/Bgwdz7733Ulxc3KTnWlGUxtGu7gDqulJvShEREfTv3x+Avn37MmXKFIQQ9O/fnzNnzpCfn8/cuXM5deoUQghMJhMAOp2O5cuXM2DAAO6//37Gjh1b43ds3bqVNWvWAHDttdfi46MNovbzzz+zf/9+hg8fDkBpaSmBgYEV+7/tttsAuPPOO7nxxhtr3P/5ZUOHDq34nkmTJnHo0KEatzGbzRw4cID//ve/jBw5kkcffZSFCxfy0ksv1XnOFEVpWe0qALQkJyenivc6na7is06nw2w2849//INJkyaxdu1azpw5w8SJEyvWP3XqFO7u7vUqk7fXbl5Kydy5c/nXv/51SdtXPwa9Xo/ZbAa0O4DHH3/8onVdXV3ZsWMHYWFhhIWFMXKkNkTEzTffzMKFC+tMh6IoLU8VATWT/Px8QkNDASrKy8/Pf/TRR9m6dSvZ2dl8/fXXNe5j/PjxFUU7GzduJDc3F4ApU6bw9ddfk5Gh9aidk5PD2bNa999Wq7Vin59//jnjxo0DwMPDg8LCwjrTff4OoPprx44dAAQHB9O5c2diY2MB7W6kT5/qI4YqitIaqTuAZvLkk08yd+5c3nzzTSZPnlwx//HHH+fBBx+kV69efPjhh0yaNInx48dXFOFU9txzzzF79myGDBnChAkT6NKlCwB9+vThn//8J1dffTVWqxWDwcCSJUvo2rUrbm5uxMTEMHToULy8vFi1ahUA8+bN44EHHqhSCXyp/vvf/zJnzhyMRiPdunXjo48+uqz9KUqrVFYAR1ZB/BYI6gcRV0DYCDA4t3TKLlmb7w30+PHjqtlhLdzd3SkqKmrpZNRK/Q2VZlOQAgc+gZzTED4Wuk0C7861b5NyEPYtg6OrwVQMXp2hIBmkFfRO0HkEREzQAkKnIeDQ+lrBtdveQBVFUWolJZz+DfZ+CCc2aBm3qy8cWakt9+sJ3SdpwSB8HDh7grEYoldrGX/KQXBwgf43w7D5WiZfXgBnd8LprXBmK/zyT/gFMLhp+xp2t7Y/XesuZVcBoBX66KOPeOutt6rMGzt2LEuWLGnwvlr71b+iNJnSXDj0hZaJZ58CF18Y/ZCWiftEQMZxSPgF4n+BA5/CnqUg9NBpMGSd1DL5gN4w/XUYcCu4eF/Yt7MXRE7TXgAlOXDmdy3QHPsWTqwH3+4w/F4YdEfVbVsRVQSktDj1N1QahakMMo9DWjSc2wnRa8BcqpXTD78H+lxfc3m9uRwSd2vB4Mzv4BOuXcV3GQUN7bHWXK4FgT3vQ9IeMLhqAWT4HyG43+Ue5SVRRUCKorQfJTmQckDL7NOjtWnWSZAWbbmjhy3TvQdCBta9PwcniBivvS6Xg5P23QNuhZRDsPd9OLwS9i+HLmPAtxuYSrSXsdg2LdHqF3QGGHGfdpfi4FTXN11+UuuzkhBiGvAW2pjAH0gpF1Zb7gV8BnSx7XORlPIjIYQzsBVwss3/Wkr5nG2b54E/Apm23Twjpfz+so9IUZT2qaxAK8OP/lq7Uj+f2XuGaVfWUddCcH/t5RPROsrfOw2CWUvgqpfg0Ao4+BkknNXuCgwu4OgGzt7g2UmrP8g7B5uegl1LYNLftXoHnb7JkldnABBC6IElwFVAErBXCLFOSnms0moPAceklDOFEAFArBBiBVAOTJZSFgkhDMDvQoiNUspdtu3+LaVc1KhHpChK62Yq04pZzm7XKmN9u2nl5T7hFxfRmMrg1A9apn9yM5jLwKsLjHkEekzRmmO6+rbIYTSIq6+W5jGP1L6elFoz05+eh7X3wfa3YMqz0Gtqw4ui6qE+dwAjgDgpZQKAEGIlMAuoHAAk4CG0x0zdgRzALLUKhvO1kAbbq+1UOiiK0jgK07WM/OQm7erdVAxCp7XIqSDAMxT8umlBwWzUKlPLC8AtAIbcBf1u1ppdtteR5ITQAlu3SXBsLWz5J3xxG3QZDde8rt3dNKL6BIBQILHS5yRgZLV1FgPrgBTAA7hNSu0va7uD2A/0AJZIKXdX2u5hIcRdwD7gCSll7iUdRQcVHh7Ovn378Pf3b/C2y5cv5+qrr6ZTp06Xta+ysjLGjx9PeXk5ZrOZm2++mRdeeKHB6VFaISkheb9WXOETDo72Oym0u11xJmTHw5ltWqafvF9b5hkGA2+HyOlak0tTKeSehuwEyEmAnHhtemwdWC3Qeyb0u0lrZ6/vQFWWOp123L2v055b2Po60DJ3APa+tfpV/FTgEDAZ6A78KITYJqUskFJagEFCCG9grRCin5QyGngXeMm2r5eAN4C7L/pyIe4D7gMqnnxVLt/y5cvp169fRQC4VE5OTmzZsgV3d3dMJhPjxo1j+vTpjBo1qpFSqjQ7qxVOfAe/vQ7pRy/M9+gEft3BN+JCsY2LD+SesWXe5zPw02A8f+MvIHQoTP479JqmFdlUvno3uGjFI6FDL06HlO33Sr++9AatInvIXdr7RlafAJAEVH5ULgztSr+y+cBCW5FPnBDiNBAF7Dm/gpQyTwjxKzANiJZSpp9fJoR4H1hv78ullEuBpaA1A601pRsXQNrRWldpsOD+ML32zs068ngAQgjc3d0BMJlMmEwmNdB7W2W1QMxa2LpIa07p2x2u+69WUVn5Cj12o3aFX5nOAby7aoGh61hbgOimtcBxv7hbk3pRv6MLmiDzh/oFgL1ATyFEBJAM3A7cUW2dc8AUYJsQIgiIBBJsFcImW+bvAlwJvAoghAiRUqbatr8BiL7so2lBcXFxfPXVVyxdupThw4dXjAewbt06XnnlFT755BO2bt2Kg4MDP/30E8888wyrV6/mscceY+LEiaxdu5aXX365XuMBPPvss2zYsIGlS5cCVccDMBgMPPjgg6xYsYK77rqrYjyAN954gxdffJEXXniBxYsXs3jxYhYtWsSwYReaBp8fD+Cdd95h0aJFfPDBB3X2BgpgsVgYOnQocXFxPPTQQxU9gypthMWsVbJuXaQ9MOUfCTd+AP1urLkFSlmBFhBKc8Gnq1Yx25GKaNqJOv9iUkqzEOJhYDNaM9BlUsoYIcQDtuXvoRXhLBdCHEUrMnpKSpklhBgAfGyrB9ABX0opz1/pvyaEGIRWBHQGuP+yj6aOK/Wm1FHHAwCt++hDhw6Rl5fHDTfcQHR0NP36tcwDL0oDWK1w9Ev4daFWDh/YF25ZDr1n1d2E0tlTa+KotGn1Ctm29vnfV5v3XqX3KcDVdrY7AgyuYZ9/aFBKW7mOOh5AZd7e3kycOJFNmzapANDaJe6BjU9pD1MF94fbVkDkNa2j7bzSbNRfu5m01/EAMjMzycvLA7Q7j59++qledQdKC8lPhtV/hA+v0nrGvP49uG8r9J6hMv8OSBXaNZP2Oh5Aamoqc+fOxWKxYLVaufXWW5kxY8Yl709pIqZS2PFf+P3fWmXvFU/AuD+Dk3tLp0xpQaozuHZOjQfQwVmtcOwb+PFZyE/U2pVf/ZLWrl/pMFRncIrSUUip9WEfvRpivoGCJAjqD9e/qw1aoig2KgC0Qmo8AKXBpNR6xYxeAzFrtIezdAatW4GrXoC+NzRpp2JK26QCQCs0f/585s+f39LJUJpDWb7WTfDJzVp/Nz5dwbsLeIfb2td3rtpBWnkRFKVDUQYUZ2jT/ETt4aysk9qAJt0mwBV/0Sp2XXxa6siUNkAFAEVpCflJsOtd2P8xGAsheAAUpkHs92AxVl3XPVgLAkUZWt/x1Qk9dB0DIx+APrPAreF9QykdkwoAitKcUo/AzsVa+byUWtHMmEcuPFRltUJhqtYvfN5ZyD2rTS1GcA/SulVwC9Sm5z+7+qniHeWSqACgKI1BSq3bYlOp9jKXVZ2W5sLBTyHhV3B0hxH3w6gHtOKeynQ68ArVXl1Ht8ihKB2HCgCKcrnKi+Dr+Vp/97VxD4Yrn4eh81TZvNIqqADQhrWG8QDOs1gsDBs2jNDQUNavt9uxa/tUkgOf36r1dz/uz+AVpnVx7OBcdWpwgYCoZhnnVVHqSwWADqqxxgM476233qJ3794UFBQ0yv7ahMI0+PQGyI6DWz/RBi9RlDakXQWAV/e8yomcE426zyjfKJ4a8VSt63Tk8QAAkpKS2LBhA3/72994880363tq27ac0/Dp9VCUCXO+gm4TWzpFitJgqvenRhIXF8ejjz7KkSNHOHHiRMV4AIsWLeKVV14hKiqKrVu3cvDgQV588UWeeeYZQAsKcXFxrF27lvnz59drPICDBw9y3XXXce7cOaDqeACHDh1Cr9dXBJ3z4wEcOHCACRMm8MILL3DzzTczbNgwVqxYwaFDh3BxcQEujAfwpz/9iUWLFgFab6CDBg266DVmzJiKdD322GO89tpr6DpKZ2Lpx2DZNCjNg7nrVOavtFnt6g6griv1ptRRxwNYv349gYGBDB06lF9//bWu09T2Je2Dz27Syvbnb4SgPi2dIkW5ZO0qALSkjjoewPbt21m3bh3ff/89ZWVlFBQUcOedd/LZZ5/VmZY2J/4XWDkH3APgrm9Vh2pKm9dB7tlbXnsdD+Bf//oXSUlJnDlzhpUrVzJ58uT2l/lbLbDzHa21j09XuHuzyvyVdkEFgGby5JNP8vTTTzN27FgsFkvF/OrjASxYsKAiI6/uueeeY+vWrQwZMoQffvjB7ngAAwYM4KqrriI1VRtuufJ4AFu2bOHZZ58FLowHMGjQIEpLS5v46NuwrFPw0XTY/LRW1j9vA3gEt3SqFKVRqPEA2jk1HsAlslq0Lht+eUVruz/tVRh4O9RShKYorVVN4wHU6w5ACDFNCBErhIgTQiyws9xLCPGdEOKwECJGCDHfNt9ZCLGn0vwXKm3jK4T4UQhxyjZVj0YqrUPGCfjwam0Qle5T4KE9MGi2yvyVdqfOACCE0ANLgOlAH2C2EKJ604eHgGNSyoHAROANIYQjUA5Mts0fBEwTQoyybbMA+FlK2RP42fZZQRsPoHqzy4ceeuiS9tXar/4b08GMgzz404PsSNlR98r2WMyw7Q343xWQkwA3fQi3r1BFPkq7VZ9WQCOAOCllAoAQYiUwCzhWaR0JeAitiYk7kAOYpVa+dD4HMthe58ucZqEFC4CPgV+BlmvH2Yqo8QAaxiqtLI9ZztsH3gZgW/I25vaZy/8N+T8c9Y7VVrbYettM1PrRzzundc2cnwiZJyH/nDZs4rVvaD1t1oOUkq1JW4nOjsbf2Z8A1wACXAIIcA3Az8UPg87Q2IesKI2iPgEgFEis9DkJGFltncXAOiAF8ABuk1JaoeIOYj/QA1gipdxt2yZISpkKIKVMFULY/W8TQtwH3AdUVHoqynl5ZXn8bfvf2Jq0lau6XsUzI5/hvcPv8fGxj9mTtoeF416hW3EenPoR4n6E1MNgNVfZh3T1Y69PMF/5exHc5w7mjHuOYLf6Zf7RWdEs2reI/en77S4XCHycfQhwCWBI0BCmR0xnYMBAdEK1v1BaXn0CgL2Cz+o1x1OBQ8BkoDvwoxBim5SyQEppAQYJIbyBtUKIflLK6PomUEq5FFgKWiVwfbdT2r/DmYf5629/JbM0k6dHPM3sqNkIIfh7v/sZa7Ty7Om13Pbt9TyVncNNRaWI0GEw+mHbSFtdMHt24of8WJbHfsHxnON4GbwoSt/JZ6unMy1iGvP6ziPSN9LudycXJfPWgbfYeHojvs6+/GPUP5jVYxb55flklmaSWZJJZmkmWSVZZJRmkFqcyppTa/jixBcEuQYxNXwq0yOm09evb63PZihKU6pPAEgCOlf6HIZ2pV/ZfGChrcgnTghxGogC9pxfQUqZJ4T4FZgGRAPpQogQ29V/CGC/7aOiVCOl5JNjn/Cf/f8hyC2Iz6Z/Rl//vpAVB988AEn7mIRktUcAzwQF84K/YPvA8Tx/xSt4OXlRbCpmzak1fLrtNVKLUwn3DOf50c8zo/sMskuz+fTYp6w+tZr1CesZHTKaef3mMTpkNEIICowFfHDkAz47/hk6oeOP/f/I3f3uxt3RHYBA10ACXQPB7+J0F5uK+SXxFzaf3sznJz7nk2OfEOYextTwqYwLHUepubRK8KiYlmYS5BrEfQPu44rQK1TAUBpNnc1AhRAOwElgCpAM7AXukFLGVFrnXSBdSvm8ECIIOAAMRLt7MNkyfxfgB+BVKeV6IcTrQLaUcqGtZZGvlPLJ2tKimoG2Tw35G+aX5/OP7f/gl8RfmNJlCi+OfRFPR0+t0vaja7WRs0Y+AD2vhOCBWAV8HPMxbx98G19nX67qehXr4tdRaCxkSOAQ5vebz/iw8RcVyeSX5/PVya/4/PjnZJZm0sunF+PDxvPVya8oKC9gZveZPDL4EYLdLq2COL88ny3ntrD5zGZ2pe7CIi1Vlns5eWn1CLa6hP3p+0kuSmZgwEAeGvQQo0JGqUCg1FtNzUDr9RyAEOIa4D+AHlgmpXxZCPEAgJTyPSFEJ2A5EIKW6S+UUn4mhBiAVsGrR2tx9KWU8kXbPv2AL4EuwDngFillTm3pUAGgqtYyHsDdd99d0SdQdPSF0r2cnBxuu+02zpw5Q3h4OF9++WVF/0WV1fY3lFJytuAsO1N3sjNlJ3vT9lJmKeOJoU8wp/ccLRPMOwcfXQPGYu1BLTv988Rkx7Bg6wLOFZ5jSpcpzOs7jwEBA+o8NqPFyIaEDXwc8zHx+fGMDBnJX4b9hSjf+vWUWh85ZTkczTyKl5MXga6B+Lv4X1R5bbKY+Cb+G/53+H+kl6QzLGgYDw9+mKFBQxstHW1FobGQ0/mnKTYVI4RAh06bCh06oUOgvTfoDDjqHbWXTps66Z0w6A04CIcOFUAvKwC0FioAVHU5mfbEiRNZtGgRw4YNu+x9bd26FXd3d+66664qAeDJJ5/E19eXBQsWsHDhQnJzc3n11Vcv2r763zC3LJfdqbsrMv3UYu2p5lD3UEaFjOKWyFvo69dXW7kgRXtStzQX7lp3YWxdO0wWE8WmYrydvRt8jFZpJbMkk0DXwBbNOIwWI1+f/JoPjn5AZmkmo0NG89DghxgYMLDF0lRdalEq25K3sSNlB6XmUoLdggl2DSbYLZggt6CKz64G+73egna+C8oLSMhPID4/noS8BOLz4onPjyej5PJLi530ToS4hRDiFkIn904XTYPdgttVRX2HCABpr7xC+fHGHQ/AqXcUwbaum2vSUuMBbNq0if379+Pv79/g8QDmzZtHaGhoxXgAvXv3Zu7cuZc0HsD5czBjxowqASAyMpJff/2VkJAQUlNTmThxIrGxsRdtGx0TTaJrIvvT93Mw4yCnck8hkXgYPBgRMoLRIaMZ3Wk0nT06V818C9Nh+TXa9K5vIazjXA2XmctYFbuKZdHLyCnLIdQ9lL5+fenr35d+fv3o49enol6ioXLLcjmadZTDmYc5kXMCbydvunp2patnV8I9w+ni2QUXB5eK9Y0WI/vT9/N78u/8nvw7CfkJAIS4heDn7EdaSRpZpVkXfY+HwQOdTofVasUszVisFqxSe1+di4ML3by6aS9vberl5IWUEonEKq0XplJikRZMVhMmi4lySzlGqxGj5cKrwFhAanEqKUUppBanklNWtfDBy8mLYUHDGB48nGFBw+jp07NNB4SaAoDqDbSRxMXF8dVXX7F06VKGDx9eMR7AunXreOWVV/jkk0/YunUrDg4O/PTTTzzzzDOsXr2axx57jIkTJ7J27Vpefvnleo0H8Oyzz7JhwwaWLl0KVB0PwGAw8OCDD7JixQruuuuuivEA3njjDV588UVeeOEFFi9ezOLFi6vcAcCF8QDeeecdFi1axAcffFBnb6C1SU9PJyQkBICQkJCKPo7KzeUUm4spMZVQYiohvSSdJ/c9iauDKwMDBnLVoKsYFTKKfv79cNDV8BMtzoZPZkFBKvxhTYfK/AGcHZyZ23cut/S6hbVxazmQfoCY7Bh+OKuNSywQhHuF09evL5E+kXg6eeLq4IqrwRU3gxuuDrapwZWMkgyOZB7RXllHOFugdSSoF3oivCKIzYllXfy6Kt8f7BZMV8+uGHQG9qfvp9RcikFnYGjQUG7seSNXhF5BhFdERcA2Woykl6STVpxGWnEa6SXpFVfyeqHXXrqqU1cHVyK8Iuju3b3Jr8jLzGWkFqeSWpRKUlESR7OOsjdtLz+f+xnQAsLQwKEMDx7OkKAhhHuG13oH01a0qwBQ15V6U+qo4wE0VH55PkmFSQA46BxwNbji6eTJqhmr6OXTq+YMv7KSHPh0FuSe1kbj6jKq7m3aKVeDK3N6z2FO7zmAdvUekx1DdFY0Mdkx7E7dzfqE+o3R7Ofsx8CAgdzQ4wYGBgykj1+fikyuxFTCucJznCk4w9n8s5wt0F6Zpkyu634dV4RewfDg4TVmio56Rzp7dKazR2e7y1uas4MzEV4RRHhFAHBr5K2AVpy1L30fe9P2sjdtL1sSt1Rs4+PkQ6h7KJ3cOxHqEUqYexih7qEEugbi6eiJh6MHLg4urbquoV0FgJbUUccDsJMYrTxeWgkKCiI1NbWiCCggMICUohRcDC6EuofiqHNECEGRoYjefrY6gLICrUWPwQUMrtrL0VUbgEUIKMvXBmTJjIXZX0DE+DqPuSPxcfZhXOg4xoWOq5hXYCyg2FhMsamYEnOJNjWVVLz3dPRkYOBAOrl1qvH34WpwJco3qlErv9uCEPcQZrrPZGZ3bbzntOI0DmUeIqkwiZSiFJKLkonNjeWXxF8wWU0Xbe+gc6gIBh4GDzydPAlyDaKzR2fCPMIqgqKno2eLBAoVAJpJfcYDePjhh/n666+5+eab7e7j/HgAf//73y8aD2DWrFk8/vjjBAYGkpOTQ2FhIV27dq0YD+D222+/rPEA6kVKbYB0o9b7x3WTR/Hxu2+yYMECPvroQ8ZP1ZpbdvboXLV7BKsZdr0LJzfBme1g5x8JodOCAQLMpXDbZ9Djyvqlq4PzdPTUmsoqly3YLZhpbtMumn++kUByUTIZpRkUGgspNBZSUF6gTY3aNL88n9icWLLLsqts72HwIMwjjG7e3bil1y0MCRzSLAFBBYBm8uSTTzJ37lzefPNNJk+eXDG/+ngAkyZNYvz48RVFOJU999xzzJ49myFDhjBhwgS74wFYrVYMBgNLliyha9euVcYD8PLyYtWqVcCF8QAqDwp/qWbPns2vv2whKzuHsAHjeOHvT3PPvfex4Mm/cutd9/Lhp6sI6NyJRR++SRdHbwzooLwIyvO1K/6CFNi8APwjYdSfIGy41p7fVALGEm1a+X2f66D75LoTpijNRCd0BLkFEeQWVK/1S0wlJBUlkVSYRGJhojYtSmR78nY2JGxgUMAg7u1/L+PDxjdpIGhXrYCUizX5eADGEq0DNVMpOHuBVxhUbsMuJSmFieQaCwmzSLwsla/uBTi6czwph95d/MG3W9OlU1HagFJzKWtPrWV5zHJSi1Pp4d2De/rfw7TwafWrH6vBZY0HoCgXsVq1K/esWLCYwCdCy8CrPcCUU5ZLrrEQfxd/vAL7gl8PcA/ShlQM7g/+PcDJQ2X+ioLW3PWO3new4cYNvDLuFaSUPL3taWasncG+tH1176CBVBFQK/TRRx/x1ltvVZk3duxYlixZ0uB9NcnVv7EEcs+ApRxcfMEzFPQX/5SKTcWkFafh7uiu9Y8jhJbZO3k0fpoUpR0x6AzM7D6Ta7tdy2+Jv7E8Zjkh7iGN/j3tIgBIKVt1U6uGatXjAZTmQd5ZEHrw7Q7O9isXjRYjiYWJGPQGwtzDavz7tKUiSEVpbjqhY1KXSUzqMqlp9t8ke21Gzs7OZGdnq4ykqUkJRRla23sHZwiIrDHzt1gtnCs8h5SSLp5d0Ov0NexSkp2djbOzc1OmXFGUGrT5O4CwsDCSkpLIzMxs6aS0X1JCWR6UF2pNMV0dta6XqzFbzZRZyig1l2KymPBz9iMhI6HWXTs7OxMWFtZECVcUpTZtPgAYDAYiIiJaOhntV3kRfH03nNoMYx6BK18EnXbjaLQY2Ze+j21J29iWvK2iC4EIrwj+2P+PTOk+pSVTrihKHdp8AFCaUEEKfH4rpB+Da9+E4fcAcCLnBO8ceoddqbsoNZfiqHNkRMgI7oi6gyvCrmi1j/srilKVCgCKfalH4PPbtGKfO77UBlgBtiVt44nfnsDFwaWiD5gRISOq9A6pKErboAKAcrHCNG2AFWdPuHsTBPcDYPXJ1by06yV6+fRiyZQlBLgGtHBCFUW5HCoAKBfbs1Trz+e+X8G/B1JK3jn8Du8dfo+xncbyxsQ3cDO4tXQqFUW5TCoAKFUZS2DfMoi6Fvx7YLKaeGHHC3wb/y3X97ieZ0c/W7UjN0VR2qx6PQcghJgmhIgVQsTZBnCvvtxLCPGdEOKwECJGCDHfNr+zEOIXIcRx2/xHK23zvBAiWQhxyPa6pvEOS7lkR1Zq3TmPfogiYxEP/fQQ38Z/y4MDH+TFMS+qzF9R2pE67wCEEHpgCXAVkATsFUKsk1Ieq7TaQ8AxKeVMIUQAECuEWAGYgSeklAeEEB7AfiHEj5W2/beUclGjHpFy6axW2PkOhAwiw787D26aR1xeHC+OeZEbet7Q0qlTFKWR1ecOYAQQJ6VMkFIagZXArGrrSMBDaM/7uwM5gFlKmSqlPAAgpSwEjgOhjZZ6pXHF/QTZp0gbeid3bvwDiYWJLJ6yWGX+itJO1ScAhAKJlT4ncXEmvhjoDaQAR4FHpZTWyisIIcKBwcDuSrMfFkIcEUIsE0L42PtyIcR9Qoh9Qoh96mnfJrZrCcWenXgoeSMFxgKWTVtWZWSp5hCdnM+nu85isaquPRSlqdUnANjrxav6f+dU4BDQCRgELBZCVHQUI4RwB1YDj0kpC2yz3wW629ZPBd6w9+VSyqVSymFSymEBAarZYZNJj8Gc8Ct/CetKfH4Cb054k75+fZs1Cd8eSuamd3fwj2+iuf/T/RSXm5v1+xWlo6lPK6AkoPKjnWFoV/qVzQcWSq1HtjghxGkgCtgjhDCgZf4rpJRrzm8gpUw//14I8T5Qv5GrlSYhdyxhYUAAv5cm8+zoZxkTOqbW9XOLjSTnlZKaX0Zqfikpeeen2ry+nTz52zV96OJnf5DwyixWyWubT/C/3xIYGeHLxMhAXt98glv/t5MP5w4n2Et1FqcoTaE+AWAv0FMIEQEkA7cDd1Rb5xwwBdgmhAgCIoEEW53Ah8BxKeWblTcQQoRIKVNtH28Aoi/9MJTLUpTBZ2c3sMrHk3l953FLr1tqXDUhs4h/bjjOlhMZVeYb9IJgL2dCvFwY2NmbX09kcNW/f+PBiT24f0I3nA32ewQtKDPx6BcH+SU2kztHdeG5mX0x6HVEBrvzyOcHmbXkdz6cO5x+oV6NesiKotRzSEhbE83/AHpgmZTyZSHEAwBSyveEEJ2A5UAIWpHRQinlZ0KIccA2tHqB83UCz0gpvxdCfIpW/COBM8D9lQKCXfaGhFQu35YND/FY5m9MCRnNG1f/D524uGSwsMzE4i1xLNt+GicHPXePi6B3sAedvF0I8XbG380Jne5CaWFafhn/3HCM9UdS6ernyvPX9WVSZNVxjhMyi7j3k32cyy7h+ev6cueorlWWH0sp4J6P95JfauLt2wdzZZ/6jbeqKEpVNQ0J2ebHBFYuT0z6QeZv/APddS4su2PrRX36WK2S1QeSeHVTLFlF5dw6LIy/To0iwMOpXvv//VQWz66LJiGzmKv7BPHszD6E+bjya2wGj3xxEINex7tzhjCym5/d7TMKyrjn431Ep+Tz92v7cPfY8HY1+I+iNAcVAJSLpBWnccc31+NQls/nY1/DP2pmleUHz+Xy/HfHOJyYx+Au3jw/sy8DO3s3+HuMZisf/J7Af3+OQyKZ1jeYdYdT6BXkwft3DaOzb+31BCVGM4+vOsTmmHT+MKorz83sg4O+cccyKjVa+D0uC29XA4M7e9d7/+VmC7/GZrLuUArxmUU4G/S4OupxMehxcdTeuzo64OKox9PZgLerAR9XA14ujrb32rSmIjJFaQwqAChVFBmLuGvjXaTmnuKTMjd63r9TG7MXKC438/y6GL7an0SghxMLpkdx/aDQKkU8lyI5r5QXv4thc0w60/sF88atA3F1rF9vJFar5NVNJ/jf1gR8XA307eRF306e9A3VphF+bg1On9Fs5fc4LfP+4Vg6JUYLAN6uBiZFBjI5KpDxvQLwcqn69LPVKtl9Ood1h5PZcCSVgjIz/u6ODOrsQ7nZQqnRQonRQqnJQonRrL03WjDX0rTV09mBmQM7MWdkV/p0sj/SmqJcKhUAlAoWq4VHtjzCjuTtvJOaxpipb8LgOYBW5HL3x3s5llLAfeO78/DkHrg7NW6XUYk5JYT5uFxSUc7mmDS2HM8gJjWf2LRCTBbt9+vqqKd3iCd9QjwJ83Eh2MuZIE9ngj2dCfZyrrjCvpB5p7AxOpW8EhNeLgau6R/MjAGdyCsx8fPxdH6JzSC3xISDTjA83JcpvQPpF+rFLycyWHc4hdT8Mlwd9UztG8ysQZ0Y18O/1rsGKSWlJgt5JSbby0heqe19qZGTaYV8H52G0WxlcBdv5ozsyowBIerOQGkUKgAoFf69/98si17GP0Qgt6afg8ejwcGJ2LRC5n+0h7xSE0vuGMKkqMC6d9aCjGYrpzIKiUkp4FhKATEp+ZxILaTQzvMD3q4Ggj2dyS0xkl5Qjqujnqv6BHHdwE5c0TMAR4eqmbfFKjmUmMtPxzPYcjyD2PRCABx0gvG9Apg1qBNX9Qmq9x1MfeSVGFl9IJkVu8+SkFmMp7MDNw/tzB0ju9Aj0L3RvkfpeFQAUAD4PuF7ntr2FLd2uZp//PYBTPobTHiS309l8afP9uPiqGfZvLbd7LKwzER6QRlp+eWk5pdq7wvKSMsvw6DXcU3/EKb0DmxQ5p2YU0JMSj4jIvzwdXNswtRrdwu7EnJYsfssm2PSMFkkkyIDeHFWvzrrSxTFHhUAFGKyY5i7cS59/frygdEDw5Ev4c/H+PJ4Kc+sOUr3AHc+mj+cTt5qdK/WIquonFV7E3nnlzisEp64uhfzxoQ3eiW40r7VFADUr6iDyCrN4tEtj+Lr7MubQ5/EcGQVcuBs3tyezZNfH2F0dz+++tNolfm3Mv7uTjw0qQc//nkCY3v48c8Nx7nhnR3EpOS3dNKUdkAFgA7AZDHx51//TH55Pm9Negu/fR8jpZWX8qfx9pY4bh0WxrJ5w/F0Vn39t1advF14/65hLLljCKn5pVy3eDuvbjpBmcnS0klT2jAVANo5KSUv736ZgxkHeWnsS/R29IEDH7PH4yqWxVj5y9W9ePWmARhUkUKrJ4Tg2gEh/PTnCdw0JJR3f41n2n+2siM+q6WTprRR6r++nfsy9ktWn1rNvf3vZVrENNjxX6TFyNOZV3P32AgentxTPVnbxni7OvLazQP5/N6RSOCO93fzyvfHVRfaSoOpANCO7U3by8I9CxkfNp6HBz0MRZmwbxnxQdeQYA3i9hGd696J0mqN6eHP5sfGc+eoLizdmsB9n+yjsMzU0slS2hAVANqpU7mneOLXJ+js2ZmFVyxEr9PDzsVgKmVR2bX0D/WiV5BHSydTuUzOBj3/vL4/L13fj19PZnLTuztIzClp6WQpbYQKAO1MmbmMtw+8za3rbwXg7Ulv4+HoASU5sPcDCnpcx6Y0L24cokbmbE/+MKorn9w9gvSCcmYt2c6e0zktnSSlDVABoB3ZmbKTG9fdyPtH32d6+HS+uf4bwr3CtYW73gFjEatcbsVBJ5g5sFOLplVpfGN7+PPNQ2PxdjUw54NdfLk3se6NlA6tcTt5UVpEdmk2i/YtYn3Cerp4dOH9q99nVMioCyuU5sHu/yF7X8cHsc5MjPTC371+3TkrbUuEvxtrHxzLw58f4MnVRziZXsjT1/RGf5kd+SntkwoAbZhVWvkm7hve2PcGJeYS7h9wP38c8Eec9NUy993/g/ICDobfS/rBIp6bGdYyCVaahZeLgY/mDeefG47zwe+nOZKcz7CuPvi5O+Hv7oifmxO+bo74uzvi4+aomgB3YCoAtFFmq5k//fQndqXuYkjgEJ4b/RzdvLtdvGJZgVb8E3kNn57xwtO5jMmtvJM35fI56HU8f11fega5s2RLHAfO5tbYHXWotwv9Q70Y0NmLgWHe9Av1uqgLbKV9UgGgjdqevJ1dqbt4fOjjzOs7z+4wjgDsfR/K8igZ/Wc2fZjGDUNCVRfDHcickV2ZM7IrUkoKSs1kFZeTXWQkp7icrCIjWUXlxGcWcyQpj00xaRXbRfi7MSDMiwFh3vQJ8SQq2AOfJu4ET2l+9QoAQohpwFtoYwJ/IKVcWG25F/AZ0MW2z0VSyo+EEJ2BT4BgtDGBl0op37Jt4wusAsLRxgS+VUqZ2wjH1CGsjVuLr7Mvf+jzh5oz//Ii2LkEelzF99khlJoyuEm1/umQhBB4uRrwcjXQPcD+OnklRo4k5XM0OZ/DiXnsTsjh20MpFcuDPJ2IDNaCQVSwB5HBHvQIdMfJQV1QtFV1BgAhhB5YAlwFJAF7hRDrpJTHKq32EHBMSjlTCBEAxAohVgBm4Akp5QEhhAewXwjxo23bBcDPUsqFQogFts9PNe7htU/Zpdn8lvgbd/a5E4Oullv1fcugJBsmPMmaTUl09XNlSBef5kuo0qZ4uzoyvlcA43tdiBAZhWWcSC3kRFoBJ9IKOZFayPL4bIwWKwB6nSDC341eQe70CvIgMsiDnkEehPu5qh5L24D63AGMAOKklAkAQoiVwCygcgCQgIfQ+hRwB3IAs5QyFUgFkFIWCiGOA6G2bWcBE23bfwz8igoA9bI+YT1maeb6HtfXvJKxBHb8F7pNJNmjPzsTtvDYlF6q2welQQI9nAn0cK4SFEwWK2eyijmeVkhsWgEn04uISSlgY3Qa53uXd9Tr6B7ozqhuvtw+vAuRweqhw9aoPgEgFKjcoDgJGFltncXAOiAF8ABuk1JaK68ghAgHBgO7bbOCbAECKWWqEMJuzaQQ4j7gPoAuXbrUI7ntm5SStafWMiBgAN29u9e84o63oTgDxi/nm4PJSAk3DFbFP8rlM+h19LRd6VPpeZJSo4W4jCJi0ws5mV7I8dQCPtt1lo+2n2FQZ29uH96ZGQM7NfoQo8qlq89fwt4lY/XmBFOBQ8BkoDvwoxBim5SyAEAI4Q6sBh47P6++pJRLgaWgDQjTkG3bo+isaOLz43l+9PM1r5S4F357Dfrfguw6hjWrf2NEuC9d/NRoUkrTcXHU0z/Mi/5hF0aTyyk2suZAEqv2JrJgzVFeXH+MmQM6cduIzgzu7K3uSFtYfQJAElC517AwtCv9yuYDC6U2vFicEOI0EAXsEUIY0DL/FVLKNZW2SRdChNiu/kOAjEs+ig5kTdwaXBxcmBo+1f4K5YWw5l7wDIVr3+BIUj7xmcX88Qo7TUQVpYn5ujly7xXduGdcBAfO5bFq7znWHU5h1b5Eega6M6qbH71DPOkd4kFUsCcujqpCuTnVJwDsBXoKISKAZOB24I5q65wDpgDbhBBBQCSQYKsT+BA4LqV8s9o264C5wELb9NtLPooOotRcysbTG7mq61W4O9YwSPjGpyDvHMzbAM5erD4QjaODjmsGhDRvYhWlEiEEQ7v6MLSrD/+Y0Yf1R1L59lAyaw8m8+muswDoBIT7u9E7xJM+IZ70DHQn0NMZf3dH/N2dVPPlJlBnAJBSmoUQDwOb0ZqBLpNSxgghHrAtfw94CVguhDiKVmT0lJQySwgxDvgDcFQIcci2y2eklN+jZfxfCiHuQQsgtzTysbU7P539iWJTMTf0uMH+CjFr4dAKGP9X6DoGo9nKusMpXN0nSI32pbQaHs4GZo/owuwRXbBaJUm5pRxLLeBYagHHUws4nJjHhiOpF23n6eyAv4cTAe5OBHg4Mb5XADcNCVPdXFwGNSh8G3L35rtJL05n/Q3rLy47zU+Cd8eAXw+4ezPoDWyOSeP+T/fz0bzhTFJP/yptSH6piTNZxWQVlZNZWF4xzSwqJ6vQSHJeKcl5pUQGebBgehQTIwNUfUItahoUXlXHtxGJBYnsTdvL/w3+v4t/6FYLrH0ALGa48X3Qa1f7aw4k4e/uyBU9/VsgxYpy6bxcDAzs7F3jciklG6PTeG3TCeYv38vobn48fU0UA8Jq3ka5mAoAbcQ38d+gEzpmdp958cId/4Uz22DWEvDTmobmFhvZciKDu0aHqwdylHZHCME1/UO4sncQX+w5x1s/n+K6xdu5bmAn/jo1ks6+tbd4s1olqQVlnM0q5kx2CWeyizmTVczZ7BLO5hTj6+pIf1tXGAPDvOkf1j77R1IBoA2wWC18G/ctYzqNIdgtuOrClEOw5Z/Q+zoYNKdi9sq9iZgsUg38orRrjg465o4J54Yhofzvt3g+2HaaTdFp3Da8M37ujhSUmikoM5FfaqKg1ERBmZmCUhOZReUYzdYq+wn3c6WrnxvjevqTWVjOkaQ8NsekV6wT7ufKgDBvhnTx5uZhndvF8wxt/wg6gF2pu0gvSefJ4U9WXWAsgdX3glsAzHwLbEVDmYXlLPkljilRgfTt5GVnj4rSvng6G/jr1CjuHNWVf/94khW7z2KV4O7kgKezA54uBjxdDIT5uOAZ4omfuyPhfm6E+7kS7u9GsKczOjuVyfklJq1vpKQ8jiTlsfdMDusOp7Dk13j+OjWSm4eE2d2urVABoA1Yc2oN3k7eTOw8seqCH/4G2XFw17fg6lsxe9HmWMrNFv52be/mTaiitLAQLxdeu3kgL1zXD4NeXHbxp5ergXE9/RlXqR7tUGIeL3wXw5NfH+HTnWd5bmYfhoX71rKX1ksVDrdyuWW5bEncwoxuM3DUV+qONy1a6+xt9EPQbULF7OjkfL7cn8i8MeF0C6jhWQFFaedcHPVNVvc1qLM3a/40hv/cNojMwnJufm8nj3xxkOS80ib5vqakAkAr9/3p7zFbzdzQs1rb/51LwOAKVzxRMUtKyQvfxeDr6sgjU3o2c0oVpeMQQnD94FC2/GUC/zelJz/EpDHljV/5948nKTVaGvW7pJTEZxZRZmrc/YIqAmrVpJSsObWGvn596eXT68KCwjQ4+hUMm1+l6Gf9kVT2nsnlXzf2Vw9+KUozcHV04M9X9eLWYWEs3HiCt34+xbLtpxndzY8regVwRQ9/uvq5NugZBS3DL2ZXQja7ErLZfTqHzMJyPrl7RJVeWRuDCgCt2PGc45zMPcnfR/696oI9S8FqhpEPVMwqNVpYuPEEfUI8uXVYZxRFaT5hPq4svmMIc8fksHp/EttOZfHDsXTbMheu6OnPuB4BjO3hh7NBT06xkewiY5UR2rKLjCTmlrDndC5ZReWANgjPmO5+jIzwIyqk8bvUVgGgFVtzag1Oeiemd5t+YaaxGPZ+CFHXVrT5B1i6NYHkvFLevHWgejReUVrI8HBfhof7IqXkTHYJ205lsu1UFusPp/LFnsRat3V00BHk6cS4Hn6M6qa9Gnr30FAqALRSJaYS1ies56quV+Hp6HlhwaHPoSwPxjxSMSslr5R3f4vj2v4hjOzm1/yJVRSlCiG0kdIi/N24a3Q4ZouVw0l57IzPBsDP3Qk/N8dKU0fcnRyavTsLFQBaqfUJ6yk2FXNb5G0XZlotsOsdCB0KnS+MyfPqphNICQumR7VAShVFqYuDXsfQrr4M7dq6mouqVkCtkJSSlbEr6e3bm4EBAy8siN0IOQkw+uGKh772n9UG7r5vfLc6H39XFEWpTAWAVuhgxkFO5Z7itsjbqt4S7lwMXl20bh/Q+jN54btjBHs686eJtQwPqSiKYocKAK3QytiVeBg8mB5RqfI3aT+c2wmjHgC9VnK3+kASR5LyeWp6JK6OqjRPUZSGUblGK5NVmsWPZ3/k9sjbcTVUKtLZuRicPDENnMO++Gy2nEjny31JDO7izayBqsM3RVEaTgWAVmbNqTWYrWZujby1Yl5+Sjwex75li/fN/Pm13RSUmXHU6xjV3Y9nZ/Rp051RKYrSclQAaEXMVjNfnfyKUSGjiPCKYHtcFm/9dIqpSW8xVy/5d8EkpvYNZkrvIMb19G8X3dEqitJy6pWDCCGmAW+hjQn8gZRyYbXlXsBnQBfbPhdJKT+yLVsGzAAypJT9Km3zPPBHINM26/xYwR3W1qStpBWnsWD4AixWyV+/OoybLOZOp60UdJ3Bd3ferq72FUVpNHVWAgsh9MASYDrQB5gthOhTbbWHgGNSyoHAROANIcT5riuXA9Nq2P2/pZSDbK8OnfkDrIpdRZBrEBM6T2DrqUxS8st4O/IoTpZifK/8s8r8FUVpVPVpBTQCiJNSJkgpjcBKYFa1dSTgIbQ2i+5ADmAGkFJutX1WanEm/ww7UnZwS69bcNA58MXucwS56og6uwK6joNOg1s6iYqitDP1CQChQOVOLJJs8ypbDPQGUoCjwKNSSit1e1gIcUQIsUwI4WNvBSHEfUKIfUKIfZmZmfZWaRe+PPklDsKBm3rdREZBGT+fyODpbqcQBclan/+KoiiNrD4BwF65g6z2eSpwCOgEDAIWCyE8qd27QHfb+qnAG/ZWklIulVIOk1IOCwho3K5QW4tScynfxH3DlV2vxN/Fn6/2J2GxSqaWbADfbtCrphI0RVGUS1efAJAEVO5fOAztSr+y+cAaqYkDTgO1dkwjpUyXUlpsdwrvoxU1dUgbT2+k0FjIbZG3YbVKVu49x7SuApeU3TDgNtCp5/UURWl89clZ9gI9hRARtord24F11dY5B0wBEEIEAZFAQm07FUKEVPp4AxBd30S3J1JKVp5YSQ/vHgwNGsr2+CwSc0r5U9AxQEKf6tUtiqIojaPOACClNAMPA5uB48CXUsoYIcQDQojzI5K8BIwRQhwFfgaeklJmAQghvgB2ApFCiCQhxD22bV4TQhwVQhwBJgGPN+qRtRFHs45yPOc4t0fejhCClXsS8XY10C//V/DvBQGqh09FUZpGvZ4DsDXR/L7avPcqvU8Brq5h29k1zP9D/ZPZfq2KXYWrgyszus8gq6icH46l8cAwT/RHtsMVf6no9VNRFKWxqcLlFpRblsum05uY2X0mbgY3Vu9PwmSR/MH7KEirKv5RFKVJqQDQgn48+yNGq5Fbet2i1QXsTWR4uA+BiZvAtzsE9W3pJCqK0o6pANCCdqfuJsg1iF4+vdiVkMPprGLuGuABp7dB3+tV8Y+iKE1KBYAWYpVW9qTtYWTISIQQfLHnHJ7ODkxz2AfSoop/FEVpcioAtJCTuSfJK89jZMhIcouNbIpO44bBoRhivwOfcAge0NJJVBSlnVMBoIXsTt0NwMjgkaw+kITRYmXOQA84/Rv0uV4V/yiK0uRUAGghu1J3Ee4ZTqBrICv3JjKosze9creB1ayKfxRFaRYqALQAk9XE/vT9jAwZyb6zucRlFHHHiC5w7Fvw7qJ6/lQUpVmoANACorOiKTWXMipkFF/sOYe7kwMzIl0h/hft6l8V/yiK0gxUAGgBu1J3IRBEeg1iw5FUZg3qhGvCD2A1aeX/iqIozUAFgBawO3U3Ub5RbDicT7nZyuzzxT+eYRA6tKWTp9ghzWZKDx+mPD4eS0EBUlbvEV1R2h41qngdpMlE4c8/43HVVQi9/rL3V2Iq4XDmYW7vNYf3f0jgip7+9PMD4n+G4X9sE8U/5sxMinftxuPKKehcXFo6OU1Kmkzkr/uOrP/9D9O5cxXzhbMzDgEB2iswEIeAAPQ+3uicnBEuztrU2QmdszPCyRmdqwvO/fqhc3Ss5dsUpXmpAFCHom2/k/zY44S88greN95w2fs7mHEQs9VMQW442cVGHruyJ5zcDBZjm2j9Yyko4Nzdd1N+Kg69jw++d/0BnzvuQO/l1dJJa1RWo5H8NWvJfv99TMnJOPfpQ8Drr4HQYc7MxJyRoU0zMymPjaX499+xFhXVuk+nyEjC3voPjuHhzXMQilIHFQDqYErVxr7J/vBDvK6fhbjMwVl2p+3GQefAxn3OXNHTn6FdfWHlt+DRCcKGN0aSm4zVaCTpoYcpP3OWoH/8neKt28h8622y3/8A79tuw3feXAxBQS2dzMtiLS8n76uvyf7gA8xpaTgPHEDws//Abfx4RB13Z9JsxlpWjiwvQ5aVYbW9ZHk5pqQk0l9+hdM33UzIK6/gOdVu57mK0qxUAKiDOT0DAGN8PEW//obH5EmXtb/dqbsJNPQitkjw6JSeUF4Ip36EYfNb9chf0moldcHTlOzdS6fXX8dr5gx858yhLDaW7Pc/IOfjj8n57DO8Zl2H39334NQtoqWT3CDSaiX3iy/Ifu9/mDMzcRkyhJCX/4nbmDF1ZvznCQcH9O4O4O528cIhQ3AdNoykxx4n+dFHKZ07l8C/PIEwGBr5SOompcSUnELpwYOUHjyItFpwjorCKTIS51690LnZSX9Dv8NopHj3brBacRkyBL2HRyOk3LZvqxVTSip6by/07u6Ntt+OSAWAOpgzMnAICEAYDGR/+OFlBYD88nyOZx9Hl3c1V/T0Z1i4Lxz9Gizlrb74J+ONNyj4/nsCnvgzXjNnVMx3jowkdNHrBDz6f+R89BF5q9eQv3oNHlOn4v/gn3Du1asFU10/pvQMUp9eQPGOnbgOH06n11/HdeSIemf89WXo1Inwzz4l/dXXyPn4Y0qPHiX03282+V2TNBopO36ckoMHKT2gZfrmzEwAdK6u4OBA3spV2spCYOjSGefIKJx7R+EUGYVzVCQOISF13wEZjRTt2EHhxk0UbtmCtbBQW6DT4RQVieuwYRUvB1/futMtJeaMDMpPnqL81CnK4+K0aXw8sqQEDAbcRo3CY8oU3CdPwhAYWPc+rVZM585hyshA5+JS8RKurtrUyanR/+6Xq/z0aXI//Qz/Rx7GwcenUfct2lJrhmHDhsl9+/Y163eeu/turMUleF57LemvvELXzz/HdcilPaj109mfePzXxyk58wCr5s7WAsCqOyFxL/z5eKu9A8j59DPSX34ZnzvuIOgff6/1H8SclUXOJ5+Su2IF1uJiWyB4EOfI1hkICrdsIfWZv2EtKyPomafxvuWWZskA8jdsIPUfz6Jzdib0jUW4jR590TrSaMSUkoIxKRlrQT7SYkGaLWDVptJiBotVK3oqLsZSkI+1oBBLYSHW/HwshYVYCgqwZGcjjUYADKGhuAwejMuQwbgOHoxTz56g12NOTaXsxAnKTpyg/EQsZSdOVKn01nl54RwZiVNUJM5RvXGOisSxRw8EaJn+ps0U/vwz1sJCdB4eeEyZgse0qeicnSnZu4+SffsoPXQIWV4OgGP37rgMGohwMGAtLUGWlmItKcVaqr1kSQnmnJwLQQTQ+/vj1LMHTj164tS9G8az5yj86SdMiYkAuAwciPuVU/C48kqcIiKQRiPl8fGUHTtO2XHtVX7iBNbi4pr/MDodOhcXHCMitGA1fBguQ4Y0esZbFyklpQcPkr1sGUU/b0EYDIS+/RYeEyde0v6EEPullMMumq8CQO3ir52BU7dudHp1IXGTJuMyfBidFy++pH09v+MlVsd+Q3/L23x+71goL4LXu8PgP8C1ixq0LyklOR8tx1pagiEoCIegIBwCgzAEB6Hz9Gy0TKzghx9IfvQx3CdPJuztt+rdEsqSl0f2xx+T+8mnWiC46ir8H3oQ56jWMcSltayMjNdeI/fzL3Dq3ZvQNxbh1K1bs6ahPD6epP97FOPp0/jOnYvOxQVTchLG5GRMScmY09OhAf+fOjc3dF6e6D080Xt6ovPUpnpfH1wGDMRl0CAMQXVfJZ9nKSqm/GRslaBQfvIksqxMW8HBAZ2jI9aSkiqZvvuYMQg7rZ2k0UhpdAwl+/ZRsn8fZUeOahmus9ZKSri4Xrgqd3VB5+WFU48eOPXsiVPPnnYzYSkl5adOUfTzzxT+9DNlMTFa0kJCsGRlIU0mAISLC85RUTj37o1zn94YOnXCWlZuJ/iUYC0uofz4cUqPHKkInE49e+I6XLt7cRk0CJ2nFzpnJ4RD4xaiSIuFwp9+JmfZMkoPH0bv5YXPnDvwueMOHPz9L3m/lxUAhBDTgLcAPfCBlHJhteVewGdAF7RipUVSyo9sy5YBM4AMKWW/Stv4AquAcOAMcKuUMre2dLREAIgdNhyv668n+O9/I/Ptt8l69z26bVh/SZnF+M+nkZnrzifXLmV4uC/ErIWv5sHc9RBxRYP2VXo0mjO33GJ3mXB2xiEoEMcuXfGacS0eU7UrsYYqOXCAc/Pvxjkqii7LP7qkJp+WvDxyPvmEnE8+xVpUhMdVV+L/4IM4RUW12K12WWwsyU88gTEuHt/58wl4/LEWa55pLS4m9bnnKVi/HoTAITgYQ2gnHEPDMISFYQgNxRAWqhWZ6PUIBwctCOv1VaY6V9dGz4zskRYLxrPnKD9xnLITsVgK8nGfOBG3MWNaRRNXU0oKhVt+oWTfPgyhnXDu3QfnPr1x7Nq1wc24reXllB09qgWsffspPXAAa0lJ1ZUMBnROTghn5wtTZ2dbsZILOhdXrSnw+fcuLkiLGVlWjrW8DGlrNGAtK0eWlWE8exZTcjKGLl3wnTcX7+uv14rpLtMlBwAhhB44CVwFJAF7gdlSymOV1nkG8JJSPiWECABigWAppVEIMR4oAj6pFgBeA3KklAuFEAsAHynlU7WlpbkDgLW4mNihwwh44s/4//GPmLOziZs8Ba/rZhLy0ksN2tfZvBRmfDuVEMst/HD3s9rMr+bDmW3wRCzoGvbjzFi0iOzlH9Pzly1Yy42YM9Ixp6VhSs/AnJ6OKT2NsugYTImJ6Dw88Jo5E+9bbsa5d+967b884TRnZ89G7+1N15VfXPYtsCU/n5xPPiXnk08qigkcO3fG0Lkzjl1s086dMXTugt7TA1NamnY8qWmY0lIxp6ZVzHMIDMRtzBjcxo7BuU+fev9jSynJ/fQzMhYtQufpSaeFC3EfN/ayjqsxSCmxZGej9/S0e+WstA7SbKbs+AnKYqKxFhdrLbzsZOTWslJkaVnFHYWsXLRVVgY6XdWAUWmq9/bG66Yb8ZgypVGeOzqvpgBQn0uGEUCclDLBtqOVwCzgWKV1JOAhtEs6dyAHMANIKbcKIcLt7HcWMNH2/mPgV6DWANDcTBlaC6DzlXQOfn543XgD+V+vxv+RR+pV6XTe2zs2AvDH4bbmf6YyOPUD9L+5wZm/lJKCjZtwGzMah4AAABzDQi9ez2qlZO8+8r7+mryvvyb3889x7tsX71tuxvPaa9F7eCCtVszp6RjPJWJKStSmiYkU79kDDg50/uD9Rin/1Ht5EfDIw/jOvYv8dd9hTIjHmJhE+YkTFG7ZArZbdbt0OhwCAjCEhOAUGYnx3Dky//MfMv/zH/ReXriOGlUREBzDwpBmM6bkZMoTEjAmnKb8tDY1xsdjyc/HfcIEQl55GQc/v8s+rsYghLis23uleQgHB1z698Olf7+6V66BtFpBiFZT0VyfABAKJFb6nASMrLbOYmAdkAJ4ALdJKa117DdISpkKIKVMFULYzU2FEPcB9wF06dKlHsltPOebgDpUyuj95s0jb9WX5H76GYFP/Lle+yk1WvjpzHb0ru7c1G+ENjN+CxiLoPd1DU5XWXQ0puRk/B96qNb1hE6H28gRuI0cgeXvfyP/u/XkffUVac+/QPrCVzEEB2NKTq4oJwVAr8fQqRPOvXsT+PhjOHbu3OD01Ubv6YnvnXOqzJMWC+a0NIyJiRgTE7EWFOIQHIQhJARDcDAOgYEXFW+Ys7Mp3rmL4h07KN6+ncLNmwFwCAjAkpdX5Zj0/v44RUTgMW0arsOG4jljRqv5B1Q6lst9jqix1ScA2PtPqV5uNBU4BEwGugM/CiG2SSkLLi95IKVcCiwFrQjocvfXEOaMdAAcAi8003Ps2hWPq68md+VK/O6/r17tkD/bdQaz40mGBwxFJ2w/gOPrwNkLIsY3OF0FGzeBwYDHlMn13kbv5YXvnXPwmXMHZdEx5K1ZjSU7B48rp2AIu1AMYwgJaZay5MqEXq+VdYeG4jZqVL22cfDzw2vGtXjNuBYpJcaEBIq376D06FEMQYE4RnTDsVsEThER6L29m/YAFKWNqs9/ehJQ+TIwDO1Kv7L5wEKpVSjECSFOA1HAnlr2my6ECLFd/YcAGQ1Id7MwpWsBoHrLCb977qZw0ybyvvwKv7vn17qPUqOFd3fsRheSzzU9bJm92Qix30PkNaBv2INAUkoKNm3EfcyYS+p+QQhx2bexrY0QAqfu3XHq3r2lk6IobUp97kf2Aj2FEBFCCEfgdrTinsrOAVMAhBBBQCSQUMd+1wFzbe/nAt/WN9HNxZyegc7d/aInI13698d15EhyPv64oplYTVbsPkshxwEYGWIrOTuzFcryL63458gRzCmpeEyf1uBtFUVRKqszAEgpzcDDwGbgOPCllDJGCPGAEOIB22ovAWOEEEeBn4GnpJRZAEKIL4CdQKQQIkkIcY9tm4XAVUKIU2gtjKo0LW0NzBkZONTwlKbfvfdgTk8nf8P3NW6fX2rivd8SCAxKJMg1iC4etjqMY+vA0R26178I57yCjZsQBgMekxu+raIoSmX1KuyVUn4PfF9t3nuV3qcAdnu3klLOrmF+Nra7htbKnJ6OQ2CA3WVu48bh1KsXOcs+xGvWdVUqd8w5ORTv2Mmvn6/n1dij/DShDN2NU7WKR6sFTmyAnleDoWFt86XVSsHmzbiNG4fe0/Oyjk1RFEX1BVQLU0YGbiNG2F0mhMDv3ntIefIpCn/4Eb2nB0Xbt1O8Yyflx7UinxCDC9LLjds3F5M609Y52tkdUJIFfRpe/FN6+DDm1FQ8H3/sUg9JURSlggoANZBWK+bMzBqLgAA8p08n4z//Ifmxx7QZBgOugwfj+fAj/OW0M1khEcwZEU3vR9+h23/XI6+Yhzi+DhycocdVDU5T4aZNCEdH3FXxj6IojUAFgBpYcnLAbMahlr5ThMFAyEsvUbx9B24jR+A6bBg6NzeeXnOU3cnn+PrWQSw9uYpTNwRz5+exZC9din/pd9DjSnBqWDe20mqlYNNm3K64QnWBqyhKo1ABoAYXmoDW3lWv+9ixuI+90J3A9rgsvthzjj9eEcHAME8ObDvAdVdfh2deLplL3sH9ykycr7yE4p9DhzCnp+P5l780eFtFURR7Wtdjaa2IvaeA61Jcbuap1Ufo5u/GE1dHklyUTKm5lL5+fQn6x9/RuzmQstsXa3jDxxQo2Ggr/pl0eQPSKIqinKcCQA0qngJuwGAdr246QXJeKa/dPABng564vDgAunt3x8Hbm5CxJsrzHMj6cEWD0iKtVgo3bcJ9wnj09kabUhRFuQQqANTAlJ6udUJWzw7DdiVk88nOs8wbE64N9ALE58UDWgAg9TAeXkl4TRxA9gcfUHr4cL3TUnrgAObMTDymqYe/FEVpPCoA1MCckYGDv3+9+sUpNVp4avURuvi68tepkRXz4/LiCHELwc3gpvX9I/QEvfA6DsFBpDy1AGtpab3SUrBxE8LJ6ZJHA1IURbFHBYAamNMz6l3+//rmWM5ml/DqTQNwdbwQMOLz4rWrfym1p3/Dx6IP6kKnl1/GeOYMmf/5T537lhYLhT/8gPuECY0yWLeiKMp5KgDUwJyeXq/y/31ncvhox2n+MKoro7tfKC6yWC2czj9ND+8ekHkCsk9V9P3jNno0PnPmkPPxJxTvrq2/vAvFP56q7x9FURqZCgA1MGVk1Gv81Pe3JeDv7sRT06uOdZtYmIjRatTuAI6tAwT0nlmxPPCJP2Po2oWUpxdQ8OOPNXYqV7BxE8LZGfcJEy7reBRFUapTAcAOa1kZ1vz8KuMA1ORIUj5juvvh7lS1rqCiAtiru1b+33kkeARXLNe5uhL6+utgMpP8yP9xasJE0l55hTJbNxKgFf8U/PAD7hMnNsq4oIqiKJWpB8HsMNuGgqyrCCizsJzU/DL6h17cL398vi0AWAWkR8PUVy5ax2XAAHr8soXi7dvJW/sNeV+sJPeTT3Hq3RvvG67HITAIS1YWnqr1j6IoTUAFADvM6edHArPfE+h50cn5AHYDQFxeHJ3cOuF66kdtRqXin8qEgwPuEybgPmEC5txcCr7/nvy135D+yr+05S4uuE9o+KhhiqIodVEBwA5TetXB4GtyJCkfIaCvvTuA8y2Ajq2DToPBu+7xjB18fPCdMwffOXMoiz1J/rpvcezcGZ2Ly6UdiKIoSi1UALCj4g6gjgBwNDmfbv5uF5X/m61mTuefZmzAYEhZAZP/3uA0OEf2wvmvf23wdoqiKPWlKoHtMGdkIFxd0dXR6+bR5DwGhHlfND+xMBGT1UR3k0WbEa6KcBRFaX1UALDDlJGOISBAG8GrBhkFZaQXlNOvhuIfgB75aVrf/50GN1laFUVRLlW9AoAQYpoQIlYIESeEWGBnuZcQ4jshxGEhRIwQYn5d2wohnhdCJAshDtle1zTOIV0+c3rNYwGfd9RWATwgzH4FMEBESjSEDQcHx8ZPpKIoymWqMwAIIfTAEmA60AeYLYToU221h4BjUsqBwETgDSGEYz22/beUcpDtVfPo6s2sPk8Bn68A7hNy8di88XnxhLqF4JoWDV3HNFUyFUVRLkt97gBGAHFSygQppRFYCcyqto4EPIRWZuIO5ADmem7bqkgpMdfjKeDo5Hx6BLjj5nRxPXpcXhzdHX1AWlUAUBSl1apPAAgFEit9TrLNq2wx0BtIAY4Cj0oprfXY9mEhxBEhxDIhhE9DE98ULHl5SJOpzqeAjyTn099O8Y/ZauZMwRm6m8ygc9CKgBRFUVqh+gQAezWhstrnqcAhoBMwCFgshPCsY9t3ge629VOBN+x+uRD3CSH2CSH2ZWZm1iO5l+fCQ2A13wGkF5SRWVhu9wGwc4XnMFvN9MjP0Cp/HVUPnoqitE71CQBJQOdKn8PQrvQrmw+skZo44DQQVdu2Usp0KaXFdqfwPlpx0UWklEullMOklMMCAmp/MrcxXHgGoOYAcCSp5grgij6AMk6p4h9FUVq1+gSAvUBPIUSEEMIRuB1YV22dc8AUACFEEBAJJNS2rRAipNL2NwDRl3MgjaU+g8EfTcpDJ6BPiP0WQAJBt/Iy6DrWztaKoiitQ51PAkspzUKIh4HNgB5YJqWMEUI8YFv+HvASsFwIcRSt2OcpKWUWgL1tbbt+TQgxCK1I6Axwf2Me2KUyZ2SCEDjUcrdxNDmfnoEeuDjqL1oWnxdPqIMbLhKtB1BFUZRWql5dQdiaaH5fbd57ld6nAFfXd1vb/D80KKXNxJyejt7PD2Ew2F0upeRocj4TetkvIorPi6e72QJB/cDFuwlTqiiKcnnUk8DVmDLSa+0FNK2gjKwio93yf5PVpLUAKshW5f+KorR6KgBUY07PwFBLE9DzFcD2moCeK7C1ACovUQFAUZRWTwWAaup6Cjg6OR+9Tth9Avh8FxDdjSYVABRFafVUAKjEajRiyc2tswloz0B3nA0XVwAn5CUggAiPruBe93jCiqIoLUkFgErMGdqDZoYaHgI7XwFs7wEwgLi8U4SZLbio5p+KorQBKgBUYs6ofSCYlPwycortVwADxGcdo3t5uWr/ryhKm6ACQCUXuoGwHwCOJuUB0N/OIDAmi4mzxan0MKnyf0VR2gYVACq58BSw/SKgo8n5OOgEUcEeFy07W3AWM1a6OXiBd2c7WyuKorQuKgBUYs7IRDg5ofOyX8RzJCmfXkEediuAz7cA6hE4oEnTqCiK0lhUAKjEnJ6OQ2Cg3aEg66oAjk/di05KIsInNnEqFUVRGocKAJVozwDYL/5Jyi0lr8Rk9wEwgPj0g4SZzThHTGy6BCqKojQiFQAqMWXU/BRwbWMAA8QVJdHdqgO/Hk2WPkVRlMakAoDN+aEga2oCejQ5H4NeEGmnAthkMXHOUkoPt05gp/hIURSlNVIBwMZaUIAsK6txJLCjSflEBnvg5HBxBfCZ5F1YBHQPUBXAiqK0HSoA2NTWBLTOCuDTPwHQI3xy0yVQURSlkakAYGNOzwDsPwWcmFNKfqmJ/qHedreNSzuATkrCI1QAUBSl7VABwKa2biCOJOcBNVcAxxeeo7Nwwsng2mTpUxRFaWwqANiYM2x3AHbqAI4m5+Oo19Er6OIKYIoyiZPldHcLuXiZoihKK6YCgI0pPR29tzc6R8eLlh1NyicqxANHh4tPl/HMVhINDnT3798cyVQURWk09QoAQohpQohYIUScEGKBneVeQojvhBCHhRAxQoj5dW0rhPAVQvwohDhlm/o0ziFdGnO6/Sag5yuA+9VQAXz69M9YhKBH2OimTqKiKEqjqjMACCH0wBJgOtAHmC2E6FNttYeAY1LKgcBE4A0hhGMd2y4AfpZS9gR+tn1uMTU9BXw2u4TCMjMDamoBlBMLQHffyCZNn6IoSmOrzx3ACCBOSpkgpTQCK4FZ1daRgIfQOtFxB3IAcx3bzgI+tr3/GLj+cg7kcpkyMzBUuwMwWaw8/10MOgHDI3ztbhdflokOiPCKaIZUKoqiNB6HeqwTCiRW+pwEjKy2zmJgHZACeAC3SSmtQojatg2SUqYCSClThRB2n8ASQtwH3Gf7WCSEiK1Hmu3xB7JqXeP33+Gf/7S7qMfC2nfuNM/pEpNVj3S1nNaaNpWuhmmt6YLWm7b2lq6u9mbWJwDY69tAVvs8FTgETAa6Az8KIbbVc9taSSmXAksbso09Qoh9Usphl7ufxtZa0wWtN20qXQ3TWtMFrTdtHSVd9SkCSgIqj3AShnalX9l8YI3UxAGngag6tk0XQoQA2KYZDU++oiiKcqnqEwD2Aj2FEBFCCEfgdrTinsrOAVMAhBBBQCSQUMe264C5tvdzgW8v50AURVGUhqmzCEhKaRZCPAxsBvTAMilljBDiAdvy94CXgOVCiKNoxT5PSSmzAOxta9v1QuBLIcQ9aAHklsY9tItcdjFSE2mt6YLWmzaVroZpremC1pu2DpEuIWWDiuQVRVGUdkI9CawoitJBqQCgKIrSQXWIAFBXVxbNmI7OQohfhBDHbV1mPGqb/7wQIlkIccj2uqYF0nZGCHHU9v37bPNatLsOIURkpXNySAhRIIR4rKXOlxBimRAiQwgRXWlejedICPG07TcXK4SY2szpel0IcUIIcUQIsVYI4W2bHy6EKK107t5r5nTV+Ldr4fO1qlKazgghDtnmN+f5qil/aLrfmJSyXb/QKp/jgW6AI3AY6NNCaQkBhtjeewAn0brIeB74SwufpzOAf7V5rwELbO8XAK+28N8xDe2BlhY5X8B4YAgQXdc5sv1dDwNOQITtN6hvxnRdDTjY3r9aKV3hlddrgfNl92/X0uer2vI3gGdb4HzVlD802W+sI9wB1Kcri2YhpUyVUh6wvS8EjqM9ad1atabuOqYA8VLKsy2VACnlVrRuTiqr6RzNAlZKKcullKeBOLTfYrOkS0r5g5TSbPu4C+0ZnGZVw/mqSYuer/Ns3dncCnzRFN9dm1ryhyb7jXWEAGCvO4oWz3SFEOHAYGC3bdbDttv1Zc1d1GIjgR+EEPuF1v0GVOuuA7A/YHLzuJ2q/5Qtfb7Oq+kctabf3d3AxkqfI4QQB4UQvwkhrmiB9Nj727WW83UFkC6lPFVpXrOfr2r5Q5P9xjpCALjs7igamxDCHVgNPCalLADeRetCYxCQinYL2tzGSimHoPXc+pAQYnwLpMEuoT1EeB3wlW1WazhfdWkVvzshxN/QOmZcYZuVCnSRUg4G/gx8LoTwbMYk1fS3axXnC5hN1QuNZj9fdvKHGle1M69B56wjBID6dGXRbIQQBrQ/7gop5RoAKWW6lNIipbQC79NEt761kVKm2KYZwFpbGlpLdx3TgQNSynRbGlv8fFVS0zlq8d+dEGIuMAOYI22Fxrbigmzb+/1o5ca9mitNtfztWsP5cgBuBFadn9fc58te/kAT/sY6QgCoT1cWzcJWvvghcFxK+Wal+ZXHk7wBiK6+bROny00I4XH+PVoFYjStp7uOKldlLX2+qqnpHK0DbhdCOAkhIoCewJ7mSpQQYhrwFHCdlLKk0vwAoY3TgRCimy1dCc2Yrpr+di16vmyuBE5IKZPOz2jO81VT/kBT/saao3a7pV/ANWg16vHA31owHePQbtGOoPWeesiWtk+Bo7b564CQZk5XN7TWBIeBmPPnCPBDG6znlG3q2wLnzBXIBrwqzWuR84UWhFIBE9rV1z21nSPgb7bfXCwwvZnTFYdWPnz+d/aebd2bbH/jw8ABYGYzp6vGv11Lni/b/OXAA9XWbc7zVVP+0GS/MdUVhKIoSgfVEYqAFEVRFDtUAFAURemgVABQFEXpoFQAUBRF6aBUAFAURemgVABQFEXpoFQAUBRF6aD+H4phRMGqPBCGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for key, df_scores in scores.items():\n",
    "    plt.plot(df_scores.num_iter, df_scores.val_auc, label=key)\n",
    "plt.ylim(0.8,0.84)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ced9b879",
   "metadata": {},
   "source": [
    "* ```max_depth = 3``` keeps growing the longest, keep this as baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "cce752e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "cc7d4994",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "xgb_params = {\n",
    "    \"eta\": 0.1,\n",
    "    \"max_depth\": 3,\n",
    "    \"min_child_weight\": 30,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=200,\n",
    "                  verbose_eval=5,\n",
    "                  evals=watchlist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "43cd78d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = f\"min_child_weight={xgb_params['min_child_weight']}\"\n",
    "scores[key] = parse_xgb_output(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "567670b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['min_child_weight=1', 'min_child_weight=10', 'min_child_weight=30'])"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "da898999",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f35a045fa60>"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABIQElEQVR4nO3deXxU1fn48c+TPSH7QgiBEPawI6sLIIggWkXQKqJFRFpqhW9tv+232k1t7a9Vi7VatZYKBRWruIPiiqCiyCZhS1hCwhISsieQPTNzfn/cAUPIMlknIc/79ZrXzNy5985zbyb3ufecc88RYwxKKaU6Hw93B6CUUso9NAEopVQnpQlAKaU6KU0ASinVSWkCUEqpTkoTgFJKdVIuJQARmSEiB0UkRUQeqOXzEBFZJyK7RWS/iCyo8bmniOwSkfeqTQsXkU9E5LDzOaz5m6OUUspVDSYAEfEEngWuBQYDc0VkcI3ZFgNJxpgRwGTgCRHxqfb5fUByjWUeADYYY/oDG5zvlVJKtRFXrgDGASnGmFRjTCXwKnBjjXkMECQiAgQC+YANQER6AN8DXqixzI3AKufrVcCspmyAUkqppvFyYZ5Y4ES19+nA+BrzPAOsBTKAIGCOMcbh/OzvwK+c06uLNsZkAhhjMkWka21fLiKLgEUAXbp0GZ2QkOBCyEoppc7auXNnrjEmquZ0VxKA1DKtZv8R1wCJwFVAX+ATEfkSmARkG2N2isjkxgR87ouMWQYsAxgzZozZsWNHU1ajlFKdlogcq226K0VA6UDPau97YJ3pV7cAeMtYUoA0IAG4ApgpIkexio6uEpGXnctkiUiMM7gYINvFbVFKKdUCXEkA24H+ItLbWbF7G1ZxT3XHgakAIhINDARSjTG/Nsb0MMbEO5f7zBjzA+cya4H5ztfzgXebtSVKKaUapcEiIGOMTUSWAB8BnsAKY8x+EbnH+fnzwCPAShHZi1VkdL8xJreBVT8KrBGRhVgJ5JZmbIdSSqlGko7UHbTWASilVOOJyE5jzJia0/VOYKWU6qQ0ASilVCelCUAppTopTQBKKdVJaQJQSqlOShOAUkp1UpoAlFKqk9IEoJRSnZQmAKWU6qQ0ASilVCelCUAppTopTQBKKdVJuTIgjFJKdXpnyqt4Z9dJPj+Uy+CYIC7tG8GouDD8vD3dHVqTaQJQSnUap4rKeXX7cY7nlTK+TzgT+kcRG+pf7zJ704tYvfUYa3dnUFppJzbUn88OZPH0Zyn4eHkwOi6My/tGcFnfCIb3CMXHq+MUrGgCUEpd1IwxfH0kj5e/OcbHSVk4jCEswIe3dp0EoE9UFyb2i2RC/ygu7RNOkJ83pZU21u3OYPXW4+xJL8LP24MbR8Ry+/g4hvcI4UyFje1p+Xx9JI8tR/J44pND8AkE+HgyoV8kP7i0FxP6ReLhUduIuu2HjgeglLooFZVW8ca36azeeozUnBLCAry5dUxPbh8fR1x4AIeyivnycA6bU3L5JjWP8ioHnh7CsNgQjmQXc6bCxoDoQO4Y34tZl8QS4u9d53cVlFSyNS2Pr1Ly+GBfJrnFlfSO7MIPLu3F90f3qHfZtlDXeACaAJRSF4XyKjuHs4pJzjzNtqP5vLcng/IqB6PiQvnBpb24blhMneX1FTY7O48VsPmwlQziwgO449JejOkVhkjjzuIrbHY+2HuKF7cc5dvjhfh7ezLrkljuvKwXg2KCW2JTG00TgFLqolFYWsnu9CKSM0+fexzJKcHusI5ngb5e3DAihjvG92JobIjb4tx3sogXtxzl3cQMKmwOxsWH0ysigLIqO2WVdkor7ZRW2SmrtFFaacfb04M7L+vF7ePj8PVqucrlZiUAEZkBPIU1JvALxphHa3weArwMxGHVKyw1xvxHRPyALwBf5/Q3jDEPOZd5GPgRkONczW+MMevri0MTgFKd1+myStYk7mVd8g4OFBwEqQAg0MebiEA/IgP96BrkR1SQH+EBfoT7hRHpH0mEfwQR/hFE+kfi71V/ha/NYaPSXom/l3+jz/zrU1hayes70lmz4wTFFTb8fTzx9/YkwMcTfx8vApyv0wvK2HY0nx5h/vxi+gBmjojFswXqEZqcAETEEzgETAPSge3AXGNMUrV5fgOEGGPuF5Eo4CDQDagCuhhjikXEG9gM3GeM+caZAIqNMUtd3QhNAEp1fOVVdr5JzWNbWj5hAT70igigd2QXeoYH4OnhoNRWSpmtjLyyPPblJrMxNZG9OQcosh9DPMudaxF8PHwRMRhjcOCwno0DQ93HtACvACL9I+ni3YVyezkVtgrr2V5Bha0Cm7EB0C+0H3cOvpPv9fkePp4+bbBXLMYYvjycy2MfHmB/xmkSugXxf9cM5KqErs1KSHUlAFdaAY0DUowxqc4VvQrcCCRVm8cAQWJFGAjkAzZjZZdi5zzezkfHKXNSSjXJqZJTbMnYwsnik5TZysgvK+ZYfgEni4rIKy3GTgXiUYlIFXhUIh6V1rM4LliXcfjgUdWdPkETmBQ/gqv7jmRA2AACvANq/W5jDDZjo6C8gLyyPHLLcskrdz6X5ZFXlkeJrQRfT1/8PP3w9XI+e/ri6+WLBx58fOxjHvz6QZ769iluS7iNOQPnEOYX1tq7DRFh0oAoJvSL5P29mTzx8UEWrtrB2Pgw/jBzKIO7t2wdgisJIBY4Ue19OjC+xjzPAGuBDCAImGOMccC5K4idQD/gWWPM1mrLLRGRO4EdwC+MMQVN2gqlVIszxpB4opAAHy/iwgPw96m7TLrKXsWu7F1sztjM5vTNHC487PxEEOOD3e4NDh+8xJfwoC5EdQknOigYHw8/bDYvKqu8KC33pLjcg6ISIb8YHLYAJvQaxpyRI5nQLwovT9fa14sI3uJN14CudA3o2qRtXzR8Ed9kfsOLSS/ybOKzvLD3BWb2ncm8wfPoHdK7SetsDA8P4YYR3ZkxtBuvbT/BPz47TAuWSJ3jShHQLcA1xpgfOt/PA8YZY/6n2jzfB64A/hfoC3wCjDDGnK42TyjwNvA/xph9IhIN5GJdETwCxBhj7q7l+xcBiwDi4uJGHzt2rOlbq5SqVZmtjKKKIooqiigsL+LzlGOs25/CqTMFgIDxINjPj8gu/kQFdqFrUADdggNwSCk7sreQWpxIlSkD44kpj6fi9ADsJQMxlV0Z0SOcqwd15aqEaAbFBLlclGGMadFy+KZKKUjhpeSXWHdkHVWOKi7vfjm9gnsR7BNMkE8QwT7B514H+QQRFRBFhF9Ei8ZeZXfg7WICrE1z6gAuAx42xlzjfP9rAGPMX6rN8z7wqDHmS+f7z4AHjDHbaqzrIaCkZrm/iMQD7xljhtYXi9YBKOWaM5VnOJB/gKS8JNKK0iipKqHMVka5rZwyW9m5cvYyWxklVSVU2Cua/F2OqlAcJQMJk+H0DRpB34hI4iMC6BXZhaHdQ4gK8m3BLXOf3LJcXjv4Gh+kfUBBeQFnKs/UWd8Q6htKv9B+9A3tS//Q/vQL60e/0H6E+LqnRVJz6gC2A/1FpDdwErgNuL3GPMeBqcCXzjP7gUCqs0K4yhhTKCL+wNXAY86AYowxmc7lZwP7mrBdSl30jDFUOapwGMe5Ss6zrwGqHFWkFqaSlJdkPfKTOHb6uyvlCL8IgnyC8Pfyx8/Lj0CfQKICovD38sffy58Ary6czBe+OVxGTpEn3YMiuH1sAjcM7UeoXzCCUOWoospRhc1ho8pRRVFZOccLzlBe4cmo7n3pERbgchFNRxXpH8nikYtZPHIxAA7joKSqhNOVpzlTeYbTFdZzZkkmKYUpHCk8wvup71NcVXzeOoJ9rH0qIniIBx7igfDda0/xxMvDC08P69lLvPDy8GLR8EUMjhjcotvUYAIwxthEZAnwEVYz0BXGmP0ico/z8+exinBWisheQID7jTG5IjIcWOWsB/AA1hhj3nOu+nERGYlVBHQU+HGLbplSHVS5rZz9eftJzE4kMSeR3dm7KahwrXospksMgyMGM7PvTAZHDGZwxGDC/cJrndfhMLyTeJKnNhzmWF4pCd2C+N0N/bl2aLcGuzDoGQRDm1a8ftHwEI9zxT51McaQVZpFSmEKKQUpHCk6QmlV6bkkXrMFk8M4sBkbdoedClsFpaYUm8OGzdiadZVWF70RTKlWVOWoYmfWTjanb6bcXk6AVwB+Xn7nzr7PPuzGzt7cvezO3k1SfhI2h9UcMT44nhFRI+gV3OvcGaKHWGfa1d/HBcUxKGJQnQf7mnYeK+CP6/azO72IwTHB3Hd1f6YNim73fdeopmlOEZBSqhFOV55mc/pmNp3YxOaTmzlTdQYfDx8CvAMos5XVeSbn6+nLkIghzB88n5FdRzI8arjLB3RXZRaV8dgHB3gnMYOuQb48ccsIZl8Sqwf+TkoTgFItIKc0h4+PfczGExvZeWonNmMj3C+cq3tdzeSek7k05tJz7dbtDjvl9vJzlbBltjKMMfQJ6YO3Z+t0GlZeZWfZF6n8c9MR7MaweEpf7p3cjy6+egjozPSvr1QzJOUl8VLSS3x49ENsDht9Q/oyf8h8JveczLDIYXh6XNh23tPDky4eXeji3aXV43M4DOv3ZfKX9Qc4WVjGtUO78ZvrBtEzvPabqFqdMVBeCEUnoTQXYkeDb91l6Kp1aQJQqpHsDjubTmzipeSX2Jm1kwCvAG4beBu3Dry1TW4Saogxhr0ni1i3O4P392SSUVTOoJhglt4ygsv6RrRFAFB0Ak5sg9zDcDrdOuCfPmk9V5V8N6+XPwy6AUbOhd5XQi0Js9FOZ0JmIsRPBN/A5q+vOlsFFKVDeB9a5c6sNqYJQCkXlVSV8Pbht1mdvJr04nS6d+nOL8f8kpv631RvS5C2YIwhOfMM7+3J4L09mRzPL8XbU5jUP4r7r03g+uHdm9apWFUZiAd4+tR9wLNVwqm9cGKr87ENzmR893lgNATHQtRA6He19TokFnyC4OD7sO9N2LsGgrrD8Fth5O3WvK5yOKwD/qEPrUfmbmt6lyiY9CsYfRd4NbE/H4cDsvZC6ueQugmOb4GqUug6GC79CQy7Fbz9mrbudkBbASnVgLSiNF47+BrvprxLcVUxI6NGMm/wPK6Kuwovj+adQ50ur+K/W4+z4UA2UYG+9Aj3p0dYAD3D/OkZHkBsqP95fdiXVNjIOVNBTnEFuc7nk4VlfJqUxZGcEjw9hMv7RnDD8O5cM6QbIQFNqFOoKoOD62HPGkj5FBw28PACn0CruMani/N1IFSVWwdfm7OTtpA46DkOeo63nrsOAq8GbgSrKrcO3Lv/C4c/AWOH7qNg4HUQEGYlCt9A5/c6X3v7W0nn0Idw6GMoPgWI9Z0DZljf+/UzcGwzhPaCq34HQ78PHg3cq2AM5KdC2ufWQT/tCyjLtz6LHAh9roSw3pC4GrL2QUAkjF0IY38Ige23XayOB6BUI9gddr5I/4L/HvgvWzK34OXhxbRe05g3aB7DooY1e/0ZhWX856s0/rvN6h54SPdgSivtnCwoo9J+fodoXYN88fP2JOdMBWVV9gvW5ekhjI0P4/rh3bl2aDciAptw563DYR0s97wGSWuh4rR1Rj70JvAPg8piqCiGyhKoPPPdaxGrHP/sQT+4e1N3iaU4G/a+AbtfsQ7wDfEJgn5TrYN+/+nQpVoRlzGQsgE2PGytK3ooTH0I+k/77mrm7AH/6Jdw9Cs4uvm7q5fgWKtYqs+V0HvS+dtmjJUcvvmnlYQ8va0Ec9m90K35v4+WpglAKRcUlBfw5uE3ef3g62SUZNA1oCu3DriVmwfcTKR/ZLPXvz+jiBe+TGPd7gwM8L1hMSya1OfcoCUOhyHrTDnpBWWcyC/lRH4ZJwpKqbI7iAr0JSrIl0jn89nX4V18mla8U1YIOQesA9ie162yep9AGHwjDJ8D8RNapky+qc4lnGKoOHPh67BeEHd5w8U7Dgfsfws+ewQKjkKvK2DQTEjfDse+gjPODgkCo63P4idYB/yIfq6V8+cdsRJB4mqreCh6GET0gdA46+ojLN56Du1pXbm4gSYApeqRWpjKin0r+CDtAyodlYzrNo65CXOZ3HOyS8U8xhjOVNgor7RTXuWg3GanvMr5uspOYVkVa7afYHNKLl18PLltXBwLroinR1grt8YxBkrzIOegdbCv/lx8yppHPK2z6OFzrGIXHze1EGpttkrY9SJsegxKsiGwm3Wwj7/CqjB29YBfl7IC+PZFq+io8Lj1qHnPR2A3GHEbTH6gTZOBJgClarE/bz8v7HmBDcc34Oflx8y+M5mbMJe+oX1dXkdJhY0lr3zLxoM59c7XNciXBVf05vZxcU0rm69NxRnY/w4UHrMO9KV5UJpf7XWeVYZ/lk+gVcEalWA9Rw60inACo1omno6gshRKcqwz9NZsyeNwQHGW9bcpPA4Fx+DUbkheZ9Uj3PCUVbzUBjQBKOVkjGFH1g5e2PsCX2d8TZBPELcn3M4dg+5o9KAfhaWVLFi5nd0nCrnnyr50d1ba+nl74Ofl+d1rb0/6Rwe23Div+amwdRnsetkqkxcP8A+HgAjno9rrLlEQNcA66AfHXhTNFzu01M9h3X1QkAaX/ACm/8mqZ2lF2hWE6vSMMXx58kv+veffJOYkEuEXwc9H/5xbB9xKoE/j24tnny5n3vJtpOWW8Nwdo5kxtFsrRF2NMVbrlG+et8rtPTxhyGwYf4/VaqahFi6qfehzJdy7BT5/DL562mrFdN3jMHhWmydnvQJQncKx08d46OuH2Jm1k+5durNg6AJm9ZuFn1fT2nAfzyvlB8u3kltcwb/vHMMV/VysILZVQt5hyE7+7pGTbN1c1CXK2Ua+h9VOPvjscyyc2gNb/wXZSVbTwzELYMxCCI5pUvyqncjcA2v/x2pKO/A6uG6p9Td3OKyiO0cV2KvAYbeavzbUpLYOWgSkOiWbw8ZLSS/xbOKz+Hj68PPRP2dWv1l4ezS9DP7gqTPMW76VCpuDlQvGcklcmPUPW3QcSvLOL38/+yjJhbwU62GcTTnF06p47DrIaiFSkmslgqJ0OJ1xYQVi9DC49B6ruWEHvvlI1WC3wdZ/wmf/z7qfQgTMhWMjc8eb0P/qJn2FFgGpTudwwWEe/OpB9uXt46qeV/G7S39HVEDzKjt3HS/grv9sx9fLgzU/voyB3YIgPw1ev8s6i6tJPL8ri4/oZ3V70HWQ9YjoV/cZnTFWQjjbjUKXSKudvZbfX3w8veDy/4GE6yHxFevg7+ltFfF5eDtfe1n1OC1ME4C66FTZq3hh3wss27OMYJ9g/nrlX7mm1zXNHqN18+FcFr20g8hAX1b/cLzVoVrye/DOvdYwSDMeg/DezspYZyWsX0jTDtoiVsucwCjofkmz4lYdRHhvuOq3bfqVmgDURWV/7n5+//XvOVxwmOt6X8cD4x5odMuemuwOw8qvj/LYBwfoHdmFlxaOo2sXT/jot7DlGYgZCbeusm74UaoD0QSgOjRjDKlFqWw8sZGNJzayN2cvUf5R/OOqfzC55+Rmr/9ITjG/emMPO48VMGVgFE/OGUloVQ6svBtOfGP1AXPNn5tcOaeUO2kCUB2OzWFjV/YuNp7YyKYTmzhx5gQAQyKGsHjkYuYOmkuwT3CzvsPuMLzwZSp/++QQvl4ePHHLCG4aFYsc+Qze+pHVgdnNy2HY91tgi5RyD5cSgIjMAJ7CGhT+BWPMozU+DwFeBuKc61xqjPmPiPgBXwC+zulvGGMeci4TDrwGxGMNCn+rMca1ka9Vp7Q/bz+vHXiNDcc3cLryNN4e3oyPGc9dQ+7iyh5XEt0lukW+53DWGf7vjT0knihk2uBo/t+soXT1KoWNf4Yv/mpV4N6yqlUq5ZRqSw0mABHxBJ4FpgHpwHYRWWuMSao222IgyRhzg4hEAQdFZDVQAVxljCkWEW9gs4h8YIz5BngA2GCMeVREHnC+v79lN091dFWOKj499imrk1ezO2c3/l7+XB13NVPipnB598tbdFQtm93Bv75I5alPDxPrU8KaScWMZROy+j6r61+AEbfD9564ePvLUZ2KK1cA44AUY0wqgIi8CtwIVE8ABggSq5lFIJAP2Ix1k0Gxcx5v5+PsjQc3ApOdr1cBm9AEoJxyy3J5/dDrvH7wdXLKcugZ1JNfjf0VN/a7sVnFO3aHIet0OScLyzhZUMbJwjIyC87gyD6MV94B+pTt5bMuKfSoTIVtWCNWxY2HKb+z7uDsMVabYqqLhisJIBY4Ue19OjC+xjzPAGuBDCAImGOMdSeD8wpiJ9APeNYYs9W5TLQxJhPAGJMpIrWOpiAii4BFAHFxca5sk+rA0s+k82zis+fG2L2i+xU8fPnDTIidgIc0vqsDu8OwJ72QTQdz2HQwm+KMQ/TlOAMknYEeJ5gqJ+nrkYk3VodpNl9/vHpeBr3usHqI7H5J00eTUqqdcyUB1Ha6U/P24WuAROAqoC/wiYh8aYw5bYyxAyNFJBR4W0SGGmP2uRqgMWYZsAysO4FdXU51PEeLjrLwo4UUVxVz64BbuS3htiaNsZtbXMEXh3LYdDCHLw/nUFBaxWUeSfwt4FX6+qScm68qOA6JHoJX9PetIf6iEvDqOsi68UapTsCVBJAO9Kz2vgfWmX51C4BHnUU+KSKSBiRgXUQDYIwpFJFNwAxgH5AlIjHOs/8YILvpm6E6urSiNBZ+tBC7sfPydS/TP6x/o9eRmlPML17fTeKJQoyByEAfbu1dwV2l/yEmcwME9IAJS62O06IG4t3SA4Yr1cG4kgC2A/1FpDdwErgNuL3GPMeBqcCXIhINDARSnRXCVc6Dvz9wNfCYc5m1wHzgUefzu83dGNUxVT/4L5++nH5h/Rq9jmN5Jdz+761U2R3879UDmNrLh0GHn0O2vwBefjD1Qbj0XreNyKRUe9RgAjDG2ERkCfARVjPQFcaY/SJyj/Pz54FHgJUisheryOh+Y0yuiAwHVjnrATyANcaY95yrfhRYIyILsRLILS29car9Sy1KZeFHC3EYByuuWdGogVjOSi8o5fZ/b6XCZufVhaMZePxVeOMxa7CUUXfClN+26wG7lXIX7Q1UuU1qYSoLP16IMYbl1yxv0sH/VFE5t/5rCyGlR3lhzEmij7xujfva9yqY/v8genDLB65UB6O9gap25UjhERZ+tBCAFdesoE9on0avI+94Eu+9/CzLKr4kQY7BDqweM697osnd5irVmWgCUG3uSOER7v7objzEg+XXLKdPiIsHf7sNcg/BoQ+w7X2LiOx9/BAo7joKRt0Dg2dag6kopVyiCUC1uKKKIg4XHCa/PL/Wx6H8Q/h6+dZ/8K8qh+z91ohJp/ZA5m7ISgJbGQCHPBN4xz6P6bcsYszw4W24dUpdPDQBqBZ1MO8Q8z9cSImt8Lzpob6hhPuFE+4XzsQeE/nJiJ8QHxJ/wfKlBzbg8fFv8C04jDhHzjJ+IdBtGDLmbsoih7DkK3++zPbn3/PHMGZA8wZ4Uaoz0wSgWsyGI7v4xZf3UmUTyk/Nx8MWxrhecdw4bAAzhnYn2K/2G6yO55XyaXIWnydn8Ej6PYgxvO24gf2OePabeNLLo/AoEvxTPRERyqvsPP+DUVypB3+lmkUTgGo2m93BXzZ8ymvpv0eMN0sS/sqEGYN5b08G7+3J5FcH9/G7d5KYPCCK60d0Z8rAKJIzz7DhQBYbkrNJyba6i7ovdDNxks3OCc8T3/VKoittXFppp6zKTlmlnVLn62uHdmNifz34K9Vc2gxUNcv+jCJ+9vZaMv2fxtezCy9Mf4FLYr67kcsYw64ThazbncH7ezLJPvPdQOfensL43hFcldCVqf2D6bV6IgTHwsKPtcM1pVqQNgNVLaq8ys7TGw7z720b8eu5ggi/UF65YSWxgbHnzScijIoLY1RcGL/73mC2H83nq5RcBsUEM7F/JEFni4W2PAenT8Ksf+rBX6k2oglANdq+k0X89L+7OFayn6D4lXTrEsnKa/9Dty7d6l3O00O4tE8El/aJOP+DimL48gnoPcnqclkp1SYa37+u6tQ+3n+KW57fQrEcJLTPSnoGd2PVtSsbPPjXa+vzUJoLVz3YcoEqpRqkCUC5xBjDis1p/PjlncTFZOOI/jc9grrznxn/ad5QjGWF8PXTMGAG9BzbYvEqpRqmRUCqQTa7g0feS2LVlmNMHxxJTvBysEWw4poVRPhHNLyC+nz9DygvsjpsU0q1Kb0CUPUqqbCx6KWdrNpyjEWT+nDVuKOkFB3ml2N/2fyDf3EOfPNPGDIbYvRuXqXaml4BqDqdKirn7pXbOZh1hj/NGsrMUWFc/9aPGR09mqvjWqCztc1PWl07TP5N89ellGo0TQCqVvszili4cgfFFTaWzx/D5IFdeXz74xRWFHL/2PuR5jbVPJ0B21+AEXMhakDLBK2UahRNAOoC2afLmfOvbwjy8+L1ey5jUEwwaUVp/Df5v9zU/yYGRQxq/pd88VcwDrjy/uavSynVJJoA1AVWbTlKSaWNtUuuoE+UNW7u0h1L8fXyZcklS5r/Bflp8O2LMGo+hPVq/vqUUk2ilcDqPGWVdlZvPc70wdHnDv5fnfyKL9K/4MfDf0ykf2Tzv+Tzx8DDCyb9X/PXpZRqMpeuAERkBvAU1pjALxhjHq3xeQjwMhDnXOdSY8x/RKQn8CLQDXAAy4wxTzmXeRj4EZDjXM1vjDHrm71Fqlne2pVOYWkVCydY/fRXOap4fPvj9AzqyR2D7mjcyoyBM6cgax+c2ms9svZZg7pctgSCY1phC5RSrmowATgHdH8WmAakA9tFZK0xJqnabIuBJGPMDSISBRwUkdWADfiFMeZbEQkCdorIJ9WWfdIYs7RFt0g1mcNhWL45jWGxIYyNDwNgzcE1pBal8tSUp/Dx9HFtRTtXwv634dQ+6w7fs0LioNswGHYrXHZvy2+AUqpRXLkCGAekGGNSAUTkVeBGoHoCMECQWE1DAoF8wGaMyQQyAYwxZ0QkGYitsaxqJz4/lENqTgl/nzMSEaGwvJDnEp9jfMx4pvSc4tpKNj8Jnz4MUYNg4AzoNhyih0L0EPAPbc3wlVKN5EoCiAVOVHufDoyvMc8zwFogAwgC5hhjHNVnEJF44BJga7XJS0TkTqzhvH9hjCmo+eUisghYBBAXF+dCuKqplm9Oo1uwH9cNs4pmntv9HMVVxfxq7K9ca/a55Tnr4D/0+3DTMvDwbN2AlVLN4kolcG3/+TUHEbgGSAS6AyOBZ0Qk+NwKRAKBN4GfGWNOOyf/E+jrnD8TeKK2LzfGLDPGjDHGjImK0kFAWsuBU6fZnJLLnZf3wsfLg5SCFNYcXMMtA25hQJgL7fS3vwAf/RoGzYTZ/9KDv1IdgCsJIB3oWe19D6wz/eoWAG8ZSwqQBiQAiIg31sF/tTHmrbMLGGOyjDF255XCv7GKmpSbLP8yDX9vT24fF4fdYefx7Y8T4B3A4pGLG17425fg/V9YHbrdvBw8tXWxUh2BKwlgO9BfRHqLiA9wG1ZxT3XHgakAIhINDARSnXUCy4FkY8zfqi8gItWbgMwG9jVtE1Rz5Zyp4N3EDG4eHcvJ0sPcvv52tmRuYfHIxYT5hdW/8J41sPZ/oO9VcMsq8HKxolgp5XYNnqoZY2wisgT4CKsZ6ApjzH4Rucf5+fPAI8BKEdmLVWR0vzEmV0QmAPOAvSKS6Fzl2eaej4vISKzipKPAj1t0y5TLXv7mGJWmhMqQN5j7/ttE+Efw+KTHmRE/o/4F978Nb/8Y4ifAnNXg7dc2ASulWoSOCdzJlVXauOwff0Ui3sPhUczchLksHrmYIJ+g+hc88D6suRNix8AP3gTfwLYJWCnVaDomsLrAkcIj/PST32OP3EvvwEE8NvkPrvXzk7wO3rgbYkbAHa/rwV+pDkoTQCe1Onk1S7cvxW73IbRsLm/Pux/PhlruGAOb/wYb/gixo60zf7/g+pdRSrVbmgA6oaNFR1m6fSkDQkaxbdt0fj/78oYP/lXlVmXv3jUw9Ga48Vnw9m+bgJVSrUITQCf05M4n8fH0wTPvNiL8hZkju9e/wJksePV2OLkDpvwOJv0SmjsegFLK7bQ30E5m+6ntfHbiM2b3mcfXhyq587Je+HrVc/afuRv+PQWyk+DWF+HK/9ODv1IXCU0AnYjDOPjr9r/SrUs38jMuxcfLgzvG19O9RtJaWOFsCnr3hzD4xrYJVCnVJjQBdCLrjqwjOT+ZuxLu5Z1vc7h5VA8iAn0vnNEYa8SuNfOg62D40UarxY9S6qKidQCdRGlVKU9/+zTDIodx6Eg/HOYE907ue+GMDges/yXsWG512zzzH3qDl1IXKb0C6CRW7V9Fdlk2Pxx8H69uT2f2JbH0DA84fyZ7Fbz1I+vgf8V9Vo+eevBX6qKlVwCdQFZJFv/Z/x+m95rO1/uDqLLnsnhKv/NnqiyF1++Cwx/B1Idg4v+6JValVNvRK4BO4B+7/oHNYWN+wr2s3nqcWSNjiY/s8t0M5UXw8s1w+GO4/kk9+CvVSegVwEUuKS+JtUfWctfQu1i/q4pym517q5/9F+fAyzdZzTy/v9y6yUsp1SloAriIGWNYumMpob6h3NJ3PjPe3c71w7vTr6uz757CE/DSLCg6CXNfhf7T3BqvUqptaRHQRWzjiY1sP7WdxSMXs2ZbLiWVdpacPfvPPWy18S/OgXlv68FfqU5IE8BFqspexd92/o0+IX24uudMVn51lGuHdmNgtyAoyYUXZ4G9Au56D3pd5u5wlVJuoEVAF6l/7fkXx04f47mpz/HylnTOVNhYclU/sNusrpxLcmDhRxAz3N2hKqXcRK8ALkIbjm/gX3v+xax+sxgRMZ4VX6Vx9aBohnQPgc8egbTP4fq/QfdL3B2qUsqNNAFcZI4UHuE3X/6GYZHD+N2lv+Olb45TVFbFT6f2g6R34au/w5i74ZIfuDtUpZSbuZQARGSGiBwUkRQReaCWz0NEZJ2I7BaR/SKywDm9p4hsFJFk5/T7qi0TLiKfiMhh53MDo4+rhpyuPM1PP/sp/l7+PDn5SWw2T5ZvTmPywCiG+2bBO/daQzjOeNTdoSql2oEGE4CIeALPAtcCg4G5IjK4xmyLgSRjzAhgMvCEiPgANuAXxphBwKXA4mrLPgBsMMb0BzY436smsjvs3P/F/WSUZPDklCeJ7hLN6q3HyC+p5L6JMfDaD6wBXG59Ebxq6QBOKdXpuHIFMA5IMcakGmMqgVeBmv0CGyBIRAQIBPIBmzEm0xjzLYAx5gyQDMQ6l7kRWOV8vQqY1ZwN6eyeSXyGzSc38+txv+aSrpdQVmln2RdpTOgbwSU7fw15R+D7/4GQ2IZXppTqFFxJALHAiWrv0/nuIH7WM8AgIAPYC9xnjHFUn0FE4oFLgK3OSdHGmEwA53PX2r5cRBaJyA4R2ZGTk+NCuJ3PR0c/4oW9L3Bz/5u5deCtAPzriyPkFlfw5+gN1iDu0/4AvSe6OVKlVHviSgKobfgnU+P9NUAi0B0YCTwjIudGCxeRQOBN4GfGmNONCdAYs8wYM8YYMyYqKqoxi3YKB/MP8vuvfs+IqBH8ZvxvAPj2eAH/+CyF/+ufSdyupTBkNly2xM2RKqXaG1cSQDrQs9r7Hlhn+tUtAN4ylhQgDUgAEBFvrIP/amPMW9WWyRKRGOc8MUB20zah8yqqKOK+jfcR6B3Ik5OtcX6LK2z87NVERgSd5ie5f4bIATDzGR3GUSl1AVcSwHagv4j0dlbs3gasrTHPcWAqgIhEAwOBVGedwHIg2RjztxrLrAXmO1/PB95t2iZ0TsYYfvXFr8guzebJKU8SFWBdHT28dj/lBRm84vMXPBw2mLMafAPdHK1Sqj1qMAEYY2zAEuAjrErcNcaY/SJyj4jc45ztEeByEdmL1aLnfmNMLnAFMA+4SkQSnY/rnMs8CkwTkcPANOd75aJd2bv4OuNr/nf0/zIiyhqu8f09mXyy8wDvh/0Nv/JsuON1iOzXwJqUUp2VS11BGGPWA+trTHu+2usMYHoty22m9joEjDF5OK8aVOO9nfI2AV4B3NT/JgAyCst45K2trAl8gsiK43D7Gogb7+YolVLtmd4J3AGVVpXy0dGPmNF7BgHeAdgdhvtf28rfzeMMsKcgt6yEvlPcHaZSqp3TBNABfXT0I8psZczuNxuA5Z8f5K70hxkvScjs5yHhe26OUCnVEWgC6IDeSXmH+OB4RkSNYN+JfLpv/BlTPXfB956A4be6OzylVAehCaCDOVp0lG+zv2VWv1mUV9o59uIirvfYQtmVDyJjF7o7PKVUB6IJoIN598i7eIonM/vOZMcLi/le1SecGHov/lN+4e7QlFIdjCaADsTusLM2ZS1XxF5B+e5NTMx5le1RN9Pz5j+7OzSlVAekCaAD+Trja7LLspkdN53gDQ9wwMTR/86n9S5fpVSTaALoQN5OeZsw3zDG7f6QwKo8Pk94kNAgvctXKdU0mgA6iILyAjae2Mj3IkYSvO8VXrB/j2umXdfwgkopVQdNAB3E+rT12Bw2ZiVv4hjd2Nv/J8RHdnF3WEqpDkwTQAfx9uG3GewVwsD8Y9xf+UPuurLmoGxKKdU4mgA6gOS8ZA4WHGRW1lHWel1DWezljO6lQygrpZrHpc7glHu9fehNfAxMtQUwtfgW/jKzN6Itf5RSzaRXAO1chb2C91PeZmpJCSt87iE4NIIZQ7q5Oyyl1EVAE0A7t3Hfy5x2VHJ14HCePzWQBVfE4+WpfzalVPPpkaQ9c9h559vn6GZ3sIElBPl6MWdsz4aXU0opF2gCaMdOffUEX0sF06Mm8MbBSuaOjyPIz9vdYSmlLhKaANorh521+1ZhRMi1zwHgrsvj3RuTUuqi4lICEJEZInJQRFJE5IFaPg8RkXUisltE9ovIgmqfrRCRbBHZV2OZh0XkZC1jBSuAI5/xuUcVwwJ68v6uKr43LIbuof7ujkopdRFpMAGIiCfwLHAtMBiYKyI170JaDCQZY0YAk4EnRMTH+dlKYEYdq3/SGDPS+VhfxzydUuW3qzjg64OP1yUUV9j40cQ+7g5JKXWRceUKYByQYoxJNcZUAq8CN9aYxwBBYjVODwTyARuAMeYL53vlqpJcDqZtoFKE/UfDGd87nGE9QtwdlVLqIuNKAogFTlR7n+6cVt0zwCAgA9gL3GeMcbiw7iUissdZTFTrra0iskhEdojIjpycHBdWeRHY8xp7fKw/TW5eN36oZ/9KqVbgSgKo7ZZTU+P9NUAi0B0YCTwjIsENrPefQF/n/JnAE7XNZIxZZowZY4wZExUV5UK4HZwx8O1L7A2NwcuEEhcSw9SEru6OSil1EXIlAaQD1Ruf98A6069uAfCWsaQAaUBCfSs1xmQZY+zOK4V/YxU1qZPfQk4yu/38KC+OZfYlPfDw0G4flFItz5UEsB3oLyK9nRW7twFra8xzHJgKICLRwEAgtb6VikhMtbezgX11zdup7HqRQp8A0quKsJf15LphMQ0vo5RSTdBgAjDG2IAlwEdAMrDGGLNfRO4RkXucsz0CXC4ie4ENwP3GmFwAEfkvsAUYKCLpIrLQuczjIrJXRPYAU4Cft+iWdUSVpbD3Tfb2mwRAtO8A+nfVEb+UUq3Dpd5AnU0019eY9ny11xnA9DqWnVvH9Hmuh9lJJL0LlWfYHtYLU5rM9QPGaq+fSqlWo3cCtye7XoLwPmwqzMJREc3MEdr6RynVejQBtBd5R+DYV5iRd3C85AABpjcJ3YLcHZVS6iKmCaC92PUyiAd7e1yBXUoZFT1Ci3+UUq1KE0B7YLdB4ivQbxqvHTkIwOzBl7s5KKXUxU4TQHtwZAMUn4JR8/g6fRc4fJnWb5i7o1JKXeQ0AbQH374IXaIo7HEV2RWHiPbth5enDteslGpdmgDcrTgbDn0Iw+fwfnIm4pfBuO4j3R2VUqoT0ATgbrtfBYcNRt3JO/u3IeJgau+x7o5KKdUJaAJwJ2Os1j89xlEU2Ie9uXsBGB413M2BKaU6A00A7pS+A3IPwiU/4NOkLPA9ToRvNFEBnaDXU6WU22kCcKf9b4OnDwyZzQf7MvHpks7obiPcHZVSqpPQBOAuxkDyOugzhdP488WRoxivfC3+UUq1GU0A7nJqDxQdh0HX81lyNnafowAMi9T2/0qptqEJwF2S3wPxgIHXsX5vJiGhmXiKJ4MiBrk7MqVUJ6EJwF2S10Hc5RR7hbLpUA6hYZkMCBuAv5e/uyNTSnUSmgDcITcFcpJh0A18diCbSpuNYpOmxT9KqTalCcAdDqyznhO+x/o9mUSGFVFmL9EKYKVUm9IE4A7J70H3Syjxj2HjwWyG9C4EYFiUXgEopdqOSwlARGaIyEERSRGRB2r5PERE1onIbhHZLyILqn22QkSyRWRfjWXCReQTETnsfA5r/uZ0AKcz4OQOSLieTQdzqLA5CAzJIMg7iPjgeHdHp5TqRBpMACLiCTwLXAsMBuaKyOAasy0GkowxI4DJwBMi4uP8bCUwo5ZVPwBsMMb0xxpI/oLEclE68L71PGgmH+zLJDLQh6yKwwyNHIqH6AWZUqrtuNLn8DggxRiTCiAirwI3AknV5jFAkFhDWAUC+YANwBjzhYjE17LeG7GSBcAqYBNwf6O3oKNJXguRAygP7cvGA59w3YgIPi48xA+H/dDdkal2oqqqivT0dMrLy90diupg/Pz86NGjB97e3i7N70oCiAVOVHufDoyvMc8zwFogAwgC5hhjHA2sN9oYkwlgjMkUka61zSQii4BFAHFxcS6E246V5sPRr2DCz/jycC4llXYSehXx4X4HwyO1AlhZ0tPTCQoKIj4+XocFVS4zxpCXl0d6ejq9e/d2aRlXyhxq+wWaGu+vARKB7sBI4BkRCXYpggYYY5YZY8YYY8ZERXXwTtIOfgDGDgnX88G+TIL9vDA+xwGtAFbfKS8vJyIiQg/+qlFEhIiIiEZdObqSANKBntXe98A6069uAfCWsaQAaUBCA+vNEpEYAOdztmshd2AH3oPgHlR2HcGnSVlcPTiapPx99AjsQbhfuLujU+2IHvxVUzT2d+NKAtgO9BeR3s6K3duwinuqOw5MdQYQDQwEUhtY71pgvvP1fOBdV4PukCqKIWUDDLqeLWn5nC63ce3QGPbk7NGzf6WUWzSYAIwxNmAJ8BGQDKwxxuwXkXtE5B7nbI8Al4vIXqwWPfcbY3IBROS/wBZgoIiki8hC5zKPAtNE5DAwzfn+4pXyKdgrYNANfLgvky4+niTEOsgqzdLyf6WUW7jU7tAYs94YM8AY09cY8/+c0543xjzvfJ1hjJlujBlmjBlqjHm52rJzjTExxhhvY0wPY8xy5/Q8Y8xUY0x/53N+a2xgu5G8DgIisPe4lI/3ZzEloSuHCq2GVHoFoDqitWvX8uijLXfeNnnyZHbs2HHB9B07dvDTn/4UgJUrV7JkyZJalw8MDGyxWKq77rrrKCwsrHeeumJPTExk/fr1jfq+u+++m65duzJ06NBGLdcU2vC8Ldgq4fDHMPA6th0rIq+kkmuHxrA/bz+e4snAsIHujlCpRps5cyYPPND6t++MGTOGp59+utW/py7r168nNDS0Scs2JQHcddddfPjhh036vsZypRmoaq60L6Di9LniH18vDyYPjGLdF0n0De2Ln5efuyNU7dQf1u0nKeN0i65zcPdgHrphSL3zHD16lBkzZjBhwgS++eYbRowYwYIFC3jooYfIzs5m9erVJCUlsWPHDp555hnuuusugoOD2bFjB6dOneLxxx/n+9//fp3rf/zxx3nppZfw8PDg2muvPXcl8frrr3PvvfdSWFjI8uXLmThxIps2bWLp0qW89957560jLS2N22+/HZvNxowZtd1r+p17772XGTNmMHPmTGbPnk1YWBgrVqxg+fLlpKWl8ac//YmXX36Zp59+msrKSsaPH89zzz2Hp6cn8fHx7Nixg8jISB555BFWr15Nz549iYyMZPTo0fzyl7+sNfbx48fz4IMPUlZWxubNm/n1r3/NnDlzGvz7TJo0iaNHjzY4X0vQK4C2kLwWfAJxxE/iw/2nuHJAFAE+niTlJTE4ouZN1Uq1DykpKdx3333s2bOHAwcO8Morr7B582aWLl3Kn//85wvmz8zMZPPmzbz33nv1Xhl88MEHvPPOO2zdupXdu3fzq1/96txnNpuNbdu28fe//50//OEP9cZ333338ZOf/ITt27fTrVu3euedNGkSX375JQAnT54kKckqft28eTMTJ04kOTmZ1157ja+++orExEQ8PT1ZvXr1eevYsWMHb775Jrt27eKtt966oMinZuw+Pj788Y9/ZM6cOSQmJjJnzhw2btzIyJEjL3hcfvnl9cbfWvQKoLU57HBwPfSfzq7McrJOV3DtsG6cKjlFQUWBJgBVr4bO1FtT7969GTbMqp8aMmQIU6dORUQYNmxYrWeos2bNwsPDg8GDB5OVlVXnej/99FMWLFhAQEAAAOHh3zWBvummmwAYPXp0g2fBX331FW+++SYA8+bN4/776+5IYOLEifz9738nKSmJwYMHU1BQQGZmJlu2bOHpp59m1apV7Ny5k7FjxwJQVlZG167n35u6efNmbrzxRvz9rTE7brjhhvM+dyX2KVOmkJiYWO92tSVNAK3txFYoyYFB1/Phvky8PYWrEqLZkf0FgCYA1W75+vqee+3h4XHuvYeHBzabrd75jal5ryjnfVZXe/Wz6/D09Kz1O2pytd17bGwsBQUFfPjhh0yaNIn8/HzWrFlDYGAgQUFBGGOYP38+f/nLX+qNuz6uxL5x40Z+/vOfXzA9ICCAr7/+2qVtaUlaBNTakt8DTx9Mv2l8sO8UV/SLJMTfm6T8JK0AVp3S9OnTWbFiBaWlpQDk5zetAeAVV1zBq6++CnBBcU1tLrvsMv7+978zadIkJk6cyNKlS5k4cSIAU6dO5Y033iA7O/tcTMeOHTtv+QkTJrBu3TrKy8spLi7m/fffb/A7g4KCOHPmzLn3Z68Aaj7ccfAHTQCt78A66DOF/XmG9IIyrh1qlVUm5SXRJ7SPVgCrTudsZeyYMWMYOXIkS5cubdJ6nnrqKZ599lnGjh1LUVFRg/NPnDgRm81Gv379GDVqFPn5+ecSwODBg/nTn/7E9OnTGT58ONOmTSMzM/O85ceOHcvMmTMZMWIEN910E2PGjCEkJKTe75wyZQpJSUmMHDmS1157zaXtmjt3LpdddhkHDx6kR48eLF++3KXlmkIauqxpT8aMGWNqa2vbbpUVwGPxMP1P/PXMNJ7/PJXtv72asABvJq+ZzMTYifxpwp/cHaVqZ5KTkxk0aJC7w1C1KC4uJjAwkNLSUiZNmsSyZcsYNWqUu8M6T22/HxHZaYwZU3NerQNoTQVHATBh8Xyw5RTje4cT3sWHUyWnyC/P1/J/pTqYRYsWkZSURHl5OfPnz293B//G0gTQmpwJ4JijK6k5uSy4PB6win9AK4DVxW3v3r3MmzfvvGm+vr5s3bq1w37nK6+80iLraS80AbSmAqsS6YMTPojANUO+K//3EA8GhmsFsLp4DRs2rM2bPLrjOzsyrQRuTQVHISCCdw+cYXRcGF2DrQrfpLwk+oT0wd/L373xKaU6NU0ArangKBVBcRw4dYYZztY/xhi9A1gp1S5oAmhNBUc57rBGMTubALJLs8krz9MEoJRyO00ArcVhh6ITJBaHMrxHCD3CrNvez1YAD4lw3y3+SikFmgBaz+mT4LCx43QI0wdHn5uclK8VwOrioOMBfKctxgPIz89n2rRp9O/fn2nTplFQUNCo9dZGWwG1FmcT0OOmK7f2jTg3WSuAVaN88ACc2tuy6+w2DK5t/oF75syZzJw5swUCqt+YMWMYM+aCe5jaTGMP4NUlJiayY8cOrrvuOpeXueuuu1iyZAl33nnnedMfffRRpk6dygMPPMCjjz7Ko48+ymOPPdbk2ECvAFqPMwFkeUQzLDb03GStAFYdwdGjR0lISOCHP/whQ4cO5Y477uDTTz/liiuuoH///mzbtu28s/G77rqLn/70p1x++eX06dOHN954o971P/744wwbNowRI0ac13X066+/zrhx4xgwYMC57ps3bdrE9ddff8E60tLSuOyyyxg7diy///3v6/2+e++9l7VrraHMZ8+ezd133w3A8uXL+d3vfgfAyy+/zLhx4xg5ciQ//vGPsdvtAMTHx5ObmwvAI488QkJCAtOmTWPu3LnndWNRM/bKykoefPBBXnvttUZ1BTFp0qTzekg9691332X+fGsY9fnz5/POO++4tL76uHQFICIzgKcAT+AFY8yjNT4PAV4G4pzrXGqM+U99y4rIw8CPgBznan5jjGl6qm1vCo5ix4NuPfvi42Xl2ezSbHLLchkUrrf5Kxe1wJl6U6WkpPD666+zbNkyxo4de248gLVr1/LnP/+ZWbNmnTf/2fEADhw4wMyZM+scEKb6eAABAQHndQZ3tk/99evX84c//IFPP/20zvjOjgdw55138uyzz9a7LWfHA5g5cyYnT54818/P5s2bue22284bD8Db25t7772X1atXn3cWXn08AJvNxqhRoxg9enS9sf/xj388N2gONK830KysLGJiYgCIiYk513FdczSYAETEE3gWa+D2dGC7iKw1xiRVm20xkGSMuUFEooCDIrIasDew7JPGmKb1BNXOVeWmkmEiGdPnuz7F9Q5g1ZHoeAA6HgDAOCDFGJMKICKvAjcC1ROAAYLE6pw7EMgHbMB4F5a9KJVlp3Lc0ZXxvb/7cSflJSEICeEJboxMKdfoeAAXxl2f1h4PIDo6mszMTGJiYsjMzLwgQTWFK3UAscCJau/TndOqewYYBGQAe4H7jDEOF5ZdIiJ7RGSFiIQ1Nvj2zLPoOOlEc0lc6LlpSXlJ9A7pTYB3gPsCU8rNdDyApo0HMHPmTFatWgXAqlWruPHGGxtcpiGuJIDaUmzNVHgNkAh0B0YCz4hIcAPL/hPo65w/E3ii1i8XWSQiO0RkR05OTm2ztD8VZ+hiK8AW3JMAn+8usrQCWCkdD6AhdY0H8MADD/DJJ5/Qv39/Pvnkk3rHXXaZMabeB3AZ8FG1978Gfl1jnveBidXef4ZVdNTgss7p8cC+hmIZPXq06QjKT+w25qFg89ZLT5+bll2SbYauHGpe3P+iGyNTHUFSUpK7Q1B1OHPmjDHGmJKSEjN69Gizc+dON0d0odp+P8AOU8sx1ZUrgO1AfxHpLSI+wG3A2hrzHAemAohINDAQSK1vWRGJqbb8bGCfaymr/Us7vB+AHn2+a+2jFcBKdXyLFi1i5MiRjBo1iptvvvniHw/AGGMTkSXAR1hNOVcYY/aLyD3Oz58HHgFWisherGKf+40xuQC1Letc9eMiMhKrSOgo8OOW3DB3yj5+kARgYMLwc9OS8q0KYG0CqjoLHQ+g/XPpPgBjtc9fX2Pa89VeZwDTXV3WOX1eLbNfFMqyj1AsXQgOizo3LSkvifiQeK0AVp2GjgfQ/umdwC2syu7Ar/gEZ/y6Q7UmaloBrJRqbzQBtLB9J4uINVlIWPy5ablluWSXZjM4XBOAUqr90ATQwral5tJTcgju3v/cNK0AVkq1R9obaAs7lJKCr1RBdN9z087eATwoQiuAlVLth14BtCC7w5CTfsh6Exp/bnpSXhK9gnvRxbuLewJTqhXoeADfaanxAMrLyxk3bhwjRoxgyJAhPPTQQ+c+0/EA2rmDp84QWZkBPkC1OoCkvCRGR4+uczml6vLYtsc4kH+gRdeZEJ7A/ePq7jjNVToeQMMaOx6Ar68vn332GYGBgVRVVTFhwgSuvfZaLr30Uh0PoL3blpZHnEc2BoHQngDkleWRVZql5f+qQ9HxANwzHoCInLuSqaqqoqqq6lyHd24bD0C5ZvvRAq73yUcCY8HL6hlQK4BVc7TEmXpT6XgA7hkPwG63M3r0aFJSUli8eDHjx48H3DQegHKNMYatafn8wjcPwnqdm342AegdwKqj0fEA3DMegKenJ4mJiRQWFjJ79mz27dt3wfjALUUTQAtJyy0ht7iCbiFZEFatCwhnBXCgT+tUUCnVWnQ8gAvjrk9LjwcQGhrK5MmT+fDDDxk6dKjbxgNQLtiWlo8vlQRUZJ9fAZyfpDeAKVWNjgdQ93gAOTk551oclZWV8emnn5KQYA0g5a7xAJQLtqXlM6xLofXGmQBOV57mVMkpBoYPdFtcSrU3Oh5A3TIzM5kyZQrDhw9n7NixTJs27VwFeGuMByANXda0J2PGjDG1tbVtD6549DPuCE/m3ozfwMJPoOc4ErMTmffBPP5x1T+Y3HOyu0NUHURycjKDBmmdUXtUXFxMYGAgpaWlTJo0iWXLlrW7LqFr+/2IyE5jzAVtabUOoAWcLCzjZGEZo3qdtiaEWpXAaUVpAPQJ6eOu0JRSLWjRokUkJSVRXl7O/Pnz293Bv7E0AbSA7WlWGWZ/n1zw8odAq3ImtSgVHw8fYgNrDqGs1MVPxwNo/zQBtICtafkE+3kRXplhlf87WyakFqXSK6QXnh6e7g1QdTj1tZTpKHQ8gLbX2CJ9rQRuAdvS8hgbH44UHDuvBVBqYaoW/6hG8/PzIy8vr9H/zKpzM8aQl5eHn5+fy8voFUAz5RZXcCSnhFtH94CvjkJvq1VBua2ck8UnuaHvDfWvQKkaevToQXp6Ojk5Oe4ORXUwfn5+9OjRw+X5NQE009ny/8tigKqSc1cAx04fw2D0CkA1mre3N71793Z3GKoTcKkISERmiMhBEUkRkQsan4pIiIisE5HdIrJfRBY0tKyIhIvIJyJy2Pkc1jKb1La2Hc3H39uTQX7Om1mcCSC1KBWA3iH6j6yUap8aTAAi4gk8C1wLDAbmikjNW1sXA0nGmBHAZOAJEfFpYNkHgA3GmP7ABuf7Diclu5gB0YF4nz5uTaiWADzEg/iQeLfFppRS9XHlCmAckGKMSTXGVAKvAjXvQTZAkFjNFgKBfMDWwLI3Aqucr1cBs5qzIe5yPL+UnuEBUGC1+Sc0DrDuAYgNjMXX07eepZVSyn1cqQOIBU5Ue58OjK8xzzPAWiADCALmGGMcIlLfstHGmEwAY0ymiNTas5GILAIWOd8Wi8hBF2KuTSSQ28Rl6/UF1mUOAH84v9M3ocGmfK0WVwtor7FpXI3TXuOC9hvbxRZXr9omupIAajuC1Wyfdg2QCFwF9AU+EZEvXVy2XsaYZcCyxixTGxHZUdut0O7WXuOC9hubxtU47TUuaL+xdZa4XCkCSgd6VnvfA+tMv7oFwFvGkgKkAQkNLJslIjEAzufmj26glFLKZa4kgO1AfxHpLSI+wG1YxT3VHQemAohINDAQSG1g2bXAfOfr+cC7zdkQpZRSjdNgEZAxxiYiS4CPAE9ghTFmv4jc4/z8eeARYKWI7MUq9rnfGJMLUNuyzlU/CqwRkYVYCeSWlt20CzS7GKmVtNe4oP3GpnE1TnuNC9pvbJ0irg7VHbRSSqmWo30BKaVUJ6UJQCmlOqlOkQAa6sqiDePoKSIbRSTZ2WXGfc7pD4vISRFJdD6uc0NsR0Vkr/P7dzinubW7DhEZWG2fJIrIaRH5mbv2l4isEJFsEdlXbVqd+0hEfu38zR0UkWvaOK6/isgBEdkjIm+LSKhzeryIlFXbd8+3cVx1/u3cvL9eqxbTURFJdE5vy/1V1/Gh9X5jxpiL+oFV+XwE6AP4ALuBwW6KJQYY5XwdBBzC6iLjYeCXbt5PR4HIGtMeBx5wvn4AeMzNf8dTWDe0uGV/AZOAUcC+hvaR8++6G/AFejt/g55tGNd0wMv5+rFqccVXn88N+6vWv52791eNz58AHnTD/qrr+NBqv7HOcAXgSlcWbcIYk2mM+db5+gyQjHWndXvVnrrrmAocMcYcc1cAxpgvsLo5qa6ufXQj8KoxpsIYkwakYP0W2yQuY8zHxhib8+03WPfgtKk69ldd3Lq/znJ2Z3Mr8N/W+O761HN8aLXfWGdIALV1R+H2g66IxAOXAGfHqlvivFxf0dZFLU4G+FhEdorV/QbU6K4DqLW7jjZyG+f/U7p7f51V1z5qT7+7u4EPqr3vLSK7RORzEZnohnhq+9u1l/01EcgyxhyuNq3N91eN40Or/cY6QwJodncULU1EAoE3gZ8ZY04D/8TqQmMkkIl1CdrWrjDGjMLquXWxiExyQwy1EusmwpnA685J7WF/NaRd/O5E5LdYHTOudk7KBOKMMZcA/wu8IiLBbRhSXX+7drG/gLmcf6LR5vurluNDnbPWMq1R+6wzJABXurJoMyLijfXHXW2MeQvAGJNljLEbYxzAv2mlS9/6GGMynM/ZwNvOGNpLdx3XAt8aY7KcMbp9f1VT1z5y++9OROYD1wN3GGehsbO4IM/5eidWufGAtoqpnr9de9hfXsBNwGtnp7X1/qrt+EAr/sY6QwJwpSuLNuEsX1wOJBtj/lZteky12WYD+2ou28pxdRGRoLOvsSoQ99F+uus476zM3furhrr20VrgNhHxFZHeQH9gW1sFJSIzgPuBmcaY0mrTo8QapwMR6eOMK7UN46rrb+fW/eV0NXDAGJN+dkJb7q+6jg+05m+sLWq33f0ArsOqUT8C/NaNcUzAukTbg9V7aqIztpeAvc7pa4GYNo6rD1Zrgt3A/rP7CIjAGqznsPM53A37LADIA0KqTXPL/sJKQplAFdbZ18L69hHwW+dv7iBwbRvHlYJVPnz2d/a8c96bnX/j3cC3wA1tHFedfzt37i/n9JXAPTXmbcv9VdfxodV+Y9oVhFJKdVKdoQhIKaVULTQBKKVUJ6UJQCmlOilNAEop1UlpAlBKqU5KE4BSSnVSmgCUUqqT+v8hJwDEjOGDhwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for key, df_scores in scores.items():\n",
    "    plt.plot(df_scores.num_iter, df_scores.val_auc, label=key)\n",
    "plt.ylim(0.8,0.84)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c34993f",
   "metadata": {},
   "source": [
    "* The values are very close, we will stay with ```min_child_weight=1```\n",
    "* Now we will train this model and choose the number of iteratins to 175"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "id": "31f9695b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "xgb_params = {\n",
    "    \"eta\": 0.1,\n",
    "    \"max_depth\": 3,\n",
    "    \"min_child_weight\": 30,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=175)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b76fa455",
   "metadata": {},
   "source": [
    "* Documentation: https://xgboost.readthedocs.io/en/stable/\n",
    "* Other useful parameters:\n",
    "    * ```subsample``` (How many rows for each tree) and ```colsample_bytree``` (How many features for each tree)\n",
    "        * Both between 0 and 1\n",
    "    * ```lambda``` and ```alpha```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "462d4ca8",
   "metadata": {},
   "source": [
    "## Selecting the final Model\n",
    "* Choosing between XGBoost, Random Forest and Decision Tree\n",
    "* Training the final model\n",
    "* Saving the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "id": "7ae86913",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The best decision tree model\n",
    "dt = DecisionTreeClassifier(max_depth=6, min_samples_leaf=15)\n",
    "dt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "id": "4cce95be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7850954203095104"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = dt.predict_proba(X_val)[:,1]\n",
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "7118d25f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=10, min_samples_leaf=3, n_estimators=200,\n",
       "                       random_state=1)"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The best random forest\n",
    "rf = RandomForestClassifier(n_estimators=200, \n",
    "                            max_depth=10,\n",
    "                            min_samples_leaf=3,\n",
    "                            random_state=1)\n",
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "ea8024b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8246258264512848"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = rf.predict_proba(X_val)[:,1]\n",
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "2e5bcc0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The best XGBoost\n",
    "xgb_params = {\n",
    "    \"eta\": 0.1,\n",
    "    \"max_depth\": 3,\n",
    "    \"min_child_weight\": 30,\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"eval_metric\": \"auc\",\n",
    "    \"nthread\": 8,\n",
    "    \"seed\": 1,\n",
    "    \"verbosity\": 1\n",
    "}\n",
    "\n",
    "model = xgb.train(xgb_params, dtrain, \n",
    "                  num_boost_round=175)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "id": "0e4a9181",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8354877576227264"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(dval)\n",
    "roc_auc_score(y_val, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18317cf8",
   "metadata": {},
   "source": [
    "* XGBoost is the best model\n",
    "* Now train the final model and use the entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "id": "7dd3435a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_full_train = df_full_train.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "id": "d239bc6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, ..., 0, 0, 1])"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_full_train = (df_full_train[\"status\"] == \"default\").astype(int).values\n",
    "y_full_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "id": "8b3ec842",
   "metadata": {},
   "outputs": [],
   "source": [
    "del df_full_train[\"status\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "id": "4546e116",
   "metadata": {},
   "outputs": [],
   "source": [
    "dicts_full_train = df_full_train.to_dict(orient=\"records\")\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "X_full_train = dv.fit_transform(dicts_full_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "id": "2072b8a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "dicts_test = df_test.to_dict(orient=\"records\")\n",
    "X_test = dv.transform(dicts_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "id": "d360e0c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfull_train = xgb.DMatrix(X_full_train, label=y_full_train, feature_names=dv.get_feature_names_out())\n",
    "dtest = xgb.DMatrix(X_test, feature_names=dv.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "id": "9d7dbda8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = xgb.train(xgb_params, dfull_train, \n",
    "                  num_boost_round=175)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "id": "d5d09d28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8302929964322369"
      ]
     },
     "execution_count": 263,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(dtest)\n",
    "roc_auc_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c38898c",
   "metadata": {},
   "source": [
    "* The auc score is similar to the auc score on the train set\n",
    "* Our model did not overfit\n",
    "* XGBoost is often very good on this type of data (tabular data), downsie: more difficult to tune, because it  has more paramters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf41aae",
   "metadata": {},
   "source": [
    "## Summary\n",
    "* Decision Trees learn if-then-else rules from the data\n",
    "* Finding the best split: select the least impure split. This algorithm can overfit, that's why we control it by limiting the max depth and the size of the group\n",
    "* Random Forests are a way of combining multiple decision trees. It shoud have a diverse set of models to make good predictions\n",
    "* Gradient boosting trains models sequentially, each model tries to fix the errors of the previous one. XGBoost is an implementation of gradient boosting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ec9e630",
   "metadata": {},
   "source": [
    "## Explore More\n",
    "\n",
    "* For this dataset we didn't do EDA or feature engineering. You can do it to get more insights into the problem.\n",
    "* For random forest, there are more parameters that we can tune. Check max_features and bootstrap.\n",
    "* There's a variation of random forest caled \"extremely randomized trees\", or \"extra trees\". Instead of selecting the best split among all possible thresholds, it selects a few thresholds randomly and picks the best one among them. Because of that extra trees never overfit. In Scikit-Learn, they are implemented in ExtraTreesClassifier. Try it for this project.\n",
    "* XGBoost can deal with NAs - we don't have to do fillna for it. Check if not filling NA's help improve performance.\n",
    "* Experiment with other XGBoost parameters: subsample and colsample_bytree.\n",
    "* When selecting the best split, decision trees find the most useful features. This information can be used for understanding which features are more important than otheres. See example here for random forest (it's the same for plain decision trees) and for xgboost\n",
    "* Trees can also be used for solving the regression problems: check DecisionTreeRegressor, RandomForestRegressor and the objective=reg:squarederror parameter for XGBoost.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d3dca9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml-zoomcamp",
   "language": "python",
   "name": "ml-zoomcamp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}