lectures/23_optional_sgd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# (Optional)<br>Lecture 23: Stochastic Gradient Descent (SGD)\n",
    "\n",
    "UBC 2022 Summer\n",
    "\n",
    "Instructor: Mehrdad Oveisi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.linear_model import Ridge, LogisticRegression, SGDClassifier, SGDRegressor\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For large datasets: `SGDClassifier` and `SGDRegressor`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note*** \n",
    "Although we speak of very large datasets here, we do *not* mean what has come to be known as *Big Data*. Some say the realm of Big Data starts with at least *1 terabyte* of data! We do not discuss the requirements for handling such datasets here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Something we haven't discussed in this course is huge data sets.\n",
    "- There could be a couple of problems arising from huge data sets:\n",
    "  1. The code is too slow.\n",
    "  2. The dataset doesn't fit in memory - we can't even load it with `pd.read_csv`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simplest strategy: subset your data for experimentation / hyperparameter tuning, then train your final model on the whole dataset (once)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- \"SGD\" (stochastic gradient descent) can help with both of these problems.\n",
    "- But we'll focus on using it to solve problem (1).\n",
    "- There is a fancy way to implement `fit` that can be a lot faster for large datasets.\n",
    "  - You can **think of it as quickly finding \"approximately\" the best coefficients** when calling `fit`.\n",
    "  - That is not quite true but it may be a useful way of thinking.\n",
    "  - Much more on this in CPSC 340 and much, much more on this in CPSC 440/540."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SGD can be used in many contexts.\n",
    "- In sklearn, it's built in as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier, SGDRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDRegressor` is basically equivalent to `Ridge`.\n",
    "- `SGDRegressor(loss='huber')` is basically equivalent to `HuberRegressor`.\n",
    "- `SGDClassifier(loss='log')` is basically equivalent to `LogisticRegression`, except the parameter is called `alpha` instead of `C` (like `Ridge`).\n",
    "- With other settings they are equivalent to other models, but this is good enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For **regular sized** datasets, use the **original functions**, as these ones can be a bit more finicky. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the [Sentiment140 dataset](http://help.sentiment140.com/home), which contains tweets labeled with sentiment associated with a brand, product, or topic. It is a large dataset. You can download the data from [here](https://www.kaggle.com/ferno2/training1600000processednoemoticoncsv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df = pd.read_csv('data/training.1600000.processed.noemoticon.csv', \n",
    "                        encoding = \"ISO-8859-1\",\n",
    "                        names=[\"label\", \"id\", \"date\", \"no_query\", \"name\", \"text\"])\n",
    "tweets_df['label'] = tweets_df['label'].map({0: 'neg', 4: 'pos'})\n",
    "tweets_df = tweets_df[tweets_df['label'].str.startswith(('pos','neg'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df_train, tweets_df_test = train_test_split(tweets_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>no_query</th>\n",
       "      <th>name</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>638487</th>\n",
       "      <td>neg</td>\n",
       "      <td>2234471696</td>\n",
       "      <td>Thu Jun 18 23:07:06 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>courtlandmeans</td>\n",
       "      <td>I just want this whole feeling to go away! You...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1321400</th>\n",
       "      <td>pos</td>\n",
       "      <td>2014685283</td>\n",
       "      <td>Wed Jun 03 02:20:50 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>sashapurse</td>\n",
       "      <td>@rowi_e hehe it's ok..same here, basics lang a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>913666</th>\n",
       "      <td>pos</td>\n",
       "      <td>1752673597</td>\n",
       "      <td>Sat May 09 22:36:42 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>sxetravis</td>\n",
       "      <td>had a great day</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>478075</th>\n",
       "      <td>neg</td>\n",
       "      <td>2178427301</td>\n",
       "      <td>Mon Jun 15 07:41:09 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>kiminikrikket</td>\n",
       "      <td>I miss you all!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1441884</th>\n",
       "      <td>pos</td>\n",
       "      <td>2061892445</td>\n",
       "      <td>Sat Jun 06 21:51:09 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>DavidBass</td>\n",
       "      <td>@ginamcnew I think she saw in my face that des...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        label          id                          date  no_query  \\\n",
       "638487    neg  2234471696  Thu Jun 18 23:07:06 PDT 2009  NO_QUERY   \n",
       "1321400   pos  2014685283  Wed Jun 03 02:20:50 PDT 2009  NO_QUERY   \n",
       "913666    pos  1752673597  Sat May 09 22:36:42 PDT 2009  NO_QUERY   \n",
       "478075    neg  2178427301  Mon Jun 15 07:41:09 PDT 2009  NO_QUERY   \n",
       "1441884   pos  2061892445  Sat Jun 06 21:51:09 PDT 2009  NO_QUERY   \n",
       "\n",
       "                   name                                               text  \n",
       "638487   courtlandmeans  I just want this whole feeling to go away! You...  \n",
       "1321400      sashapurse  @rowi_e hehe it's ok..same here, basics lang a...  \n",
       "913666        sxetravis                                   had a great day   \n",
       "478075    kiminikrikket                                   I miss you all!   \n",
       "1441884       DavidBass  @ginamcnew I think she saw in my face that des...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 6)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holy cow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = CountVectorizer(stop_words='english')\n",
    "\n",
    "X_train = vec.fit_transform(tweets_df_train['text']) \n",
    "y_train = tweets_df_train['label']\n",
    "\n",
    "X_test = vec.transform(tweets_df_test['text']) \n",
    "y_test = tweets_df_test['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 563972)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse._csr.csr_matrix"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the **fraction of elements that are nonzero**, showing that having a sparse matrix really helps!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2338687322538471e-05"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.nnz / np.prod(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train a classifier. I'll use `%time` instead of `%timeit` because I want to keep the output, and it gets lost with `%timeit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5002633333333333"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dc = DummyClassifier()\n",
    "dc.fit(X_train, y_train)\n",
    "dc.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 47s, sys: 2.97 s, total: 1min 50s\n",
      "Wall time: 30.6 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `LogisticRegression`, the hyperparameter `max_iter` is set to 100 by default. We could increase it to get rid of this warning, but it would then take even longer to run `fit()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8108816666666666"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7764"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([100], dtype=int32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.n_iter_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try to use `SGDClassifier` on the same data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_sgd = SGDClassifier(loss=\"log\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.23 s, sys: 40.3 ms, total: 4.27 s\n",
      "Wall time: 3.66 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SGDClassifier(loss='log')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time lr_sgd.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.n_iter_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7695691666666666"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7667625"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDClassifier` was **about 10x faster** than `LogisticRegression`, but the accuracy is slightly lower.\n",
    "- In fact, we can control the speed vs. _train_ accuracy tradeoff in both cases using the hyperparameters.\n",
    "  - The main ones are `max_iter` (higher is slower) and/or `tol` (lower is slower)\n",
    "  - (This is the same for both `LogisticRegression` and `SGDClassifier`)\n",
    "  - This is analogous to `n_iter` in `RandomizedSearchCV` !\n",
    "- In general, `LogisticRegression` will get slightly higher _train_ accuracy (may or may not correspond to better validation/test)\n",
    "- But in some cases your **dataset is so big** that `LogisticRegression` is **not feasible**, and then `SGDClassifier` can save the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 4 candidates, totalling 8 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n",
      "/home/moveisi/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "# from https://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_early_stopping.html#sphx-glr-auto-examples-linear-model-plot-sgd-early-stopping-py\n",
    "\n",
    "# from sklearn.utils._testing import ignore_warnings\n",
    "# from sklearn.exceptions import ConvergenceWarning\n",
    "\n",
    "# @ignore_warnings(category=ConvergenceWarning)\n",
    "def fit_grid_search_lr(iters):\n",
    "    grid_search_lr = GridSearchCV(\n",
    "        LogisticRegression(), \n",
    "        {\"max_iter\" : iters}, return_train_score=True, verbose=2, cv=2, n_jobs=-1)\n",
    "    # n_jobs=-1 might mess up the timing a bit but otherwise it takes too long\n",
    "    grid_search_lr.fit(X_train, y_train)\n",
    "    return grid_search_lr\n",
    "\n",
    "grid_search_lr = fit_grid_search_lr([3, 10, 30, 100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_max_iter</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.009968</td>\n",
       "      <td>0.742118</td>\n",
       "      <td>4.775916</td>\n",
       "      <td>0.341908</td>\n",
       "      <td>3</td>\n",
       "      <td>{'max_iter': 3}</td>\n",
       "      <td>0.687295</td>\n",
       "      <td>0.687427</td>\n",
       "      <td>0.687361</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>4</td>\n",
       "      <td>0.687573</td>\n",
       "      <td>0.688147</td>\n",
       "      <td>0.687860</td>\n",
       "      <td>0.000287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11.685373</td>\n",
       "      <td>1.088875</td>\n",
       "      <td>3.881189</td>\n",
       "      <td>0.089727</td>\n",
       "      <td>10</td>\n",
       "      <td>{'max_iter': 10}</td>\n",
       "      <td>0.754632</td>\n",
       "      <td>0.753495</td>\n",
       "      <td>0.754063</td>\n",
       "      <td>0.000568</td>\n",
       "      <td>3</td>\n",
       "      <td>0.757448</td>\n",
       "      <td>0.756968</td>\n",
       "      <td>0.757208</td>\n",
       "      <td>0.000240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.471042</td>\n",
       "      <td>0.636646</td>\n",
       "      <td>2.475081</td>\n",
       "      <td>0.002605</td>\n",
       "      <td>30</td>\n",
       "      <td>{'max_iter': 30}</td>\n",
       "      <td>0.770763</td>\n",
       "      <td>0.771062</td>\n",
       "      <td>0.770913</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>2</td>\n",
       "      <td>0.786268</td>\n",
       "      <td>0.784358</td>\n",
       "      <td>0.785313</td>\n",
       "      <td>0.000955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33.336141</td>\n",
       "      <td>0.401457</td>\n",
       "      <td>2.271428</td>\n",
       "      <td>0.067878</td>\n",
       "      <td>100</td>\n",
       "      <td>{'max_iter': 100}</td>\n",
       "      <td>0.772603</td>\n",
       "      <td>0.770943</td>\n",
       "      <td>0.771773</td>\n",
       "      <td>0.000830</td>\n",
       "      <td>1</td>\n",
       "      <td>0.835227</td>\n",
       "      <td>0.832345</td>\n",
       "      <td>0.833786</td>\n",
       "      <td>0.001441</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       8.009968      0.742118         4.775916        0.341908   \n",
       "1      11.685373      1.088875         3.881189        0.089727   \n",
       "2      19.471042      0.636646         2.475081        0.002605   \n",
       "3      33.336141      0.401457         2.271428        0.067878   \n",
       "\n",
       "  param_max_iter             params  split0_test_score  split1_test_score  \\\n",
       "0              3    {'max_iter': 3}           0.687295           0.687427   \n",
       "1             10   {'max_iter': 10}           0.754632           0.753495   \n",
       "2             30   {'max_iter': 30}           0.770763           0.771062   \n",
       "3            100  {'max_iter': 100}           0.772603           0.770943   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  split0_train_score  \\\n",
       "0         0.687361        0.000066                4            0.687573   \n",
       "1         0.754063        0.000568                3            0.757448   \n",
       "2         0.770913        0.000149                2            0.786268   \n",
       "3         0.771773        0.000830                1            0.835227   \n",
       "\n",
       "   split1_train_score  mean_train_score  std_train_score  \n",
       "0            0.688147          0.687860         0.000287  \n",
       "1            0.756968          0.757208         0.000240  \n",
       "2            0.784358          0.785313         0.000955  \n",
       "3            0.832345          0.833786         0.001441  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_lr = pd.DataFrame(grid_search_lr.cv_results_)\n",
    "results_lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 3 candidates, totalling 6 fits\n"
     ]
    }
   ],
   "source": [
    "grid_search_sgd = GridSearchCV(\n",
    "    SGDClassifier(loss=\"log\", max_iter=100_000),\n",
    "    {\"tol\" : [1e-3, 1e-4, 1e-5]},\n",
    "    return_train_score=True, verbose=2, cv=2, n_jobs=-1)\n",
    "\n",
    "grid_search_sgd.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_tol</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.184362</td>\n",
       "      <td>0.053181</td>\n",
       "      <td>3.849718</td>\n",
       "      <td>0.054271</td>\n",
       "      <td>0.001</td>\n",
       "      <td>{'tol': 0.001}</td>\n",
       "      <td>0.765213</td>\n",
       "      <td>0.764945</td>\n",
       "      <td>0.765079</td>\n",
       "      <td>0.000134</td>\n",
       "      <td>3</td>\n",
       "      <td>0.772183</td>\n",
       "      <td>0.772630</td>\n",
       "      <td>0.772407</td>\n",
       "      <td>0.000223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.374817</td>\n",
       "      <td>0.242586</td>\n",
       "      <td>5.378076</td>\n",
       "      <td>0.235454</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>{'tol': 0.0001}</td>\n",
       "      <td>0.765213</td>\n",
       "      <td>0.765028</td>\n",
       "      <td>0.765121</td>\n",
       "      <td>0.000092</td>\n",
       "      <td>2</td>\n",
       "      <td>0.772127</td>\n",
       "      <td>0.772695</td>\n",
       "      <td>0.772411</td>\n",
       "      <td>0.000284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.963195</td>\n",
       "      <td>0.089307</td>\n",
       "      <td>1.887105</td>\n",
       "      <td>0.060743</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>{'tol': 1e-05}</td>\n",
       "      <td>0.765393</td>\n",
       "      <td>0.765207</td>\n",
       "      <td>0.765300</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>1</td>\n",
       "      <td>0.772352</td>\n",
       "      <td>0.772847</td>\n",
       "      <td>0.772599</td>\n",
       "      <td>0.000248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_tol  \\\n",
       "0       5.184362      0.053181         3.849718        0.054271     0.001   \n",
       "1       8.374817      0.242586         5.378076        0.235454    0.0001   \n",
       "2      19.963195      0.089307         1.887105        0.060743   0.00001   \n",
       "\n",
       "            params  split0_test_score  split1_test_score  mean_test_score  \\\n",
       "0   {'tol': 0.001}           0.765213           0.764945         0.765079   \n",
       "1  {'tol': 0.0001}           0.765213           0.765028         0.765121   \n",
       "2   {'tol': 1e-05}           0.765393           0.765207         0.765300   \n",
       "\n",
       "   std_test_score  rank_test_score  split0_train_score  split1_train_score  \\\n",
       "0        0.000134                3            0.772183            0.772630   \n",
       "1        0.000092                2            0.772127            0.772695   \n",
       "2        0.000093                1            0.772352            0.772847   \n",
       "\n",
       "   mean_train_score  std_train_score  \n",
       "0          0.772407         0.000223  \n",
       "1          0.772411         0.000284  \n",
       "2          0.772599         0.000248  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_sgd = pd.DataFrame(grid_search_sgd.cv_results_)\n",
    "results_sgd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEQCAYAAACJLbLdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABB8UlEQVR4nO3dd3xUVfr48c+THkpCCZBQkwDSi0gJVUARQcQCqCigKK5td9l111V0/YG4tl0X3V13/aqISLGLooIKKCiEAAaRJk1IoZOE0AJpk/P7405iEibJhMxkJpnn7Suvmdx77r3PcGOenHNPEWMMSimllLv4eToApZRStZsmGqWUUm6liUYppZRbaaJRSinlVppolFJKuVWApwPwRhERESY6OtrTYSilVI2yefPmdGNMk9LbNdE4EB0dTWJioqfDUEqpGkVEUhxt16YzpZRSbqWJRimllFtpolFKKeVWmmiUUkq5lSYapZRSbqWJRimllFtpolFKKeVWmmiUUsrHGWP4Ifkk//h6t1vOrwM2lVLKR+XmF/DFtiPMi09ix+EzhIcGMqV/NM3CQlx6HU00SinlY9LP5fDOxlQWbkgh7WwObZvU5ZmbunLT5S2oE+T6tKCJRimlfMTPR87wVnwSS7ceITe/gKEdmjB1YAyD20Xg5yduu64mGqWUqsVsBYZvdh3nrfhkEg5kEBrozy29W3LXgBjaNa1XLTFoolFKqVrobHYeHyYeYv76ZFJPnqd5eAgzRnXktj6tCa8TWK2xaKJRSqlaJDXjPPPXJ/NB4kHO5eRzRZuGPHptR0Z2aUaAv2c6GmuiUUqpGs4Yw4YDJ5kXn8SqXcfxF2FM9yimDoyhR6sGng5PE41SStVU2Xk2Ptt6hHnrkth97CyN6gbx0NB2TO7fxuVdlKtCE41SStUwJ85ms2hDKos3pJCRlUuHZvV5YVw3bujZgpBAf0+HdxFNNEopVUPsOHyaeeuS+HzbEfILDFd1bMrUgTEMaNsYEfd1T64qTTRKKeXF8m0FrPzZ6p68KfkkdYL8uaNfG+4cEE1MRF1Ph+cUTTRKKeWFTl/I44MfDjJ/fTKHT12gZcNQ/npdJyb0bkV4aPV2T64qTTRKKeVFDqSdY/76ZD7afIjzuTb6xjTiyTGdGdG5Gf5uHL3vTppolFLKw4wxxP+Swbz4JL7dfYIgfz+u79GcqQOj6doi3NPhVZkmGqWU8pDsPBufbDnMW/FJ7D1+joh6QUy/qj13xLWmaX3v6Z5cVZpolFKqmh07nc3CDcm8szGVzPN5dI4K48UJPbi+RxTBAd7XPbmqNNEopVQ1+engKeatS2L59qPYjOGazs24e2AMfWMaeXX35KrSRKOUUm6UZyvgqx3HeCs+iR9TT1E/OIA7B0RzZ/9oWjeu4+nwqoUmGqWUcoNT53N5d9NBFiQkc/R0Nm0a12HW9Z0Z37sV9YJ961evb31apZRys19OnGVefDJLfjxEdl4BA9o25ukbujKsY9Ma2z25qjTRKKVUFRUUGL7fl8a8+GS+35tGUIAfN/VswV0Do+kUFebp8DxOE41SSl2i87n5fPyj1T35QFoWTesH86cRl3F7v9Y0rhfs6fC8hiYapZSqpMOnLrAgIZl3N6ZyJjuf7i3DefnWnozuFkVQgGcWF/NmmmiUUsoJxhh+TM1k3rpkvtp5DGMMo7pGMXVgNFe0aViruydXlSYapZQqR25+Acu3H+Wt+CS2HjpNWEgA0wbFMLl/G1o29I3uyVWliUYppRw4mZXLOxtTWJCQwomzOcQ2qcvTN3ZlXK8W1AnSX52Vof9aSilVzJ5jZ3krPolPthwmJ7+Awe0jeGF8d65s3wQ/H+2eXFWaaJRSPq+gwLB6zwnmxScR/0sGIYF+jLuiJVMHRNO+WX1Ph1fjaaJRSvmsczn5fJRoLS6WnHGeyLAQ/nJtByb2aU3DukGeDq/W0ESjlPI5B0+e5+31ybz/w0HO5uRzeesG/OmaDlzbNZJAf+2e7GqaaJRSPsEYw6akk8yLT2Llz8fxE2F0N6t78uWtG3o6vFpNE41SqlbLybfx+Vare/LOI2doUCeQ+69sy+T+bYgKD/V0eD5BE41SqlZKO5vD4o0pLNqQSvq5HNo3rcdzN3fjxp4tCA2qfYuLeTNNNEqpWmXnkdO8FZ/MZz8dIddWwLAOTbh7UAyD2kXo6H0P0USjlKrxbAWGVbuOM29dEhuTThIa6M9tfVtx54Bo2jap5+nwfJ4mGqVUjXUmO48PfjjI2wnJHDx5gRYNQnl8dEdu7d2a8DqBng5P2WmiUUrVOMnpWcxfn8yHiQfJyrXRJ7ohj4/qxIjOzQjQ7sleRxONUqpGMMaQsD+DefFJfLP7BAF+wvXdmzN1YAzdWoZ7OjxVDk00Simvlp1n47OfjjAvPondx87SqG4QvxvWjklxbWgaFuLp8JQTNNEopbzS8TPZLNqQwuKNqZzMyqVjZH3+Pr47Y3s0JyRQuyfXJJpolFJeZduhU8xbl8Sy7UfJLzBc1bEZdw+Kpn9sY+2eXEM5nWhEpC5wDzAEaAz8xhizT0RuA34yxux28jytgJeAEYAAq4A/GGNSnTi2NfA0MAyIAA4BHwDPGWOyipXzAx4F7gMigT3AbGPMx05+XKVUNcq3FbDiZ6t7cmJKJnWD/JkU14a7BkTTpnFdT4enqsipRGNPDmuAlsBuoCtQOHf2MOBqYJoT56kDfAvkAHcCBvgbsFpEuhdPFg6OrYuVlAKBJ4FUoA/wFNAeuLVY8aeBPwNPAJuB24APRWSMMWa5M59ZKeV+p8/n8d4PqSxISOHwqQu0blSHJ8d0ZkLvloSFaPfk2sLZGs0/sZJDe+AIkFts33fALCfPcy8QC3QwxvwCICLbgH1YtY855Rw70H79kcaYFfZtq0WkEfBnEaljjDkvIk2xkszzxpgXi5VrBzwPaKJRysP2p51jfnwyH20+xIU8G3GxjZh5fWeu6tQMf11crNZxNtGMwGoqSxWR0k/hDgMtnDzPWGBDYZIBMMYkiUg8cAPlJ5rCxSHOlNp+CvDDaoYDGGkvu6hUuUXAPBGJMcYkORmvUspFjDGs3ZfOvPgk1uxJI8jfjxt6NueugdF0aa7dk2szZxNNEHC2jH3hQJ6T5+kCLHWwfScwoYJjV2HVfF4QkQewms76AtOB/yvW7NYFq/b1S6njd9pfOwOaaJSqJhdybXyy5TBvxSex78Q5IuoF88erL+P2fq1pUj/Y0+GpauBsotkGjAO+crBvFNZzEGc0AjIdbD8JlLsghDEmW0QGAR/za9IAmAv8ttQ1ThljjINrFO6/iIj8BvgNQOvWrcsLRSnlhKOnL7AgIYV3N6Vy6nweXZqHMeeWHlzXPYrgAO2e7EucTTT/AD6ydy18x76ts4jcgNUTbWwlrlk6AcCvzV5lEpEQ4H2gKTCZX2s0/w/IBx4odq5KX8MY8zrwOkDv3r0dHa+UcsKPqZm8FZ/M8u1HMcZwTedI7h4UQ5/ohto92Uc5lWiMMUtE5EGsh+l32zcvwGpO+60xxlFNx5FMHNcoGuK4plPcPcBQoJ0xZr992/cichp4XUT+zxizFXvtSESkVK2msMZ0EqWUS+XZCvhyxzHmrUvip4OnqB8cwN0Do5nSP5pWjep4OjzlYc52bw4H3gIWAv2xahUZwHpjTFnPbhzZifUMpbTOwM8VHNsNyCyWZAptsr92ArbarxEMtKXkc5rO9teKrqOUclJmVi7vbEplYUIKx85kExNRl6fGdmHcFS2pF6zjwZWlwp8EEQnASio3GWM+x3oof6k+A14UkVhjzAH7+aOxui4/VsGxx7BqKu2K91oD+tlfD9tfv8Lqfn0H1hibQpOAHdrjTKmq23v8LG/FJ/PJlkNk5xUwqF0Ez97claGXNcVPuyerUipMNMaYfBE5DthccL03sB7cLxWRv2I9S3kaOAi8VlhIRNoA+7FG88+2b54PPAwsF5FnsJ7R9MYavLkZiLfHe0JEXgJmiMhZ4EeswZzDsbpQK6UuQUGB4bu9acyLT2LtvnSCA/y4uVcL7hoQQ4fI+hWfQPksZ+u2i7BG/ldpsKMxJktEhmNNQbMQ6wH9N1hT0JwrVlQAf6zxMYXHJotIHNbg0L9hTUFzEOsB/jPGmIJixz8BnMPq+lw4Bc0t9hqZUqoSsnLyWfLjId6KT+ZAehbNwoJ5ZGQHJvZtTaO6QRWfQPk8ubgXsINC1riVx7Gar5YCRynVs8sYM88dAXpC7969TWJioqfDUMqjDmWeL+qefDY7nx4tw7l7UAyjukYRFKCLi6mLichmY0zv0tudrdH81/7aArjCwX4D1JpEo5SvMsaQmJLJW/FJfLXjGCLCtV0juXtgDL1aN9DuyeqSOJtoYtwahVLKo3LzC1i2/Qjz1iWz/fBpwkMD+c2Qtkzp34bmDUI9HZ6q4ZwdR5Pi7kCUUtUv/VwO72xMZeGGFNLO5tC2SV3+dmNXbu7VgjpB2j1ZuUalfpJEpCtwJdagywzge2PMDncEppRyn11Hz/BWfBKf/nSE3PwCrrysCXdPiGFwuwjtnqxcztkBmwFY3YsnUnIqFyMi7wB3GWNc0f1ZKeUmtgLDt7tPMG9dEgkHMggN9OeW3i25a0AM7ZrW83R4qhZztkYzE7gFa16xRVi9zyKxBkHOBA7YX5VSXuZsdh4fJh7i7YRkUjLOExUewmOjOnJbn1Y0qKPdk5X7OZtoJgFPG2OeKbYtBXjGvj7NVDTRKOVVUjPOM399Mh8kHuRcTj5XtGnIIyM7MLJLJIH+2j1ZVR9nE01zIKGMfeuxBkgqpTzMGMPGpJPMW5fEyl3H8Rfhuu5RTB0YQ89WDTwdnvJRziaaI1jzkTma52yAfb9SykOy82x8vvUI8+KT2XX0DA3rBPLQ0HZMimtDZHiIp8NTPs7ZRLMYeEJECuzvj2I9o7kNqzbzgnvCU0qV58TZbBZtSOWdjSmkn8ulQ7P6PH9zN268vAUhgbq4mPIOziaaWUAs1mzIs4ptF+BdSs6SrJRysx2HTzMvPonPtx4hz2a4qmNT7h4Uw4C2jXX0vvI6zg7YzAdut8+aPARrHM1J4DtjjK7volQ1KLB3T35j7QE2Jp2kTpA/d/Rrw50DoomJqOvp8JQqU6UGbBpjdmItLKaUqibZeTaW/HiYuesOcCAti+bhITwxuhO39GlFeGigp8NTqkLODticCrQxxsxysG8WkGSMedu1oSnl2zLO5bBwQwoLE1LIyMqla4sw/nVbT0Z3i9LuyapGcbZGMx14s4x9J4A/AJpolHKB/WnneHNdEh9vPkROfgHDOzbl3sGxxMU20ucvqkZyNtG0o+wms11AW9eEo5RvMsawKekkb6xNYtWu4wQF+DGuVwvuGRRDu6a6eqWq2ZxNNPlYK1o60sRFsSjlc/JtBXy54xhvrD3AtkOnaVgnkN9f1Z7JcW1oUj/Y0+Ep5RLOJppNwP3ABw723Q/84LKIlPIB53LyeW9TKm/FJ3P41AViIqzp+cf1aklokI5/UbWLs4nmGWCViGwE5gKHsVbbnAb0Aka4Jzylapejpy8wPz6Zd+zLI/eJbsjM6ztzdadmOj2/qrWcHUfznYiMB14GXiu2KxkYZ4xZ4/LIlKpFdh45zdy11gDLAmMY1S2KewfH6vxjyic4PY7GGLMUWCoiHYDGQLoxZq/bIlOqhjPGsGZvGnPXHiD+lwzqBPkzuX8b7h4YQ6tGdTwdnlLVptJrtRpj9hS+F5HGxpgM14akVM2Wk29j6ZYjvLH2APtOnKNZWDCPXtuR2/u2JryODrBUvsfZAZv3Ag2MMf+wf98N+BKIEpEtwBhjzDH3hamU98vMymXxxhTmr08h/VwOHSPr888JPbi+R3OCAnSApfJdztZofge8Xuz7OcAprFmbfw/MBn7j0siUqiFSMrJ4c10SHyYe4kKejSGXNeE3g2MZ2E4nuFQKnE80rYHdACISDlwJ3GiMWS4iGcBzbopPKa+1OeUkb3yfxNc/HyPAT7ixZwumDY6lQ6QOsFSqOGcTjT9QYH8/CDDAGvv3B4Gmrg1LKe9kKzCs2HmM19ceYEvqKcJDA3lwaFvu7B9N0zBdYEwpR5xNNPuA64BvsRY7W2+MOW/f1xxryQClaq2snHw+TDzIvPhkUk+ep3WjOjw1tgsTerekTlCl+9Qo5VOc/T/kRWChiNwJNAQmFNs3DNjm6sCU8gYnzmQzf30yizemcvpCHr1aN2DGqI5c0yUSfx1gqZRTnB2w+Y6IpAL9gB+MMd8X230c+MwdwdU0cxLncOx8FTvfGdfEYuwnKno1Jb8vKldse+kyRWWLXso/Z5nlK3Hei8qaUvG66NqOzlu8bE5eARlZOZzJzsMYQ/3oQKLqBGKC/Hltv+G1/c6d02EMDu5FRXE6ey8e6PEAt3a8FaW8SWUGbK4D1jnYPtOlEdVg+07t4+DZg1U+j+Dav5QLez4VnrfotVSPKBEps0zpmCo6Z1H5ohfnz1v0nwjIxcc6fU4nyxWP4eT5PFIyskg/l4u/BNOiYR2iG9WhbnCg43+zis5ZOgYn7kWZZZ24F63CWlEjGAOmoIKv6ipjqvlaXngOin1/9woICHLp7dbGZRd69epXPR2CugS5+QV8vtUaYLn72Fki6gXz2wFtuKNfGxrWde3/cFViDGTsh/3fwtGfwJZ38S+QlD1gXnHulwzl/TKq6i+7CvYrrL+i/Mr5cvN+P3+QgIu3u6pZpRhNNMpnnT6fxzubUpm/PonjZ3Jo37Qefx/XnRsub05wgJfMoJyVAUlrYP9qOLAGTttrzPWaQWCdcn6hOPNLxt2/6KrrHOWUKfHL3Nvi9Z1nfJpolM85ePI8b65L4oPEg5zPtTGwXWNeGNedKy9r4vkBlvk5kLrBqrUcWA1HtwEGQsIhZggM+iO0HQaNYj0bp1KVoIlG+YwtqZnMXZvElzuO4ifC9T2aM21wDF2ah3suKGPg+E4rqexfDSnrIf8C+AVAy74w7HFoOxyieoK//u+qaib9yVW1mq3AsGrXceauPcAPyZnUDwng3iGx3DUgmqjwUM8Edeao1QxWmFyyTljbIy6DK+6E2GEQPRCCdYYBVTs4O6lmPPB/wAfGmBz3hqRU1V3ItfHRj4eYty6JpPQsWjQI5ckxnbm1TyvqBVfz31e5WVZNZf+3VmJJ22VtrxMBsUOtprDYYRDeonrjUqqaOPt/XB7wNvCyiLwNvG6M2e2+sJS6NGlnc1iYkMzCDSlkns+jR8tw/jPxckZ1jSTA3696giiwWb3CCh/gH9wItlzwD4Y2/aHnRCuxNOsKftUUk1Ie5OyAzaH2Bc/uA6YA00VkLfAqsMQYk+fGGJWq0C8nzjJ3bRJLthwmz1bAVR2bce/gGPrGNKqeB/yZKb82hSV9Bxcyre3NukG/+zGxQ6F1fwiwz4dWOHYjP//X74u9FnUwLbW9rNfCb/2CApEgL+qSrRQgpUczV3iASDBwC9ayAAOAdOAtrFrOAZdH6AG9e/c2iYmJlT7uyKOPkXf48EWjuK33ZfyiKPbeqeMcbCsx2t+UcdxF5yk1mt3J4y6K0dnybrpGXoEhJzefPJtBMAQF+BHs74efnzhxHeficVzeQIH9tfB7A7h4sG1lNf3LX2h891SPxqB8l4hsNsb0Lr39UlbYzMGa92wn1ro0Q4C/AH8WkU+A3/nsImgi4Odn/aop/Cu6+F/TRZuk5AZH5R0ch4PR447Ll95XxnXLuqa7yjs4pvD7iq/1a/mCAkNq5gV2HjlDRlYuIYH+dIwMo0NkfUIKJ7h01WcpfDUGzh6FU6lwKsV6jwG/QGjQChrGQKM2SJ0I+zku4bM7+W9QXvk6ffqglLepVKIRkVBgInA/cAXWGjXTgQ+B64FZwGLgKpdGWUM0f16X5XGnM9l5vLcplbfikzlaJ5vY/nW5d3AsN13egpBAFw+wNAYyfrE/Z1kNSWtBzkIjP+jaC9qOtZ6ztOzj8uk6lKptnO111g3r+cwdQF1gKfCoMWZ1sWJviMgxrKSjlMscPnWBt9Yl8d4PBzmXk09cbCP+dmNXhnVoajWRuUrRKPxvYf8aOHPI2t4wGrqNt3qHxQyB0Iauu6ZSPsDZGs1W4AjwMtazmKNllPsFSHBBXEqx/dBp3lh7gGXbrR+367pFce/gWLq1dNEAy7xsOLjh11pL6VH4gx/WUfhKuYCziWYC8KkxxlZeIWPMLqz1aZS6JAUFhtV7TvDG2gNsOHCSesEBTB0QzdRBMbRoUMUBluWNwm/VD4Y9YSWW5pdbc4EppVzC2UTzGRACZJXeISJ1gVzt4qyqIjvPxidbDjN37QH2p2URFR7C46M7clvf1oSFBF76ic8c/TWxHFhTbBR+Bx2Fr1Q1cTbRzAUCgdsd7HsNyAXudlVQyndknMth4YYUFiakkJGVS5fmYbx8a0+u6x5F4KUMsMzNguT4X5OLjsJXyuOcTTTDgEfK2PcZ8A/XhKN8xf60c7y5LomPNx8iJ7+AYR2acO/gWPq3bVy5AZalR+GnboCCPPso/AE6Cl8pL+BsomkKnChjXxrQzDXhqNrMGMOmpJO8sTaJb3YfJ9DPj5sub8G0wTG0b1aJpqvM5GLdjr//dRR+ZDeIe8CqtbTuD4EemjRTKVWCs4nmBNANWO1gXzcgw2URqVrHVmBYvv0oc9ceYOuh0zSoE8hvh7Vjcv82NK0fUvEJLpyC5LW/JpeT9gko6jeHDqOtGkvsUKjXxJ0fQyl1iZxNNF8AT4rIGmPMtsKN9vE1TwCfuCM4VTv8/evdvPbdAaIb1+HpG7syvldLQoPK6dVly4NDifbnLN/C4c3W8r+BdSF6EPS9z6q1RFzmU6sUelp2djZpaWlkZ2eTXzhHm6r1AgICCAkJoUmTJoSEOPGHoaNzOFnu/wEjgM0i8gNwCGgB9AWSgL9e0tVVrZeVk887G1IZ1TWSV27vhb+jAZbGQPq+Xx/gJ6+D3LPWkrfNe8HgP+kofA87ffo0x48fp0mTJkRGRhIQEOD51UiV2xljyM/P59y5c6SmptKsWTPCwys/js3Z2ZvTRaQP8DBWwumJNZnmM8BLxpjTlb6y8glLfjzE2Zx8pg2OLZlkstKLLf61Rkfhe7n09HRatmxJnTp1PB2KqkYiQmBgIA0bNiQ4OJhjx465L9EAGGNOYdVs/l+lr6J8kjGGtxNS6NoijF7NQ6zEst/eHHbM3gIbEg4xV8IQe62lUYxHY1aO5ebmEhqqnSt8WWhoKDk5l7bupS7lrNxm/f4Mfjlxjvf6H0RemFBqFP5fdRR+DaNNZb6tKvff6UQjIl2Be4AOWLMEFGeMMU7N2CwirYCXsJrgBFgF/MEYk1rBcbOAmWXszjHGhBQrmwy0cVDuJmPMp87Eqapu/vpkIkL96Lf/PxDRzkouOgpfKZ/j7OzN/YDvgGSgPbANaAi0xuoY8IuT56kDfAvkAHdiLRX1N2C1iHQ3xlw0xU0xc4GvSm2ra9/2mYPyX2MtW1DcHmfiVFV38OR5vtl1nJe6JiH7DsHof0CHaz0dllLKA5wdKv0ssAToglULuccYEw1cDfhjJQtn3AvEAjcaYz41xiwFxmLVPu4r70BjzCFjzIbiX0BzrGT5toND0kuXN8ZkOhmnqqJFG1MAGHV2iTX78WWaZJR3mTVrVpWag5KTk5k1axYHDrh+YeE1a9YgIqxZs8bl5/YEZxNNd2ARv65m6w9gjPkWK8k4u+LXWGCDMaaoBmSMSQLigRucPEdxdwLHsWovyktk59l4/4eD3Bd7kqBjm6HfAzr9i6p1kpOTeeqpp9ySaHr16kVCQgK9evVy+bk9wdn/+wOBLGNMAXASiCq2bw/Q1cnzdAF2ONi+E+js5DkAEJGWWHOwLTbGOBo9dr2InBeRHBHZICI3Vub86tJ99tMRTp3P456A5Vavsp6O5mJVyvvZbDaXDE41xpCbm+t0+bCwMOLi4ggLC6vytb2Bs4lmP9YATbCez9wtIn4i4gdMBY45eZ5GgKPmq5NYz3wqYzJW/I6azT4HfgeMxFoVNBv4REQmlXUyEfmNiCSKSGJaWlolQ1GFjDHMX5/MoCYXaJz6NfS6E4LreTospZwiIjzxxBM8//zzxMTEEBQUxPbt2y8qt2bNGoYNs5beGjFiBCJSoqkrOjqaSZMmMW/ePDp27EhQUBDLli0DYObMmfTq1Yvw8HAiIiIYPnw4GzZsuOj8pZvOhg4dyqBBg1i1ahW9evWiTp06dO3alU8//dQt/xauVJkpaIYC72A9r1kGnAFsQD3g95W4pnGw7VIaSqcAW4pPiVN0AWN+V+LkIp8AG7Ca+BY5DMqY14HXAXr37u0oRuWExJRMfj56huWdvkfOAf3KffSmlNeZP38+sbGxvPjii9StW5fmzZtfVKZXr17897//5aGHHuLf//43ffr0AaBz518bZlavXs1PP/3EzJkzadq0KdHR0QAcPnyYP/7xj7Rs2ZKsrCwWLVrEkCFDSExMpHv37uXGtn//fqZPn86MGTOIiIjgn//8J+PHj2f37t20a9fOdf8ILubszAAzi71fJSJxwDigDvCVMWaFk9fLxKrVlNYQxzUdh0SkL9AR+IMz5Y0xNhH5EHhBRKLKWYpaVdHb65NpGpJPpyOfQOcbILylp0NSbvTU5zv5+cgZj8bQuXkYM6/v4rLzGWNYsWJFuQNUw8LCipJKp06diIuLu6hMZmYmmzdvJjIyssT2uXPnFr232Wxce+21dOnShTfffJN//etf5caWnp7O999/T/v27QEr4UVFRfHBBx/w+OOPO/0Zq1uFiUZEAoHRwDb7g3uMMVuALZdwvZ1Yz2lK6wz8XInz3AnkY9WwnFVYa9LaipscP5PNVzuO8Urbn5CDZ6D/Q54OSalKu/baa10yC0JcXNxFSQZg1apVPPPMM2zbto2TJ08WbY+JqXhWjPbt2xclGYCmTZvStGlTUlPLHYbocRUmGmNMnoh8AFyLNYFmVXwGvCgiscaYAwAiEg0MBB5z5gQiEgTcBiw3xjj1MEVEAoAJQKoxxtnnSaqSFm9IwRgbV51ZAi37Qsveng5JuZkraxLeIioqquJCl3ieH3/8kdGjRzNy5EjefPNNoqKi8Pf3Z9q0aWRnZ1d4zkaNLm4QCg4OdupYT3L2Gc0BrMXPquoN4LfAUhH5K1bt4mngINaS0ACISBusDgizjTGzS51jDFbzm6NOAIjIRKyu0svt520GPARcAUx0wWdQDuTk23hnUyrTW+4nMC0Zrpnl6ZCUuiSummrH0Xk+/vhjAgICWLJkCYGBgUXbMzMzadCggUuu642c7XX2d+AJEanSylL2kf/Dgb3AQmAxVi1puDHmXLGigjVWx1F8d2L1UvuijMskYSXFfwArsBJYDnCtMea9qsSvyvbl9mOkn8tliiyD8NbQ8XpPh6SUWwUHBwNw4cIFp485f/48/v7+JZLQt99+6/VNX1XlbI1mOFYtIklENgBHKfmswxhj7nTmRPY5zcZVUCaZMnqiGWPKHdhpnzFguDOxKNeZvz6ZEQ2P0+DEJrjmb+Cv87Wq2u2yyy4jICCAefPm0ahRI4KDg+nQoQP165c9l9+1117Lyy+/zF133cXUqVPZu3cvTz/9NC1atCjzmNrA2RrNICAPSAPa2r8fXOpL+aitB0/x08FTPBr+DQTVg15TPB2SUm7XuHFjXnnlFbZu3cqVV15Jnz592Lx5c7nHjBw5kn//+9/Ex8czZswY5s2bx4IFC7y6a7IriDHaCau03r17m8TERE+HUWM8/MFPJO7YxXcBv0P63AOjXvB0SMrFdu3aRadOnTwdhvKwin4ORGSzMeaiXkA6AZWqkoxzOXyx9SizoxKQgnwdoKmUuoizywS0rqhMRevJqNrpvR8OIrZsBp/+DDpeZ83UrJRSxTj7xDaZigc66jKJPibfVsCiDSk8ErkF/1OZEPeAp0NSSnkhZxPN3VycaBoD12GtL/O0K4NSNcPKn49z9PQFbgv6AiK7Q5uBng5JKeWFnJ3rbH4Zu+aIyEKsZKN8zPz1ydxcfzf1zu6Hq18DXVNeKeWAKzoDLMKq8SgfsuvoGTYmneSP9VZCvUjocrOnQ1JKeSlXJJqmQIgLzqNqkAUJKXQJOEyrzA3QdxoEBHk6JKWUl3K219kQB5uDsFbWnAGsdWVQyrudPp/Hp1sOM7/Jd3A2BK7QCq1SqmzOdgZYw8WdAQob5L8DtLuRD/kg8SCheZn0ObMSek6Euo09HZJSyos5m2iGOdiWDaTotPu+xVZgWLghhb9ExON3LgfiHvR0SEopL+fUMxpjzHcOvjZqkvE9a/ac4NjJ09yY9yW0uxqadPB0SEpdklmzZrlsSQBVPqcSjYjEicgtZeybICL9XBuW8lbz1ydzR90fCMlJ19qMUsopzvY6ew7HSzADdLLvV7Xc/rRzrN2XxoMhK6BJJ2irqzGo2s1ms5Gfn+/pMGo8ZxNND2BDGfs2Ad1dE47yZgsTUhgUsIsmWXut6Wa02UHVMiLCE088wfPPP09MTAxBQUFs3769zPJZWVk89thjtG3bluDgYCIjIxk3bhzHjx9n06ZNiAiff/75Rcc98MADNGnShLy8PHd+HK/hbGeAEMpOSv5AXdeEo7zVuZx8Ptp8iPcafAsFjaG7w5ZUpWq8+fPnExsby4svvkjdunVp3ry5w3K5ubmMGDGCn376iRkzZhAXF8fp06f5+uuvyczMpG/fvnTo0IGFCxdy/fXXlzjugw8+4Pbbby+xnHNt5myi2QWMBZY52DcW2OOyiJRXWvLjISJyD9LlXAIMeQQCQz0dkvIGXz4Gx8r+i79aRHaDUc+77HTGGFasWEFoaPk/44sWLSIhIYGlS5cyduzYou3jx48vej958mT+9re/cfr0acLDwwFYvnw5J0+eZPLkyS6L2ds523T2f8C9IvIPEblMROqISHsR+QdwD/A/94WoPM0Yw9vrk3kkfDX4B0KfaZ4OSSm3ufbaaytMMgArVqwgMjKyRJIpbdKkSeTk5PDhhx8WbVu4cCEdOnSgb9++Lom3JnB2Us03RKQD8Efg4eK7gJeMMa+7IzjlHdb9kk5a2nFG1v0G6Toe6jfzdEjKW7iwJuEtoqKinCqXkZFBixYtyi3Tpk0bhgwZwsKFC5k2bRqnTp1i2bJlPPnkk64ItcZwtukMY8yfReRV4GqsJQLSgVXGmAPuCk55h7fXp3BP6PcE2C5Af+3SrGo3Z8fWREREsGPHjgrLTZ48mXvvvZeUlBS+/vprcnNzueOOO6oaZo1SqUk1jTH7jTGvGWOeNca8rkmm9jt48jzf7T7M1MAVED3Yag9XSnHNNddw7Ngxh73KipswYQIhISEsXryYhQsXMmTIEKKjo6snSC/h7IDNqSIyq4x9s0TkTpdGpbzGog0pjPb/gbDcE9D/IU+Ho5TXmDRpEv3792fixIk888wzrFq1ik8++YT777+f3bt3F5ULCwtj7Nix/Pe//yU+Pt6nOgEUcrZGMx3IKGPfCeAPLolGeZULuTbe25TKH+quhEZtof1IT4eklNcIDAxkxYoVPPDAA7z++uuMHj2aBx98kPT0dBo1alSi7OTJkzly5AjBwcEleqX5CjGm9KTMDgqJnAPGGmO+dbBvGLDUGBPmhvg8onfv3iYxMdHTYXjce5tS+eCTj1kSPAtGvwh97/V0SMpDdu3aRadOnTwdhvKwin4ORGSzMaZ36e3O1mjygYgy9jVx8hyqBjHG8HZCCn+stxITEg49Jno6JKVUDeVsotkE3F/GvvuBH1wTjvIWPyRncvbYfgbmJSBX3AXB9TwdklKqhnK2e/MzwCoR2QjMBQ4DLYBpQC9ghHvCU57y9vpk7g1eaXX17PsbT4ejlKrBnB2w+Z2IjAdeBl4rtisZGGeMWePyyJTHHDudzdqdSfwzdDXS6UYIb+npkJRSNVhlBmwuBZbaZwhoDKQbY/a6LTLlMYs3pjDObzUhtiyI0y7NSqmqcTrRFDLG6ASatVhOvo33NyaxLGQlRPWDlld4OiSlVA1XqUQjIj2ADljLBpRgjFngqqCU5yzffpTLL2ygSdAxiHvB0+EopWoBpxKNiDTAWiIgrnCT/bX4IBxNNLXA/PUpzA5dganXCuk4xtPhKKVqAWe7Nz+L9VxmCFaSuQkYDiwGDgC+M991LfbTwVPkH9pCD9tOpN/94F/pllWllLqIs4lmJFayKVzO+ZAxZo0xZgqwCmuKGlXDLVifzH1BX2GC6kIv35uPSSnlHs4mmijggDHGBmQD9YvtWwJc5+rAVPVKP5fDpm07uU7WI5dPhpBwT4ekVI22Zs0aRIQ1a9aUW27+/PnMmzfPLTEMHTqUoUOHuuXcleFs28gxoIH9fQrQH1hj/76da0NSnvDuxlRuk6/xowD63efpcJTyGfPnzyc/P5+7777b5ef+3/+8Y/FjZxPNOqzk8gWwEJgpItFYc6DdCXzmluhUtcizFfDRhn18EfQtctl10CjW0yEppRzIyckhODjY6fKdO3d2YzTOc7bp7CngK/v7fwD/xWoum4iVZH7n+tBUdVmx8zgDz6+ifsFZiNMVNJVv2Lt3LzfddBNNmzYlJCSE1q1bM2HCBPLz84vK/PjjjwwePJiQkBBatWrFs88+y8yZMy9ahTMtLY3bb7+dsLAwGjRowJQpUzh16lSFMQwdOpTvvvuO+Ph4RAQRKWrqmj9/PiLC999/z4QJE2jQoAH9+vUD4IcffmD8+PG0bNmS0NBQOnTowOOPP86FCxcuOn/xprPC5rzPPvuM3/72t0RERNCkSRMmTZrkVLyXytkpaPYD++3v84A/2b9ULbAg/gAvBH2FieyBtBng6XCUqhZjxoyhQYMGvPrqq0RERHD48GGWL19OQUEBAOnp6Vx11VU0b96cBQsWEBQUxEsvvURycvJF57r55pvZunUrzz77LO3bt+f999/nd7+r+O/v//3vf0yaNAmbzcZrr1mze4WFlVxx5Y477mDixIl89NFHRUkwNTWVnj17ctddd1G/fn127tzJ7NmzOXDgAO+9916F150+fTpjxozhnXfeYc+ePfzlL3/B39+ft99+u8JjL4X2X/VxPx85Q+jBNUQHHYa4meDkeulKAbyw6QV2n9xdcUE36tioI4/2fbRSx6Snp7Nv3z6WLl3K2LFji7bffvvtRe/nzJlDVlYWX3/9NS1bWvP9jRw58qJlmFeuXMm6det49913ue2224rKjRo1ikOHDpUbR+fOnQkLCyM/P5+4uDiHZcaPH8/f//73EtvGjRtX9N4Yw8CBAwkLC2PKlCn897//pXHjxuVed8iQIfznP/8BrCWp9+zZw9y5c4tqUa7mbNOZqqUWJCRzb+CXFNRtBl1u8nQ4SlWLxo0bExsby2OPPcYbb7zBvn37LiqzYcMG+vfvX5RkAEJDQ7nuupKdbBMSEvD39y/xyx8oSjpVddNNF/9/eebMGR599FHatm1LcHAwgYGBTJ48GWOMw89SWunP0K1bN3Jycjh+/LhLYi5NazQ+7NT5XHb+tIHn/bdDvychIMjTIakaprI1CW8hIqxcuZJZs2YxY8YMMjIyiImJ4ZFHHuGBBx4A4OjRo3Tt2vWiY5s1a1bi+6NHj9KwYUMCAwPLLXepoqKiLto2depUVq1axezZs+nZsyd169Zl06ZNPPTQQ2RnZ1d4ztJLTRd2MHDm2EuhicaHfZB4kDvMMgoCQvDr7fqulUp5s9jYWBYsWIAxhq1bt/LKK6/w4IMPEh0dzahRo4iKiuLEiRMXHVf6r/6oqCgyMzPJy8srkWxcVTso3ZSVnZ3N0qVLmTVrFtOn/zpWfvv27S65njto05mPshUYPlu/lZsD4vHrMRHqNKr4IKVqIRGhZ8+ezJkzB4AdO3YAEBcXR0JCQonnLBcuXGDZsmUlju/fvz82m42PP/64xHZnHsqDVZso3VusPDk5OdhstotqUPPnz3f6HNVNazQ+avXuEww/u4ygwDzt0qx8zrZt25g+fTq33nor7dq1w2azMX/+fAICAhg+fDgADz/8MK+++iojR45k5syZBAcHM2fOHIKDg0vUMkaMGMGgQYO47777SE9PL+p1VpiwKtK5c2f+97//8f7779O2bVvq169Phw4dyiwfHh5OXFwc//znP4mKiiIiIoJ58+Zx+PDhqv2juJHWaHzU4vV7uTNwFQXtroYml3k6HKWqVWRkJK1bt2bOnDmMHTuWiRMncuTIEb744guuuMJagykiIoJvvvmGhg0bMmXKFB588EGuvvpqbrrpJsLDS07RtGTJEkaPHs2MGTO49dZbyc/P55VXXnEqlkcffZSrrrqKadOm0adPH+67r+KZOd59912uuOIKHnroIe666y4iIyP517/+Vfl/iGoixpiKS/mY3r17m8TERE+H4Ta/nDjH//1rNi8GvgaTP4G2wz0dkvJyu3btolOnTp4Ow+NsNhu9evUqSkK+pqKfAxHZbIzpXXq7Np35oIXrk5gW8CX5EZ0IiB3m6XCU8lpPPvkk7dq1o02bNmRkZDB37ly2bdvG8uXLPR1ajaKJxseczc7j4I9f09EvFQb8RwdoKlUOEWH27NkcOXIEEaF79+58+umnjBo1ytOh1SiaaHzMx5sPcbv5gryQxgR2u8XT4Sjl1WbPns3s2bM9HUaNp50BfEhBgeHb+PVc7b+FwH7TIDDE0yEppXyAJhofsu6XdK4+8wk2v0DoM83T4SilfIQmGh/y0dptTAj4HrqOh3pNPR2OUspHaKLxEakZ52me9AGh5OA/4CFPh6OU8iHVnmhEpJWIfCQip0XkjIgsEZHWThw3S0RMGV/Zpcr6icgMEUkWkWwR2Soi48o6ty9YvP4X7vJfQU6rQRDZzdPhKKV8SLX2OhOROsC3QA7WEtAG+BuwWkS6G2Oyyjl8Lr+u8lmorn1b6aWknwb+DDwBbAZuAz4UkTHGGJ/rAH8+N59Tmz8kUk7CIF0MVSlVvaq7e/O9QCzQwRjzC4CIbAP2AfcBc8o60BhzCCixipCITMb6DG8X29YUK8k8b4x50b55tYi0A54HfC7RLN1ymIkFX3ChQQyh7a/xdDhKKR9T3U1nY4ENhUkGwBiTBMQDN1zC+e4EjgNfF9s2EggCFpUquwjoJiIxl3CdGssYww9rv6Sn3wFCBj0EfvpYTqnqsGbNGkSENWvWeDoUj6vu3zpdAEdTmu4EOlfmRCLSEhgGLDbG5Je6Rg7wS6lDdtpfK3Wdmm5j0kmuPv0xOYFhSM/bKz5AKaVcrLoTTSMg08H2k0DDSp5rMlb8b5fa3gg4ZS6eLfRksf0XEZHfiEiiiCSmpaVVMhTvtez7BEb6/4B/76kQVNfT4SilfJAn2lEcTRd9KRNuTQG2GGO2OThXpa9hjHndGNPbGNO7SZMmlxCO9zly6gLR+xeD+BEQV/HU40r5kr1793LTTTfRtGlTQkJCaN26NRMmTCA//9cGkh9//JHBgwcTEhJCq1atePbZZ5k5c+ZFq16mpaVx++23ExYWRoMGDZgyZQqnTp1yOpbvvvuOESNGEB4eTt26denRowdvvvkmAKNHjy5auqC4o0ePEhAQwMsvv3xJn786VXdngEwc1yga4rim45CI9AU6An9wsPsk0FBEpFStpmGx/T7ho/ifmeq3muz2Y6kb3sLT4SjlVcaMGUODBg149dVXiYiI4PDhwyxfvpyCggIA0tPTueqqq2jevDkLFiwgKCiIl156ieTk5IvOdfPNN7N161aeffbZooXPfvc753p4Ll26lHHjxjFw4EBee+01IiIi2LlzJykpKQBMmTKFiRMn8vPPP9O5868t/++88w4AEydOrOK/hPtVd6LZifUMpbTOwM+VOM+dQD7wThnXCAbaUvI5TeEdqsx1aqzsPBu5iQuoLxfgSu3SrNzj2LPPkrNrt0djCO7UkcjHH6/UMenp6ezbt4+lS5cyduzYou233/7rc8w5c+aQlZXF119/TcuWLQEYOXIk0dHRJc61cuVK1q1bx7vvvsttt91WVG7UqFElloF2xBjD9OnT6dmzJ6tXr8bP3lnn6quvLipzww03EBYWxsKFC3nuueeKti9cuJBrrrmGZs2aVeqze0J1N519BsSJSGzhBhGJBgZy8VgYh0QkCGtczHJjjKOHKV8BucAdpbZPAnbYe7nVesu2HuIW2zJON7kCWlxc7VbKlzVu3JjY2Fgee+wx3njjDfbt23dRmQ0bNtC/f/+iJAMQGhrKddddV6JcQkIC/v7+jBtXckx4YdIpz549e0hJSWHatGlFSaa00NBQxo0bx+LFiylspNm+fTtbt25lypQpFV7DG1R3jeYN4LfAUhH5K9azlKeBg8BrhYVEpA2wH5htjCk9R/cYrOa30p0AADDGnBCRl4AZInIW+BG4FRjOpXWhrpH2fvce4/zSMENfrLiwUpeosjUJbyEirFy5klmzZjFjxgwyMjKIiYnhkUce4YEHHgCsZyBdu3a96NjSNYijR4/SsGFDAgMDyy3nSEZGBkCJZObIlClTeOutt1izZg3Dhg1j4cKF1K9fnxtuqBm/0qq1RmMf+T8c2AssBBYDScBwY8y5YkUF8C8jvjuxnrN8Uc6lnsCacWA61hibgcAtxpjPq/oZaoItqZlcdfpjzoU0RzqO8XQ4Snml2NhYFixYQFpaGlu2bGH48OE8+OCDfPnllwBERUVx4sSJi447fvx4ie+joqLIzMwkLy+v3HKOREREAHD48OFyy1155ZW0bt2aRYsWUVBQwLvvvsv48eMJDQ2t8BreoNp7nRljUo0x44wxYcaY+saYG40xyaXKJBtjxBgzy8HxNxhjGhtjcsu5hs0Y8zdjTBtjTLAxprsx5iPXfxrv9O03X9HXbw+BAx8Af13bTqnyiAg9e/ZkzhxrYpIdO6yhfnFxcSQkJJR4znLhwgWWLVtW4vj+/ftjs9n4+OOPS2x/7733Krz2ZZddRnR0NHPnzsVcNCKjZIx33HEHH330EcuXL+fQoUM1ptkMdIXNWiftbA7tkxaSE1CH4D53ejocpbzStm3bmD59Orfeeivt2rXDZrMxf/58AgICGD58OAAPP/wwr776KiNHjmTmzJkEBwczZ84cgoODS3RvHjFiBIMGDeK+++4jPT29qNdZYcIqj4jw8ssvc/PNNzN8+HDuv/9+mjRpwq5duzhx4gRPPfVUUdkpU6bw3HPPcf/999OqVSuuvPJK1//DuInOR1LLfLY2kVGygeyuEyEk3NPhKOWVIiMjad26NXPmzGHs2LFMnDiRI0eO8MUXXxSNWYmIiOCbb76hYcOGTJkyhQcffJCrr76am266ifDwkv9vLVmyhNGjRzNjxgxuvfVW8vPzeeWVV5yK5YYbbmDlypUA3HPPPYwdO5bXX3/9ot5tHTt2pHfv3hw+fJhJkyZdNJbHm0l51TVf1bt3b5OYmOjpMCotz1bAO8/czeSCT/H7/RZo5FPTuik32rVrF506dfJ0GB5ns9no1atXURLyNRX9HIjIZmNM79LbtemsFlm19QA32FaQ3upqmmqSUarKnnzySdq1a0ebNm3IyMhg7ty5bNu2jeXLfW4S+CrRRFOLHFozjwaSRcFVf/R0KErVCiLC7NmzOXLkCCJC9+7d+fTTTxk1apSnQ6tRNNHUEjsPZzL81BLSwjvRJHqAp8NRqlaYPXs2s2eXHsqnKks7A9QSm1a8T1u/o9S98vdQgx4SKqVqP000tUBmVi4dkhZyOiCCOj3HezocpZQqQRNNLbBnawID/HaQ0+seCAjydDiqltIeqr6tKvdfE00tEBcdRl70UJoOvd/ToahaKigoiAsXLng6DOVBFy5cIDg4+JKO1c4AtUHzywm8a6mno1C1WEREBIcOHSIiIoL69esTEBBQowYMqktjjCE/P5+zZ8+Snp5+yUsSaKJRSlUoPDyc4OBg0tLSyMjIKLEKpardAgICilYgDQkJubRzuDgmpVQtVbicsVKVpc9olFJKuZUmGqWUUm6liUYppZRbaaJRSinlVppolFJKuZUmGqWUUm6lC585ICJpQIqn4/ASEUC6p4NQTtP7VbPUtvvVxhjTpPRGTTSqXCKS6GjFPOWd9H7VLL5yv7TpTCmllFtpolFKKeVWmmhURV73dACqUvR+1Sw+cb/0GY1SSim30hqNUkopt9JEo5RSyq000fg4EWkpIv8RkQQROS8iRkSiHZRrKCJzRSRdRLJEZJWIdPNAyD5LRMaLyMcikiIiF0Rkj4g8JyL1S5XTe+UFRGSkiHwrIsdEJEdEDonIByLSuVS5Wn+/NNGodsAtQCaw1lEBsZZS/Ay4FvgdMA4IBFaLSMtqilPBnwEb8DjWvXgVeABYKSJ+oPfKyzQCNgO/Ba4BZgBdgA0i0gZ86H4ZY/TLh78Av2LvpwEGiC5V5gb79mHFtoUDJ4F/e/oz+MoX0MTBtin2ezNc75X3fwEd7PfnT750v7RG4+OMMQVOFBsLHDHGrC523Gngc6z/UVQ1MMakOdj8g/21hf1V75V3y7C/5tlffeJ+aaJRzugC7HCwfSfQWkTqVXM86ldX2l932V/1XnkZEfEXkSARaQ+8BhwD3rPv9on7pYlGOaMR1jOc0k7aXxtWYyzKTkRaALOBVcaYRPtmvVfeZyOQA+wFumM1c56w7/OJ+6WJRjlDsNqRHW1XHmD/S3cpkA9MLb4LvVfeZjIQB9wOnMHqvBFt3+cT90sTjXLGSay/vEor/GvL0V9kyk1EJASrp1IsMNIYc6jYbr1XXsYYs8sYs9EY8y5wFVAPeMy+2yfulyYa5YydWG3JpXUGUo0x56o5Hp8lIoHAx0BfYLQxZnupInqvvJgx5hTwC9awAvCR+6WJRjnjM6CFiBQ+eEZEwoDr7ftUNbCPlVmM9VfxDcaYDQ6K6b3yYiLSDOgI7Ldv8on7pZNqKkRkvP3tVcD9wINAGpBmjPnO/gtuHdAKeASrOj8D68FmD2PMweqP2veIyKtY9+cZ4ItSuw8ZYw7pvfIeIvIJ8COwDevZzGXAH4FIoK8xZq+v3C9NNAoRKeuH4DtjzFB7mUbAi8CNQAiQADxsjNlaHTEqEJFkoE0Zu58yxsyyl9N75QVE5FGsWTfaAkHAQWAN8JwxJrlYuVp/vzTRKKWUcit9RqOUUsqtNNEopZRyK000Siml3EoTjVJKKbfSRKOUUsqtNNEopZRyK000SpVBRO4WkX0ikisip+zbkkVkfrEyQ0VkVuEKlxWcr4G9bC8H+9aIyBoXhu80EfETkZ9E5E+VOOZy+9Lfrd0Zm6oddByNUg6ISHMgFWvKlzeAbGNMoohcDpwxxuy3l5sFzAQCjTH5FZwzGkgC7jXGzC21rzOAMeZnF3+UConIFKwBg9HGmPOVOG4pcMoYc6fbglO1QoCnA1DKS7UH/IG3jTHrCjcaY7a442KeSDDF/BnrczqdZOxeA5aKyAxjzBE3xKVqCW06U6oUe9PYGvu334iIKWwuK950Vqw2A5BnL+ewiaBYbQbgjcKyInKXfX+JpjN7k5wRkRtF5DUROSkimSLykn3Fxj4isk5EskRkp4iMdHDNK0XkGxE5ay/3tYh0LVWmH9ANeKfU9stE5BMROSEi2SKSKiIfikjxP05XYM3hdVfZ/5pKaaJRypGngd/b3z8E9LdvK20u8Kb9/SB7uf5lnPMocLP9/XPFyi6rIJaXgSzgVuAV4A/2bQuAefZzngSWiEhE4UEich3wDXAOmIS16FZ9YK2ItCp2/muBs0DpebW+AFoADwAjsdZPyaHY7wx7U2GC/RxKlUmbzpQqxRizX0R22b/9uYzp+LHPlly46NjG8p7RGGNyRKSw2e1AWed04FtjzMP29yvtCeS3wODCJj0ROYqVKK4D3raX/RfWpKg3FJ5IRFYDB4A/YSUssFZ+3G6MKShWLgKr6fAGY0zxqepL1HrstgCPiIhf8XMoVZzWaJTybl+W+n43kFX8uZF9G1hTzSMi7bFmDF4sIgGFX8B5rBrIkGLHNsdaEqK4DKyE9LyI3Gs/X1nSgGAcrxKpFKCJRilvV3op31zgVPENxphc+9sQ+2tT++ubQF6przFA42KHh2A1iRU/nwFGAIlYzXx7ReSAiDzgIL4L9tdQ5z6O8kXadKZU7ZNhf50BrHKwP7dU2YalCxhjDgBTRESAHljNdf8TkWRjTPFaVmFNJr3KUataS2s0SlVNYW3Amb/oK1O2KvYAyUAXY0yig69txcruBmLLOpGx/AQUPifqWqpIDHDQGHMBpcqgNRqlqqZw/MufRORLwGaMSSyj7HGsGsRtIrINqzdZkjEmo4zyl8QYY0TkIawxLkHAB1g1jmbAACDVGDPHXvx7YKqINC6MQ0S6Y3UmeB/4BWs80V1APvBtqcv1s59DqTJpjUapqvkC+B/wINaD9h/KKmjvlTUNq6lqlb3s9e4IyhizHOuhf12sbthfA3/HWq8+oVjRpUA21rObQsewZkV4GPgMeBer08AYY8zmwkL2btI9gPfc8RlU7aFT0Cjl4+wDUFsaY66u5HGPYo2zaWuMsbkjNlU7aKJRyseJSAywCxhUTrNf6WNCsLpAP2aMWeDO+FTNp01nSvk4Y0wS1jOYphUULS4a6znOQjeEpGoZrdEopZRyK63RKKWUcitNNEoppdxKE41SSim30kSjlFLKrTTRKKWUcqv/D5LgaAccBIfsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(results_lr[\"mean_fit_time\"], results_lr[\"mean_train_score\"], label=\"lr train\")\n",
    "plt.plot(results_lr[\"mean_fit_time\"], results_lr[\"mean_test_score\"], label=\"lr cv\")\n",
    "plt.plot(results_sgd[\"mean_fit_time\"], results_sgd[\"mean_train_score\"], label=\"sgd train\")\n",
    "plt.plot(results_sgd[\"mean_fit_time\"], results_sgd[\"mean_test_score\"], label=\"sgd cv\")\n",
    "plt.xlabel(\"fit time(s)\");\n",
    "plt.ylabel(\"accuracy score\");\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TL;DR if you don't want to wait that long, `SGDClassifier` will probably do better on a large dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}