ensemble-methods-notebooks-master/Ch5.3and5.4-practical-boosting-with-lightgbm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains code and comments from Section 5.3 of the book [Ensemble Methods for Machine Learning](https://www.manning.com/books/ensemble-methods-for-machine-learning). Please see the book for additional details on this topic. This notebook and code are released under the [MIT license](https://github.com/gkunapuli/ensemble-methods-notebooks/blob/master/LICENSE)._\n",
    "\n",
    "## 5.3  LightGBM: A Framework for Gradient Boosting \n",
    "\n",
    "[LightGBM](https://lightgbm.readthedocs.io/en/latest/), or Light Gradient Boosted Machines, is an open source gradient boosting framework that was originally developed and released by Microsoft. \n",
    "At its core, LightGBM is essentially a histogram-based gradient boosting approach. However, it also has several modeling and algorithmic features that enable it to handle large-scale data. In particular, LightGBM offers the following advantages:\n",
    "\n",
    "* Algorithmic speedups such as gradient based one-sided sampling and exclusive feature bundling that result in faster training and lower memory usage; these are described in more detail in Section 5.3.1;\n",
    "* Support for a large number of loss functions for classification, regression and ranking as well as application-specific custom loss functions (Section 5.3.2);\n",
    "* Support for parallel and GPU learning, which enables it handle large-scale data sets (parallel/GPU-based machine learning is out-of-scope for this book).\n",
    "\n",
    "\n",
    "### 5.3.2 Gradient Boosting with LightGBM\n",
    "LightGBM is available for various platforms including Windows, Linux and MacOS, and can either be built from scratch or installed using tools such as ``pip``. See the documentation of LightGBM for [installation instructions](https://lightgbm.readthedocs.io/en/latest/). Its usage syntax is quite similar to ``scikit-learn``’s. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing with the breast cancer data set from Section 5.2.3, we can learn a gradient boosting model using LightGBM as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "Xtrn, Xtst, ytrn, ytst = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a gradient boosting classifier using LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(max_depth=1, n_estimators=20)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lightgbm import LGBMClassifier\n",
    "gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=20, max_depth=1)\n",
    "gbm.fit(Xtrn, ytrn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9473684210526315"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "ypred = gbm.predict(Xtst)\n",
    "accuracy_score(ytst, ypred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 5.4 LigthGBM in Practice\n",
    "\n",
    "As with any ML algorithm, we need to ensure that we avoid overfitting during training. We look to set the learning rate (Section 5.4.1) or employ early stopping (Section 5.4.2) as a means to control overfitting. Specifically,  \n",
    "* by selecting an effective learning rate, we try to control the rate at which the model learns so that it doesn’t rapidly fit, and then overfit the training data. We can think of this a proactive modeling approach, where we try to identify a good training strategy so that it leads to a good model. \n",
    "* by enforcing early stopping, we try to stop training as soon as we observe that the model is starting to overfit. We can think of this as a reactive modeling approach, where we contemplate terminating training as soon as we think we have a good model.\n",
    "\n",
    "### 5.4.1 Learning Rate\n",
    "As with AdaBoost, we can aim to avoid overfitting by training with an appropriate [learning rate](https://lightgbm.readthedocs.io/en/latest/Parameters.html#learning_rate), which adjusts the contribution of each estimator to the ensemble. ``LightGBM`` plays nicely with ``scikit-learn``, and we can combine the relevant functionalities from both packages to perform effective model learning.\n",
    "\n",
    "LightGBM provides its own functionality to perform cross validation (CV) with given parameter choices through a function called ``cv``. \n",
    "\n",
    "**Listing 5.5**: Cross Validation with LightGBM and scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/H0lEQVR4nO3dd3ykd3Xo/895phf1stpe7F0XcMFeTA2YlpjgYEKoIQklucS5lxjyS24upJLkld+FJCQhgOPr+BITApgSCDYBE4ptIMb2rsuu7bXXXm/VFvUy0vSZc/94ntHOakfSaFejGWnO+2W9VvPMzDNnHsk6823nK6qKMcYY02icegdgjDHGVGIJyhhjTEOyBGWMMaYhWYIyxhjTkCxBGWOMaUj+egewWN3d3bply5Z6h2GMMWaJPPzww8Oq2jP7+IpLUFu2bGH37t31DsMYY8wSEZEjlY5bF58xxpiGZAnKGGNMQ7IEZYwxpiFZgjLGGNOQLEEZY4xpSJagjDHGNCRLUMYYYxqSJShjjDENyRKUMcaYhtSUCSpXKNY7BGOMMQtoygSVzBbqHYIxxpgFNF2COjGe4uHDY/UOwxhjzAKaLkHdfO8BPvTlR+sdhjHGmAU0XYLqjIVIpPMUilrvUIwxxsyj+RJUNIACE6lcvUMxxhgzj6ZLUB2xIACj09k6R2KMMWY+TZegOr0ENZa0BGWMMY2s6RJUR9RaUMYYsxI0XYIqtaBGpyxBGWNMI2u6BFVqQQ0k0nWOxBhjzHyaLkFFgj5CfofBRKbeoRhjjJlHTROUiFwnIvtF5ICIfLjC/deKyISIPOZ9/Ukt4ylpCfsZmbIEZYwxjcxfqxOLiA/4DPA6oB/YJSJ3quq+WQ/9sapeX6s4KmmLBGyShDHGNLhatqCuAQ6o6kFVzQJ3ADfU8PWq1hENMp60hbrGGNPIapmg1gPHym73e8dme4mI7BGR74jI8yqdSETeLyK7RWT30NDQeQfWGQ9aJQljjGlwtUxQUuHY7AJ4jwCbVfUK4FPAv1c6kareqqo7VXVnT0/PeQfWFQsymbYEZYwxjayWCaof2Fh2ewNwovwBqjqpqlPe998GAiLSXcOYAOiOh0jnimTztnGhMcY0qlomqF3AdhHZKiJB4B3AneUPEJE+ERHv+2u8eEZqGBMAvS0hAMat3JExxjSsms3iU9W8iHwA+C7gAz6rqk+KyI3e/bcAbwF+S0TyQAp4h6rWfB+MrriboEaTWXpbw7V+OWOMMeegZgkKZrrtvj3r2C1l338a+HQtY6hkph6flTsyxpiG1XSVJOB0PT6rJmGMMY2rKRNURywAwMCk1eMzxphG1ZwJKmotKGOMaXRNmaACPodIwMew1eMzxpiG1ZQJCtyCsVaPzxhjGlfTJqj2aMDWQRljTANr2gTVFgkwkcrXOwxjjDFzWDBBedtmrDqdsSAJq8dnjDENq5oW1AER+WsRubTm0SyjzliQqYy1oIwxplFVk6AuB54BbhORB7ytL1prHFfNdcVD5ApKKluodyjGGGMqWDBBqWpCVf9JVV8K/D7wp8BJEfmciFxY8whrpKesHp8xxpjGU9UYlIi8UUS+AXwS+ASwDbiLWXX2VpJSRfNBqyZhjDENqZpisc8C9wB/rar3lx3/moi8ojZh1V6Pl6BOTViCMsaYRlRNgrq8tKngbKp60xLHs2w6vIKxAwlLUMYY04iqmSTRKyJ3iciwiAyKyDdFZFvNI6uxTq8e39CklTsyxphGVE2C+iLwFaAPWAd8FfhSLYNaDm2RAAIM255QxhjTkKpJUKKqn1fVvPf1r0DNd72tNccRYiE/o9PWgjLGmEZUzRjUPSLyYeAO3MT0duA/RKQTQFVHaxhfTbWE/YwlrZqEMcY0omoS1Nu9f39z1vH34SasFTse1RoOMJ6yBGWMMY1owQSlqluXI5B6aIsGODwyXe8wjDHGVLBgghKRAPBbQGnN073A/1HVFd/06IgEeDKdR1URkXqHY4wxpkw1kyT+EbgauNn7uto7tuJ1xIKkcgWrx2eMMQ2omjGoF6rqFWW3fygie2oV0HLqigcpKpycSHNBb7ze4RhjjClTTQuqICIXlG54i3RXRZOj2ysY2z+WrHMkxhhjZqumBfV7uFPNDwICbAbeW9OolkmpYOzx8VSdIzHGGDPbvAnK2033CmA7cBFugnpaVVfF6tbe1jAAA1bR3BhjGs68XXyqWgDeqKoZVd2rqntWS3IC6Gv1KppbgjLGmIZTTRff/SLyaeDLwMyiIVV9pGZRLZM1rREABqxgrDHGNJxqEtRLvX//vOyYAq9e+nCWV9DvEA/5GZnK2FooY4xpMNUkqF9X1YPlB1bDdhsl7dEAk6k809kC8VA1l8MYY8xyqGaa+dcqHPvqUgdSL52xIFOZPJNWk88YYxrKnE0GEbkYeB7QJiJvLrurFQjXOrDl0hUL8vTJBJPpHOuI1DscY4wxnvn6tC4CrgfagV8oO54A/lsNY1pWPS1hsoUiJ8dTXNzXWu9wjDHGeOZMUKr6TeCbIvISVf3pMsa0rEpTzY+N2mJdY4xpJNXMCjggIn8AbCl/vKq+r1ZBLae+Nre38tiYJShjjGkk1SSobwI/Br7PKqnBV25TZxSAUxOWoIwxppFUk6Ciqvq/zuXkInId8EnAB9ymqh+b43EvBB4A3q6qlWYN1sxGL0ENTWUpFhXHsbVQxhjTCKqZZv4tEfn5xZ7Yq+P3GeD1wKXAO0Xk0jke93Hgu4t9jaXQHg0SDfpIpHMkMvl6hGCMMaaCahLUB3GTVFpEJkUkISKTVTzvGuCAqh5U1SxwB3BDhcf9NvBvwGDVUS+hWNBHPORnKm1roYwxppEs2MWnqi3neO71wLGy2/3Ai8ofICLrgV/ELZv0wnN8nfPi9zm0RQKMTmeZTFuCMsaYRrFgC0pcvyIif+zd3igi11Rx7kqDOTrr9t8D/8urmj5fDO8Xkd0isntoaKiKl16crliQRCbPhLWgjDGmYVTTxXcz8BLgl73bU7hjSwvpBzaW3d4AnJj1mJ3AHSJyGHgLcLOIvGn2iVT1VlXdqao7e3p6qnjpxemKh8jmiwxaVXNjjGkY1czie5GqXiUijwKo6piIBKt43i5gu4hsBY4D7+B0ksM719bS9yJyO/AtVf33KmNfMmu8jQuPjdrW78YY0yiqSVA5b6adAohID1Bc6EmqmheRD+DOzvMBn1XVJ0XkRu/+W8497KW1rt1NUCdsLZQxxjSMahLUPwDfAHpF5C9xu+L+qJqTq+q3gW/POlYxManqe6o5Zy2s73CLxA4nMuQLRfy+ano+jTHG1FI1s/i+ICIPA6/BnfjwJlV9quaRLaONHe5i3cl0nrFkjp6WUJ0jMsYYU9UOfar6NPB0jWOpm/ZogGjQx1Qmz1gyawnKGGMagPVlAfGQ312sm8kzMpWtdzjGGGOwBAVANOinJexWkxidtgRljDGNoJqFujERcbzvd4jIG0UkUPvQlk/Q79AaCZDI5BmdtrVQxhjTCKppQf0ICHtliX4AvBe4vZZB1UNXLEg2X2RgMkOxOLvghTHGmOVWTYISVU0CbwY+paq/iFudfFXp9SZGTKRyVvLIGGMaQFUJSkReArwL+A/vWFWz/1aSUjWJRDrHiI1DGWNM3VWToD4EfAT4hlcJYhtwT02jqoP13lqoqYxNlDDGmEZQzULd+4D7ALzJEsOqelOtA1tuGzrcFpTN5DPGmMZQzSy+L4pIq4jEgH3AfhH5n7UPbXm1Rbydda0FZYwxDaGaLr5LVXUSeBNuXb1NwK/WMqh6KF+sO5bMomoz+Ywxpp6qSVABb93Tm4BvqmqOszceXPGiQR+t4QATyRzZfJHJdL7eIRljTFOrJkH9H+AwEAN+JCKbgclaBlUPsZCfjliAiXSOfLHImHXzGWNMXS2YoFT1H1R1var+vLqOAK9ahtiWVcjv0BMPoQoTSZtqbowx9VbNJIk2EflbEdntfX0CtzW1qogI69rdfaFGk1mbKGGMMXVWTRffZ4EE8DbvaxL451oGVS+butwENTads5p8xhhTZ9VUhLhAVX+p7PafichjNYqnrjpjIeIhP2PJLKPTOVQVEal3WMYY05SqaUGlROTlpRsi8jIgVbuQ6icW8tMZCzI6nSWdK5DMFuodkjHGNK1qWlA3Av8iIm3e7THg3bULqX5iQT+d0SBPnpxAVRmdzhILrbqyg8YYsyLM+9dXRHzAr6jqFSLSCuAt2l2VYiEfHbEAuYK6u+tOZ9nYGa13WMYY05Tm7eJT1QJwtff95GpOTuC1oGJBAEans5yaWJU9mcYYsyJU03/1qIjcCXwVmC4dVNWv1yyqOomF/HRE3QQ1lsxxbDRlEyWMMaZOqklQncAI8OqyYwqswgTlIxr0EfQ7jE5nmcrkGU/m6PBaVcYYY5ZPNdttvHc5AmkEkYAPv8+hMxpkLOku1O0fS1mCMsaYOqimksTnRKS97HaHiHy2plHViYjQHg3QEQvM1OLrH0vWOSpjjGlO1ayDulxVx0s3VHUMeEHNIqqzjmiQzliQ6WyBTL5A/1jKtt4wxpg6qCZBOSLSUbohIp1UN3a1InVEg3SWJkpM52bGoYwxxiyvahLNJ4D7ReRruJMj3gb8ZU2jqqOOWGBmzGl0OktfW9jGoYwxpg6q2W7jX4BfAgaAIeDNqvr5WgdWL52xIG3hAI64Vc3BxqGMMaYequqqU9V9wL4ax9IQOqJBHEdojwbLJkrYeihjjFlu1YxBNZVwwF0L1RkNzrSgbBzKGGOWnyWoCjpiQXf791SOQtGdwdc/ZmWPjDFmOVmCqqA0k08Vxm0cyhhj6mLOMSgRSeDO2qtIVVtrElED6IwF6G0NA3BqMk1XPMSxsaSNQxljzDKaM0GpaguAiPw5cAr4PCDAu4CWZYmuTjqiQTqiAcIBh+PjKZ63ro3pTIGhRGYmcRljjKmtarr4fk5Vb1bVhLflxj/iTjtftTqiQUSEdW0RToynZ44fHrFuPmOMWS7VJKiCiLxLRHwi4ojIu4Cq9kIXketEZL+IHBCRD1e4/wYR2Ssij4nI7vKt5eupLRLA5wjr2yNMpHJMZ/IAHB6eXuCZxhhjlko1CeqXcatHDHhfb/WOzcvbjfczwOuBS4F3isilsx72A+AKVb0SeB9wW9WR15C7DirAuvYIACfG3Rl8JyZSpHNV5WZjjDHnqZpKEodV9QZV7VbVHlV9k6oeruLc1wAHVPWgqmaBO4AbZp17Sk9XYo0xz6SM5dYeDdLTEsLvCMe9BKUKR0etm88YY5bDfLP4PsX8s/huWuDc64FjZbf7gRdVeJ1fBP430Au8YY5Y3g+8H2DTpk0LvOzS6IwG8TlCX1uYExOnx6EODU+zY82qniNijDENYb5SR7vP89yV5mOflfBU9RvAN0TkFcBfAK+t8JhbgVsBdu7cuSytrI5YAID17REeOjRKJl8g5PdxZGTappsbY8wymG+a+efKb4tIi3tYp6o8dz+wsez2BuDEPK/3IxG5QES6VXW4yteomQ5vy4117REUODmRZktXjOlMgcFEhjU23dwYY2qqmh11ny8ijwJPAPtE5GEReV4V594FbBeRrSISBN4B3Dnr3BeK1xQRkauAIDCy2DdRC53e9hp9rWFETk+UAJvNZ4wxy6Gaaua3Av+fqt4DICLXAv8EvHS+J6lqXkQ+AHwX8AGfVdUnReRG7/5bcNdT/ZqI5IAU8PaySRN1VSoaC9DbEpq1HmqaF23rqldoxhjTFKpJULFScgJQ1XtFJFbNyVX128C3Zx27pez7jwMfrzLWZdcRDZLMpljXFmHv8QnyxSJ+x+HkRJp0rkA44Kt3iMYYs2pVsw7qoIj8sYhs8b7+CDhU68AaQWkX3XXtEQpFZXAyA7jTzQ8MVjsUZ4wx5lxUk6DeB/QAX/e+uoH31jKoRtHbEgJgXbs7IaJ8HOrBQ6MzW3EYY4xZevOtg/q8qv4q8GtVrHlalda2uYkpGvTTEQ3QP55ip3ffZCrHE8cnuGJje93iM8aY1Wy+FtTVIrIZeJ+IdIhIZ/nXcgVYT93xEEG/e4k2dkQ5MZ46o9X00KFRcoVivcIzxphVbb4EdQtwN3Ax8PCsr/NdxLsiOI7MrHfa0BkhV1AGJk/P5pvK5NlzbLxO0RljzOo2Z4JS1X9Q1Utwp4dvU9WtZV/bljHGulrndfNt6IgCcGzWzrq7Do9ZAVljjKmBaorF/lbpe68mXlPp8xJUJOCjpyVE/2jqjPvTuQKPHB2rR2jGGLOqVTOLr9yNNYmiga1ti8x8v7EjwsmJ9FnjTo8dGyebt7EoY4xZSotNUE1XITUS9M2UPdrYEaWgysmy6uYAmVyRJ09M1CM8Y4xZtRaboH6hJlE0uFI337r2CI7AsQp7Qj12bJyirYsyxpglU3WC8rZjf5uI/GwN42lI67xuvqDfYU1r+KyJEgDjyRwHrYisMcYsmTkTlIg8VPb9fwM+DbQAfyoiH16G2BrG2vbTW2ts7IgyOJkhkz975p5NljDGmKUzXwsqUPb9+4HXqeqfAT8LvKumUTWYrljw9ILdTnd/qONjqbMed3wsdcY6KWOMMeduvgTleBUkugBR1SEAVZ0G8ssSXYMQkZmyR31tYfyOcKxCggJ41FpRxhizJOZLUG2crhrRKSJ9ACISpwln85Wmm/sdh3XtkYoTJQD2n5piIpVbztCMMWZVmq+SxJayChLbVPWUd1cR+MXlCa9xrDtjHCrCyHS2YiIqqnLXnhMVx6iMMcZUb7HTzFHVpKo2xX5Q5fra3K3fAS7qa0EEHu+vvPZpKJHhO4+fsmnnxhhzHhadoJpVyO+jO+7uD9USDnBhT5wnTkzMWc380PA09z4zSIPsYG+MMSvOfNPMQ8sZyEqwvuN02aMrNrSTyRfZP5CY8/F7jk3wmFU7N8aYczJfC+qn4G5cuEyxNLz17acT1Lr2MF3xIHuPTczbSrr/uRGrdm6MMedgvgQVFJF3Ay8VkTfP/lquABvJurIEJSJcuaGdoakMJybmXvuUzRd55IhNPTfGmMWaL0HdCLwYaMetwVf+dX3NI2tA8ZCf9ujp9csX9bUQ8jvsXaAb79Fj46Sy1ooyxpjF8M91h6r+BPiJiOxW1f+7jDE1tHXtEcaT7vTygM/h0nWt7Dk2zlQmTzxU+XJm80UeOTrGyy7sXs5QjTFmRatmFt/nReQmEfma9/XbIhJY+GmrU/k4FLiTJYrKglu/P2atKGOMWZRqEtTNwNXevzcDVwH/WMugGtnsBNUWCbCjN86e/nGS2bkrQGXzRR62sShjjKlaNQnqhar6blX9off1XuCFtQ6sUbVHA0SDvjOOvWhbF/mCLpiA9vSPMzyVqWV4xhizalSToAoickHphohsA5q2r0pEzlgPBdAZC3JxXwt7+ieYzszfivryrmM8NzRV6zCNMWbFqyZB/U/gHhG5V0TuA34I/G5tw2ps62Z18wFcs7WToiq7Do/O+9xsvshde07w4MERqzJhjDHzmHMWX4mq/kBEtgMX4VYxf1pVm7qfavY4FEB7NMila1t54vgkV2/uoCU89zwSVXcB7/BUltddumZmryljjDGnVfWXUVUzqrpXVfc0e3IC6ImHKiaVa7Z0oigPHZq/FVXyzECCr+w+xmTatucwxpjZ7KP7OXAcOWP7jZLWSIDL1rfx5InJqnfWHUpk+NKDR+kfq7y/lDHGNCtLUOdoe29LxeMvuaCLaMjH954aoFDldhvJbIGvP3LckpQxxpSZr5r5VfN9LWeQjejC3jg+5+yNhUN+H6++qJeRqeyCEybKFYrKt/aetN14jTHGM98kiU94/4aBncAe3EkSlwMPAi+vbWiNLRzwsbU7xoHBs6eMb+uJc9GaFnYdHuXC3vjMPlILSWUL3LXnBG/budEmThhjmt58W76/SlVfBRwBrlLVnap6NfAC4MByBdjILllbuZsP4JU7egj5fXxv38CidtYdSmT43r4Bm4JujGl61XxMv1hVHy/dUNUngCtrFtEKsqUrRihQ+RJGgj6uvaiHwUSGRxe5aeEzAwm+/9QgmXzTroc2xpiqEtTTInKbiFwrIq8UkX8Cnqp1YCuB3+fMOVkCYHtvnG3dMR44OLLosaUnjk/wrw8c5eiITZwwxjSnahLUe4AngQ8CHwL2eccWJCLXich+ETkgIh+ucP+7RGSv93W/iFxRdeQN4uK+uROUiHDtRT2IwA+fHlx0t91kKse/PdLP9/cNkM0XzzdUY4xZUapJUL+pqn+nqr/off0d7maG8xIRH/AZ4PXApcA7ReTSWQ87BLxSVS8H/gK4dXHh19+Gjggt4bnnmrSEA7zsgm6OjibZfypxTq/x+PEJvvDgEU7Ns3OvMcasNtUkqHdXOPaeKp53DXBAVQ+qaha4A7ih/AGqer+qlkqAPwBsqOK8DUVE2LFm7lYUwGUb2uhrDfOjZ4fPeU+o8WSOL+86xq7Do4uadGGMMSvVfOug3ikidwFbReTOsq97gZEqzr0eOFZ2u987NpdfB74zRyzvF5HdIrJ7aGioipdeXpetb6u4JqrEEeE1l/SSyRe4Z//iu/pKiqr85Nlh7nu28a6BMcYstfnWQd0PnAS6Ob0mCiAB7K3i3JX+Ylf8yywir8JNUBXXVqnqrXjdfzt37my45kNHLMhVmzrmXZjbHQ/xkm1d/NdzI8SeGeYVO7oRmTupzeexo+P0xEM8f33buYZsjDENb84EpapHgCMi8logpapFEdkBXAw8PtfzyvQDG8tubwBOzH6QiFwO3Aa8XlWraZk1pGu2dvL0qUkS6bn3g7p6cwfJbIFHj40TCji8eFvXOb/eD58epCMWrFhZ3RhjVoNqxqB+BIRFZD3wA+C9wO1VPG8XsF1EtopIEHgHcGf5A0RkE/B14FdV9ZnFBN5ogn6Hay/qnfcxIsLPbO/m0rWtPHholEePnvsW8IWi8q09J6wSujFm1VpwPyhAVDUpIr8OfEpV/0pEHl3oSaqaF5EPAN8FfMBnVfVJEbnRu/8W4E+ALuBmr7srr6o7z/XN1NsFPTG29cQ4ODQ952NEhNdc7I5H/ejZYWIh/4KTLOaSzBb4yq5jbO6KsaY1RG9LmJ6W0LzjYcYYs1LIQgP2XjL678DfAb/uJZnHVfWy5Qhwtp07d+ru3bvr8dJVmUjm+JefHia/wEy7fLHI1x85zlAiw1uu3sCa1rO37zgXQb/Dxs4oW7tibO2JEQ9V8xnEGGPqR0QertQ4qaaL70PAR4BveMlpG3DPEse3arRFA7xgU8eCj/M7DtdfvpZI0Mdde08wlZl77Goxsvkizw1O8f2nBvjsTw5x7/5BktmlObcxxiyn+aaZf0REXqCq96nqG1X14wDeuqabli/ElWfnlo45a/SViwb9/MLl68jmi3xr7wnyhaWtFlEoKo8eHeef/+swDxwcIbfE5zfGmFqa76/oIeCDIvKoiNwuIm8XkYWbBoZwwMfOzZ1VPbanJcR1z+tjYDLDf9aoink2X+Snz43w+Z8e4cjI3ONjxhjTSBYcgwIQkRcA1wE/izvh4fvA3ar6UG3DO1ujj0GVZPNFbr//ENOZ6ipHPHJkjB8fGOaqTe38zPaemsZ2cV8LO7d0IgLFolJQRRVKy7LaI0EiQV9NYzDGmJK5xqCqGkFX1UeBR4H/LSKtwOuA3wCWPUGtFEG/wzVbu7jn6cGqHv+CTe1MpnM8cnSclnCAKze21yy2p08leHqeuoChgMPLL+zmsvVt57yY2BhjzldVCUpEXgpsKX+8qr6/RjGtGpetb+PhI2NMVrHVhojwih09TGXy3PfMEPGQnwt748sQ5dkyuSI/eGqQfScmefXFvfQu0QxDY4xZjAVH8kXk88Df4JYheqH3tWLXKi0nnyO8ckd31Y93RPi55/XR1xrm7idO8dDhUQp1LAx7ciLNFx86yn3PDNl2H8aYZVfNOqingEu1QfYgXyljUOUePDjC/c9VX8UpnSvww6cHeXZwip6WEK+7ZA09LaEaRriwlrCfay/qrbpVp6rWPWiMqcr5jEE9AfThFo415+CarZ2Mp3LsOzFZ1ePDAR8/f9lanh1McO/+Ie7YdZRrtnbywi2dOHX6o59I57lrzwk6ogE2d8XY1BVlQ0eEkP/0ZApV5cREmkeOjHFkZJrOWIi+thBrWsNsaI/SGvFb0jLGVK2aBNUN7BORh4BM6aCqvrFmUa0yIsJrL1lDIp3n2Gj1W7hv721hY0eUe/YP8sDBUfpHU/zc8/vqWh1iLJljLDnOY8fGAbdl1REN0hELMDCZOWNTxYHJNAOTaWBi5rEbOtzEtrYtTGcseEbCyuQLCELQX836cWPMaldNF98rKx1X1ftqEtECVmIXX0k6V+COh44yllxcgVdV5amTCe7ZP0jA5/DaS3vZ1l2fCRRLKRRw6ImHyOSLJNJ50rkCItAaDtDdEqI7FqS3NUxfW3jepFwoqtUfNGYFm6uLr6p1UI1kJScogMFEmi8/dGzBWn2VjE5n+c4TJxmeynJxXwuv2NFDJNAc65XiIT8tYT9Bv0PA5+CIkEjnSKTzTGXytEYCbOiIsL49QmcsSKGoFL3f7TWtYcJNcp2MWYnOOUGJyIuBTwGXAEHchbrTqtpai0AXstITFMATxyf43r6Bc3puvlhk16Exdh8ZJeT38codPexYE7exnXk4IqxtC7O5K0pXPIjfcfA5bldiWyRgycuYOjufSRKfxt3L6au408t/Ddi+tOE1l+eta6V/LMlTJ+deLDsXv+Pwkgu6uLA3zg+eHuDuJ09xZLSFV1/Ui99nYzeVFFU5Pp7i+Hiq4v3RoI+OaJCelhBr28OsbY3YhA5jGkC1lSQOiIhPVQvAP4vI/TWOa1UTEV598RoGExlGprLndI6elhBv27mRhw6N8uChUUamsrzh8rW0hgNLHO3ql8wWSGbdBPbYMfdY0O92I/octwXWGg7QEQvSGQsSDfrIFYrkCkWKChf1tdh1N6YGqklQSW9H3MdE5K9wp5vHahvW6hf0O/zspX3cseso5zoM6Ijw4m1d9LaE+O6TA9zx0DFee2kvW7ti9un/PM1emJxI5+dsgd1/YISL+lq4enMHLWE/x8dTHBtNMjCZJuT3EQv5iQV9dMVD9LWGz2qd5QpF/I4s+c8sXygyMp1leCpDT0uInnjorNew9WqmkVUzBrUZGMAdf/odoA24WVUP1D68s62GMahy9+wf5LGj4+d9nrHpLN/ae5LRZJaOaIArNrRzydpWm7K9zERY8ANHJOijPRIglSuQzBbI5ouEAg7dsRBd8SDrOyJc0BMnMKvLVr2ivs4cMxbTuQL9YymOjSU5PpZiZCo7M1EEoCse5KI1LXTGgpyaTHNyPM1gIo3PcYiFfEQCPtoiAbriIbrjQTpiQWJBv82QNDV3XrP4RCQCbFLV/bUIbjFWW4LK5At8/qdHSKTPf1PBfKHIs4NT7OkfZ2AyQ8AnbOmKcUFPnC3d0TMW1ZrGFvQ77FjTwvbeOBOp3EzSSeUKxIJ+4mE/0aCPbL5IJl8knSswlcmfc2t8PqGA475myE9rJEBbJEBnLMC69gjRoO3YbM7f+czi+wXcWnxBVd0qIlcCf16vhbqrLUEBHBic4q49J5b0nKcm0jx5YoKDw9MkswUcga3dMa7Y0M6Gjoh165gl0RkLsr49wuauKBs7o/POiCxtmDm7ZVgL+UKRoakMPfHQWZOHktk8pybStEUCtEeDFVuI+UKRkxNpTk6k6WsNs6EjMmfL1Zy/85nF91HgGuBeAFV9TES2LGVwze7C3jgX9sY5MDi1ZOfsa3MXuL5alZMTaQ4MTfHUyUmeG5qmKxbk8g1t7FjTYlOszXkZnc4yOp3l8eMT7nT+9jBrWsMEfDKzXm1sOsupyTQjU1n8PmFrd4ztvXE2dkZJZguMJbOMJ3P4HKEj6iaNkN/h5ESafq/l6IjQEnZbcNGgj6DfIeStiQv4HPw+IeA4TKRy7B9I8NzQFJlckaDfYVt3jAt646RzBZ4dmOLYWHKmpelzhPaou9Qg4BP8jkMmX+TkeOqMtYqRoI9t3TE2d8VY0xqiLRKY+ZCnqjNdtYo7azTkd2g5j4kzqkomXyTkd875w+REMseBoSlCfof2aICOaBAFTk2kODWRYTyVZXNnjAt6YzMt4VyhyPGxFENTGYI+h0jQR9jvI1csksq6XdLT2TxT6TzTmTzr2iO8Ykft9q+rpgX1oKq+SEQeVdUXeMf2qurlNYtqHquxBQUwlclzx0NHl6Srby75QpFnBtwuwMFEZqZVddGaFrZ0x5blk60xq0Eo4NAZDZLKFUik8xV3HehrC7Pd+/CZzhU5Pp7k+HiaRDpHW8RNGK3hAEUvGWXyBabSea+cWJZs3p080xZ1u1WjQXehetBLyLl8kUyhSC5fJOB3iAbcCTmZfJFnBhJnlB2bjyPCho4IBVVOTaQXtYPCJWtbue75fVU/fi7nVSxWRH4Z8InIduAmwKaZL7F4yM+br9rAV3cfI5mtbhfexfL7HC5d18ola1sYSmR4eiDB/lMJnhuaxucI69sjbOmKsrU7Rns0WJMYjFkNMjm3C3A+pybSnJpI8+Nnh8+6b3AyU+EZZ8sXlZGp7DkvR6lGUZWji6gRupyqaUFFgT/E3e5dgO8Cf6Gq1aXnJbZaW1Alg4k0X3u4n0xuefZfKqrSP5bi0PA0R0amZ+oE9raEuLivhR1rWojVsTitMaZx1boFZbX4GtDJiRRff+R4XTYJnEjleG5wiqcHEgwlMgiwrSfGVZs6WNsWtskVxpgZdeviE5E75zuhbbdRO2vbIrzzmk18f9/AnItDa6UtEuCqzR1ctbmD0eks+05O8sTxCZ4bmmZNa4grN7ZzYU/cyioZY2puvr6blwDHgC8BD+J275ll0hkL8tadG9jTP8F/HRiuS2uqMxbk5Rd286Ktnew7OcljR8f57pMD3Ocf4uK1rVy6tpXuePCsVlW+UCSZLbgDul7JIGOMWaz5ElQf8DrgncAvA/8BfElVn1yOwIxbs+/Kje1s7Y5x9xMnOTFel2E/Aj6HKza0c/n6No6NpXji+AR7+91NCwM+oSvmVh7IFZThqQyjyewZC0ZDfofueIgNHRE2dEToawvjd6wFZoyZ35wJyisMezdwt4iEcBPVvSLy56r6qeUK0Ljdbm+5eiP/dWCYh4+M1S0OEWFTZ5RNnVGS2TwHh6YZnsowPJXlwNAUfsehOx5kW0+M1kiAXL5IOl8knS1wajLNg4dGefAQOAIdsSA98RBdsSCRoG9mPUvI7669iAbctS425mVM85p3epaXmN6Am5y2AP8AfL32YZnZfI7wih09rO+I8NPnRkjnCmTyxbp0/QFEg36ev75tUc/J5AocH09xciLN8FSG/rEUT5+ae8sRnyP0xEP0tobcTQf9DtmC+57zRQX3P8DtVswVlGyhiOBuL98SDtAa8dMVC1lNQmNWoPkmSXwOeD7wHeDPVPWJZYvKzOmCnjgX9Jze7n0ileMLDx5Ztmnp5yMU8LGtJ862svgz+QKZnLt1Rbbg1pVLZQukcu6ixcFEhqdOTrK3f2LB8/scIehzZhY+lgjQGQ/S1xqmMxqcqUIQCviIh9wac5bAjGk887WgfhWYBnYAN5V1tQig9dpR15ypLRLgtZes4T/2nqx3KOck5PctWMS2qMp4MkeuUDxjJb2UzdvxO3JGrbRsvkginWMilWMwkeHUZJrnBqd4co4WZ9Dn0Brx0x4J0h4NEA/5vd909/5I0K30bTvwGrN85huDso+UK8SONS0cWjfNvhOT9Q6lJhwROmOLq2wR9Dt0xUN0xUMzLTZVnekizOSLZHJFpjJ5EpkcU+k8E6kcI9MZDg5PMV+1l7DfoTMWpDMepCsWosOrc9YSdvd5UlVyBXVbh6XXyhcI+Xy0R91acqXHJbNuFfJIwDfzfGOMy0oErBLXXtTDifEU414lCHM2EZlpsbXM87iiKimv3FQpX0xnCkx6LbKxpFsg9cDAFE/kT38o8DtugdRMvjBvgisV4ZzKnFnDLehz6Iq7k0dKxX7bIwGK6tZqTKRzqDLTwisls2JRSeUKMzv8FopKvuhO9Z/O5ElmC0SDPrrj7n5TIb+7I3AyWyCdK1ir0DQsS1CrRMjv4/XPX8vXHj5GrrCyqoM0GkfkrPJO0aCfnpbQGcdKLaDxZI7RZJaxZJZcoUjY7yMUcAj5fYS9tWAhv49MvsBYMsd4MksqW+CCnhit4QDxsJ9kpsDwdIbhqQxPn0qw97g75hbwCfmCMvsn6nOEeMhPNl8klVtc7Ua/I2dU6gZ3UklPPER7NDDTjRrwO/hEEHGvScjvuGN2Yb/tLWaWhSWoVaSvLcy7X7qF+58b4amTkzXZvM6cJl4ii4X8rO+IVPWczV0LP6aoyth0lpOTaYYTGUJ+H60R/8z2DRPJHOOpLFPpPKGAj2jQRyToI+htb+E44HccokG3unUk4GM6m2d4KsPIlJscI0H3eSG/j/FUlqFEhqFEhqOjybOSVyUh/+nWXrc307IrFrLdd82SsgS1yrSEA/zc8/p4wcZ2Hjo8yngyRzLrdvNYwloZHJGZ8bOKOhd/ztZwgNZwgG3dCz+2WPTG6gpFikW39VYsKmlvzG4q447XDU9l2HdycqbF7nOE7niQzlgQv+NOZPE7bguwNEvT5wgtoQAtET/xoJ+Cuvfl8orfJ0SDPqJBPwGfkEjn3a9MjpD/9CSV1rD/vEptlcYifSL4HJl33E9VbVywjixBrVK9rWGuv3zdzO1iURlNZjk5nubkRIojI0mmMrXbe8qsXI4jhB1fVeNSqjozU3JwMsNAIs2x0dTMOFi+qDNjcwGfQ6GoTGXmXvtWrdKkkpawn45okO54aKbs1uBkmoFEholUjo5ogN6WML2tIabSeQ4OT3NoeJqJ1Omx2lJ8pdJcAqRzBdI5N6mG/I7XUva5ydXbODEW9JHKFZjOuGN9KW9tYibnjkF2xoJ0x4N0xUMUi8pkOkcinSeTL7pdqN6mjrni6Yk7RVX8joPPcZN7wOtuLT3e73PwO0I44KMjFpizIktR3W06BhNpAj5nZjmFz5Ez3lss5KMlHCDsdygqjCezjHpd0O3RIF2x4MyknlyhyHQmPxOnqvuh5KrN7fS2hM/7Z1pJTROUiFwHfBLwAbep6sdm3X8x8M/AVcAfqurf1DKeZuY44v1PHOKyDW2kcwW+t29gSXfxNc1HRGiPBmmPBtmxZr6pJ6e5SSpPMpvHV5a8SjUcp7N5cgUlHvLTGnbHvLL5IhOpHBPJHBNpd9ZlIpNnbDrHoeHpsyal+B2hNRLg6GiSQnF85rjPETZ2RHj+ulYUyBfcRFqa3ZkrKEVVOmJBIn63mkkmX5hpOR6Zmma6wn5tAZ8QCfgIBXyE/A4OcGh4mn0nz5xZ64g7GSZX1DMmyPgdmdk9t+Ddly8W551s4xOhKx6ktzVE0Od470VJZHKcmkgvaiw64HNft9LrhfwOqpAtVF6isbYtzFt3bqz6tRajZglKRHzAZ3Dr+fUDu0TkTlXdV/awUdwNEN9UqzhMZeGAj+svX8ue/gl+9MzQonbRNOZ8+ByZ6a6brT1a+Tn+oEM06Gdt29ljfflikbFpt8tRcfcy64wGcRz3j+7otNuSCAd8bOqMnvfO0flCkUQmTzLjjuXFQnOv5ZvO5BmZzhLwuV2b0ZBvpnhysajkisWZFlPF1yq63Z/ZgruYvZRQk9kCg4kMA5Npnh2YolBUr0vVHXu8ZG0ra9vCrGkNz3wgmMrkKRaVcMBtHfsdIZktkEjnmEznCfjc5RydUbf82Hgyx8i0O2PVEWbGW8NeAWgR2NId45U13PK9li2oa4ADqnoQQETuAG4AZhKUqg4CgyLyhhrGYeZQKka7ri3MPfsH61aM1pjz4XccelpCZ82yBK9c1hz3nfPr+Rw6okE65kim5Up/1CtxHCHkzN+N6ncc/EGIcPbjqm2xAnTPNZ45j5ZwgI2d87/Ji/ta6W2tTfceQC0X467H3a6jpN87tmgi8n4R2S0iu4eGhpYkOHNab2uYt+3cyM9ftpbWCp9qjTGmHmrZgqrUZj2nfiRVvRW4Fdwddc8nKFOZiHBRXwsX9MQ4PJJkOpNnOptnKp3nxHhqZit4Y4xZLrVMUP1A+cjZBuBEDV/PLAG/z+HC3vhZxyfTOY6OJNl3YnLZd/k1xjSnWiaoXcB2EdkKHAfegbvxoVmBWsMBnr++jeeta+XISJL7nxthYNLGrIwxtVOzBKWqeRH5APBd3Gnmn1XVJ0XkRu/+W0SkD9gNtAJFEfkQcKmqrs6qp6uAiLClO8bmrijHRlP0jyfd9S+TaZIVpt8aY8y5quk6KFX9NvDtWcduKfv+FG7Xn1lhRIRNXVE2dZ2e5ZMruHXh0rkCuYIS9nbHDfl9jEy7GxQeG01yasKSmTFmYVZJwiyZ0oLL1vDZMwF7W8L0toS5alMH4G5UOJHMMZbMMTCZ5tREmsHE4hYXzuaIsLkrSls0QP9okuGp7DmfyxhTf5agTF2E/D56W330toa5qM9dz1EoKulcYWYRYKGoDHmLEQcSGRyBzmiQDq/8irsFhTvTsK8tzAU9cSLB0+tFpjJ5+seSjE5lGfe2yUik86RzVpfQmJXAEpRpGD7n7G0uYiE/W7pj53S+eMjPxX1nb/xc2j+ptB9SqTbZ6Q0GC2Tzbg05n1fw1C2U6pbYGU/mqqr4bYw5P5agTNNxHJl3hf9C0rkCzw5M8eSJCU5OuDMZo0EfHbEgPpGZ0jGFotIWCbDW23ww4HNI5QqksgX3vmiAzqi7xXwyW2BkKsvwVIaxZHam6GgmX0RhpsWX9Qp1GtMMLEEZs0jhgI/LNrRx2YY2JlI5Qn7nrMrfpS0dqt3Yrz0K69oX3lMqXygymswynMgykcoRCfqIh9yq3qlsgZMTaU5NphipMP6mCopSKLrnsVagaXSWoIw5D5UKnsLp7eWXmt/nzEw4qWQx3aHFopLJFzk+nuTQcJLDw9NMZdwK4yG/Q8jvEPT7vH8diqpMZwoz+4tZgWFTa5agjGlSjiNEgj4u7G3hwt4WVNWrir1wiU5VZTpbYDKVYzKdYzKV97o2c0xnChRVyRfc8yWzBeuWNOfEEpQxBnBbfX5fdbvHisjMJnjrmL9rslhUprPujMtCUWe2og/5HfJFJZv39mIqettJeFtKAN74mzKVKTA2nWUs6W7/kEivns02gzOtVYe0twGicVmCMsbUlOMILeEALRXWx5V2lY0tcjeIdK7AyHSW4USGwyPTHBlJVt3lGA366GsLEw/5iXhbzBeKOjMpJZl1d8hNZPJMZ/Jlu/cG8PvcnWVLezNFgj5iQf/MWGBr2N1xNxL0UfRapIWizkxyUdxNC0trBgO+M7ecV1Um03lOTaQZSmQQYWZ3XXfphbv8IpHOsf9UYtWPI1qCMsasOOGAj/XtEda3R7hiYzuZfIHDw0lGp7OEAu4W6X6fkMkVZ5YTtEb8bOyM0hMPnZEUGonI6c0cS+sD5/KyC7vZc2ycx/rHyRfcXYA7o0E6ogHiYbd1Gwt5yddbPpHOubND016rNRLwEQ15STbgm2nNFVU5Pp7i6GiSY6NucWj3nD4CPod0rkgqmyccqOWOTSC6wvqGd+7cqbt37653GMYY0xDyhSKOCM4cu/KeL1WteUIXkYdVdefs49aCMsaYFayaSS3no56tzdq+M2OMMeYcWYIyxhjTkCxBGWOMaUiWoIwxxjQkS1DGGGMakiUoY4wxDckSlDHGmIZkCcoYY0xDsgRljDGmIa24UkciMgQcOc/TdAPDSxDOamLX5Ex2Pc5m1+RMdj3Odq7XZLOq9sw+uOIS1FIQkd2V6j41M7smZ7LrcTa7Jmey63G2pb4m1sVnjDGmIVmCMsYY05CaNUHdWu8AGpBdkzPZ9TibXZMz2fU425Jek6YcgzLGGNP4mrUFZYwxpsFZgjLGGNOQmi5Bich1IrJfRA6IyIfrHc9yE5GNInKPiDwlIk+KyAe9450i8j0Redb7t6PesS4nEfGJyKMi8i3vdrNfj3YR+ZqIPO39rrzEron8jvf/zBMi8iURCTfbNRGRz4rIoIg8UXZszmsgIh/x/tbuF5GfW+zrNVWCEhEf8Bng9cClwDtF5NL6RrXs8sDvquolwIuB/+Fdgw8DP1DV7cAPvNvN5IPAU2W3m/16fBK4W1UvBq7AvTZNe01EZD1wE7BTVZ8P+IB30HzX5HbgulnHKl4D7+/KO4Dnec+52fsbXLWmSlDANcABVT2oqlngDuCGOse0rFT1pKo+4n2fwP3Dsx73OnzOe9jngDfVJcA6EJENwBuA28oON/P1aAVeAfxfAFXNquo4TXxNPH4gIiJ+IAqcoMmuiar+CBiddXiua3ADcIeqZlT1EHAA929w1ZotQa0HjpXd7veONSUR2QK8AHgQWKOqJ8FNYkBvHUNbbn8P/D5QLDvWzNdjGzAE/LPX7XmbiMRo4muiqseBvwGOAieBCVX9T5r4mpSZ6xqc99/bZktQUuFYU86zF5E48G/Ah1R1st7x1IuIXA8MqurD9Y6lgfiBq4B/VNUXANOs/q6reXnjKjcAW4F1QExEfqW+UTW88/5722wJqh/YWHZ7A24zvamISAA3OX1BVb/uHR4QkbXe/WuBwXrFt8xeBrxRRA7jdvm+WkT+lea9HuD+f9Kvqg96t7+Gm7Ca+Zq8FjikqkOqmgO+DryU5r4mJXNdg/P+e9tsCWoXsF1EtopIEHcA7846x7SsRERwxxaeUtW/LbvrTuDd3vfvBr653LHVg6p+RFU3qOoW3N+HH6rqr9Ck1wNAVU8Bx0TkIu/Qa4B9NPE1we3ae7GIRL3/h16DO37bzNekZK5rcCfwDhEJichWYDvw0GJO3HSVJETk53HHHHzAZ1X1L+sb0fISkZcDPwYe5/SYyx/gjkN9BdiE+z/jW1V19mDoqiYi1wK/p6rXi0gXTXw9RORK3EkjQeAg8F7cD7TNfE3+DHg77kzYR4HfAOI00TURkS8B1+JuqzEA/Cnw78xxDUTkD4H34V6zD6nqdxb1es2WoIwxxqwMzdbFZ4wxZoWwBGWMMaYhWYIyxhjTkCxBGWOMaUiWoIwxxjQkS1BNSERURD5Rdvv3ROSjS3Tu20XkLUtxrgVe561ele17Zh2/tlSRvEav+x4RWVd2+7alKjgsIvd7/24RkV9einOWnfsPKr1WrYjIm0TkT87heWe8dxHZKSL/sEQxnfGzO89zXe9NOzc1ZAmqOWWAN4tId70DKbfISse/Dvx3VX1VreKZw3twS90AoKq/oar7luLEqvpS79stwKISVBXX7owEVfZatfL7wM3n8LwtlL13Vd2tqjctUUzvoexnV415rut/4FYgiZ5vUGZulqCaUx64Ffid2XfMbgGJyJT377Uicp+IfEVEnhGRj4nIu0TkIRF5XEQuKDvNa0Xkx97jrvee7xORvxaRXSKyV0R+s+y894jIF3EXD8+O553e+Z8QkY97x/4EeDlwi4j8dYX31yoi3xCRfSJyi4g4c51rntfwedfiCe++3/Guy07gCyLymIhEROReEdlZulYi8pciskdEHhCRNd7xC7zbu0Tkz0vXtMJ7LR3/GPAz3mv8zmKunYj8u4g8LO6+Re/3jn0Mtwr3YyLyhVk/V/HOXXqfby87971yek+oL4iIlM7nXdu9IvI3Fd7HDiCjqsPe7R4R+Tcv/l0i8jLv+Cu9mB4TtyhtS4X3PtMiFpGPisjnROQ/ReSwiLxZRP7Ki/tucUt4ISJ/4r3OEyJyq/ceK/3sXuO97uPi7nMU8p5/2DvHT4C3ishNZe/3DgB1F5DeC1xf6Wdploiq2leTfQFTQCtwGGgDfg/4qHff7cBbyh/r/XstMA6sBULAceDPvPs+CPx92fPvxv3wsx23HlcYeD/wR95jQsBu3MKb1+IWI91aIc51uCvTe3ALmP4QeJN33724e/PMfs61QBq3IrcP+B7wlrnONc/xq4HvlZ23vdLrlt/GLYT5C973f1X2fr8FvNP7/sbSNa30cyl7D98qO171tQM6vX8jwBNAV/m5K7zWL3nXyAes8a7FWu/cE7j10xzgp7gfCjqB/Zxe5N9e4X28F/hE2e0vAi/3vt+EW2YL4C7gZd73ce/6z37vM7eBjwI/AQK4e1Qlgdd7932D078bnWXP/3zZz6T8ZxXGrbS9w7v9L7iVDsD9/+L3y85xAgjNfr/Au4BP1fv/59X8ZS2oJqVuBfN/wd2ErVq71N1PKgM8B/ynd/xx3K6Zkq+oalFVn8Utk3Mx8LPAr4nIY7hllbpwExjAQ+ruFzPbC4F71S3QmQe+gLtP0UIeUnfPrwLwJdw/rHOda67jB4FtIvIpEbkOqKbiexY3GQE8zOlr8hLgq973X6ziPLMt5trdJCJ7gAdwC3VuZ34vB76kqgVVHQDuw70mpXP3q2oReMx7P5O4HwBuE5E34yaJ2dbibtdR8lrg0178d+K2cFuA/wL+VkRuwv3Dn18gVoDvqFus9XHcpHq3d7z8d/BVIvKgiDwOvBp3w7zZLsIt/vqMd/tznPm79eWy7/fitrx+Bbf3oWSQRXYZmsWxBNXc/h53LCdWdiyP93vhdekEy+7LlH1fLLtdxP30WzK7fpbilt7/bVW90vvaqu5+OuC2AiqpVK6/GnO9ftWvoapjuJ/S7wX+B2duZjiXnHofrYECZ16T81HVtRO3luBrgZeo6hW49eLCVZx7LuU/7wLg95LINbjV8N/E6QRRLjXrdR0vplL861U1oaofw61nFwEeEJGLF4h1JiYvaZZf7yLgF5Ew7tjXW1T1MuCfqHwNFvrdKv+dfAPuTtxXAw+Lu2Eh3nlTVcRszpElqCambkHHr+AmqZLDuP8jgrv/TeAcTv1WEXHEHZfahtsl9F3gt8rGCXaIuwnefB4EXiki3eIOVr8T9xP+Qq4Rt2K9g1vc8yfznKvicXEnkDiq+m/AH+NuNwGQAFqqvRCeB3C70sCtmL6Q2a9R7bVrA8ZUNen9sX9x2X250vNn+RHwdm+cqwe3FTFnxWlx9xFrU9VvAx8CrqzwsKeAC8tu/yfwgbJzXOn9e4GqPq6qH8fttry4wntfrFIyGvZiLZ9RWn7up4EtIlKK81ep8Lvl/Q5tVNV7cCd+tON2RwLswO1GNTWyVJ/wzMr1Ccr+eOB+4vymiDwE/IC5Wzfz2Y/7P/sa4EZVTYvIbbhdMI94LbMhFtgeW1VPishHgHtwP/F+W1Wr2c7gp7iD7Zfh/gH+hqoW5zpXpeMicgXujrKlD3Ef8f69HXdyRgq3664aHwL+VUR+F3f218QCj98L5L2uutuBT1LdtbsbuFFE9uL+DB4ou+9WYK+IPKKq7yo7/g3vfezBbWn+vqqemqc104L7+xHGvV5nTbTBveafEBHxWjg3AZ/x4vJ7998IfEhEXoXbOtsHfAe3JVT+3h+dI46KVHVcRP4Jt8vvMO4WOyW3c+bP7r3AV70W0S7glgqn9OH+7Nq89/t3qjru3fcqTv9emBqwaubG1Ji4U5FTqqoi8g7cCRM31DuuWhKRTwJ3qer36x1LLYg7Q/OLqvqaeseymlkLypjauxp3koDgzoR8X33DWRb/P/CiegdRQ5uA3613EKudtaCMMcY0JJskYYwxpiFZgjLGGNOQLEEZY4xpSJagjDHGNCRLUMYYYxrS/wMyt5PSXnXAXwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from lightgbm import cv, Dataset\n",
    "\n",
    "trn_data = Dataset(Xtrn, label=ytrn)\n",
    "params = {'boosting_type': 'gbdt', 'learning_rate': 0.25, \n",
    "          'max_depth': 1, 'objective': 'cross_entropy'}\n",
    "\n",
    "cv_results = cv(params, trn_data, \n",
    "                num_boost_round=100,\n",
    "                nfold=5, stratified=True, shuffle=True)\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 4))\n",
    "m = np.array(cv_results['cross_entropy-mean'])\n",
    "s = np.array(cv_results['cross_entropy-stdv'])\n",
    "ax.fill_between(range(100), m + s, m - s, alpha=0.5)\n",
    "ax.plot(range(100), m)\n",
    "ax.set_xlabel('Number of boosting iterations (estimators)')\n",
    "ax.set_ylabel('Mean/std of 5-fold cross entropy');\n",
    "\n",
    "fig.tight_layout()\n",
    "pngFile = './figures/CH05_F19_Kunapuli.png'\n",
    "plt.savefig(pngFile, dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternately, LightGBM plays nicely with scikit-learn, and we can combine the relevant functionalities from both packages to perform effective model learning.\n",
    "\n",
    "In the experiment below, we combine ``scikit-learn``'s [``StratifiedKFold``](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) class to split the training data into several folds of training and validation sets. ``StratifiedKFold`` ensures that we preserve class distributions, that is, the fractions of different classes across the folds.\n",
    "\n",
    "**Listing 5.4**: Cross Validation with LightGBM and scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "import numpy as np\n",
    "\n",
    "# Initialize choices of learning rates, number of cross-validation folds\n",
    "n_learning_rate_steps, n_folds = 10, 10\n",
    "learning_rates = np.linspace(0.1, 1.0, num=n_learning_rate_steps)\n",
    "\n",
    "# Split the data into training and validation folds\n",
    "splitter = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=42)\n",
    "\n",
    "# Initialize some structures to save training and validation errors\n",
    "trn_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "val_err = np.zeros((n_learning_rate_steps, n_folds))\n",
    "\n",
    "# Train LightGBM classifier for different learning rates and plot the results\n",
    "for i, rate in enumerate(learning_rates):\n",
    "    for j, (trn, val) in enumerate(splitter.split(X, y)):\n",
    "        gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=10, max_depth=1, learning_rate=rate)\n",
    "        gbm.fit(X[trn, :], y[trn])\n",
    "\n",
    "        trn_err[i, j] = (1 - accuracy_score(y[trn], gbm.predict(X[trn, :]))) * 100\n",
    "        val_err[i, j] = (1 - accuracy_score(y[val], gbm.predict(X[val, :]))) * 100\n",
    "\n",
    "        \n",
    "trn_err = np.mean(trn_err, axis=1)\n",
    "val_err = np.mean(val_err, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEYCAYAAABiECzgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABBCklEQVR4nO3dd3hUxdfA8e9sKin03oOR3kkoggiKIEUEpIi0AIL0olIsFBERFBHpHaQIItJEpP5EqpJQDb33Gkp6n/ePDXkpCQSyuzebnM/z7MPu3b13ziVwMjt37hmltUYIIYTtmYwOQAghMipJwEIIYRBJwEIIYRBJwEIIYRBJwEIIYRBHowN4WM6cOXXRokWNDkMIISxq3759t7XWuR7fnqYScNGiRQkICDA6DCGEsCil1IWktssQhBBCGEQSsBBCGEQSsBBCGCRNjQELIVImJiaGy5cvExkZaXQo4iGurq4ULFgQJyenFH1eErAQdujy5ct4enpStGhRlFJGhyMArTVBQUFcvnwZLy+vFO0jQxBC2KHIyEhy5MghyTcNUUqRI0eO5/pWYrcJODIq5rm2C5HeSPJNe573Z2K3QxCuLk44V+ryxPboA/MMiEYIIZ6f3faAhRDGCQoKomLFilSsWJG8efNSoECBxNfR0dFP3TcgIIB+/frZKNK0zW57wEKIlIuPjyc6LBwXdzeiwsJxdnfDZHrx/leOHDk4ePAgACNHjsTDw4NPPvkk8f3Y2FgcHZNOLz4+Pvj4+Lxw28mJi4vDwcEh2ddpkfSAhUjn4uPjCbkZxLSm3ejtUpxpTbsRcjOI+Ph4i7bj5+fHRx99RN26dRkyZAh79+7llVdeoVKlSrzyyiucOHECgG3bttGkSRPAnLy7dOlCnTp1KFasGJMmTUry2Js2baJGjRpUrlyZVq1aERoaCpjLF4waNYpatWrx66+/PvE6rZMesBB2rocq+tT3h/23kV/6juTktj0AnNy2h7lt+9Fm8ki+Ktcg2f1m6PPPHcvJkyfZsmULDg4OBAcHs337dhwdHdmyZQufffYZv/322xP7HD9+nL/++ouQkBBKlChBz549H5lHe/v2bUaPHs2WLVtwd3dn3LhxTJgwgeHDhwPmubc7d+4EYOjQoY+8TuvsNgFHRsUkecHtwtXb/HfqMk1eq2j7oIRIg/KV8ub0Tv9Htp3e6U++Ut4Wb6tVq1aJX/vv379Pp06dOHXqFEopYmKSnqHUuHFjXFxccHFxIXfu3Ny4cYOCBQsmvv/PP/9w9OhRatasCUB0dDQ1atRIfL9NmzaPHO/x12mZ3SZgV5cn7zT5fNIKvpu/HhdnR9ZOHkjdqqUMiEwI23pWTzUyJBTvWr6JPWAA71q+RIdHvFAv92nc3d0Tnw8bNoy6deuyatUqzp8/T506dZLcx8XFJfG5g4MDsbGxj7yvtebNN99k6dKlz2wzqddpmdXGgJVSJZRSBx96BCulBlirPYDRfd+lR+u6REXH0mLAJPb+d8aazQlhF5zd3ei6dBLF69TA5OhI8To16Lp0Es7ublZt9/79+xQoUACABQsWvPBxqlevzq5duzh9+jQA4eHhnDx50hIhGs5qCVhrfUJrXVFrXRGoAoQDq6zVHpgnQU8c0o73G9cgLCKKt/tM5L9Tl63ZpBBpnslkwjN3Dnqtnc3UqJP0Wjsbz9w5UjULIiUGDx7Mp59+Ss2aNYmLi3vh4+TKlYsFCxbQtm1bypcvT/Xq1Tl+/LgFIzWO0lpbvxGl6gMjtNY1n/Y5Hx8fbYmC7LGxcbQZNI3ftx0gT47M/DXvU7wL50n1cYVIK44dO0apUjLElhYl9bNRSu3TWj8x985W09DeA5IcwFFKdVdKBSilAm7dumWRxhwdHVgytgevVy3FjaBgGvYYz6XrdyxybCGEsBSrJ2CllDPQFEhyUp7WepbW2kdr7ZMr1xNLJr0wVxcnVvzQl2rlXuLCtSAa9hzPzTvBFju+EEKkli16wA2B/VrrGzZo6xEebq6snTKAci8X5OT56zTuNYF7IeG2DkMIIZJkiwTclmSGH2whW2Z31k//GO/CeTh04iLN+k0kLCLKqHCEECKRVROwUsoNeBNYac12niVPjixsmPEJhfJmZ/fB07T+eApR0VK2UghhLKsmYK11uNY6h9b6vjXbSYnC+XKwfvrH5MrmyeY9R+jw6SxiY198aowQQqRWhirGU6JoPv6Y/jFZPDKx+n/7+HDUAosXJBEiI6hTpw4bN258ZNvEiRPp1avXU/d5MM20UaNG3Lt374nPjBw5kvHjxz+17dWrV3P06NHE18OHD2fLli3PEX3akaESMEDFEoVZO3kgbq7OLPp9Fx9/txRbzIUWwijWWD2mbdu2LFu27JFty5Yto23btinaf/369WTNmvWF2n48AY8aNYp69eq90LGe1+M3lKTmBhPIgAkYoEZFb1b80BdnJ0emLtvKl9NXGx2SEFbzYPWYxx9J1VNJqZYtW7Ju3TqioswXtM+fP8/Vq1epVasWPXv2xMfHhzJlyjBixIgk9y9atCi3b98G4Ouvv6ZEiRLUq1cvsWQlwOzZs/H19aVChQq8++67hIeHs3v3btauXcugQYOoWLEiZ86cwc/PjxUrVgCwdetWKlWqRLly5ejSpUtifEWLFmXEiBFUrlyZcuXKJXknXVxcHIMGDcLX15fy5cszc+ZMwFw+s27durz//vuUK1fuidepYbfFeFKrXvUyLB77IW0HT2fM7N/J4pGJgR3fMjosIZ5bUktzPexpy3Q9bd+n7ZcjRw6qVq3Khg0beOedd1i2bBlt2rRBKcXXX39N9uzZiYuL44033uDw4cOUL18+yePs27ePZcuWceDAAWJjY6lcuTJVqlQBoEWLFnTr1g2AL774grlz59K3b1+aNm1KkyZNaNmy5SPHioyMxM/Pj61bt1K8eHE6duzI9OnTGTBgAAA5c+Zk//79TJs2jfHjxzNnzpxH9p87dy5ZsmTB39+fqKgoatasSf369QHYu3cvgYGBeHl5sW3btkdep0aG7AE/0Oz1Kswa0RmAIT8sZ+7Kvw2OSAj78fAwxMPDD8uXL6dy5cpUqlSJI0eOPDJc8LgdO3bQvHlz3NzcyJw5M02bNk18LzAwkFdffZVy5cqxZMkSjhw58tR4Tpw4gZeXF8WLFwegU6dObN++PfH9Fi1aAFClShXOnz//xP6bNm1i4cKFVKxYkWrVqhEUFMSpU6cAqFq16iPJ9vHXLyrD9oAf6PB2TULDI+k/dgm9Ri/Ew82VNm9VMzosIVIsNQvRpmbfZs2a8dFHH7F//34iIiKoXLky586dY/z48fj7+5MtWzb8/PyeuUx7cisJ+/n5sXr1aipUqMCCBQvYtm3bU4/zrGs5D8peJlXy8sH+kydPpkGDR4vUb9u2zWolLzN0D/iBnm3eYFSfFmit6TxsDut3HDI6JCHSPA8PD+rUqUOXLl0Se7/BwcG4u7uTJUsWbty4wZ9//vnUY9SuXZtVq1YRERFBSEgIv//+e+J7ISEh5MuXj5iYGJYsWZK43dPTk5CQkCeOVbJkSc6fP59YtnLRokW89tprKT6fBg0aMH369MTC8SdPniQsLCzF+7+IDN8DfmBIl8bcCwlnwk8beG/QNH6fMpDXfEoaHZYQqZbc6jGRUTGpuhAH5mGIFi1aJA5FVKhQgUqVKlGmTBmKFSuWuIpFcipXrkybNm2oWLEiRYoU4dVXX01876uvvqJatWoUKVKEcuXKJSbd9957j27dujFp0qTEi29gXppo/vz5tGrVitjYWHx9fenRo0eKz+WDDz7g/PnzVK5cGa01uXLlYvXq1c/xt/H8bFKOMqUsVY7yRWmt6fP1Imb/tg0PNxc2zRqMT5nUj/MIYWlSjjLtSovlKO2CUopJn7bnvYbVCQ2PoknvCQSeloLuQgjrkAT8GAcHE3O/7ELj2hW5cz+MRj2/58ylm0aHJYRIhyQBJ8HJyZGl3/akjm9Jrt++T8Me47l8Qwq6i7QlLQ0fCrPn/ZlIAk6Gq4sTv/3QF9+yXpy/epuGPb/nlhR0F2mEq6srQUFBkoTTEK01QUFBuLq6pngfuQj3DHfuh/LGB+M4cvoKlUoVYdPMQWTxtO5qskI8S0xMDJcvX37mHFthW66urhQsWBAnp0dnlyR3EU4ScApcu3WP17uO5cylm9Ss9DJ/TP0It0wuRoclhLATMgsiFfLlysqfMz6hQO5s7DpwijafTCU65sk7aYQQ4nlIAk6hovlz8ueMT8iZ1YONuwPp+JkUdBdCpI7cCfccSnrl449pH5MjqweF8+V44n1L3FkkhMg4JAE/p0qligBJl/FLTWETIUTGI0MQFtZr9E/MX7Wd/05dJi5OljsSQiRPesAWNue3v5nzm7musHsmFyqXKoJv2WL4lPXCt4wXhfPlSLb8nhAiY5EEbGHjBrbG/8g5AgLPcf7qbXbsP8mO/ScT38+TIzM+ZYrhW9YL37Je+JTxIltmy9QWFULYF0nAFvbwskY37wQTcOQc/oHnCAg8i/+Rc9wICuaP7Qf5Y/vBxM95F85jTshlvPAtW4wKJQrLxTwhMgC5EeMFJDfb4VmzILTWnL18C//As4mJ+cDxC0+sTuvo6ED5lwviW7ZYYi+5pFc+TCZTqtoXQhhD7oRLo2JiYgk8fQX/hB6yf+A5jp29+sQ9/p7urviUMSfjr/u1lFkYQtgRScB2JCQsgv3HLuAfeC6xt3zp+v9XY4s+ME8SsBB2JLkELGPAaZCneyZe8yn5yJJI127dMw9bHDlnYGRCCEuSBGwn8uXKytt1KvF2nUpGhyKEsBCr3oihlMqqlFqhlDqulDqmlKphzfaEEMKeWLsH/COwQWvdUinlDFi0kG58fDzRYeG4uLsRFRaOs7tb4kyB9Myaq9wKIWzHatlKKZUZqA3MBdBaR2ut71nq+PHx8YTcDGJa0270dinOtKbdCLkZRHx8+r/99+Eku37HIZwrdSFf3X5SIlNYzeNTJZ+1XaSMNbuLxYBbwHyl1AGl1ByllMVu+YoOC2du236c3LaH+NhYTm7bw9y2/YgOC7dUE3ahYa3yvFLRm6B7ofy4ZJPR4Yh0ytXFCedKXZ54yDeu1LFmAnYEKgPTtdaVgDBg6OMfUkp1V0oFKKUCbt26leKDu7i7cXqn/yPbTu/0x8U9Yy0XpJTiqz7vAjBx0UZu3w0xOCIhREpZMwFfBi5rrf9NeL0Cc0J+hNZ6ltbaR2vtkytXrhQfPCosHO9avo9s867lS1QG6wEDvFqlBA1eKUtIWCTfzV9vdDginXl4DrqwLKslYK31deCSUqpEwqY3gKOWOr6zuxtdl06ieJ0amBwdKV6nBp3mf4fJMWPOrBvV19wLnvbLVi7fkP8wwjL8A89Ss8NXRoeRbll7ykBfYIlS6jBQERhjqQObTCY8c+eg19rZTI06yfvTv2LVp98yu3Vv4mIz3sWoSiWL0LK+L1HRsXw9a63R4Yh0YMVmf974YBzXb983OpR0y6rdRa31QeCJ2+8sxWQy4erpkfj62KbthN25x7Lew3h/xpgMV3d3RM9mrNq6jwVrdjKw41sUL5LX6JCEHdJaM3buOkZMXQVAl+a1iYiKTnLq46Xrd/Bwc5GSqi8o3UyazVvSm55r5+Dk6sKOWUv58+spRodkcyWK5qPj2zWJi4tn1PTVRocj7FBUdAydh81hxNRVKKUYN7A104d1IpOL8yOfi4iMptr7X/JSw0/w+3x2hpj+aQ3pJgEDeNf0ocvPP6KUYu2w79m94FejQ7K5Lz58B2cnR5Zv3MvBExeNDkfYkVt3gmnw4Xf8/Mce3DO5sGJCHwZ2fCvJb5KZXJ1ZPr43ObJ68OfOw4yWYa8Xkq4SMECl5m/RetJIABZ3+5Sjm7YbGo+tFcqbnR6t6wIwYspKg6MR9uLomSvU6jia3QdPUzBPNv6a/+kz644UyZ+TRd98iMmkGD1z7SOLDIiUSXcJGKBun07UH/wh8bGxzHy3Jxf3Bxodkk0N6dIYDzcX/tx5mF0HTj57B5GhbdodSG2/MZy7cpsqpYuya9EwKpYonKJ961Uvw6jeLQDw+3w2py/esGao6U66TMAAzb4ZQtV2zYgKDWNK487cPn/J6JBsJlf2zPRv3wCAYZNXPlHcXYgHZiz/H+/0m0hwaAQt6vmwdc4Q8uXK+lzHGNS5Ee+8Xpn7oRG0+ngKYRFR1gk2HUq3CdhkMtFx3reUeP0Vgq/fYvJbnQgNumt0WDYzsEMDsmdxZ+eBk2zc9Z/R4Yg0JjY2joHjltDvm8XExcUz9IMm/DyuB26ZXJ77WEop5n7ZleJF83Lk9BV6jFogv/RTKN0mYABHZ2d6rJxBgfIluXHiLNOafkB0RKTRYdlEZo9MDO7SGIBhU1bKVWqRKDg0ghYDJjF12VacHB2YO6oro3q3SFUlwcwemfj1+z54uLnwy4Z/mfzzZgtGnH6l6wQMkClLZvqsX0C2Qvk5u3sf89r1Jz4uzuiwbKJn69cpkDsbh05c5LctstSTgPNXb1Pbbwwbdv1HjqwebJw5iA5v17TIsUsVy8/skV0BGPLDcrYHnLDIcdOzdJ+AAbIVyEvfPxfgljUzB1dtZPmAURniK1ImV2c+7/42ACOnrSI2NmP84hFJ++fQaWq2/4qjZ65QwisfOxd+Qa3KxS3axrtv+vCxX0Pi4uJ5f8h0rtzMOMN+LyJDJGCA/GWK02P1LBydndk25Sc2fTfT6JBsolPTWngXys2pCzdYuHaX0eEIgyz98x/e7P4tt+6GUK96GXb89DkvFcptlba+6t2CulVLcfNOMO99Mo2oaKkZnJwMk4ABir9WHb9FEwBYNWQse39eY3BE1ufk5Mjwns0AGD1rjRTQzmC01oyasZpOn80iKjqWD1vVZe3kAWT1tF7ZVkdHBxZ/8yGF8mbn3//O8Mn4ZVZry95lqAQM4NO6CS0nfAHAT36fcHxr+u8Vtm5QlfLFC3H5xl1m/vqX0eEIG4mIjKbDpzMZPXMtJpPih8HvM+nT9jg6Oli97VzZM/PL+N44Ozky89e/WLh2p9XbtEcZLgED1Bv4AW8M7EpcTAwzWvTg8uFjRodkVSaTiVF9zJPlx81dR0hYhMERCWu7fvs+b3b/luUb9+Lp7sqqif3p3baeTQtU+ZTxYvKn7QHo/fVCDhy7YLO27UWGTMAA747/nCqtGxMZHMLkhn7cuXjF6JCsqmGt8tSo4M3te6FMXCxLF6Vnh09eolaH0ez97yxF8uVg2/zPaPhqeUNi6dy8Nl1b1CYqOpbWH08h6F6oIXGkVRk2AZtMJvx++p6Xa1fl/tUbTG7oR9jd9Fv3VCnF6L6ydFF6t37HIep0HsPF60FUK/cSOxd9QbmXCxoa08Qh7fAt68WFa0F0/GwmcXEyJ/2BDJuAAZxcXemxejb5yxTn2tFTzGjWjZjI9HujhixdlH5prZn882ZaDJhEaHgUrRtUZdOsQeTJkcXo0HBxdmLZd73Jlc2TzXuOMHL6KqNDSjMydAIGcM+WhT7r55Mlfx5Obd/Lgk4fp+u7xkb1kaWL0puYmFj6jlnEx98tJT5eM6zHOyz65kMyuTo/e2cbKZQ3O0vG9cBkUoyb+wdr/tpvdEhpQoZPwADZCxeg7/r5uHp6sG/5H6wcZLGVk9KcSqWK8O6bPkRFxzJm9u9GhyNS6V5IOO/0+5FZK7bh4uzIwjHdGfbhO2lyNZg6vqUY078VAF2GzeHE+WsGR2Q8ScAJClYoTY9VM3FwcmLLhDlsnTjX6JCsZmSv5jg4mJi/egenLkj5QHt15tJNXu30NVv+OULu7JnZPGsw7zWsbnRYTzWwQwPefdOHkLBIWn88NcPPyJEE/JCSb9Sk4/zvAFjx0Wj2/fqHwRFZx8NLF30p43EpltxNLLa4uSWpNl4qlJvfpwykjHcBdi76guoVvK0eR2oppZg9sguliuXn2NmrdBs5P0OUBUiOSksn7+PjowMCjC8as3HcdFYNHYejszP9Ny/i5drVjA7J4i5eC6L0O58SHRPL3mUjU1yAO6NzrtTliW1JLVZpy7aDQyPI7JHJJjFYyonz16jZYTTBoRGMHdiajzq+ZXRIVqWU2qe1fmKBYquuimyv6g/uwZ1L1/h76kKmv9ONT3auIH8ZyxYtMVrhfDno0bouk5ZsZuTUlayeNMDokOza273Nt7hrzDMStH7wp056mwaNfmx7UtvM+//784hk27a35Avmb2HzRn1Ay48m89mPv1KpZBHqVi1ldFg2Jwk4CUop2vw4gvtXb3Bw1UYmN/Rj8J6VZCuQvpZ5H9KlMfNWbWf9jsPsPniKVyq+bHRIaVZcXDwODsmP2G3cnbGWvbKEpnUrMfSDJoyds452Q6bz79KRFMqb3eiwbEoScDJMDg50WfIjE+u14+zufUxp5Mcn25eTKUtmo0OzmFzZM9OvXX3GzP6dLyb9xta5Q9Lk1XOjHT55iZ5fLWDXomHJfmbNpAEoZf7lbX4kPCepbQ89f9q2hJ9Fev6JjOjRjP1Hz7NpdyBtPpnK/+YOxdXFyeiwbEbGgJ8hNOgu39V8lxsnzlLyjZr0WT8fR+e0M78yte6HhFPi7SHcuR/G71MG0qBmOaNDSjMiIqMZPWstPyzaSGxsHNEH5qXJMWB7F3QvlBrtRnH+6m26tqjN9GF+RodkccmNAUsCToHb5y7xbY3mBN+4TdV2zfBbOCFVy7ekNRMWbmDoD8upWLIw/ywZnq7O7UVt/ecIfcYs4sylmyil6Nn6dcYMaIVbEjc3REbFWL3XllwbtmjbFg4cv8BrfmOIjIphxjA/urSobXRIFpVcApb/aSmQ06sQfdYvwMXDnb1LVrPms2+NDsmierZ+nfy5snLwuCxddPtuCF2Hz6Vhz+85c+kmZbwLsH3BZ0wc2i7J5AvYJAEm10Z6SL4AlUoWYcpnHQHoN3Yx/oFnDY7IzNpTDyUBp1DhymXpvmIaJkdHNo6bwbapC40OyWLMSxc1BTLu0kVaa5b8sYfyLT5n0e+7cHF25Ks+77L35xFUK/+S0eFlCB2b1qRH67pEx8Ty3qBp3LoTbHRIuLo44VypyxMPS/3ikwT8HMo0eI32s78B4PTOAIJv3ELHxxMZEmr39SP83vn/pYsW/Z7+i9Q/7OzlmzTpPYHOX8zm9r1Q6lYtxf7lXzGka2OcnOQ6tS2N/6Qt1cu/xKXrd2j/6UzDOgMhYRH8HXDc6u1YdQxYKXUeCAHigNikxkAellbHgB934LcNFKxUmkVdh3B6pz/etXzpunQSnrlz2PX46bI//6HjZ7MomCcbR9eMTTdfb5MTExPLj0s289XMNURERpM9izvfftSGDm/XlNkgBrp68y7V3v+SG0HBfNTpLcYOaG3V9mJiYgk8c4WAwHP4B55lb+A5jp29itbaYhdejbwRo67W+rYN2rGZUvVrMa1pN05u2wPAyW17mNu2H73WzsbV08Pg6F5c6wZV+W7+ev47dZmZv/5F//b1jQ7JagKOnKPHqAUcPnkJgLaNqvPdx++RO3v6mWZor/LnzsbSb3tR/8PvmPDTBnzKeNHyTV+LHFtrzdnLt/APPEvAkXP4B57jwPELT4zpOjo6UN4GdZTl+9ULcHF34/RO/0e2nd7pj4u79RY6tIUHSxc17z+JcXPX0aX5q3i6299dVk8TGh7JyGmrmLJ0C/HxmqL5czLl847Uf6Ws0aGJh9SqXJxxA1vz8XdL6TZiHqWL5af0SwWe+zi37gQTcOQ8ewPPEhB4Fv8j57hzP+yJz3kXzoNvWS98y3jhW7YYFUoUtsk3QGsnYA1sUkppYKbWetbjH1BKdQe6AxQubB/1CKLCwvGu5ZvYAwbwruVLVFi4XfeAARq9WoEaFbzZc+g0Py7ezBcfNjU6JItZv+MQ/cYs5uL1IBwcTHzUqQHDPnwH90wuRocmktCnbT38A8+x7M9/aPXxFHYvGkaWp6zmHB4RxYHjF/APNPdsA46c5dyVJ798586eGd+yxcwJt6wXVUoXJXuWpP/fRkbFJDncYKnpf9YeA86vtb6qlMoNbAb6aq23J/d5exkDjo+PJ+RmEHPb9kscA+40/zuyFsqHg4P1V5y1th37TvDGB+PwdHflxO/jyJnN0+iQUuX67ft89N3PrNhk/tZSuVQRpg/3o1LJIgZHJp4lLCKKu8FhFMzz5C3KIWGR/LpxL/5HzuIfeI4jZ648sdyReyYXqpQuik9ZL3zLmJNuobzZbT7Gb/iNGEqpkUCo1np8cp+xlwQM5iQcHRaOi7sb10+c5Y9RkyjbqA7VO7QwOjSLaNJ7Apt2BzKwQwPGfdTG6HBeSHx8PAtW72ToxOXcCwnHzdWZL3u3oPd7b9hkaXZhOcldCHt4u4ODibLeBROHEnzKFqN0sfxPreFhKza/CKeUcgdMWuuQhOf1gVHWas/WTCZT4nDD+b0HCVi2llN//0PFZvXtfhgC4Ks+77JpdyDTftlK33ZvJtkDScuOn7tG79E/sWP/SQDeqlmOSZ91oGj+nAZHJiypdYOqicMJFUsUxs3OhpOe+atBKeWqlGqplPpRKfWrUmqhUmqwUqrMM3bNA+xUSh0C9gJ/aK03WCLotKZahxYUrVqB+9du8ueYqUaHYxH2unRRVHQMo2euxafNCHbsP0nu7JlZ9M2HrJk8QJJvOrR4bA/6t6/PKxVftrvkC89IwAnDBruAGsC/wExgORALjFVKbVZKlU9qX631Wa11hYRHGa3115YNPe0wmUy0mTQSgK0T5nLz9Hkjw7GYET2bYzIpu1m6aNeBk/i+N5JRM1YTHRNL52avcnjlaNq8VU3m9Yo06Vk9YH+tdRWt9cda65+11lu01uu01hO01m8D7YD0UxosFbyqVaKGX0tio6NZ8dFoo8OxiJJe9rF00b2QcPp8vZC6XcZy/Nw1Xi6Sh82zBzNzROdkr24LkRY890U4pZQr4Ky1tviN2vZ0ES4p96/fZETx14kMCaXvhp8o0+A1o0NKtYeXLvJfNpIKBi5dlNzUn0vXgnip0SCcHB0Y1LkRQ7s2Sfd38WUk6aESnEWqoSmlPgA2An8opdLv2u0vKEve3DQa1heAXweMIi7G+os1WtuDpYsARkxdaWgsyRVGKZQvBzUqeLN32UhG9mpuN/8pRcqk50pwzxoDfvuxTfW01q9prV8FGlsvLPv1ev/O5H7Zi+vHz/DXlJ+MDsciBndpjHsml8Sli4wQFf30X2Z/zRtKmRe4U0oIIz2rB1xBKbVGKVUh4fVhpdQSpdRi4IiVY7NLjs7OtPrBvHTNupE/EnzT/stg5M6eObEuxBeTfrP6MuLx8fGcOH+Nxet203/sYl5p/xXZa/Z66j72XARJZFxPnQestR6tlMoLjEq4ijwc8ADctNaHbRCfXSrX+HXKNqpL4Pq/WPPZd3SYM87okFJtYIcGzFj+P3YeOMmm3YEWXbro2q175ttHj5wlIPAcAUfOcT80wmLHFyKtSsmNGGHAAOBlYBbgD3xnxZjShVY/DOPY5p3snrec2j3bU6SKfa+1lsXTjUGdG/HpxF8ZNuU33qxR5oV6ncGhEew/dv6R+/Uv37j7xOfy58qKb7liicVRKpeS24ZF+vPUWRBKqdFAbcAJ+EVrPVEp1RToDyzQWi+yZDD2Pgvicb8NGsPm8bMo9koVBu1cYfdzUSMioynVdChXb91jybgetKpf9amfj4mJJfD0lYQaq+b79Y+fu/bEEIanuys+ZbzwKeOVUCClGAVyZ3vieOnharjImF6oFoRS6qDWuqIyZ459WuvKCdsdgd5a6x8tGWR6S8ARwSGMKF6X4Bu36bx4ItXaNTM6pFS7cz+M7Fncn9geGRXN5Rt38T9yzlz2L6HOalR07COfc3J0oHzxQviW9cInoThKiaJ5ZQxXpGsvmoAXYy4pmQm4pLUeaL0Q018CBtg9fzkLuwwmS/48fHnif7h6PJm87E1KCqM88HKRPOZ79RN6t+WL26bOqhBpyQsV49Fat1dKlQNitNbWXyApHareqSV/T1/CBf9DbBgzlWZjBhsdktXkyfFwndViVCldlGyZ7f8XjhDW8tQErJSqpbXe+ZT3MwOFtdaBFo8snTCZTLw3eSTjqjdny/dzqNm1DbleSp8XlC5u/sHux7mFsKVnDby9q5TarZQarpRqrJSqqpSqrZTqopRaBKzDPDwhnsKrWiWqd2xhrhPxcfqoE5EUSb5CPJ+nJuCEMd/GwDWgFfAV8BHmKWkztda1tdb+TzmESNB87FBcPNw5tGYzRzcluyiIECIDsdmKGCmRHi/CPWzjuOmsGjqOvKW8GXboTxyc7O9ilEwFE+L5WaQYj0id1wd0IZd3Ua4fO822qQuNDueFpOfCKELYmiRgG3JycXmoTsREQm4FGRyREMJIKVmSyKSUesUWwWQE5Rq/Tpm3XiPifghrPk92fVIhRAbwzASstY4HvrdBLBmCUopWPwzD5OjIrjnLuLhfZvAJkVGldAhik1LqXSXzjCwib0lvXu/fGa01v/QbafXyjkKItCmlCfgj4FcgWikVrJQKUUpZfEmijKTxsL545s7JmV0B+C9da3Q4QggDpCgBa609tdYmrbWT1jpzwuvM1g4uPcuUJTPNvjHflrxy8DdEhYUbHJEQwtZSPAtCKdVUKTU+4dHEmkFlFDX8WlLEpzz3rlxnwzfTjA5HCGFjKUrASqmxmGsAH0149E/YJlLBZDLRZtIIADaPn8WtsxcNjkgIYUsp7QE3At7UWs/TWs8D3krYJlKpWI0qVOvQgtio9F0nQgjxpOe5ESPrQ8+zWDiODK352CG4uLtxaPUmjm1JtvicECKdSWkCHgMcUEotUEr9BOxL2CYsIGv+PDT8og8Ay/t/SVzM05dgF0KkDym6Ew6IB6oDKxMeNbTWy6wcW4byxsCu5HqpCNeOnuLv6YuNDkcIYQMpvROuj9b6mtZ6rdZ6jdb6ug1iy1CcXFxoOeELAH4fPkHqRAiRAaR0CGKzUuoTpVQhpVT2B4+U7KiUclBKHVBKrUtFnBlC+bfrUbpBbSLuh7D2C6kTIUR6l9IE3AXoDWzHPP67D0hp4d7+wLHnDy3jUUrReuJwTI6O7Jy9jIsHpE6EEOlZSseAh2qtvR57FEvBvgUxr6gxxwKxZgh5S3pTt28ntNYs7/el1IkQIh1L6Rhw7xc8/kRgMOaLeElSSnVXSgUopQJu3br1gs2kL42H98MzVw5O7/QnYJnUiRAivbLaGHDC7co3tdb7nvY5rfUsrbWP1tonV65cKY07XXPLmoV3xgwC4LdBUidCiPTKmmPANYGmSqnzwDLgdaWUzK9KoVc6t6JwlXLcu3KdjWOnGx2OEMIKUloN7fHx32eOAWutP9VaF9RaFwXeA/6ntW5vgZgzBJODA20mjQRg03czuX3ukqHxCCEs76kJWCk1+KHnrR57T+6Es7KXXqlC1XbNiI2K5rdPvjY6HCGEhT2rB/zeQ88/fey9t1LaiNZ6m9ZaSli+gObjhuLi7saBlRs4vnWX0eEIISzoWQlYJfM8qdfCCrIVyEvDz82TUJb3/5K42FiDIxJCWMqzErBO5nlSr4WVvDGwKzmLFebqkZNslzoRQqQbz0rAFR6sAQeUT3j+4HU5G8QnACdX10fqRITevmNwREIIS3hqAtZaOzy0BpxjwvMHr51sFaSACk3fpNSbrxJ+L5g1UidCiHTheQqyCwMppWg1cTgmBwd2zlrKpYNHjA5JCJFKkoDtSP7SL1MnoU7EL1InQgi7JwnYzjQZ0R/PXDnIWiAvITduo+PjiQwJJT4+2XIbQog0ytHoAMTzccuahS4//0jOYoWZ07Yfp3f6413Ll65LJ+GZOwcmk/xOFcJeyP9WO+RVtSKLug7h5LY9xMfGcnLbHua27Ue0FO0Rwq5IArZDLh5unN7p/8i20zv9cXF3MygiIcSLkARsh6LCwvGu5fvINu9avty/dlMuzAlhRyQB2yFndze6Lp1E8To1MDk6UrxODTrMHceKT8Ywo3l3wu7eNzpEIUQKyEU4O2QymfDMnYNea2fj4u5GVFg4146e4siffxFxP4QxlRvzwS9T8Kpa0ehQhRBPIT1gO2UymXD19EAl/OlVrRKf7f+DIj7lCTp/mfG1WrH1x3kyJCFEGiYJOB3JVawwn+z8lbp9/YiLieHXAaOY1bIn4fdkSEKItEgScDrj5OJCm0kj6fbrNFwze3Jg5QbGVG7ChYDDRocmhHiMJOB0qkrLRny+fx2FK5fl9rlLfPvKu/w1eYEMSQiRhkgCTsdyvVSEQbt/47XeHYmLieGXfiOZ1aoXEfeDjQ5NCIEk4HTPycWFtlNG0W35VFw9PTjw2598XbkJF/cHGh2aEBmeJOAMokqrxny2fx2FKpbm9tmLfFujBdumLZIhCSEMJAk4A8ntXZTBe1ZSu2d7YqOjWdZ7GHPe60NEcIjRoQmRIUkCzmCcXF15f9poPlg2GVdPD/Yt/4MxlZtw8YAMSQhha5KAMyifNm/z6b7fKVihFLfOXODbGi3YPmOxDEkIYUOSgDOwPC97MXjPKl798H1io6L5uecXzH2/nwxJCGEjkoAzOOdMrrSbMYauP/+Ii4c7Act+5xufplw+dNTo0IRI9yQBCwB8277DpwFrKVC+JDdPnWNstWbsmPWzDEkIYUWSgEWivCVeYsg/q6nVrS2xUdEs+fAz5rUfQGRIqNGhCZEuSQIWj3DO5Er7Wd/QefFEXNzd8P95jXlI4vAxo0MTIt2RBCySVK1dMz4NWEv+siW4cfIs46o1Y+ecZTIkIYQFWS0BK6VclVJ7lVKHlFJHlFJfWqstYR15S3oz9N/V1OzahpjIKBZ3G8qCjh8RFR5BZEgoOj6eyJBQ4uPjjQ5VCLtkzR5wFPC61roCUBF4SylV3YrtCStwdstEhznj8Pvpe5zdMhEXG0fIjdtMa9qN3i7Fmda0GyE3gyQJC/ECrLYkkTZ/V31w9cYp4SHfX+1U9Y7vUsS3AiaTiYVdBnNy2x4ATm7bw9y2/ei1djaunh4GRymEfbHqmnBKKQdgH+ANTNVa/5vEZ7oD3QEKFy5szXBEKuUr5Y2Oj+f0Tv9Htp/e6Y+Lu5tBUQlhv6x6EU5rHae1rggUBKoqpcom8ZlZWmsfrbVPrly5rBmOsICosHC8a/k+ss27li9RYeEGRSSE/bLJLAit9T1gG/CWLdoT1uPs7kbXpZMoXqcGJkdHitepQce547h76arRoQlhd6w2BKGUygXEaK3vKaUyAfWAcdZqT9iGyWTCM3cOeq2djYu7G3cuXmXVp98SuP4vPtq2lMKVnviSI4RIhjV7wPmAv5RShwF/YLPWep0V2xM2YjKZcPX0QJlMZCuUDx0fR2RwCJPf8uPGqXNGhyeE3bBaAtZaH9ZaV9Jal9dal9Vaj7JWW8I4JgcHOi/6gdL1XyXk5m1+fLM9d69cNzosIeyC3AknUs3R2ZkPV87Eq3ol7ly4wqT6HQgNumt0WEKkeZKAhUW4uLvR54/55C9TnGtHTzG1cWciQ8OMDkuINE0SsLAY9+xZ6bdpETmKFuTcvweZ0fxDYqKijA5LiDRLErCwqKz589B/82Iy58nJ8S07md9+IPFxcUaHJUSaJAlYWFxu76L027iQTFk82b9iPUt6fC5V1IRIgiRgYRUFK5Sm97p5OGVyZdecZaz+7FujQxIizZEELKzGu5Yv3VdMx+ToyMax09k0fpbRIQmRpkgCFlZVrlFdOi0YD8DKQWPYNW+5wREJkXZIAhZWV61dM9pMNtfjX9xtKAdWbjA4IiHSBknAwibq9ulEk5ED0PHxzG3bj+NbdxkdkhCGkwQsbKbx8P7U7etHbHQ005t157z/IaNDEsJQkoCFzSilaDVxONXaNycqNIzJDf24duy00WEJYRhJwMKmTCYTHed9S7nGrxMWdJcf32xP0IXLRoclhCEkAQubc3ByotvyqXjX8uXeletMqt+R4Ju3jQ5LCJuTBCwM4eyWiV6/z6FghVLcOHmWKQ39iAgOMTosIWxKErAwjFvWLPTbuJBc3kW5uD+QaU0/IDoi0uiwhLAZScDCUJnz5KL/5kVkyZ+HU3//y5z3+hAXG2t0WELYhCRgYbicRQvRf9Mi3LJl4fDaLSzqOpj4+HijwxLC6iQBizQhf5ni9Fk/Hxd3N/5ZuJIVH4+WCmoiTYiPjycyJBSd8KclOweSgEWaUax6ZT5cNRMHJyf+N3Eef46ZanRIIoOLj48n5GYQ05p2o7dLcaY17UbIzSCLJWFJwCJNKf3mq3RZMhGlFGu/GM/f0xcZHZLIwKLDwpnbth8nt+0hPjaWk9v2MLdtP6LDwi1yfEnAIs2p0qox788cA8Cy3sPxX7bW4IhERuXi7sbpnf6PbDu90x8XdzeLHF8SsEiTXu3WlmbfDEZrzfwOH3FkwzajQ8rwrDkWmhbFREURdP4y3rV8H9nuXcuXKOkBi/SuwZCevPlJd+JjY5nRogdndu8zOqQMy9pjoWlNxP1gpjT0Y/Xn4+k4dxzF69TA5OhI8To16Lp0Es4W6gGrtHSl2cfHRwcEBBgdhkhDtNYs6jqY3fN/xS1rZj7evpwC5UoaHVaGEh8fT+jNIOYkjIU+ULxODXqtnY2rp4eB0Vnevas3mNywE1cOHydz3lwM3v0bHjmz4+LuRlRYOM7ubphMz9d3VUrt01r7PL5desAiTVNK0W7WN1RoVp/we8Fsm7qIsLv3M8zXYCPFx8Xhv3QNX5VrgEeu7FYdC00rrh07zbc1WnDl8HHylCjG4D0ryelVGFdPD5TJhKunx3Mn36eRBCzSPAdHRz5YOom3v/qY+oM/ZGaLHhnia7BR4mJj+XfxKr4s8yZz3+/PtaOnuHnyXJJjoXcvX083dy6e3hXAdzXf5c7FK3hVr8SgXb+Rs2ghq7YpCVjYBSdXV17v68eirkOsNiUoo4uLjWXPTyv4svSbzO8wkBsnzpKjaEHazx5LzpcK03XppEfGQjvMHcfKIWOZVL8DwTduGR1+qhxcvZEf67Uj/O59yjetx8CtP+ORI5vV23W0egtCWIirp3uSX4Od3TLx7+JVlG1UF/fsWY0Jzo7FxcTwz8KV/DlmKrfPXgQg10tFaPh5b6q1b46DkxMAnrlz0Gvt7MSx0LuXrnJy2x6Cr9/i68pN6P7rNF56pYqRp/JC/p6+iGV9RqDj43m1e1vem/oVDo62SY1W6wErpQoppf5SSh1TSh1RSvW3VlsiY4gKC0/ya/C1Y6eZ32Egg3JX4fs6bdgyYQ63zlwwKEr7ERsdzY5ZPzO8eF0WfTCE22cvkvtlL/x++p6Rx7fySufWickXzMX0Hx4LzVe6OJ/v/wPvWr7cv3qD719rw1+TF9jNLeRaa9Z8MZ6lvYah4+N5e9RHvD9jjM2SL1hxFoRSKh+QT2u9XynlCewDmmmtjya3j8yCEE/zYCrU3Lb9OL3TH+9avnT9+UcuHzzK5glzOLntH+IfGo/MV/plyjetR4Wm9SharZJFL57Ys5ioKHbPW86Gb6Zz99JVAPKWfIlGw/rh06YJJgeH5zpeXEwMq4aOY8uEOQD4tm1Ku1nf4OrhbvHYLSUuJobF3T9lz4IVmBwcaDdzDDW7trFae8nNgrDZNDSl1BpgitZ6c3KfkQQsniU+Pp7osPAkpwRF3A8m8M9tHF67hcD1fxFx//8LvHvmzkn5t9+gfNN6lKpXC2e3TEadgmFiIiPZOecXNo6dzr0r1wFzEaRGw/pSuWWj5068j9v36x8s7DKYqNAw8pV+mQ9/m07ekt6WCN2iIkPDmN26N0f+3IZTJle6/zqNco1ft2qbhiZgpVRRYDtQVmsd/Nh73YHuAIULF65y4YJ8dRSpFxcTw6kd/hxeu5lDazYTdP7/151zyuRKqTdrUaHpm5Rr8jqZ8+QyMFLriw6PYMesn9n07UzuX7sJQIFyJWk0vB+VWrxl0W8G14+fZkaLHlw/dhoXD3c6zv+OKi0bWez4qRV88zZTG3fhQsBhPHJmp/e6uXhVq2T1dg1LwEopD+Bv4Gut9cqnfVZ6wMIatNZcDTzBobVbOLRmMxf8DyW+p5SiaLWKVGhaj/Lv1CdfKW+UUgZGazlRYeFsn7GEzd/NJPiGec29QhVL02h4Pyq8U99qQzKRoWEs/mAIAb+sA6Dex91o/s3gR8aTjXDrzAUmNejIrTMXyOlViL4bfiJP8WI2aduQBKyUcgLWARu11hOe9XlJwMIW7l29wX/rtnJ47RaObdlJbFR04nu5XipC+ab1KN+0Ht61fG16QcZSIkPD+HvaIraMn03IrSAAClcpR+Ph/Sj/dj2b/ILRWvPX5AWs+Phr4mNj8X61Kt1+mUKWfLmt3nZSLgQcZkqjzoTcCqJw5bL0WT/fpt98bJ6Alfmn/BNwR2s9ICX7SAIWthYVFs7RTds5vHYL/637H6G37yS+55YtC+Uav075pvUo3aA2mTJ7PnUM2mgRwSH8PXUhm7+fQ1jQXQCKVq1A4xEDKNuwjiE9+9O7Apjdujf3r94gc95cdFs+lZdfrWrTGI5s2Maslr2ICgundP1X6b5ius1vnzYiAdcCdgD/AQ9uVfpMa70+uX0kAQsjxcfFcfafAxxas5nDazdz48TZxPccnJxoMnIA1Tu2YH6Hj/5/FsbSSXjmzmFoEo64H8xfk39iy4Q5hN+9D0CxGpVpPKI/pevXNnxIJfjGLea815eT2/7B5OBAi28/5Y2BXW0S156fVrDog6HEx8ZSrUMLOswZi6Ozs9XbfZzhsyBSQhKwSEuunzjD4d+3cnjNZs7s3scXh/7kl74jnyhI02HOWNYOn4B79qy4Z8+CW/asCc8ffe2WLUuqhjQe731rYMv3s/nfxHmJMz68a/nSeER/Sr5R0/DE+7C42FjWfjGejeNmAFC5ZSM6zB1HpsyeVmlPa82Gb6ax5vPvAGgwtCfNxgw27O9EErAQqRByKwj37Nno41r8kbnGJkdHpkSeoJfjSyk6TqYsnuZknESCNm/P8tBz8/vu2bNicnR8Yg50x7njWP35eAKWraV4neo0Ht6P4nVqpKnE+7iDqzeyoNMnRAaHkKdEMT78bQb5yxS3aBvxcXH80m8kf09bhFKK1pNGUrdPJ4u28bwkAQuRSpEhoUxr2u2JHvAHv0zm2KYdhN25R9ide4Tfuf/Q83uE3blPWNBdwu/ef+G7xIYHbmJZnxFPtN1+9jfcv3qDl2tXS/X52cqNU+eY2aIHVwNP4OyWiQ5zxuLb9h2LHDs6IpJ57fpzcNVGHF2c6bx4YpqYBicJWIhUSvJOvOcYA46PjyfifkhCUn4sQSf1OuhuYjKfEnWSPq4lnuh9T406iUojFwGfR1RYOD/3+Jx/F68CoG5fP94d/1mqxmfD7txjWtMPOLMrALesmem5Znaa+cUkCVgICzBiFoTWmsjgUKY3656uCqJrrdk+YzHL+48iLiaGYjUq0235VLIVzPfcx7pz8QqT3urE9WOnyVYwH303/GTxoY3UkILsQljA4wVpbDH7QSmFi6f7E+UgLbk0jhGUUrzWswOf7FhOtoL5OLtnP2MqN+H4/3Y/13EuHz7GuBotuH7sNPnLFGfQ7t/SVPJ9GukBC2En0vIc5NQKuRXE3Pf7c3zLTpTJRLMxg6g/uMczLyie2LaH6e90JzI4hJdrV6Xnmtm4Zc1io6hTTnrAQtg5I3rftuKZKwf9NvxEw8/7oOPjWTV0HDOadyfifnCy+wT88juTG3QiMjiEyi0b0W/jwjSZfJ8m/fwEhRB2zeTgwDujP6HX73Nxy5qZQ2s2M8anKZcPH3vis1snzmXOe32JjY6mbl8/Plg2GSdXVwOiTh1JwEKINKV8kzf4dN86ClUsza3T5xlXvTknt/9LZEgoOj6eOxevcO5fc0Gl5uOG0vrHEakupWkU+6s0IoRI93IVK8yg3StZ1nsY0RFRZCuYj2lNuyVO/+swdxy1e7ajeBqZZvai5CKcECJNu3/tJnPf72/XU/DkIpwQwi5lzpMzycVYXex4Ct4DkoCFEGlacouxRoWFGxSR5UgCFkKkac7ubunuJpQH5CKcECJNM5lMeObOQa+1s9PdTSiSgIUQad6Dm1AAu7nwlhL2/ytECCHslCRgIYQwiCRgIYQwiCRgIYQwiCRgIYQwiCRgIYQwSJqqBaGUugVcMDqO55QTuG10EAaRc8+YMuq5p+a8i2itcz2+MU0lYHuklApIqshGRiDnLueekVjjvGUIQgghDCIJWAghDCIJOPVmGR2AgeTcM6aMeu4WP28ZAxZCCINID1gIIQwiCVgIIQwiCTiFlFJvKaVOKKVOK6WGJvF+O6XU4YTHbqVUBSPitIZnnftDn/NVSsUppVraMj5rScl5K6XqKKUOKqWOKKX+tnWM1pKCf+9ZlFK/K6UOJZx7ZyPitAal1Dyl1E2lVGAy7yul1KSEv5vDSqnKL9yY1loez3gADsAZoBjgDBwCSj/2mVeAbAnPGwL/Gh23rc79oc/9D1gPtDQ6bhv9zLMCR4HCCa9zGx23Dc/9M2BcwvNcwB3A2ejYLXT+tYHKQGAy7zcC/gQUUD01/9elB5wyVYHTWuuzWutoYBnwzsMf0Frv1lrfTXj5D1DQxjFayzPPPUFf4Dfgpi2Ds6KUnPf7wEqt9UUArXVGOncNeCqlFOCBOQHH2jZM69Bab8d8Psl5B1iozf4Bsiql8r1IW5KAU6YAcOmh15cTtiWnK+bfkOnBM89dKVUAaA7MsGFc1paSn3lxIJtSaptSap9SqqPNorOulJz7FKAUcBX4D+ivtY63TXiGe958kCxZkihlVBLbkpy/p5SqizkB17JqRLaTknOfCAzRWseZO0TpQkrO2xGoArwBZAL2KKX+0VqftHZwVpaSc28AHAReB14CNiuldmitg60cW1qQ4nzwLJKAU+YyUOih1wUx/+Z/hFKqPDAHaKi1DrJRbNaWknP3AZYlJN+cQCOlVKzWerVNIrSOlJz3ZeC21joMCFNKbQcqAPaegFNy7p2Bsdo8KHpaKXUOKAnstU2IhkpRPkgJGYJIGX/gZaWUl1LKGXgPWPvwB5RShYGVQId00AN62DPPXWvtpbUuqrUuCqwAetl58oUUnDewBnhVKeWolHIDqgHHbBynNaTk3C9i7vmjlMoDlADO2jRK46wFOibMhqgO3NdaX3uRA0kPOAW01rFKqT7ARsxXiOdprY8opXokvD8DGA7kAKYl9ARjdTqoGJXCc093UnLeWutjSqkNwGEgHpijtU5y6pI9SeHP/CtggVLqP8xfyYdordNFiUql1FKgDpBTKXUZGAE4QeK5r8c8E+I0EI7528CLtZUwrUIIIYSNyRCEEEIYRBKwEEIYRBKwEEIYRBKwEEIYRBKwEEIYRBKwMIRSKtTG7e22cXtZlVK9bNmmsD+SgEW6oJR66px2rfUrNm4zKyAJWDyV3Igh0gyl1EvAVMzlDcOBblrr40qpt4EvMJdGDALaaa1vKKVGAvmBosBtpdRJoDDmMoqFgYla60kJxw7VWnsopeoAI4HbQFlgH9Bea62VUo2ACQnv7QeKaa2bPBajH9AYcAXclVJNMd8Rlw3zZP0vtNZrgLHAS0qpg8BmrfUgpdQgoDXgAqzSWo+w3N+esEtG196UR8Z8AKFJbNsKvJzwvBrwv4Tn2fj/m4Y+AL5PeD4ScwLN9NDr3ZgTXE7Mydrp4fYw3+F0H/P9+yZgD+bCSa6YK1x5JXxuKbAuiRj9MNcCyJ7w2hHInPA8J+a7oxTmXwqBD+1XH/Oijiqh3XVAbaN/DvIw9iE9YJEmKKU8MBe1//WhimouCX8WBH5JqLnqDJx7aNe1WuuIh17/obWOAqKUUjeBPJgT5sP2aq0vJ7R7EHOyDAXOaq0fHHsp0D2ZcDdrrR/Ui1XAGKVUbcy3IxdIaPNx9RMeBxJeewAvA9uTaUNkAJKARVphAu5prSsm8d5kYILWeu1DQwgPhD322aiHnseR9L/xpD7zPHU0H26zHeYhkypa6xil1HnMvenHKeAbrfXM52hHpHNyEU6kCdpcR/acUqoVJK679WBdvSzAlYTnnawUwnGgmFKqaMLrNincLwtwMyH51gWKJGwPATwf+txGoEtCTx+lVAGlVO7Uhy3smfSAhVHcEipNPTABc29yulLqC8wXtJZhXo9sJOahiSuYl3vysnQwWuuIhGljG5RSt0l5XdslwO9KqQDMBcqPJxwvSCm1K2Fhxz+1+SJcKcxF28E85NGe9LOEk3gBUg1NiARKKQ+tdWjCOmdTgVNa6x+MjkukXzIEIcT/65ZwUe4I5qEFGa8VViU9YCGEMIj0gIUQwiCSgIUQwiCSgIUQwiCSgIUQwiCSgIUQwiD/B2bEVASPLeH1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from visualization import get_colors\n",
    "cm = get_colors(colormap='RdBu', n_colors=2)\n",
    "\n",
    "%matplotlib inline\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5, 4))\n",
    "\n",
    "ax.plot(learning_rates, trn_err, marker='o', c=cm[0], markeredgecolor='w', linewidth=2)\n",
    "ax.plot(learning_rates, val_err, marker='s', c=cm[1], markeredgecolor='w', linewidth=2)\n",
    "ax.legend(['Train err', 'Validation err'])\n",
    "ax.set_xlabel('Learning rate')\n",
    "ax.set_ylabel('Error (%)')\n",
    "\n",
    "fig.tight_layout()\n",
    "pngFile = './figures/CH05_F18_Kunapuli.png'\n",
    "plt.savefig(pngFile, dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.2\tEarly Stopping\n",
    "\n",
    "As with AdaBoost, the other important consideration for practical boosting is the number of base learners, which can be hard to specifiy ahead of time. Identifying the least number of base estimators in order to build an effective ensemble is known as early stopping. \n",
    "\n",
    "[Early stopping](https://lightgbm.readthedocs.io/en/latest/Parameters.html#early_stopping_round) with ``LightGBM`` works slightly differently than AdaBoost. In ``LightGBM``, we specify a value for the parameter ``early_stopping_rounds``. As long as the overall score (say accuracy) improves over the last ``early_stopping_rounds``, ``LightGBM`` will continue to train. However, when the score does not improve after ``early_stopping_rounds``, ``LightGBM`` stops.\n",
    "\n",
    "Small values of ``early_stopping_rounds`` make ``LightGBM`` very \"impatient\" and aggressive in that it does not wait too long to see if there is any improvement before stopping learning early. This may lead to underfitting. Large values of ``early_stopping_rounds`` make ``LightGBM`` overly patient and too passive in that it waits for longer periods to see if there is any improvement. This may lead to overfitting.\n",
    "\n",
    "``LightGBM`` needs us to **explicitly specify a validation set as well as [scoring metric](https://lightgbm.readthedocs.io/en/latest/Parameters.html#metric)**. In the experiment below, we split the training data further into a smaller training set and a separate validation set for early stopping. The metric we will use is accuracy score.\n",
    "\n",
    "**Listing 5.6**: Early Stopping with LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 5 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(early_stopping=5, max_depth=1, n_estimators=50)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a single split of training and validation sets\n",
    "Xtrn, Xval, ytrn, yval = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=42)\n",
    "gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=50, max_depth=1, early_stopping=5)\n",
    "gbm.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric='auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also visualize the training and validation errors as well as the ensemble size for different choices of early_stopping_rounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 1 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 2 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 3 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "Early stopping, best iteration is:\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 4 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "Early stopping, best iteration is:\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 5 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 6 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 7 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 8 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "[35]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132684\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[1]\tvalid_0's auc: 0.885522\tvalid_0's binary_logloss: 0.602321\n",
      "Training until validation scores don't improve for 9 rounds\n",
      "[2]\tvalid_0's auc: 0.961022\tvalid_0's binary_logloss: 0.542925\n",
      "[3]\tvalid_0's auc: 0.96528\tvalid_0's binary_logloss: 0.497678\n",
      "[4]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.451922\n",
      "[5]\tvalid_0's auc: 0.986243\tvalid_0's binary_logloss: 0.421817\n",
      "[6]\tvalid_0's auc: 0.985588\tvalid_0's binary_logloss: 0.393713\n",
      "[7]\tvalid_0's auc: 0.989846\tvalid_0's binary_logloss: 0.364781\n",
      "[8]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.338063\n",
      "[9]\tvalid_0's auc: 0.989191\tvalid_0's binary_logloss: 0.319267\n",
      "[10]\tvalid_0's auc: 0.990501\tvalid_0's binary_logloss: 0.299509\n",
      "[11]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.286114\n",
      "[12]\tvalid_0's auc: 0.989519\tvalid_0's binary_logloss: 0.269481\n",
      "[13]\tvalid_0's auc: 0.992303\tvalid_0's binary_logloss: 0.255033\n",
      "[14]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.241218\n",
      "[15]\tvalid_0's auc: 0.992958\tvalid_0's binary_logloss: 0.231917\n",
      "[16]\tvalid_0's auc: 0.993285\tvalid_0's binary_logloss: 0.221174\n",
      "[17]\tvalid_0's auc: 0.994923\tvalid_0's binary_logloss: 0.210735\n",
      "[18]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.204185\n",
      "[19]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.195894\n",
      "[20]\tvalid_0's auc: 0.994595\tvalid_0's binary_logloss: 0.187605\n",
      "[21]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.183288\n",
      "[22]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.177604\n",
      "[23]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.170961\n",
      "[24]\tvalid_0's auc: 0.995087\tvalid_0's binary_logloss: 0.167818\n",
      "[25]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.165031\n",
      "[26]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.158768\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n",
      "[28]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.153942\n",
      "[29]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.15031\n",
      "[30]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.145113\n",
      "[31]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.143901\n",
      "[32]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.139801\n",
      "[33]\tvalid_0's auc: 0.995742\tvalid_0's binary_logloss: 0.13718\n",
      "[34]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.135961\n",
      "[35]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132684\n",
      "[36]\tvalid_0's auc: 0.995414\tvalid_0's binary_logloss: 0.132215\n",
      "Early stopping, best iteration is:\n",
      "[27]\tvalid_0's auc: 0.996069\tvalid_0's binary_logloss: 0.156152\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n",
      "c:\\users\\gauta\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\lightgbm\\engine.py:153: UserWarning: Found `early_stopping` in params. Will use it instead of argument\n",
      "  warnings.warn(\"Found `{}` in params. Will use it instead of argument\".format(alias))\n"
     ]
    }
   ],
   "source": [
    "number_of_steps = np.arange(1, 10, 1)\n",
    "\n",
    "\n",
    "trn_err = np.zeros((len(number_of_steps), ))\n",
    "val_err = np.zeros((len(number_of_steps), ))\n",
    "n_iters = np.zeros((len(number_of_steps), ))\n",
    "\n",
    "for i, rounds in enumerate(number_of_steps):\n",
    "    gbm = LGBMClassifier(boosting_type='gbdt', n_estimators=50, max_depth=1, early_stopping=rounds)\n",
    "    gbm.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric='auc')\n",
    "                     \n",
    "    trn_err[i] = 1 - accuracy_score(ytrn,  gbm.predict(Xtrn))\n",
    "    val_err[i] = 1 - accuracy_score(yval,  gbm.predict(Xval))\n",
    "    n_iters[i] = len(gbm.evals_result_['valid_0']['auc']) # Get the number of estimators in the ensemble in a roundabout way  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAEYCAYAAADMEEeQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABVF0lEQVR4nO3deXxU9dX48c9JQhIIO4RF9hB2lC0gICBo3RWXSpXWrVpxwaq11tI+1mpr+zxaa3/VKooFUWul7qKliloRkC1h3wQCBIjsIGEJZD2/P+5NnITJBrlzMzPn/Xrd18xdv2dIuDlzv5uoKsYYY4wxJnrE+B2AMcYYY4wJLUsAjTHGGGOijCWAxhhjjDFRxhJAY4wxxpgoYwmgMcYYY0yUifM7gNrUsmVL7dy5s99hGGPCxNKlS/erarLfcVSX3eOMMTVV0X0uohLAzp07k5GR4XcYxpgwISLb/I6hJuweZ4ypqYruc1YFbIwxxhgTZSwBNMYYY4yJMpYAGmOMMcZEmYhqA2iM3woKCsjOzubEiRN+h2ICJCYm0r59e+rVq+d3KMYYUydYAmhMLcrOzqZRo0Z07twZEfE7HAOoKgcOHCA7O5suXbr4HY4xxtQJVgVsTC06ceIELVq0sOSvDhERWrRoYU9ljTEmgCWAp6C4uJgTR46i7mtxcbGVaUpZ8lf32M/EGBPuavtvsiWANVRcXMyRvQd4fuztTEzozvNjb+fI3gOeJkfRUqYxxhhjTubF32RrA1hD+cdymTr+XjbOWQjAxjkLmTr+Xq7/2+94cvg1npT50IJ3mXHPI3WizLtnvkRio4aelGlO34EDBzj//PMB2L17N7GxsSQnOwPAL1myhPj4+ArPzcjI4NVXX+WZZ54JSazGGGOqdmjnHmJiYoLmHqfzN9kSwBpKSGpA5vz0Mtsy56fTpmcKJw4f8aTMNj1T6kyZCUkNPCkvWhUXF5N/LJeEpAbkHcslPqkBMTGn/mC+RYsWrFixAoBHH32Uhg0b8uCDD5buLywsJC4u+H/7tLQ00tLSTrnsihQVFREbG1vhujHGGIeqsm/zNjLnLWHT3CVkzktn3+ZtPF+4udb/JlsCWEN5x3JJHTG4NAsHSB0xmLyjufzl0Cpvyjxah8o8lmtPAGtJySP9qePvJXN+OqkjBnPbG8/QqFWL00oCy7vlllto3rw5y5cvZ+DAgVx33XXcf//9HD9+nPr16/Pyyy/To0cP5syZw1NPPcVHH33Eo48+yvbt29myZQvbt2/n/vvv59577z3p2rNnz+a3v/0teXl5dO3alZdffpmGDRvSuXNnbr31VmbPns0999zDpEmTyqxff/31tfb5jDEmXBUXF7Nr7UY2zV3CprmLyZyXTs6uvWWOSWiYxP6t2bX+N9kSwBqKT2rAbW88c9If7YRGSbX6RztQcXFxnSkz3p4AVtud0rnS/b9Z/Qn/+umjJz3Sv+7ZR/n9mRdVeN4LmlXjWDZu3Mhnn31GbGwshw8fZu7cucTFxfHZZ5/x61//mnfeeeekc77++mu++OILjhw5Qo8ePbjrrrvKjKO3f/9+Hn/8cT777DOSkpJ44oknePrpp3nkkUcAZ+y9+fPnAzBp0qQy68YYE42KCgrYtnQ1mfPSnWV+Ornf5pQ5pmHL5qSOHEzqyMF0G3U27fv1QmJiav1vsiWANRQTE0OjVi24e+ZLtVZtV5fLvOGl/6Vllw4c3f8tDZObe1pmtGnbKzXoI/22vVJrvaxx48aVVrvm5ORw8803s2nTJkSEgoKCoOdcdtllJCQkkJCQQKtWrdizZw/t27cv3b9o0SLWrVvHOeecA0B+fj7Dhg0r3X/dddeVuV75dWOMiXT5ucfZsmh5aZXu1kXLyc89XuaYZh3OoNuoIXQbNYTUkUNo07Nr0JELajsPsATwFMTExJQ+cg1VdahfZS6Y9iYf/+/zXDTpLq7+31+GpNxIUdWTuhNHjgZ9pJ+fe/yUnvJVJikpqfT9b37zG8aMGcN7771HVlYWo0ePDnpOQkJC6fvY2FgKCwvL7FdVLrjgAt54440qywy2Hi1EJBGYCyTg3HPfVtXfisijwO3APvfQX6vqLH+iNMbUVLA23MdzjrD5q4zShG/70jUUlfuS3aZnV1JHliR8g2nRqX0FJZRV23mAJYCmUinDBwGwZcFSnyOJPBU1J/C6mj0nJ4d27doBMH369FO+ztChQ5k4cSKZmZmkpqaSm5tLdnY23bt3r6VII0YecJ6qHhWResB8EfmPu+8vqvqUj7EZY05BsDbcN738JO//6k9kzJhZepzExNBxYN/S6tyuI9Jo3Kqlj5F/xxJAU6mUoQMAyFqykqKCAmJtLtVa40fVPsBDDz3EzTffzNNPP8155513ytdJTk5m+vTpjB8/nry8PAAef/xxSwDLUVUFjrqr9dxF/YvIGHMqVJX9W3ewae5iuo8exqs//kWZNtyv/vghrn/2MQ5l7yJ11BC6jRxMyvBB1G/cyOfIgxPn3hQZ0tLSNCMjw+8wIs5ve57Hng1bmLTkAzoP7ud3OHXa+vXr6dWrl99hmCCC/WxEZKmq1v7YN+WISCywFEgFnlPVX7pVwLcAh4EM4Oeq+m2QcycAEwA6duw4aNu2bV6Ha4whoIfuvHQy5y5m07x0cnbuAeD5ws3ck9iD4oCmMTFxcTyXtxGpY23lK7rPeRqliFwsIhtEJFNEJgXZ31NEFopInog8WG7fNBHZKyJrvIzRVK2rWw282aqBjTklqlqkqv2B9sAQEekLTAa6Av2BXcCfKzh3iqqmqWpayaDexpjaV1RQwNbFy5n91BSeH/sTHmw5gN+fdTEzJv6GjH99RM7OPSS1aEa/qy7k8O59pI4YXOb8kmFZwoVnVcDuN97ngAuAbCBdRGaq6rqAww4C9wJXBbnEdOBvwKtexWiqJ2X4IBa8/BZbFy6D+271OxxjwpaqHhKROcDFgW3/ROQl4CPfAjMmCuXnHmfr4uWlAy5vWbgsaA9dp/2e00O3ba9URCQihkrzsg3gECBTVbcAiMgM4EqgNAFU1b3AXhG5rPzJqjpXpIqB1ExIpJQ+AVzmcyTGhB8RSQYK3OSvPvA94AkRaauqu9zDrgastsMYD+UeymHzV0tLB1zelrH6pB66rXuk0G3U2aVJX0U9dP1qw12bvEwA2wE7AtazgbNru5By7WNq+/IGp8t6g6aN+XbHTg7u2EnzDmf4HZIx4aQt8IpbKxIDvKmqH4nIayLSH6dDSBZwh38hGhPegg3JcmTvfjLnpbtP+JbwzaqvCez3IDExdBjQp/TpXuqINBq3rn4zCz+GZ6tNXiaAJ49i6EHPN1WdAkwBpxNIbV/fOL/kXYYOYO3HX7Jl4TJLAI2pAVVdBQwIsv1GH8IxJuIEHZJl2pO8/+uyQ7LExcfTafBZpePvdR0+iPpNGvsYub+8TACzgQ4B6+2BnR6WZzyUMnyQkwAuWEraDy73OxxjjDFRrLi4mF3rNrFp7hL6Xnwur972y7JDstz6ENf/7TGOHfi2NOHrPKQ/8fUTfY687vCysjod6CYiXUQkHrgemFnFOaaO6mrtAMPC6NGj+eSTT8ps+3//7/9x9913V3pOyfBJl156KYcOHTrpmEcffZSnnqp8vOL333+fdeu+6+P1yCOP8Nlnn9UgemOMCS5oD90zL2LGxN/QvFO7CqfVvG/2a1z68E/pfu5QS/7K8ewJoKoWisg9wCdALDBNVdeKyJ3u/hdEpA3O+FeNgWIRuR/oraqHReQNYDTQUkSygd+q6lSv4jWV6zykHxITw47la8k/fsL+I9WCE3kFJCacPLB2RdurY/z48cyYMYOLLrqodNuMGTP405/+VK3zZ8069ZnI3n//fS6//HJ69+4NwO9+97tTvlZNFRUVlc51HGzdGBNeSnrolrThC9pDt31bUkcN4fCe/UGn1cw7lhuWbfNCxdOZQNx5LWeV2/ZCwPvdOFXDwc4d72VspmYSGzWk/Vk92bFiHdsyVtFt5BC/Qwp7iQn1iB9w8rA6+cunnfI1r732Wh5++GHy8vJISEggKyuLnTt3MmLECO666y7S09M5fvw41157LY899thJ53fu3JmMjAxatmzJH/7wB1599VU6dOhAcnIygwY5T4FfeuklpkyZQn5+Pqmpqbz22musWLGCmTNn8uWXX/L444/zzjvv8Pvf/57LL7+ca6+9ls8//5wHH3yQwsJCBg8ezOTJk0lISKBz587cfPPNfPjhhxQUFPDWW2/Rs2fPMjEVFRUxadIk5syZQ15eHhMnTuSOO+5gzpw5PPbYY7Rt25YVK1bw/PPPl1kPfBppjKnbSnrolsyhG7SHbveU0hk2Ut0eupEyJIsfbCo4U21dhg1kx4p1bFmw1BLAagiW3AWqLNGr7NzKzmvRogVDhgzh448/5sorr2TGjBlcd911iAh/+MMfaN68OUVFRZx//vmsWrWKs846K+h1li5dyowZM1i+fDmFhYUMHDiwNAG85ppruP322wF4+OGHmTp1Kj/96U8ZO3ZsacIX6MSJE9xyyy18/vnndO/enZtuuonJkydz//33A9CyZUuWLVvG888/z1NPPcXf//73MudPnTqVJk2akJ6eTl5eHueccw4XXnghAEuWLGHNmjV06dKFOXPmlFk3xtRdObv3Vt5DV4QOA/qQOnIwqSOdpK+iHrqRMCSLHywBNNXWdfgg5k7+h80IUseVVAOXJIDTpjkJ45tvvsmUKVMoLCxk165drFu3rsIEcN68eVx99dU0aOB8gx47dmzpvjVr1vDwww9z6NAhjh49Wqa6OZgNGzbQpUuX0jmCb775Zp577rnSBPCaa64BYNCgQbz77rsnnT979mxWrVrF22+/DUBOTg6bNm0iPj6eIUOGlEn2yq8bY0Iv2JAsB7KyS5/uZc5LZ++mrWXOia1Xj85D+pWOv1fTHrrhPiSLHywBNNVWMiD0lgXLUFVEgo30Y0qcTlXu6Zx71VVX8cADD7Bs2TKOHz/OwIED2bp1K0899RTp6ek0a9aMW265hRMnTlR6nYp+vrfccgvvv/8+/fr1Y/r06cyZM6fS61Q133hCQgIAsbGxFAbMqxl4/rPPPntSojlnzhySkpLKbCu/bowJrYqGZPngf54qMyRLQlIDUoYPKk34rIdu6NnzUVNtLbt0oHHrlhzdf5C9mVl+h2Mq0LBhQ0aPHs2tt97K+PFOU9rDhw+TlJREkyZN2LNnD//5z38qvcaoUaN47733OH78OEeOHOHDDz8s3XfkyBHatm1LQUEBr7/+eun2Ro0aceTIkZOu1bNnT7KyssjMzATgtdde49xzz63257nooouYPHkyBW57oI0bN3Ls2LFqn2+M8VZRQQFbl6xg9lNTOLB1B1PH38vGOQspLiwsHZLlst/8lH5XXsC1f36YSUs+4OlDq7hv9mtc9pt7rYeuT+wJoKk2ESFl+CBWvPcJWxYspXU3q2o7HSfyCoI+6TudXsAlxo8fzzXXXMOMGTMA6NevHwMGDKBPnz6kpKRwzjnnVHr+wIEDue666+jfvz+dOnVi5MiRpft+//vfc/bZZ9OpUyfOPPPM0qTv+uuv5/bbb+eZZ54pra4FSExM5OWXX2bcuHGlnUDuvPPOan+Wn/zkJ2RlZTFw4EBUleTkZN5///0a/GsYY2pT/vETTg/duUvYNC+drQuXkXcsF4Dv/ey2oEOytOnZlbvef8mPcE0FpKrqmXCSlpamJeOZGW/MfmoK7/7ij4ycMJ4fvfi/fodT56xfv55evXr5HYYJItjPRkSWqmqaTyHVmN3jjB9yD+WwecEyMucuZtO8dLalr6qwh+7Y3z3A1B/eV2ZIlu6jh3H3zJesbZ5PKrrP2RNAUyMlA0JvWbjc50iMMcZ4oaSHrtNLd3HwHrr9e5PqzqEb2EPXhmQJH5YAmhrpOLAPsfXqsXPNBo7nHI7qeRSNMSbcqSoHsrLZNHdx6bAswXrofjeH7hC6Dh9Ig6ZNgl7PhmQJH5YAmhqpl5hIx0F92bpoOVsXr6D3haP8DqnOsR7SdU8kNXUxpqbKDMtyNJfcQzms/vd/SxO+Q9/sLnN8QlIDugwbWDqHbpezB9Sok4YNyRIeLAE0NdZ1+CC2LlrO5gVLLQEsJzExkQMHDtCiRQtLAusIVeXAgQMkJlovQxN9gg3LcuPUJ9g0N710WJak5k3pOiKNbqPOJnXkYDoOcGp6TGSzBNDUWMrwQfD039myYJnfodQ57du3Jzs7m3379vkdigmQmJhI+/ZBZ500JqLlH8stHZYFYOOchbx22y+5adqTdBs1mG6jzqZNr1Sroo1ClgCaGksZNhCArYuWU1xURExsrM8R1R316tWzmSiMMXVGQlKDoMOytOjUjnPvutGnqExdYCm/qbGmZ7SmRef2nDhylF3rNvkdjjEhIyIDgyxdRcS+TJs6Ke9YLqkjBpfZljpicOm4fSZ6WQJoTknJU0CbF9hEmeeBRcAU4CVgITAD2CgiF/oZmDHBKHDj1CfoPnoYMXFxdB89zIZlMYAlgOYUfTcvsCWAJqpkAQNUNU1VBwEDgDXA94An/QzMmGC2LlrOB//zFDdM+SPP5W3k7pkv0ahVC2vzZywBNKemZEDozdYRxESXnqq6tmRFVdfhJIRbfIzJmAplLVlJxoyZfPG3VxB3eBZL/gxYJxBzitqd1ZP4BvXZl5nF4b37adyqpd8hGRMKG0RkMk61L8B1ONW/CUBBxacZ449t6SsB6Dykv7+BmDrH068BInKxiGwQkUwRmRRkf08RWSgieSLyYE3ONf6KjYujy9n9Adiy0J4CmqhxC5AJ3A/8DNjibisAxvgVlDEVyUpfBUDnwWf5HImpazxLAEUkFngOuAToDYwXkd7lDjsI3As8dQrnGp9ZO0ATbVT1uKr+WVWvVtWrVPUpVc1V1WJVPep3fMYE+vab3eTs3EP9Jo1ITu3sdzimjvHyCeAQIFNVt6hqPk6VyZWBB6jqXlVN5+SqkyrPNf4rTQAXLvc5EmNCQ0TOEZFPRWSjiGwpWfyOy5hgSqp/O6WdZe3+zEm8bAPYDtgRsJ4NnB2Cc02IpAwdADg3mcL8fOLi432OyBjPTcWp+l0KFPkcizGVKq3+HdLP50hMXeTlV4JgE6FWd0b2ap8rIhNEJENEMmz6rdBKat6UNj27UnAijx0r1vkdjjGhkKOq/3FrLw6ULJWdICKJIrJERFaKyFoReczd3tx9mrjJfW0Wmo9gokXWEvcJoLX/M0F4mQBmAx0C1tsDO2v7XFWd4o7JlZacnHxKgZpTZ+0ATZT5QkT+JCLDAmcDqeKcPOA8Ve0H9AcuFpGhwCTgc1XtBnzurhtTK1SV7RklHUDsCaA5mZcJYDrQTUS6iEg8cD0wMwTnmhCy8QBNlDkbSAP+CPzZXZ6q7AR1lHQQqecuitOu+RV3+yvAVR7Ea6LU3swscg8dpknbVjRt18bvcEwd5FkbQFUtFJF7gE+AWGCaqq4VkTvd/S+ISBsgA2gMFIvI/UBvVT0c7FyvYjWnruQJ4OavMlBVRILV3hsTGVT1lIZ6cUc2WAqkAs+p6mIRaa2qu9zr7hKRVhWcOwGYANCxY8dTC9xEncDqX7svm2A8HQhaVWcBs8pteyHg/W6c6t1qnWvqntY9UmjQrAk5O/fw7Y6dNO/Yzu+QjKl1InKDqv5DRB4Itl9Vn67sfFUtAvqLSFPgPRHpW92yVXUKztzDpKWlVbcdtYlypQNAW/WvqYD1CzenJSYmhi5ub2AbENpEsCT3tVEFS7Wo6iFgDnAxsEdE2gK4r3trL1wT7awHsKmKJYA1dCIv+GxPFW2PhjK/awdoHUFMZFLVF93Xx4ItlZ0rIsnukz9EpD7wPeBrnHbNN7uH3Qx84NkHMFGlqKCAHcudVlOd0qwHsAnO5gKuocSEesQPuPWk7fnLpzFr3kpPyrx0ZL86VWZ53/UEtieAJrKJSArwV2AoTkeOhcDPVLWywaDbAq+47QBjgDdV9SMRWQi8KSK3AduBcd5Gb6LFzrUbKTh+guSunUhq3tTvcEwdZQlgLbrq3r96ct1gSZefZZbXeUg/JCaGHcvXkncsl4SkBp7EZEwd8E+caSqvdtevB96gkoHqVXUVMCDI9gPA+R7EaKKcVf+a6rAEsBZdMiL0j9r9KLO8xIZJtO/Xix3L17ItYxXdzx3qd0jGeEVU9bWA9X+4IxYYU2dscxNAGwDaVMYSwFr0wbP3R0WZwaQMH8SO5WvZsmCpJYAmkn0hIpNw5idX4Drg3yLSHEBVD/oZnDEAWUtWANYD2FTOEkBTK7oOH8SXz71qA0KbSHed+3pHue234iSEKaENx5iy8nOPs3PNRmJiY+kwoI/f4Zg6zBLAGjqRVxC0fdyJvAISE+pFbZkpw5zZsLYsXGYDQpuIpapd/I7BmMrsWLGO4qIi2p3V09pjm0rZMDA1VFHC5VUi5neZBw4dpX7aT2gw+HYOHcmtsMwWndvTuE0yxw58y95NWz2Lyxg/iUiiiDwgIu+KyDsicr+IJPodlzElrPrXVJclgKZSLZo2ZMSA7hQWFvGf+asqPE5EbDxAEw1eBfoAzwJ/A3oDr1V6hjEhlGUdQEw1WQJoqjR2jDOCxcwvKm/fZ+MBmijQQ1VvU9Uv3GUC0N3voIwpsc2GgDHVZAmgqdIVo50E8JOvVpOXX/HsI12Hu+0A7QmgiVzLRaS0m7uInA185WM8xpQ69m0OezdtpV5iAu369vA7HFPHWQJoqtT5jJb069GRo7l5fLFkfYXHdRjYl7j4eHau3UjuoZwQRmhMyJwNLBCRLBHJwpkJ5FwRWS0iFbeRMCYEtmU4v4Lt+/cmtp53bcRNZLAE0FTLFaP7AzBzzvIKj6mXkEDHQX0B2Lp4RQiiMibkLga6AOe6SxfgUuBy4Aof4zLGqn9NjVgCaKpl7BinevejL1dQXFxc4XHftQO0amATeVR1G9AUJ9m7AmiqqttKFl+DM1HPegCbmrAE0FRLv+4d6NS2Bbv355C+puJhXr7rCWwdQUzkEZH7gNeBVu7yDxH5qb9RGeOwHsCmJiwBNNUiIqWdQT6opDdwyYDQWxctp7ioKCSxGRNCtwFnq+ojqvoIMBS43eeYjOHQzj3k7NxD/SaNaNXNxis3VbME0FRb6XAwlbQDbNK2FS27dCDv6DF2rtkQqtCMCRUBAr/ZFLnbjPFVVvpKADqlnUVMjP1pN1Xz9LdERC4WkQ0ikulOoF5+v4jIM+7+VSIyMGDffSKyRkTWisj9XsZpqmfEgO40a5zExqzdfL11V4XHpdiA0CZyvQwsFpFHReRRYBFw8pyJxoRY1hI3AbTqX1NNniWAIhILPAdcgjNa/ngR6V3usEuAbu4yAZjsntsXp1plCNAPuFxEunkVq6meuLhYLhvlNC7+sJKngDYgtIlUqvo08GPgIPAt8GNV/Yu/URljPYBNzXn5BHAIkKmqW1Q1H5gBXFnumCuBV9WxCGgqIm2BXsAiVc1V1ULgS+BqD2M11VTSDnDmF5UkgMOcY7YstATQRBYReU1Vl6nqM6r6V1VdLiI2FZzxlaqyza0Cth7Aprq8TADbATsC1rPdbdU5Zg0wSkRaiEgDnHG2OgQrREQmiEiGiGTs27ev1oI3wV04vC+JCfVYvHozu/YdCnpMuzN7kpDUgH2bt3F4j/1MTETpE7ji1nQM8ikWYwDYm5lF7qHDNG6TTNN2bfwOx4QJLxPAYA2jtTrHqOp64AngU+BjYCVQGKwQVZ2iqmmqmpacnHw68ZpqSKqfwPlnO38D/z13RdBjYuPi6Hx2f8CeAprIICK/EpEjwFkicthdjgB7gQ98Ds9EucDqXxHrk2Sqx8sEMJuyT+3aAzure4yqTlXVgao6Cqe9zSYPYzU1MLZkVpBKqoFtPEATSVT1f1W1EfAnVW3sLo1UtYWq/srv+Ex0y7LqX3MKvEwA04FuItJFROKB64GZ5Y6ZCdzk9gYeCuSo6i4AEWnlvnYErgHe8DBWUwOXndsfEeG/S9Zz+OjxoMekDHc6dNuMICbCfCQiSQAicoOIPC0infwOykQ36wFsToVnCaDbeeMe4BNgPfCmqq4VkTtF5E73sFnAFiATeAm4O+AS74jIOuBDYKKqfutVrKZmWjVvzPD+qeQXFPLJgtVBj+ky1EkAt2WsoiAvL5ThGeOlyUCuiPQDHgK2Aa/6G5KJZkWFhexYvhZwxgA0pro8HQdQVWepandV7aqqf3C3vaCqL7jvVVUnuvvPVNWMgHNHqmpvVe2nqp97GaepubFub+AP56wIuj+pWRPa9EqlMC+/9OZkTAQoVFXFGcHgr6r6V6CRzzGZKLZr7UYKjp8guWsnGrZo5nc4JozYcOHmlJQMB/OfeSvJLwjaP6e0HaB1BDER5IiI/Aq4Afi32wu4ns8xmSi21ap/zSmyBNCcktSOrendtR05R48zd2nwKd9sQGgTga4D8oDbVHU3zrBVf/I3JBPNSnsAWwcQU0OWAJpTVjI3cEWzgnR1O4JsXrAUp9bMmPCmqrtV9WlVneeub1dVawNofFM6ALTNAGJqyBJAc8rGBswKEizBa9U9haTmTcnZuYeD278JdXjG1Aki0kFEvhCR9e7c5ve52x8VkW9EZIW7XOp3rCa85Oce55vVG5CYGDoM6FP1CcYEsATQnLKBvTrRrlUzvtn7LcvWZZ20PyYmhpRhNhyMiXqFwM9VtRcwFJgYMC/6X1S1v7vM8i9EE452rFhHcVERZ/TpTkJSA7/DMWHGEkBzymJiYrjCHRT6gwqqgVNsQGgTQUTkchGp0X1TVXep6jL3/RGcYbHKT4tpTI1lWfWvOQ2WAJrTMnaM84SvonaAKcOcamLrCWwixPXAJhF5UkR61fRkEekMDAAWu5vuEZFVIjJNRIKO4WHznZuKlHQAsR7A5lRYAmhOy6hBPWjcsD5rM78hc/uek/Z3GtyPmNhYslesI+9Yrg8RGlN7VPUGnARuM/CyiCx0E7QqxwIUkYbAO8D9qnoYZ1DprkB/YBfw5wrKtPnOTVAlM4BYD2BzKiwBNKclvl4cl4xwvn0GewqY2DCJ9v16UVxUVFpdYUw4c5O3d4AZQFvgamCZiPy0onNEpJ57zuuq+q57nT2qWqSqxTgzIQ3xPHgTMXIP5bB301biEuJpd2YPv8MxYcgSQHPavhsOZkXQ/TYeoIkUInKFiLwH/BdnAOghqnoJ0A94sIJzBJgKrFfVpwO2tw047GpgjWeBm4izLcOZhrPDgD7E1rOxyE3NWQJoTttFw8+kXlwsC1ZuYu/BwyftLxkP0HoCmwgwDqfn7lmq+idV3QugqrnArRWccw5wI3BeuSFfnhSR1SKyChgD/CwUH8BEBqv+Nacrzu8ATPhr3LA+5w3pxScL1jBr7kpuuWpkmf0pAVPCFRcXExNj3ztMeFLVmyrZF3TOclWdD0iQXTbsizll1gPYnC77S2xqRUlv4A++OLkdYPOO7WjSthXHDh5i78YtoQ7NmNMmIkdE5HCQ5YiInPzY2xiPWQ9gc7osATS14vJz+wPw+eK1HDueV2afiJR5CmhMuFHVRqraOMjSSFUb+x2fiS6Hdu7h0De7SWzciFbduvgdjglTlgCaWtE2uSlDzkzhRF4Bny48uS17VxsQ2oQ5EYkREeuoYXxXUv3bKe1Ma1JjTpn95phaEzg3cHkp1hHEhDl3uJaVItLR71hMdCup/rX2f+Z0WAJoak1JO8BZ81ZSWFhUZl+HAX2IS4hn17pNHPs2x4/wjKkNbYG1IvK5iMwsWfwOykQX6wFsakOVCaCIJIrItSLyVxF5S0ReFZGHRKRPNc69WEQ2iEimiEwKsl9E5Bl3/yoRGRiw72cislZE1ojIGyKSWPOPZ0KpZ5e2dOvUmoM5x/hqxaYy++olJNApzWmsvHWRVQObsPUYcDnwO5yZO0oWY0JCVdmWYR1AzOmrNAEUkUeBr4BhOHNXvgi8CRQC/ycin4pI0N9AEYkFngMuAXoD40Wkd7nDLgG6ucsEnKmREJF2wL1Amqr2BWJx5uA0dVzJU8CZX5yc5NmA0CbcqeqXwRa/4zLRY9/mbeR+m0PjNsk0a9+26hOMqUBVTwDTVXWQqv5cVf+pqp+p6keq+rSqXgH8CIiv4NwhQKaqblHVfJxpk64sd8yVwKvqWAQ0DRgdPw6oLyJxQANg56l8QBNaVwa0A1TVMvtShjn7Nls7QBOmRGSoiKSLyFERyReRIhsGxoRSYPWvM8mMMaem0gRQVf9dfptbJdzY3b9XVTMqOL0dsCNgPdvdVuUxqvoN8BSwHWeS9BxVnR2sEHci9gwRydi3b19lH8eEwJAzU2jdojHbdh1g1cYdZfalDHOeDmYtWUlRYaEf4Rlzuv4GjAc2AfWBn7jbjAmJ0h7AVv1rTlONOoGIyE+AT4B/i8gfqzo8yDatzjEi0gzn6WAX4AwgSURuCFaIqk5R1TRVTUtOTq4iJOO1mJiY0jEBZ84p2xu4SZtWtEzpSN7RY+xcs8GH6Iw5faqaCcSqapGqvgyM9jkkE0WsB7CpLVW1Abyi3Kbvqeq5qjoSuKyKa2cDHQLW23NyNW5Fx3wP2Kqq+1S1AHgXGF5FeaaOGDu6pB3gycPB2HiAJszlikg8sEJEnhSRnwFJfgdlokNRYSHblzlDUZZ0qjPmVFX1BLCfiHwgIiVfNVaJyOsi8g9gbRXnpgPdRKSLe8O8Hig/XMJM4Ca3N/BQnKreXThVv0NFpIE4jRzOB9bX5IMZ/4wZ0ouGDRJYuWE7WTv3l9ln4wGaMHcjzn3zHuAYzhfY7/sakYkau9ZupOD4CVqmdKRhi2Z+h2PCXFxlO1X1cRFpA/zObWz6CNAQaKCqq6o4t1BE7sGpMo4FpqnqWhG5093/As5k6JcCmUAu8GN332IReRtYhtPjeDkw5ZQ/pQmpxIR6XDj8TN79LIOP5iznnh9eULqva2lPYEsATfhR1W0iUh9oq6qP+R2PiS5ZVv1ralF12gAeA+7HGdJlCk4D6I3VubiqzlLV7qraVVX/4G57wU3+cHv/TnT3nxnYoURVf6uqPVW1r6reqKp5FZVj6p6xY9zewOXaAZ7RtwcJDZPYv3UHObv2+hGaMafMbRazAvjYXe9vA0GbUNmWbgNAm9pTVRvAx4F/A58DY1R1LLASpxPIjSGIz4SpS0acRVxcLPOWbeRgztHS7TGxsXQ5uz8AWxZaO0ATdh7FGeLqEICqrgA6+xaNiSolQ8BYD2BTG6p6Ani5qo7C6YBxE4CqzgQuApp7HJsJY80aJ3HuoB4UFRUza17Z1gKlA0JbAmjCT6Gq2lyGJuTyj5/gm9UbkJgYOg7s63c4JgJUlQCuEZHXgLeA0tHuVbVQVf/qaWQm7F0RMCh0oK5uRxAbENqEoTUi8kMgVkS6icizwAK/gzKRL3vFWoqLijijT3cSkhr4HY6JAFUNBH0D8CTwsKr+LDQhmUhRkgDOXrCa4yfyS7d3Geps356xmoI8a9ppwspPgT5AHvBPIAenjbQxntpq1b+mllXVBnCEqq5W1a8r2N9YROxZtAmqQ5vmDOzVidwT+Xy+eF3p9gZNm3BGn+4U5uezwx3TyphwoKq5qvo/qjrYXR5W1RN+x2UiX+kA0JYAmlpSVRXw90VkgYg8IiKXicgQERklIre6VcMf4UyHZExQJU8BPyzXGzjFBoQ2xphq+24GkP7+BmIiRlVVwD/DmfFjFzAO+D3wANANeFFVR6lquudRmrA1dozT3u+jL1dQVFRcuj1lmJMY2niAxhhTudxDOezZuIW4hHjandnD73BMhKh0IGgAVf0WeMldjKmRvqntSGmfzJbsfSxalck5A7oDgU8Al6KquAONG1Onicg5qvpVVduMqU3bMlYD0KF/b2Lr1fM5GhMpqjMQtDGnTEQYW1oNvKJ0e+vuKSQ1b8rh3fs4sC3bp+iMqbFnq7nNmFpTOgC0Vf+aWmQJoPFcSTvAD75YhqoCTmJYOh6gtQM0dZyIDBORnwPJIvJAwPIozlSXxnimZAo46wFsalOVCaCIxIjI8FAEYyLTsH6ptGzakM079rJuy87S7TYeoAkj8TjzoMcBjQKWw8C1PsZlokDJDCDWA9jUpuq0ASwWkT8Dw0IQj4lAcXGxXHZuf175YD4zv1hGn67tgIAZQSwBNHWcqn4JfCki01V1W03OFZEOwKtAG6AYmKKqfxWR5sC/cKaSywJ+4La5NqZUzq69HPpmN4mNG9Gqe4rf4ZgIUt0q4Nki8n2xlvrmFF1x7sntADulnUVMbCzZK9dz4ugxnyIzpkZyReRPIjJLRP5bslRxTiHwc1XtBQwFJopIb2AS8LmqdsOZb32St6GbcJTltv/rlHYmMTHWasvUnur+Nj2AMx1cvogcFpEjInLYw7hMhPne0N7UT4wnY+1WsvccBCAhqQHt+/dGi4tLqziMqeNeB74GugCP4Ty5q3QoLFXdparL3PdHgPVAO+BK4BX3sFeAqzyJ2IQ1q/41XqlWAqiqjVQ1RlXrqWpjd72x18GZyNGgfgIXDOsDOGMCluhaUg280DqCmLDQQlWnAgWq+qWq3orzVK9aRKQzMABYDLRW1V3gJIlAqwrOmSAiGSKSsW/fvtP+ACa8WA9g45VqP08WkbEi8pS7XO5lUCYyjQ0yK0iK2xHE2gGaMFHgvu5yZ0caALSvzoki0hB4B7hfVatdg6KqU1Q1TVXTkpOTax6xCVuqaj2AjWeq7AQCICL/BwzGqf4AuM+dJ9jarJhqu3RkP2JihC/Sv+bQkVyaNmpQ5glgcXGxtXExdd3jItIE+DnO+H+NgZ9VdZKI1MNJ/l5X1XfdzXtEpK2q7hKRtsBer4I24Wnf5m3kfptD49Ytada+rd/hmAhT3b+2lwIXqOo0VZ0GXOxuq5SIXCwiG0QkU0ROShbF8Yy7f5WIDHS39xCRFQHLYRG5vwafy9RBLZs1YsSA7hQWFvHxfOdbbbMOZ9C0XRtyv81hz4YtPkdoTOVU9SNVzVHVNao6RlUHqerMys5xO89NBdar6tMBu2YCN7vvbwY+8CZqE64Cq3+tD6apbTV53NI04H2Tqg4WkVjgOeASoDcw3u35FugSnHmFuwETgMkAqrpBVfuran9gEJALvFeDWE0dNXaMUw08060GLjsgtFUDm4h0DnAjcF7Al9pLgf8DLhCRTcAF7roxpaz613ipugngH4HlIjJdRF4BlrrbKjMEyFTVLaqaD8zA6fUW6ErgVXUsApq6VSGBzgc213TsLVM3lcwK8slXq8nLd5pTpQxzttmA0CYSqep8VRVVPavki62qzlLVA6p6vqp2c18P+h2rqVusB7DxUrVmAsEZvHQo8K67DFPVGVWc2g7YEbCe7W6r6THXA29UEp/1kAsjXdolc2a39hw5doI56V8DAT2BLQE0xhgAigoL2b5sDeCMmWpMbasyAVTVYuAedyyrmar6garursa1gzVY0JocIyLxwFicMQgris96yIWZsWOcnr8zv3CqgTsM6ENcQjy7v97MsYOHfIzMmMqJSGsRmSoi/3HXe4vIbX7HZSLPrnWbKDh+gpYpHWnYsrnf4ZgIVN0q4E9F5EER6SAizUuWKs7JBjoErLcHdtbwmEuAZaq6p5pxmjBQMhzMR18up7i4mLj4eDoP7gfAlkXLKzvVGL9NBz4BznDXNwL3+xWMiVxW/Wu8Vt0E8FZgIjAXp/3fUiCjinPSgW4i0sV9knc9Tq+3QDOBm9zewEOBnJKBUV3jqaT614Sn/j070qFNc3btzyFj7VbAxgM0YaOlqr6J0ywGVS0EivwNyUSikh7Andwvx8bUtuq2AZykql3KLZXOSu3eGO/B+ba8HnhTVdeKyJ0icqd72CxgC5AJvATcHVBuA5yece9iIoqIlD4F/MCtBraewCZMHBORFrhNVUq+uPobkolEJT2AOw+xBNB4o7ptACeeysXdnm7dVbWrqv7B3faCqr7gvldVnejuP1NVMwLOzVXVFqpqN9cIVNoO0B0OJmWYs7518QqKCgt9i8uYKvwcp+aiq4h8BbwK3OtvSCbS5B8/wTerNyAxMXQc2NfvcEyE8rINoDEVGjGgG00bNWDD1l1syNpF41YtSe7aifzc43yz6mu/wzMmKFVdCpwLDAfuAPqo6kp/ozKRJnvFWooLC2nbuxsJSQ38DsdEKC/bABpToXr14rh0lFO18eGcFcB31cA2HqCpq0RkM/ATVV3rzgZSICIf+R2XiSxW/WtCoVoJYJD2f1W2ATSmKiXtAGd+sQyAs2+4it+s/oRz77qBE0eOUlxc7Gd4xgRTAIwRkZfdzm1w8tilxpyWbSUJoPUANh6qNAEUkYcC3o8rt6+qmUCMqdSFw/uSEB/H4tVb2Ln3W87o24N//fRR7knswfNjb+fI3gOWBJq6JldVr8Pp2DZPRDpx8vimxpyWkiFgrAew8VJVTwCvD3j/q3L7Lq7lWEyUadggkfPP7o2qMvOzdKb96H42zllIcWEhG+csZOr4e8k/lut3mMYEEgBVfRL4Nc4oB+19jchElNxDOezZuIW4hHjandnD73BMBKsqAZQK3gdbN6bGxo52ev/OWrCWzPnpZfZlzk+3BtCmrnmk5I2qfg5cBPzNv3BMpNm+1Jn+rUP/3sTFx1dxtDGnrqoEUCt4H2zdmBq77Nx+iAj/XbKODucMLrMvdcRg8uwJoKkDRKSn+/YbERlYsgAtAOsEYmpN1pIVgFX/Gu/FVbG/n4gcxnnaV999j7ue6GlkJiq0btGEoWd1ZeHKTNpP/AkJ4jz5Sx0xmNv++Vfi7QmgqRseACYAfw6yT4HzQhuOiVTWA9iESqUJoKrGhioQE73GjhnAwpWZfLo8kykzXyK+QX12rc8ke+U6+lw82u/wjEFVJ7ivY/yOxUQ26wFsQqW64wAa45mSdoAfz19NbGIi6W/M5PdnXsTH/zfZ58iMKUtExolII/f9wyLyrogM8DsuExlydu3l2+xdJDZuRKvuNtKa8ZYlgMZ33Tq1pmeXthw6ksu8ZRs564rziYuPJ3PuEnJ27/U7PGMC/UZVj4jICJwOIK8AL/gck4kQWenu8C+D+hITY3+ejbfsN8zUCaVzA3+xnPpNGtP7olGoKsvf/cTnyIwpo8h9vQyYrKofANZV09SKbdb+z4SQJYCmTrhyjDsryJxlqCoDx10KwLK3/u1nWMaU942IvAj8AJglIgnYfdTUEusBbELJblymThjUuzNnJDcle8+3LF+/jX5jv0dsvXpsmruEw3v2+R2eMSV+gDP488WqeghoDvzC14hMRFBVtmWsBqwDiAmNqoaBMSYk8guKyJr9dJltz+VvYs+2nSx/92POvetGnyIz5juqmgu8G7C+C9jlX0QmUuzfsp1jBw/RuHVLmnU4w+9wTBSwBNDUCYkJ9YgfcOtJ2/OXT+Ofb82yBNAYE9FKO4AMdgbHN8ZrVgVs6ryNXy7m8N79fodhjDGeyVriJIBW/WtCxdMEUEQuFpENIpIpIpOC7BcRecbdv8qdWqlkX1MReVtEvhaR9SIyzMtYTd2lxcWseM96A5u6QUQ6icj33Pf1S8YFNOZ0WA9gE2qeJYAiEgs8B1wC9AbGi0jvcoddAnRzlwlA4Mi/fwU+VtWeQD9gvVexmrpv2Vuz/A7BGETkduBt4EV3U3vgfd8CMhGhqLCQ7cvWANApzZ4AmtDw8gngECBTVbeoaj4wA7iy3DFXAq+qYxHQVETaikhjYBQwFUBV890edyYKxcTFseGLhRzZd8DvUIyZCJwDHAZQ1U1Aq6pOEpFpIrJXRNYEbHtURL4RkRXucqlnUZs6bde6TeTnHqdllw40bNnc73BMlPCyE0g7YEfAejZwdjWOaQcUAvuAl0WkH7AUuE9Vj5UvREQm4Dw9pGPHjrUWvAmtE3kF5C+fFmR7Pr2+dw5rP/6SFe99wsgJP/QhOmNK5alqfkkjfRGJA7Qa500H/ga8Wm77X1T1qVqN0IQdq/41fvDyCWCwbkzlb5QVHRMHDMQZaX8AcAw4qQ0hgKpOUdU0VU1LTk4+nXiNjxIT6pW+P5FXwBlj7iV+wK2s+Ho7A8ddBsBSqwY2/vtSRH4N1BeRC4C3gA+rOklV5wIHvQ7OhKfAHsDGhIqXCWA20CFgvT2ws5rHZAPZqrrY3f42TkJookBiQj1uuWokAJPf/C/9r7qQmLg4Nn6xkKP77W+o8dUknNqJ1cAdwCzg4dO43j1uB7hpItIs2AEiMkFEMkQkY98+GxQ9ElkPYOMHLxPAdKCbiHQRkXjgemBmuWNmAje5vYGHAjmquktVdwM7RKSHe9z5wDoPYzV1zO3XjkZEeOfTDI4RQ8/zh1NcVMSK92f7HZqJYqparKovqeo4Vb3WfV+dKuBgJgNdgf44g0n/uYIyrZYjghWcOME3qzcgMTF0GNjX73BMFPEsAVTVQuAenGmT1gNvqupaEblTRO50D5sFbAEygZeAuwMu8VPgdRFZhXOD/KNXsZq6p0u7ZC4ZeRb5BYVMf38eg9xqYJsb2PhBRFa7T+qCLqdyTVXdo6pFqlqMc/8bUrtRm3CwY8U6igsLadu7G4kNk/wOx0QRT2cCUdVZOEle4LYXAt4rTq+6YOeuANK8jM/UbXeOG8OsuSt56e053DF9EjGxsXz9+QKOHviWhi2C1pYZ45XLa/uCItLWnUoO4GpgTWXHm8hk1b/GLzYTiKmzLhzel5T2yWzbdYB563fQ4zynGnilVQObEFPVbSULkIczNulZOL2Ct1V1voi8ASwEeohItojcBjxZ8mQRGAP8zMOPYOqokh7AnSwBNCFmCaCps2JiYpgwbgwAL775XwaOc4ZJW2rVwMYnIvITYAlwDXAtsEhETp7EuhxVHa+qbVW1nqq2V9Wpqnqjqp6pqmep6tiAp4EmipT0AO48pL+/gZioYwmgqdNuHjuChPg4PlmwhqZnDyqtBj528JDfoZno9AtggKreoqo3A4OAX/ockwlTuYdy2LNhC3Hx8bQ7s0fVJxhTiywBNHVai6YN+cFFzvjhr3++gu5jhlFcWMjKD6wa2PgiGzgSsH6EsoPZG1Nt25c6zT7b9+9FXHy8z9GYaGMJoKnz7rruPACmfzCPvldfBNig0Ca0ROQBEXkA+AZY7E7j9ltgEc4oBsbUmFX/Gj9ZAmjqvLQ+XRjUuzMHc46xuUkyEhPD1599xbFvc/wOzUSPRu6yGXif72Y1+gBnDD9jaqx0CjjrAGJ84OkwMMbUljt/MIbbH32ZV/6TzrjRQ9nw3wWs/GA2w28Z53doJgqo6mN+x2AiT8kQMDYFnPGDPQE0YWHchUNo1jiJjLVbSTp3BADLrBrYhJiIpInIeyKy7HQHgjbRLWf3Xr7N3kVio4a07pHidzgmClkCaMJCg/oJ3HKlk/h9dRQkJob1n84n95BVA5uQeh14Gfg+cEXAYkyNlI7/l3YmMTH2p9iEnv3WmbBx+7WjAXhv7krajTybooICVs38zN+gTLTZp6ozVXVrucGhjamR76p/rf2f8YclgCZspHZszYXD+3Iir4DdvZxJ0603sAmx34rI30VkvIhcU7L4HZQJP9YD2PjNEkATVu78gTMkzKfbDoEI62fP43jOYX+DMtHkx0B/4GK+q/6t9XmCTWRTVesBbHxnvYBNWLlkxFl0bNOCrbsOcO6wYRQuWMDKmZ8x9EZ7CGNCop+qnul3ECa87d+ynWMHD9GoVUuadTjD73BMlLIngCasxMbGlLYF/LplO8B6A5uQWiQivf0OwoS376p/+yEiPkdjopUlgCbs/PjqkcTXi2PxjoMcja/Puk/mcvzwkapPNOb0jQBWiMgGdwiY1TYMjKmJ4uJiug4fxPOFm/nh849TXFzsd0gmSlkCaMJOq+aN+f4Faagqe/oPojA/n1Uffu53WCY6XAx0Ay7ku/Z/NgyMqZbi4mKO7D3A9Jsf5J7EHrx80wMc2XvAkkDjC0sATVi6c5zTGWRFTBJFIix7698+R2SigTvkSwfgPPd9LnYfNdWUfyyXqePvZeOchRQXFrJxzkKmjr+X/GO5fodmopCnNy4RuditKskUkUlB9ouIPOPuXyUiAwP2ZbnVKytEJMPLOE34GdqvK2d170DOiQKymrZm7cdfWjWw8ZyI/Bb4JfArd1M94B/+RWTCSUJSAzLnp5fZljk/nYSkBj5FZKKZZwmgiMQCzwGXAL2B8UEaT1+CU53SDZgATC63f4yq9lfVNK/iNOFJRLjrOucp4NYuPSjMy2f1R//1OSoTBa4GxgLHAFR1J9DI14hM2Pg2exepIwaX2ZY6YjB59gTQ+MDLJ4BDgExV3aKq+cAM4Mpyx1wJvKqORUBTEWnrYUwmglx/yVAaN6zP9uJYDtRvyFKrBjbey1dVBRRARJJ8jseEiQXT3+LdXz7BjVOfoPvoYcTExdF99DBue+MZ4u0JoPGBl+MAtgN2BKxnA2dX45h2wC6cG+xsEVHgRVWd4mGsJgwl1U/gpivO4W9vfMb6Fu1o/Z85nDhylMRGDf0OzUSuN0XkRZwvq7cDtwIv+RyTqeMy56fz+oRfU1RQwDm3/YC7Z75EQlID8o7lEp/UwOYCNr7w8rcu2OBGWoNjzlHVgTjVxBNFZFTQQkQmiEiGiGTs27fv1KM1YemOH4wBYGvLMzhWWMzqf1s1sPGOqj4FvA28A/QAHlHVZ/2NytRl+7N28MLVd1BUUMCYe2+h1/dGkNioIRITQ2Kjhpb8Gd94+ZuXjdNbrkR7YGd1j3Hb1qCqe4H3cKqUT6KqU1Q1TVXTkpOTayl0Ey56dG7LeUN6UYCQ2ayNDQptPOVW+f5XVX+B8+SvvojU8zksU0edOHKU56/4CUf3H6T3RaO49s8P+x2SMaW8TADTgW4i0kVE4oHrgZnljpkJ3OT2Bh4K5KjqLhFJEpFGUHrDvRBY42GsJozd4c4PvL5lO1bP+oITR4/5HJGJYHOBBBFpB3yGMzfwdF8jMnVScVERU394HzvXbKBNz67c/q+/ERtns6+ausOzBFBVC4F7gE+A9cCbqrpWRO4UkTvdw2YBW4BMnG/Td7vbWwPzRWQlsAT4t6p+7FWsJrxdcW5/2rVqxuHEJLbHNWCNVQMb74iq5gLXAM+q6tU4oxwYU8Z7v3qS1R99TlLzptz94VTqN2nsd0jGlOHp1xFVnYWT5AVueyHgvQITg5y3BejnZWwmcsTFxfKT75/LY5PfZ33Ldix9axZp19nkDMYTIiLDgB8Bt7nb7LGOKWPB9Lf49E8vEhMXx4S3n6dVame/QzLmJNb61ESEW68eRVxsDNubJLNo9lc2rpbxyn04g0C/59ZopABf+ByTqUNKevwCXP+3x+gxZrjPERkTnCWAJiK0TW7KVecPQkVYm9SCNbPsb7Kpfao6V1XHquoT7voWVb3X77hM3VC+x++oO37kd0jGVMgSQBMx7nI7g2xocQaL37RBoU3tE5HuIjJFRGaLyH9LlmqcN01E9orImoBtzUXkUxHZ5L428zZ64yXr8WvCjSWAJmKMGNidHh1bcbxeAh/NXUV+7nG/QzKR5y1gOfAw8IuApSrTgYvLbZsEfK6q3YDP3XUThgJ7/LbukcJPZjxrPX5NnWcJoIkYIsLEH10IwJrGyay2amBT+wpVdbKqLlHVpSVLVSep6lzgYLnNVwKvuO9fAa6q3VBNqJT0+G3QrAkTP5pGg6ZN/A7JmCpZAmgiyo8uG0b9uBh2N2zGR//40O9wTOT5UETuFpG2bhVucxFpforXaq2quwDc11bBDrLZjuq2wB6/d7wz2Xr8mrBhCaCJKI2S6vOD7w0C4L2V26wa2NS2m3GqfBcAS90lw8sCbbajuivzqwz+ecf/ANbj14QfSwBNxLnvNmcMwI2NWrL4g898jsZEElXtEmRJOcXL7RGRtgDu697ai9R4bX/WDl68+g4K8/MZ81Pr8WvCjyWAJuL0TW1P31aNKIyNY8or1hvYnD4ReSjg/bhy+/54ipedifNEEff1g1O8jgmxkh6/R/YdoPeFI7n2aevxa8KPJYAmIt1zo9Phcvb2Q+RZNbA5fdcHvP9VuX3le/eeRETeABYCPUQkW0RuA/4PuEBENgEXuOumjisuKmLaj+7/rsevzfFrwpQlgCYi3XDdBTSkmG8TGvCPF9/yOxwT/qSC98HWT6Kq41W1rarWU9X2qjpVVQ+o6vmq2s19Ld9L2NRB7//6SVZ9+BkNmjXh7g+nWo9fE7YsATQRKb5eHJf1aQ/A39+b73M0JgJoBe+DrZsItfCVt5n9ZMkcv5Np3a2L3yEZc8osATQR68H7rkO0mOWH89m+Y4/f4Zjw1k9EDovIEeAs933J+pl+B2e8l/lVxndz/D77KD3Psx6/JrxZAmgiVr/BZ9JT8imWGJ58+nW/wzFhTFVjVbWxqjZS1Tj3fcl6Pb/jM946qcfvnTf4HZIxp80SQBPRfjj6LAD+NX8tBQWFPkdjjAk3J44cZfLY263Hr4k4lgCaiPbjidfT5MQxcgqVDz5P9zscY0wYKS4qYtoN9/PN6q+tx6+JOJYAmojWKrUzwxKLAPjLSzN9jsYYE07e//WTrJppPX5NZLIE0ES8G68cSVxREelb9vD11l1+h2OMCQPW49dEOk8TQBG5WEQ2iEimiEwKsl9E5Bl3/yoRGVhuf6yILBeRj7yM00S2UT8aS8q3uwGYPMOmhjPGVG7zgqXW49dEPM8SQBGJBZ4DLgF6A+NFpHe5wy4BurnLBGByuf33Aeu9itFEh9bdujC6RTwAr34wn2PH83yOyBhTVx3Yls0LV02gMD+f0ffcbD1+TcTy8gngECBTVbeoaj4wA7iy3DFXAq+qYxHQNGBy9PbAZcDfPYzRRIlLx11Eq6OHOJZXwBuzFvodjjGmDgqc47fXBSMZ95ff+B2SMZ7xMgFsB+wIWM92t1X3mP8HPAQUV1aIiEwQkQwRydi3b99pBWwi16Bxl9LrwDcAPD/jc1Rt8gZjzHeKi4utx6+JKl4mgMHmxyz/VzfoMSJyObBXVZdWVYiqTlHVNFVNS05OPpU4TRRo3T2F4R2akViQz5rMb1i0crPfIRljfFZcXMyJI0fR4mIOZe8mvkGD0h6/Sc2sx6+JbF4mgNlAh4D19sDOah5zDjBWRLJwqo7PE5F/eBeqiQZDrr2U7gedX8EX3vqvz9EYY/xUXFzMkb0HeH7s7UxM6M70m3/OlX94kJ9/+S/r8WuigpcJYDrQTUS6iEg8cD1QfiC2mcBNbm/goUCOqu5S1V+pantV7eye919VtZa45rQMGncpPffvRFR559MM9h487HdIxhif5B/LZer4e9k4ZyHFhYVsnLOQ1277JS06t/c7NGNCwrMEUFULgXuAT3B68r6pqmtF5E4RudM9bBawBcgEXgLu9ioeY9r0TKV7j050OLyf/IJCXn5vnt8hGWNCRFXZtT6TuS++zrQb7ie+QX0y55edHShzfjoJSQ18itCY0PK0hauqzsJJ8gK3vRDwXoGJVVxjDjDHg/BMFBo07lKWPPUK25sk89Lbc3jwlkuIjbXx0I2JNEWFhWSvXM+muYvJnJdO5rx0ju4/WLr/okl3kTpiMBvnfDcqQOqIweQdyyWxUUM/QjYmpKyLk4kqA8ddRrvf/oUmBSfYvvsAs+at5IrRA/wOyxhzmgpOnCArfRWb5i4hc94SNn+1lLyjx8oc06RtK1JHDqbbqLNJbNiA2954hqnj7yVzfjqpIwZz2xvPEG9PAE2UsATQRJW2vVJp16c73ffuIL1dN1586wtLAI0JQyeOHGXzgqVuwpdO1uIVFObnlzkmuWsnuo0aQurIwaSOHEJy106IfDf4RHFxMXfPfImEpAbkHcslPqkBMTFWI2CigyWAJuoMuPYStj7+HCvad2P2gjVkbt9DasfWfodlTNQrLi4m/1hu0ITsyL4DZM5PL034dixfixZ/N0ysiNDuzJ6kjhpCNzfha3pG5f+vY2JiSqt7rdrXRBtLAE3UGTTuMv792F9JPbKf9Q1b8tLbc3jigev8DsuYqFYyLEtgleyPX/sLa//zBZ/9ZSq712eWOT4mLo7Og88iddTZdBs5mK7npJHUvKk/wRsThiwBNFHnjD7dadMrlU8+fpE2HduetP9EXgGJCfVqvdyKrutVedFSph+f0dQuVeX4ocOlw7IAbJyzkJdv/BnXPfsouyf8inr1E0kZNtBpwzdyCF2GDrAeu8acBksATVQaNO5S2nRsS/yAW0/al798Gu9+llHrZV7zvbSQlhctZVZWnqmbiouKTuqh+8SuJUGHZWnbqxsPLXyXjgP7Ehcf71PExkQeSwBNVBo47rJK91//i+drvczKEhIvyouWMi3Rq/tKeuhmzktn09zFbFmwjBNHjpY5Zs/GrUGHZcnPzSVl6MBQh2xMxLME0ESlM/p0r3T/VecNClEk/pQXTWWa0CvpoeskfEvIWrKCwryyPXRbpnR0e+gOoduoIbRM6WjDshgTQpYAmqgUOBREMG/+udLxyWtdqMuLpjKN90p66JYkfOV76AKc0bdHmSFZmrVrc9J1GrVqYcOyGBMilgAaY4yPRCQLOAIUAYWqmuZvRI7KhmQ5uP0bNs1LJ3PuYjbNS6+4h+5IN+EbMbhaPXRtWBZjQscSQBO1juflB20/djwvn/oJtd/YPNTlRUuZfnxGD4xR1f1+B1Ei+JAsT7Pyg0+Z/acXObjtmzLH10tM+K6H7qizrYeuMWHAEkATtSQ/n6cvvrlMo/Puo4dxw5Q/8rcf/6LWy7v55T8xecKvQ1ZetJRZUXl3z3wJwicBrFPyj+UGGZLlAa579lFm3PMI9Zs0InXEYDfhG0LHQWdaD11jwowlgCZqJSQ1CDrsRMuUjmz+qvaHK2mZ0jGk5UVLmRWVF0ZPoBSYLSIKvKiqUwJ3isgEYAJAx44dQxJQRf832vbqxsMrZnFG3x7ExMaGJBZjjDcsATRRK+9YbtBhJ3IP5vDgvLdqvbzcgzkhLS9ayqyovLxjueHSjuwcVd0pIq2AT0Xka1WdW7LTTQinAKSlpWkoAqro/0Z+bi7t+/UORQjGGI+JakjuJyGRlpamGRnePNUwkSdYO6fb3niGRq1aeNLzMNTlRUuZp1OeiCytK50uAETkUeCoqj4VbH+o7nF+/N4YY7xR0X3OEkAT1Srr6RgJ5UVLmadant8JoIgkATGqesR9/ynwO1X9ONjxobzH+fF7Y4ypfRXd56wK2ES1UA874ccwF9FQZhgPH9IaeM8dlzIO+GdFyV+ohfG/qTGmGjz9OiciF4vIBhHJFJFJQfaLiDzj7l8lIgPd7YkiskREVorIWhF5zMs4jTHGD6q6RVX7uUsfVf2D3zEZY6KDZwmgiMQCzwGXAL2B8SJSvvXwJUA3d5kATHa35wHnqWo/oD9wsYgM9SpWY4wxxpho4uUTwCFApvsNNx+YAVxZ7pgrgVfVsQhoKiJt3fWSmcLruUvkNFY0xhhjjPGRlwlgO2BHwHq2u61ax4hIrIisAPYCn6rq4mCFiMgEEckQkYx9+/bVVuzGGGOMMRHLywRQgmwr/xSvwmNUtUhV+wPtgSEi0jdYIao6RVXTVDUtOTn5dOI1xhhjjIkKXiaA2UCHgPX2wM6aHqOqh4A5wMW1HqExxhhjTBTybBxAEYkDNgLnA98A6cAPVXVtwDGXAfcAlwJnA8+o6hARSQYKVPWQiNQHZgNPqOpHVZR5BNjgyQcKriUQ6gnco6FM+4xWZqjK66SqYVN14MM9Duz3xsoMn/Kipcxauc95Ng6gqhaKyD3AJ0AsME1V14rIne7+F4BZOMlfJpAL/Ng9vS3wituTOAZ4s6rkz7UhlIO6ikhGqAeRjYYy7TNameFSng9Ceo8D+72xMsOnvGgps7bK83QgaFWdhZPkBW57IeC9AhODnLcKGOBlbMYYY4wx0crm9THGGGOMiTKRlgBOifDyoqVM+4xWZriUF2rR8DP0o8xo+Ix+lBkNn9GPMmulPM86gRhjjDHGmLop0p4AGmOMMcaYKlgCaIwxxhgTZSIiARSRi0Vkg4hkisikEJQ3TUT2isgar8tyy+sgIl+IyHoRWSsi94WgzEQRWSIiK90yH/O6TLfcWBFZLiLVGfanNsrLEpHVIrJCRDJCVGZTEXlbRL52f6bDPC6vh/v5SpbDInK/x2X+zP29WSMib4hIopfluWXe55a31uvP5we7z3lSpt3nvCszZPc5P+5xbrnhfZ9T1bBecMYY3AykAPHASqC3x2WOAgYCa0L0GdsCA933jXAG2Pb6MwrQ0H1fD1gMDA3BZ30A+CfwUYj+bbOAlqEoK6DMV4CfuO/jgaYhLDsW2I0zMKhXZbQDtgL13fU3gVs8/lx9gTVAA5zhrT4DuoXy5xqCn5vd52q/TLvPeVemL/e5UNzj3HLC/j4XCU8AhwCZqrpFVfOBGcCVXhaoqnOBg16WUa68Xaq6zH1/BFiP88vnZZmqqkfd1Xru4mmPIRFpD1wG/N3LcvwkIo1x/rBOBVDVfHWmOwyV84HNqrrN43LigPrujEANOHkayNrWC1ikqrmqWgh8CVztcZmhZPc5b8q0+5wHfL7PheoeB2F+n4uEBLAdsCNgPRuPbxp+EpHOOINkLw5BWbEisgLYC3yqql6X+f+Ah4Bij8sJpMBsEVkqIhNCUF4KsA942a0C+ruIJIWg3BLXA294WYCqfgM8BWwHdgE5qjrbyzJxvhWPEpEWItIAZ4ahDlWcE07sPuddWXafq31+3uc8v8dBZNznIiEBlCDbInJsGxFpCLwD3K+qh70uT1WLVLU/0B4YIiJ9vSpLRC4H9qrqUq/KqMA5qjoQuASYKCKjPC4vDqdabbKqDgCOAZ635wIQkXhgLPCWx+U0w3k61QU4A0gSkRu8LFNV1wNPAJ8CH+NUkRZ6WWaI2X3OI3af84Qv97lQ3ePcssL+PhcJCWA2ZTPg9nj/GDbkRKQezk3xdVV9N5Rlu4/u5wAXe1jMOcBYEcnCqd46T0T+4WF5AKjqTvd1L/AeTlWbl7KB7ICnDG/j3ChD4RJgmaru8bic7wFbVXWfqhYA7wLDPS4TVZ2qqgNVdRRO1eUmr8sMIbvPeczuc7XKr/tcqO5xEAH3uUhIANOBbiLSxc3+rwdm+hxTrRIRwWlLsV5Vnw5Rmcki0tR9Xx/nl/1rr8pT1V+pantV7YzzM/yvqnr6bUpEkkSkUcl74EKcR+yeUdXdwA4R6eFuOh9Y52WZAcYTgqoRnCqRoSLSwP3dPR+nPZenRKSV+9oRuIbQfNZQsfucN2Xafc4DPt7nQnWPgwi4z8XVVlB+UdVCEbkH+ASn9880VV3rZZki8gYwGmgpItnAb1V1qodFngPcCKx226oA/FpVZ3lYZlvgFRGJxfmi8KaqhmTIghBqDbzn/N8lDvinqn4cgnJ/Crzu/iHfAvzY6wLd9iIXAHd4XZaqLhaRt4FlONUTywnNVEnviEgLoACYqKrfhqDMkLD7nGfsPuedkN7nQnmPg8i4z9lUcMYYY4wxUSYSqoCNMcYYY0wNWAJojDHGGBNlLAE0xhhjjIkylgAaY4wxxkQZSwCNMcYYY6KMJYDGGGOMMVHGEkBz2kRkuohcW0vXukVEzqiNa5W7bpqIPFPb1w0FEXlURB70Ow5jopnd57xl97nQC/uBoI2/3AFUa9MtOKPU1+o0V6qaAWTU5jUDiUicqkbS3LPGGJfd5xx2n4ss9gTQACAiN4jIEhFZISIvikisiEwWkQwRWSsijwUcmyUij4jIfGBcwPbzReS9gPULRCTofJ7u9aeLyBoRWS0iP3O/XafhjB6/QkTqu9dc7h4zTUQSAmJ4wo15iYikutuni8gLIjJPRDa6k68jIqNF5CP3/aPuteaIyBYRuTcgrt+IyNci8qmIvFHZN1L3/D+KyJfAfVXE2tJ9nyYic6oRx/+IyAYR+QzoEbD9XhFZJyKrRGRG1T9ZY0wJu8+VxmX3OQOqakuUL0Av4EOgnrv+PHAT0Nxdj8WZJP0sdz0LeCjg/OnAtYDgzKOZ7G7/J3BFBWUOAj4NWG/qvs4B0tz3icAOoLu7/ipwf0AM/+O+vwn4KCCWj3G+3HTDmZQ8EWdKq5JjHgUWAAlAS+AAUA/nprwCqA80wplk+8FK/t3mAM9XM9aW7vs0YE4VcQwCVgMNgMZAZkkcOE8MEgL/zWyxxZaqF7vP2X3OlrKLPQE04ExiPQhIF2cOzvOBFOAHIrIMZ47DPkDvgHP+Vf4i6vxvfQ24QZwJ1ocB/6mgzC1Aiog8KyIXA4eDHNMD2KqqG931V4BRAfvfCHgdFrD9TVUtVtVNbjk9g1z736qap6r7gb0482WOAD5Q1eOqegTnj0VVSv4dqoq1IsHiGAm8p6q5qnoYmBlw/CqcJwc34Mw/aYypHrvP2X3OBLA2gAacb7SvqOqvSjeIdAE+BQar6rciMh3n21+JYxVc62WcG8oJ4C2toL2Ie81+wEXAROAHwK1B4qqMVuN9sHWAvID3RTj/F6oqL5iSf4fKzi3ku+YWieX2BYsDgscMcBnODXcs8BsR6VPRv7Expgy7z9l9zgSwJ4AG4HPgWhFpBSAizYGOOP/pc0SkNXBJdS6kqjtxHt8/jFNNEZTbViRGVd8BfgMMdHcdwamWAKeapXNJuxfgRuDLgMtcF/C6MGD7OBGJEZGuON/wN1QndmA+cIWIJIpIQ5ybUHVVFmsWzpMHgO9X41pzgavdtkGNgCsARCQG6KCqXwAPAU2BhjWI0ZhoZvc5h93nDGBPAA2gqutE5GFgtvufrwDn2+pyYC1O9cJXNbjk6zjtY9ZVckw74GW3PICSb+XTgRdE5DhOdcePgbdEJA5IB14IuEaCiCzG+SIzPmD7BpybUmvgTlU9IVL1l15VTReRmcBKYBtOb7qcKk90zj0hIhXF+hgwVUR+DSyuxrWWici/cNrpbAPmubtigX+ISBOcb+J/UdVD1YnPmGhn9zmH3edMCXGaMxhTe0Tkb8ByVZ3qYRlZOI2o95fbPh2nEfTbp3jdhqp6VEQa4HxDnaCqy043XmNMZLH7nAl39gTQ1CoRWYpTpfJzv2M5RVNEpDdOG5ZX7KZojCnP7nMmEtgTQOM5t/oiodzmG1V1tR/x1JSIPAecU27zX1X1ZT/iMcbUPXafM+HGEkBjjDHGmChjvYCNMcYYY6KMJYDGGGOMMVHGEkBjjDHGmChjCaAxxhhjTJT5//WAchsDAOUJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 648x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(9, 4))\n",
    "\n",
    "ax[0].plot(number_of_steps, trn_err, marker='o', c=cm[0], markeredgecolor='w', linewidth=2)\n",
    "ax[0].plot(number_of_steps, val_err, marker='s', c=cm[1], markeredgecolor='w', linewidth=2)\n",
    "ax[0].legend(['Train err', 'Validation err'])\n",
    "ax[0].set_xlabel('early_stopping_rounds')\n",
    "ax[0].set_ylabel('Error (%)')\n",
    "ax[0].set_xticks(range(10))\n",
    "\n",
    "ax[1].plot(number_of_steps, n_iters, marker='o', c=cm[0], markeredgecolor='w', linewidth=2)\n",
    "ax[1].set_xlabel('early_stopping_rounds')\n",
    "ax[1].set_ylabel('Ensemble size at early stopping')\n",
    "ax[1].set_xticks(range(10))\n",
    "\n",
    "fig.tight_layout()\n",
    "pngFile = './figures/CH05_F20_Kunapuli.png'\n",
    "plt.savefig(pngFile, dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.3. Custom Loss Functions\n",
    "The **training loss** is the function that is optimized during training. Recall that we used squared error as the loss while building our own gradient boosting algorithm; ``LightGBM`` uses cross-entropy, otherwise known as the logistic loss. \n",
    "\n",
    "The **focal loss** is a new loss function introduced by [Lin et al.](https://arxiv.org/abs/1708.02002) for object detection. It is a modification of the classical cross-entropy loss that puts more focus on harder-to-classify examples, while ignoring the easier examples. The extent of this \"focus\" is determined by a user-controllable parameter, $\\gamma \\geq 0$.\n",
    "\n",
    "Let's define the model's predicted probability of $y_{pred} = 1$ as $p$. For a binary classification problem, since the only other label is $y=0$, the probability of $y_{pred} = 0$ will be $1-p$. The classical cross-entropy loss (for binary classification) is:\n",
    "\n",
    "\\\\[\n",
    "L(y_{true}, y_{pred}) = - y_{true} \\log{p} -  (1 - y_{true}) \\log{(1 - p)}.\n",
    "\\\\]\n",
    "\n",
    "The focal loss introduces a **modulating factor** to each term:\n",
    "\n",
    "\\\\[\n",
    "L(y_{true}, y_{pred}) = - (1 - y_{true}) p^\\gamma \\log{(1 - p)} -  y_{true} (1 - p)^\\gamma \\log{p}.\n",
    "\\\\]\n",
    "\n",
    "The modulating factor essentially suppresses the contribution of well-classified examples, forcing the algorithm that uses this loss to learn a model that focuses on poorly-classified examples. This effect can be seen below, where the focal loss is plotted for different values of $\\gamma$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEYCAYAAABiECzgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABR5klEQVR4nO3dd3wcxd348c/sXpVOvVdLttx7AwzYYJrBJBBCQgIkPJSEh/x4EtJ5kich/QFSngcIAcIDpFJDQgkYh2aaMTZucq+SbPXepWu78/tjZVmyik/2nc6S5/16LXe3tzsza+GvxrOz3xFSShRFUZTRp0W7AYqiKKcrFYAVRVGiRAVgRVGUKFEBWFEUJUpUAFYURYkSW7Qb0FdqaqosKCiIdjMURVHCatOmTQ1SyrRj959SAbigoICNGzdGuxmKoihhJYQ4NNh+NQShKIoSJSoAK4qiRIkKwIqiKFESsTFgIcRU4Nk+uyYCd0kp74tUnYqiRF4gEKCiogKv1xvtppxyXC4Xubm52O32kI6PWACWUu4F5gEIIXSgEnghUvUpijI6KioqiIuLo6CgACFEtJtzypBS0tjYSEVFBYWFhSGdM1pDEBcCB6WUg94JVBRl7PB6vaSkpKjgewwhBCkpKSP6l8FoTUP7PPD0YF8IIW4FbgXIz88PuUDTlPiCJt0Bg+6AgV0XpMe5wtJYRVGGp4Lv4Eb65xLxACyEcABXAN8b7Hsp5aPAowCLFi0KOTfmK9ur+e9nV5Gq16CLAHnTruTB6xaEpc2KoiijYTR6wJcBm6WUteEs1GXTuOnwI8zZ3UlQhz9N+kQ4i1cURYm40RgDvpYhhh9OhtuhYw8I2hPPwxlIpDtghLsKRVGUiIpoABZCxAAXA/8Id9luu05At1GWlkqXOwW/tyPcVSiKchpYvXo1U6dOpaioiHvuuWdU645oAJZSdkkpU6SUreEu22XX8dtseGUZXrsO3U3hrkJRlHHOMAxuv/12XnvtNXbt2sXTTz/Nrl27Rq3+UyoZz0i4HToBzYZdCoK6DbtXBWBFGU0F//lqRMsvu+fyIb/bv38/F198MW+++SZFRUUEAgEWLVrEq6++Sm5ubsh1bNiwgaKiIiZOnAjA5z//eV566SVmzJhx0u0PxdgNwHYdv+7AKQWGpmPztUS7SYqijJLJkydz66238q9//YuioiIefPBBrrzyyt7gu3TpUtrb2wec9+tf/5qLLrqo93NlZSV5eXm9n3Nzc1m/fn3kL6DHmA7APt1JgoSgbkP3h32UQ1GUU9isWbN48803aWpq4vHHH+8XON9///2QyhhsVfjRnOM8dgOwQ8enOdEkGJqGPTDwt52iKOPXlClT+N3vfsePf/xjvv3tbxMbG9v7Xag94NzcXMrLy3s/V1RUkJ2dHdmG9zFmA7DTpuHVXT0BWMcR7MQwJbqmntBRlNEw3BjtaJg0aRKbN2+mtbWV++67r993ofaAFy9ezP79+yktLSUnJ4dnnnmGp556KgKtHdyYDcBCCPy2WDTZiaHpuE0v3oBBrHPMXpKiKCNgt9uJj4/nnnvuQdNObEKXzWbjwQcfZMWKFRiGwc0338zMmTPD3NJh6h+1miIgaPegmR0YmsBpeulWAVhRTiuBQIDzzjvvpMpYuXIlK1euDFOLRmZMJ2Q37PHopsQUApfho9uvnoZTlNNFWVkZEyZMGNOJgcZ0AA44E9CliakJHKYfr3ocWVFOGwUFBSGP9Z6qxnQANtzJaKaJIQQOI6DyQSiKMqaM6QBsulOwGQZSgN0IqCEIRVHGlDEdgEVMIrppYgiwG0HVA1YUZUwZ0wE4xuEAaSA1sBmGGgNWFGVMGdMB2O3QEcLARGI3DNUDVhRlTBnbAdiugwwghYktaNLtN6PdJEVRlJCN6QDssutAEInEZpiqB6woypgypgOw26GDCCAx0YNSjQErijKmjO0AbNcxRRCJiT0o1TQ0RVFGLJQliQoKCpg9ezbz5s1j0aJFYat7TCdOcNt1EEEkBroh1RCEoigjcmRJojfeeIPc3FwWL17MFVdcMeiKGGvWrCE1NTWs9Y/pHrDLoSNFAImBLYgKwIpymti/fz8FBQUcOHAAsJLyzJ07l4qKihGV03dJIofD0bsk0WgZ8z1gUzeBIHpQw6uGIBRldL32n1CzPfzlZs6Gy4ZeoXi0lyQSQnDJJZcghODf//3fufXWW0/m6npFNAALIRKBx4BZgARullKuC1f5brtOm02ANNBNB96gCsCKMqpqtsOhD6JS9WguSbR27Vqys7Opq6vj4osvZtq0aSxbtuzEG98j0j3g+4HVUsrPCCEcQEw4C3c7NJp1AQTRTIe6Cacooy1zdtTKHc0liY7sS09P56qrrmLDhg2ndgAWQsQDy4AbAaSUfsAfzjpcdp2grgMammmn2x8MZ/GKohzPMMMEkTZaSxJ1dnZimiZxcXF0dnby+uuvc9ddd4XlGiLZA54I1AN/EELMBTYBd0gpO8NVgduuE9BtCKkjsOP3d4WraEVRTnGRXpJo5cqVPPbYY3i9Xq666ioAgsEg1113HZdeemlYriGSAdgGLAC+KqVcL4S4H/hP4Id9DxJC3ArcCpCfnz+iCtwOKwAT1DA1O8LbHJ6WK4oyJkRySaJVq1b1vi8uLj6pOoYSyWloFUCFlPLIyPjzWAG5Hynlo1LKRVLKRWlpaSOqwG3X8et2BBpBmwPN13TyrVYUZUxQSxINQ0pZA5QLIab27LoQ2BXOOtx2nYBmR0iNoGbD5msJZ/GKopzCxsOSRJGeBfFV4MmeGRAlwE3hLNzl0PHanDgRBHUbDn9rOItXFEWJqIgGYCnlViB8D04fw23X8WlOhNQwdBu2wMBpJ4qiKKeqMf0osl3X8GtuNCCo6ziNLgKGygmsKMrYMKYDMIDfEYMmBYam4za6VT4IRVHGjDEfgIMOD0KCoek4pVflg1AUZcwY+wHYHodmSgxNw2X4VA9YUZQxY8wHYMOZiC7B0DQcpl8FYEVRxowxH4CDrmQ0U2JqGg7DrxLyKIoyZoz5AIw7CV2aGJrAbgRUD1hRlBG5+eabSU9PZ9asWaNe95gPwFpMEropMYXAbhhqYU5FUUbkxhtvZPXq1VGpe8wHYKfbhWYaPQE4SLdfzQNWlPEuXEsSASxbtozk5ORwNzEkY3pJIgCXwwaYGBo4DFMNQSjKKLp3w73sadoT9nKnJU/jzjPuHPL7cC1JFG1jPwDbdYQ0kAJ0w1ABWFFG0Z6mPWys3RiVusOxJFG0jfkA7LbrgIGJxBaUtKtZEIoyaqYlT4taueFYkijaxkcAFgamkNiCaghCUUbTcMMEkRaOJYmibczfhHM7dCCIFBKbIVUAVpTTRDiWJAK49tprWbJkCXv37iU3N5fHH388jK0c3pjvAbvsOhBAIrEFUQ9iKMppJBxLEj399NNhas3Ijf0esF0HzUBioAelmgesKKcJtSTRKcDt0DBFEInEHkQNQSjKaWI8LEk09gOwXccUR3rAaghCUZSxY8wHYJddR+oSCGIzdNUDVhRlzBjzAdht1wnqGsggumHH6w9Gu0mKoighGfsB2GEFYCmD2E0H/kB3tJukKIoSkrEfgO06hmbNBdalHelriXaTFEVRQhLRecBCiDKgHTCAoJQy7EvUu+w6ft0GQYOg5kD3N4e7CkVRlIgYjQcxlkspGyJVuNuhE7A5IKhh6HZsXhWAFUUZG8bFEIRPdyDQCNrs2P2t0W6SoihKSCIdgCXwuhBikxDi1sEOEELcKoTYKITYWF9fP+IKXHYdr+ZCSI2AZsMeaEdKebLtVhTlNFBeXs7y5cuZPn06M2fO5P777x/V+iM9BHGOlLJKCJEOvCGE2COlfK/vAVLKR4FHARYtWjTiyKlrAr/uRuDF0HViZBe+oNmTI0JRFGVoNpuN3/zmNyxYsID29nYWLlzIxRdfzIwZM0al/oj2gKWUVT2vdcALwBmRqCdoj0EgCNhsuMxu9TScooxz4VqSKCsriwULFgAQFxfH9OnTqaysDHt7hxKxHrAQIhbQpJTtPe8vAX4aiboCDg+abMHQbLiMLroDBkmRqEhRlH5q/vu/8e0O/5JEzunTyPz+94f8PhJLEpWVlbFlyxbOPPPM8FxECCI5BJEBvNCTqcgGPCWljMjSo0FHPEIKDE3HIX3qcWRFGSW+3Xvo+vjjqNQdziWJOjo6uPrqq7nvvvuIj48Pd1OHFLEALKUsAeZGqvy+DGc8mhQYuo4z6FdDEIoySpzTI7MkUSjlhmtJokAgwNVXX83111/Ppz/96ZNv/AiM+YTsAKYzCU1KgpqOw/SrnMCKMkqGGyaItHAsSSSl5JZbbmH69Ol885vfjEArhzfm5wEDyNgUNAmmpmE3AmoIQlFOA+FYkmjt2rX85S9/4e2332bevHnMmzePVatWhbmlQxsXPWDdnYAmJYamYTeCaghCUU4TJ7sk0bnnnhvV5wbGRQ/YHhODbloB2GYYqgesKKcBtSTRKcLldCCkxNQENsNQY8CKchpQSxKdItwOHSFNDCGwGaYaglAUZUwYHwHYriMwMQXYgibeoBntJimKohzXuAjAVt4Hs2cIQvWAFUUZG8ZFAHY7dAQGJhI9KNUYsKIoY8K4CMBxLhsSA6mBLQj17b5oN0lRFOW4xkUAnpAcC8LqAduCktLGzmg3SVEU5bjGRQAuSI1BIpFCIgyoq6+LdpMURVGOa1wE4OwEN4YuMDGJ7Ya4wEFauvzRbpaiKMqwxsWjyJom6HZ5kEhS2iDNdpDShk7m5zui3TRFUU5xBQUFxMXFoes6NpuNjRs3jlrd4yIAA/iSsqE1gCYhK3CYssZO5uertOyKohzfmjVrSE1NHfV6x8UQBIAzewoQRAobSd5GShu6ot0kRVEiJFxLEkXbuAnASQWTEDKAoTuI7eqkrEHNhFCU8arvkkTAoEsSHUkv2Xd78803B5QlhOCSSy5h4cKFPProo6N6HeNmCCKzKJ9aM4DPmYC7o5oyNRVNUUbF7257O+xl3v7IBcc9JlxLEq1du5bs7Gzq6uq4+OKLmTZtGsuWLTuhdo/UuAnABVkpFBPA60zG1VZNTUMjUsoxnapOUcaCUIJlJIRrSaLs7GwA0tPTueqqq9iwYYMKwCOVmexBYuB1pZPYCon+QzR1+knxOKPdNEVRIiAcSxJ1dnZimiZxcXF0dnby+uuvc9ddd0WgtYMbNwHY7nQiMfE5k0ivgQznQcoaO1UAVpRxKhxLEtXW1nLVVVcBEAwGue6667j00kvD2cxhRTwACyF0YCNQKaX8RKTqsdkdgEm3K4n4bkjhMKUNXSyckBypKhVFibKTXZJo4sSJFBcXh7FFIzMasyDuAHZHuhKhaSA0vI4EAJJ8tZQ2dES6WkVRokQtSXQcQohc4HLgsUjWc4Sm2/G64gHwdHZQpuYCK8q4pZYkOr77gO8CQy5RIYS4VQixUQixsb6+/qQqc8TF47PpmELD2RmgVM0FVhTlFBaxACyE+ARQJ6XcNNxxUspHpZSLpJSL0tLSTqrOhPR09EA9fkcCjg6oaWyK6pLTiqIow4lkD/gc4AohRBnwDHCBEOKvEayPpNRUCDbgdSUT1wYpgQqVnF1RlFNWxAKwlPJ7UspcKWUB8HngbSnlFyJVH4AnOQWTNrzOJNJbINNRooYhFEU5ZY2bXBAAnqRkTM2L15VMWisk2Ms5WK8CsKIop6ZRCcBSynciOQf4CE9yCthMfM5k7AYkBWrYdKg50tUqiqKckHHVA45NSsZmN/C6rDzACd4mNpQ1RrlViqIogxtXAdiTlAL48DqtAKx1+Wloaqa6tTu6DVMURRnEuArAsUlJ+Lo66HYmIQF/p85cbT8bSpui3TRFUU5RN998M+np6cyaNavf/tWrVzN16lSKioq45557IlJ3SAFYCBErhNB63k8RQlwhhLBHpEUnwe5wYne5kPgI2tyktAgmOYtVAFYUZUg33ngjq1ev7rfPMAxuv/12XnvtNXbt2sXTTz/Nrl27wl53qD3g9wCXECIHeAu4Cfhj2FsTBp6kZGyyEZ8zicIaSUzMAT4uUwFYUcaTcC5JtGzZMpKT+yft2rBhA0VFRUycOBGHw8HnP/95XnrppbC0va9Qs6EJKWWXEOIW4LdSyl8KIbaEvTVhEJuUjOjy4XUlk9NYRZdo4UBtG82dfpJi1SrJijIe9F2SqKioaNAliUJNyD6YyspK8vLyej/n5ub2W3EjXEIOwEKIJcD1wC0jPHdUeZJSMNpk7424zhadaeIwH5c1ccnMzCi3TlHGn998LvwzTL/17CvHPSZcSxINZrAUBpHIuhZqEP068D3gBSnlTiHERGBN2FsTBp7kFDoam/G6rH9SJNTpzMrcwYbS5SoAK0oEhBIsIyFcSxINJjc3l/Ly8t7PFRUVvUsXhVNIAVhK+S7wLkDPzbgGKeXXwt6aMPAkJYNWQ2eCFWyLqiSVE3fxoRoHVpRxJRxLEg1l8eLF7N+/n9LSUnJycnjmmWd46qmnTqrMwYQ6C+IpIUS8ECIW2AXsFUJ8J+ytCYPY5GSC/jaMxBwAiqolAVcNO6va6PAFo9w6RVHCJRxLEgFce+21LFmyhL1795Kbm8vjjz+OzWbjwQcfZMWKFUyfPp1rrrmGmTNnhrH1llCHIGZIKduEENcDq4A7gU3Ar8LeopPkSUrG19WKYbNWxkjshLqASYZZz7qDjVw8IyPKLVQUJVxOdkkigKeffnrQ/StXrmTlypUnVfbxhPprw94z7/dTwEtSygBwSiba9SSn0N3WQgAbAZsbAKPBzgJtN6u2V0e5dYqihMvptCTR74EyIBZ4TwgxAWiLVKNORmxCEt3tbSRnx9IeZ01JmVADRe7NvLGrFm/AiHILFUUJh9NmSSIp5QNSyhwp5UppOQQsj3DbToim67jj4khIF3QVzAWsG3HumAN4fV7e23dyyx4piqKES6g34RKEEP9zZO02IcRvsHrDp6TYpGRi4wP4MqcDMLEGtrg0ztD2qGEIRVFOGaEOQTwBtAPX9GxtwB8i1aiT5UlOweH20m5PBcAVgOoOJ8ttH6lhCEUJA7XW4uBG+ucSagCeJKX8kZSypGf7CTBxxK0bJZ6kZDTRRWuXHdPKIURhNcR5ttHlD/CuGoZQlBPmcrlobGxUQfgYUkoaGxtxuVwhnxPqNLRuIcS5UsoPAIQQ5wCnbJJdT1IK3e0teJJT8aYVElN3kJmHJZsnS+Y3H+DVbbmsUE/FKcoJyc3NpaKigvp61ZE5lsvl6s1HEYpQA/BtwJ+FEAk9n5uBfxth20ZNbFIyVft2k5o7n8Dc8+GNg8wrkfzhchdf1Nfz293T6fQFiXWekuksFOWUZrfbKSwsjHYzxoVQZ0EUSynnAnOAOVLK+cAFEW3ZSYhLTqGzuYnUXA9dOdbTKx4vZNVoZMdupssf5B9bKqPcSkVRTncjen5PStkmpTwy//ebEWhPWMQmJdPR1Ehqrod2W0rv/vkHTbZ7/MwUh/jTh2VqDEtRlKg6mSWJTtnHTzzJKbQ3NpCSG0tjnR9XzzPc8w9K3olxc7X+DgfqOlh7QC3YqShK9JxMAB62+yiEcAkhNgghioUQO4UQPzmJukbEHRePEAJN82IaJvo51mjJxFrwd+vMdH+EGy9//LB0tJqkKIoywLB3oYQQ7QweaAXgPk7ZPuACKWVHTx6JD4QQr0kpPzqxpoZOCEFyTh5NVRWk5nrwFS3q/W5eieSdAo1Pdq3jb3tcHGrsZELKKftMiaIo49iwPWApZZyUMn6QLU5KOWzw7nlkuaPno71nG7VB1+ScPJoqy0nNjaPdnoaeZK2QMf+g5J+eWK6xvYGU8Od1h0arSYqiKP1EdFl6IYQuhNgK1AFvSCkHLKokhLj1yCPO4ZxXmJyTS2NlOam5Hhoqu4hdei4Ac0olHWjUe+qYLUp4ZsNhGjt8YatXURQlVBENwFJKQ0o5D8gFzhBCzBrkmEellIuklIvS0tLCVndKTh5NlRWk5cdRV9aGZ5mVMzTGD9MqJH+P8/AF/U06/QYPv3MwbPUqiqKEKqIB+AgpZQvwDnDpaNQHR4YgKkjOisXXFYTZi8FmjZqcvUuy3u1igXM98XTw548OUd16yj7YpyjKOBWxACyESBNCJPa8dwMXAXsiVd+x4tPS6G5vI+D3kj05kdpaE8+51jDEkj0Se1DyWryDm/R/4Q+aPPDWgdFqmqIoChDZHnAWsEYIsQ34GGsMeNSWT9U0naTMLJqrKsmekkjl/hYSrrwCsJ6KW3BA8qInlhtsrxFHF89tLKesoXO0mqcoihK5ACyl3CalnC+lnCOlnCWl/Gmk6hpKck4ejZXl5ExJpGpfC57ly9E8HgCW7ZDU22x87IEb9dUYpuRX/9o72k1UFOU0NipjwNFyZCpaSraH7g4/3T5B3KUrAFhwUBLXJXksIZ6bba/hoYtXt1erVJWKooyacR6Ac2mqrEBoguwiqxeccIU1DKGbcPZuyV6ng+0xBv+mvw7AXS/tUAnbFUUZFeM6AKf0DEEA5ExJonJfMzGLFmHPzgbg/J1WOovHE+P5su1V4ungUGMXD61RN+QURYm8cR2Ak7JyaK2rwQgGrRtx+1oQmkb8FZ8EYFKlQXajZLPLRYkryDdsfwfg4XcPcqCuY7iiFUVRTtq4DsA2hwNPcgottdWk5njobvfT2eoj4core4+5fJPVC34sMZ4v6m8wWVQQMCTf+lsxAcOMVtMVRTkNjOsADD1PxFVZ48BZRYlU7W/BWVhI7LKlACzfLontlrwf42aHy8aPbH8CJMXlLTzw1v7oNl5RlHFt3Afg5Jw8miqOjAMnUrm32dp/g7Wiks1vcMk264/h18lJnKPvZIW2EYDfrTnAhtKmKLRaUZTTwfgPwNm5NPXciJswK4Wy7dZqrrHnnI1j0iQAPlXsQDMlxS4nb8S4+aH9L8TSjSnhG89upbU7EM1LUBRlnBr3ATglN5+GisMAJGXG4nDp1B9ut3IG33ADAO7GTpaXxADwv8mJZIgGvmd7CoDKlm6+/swWDFMtX6QoSniN+wCcVlBIU2UFAb+VcrJwbiolW62HLRKuvAI9MRGA67dZCz5X2O08Ex/HF2xvcba2A4A1e+v53zf2jX7jFUUZ18Z9ALY7nKTk5lFXWgJA4dw0SosbANBcLhI/9zkAPHsrWN6QDsDDSQk06Bq/tD+Khy4AHlxzgFXbq6NwBYqijFfjPgADZBZNpXq/lYgtoyAeb0eAllorsCZ/4XqEywXALetiQEraNY17kpPIFQ38wPbX3nK+9Vwx2ypaRr39iqKMT6dFAM4qmkL1AWsIQWiCwrmpvb1gW1oaSddfB4BjxwFu7JoPwL88sbzrdvF52ztcqX0AQHfA4KY/fKyypimKEhanRwCePI2aA0cznRXOS6O0+GjSnZQvfQktxroJd8XbXcTZrIxpP09LpVMI/tv+OJNEJQCNnX5ueGID9e1qGSNFUU7OaRGAk7Ky8Xd10dlizQHOnZJEY1UnXW1+AGxJSSTd8EUAgjt28UMuB6BG17g/KZFY4eMh+/24sILu4aYubvzDBlq71PQ0RVFO3GkRgIUQZBZNoXq/1QvW7Rr5M5P794JvugktLg6Aqc9vYmGqNRTxdEIcH7hdTNUquNf+fxxZ2HlnVRtffGK9miOsKMoJOy0CMEDW5KlU9xmGmLwwg30bans/6wkJJN90IwC+3Xu4q/FcYmzWsMQPMjJp1DSu1D/kq/oLvedsq2jlhsdVEFYU5cScPgG4aGq/ceAJs1Noqu6ktf7oYpwpN92ELSsLgOAjf+IHM78BQKMw+WF2LhL4lv15Ltc+6j2nuKKVLzy2Xi1tryjKiJ02ATizaAo1B/djmlaydd2mMWVxBns+Ojq3V3O7yfjudwAwmppYvKqUywouA+B9O/w1OQ2A/3U8wnxxNFHP9spWPvv7dVS2qJWVFUUJ3WkTgN1x8cQmJvUm5gGYdnYWe9fVIPs8Zhx36aXELF4MQPOTT3Jn+nVkx1oJ3H+TEMtGlxsHfv7i+hWTRUXveSX1nVz90Ifsq20fpStSFGWsO20CMPQ8kHHg6CPFaXlxOGJsVO5r7t0nhCDjB/8FmgaGQcfd/8uvl/4Su2bHwORbuROo0XU8soNnY+4lVxy9kVfT5uXqhz7k/f1qXTlFUY4vYgFYCJEnhFgjhNgthNgphLgjUnWFKmvyVKr27em3b/qSLHav6/+IsWvqVJKuvRaArg0byF2zmx+e9UMAmowuvjl5Lj4ByUYjL3juJZ2jAbzdF+TGP3zMXz46FOGrURRlrItkDzgIfEtKOR04C7hdCDEjgvUdV/7MORzesRUpjw45TDkjg7Jtjfi7g/2OTfvG13tvyNX98ld8IuYMPjfVyhux3dfAD6edjQmkBapYlXA3GRzNG2yYkh++uIMfvLgdf1CtqqEoyuAiFoCllNVSys0979uB3UBOpOoLRXJOHqZp0lxd2bvPHecgd1oS+zbU9DtW93jI+ulPATC7uqj+4V18d9F3WZC+AIDXvBU8MH0ZAKm+Cl5Pupcs0divjL9+dJjPPbqOmlZvJC9LUZQxalTGgIUQBcB8YP0g390qhNgohNhYXx/ZsVMhBAVzFlBWvLnf/jnn51L8dkW/m3EAnqXnknD1pwHo/PBDOl94ifuX38+E+AkAPO4t47lZKwBI6C7nreRfMtne0K+MLYdb+MRv31fjwoqiDBDxACyE8AB/B74upWw79nsp5aNSykVSykVpaWmRbg4FcwcG4OwpidgcGod2Ng44PuPOO7GlW2kqa//7btxVTTx84cMku5IB+EXXXt6c+ykAYjrLec3zM5Yn9u9NN3T4+eLjG7jntT1qoU9FUXpFNAALIexYwfdJKeU/IllXqCbMnkflnp0EA0efXhNCMO/CPIrfKh9wvB4fT/Y9d4MQyO5uKr/xTXKc6Tx4wYO4dBemNPlOxzbeW/R5AGzd9Txh/ojbJlQNKOuRdw/ymUfWUVKvlrxXFCWysyAE8DiwW0r5P5GqZ6RcHg8puflU7tnZb3/RogyaqztpqBgYHGPPPpuUW28FwLd3L3X3/pLZabO5f/n92DU7QTPIN5o3sn7p7QAIfzt3Nnyf38/ehyb6l1Vc3sLKB97nj2tLMdUyR4pyWotkD/gc4IvABUKIrT3bygjWF7LBhiF0m8as83MpfuvwoOek/cftuOdbCXqan3qKttWrOTvnbH5z3m+wCRt+089Xa95k/cXfB82GMPys2P9j3lu0lqw4R7+yvAGTH/9zF9c/tl7lFlaU01gkZ0F8IKUUUso5Usp5PduqSNU3EgVzF3DomAAMMGtpDqXFDXS2DszrIOx2cn79K7T4eACqvvd9vHv3sjx/OXcvuxtNaHQHu7m99G+8f/kvwJ0EQO723/FuwR/45LS4AWWuK2lkxX3v8ft3DxJUY8OKcto5rZ6EOyJz0hTaGxvoaG7qt9/lsTP1rEw2/2vwhyjsOTnk/PpXvePBFf/vdoLNzVxacCn3Lr0Xm7DhM3x8bdfveevyX0BKEQCO/a/yQMe3+P2lccQ5bf3K9AVN7n5tD1c8uJbNh5sHq1ZRlHHqtAzAmq6TP2suh7ZtGfDdghUT2Lu+ho7mwefuepYtI+2bVpa0QGUllV//BjIQ4NLCS/nN+b/pHRP+5uZf8rflX4OiiwAQDftY8eF1vPOJVs4tSh1Q7q7qNj790Ifc+fw2mjr9YbxaRVFOVadlAAaYuPAM9m9YN2B/bIKTGWdns/G1oR8lTvnSl4hfaQ1nd61fT/WPfoyUkgvyL+C3F/y2d3bETzf9mt/NPB+51Mqwhr+dlFe/zF8ynuF/Pj2VBLd9QNnPbizn/F+t4fEPStVTdIoyzp22Abho8VmU79yGt3PgrIf5K/I5sKmWtobB00sKIcj6xc9xzZwJQOs//kHDbx8E4Jycc3h8xeMkOa0x4Ee2PcpdLh+Bzz0JrkTr/E1P8OmNX2TNDRlcMTd7QPlt3iA/e2UXl973Hq/vrOn36LSiKOPHaRuAnTGx5M+ay4FBesFuj4PZ5+Xy8aqyIc/X3G7yfv8I9txcABoeeojmZ58DYE7aHP582Z/J8VhPXr944EW+VPY3mm56FfLOsgqo20XyXy/mgQlreeqWxUxKix1QR0lDJ7f+ZROffWQdmw6p8WFFGW9O2wAMMO2c89jz4XuDfjfvojzKtjXQWDX0QxO21FTy/u9R9MREAGp+8hPaXnsNgIKEAv668q/MSZ0DwOa6zVz7/rfYe+X/wrLvgNDA8MHr/8XZa2/mtRsm8P2V0wbcpAPYeKiZqx/+kC/96WN2VQ14mFBRlDHqtA7AExcsoubAPrpaWwZ854yxs/jyAj54bv+wQwDOwkLyfv8Iwu0G06TyO9+l/a23AEh1p/LEpU9w+URrleWqziq+sPrf+OeEuXDTakgqsAopex/Ho+dwq+MN1nxrKdeekT/gAQ6AN3fXsfKB97n9yc3srVGJ3xVlrDutA7Dd6aJw/iL2rf9w0O9nLcuhq81PydbhE+m4584l76HfIRwOCAap+Po36HjP6lk7dSd3n3s3dyy4A4HAa3j5/gff5+dVb+D/8hpYeJNVSKATVt9J6t8+xd3n6Lx2xzIumJY+aH2vbq9mxX3v8ZW/bmJnVeuJ/wEoihJVp3UABph2zjL2rH130O80XWPpNZNZ+7cDBP3GsOXELllC7oO/BbsdAgEq/uOrtK9ZA1g37b40+0s8cvEjJDoTAXh277N84a3bKFv2dbjhZUi0MqxRvh5+v5Sp23/NE9fN4Okvn8X8/MRB63xtRw2XP/ABN/1hAx+XNQ16jKIop67TPgBPmLOAxorDtDc2DPp97rRk0gvi2PLG4I8o9+VZtozc+/4XbDak30/FV7/WOyYMcHb22Tz3ieeYnTobgN1Nu7nmlWt4yWxBfuVDOOt2a2zYDMLa++B3Z7LE+z7/uG0Jf7hxMTOz4wetd83eej77yDqufvhDVu+owVA5JhRlTBCn0hSnRYsWyY0bN456va8/+lsS0jI486prBv2+vcnLc7/4mE9/ZwFJmQNnKww4fs0aKu/4OtLvB00j62c/JfHqq3u/DxgBHtjyAH/c+cfefZdMuIQfnPUDkpoPwyvfgMpNRwssWAqX3YtMn8Ebu2q5/6397BzmZlxBSgw3nVPIZxbmEjvITT1FUUaXEGKTlHLRgP0qAENt6UFe+tXP+dJvH0PT9UGP2bamgv0f13LVtxegDXaH7Bid69ZR/v9uR3Zbc4nT7vgaKbfdhpUkzrK2ci3f/+D7NHmt4YMUVwo/PvvHnJ+zFDb/Gd76KXT3DC0IDeZ/EZZ/H+nJ4K3ddfzunQNsOdwyZBvinDauWZzHDUsmMCHl+L84FEWJDBWAj+OpH36bxVdczeTFSwb9XpqSF/93C4VzU5l3UX5IZXZt3kL5V76C2WrdKEu85hoy7/ohwna0V9rY3chP1/2Ut8vf7t13+cTLuXPxnSRJYM3d8PFjIHvGoO2xcPZX4ez/QDo8rC9t4uF3DvLuvqFvFAoB501J44tnTeD8qenoIfwCURQlfFQAPo7dH7zDjjVv8Nkf/mLIY1rru3j+nk1c/d2FJGbEhFSu7+BByr98K4EqK0F77NKl5Pzm1+jxR8dzpZS8UvIKd6+/m/aANb0syZnEnWfcycrClYj6vfDGXbD/X0cLjkmBpd+CRbeA3cWemjYee7+Ul7ZWEjCG/plmJ7j43OJ8rlmcS1aCO6RrUBTl5KgAfBzBQID/u/0mrrnrblJy84Y8rvjt8t6hCF0P7R5moK6O8ttuw7drNwCOwkJyH/odzsLCfsfVdtby8/U/553yd3r3nZV1Fv915n9RkFAAJe9agbh669GT4rJh6TdhwQ1gc1LX5uWvHx3iyfWHaRwmqY/W0yv+3OI8LpiWgcN22t+PVZSIUQE4BB888xd8XZ1cePNtQx4jTckrvysmNdfDkquKQi7b6Oik6j/vpONN6yENLS6OnF//Cs955/UvX0r+dehf3L3+7t6xYbtm56ZZN3HLrFuIsblh10vw9s+g8cDRE+Nz4dyvW+PEdhfegMEr26r5y0eHKC5vGbZtKbEOrpiXzWcW5jIzOyHka1IUJTQqAIegraGev3z3q3z5d0/gcA89xNDd7ue5//6Y86+fxoRZKSGXL02ThgcfpOGhh3v3pdz276R99auIY27+tfpaeWDzA/xt39+QWD+j9Jh0vrnwm9awhGnAtmfhvV9Cc9nREz0ZsOQ/YNFN4LSSwG+raOGvHx3i5eIqvIHhM6xNy4zjqvk5XDkvh8wEV8jXpijK0FQADtEr991LeuEkzrjyM8MeV7W/hdX/t4NrvrcIT9LIAlXb6tVUf/+/MLu6AIg56yxyfvVLbIOsCr29fjs/X/9zdjXu6t03J20O3170beanzwcjAMVPw/u/6R+IXQmw+Etw5m3gsZ6oa/MGeHFLJU+tP8ye4zzKLAScVZjCFfOyuWxWJokxjmGPVxRlaCoAh6ix4jDP/uR7fOmB/xu2FwywaXUZpcUNfOqb87HZB5++NhRfSQmVd9yBb781jKAnJ5P1i58Tt3z5gGNNafLSgZe4b/N9vcMSABfmX8jXFnyNiQkTwQjCjr/D+7+Ghn1HT9adMOez1kMeGTMAa5hjR2Ubz248zEtbq2j3Bodtq10XLJ2cxifmZHHRjAziXQPzGCuKMjQVgEfglft/SVp+wZAPZhwhTcnrj+9E0wUX3TSj3xzfUJhdXdT89Ge0vvhi776k664j/TvfRnMPnKHQ4e/gse2P8dfdf8VnWOvWaULjyklXctvc28j2ZINpwt5V1pN0FR/3L2Di+VaPePIloFm/MLwBgzd21fL8pgre31/P8R6ic+gaSyenctnsLC6enkFCjArGinI8KgCPQGNFOc/+5D+55f7/wxkzfC844Dd48TebKZyXxqLLCk6ovtZ/vkLNT36C2WGlvrRPyCf77ruJWbBg0ONrOmv47Zbf8s+D/+wdH7Zrdj49+dN8afaXyIzNBCnh8Eew7kHY8yrQ5+ecOMEanpj/BYhJ7t1d1+7ln8XVvLilku2Vx0/yY9MESyalcMnMTC6ZkUFGvBozVpTBqAA8Qq8+8CtScvM569OfO+6xnS0+nr93I+d8ZjJFCwfPYHY8/ooKqr57J92be1ZrFoLkG24g7Y6voQ3xS2B/834e3PJgv4c4jgTim2fdbPWIAZpKYP3vYcuT4O8z9qs7YeZV1g27vDOtgd8eJfUdvFxcxcvFVZTUd4Z0DXPzErlkRgYXTc9gSoZnxP8iUJTxatQDsBDiCeATQJ2UclYo55xKAbipqoJn7vouN933e9yegUvKH6u+vJ1/PrCVi2+aSd6M5OMePxhpGDT96c/U33eflUcCsGdnk/njH+FZtmzI87bVb+Oh4odYW7m2d59N2Fg5cSW3zLqFiYkTrZ2+dih+BjY82n+cGCB1qjWXeO7nIfbooqFSSnZXt/Pq9ipe2VbNocaukK4lN8nNhdPSuWB6BmcWJuMa4Ri5oown0QjAy4AO4M9jMQADvPnYQyAEF93ylZCOrzrQwurfb2flV+aQOfHE59P6Skqo/v5/0b11a+++uMsuJePOO7FnZg553rb6bTxc/DAfVH7Qb//5uedz46wbWZC+wOqVSgllH8DGx2H3P63sa0dodpiywhqeKLoI9KNjvEeC8Ws7qlm1vZqDIfaM3Xadc4pSOX9qGudPTSM3KbSnCBVlvIjKEIQQogB4ZawG4O6Odv74za/w6f/8MRkTQ3voomx7A2//eTdX3DGP1Nzj95yHIk2Tlmefpe43/9M7NixiYki97TaSb/w3NMfQ08J2Ne7ise2P8eahN3vHiAFmp87mC9O/wMUFF2PXegJrR501jW3Tn6DpYP+CYtNg1mesXnHW3H5DFAAH6jr4184aXt9ZQ3FF6Inhi9I9LJucxrIpqZxZmILboXrHyvh2ygZgIcStwK0A+fn5Cw8dGno5+GjYvuZ1tr/5L6792a8QWmiP6x7YVMd7z+7jE7fPIX3C4Dl8QxWoraP2nrtpf2117z57fj7p3/oWcZdcPOw4a0lrCX/e+Wf+efCf+M2jjyWnu9O5Zuo1XD3lalLdPcMNUsLhddY48c4XrBU6+kqdArOvgdlXQ/LEAXXVtnl5c3ctb+6qZe3BRvzB4R/4OMJh01g0IYlzJ6eytCiNGdnxKlmQMu6csgG4r1OtBwxWT/Tpu77D7AtXMHv5JSGfV1pcz5q/7jnp4YgjOj/6iNpf/KJ33jCAe8EC0r/zbWLmzx/23IbuBp7Z8wx/2/e3fvOIbZqNiydczOemfu7o8ASArwN2v2yNF5e+R78ZFADZ82Hmp60beIkD82Z0+YN8sL+BNXvrWLOnnpo2b8jXmeC2c/akFM4uSuXsSSlMTI1VN/OUMU8F4JNQW3KAf9zzY7547wN4kkK/wXZoZyNv/XEXF900g/wZoT+yPBQZCND87HM0PPggRktL737PBReQdscduKZOGfZ8n+FjVckqntz9JHub9/b7rjChkM9M/gyfnPRJklxJR79orYDtz1tb7faBheYsghlXwowrji4y2rfNPePG7+yr45299Ww61DyiFTvS45wsmZTCWROtrSAlRgVkZcxRAfgkrX3uSar37+Hq7/0k5KEI6Hlk+dHtLLlqEtPPzg5LW4y2Nhoe+T3Nf/kLMhCwdgpB/GWXkvqVr+CcPHnY86WUFNcX8/Sep3n90OsE+9yEs2k2luct56qiq1iSvQSb1mdFjbrdsOMf1hN3x44XA2TOhulXwLTLIX3GgDFjsB6H/vBAI+/tr+e9ffVUNHeP6Noz4p2cUZjCGYXJnFmYTFGaJ6QE+YoSTdGYBfE0cD6QCtQCP5JSPj7cOadyADYNg2d+9F2mnb2MBSuvHNG5zTWdvPJgMVPOyOSMTxaGrQcXqKqi/qGHaP3HC9YTcABCEHfpClL//d9xTZt23DKavE28fOBlnt//PIfa+o+/p7pTubzwcj456ZNMTZ569AspoWYb7HwRdr1ozTM+VuIEKxBPvQzyl/SbTXG0GMmhxi4+ONDAB/sbWFfSSGt3YAR/ApAYY2fRhGTOKExiUUEys7ITVGpN5ZSjHsQIg5aaap76wbf47F3/TVp+wYjO7Wrzs+rhbcQmOLnwxuk4XOFbq81XUkrDww/T9uqrRwMxELtsKalf/jLuRYuOG/SllGys3ciLB17kjUNv0B3s3zMtSizi8omXc1nhZeR4cvqeCLU7YPcrsOcV6/2xnPFQdCFMXmFNbfMMTDoEYJiSnVWtrD3QyLqSRj4ubaI7MPxq1AOqsmnMzU1kYUESC/OTWDAhieRYlUhIiS4VgMNk57tvseGl57nu57/GGTOyddaMgMl7z+2jen8Ll902O6QFPkfCV1JKwyMP0/bqKjCOBi7XrFkk33gj8SsuQdiPn7uhw9/BG4fe4OWDL7OxduDPY07aHC4tuJRLJlxCRmxG/y+bSmDva7BnFRz+EOSxsyGEdROv6CJry1kI+uC/jPxBk20VLXxU0shHJU1sPNR03HSagylMjWV+fiIL8pOYn5/I1Iw4bCEm01eUcFABOIzeeuJhWmpruOq7dw25iOdwdn1QxboXD3LuZycz9cyhH6w4Uf6KCpqeeIKW5//e+0QdgC0jg6RrP0/iZz+LLSW0m4KVHZWsKlnFqyWvcrB14Ljv3LS5XDzhYi7Mv5DcuNz+X3Y1wYE3Yd9q69U7yFxhVwIUngeTLoBJywe9kdd7XUGT7ZWtrC9tZENpE5vKmmn3DZ/JbTBuu87snATm5ScyLy+RuXmJZCe41M09JWJUAA4j0zD4+90/IjVvAsv/7csnVEZDRTuvP7aTtPw4zrt2Kg53+JePDzY00Pz0MzQ/9RRGc3PvfmG3E7diBUmfuyak4Qmwhij2Ne9jddlqVpeupqKjYsAxU5OmckH+BSzPW8605Gn9yzWCULEB9v0LDrw1+IwKsAJw4Xkw8TwoWDbkcAVYQxZ7atrYWNbMx2VNbCxrHtGUt75SPQ7m5CYyJzeBubmJzM5NINXjPKGyFOVYKgCHmbejg6d++G0WXHYF8y5ZeUJlBPwGa58/wOGdjSy/ftoJ55A4HtPrpfXll2l+8il8e/tPP3NMnEji1VeTcOUV2FJThyihPykluxp38fqh13nj0BuUt5cPOCYjJoPzcs/jvLzzWJy5GLftmPSabdVw8G1rK1kDXY2DV5Y2HQqXQsG5MOGcfnkqBmtXVauXjWVNbDrUzObDzeyubh/RtLe+chLdzMqJZ3ZOArNyEpidk0CKCsrKCVABOAKaa6p47iff49zP38DM8y484XIO72xkzZN7yJuWzDmfKcIZoRy7Ukq6N2+m+cmnaHvjDQj0mXGg63iWLSPhU5/Cc/55aM7QAs2RnvHbh9/m7fK32dO0Z8AxTt3J4szFnJtzLudkn8OE+An9e8emac2qKH0XSt6BQx9CcIiebNo0mHC2FYzzl0BCzuDH9ejyBykub2VLeTNbDrew5XALDR2+kK5tMFkJLmZmJzAzO97achLU8IVyXCoAR0hjZTnP/+y/WHr9TcxYOnA1i1D5vUHWvXCQki31LLlqElPPzEREcH5rsKmJ1hdepOVvf8NfVtbvOy0ujvhLVxB/+eXELF48YL264VR2VPJu+bu8W/EuG2o29JtjfESOJ4cl2Us4O/tszsg8gwTnMU8KBn1WMvnS960n8So3gjHECs8J+ZB/FuSfaaXUTJ/Rm2x+MFJKKlu62VrewtbDLWyraGV7ZeuIZ1v0lRRjZ0Z2PDOy4pmRHc/0rHgmpXmwqxt9Sg8VgCOoofwQz//8Byy7/iZmLLvgpMqqLW3jvWf2oumCpZ+bctK5JI5HSkn3lq20vvAP2la9htnZPweEnppK/CWXELdiBTGLFo4oGHcGOvmo6iPer3yf9yvep667bsAxmtCYnjydM7LO4MzMM5mfPp8Y+zHZ0gLdVkAuWwuH1lrvh+ohO+IgdyHkLobcM6xZFrHD33AMGib76zrYVtFCcUUr2yta2VPTRsA48b8bDl2jKN3D9Kx4pmfFMS0znmlZcWpc+TSlAnCENZQf4h/3/Jg5F17KmVddc1L/JJWmZPeH1az/ZwnZkxM568qJJKRFPoWj6fXSsWYNrS//k44PPug/RIG1bl3chRfiufACYpcsCXmYAqxAf6DlAGsr1/Jh1Ydsrtvcu6xSXzZhY2bqTBZnLmZB+gLmp8/H4/D0Pyjoh6otUP4RHFoH5euhu2lAWb2SCiF3kRWMcxZaT+zZBy751JcvaLC3pp3tla3sqLR6yXtr2k8qKIN1s29qZhxTM+KZlhnH1Mw4Jmd4iHGE/yascupQAXgUdDQ18sK9PyVj4iQuvOX/odtO7i9VwGdQ/NZhit+qYNLCdBasyCc+ZfjAES5GSwvtb75J26pVdK7f0G9eMVipMT3nnI3nvPOIXbYMe/rIVgLxBr1srtvM+ur1rK9ez67GXf1SZx6hCY2pSVOZnz6f+enzmZc+z1pyqS8poWG/FZDLN1g95PqBY9FHG69bQxU58yFrHmTPg/SZYB9+SSV/0GRfbTs7q1rZWdXGzqo2dle30eU/8eELsJ7Yzk+OYXJ6HFMzPUzJiKMo3cOkNI9KZD9OqAA8Svzebl69/5d4Ozu5/GvfIT516GlUoeru8LP1zXJ2vl/JpHlpzL9kAokZo5fUPNjcTMfbb9P2+ut0fbjuaP6JPpzTp+M59xxiz11KzPx5iGHyFQ+m1dfKlrotbKjZwMaajexp2jNoQAZrhsW89HnMSZ3DnLQ5TE+ZjlM/pjfe3QJVm6FikxWQKzdBV8PQDdBs1oyLrLmQNcfqJWfMAtfwQ0CmKSlr7GRXdRu7egLyruo2attO/EZfb5METEiJpSjdw5QMD5PTjwZmlUN5bFEBeBRJ02TDy39n86qXWHHbHUxcsDgs5Xo7AhS/Xc6O9yrJmpTAvIvzyZqUMKp34I2OTjo/+ID2t9+i8733+2VlO0K43cQsWkTsWWcRc9aZuKZNG9HYMUCbv42tdVvZVLuJLXVb2NGwg4A5eJ4Im2ZjStIUZqfOZmbKTGamzmRiwsT+iYSkhJbDViCu3moNYVRtBV/b8A1JKrAC8ZGAnDHTynNxnIRMjR0+9ta0s6u6jT017eypaWNfbUfIeZKHI4Q1RW5SmoeidE9vUC5K96jHrk9RKgBHQcWenaz67a8pnLuQpdffiCvWc/yTQhDwGexZV03xW+XYnDqzz8th8uKMsOaXCIU0DLq3baPj3Xfp/GAt3p07rUB3DC0+npiFC4lZtIiYRQtxzZgR0iPRffkMH7sad1FcV8zW+q0U1xfT0D10j9alu5iaPJXpydOZkTKD6SnTmZQwCXvfpECmCc2lVjCu3grV26zpcN3NQ5YLgMMD6dOtYYz0GT3vp1sriAzzyzBomJQ1drG3pp29NW3srW1nb007h5q6BvtjOyFJMXYmpnmYmBrLpHTrdWKah/zkGJWkKIpUAI4SX1cn7z/1Rw5u2sAFN/47RWcsCVuPVZqS8j1N7Hi3kqr9LUycn8b0JVlkjnKv+IhgczOdaz+k86N1dH24jkBV1aDHCbcb9+zZuOfPxz1vLu65c7Elj+whFCklNZ01FNcXs71hOzsadrC7afeAJEJ92TQbkxImMTV5KlOSpvRuKe6UvgVbOZBrtlkBuXYH1GyHlhBWanEnW4E4bao1Xzl1ivUalzlsYO72Gxyo62BfbXufrYPKlpGl6hyOrgnyktxMTPNQmBpLYWosE1NjKUiNJTPepVJ6RpgKwFFWsXsHb/zf7/AkJbHs+ptDXmMuVJ0tPvaur2HPumpMQzJ5cQZFi9JJyQ5Pr3ukpJQEKiro2rCBzvXr6fp4I8Hq6iGPt+fl4Z49G9fs2bhnzcQ5fQa6Z2TJioJmkJLWEnY17mJHww72NO1hb9NevMbwjycnu5KZnDiZoqQiJiVOoiixiIkJE/vPT/a2WfmQa3dA7U6o22Vtg+W3OJYzHlInWwE5pch6nzLZWtppmBt/nb4gB+s72F/bwb66dg7WdbC/roPypi5O8OG+QbnsGhOSraBckBpLQUpMz2ss6XFOFZzDQAXgU4BpGGx/+3XW/f1p8mbM5qxPf46U3Pyw1iGlpO5QOwc21nJgUx12p87EeWkUzksjfUJcVJ/YClRW0rVpE12bN9O9ZSu+ffsGHbIAQAgcBQW4pk/HOX0armnTcE6dii0tbUTXEDSDlLWWsafZCsZ7mvawr3lfv6WZhpLqTmViwkQKEwqtLb6QgoQCMmMz0YRmtb2tCup3W8G5bjfU77U2f3sIrRPWkk4pRZA8yQrIKZOs94n5YBt8PNcbMCht6GR/XQcH6zo4WN/BgboOShs68YVhjLkvl12jICWW/GQrKE9IiWFCsvWaleBSWeVCpALwKcTf3cWW1a+w+bWXyZ4yjcVXXE3W5GlhD47SlNQeaqN0az0lWxvwe4Pkz0xhwswUcqcl4YqNzCPPoTLa2/Fu3073tm10F2+je8d2jPphZioAelISzilTrK2oCOfkIpyTJqEnhL7unpSShu4G9jXvY3/zfva37Gd/835KW0uP21sGa3w5Pz6fCfETmBA/gfy4fPLj88mPyyfVnYoAKzA37IX6fdZrw35o2AcdtaE1UmiQkAfJhdY85qSCnvcF1k1Ad+KAU0zTesrvYH0HJfWdlDT0vNZ3nnCSouHYdUFOopv8lFjyk91MSI4lLzmGCSkx5CXH4HGquc1HqAB8Cgr4vOxY8wabV72M3e1m7kWXMf3c83C4IzPFrKWui8M7Gzm0o5Hqg60kZcSQOy2J7MlJZE5KwBmBjGwjIaUkWFtL9/bt+HbvxrtzF95duwjW1x/3XD01FefEiTgmFuIoKMBZaL3as7NDvuFnmAZVHVUcaDlASWsJB1sOUtJaQmlrKV3BrpDKcOkucuNyrc1jveZ4cnq3GCMAjQeg4YD1emRrKgF/R0h1WBUlWIE4aYL1mjjB6k0n5Fm952Omz3X6gpQ2dPZuJfUdlDZ2UVrfQZt35Ck9Q5ES6yA3OYb85Bjyktzk9b6PISvRdVo9qq0C8ClMmiaHdhRT/PoqDu8opmDeQqafcx4F8xZiG+FsgVAZQZPa0lYq9jRTdaCF2rJ2EtPdZBYmkDExnszCBBLS3BHNRxGqYFMTvj178O7Zi2/fPms7eBDpC2Gurc2GPScbR14+jvw87Ll52HNzcOTmYs/NRY8//qPeUkpqu2opayujrLWMsrYyDrUd4nDbYSo7KjFk6A9iJDoTyYrNItuTTVZslrV5ssiMySBD6qR2NaM1l1kBuakEmsugqRS8LSHXAVgBOiEPEnIhPsd67X2fA3HZYHMgpaS5K0BpQydlDZ0cauyktLGLsoZOyho7aY9QcNYEZCW4ye0JzLlJbnKTjry6yYwfX8MbKgCPEd3tbez7aC17PnyXutIS8mfNZdLCMyiYuwBP8smvrDwUI2hSf7idmpJWasvaqC1pw9cVIDUvjrT8OFJzPaTkekjOjEW3R/8vhjQMAhUV+A4cwHfgIP6SEnwlJfhLSzE7Qu9Jah4P9uzsni0LW1YW9sws7JkZ2DIzsaWnD/vIdcAMUN1RzeH2wxxqO0RFewXl7eVUtFdQ2VEZ0pBGXzZhIy0mjfSYdNJj0smIybDe22JJCwZJ9XaQ1tmMp7UK0XrYmtvcUg6DPNZ9XLHpEJ9lBeMjr3GZPe+zkJ5MmqWHsqYuDjd2UdbYyaHGLg71vDZ2DpEgKQx0TZAZ7yI3yU1OkpvcRCtA5yS5yUl0k5XowmkbOw+jqAA8BnW1tVK6ZSMlmz/m8I5iYuITyJs5h5xpM8iaPI2E9IyI3lTr7vBTf7idhvIOGiqsra2+m7gUF8lZsSRlxpCYGUNiurU5Y21RT8sopcRobMRfWor/0CH8hw5br+XlBA4fHpBsKBR6QgK2jAxsaWk9Wyq21FT01FRsKanYUlPQU1LQExL6rZgtpaTR20hFewVVHVVUdlRS1VlFdUc1VZ1V1HTWDDttbjgu3UWKO4U0dxoprmRSbTGkoJEcDJLs95LkbSO5o4nktjri2yrRQpmtMejFO8CT0WdL733tdqZSY8Rz2BfDgc4YStrgcFMX5U1dVDR3EwznVI1BpMU5yU60gnN2oovsRDfZiVaAzk50kxRjj/r/j0eoADzGSdOk7lAp5TuKqdq/h+p9ezBNk/TCSaQXTCS9YCIpufkkZWWj2yJ3c80ImrTUddFU1UlLbRfNNV201HbRWm8FkoQ0N/EpLuJSXMSluIlLduJJcuFJduKKje5fCCklRksLgYoKa6usJFBVRaCyikB1NYGaGsy24zwZNxxdR09MxJachJ6UjJ6YaG1JSegJCT2fE9Dj49F6tk4nVBuN1HbVUdNVQ01nDXVdddR21VLbWUt9d/0JB+neZgmdBEc8SfZYEjUniWgkGiYJQT+J/m4SvR3EdzUT720j3jBJME3iTRO3lIzop2WPsRLmx6QiY1LpsifSKhKoN+OoDsRQ7nVT0uVkf7uDgx0OWvFgErl/TbnsGtkJVjDOSjgSoF1kJVivmQnuUbtRGJUALIS4FLgf0IHHpJT3DHe8CsChk1LS3thAXVkJdaUHqT9UQmNlBW31tcSnppOYmWVt6ZnEpaUTn5pOXEoq7rj4iARBKSW+ziCt9d20NXbT3uilrdFLR7OXjiYfHc1egn6TmAQHsQkOYhKcxMQ7iIl34I5z4I6z4/bYccU6cHnsOGNt6FEYAzQ6OghWVxOorSNYW0Owro5AbS3B2jqC9fXW1tgIwfCNjQq7HS0uDi3Og+6Js957YtFjPQhPLIbLQZfDpMNu0KYFaNV9NIsuGkUnDbKDetlOrdlMvWzHZ4egzrAPfoTKhkacZiNOanikSVwwiCfowxPw4TElHtMkVprEmpJYs+dVmsSYkpgjrz2BfLAwJxEEHfF02xLoEB6apYcGw02t30VtwE2rjKWVWNpkDO3E9L52SDftxODn5DsacS4b2QluMhNcVlCOt4J1ZoKr9zXOdfL1jHoAFkLowD7gYqAC+Bi4Vkq5a6hzVAA+ecFAgJaaKlpqa2ipqaK1roa2hnra6+tob2ok4O0mNimZmIREa4tPxB0fj9sThzsuHqfHgysmFmesB4c7BofbjcMdg247+eGFoN+gs9VPZ4uPrjZ/z+ajuyNAd5sfb2cAb0eA7vYAvu4gNoeGK8aOI8aG023DGWPD4e7ZXDoOlw27U8fh0rE7rfc2p47dqWFz6NgdOjaHhs2uo9lEGJ9ANDFaWzEaGgg2NBBsbMJoaux5bSLY3ITR1IzR0tK7YYZ3fu6wNA3psGM6bAQdOkG7wG8T+G0Sr27i1Uy6tQDdIohflwRsENCtwH10EwS1o58Nzdp63+tgiJ5XDQxNYAowNTCFte/Ie1MDDYkDEwfglBIHPZswcUqJHdn72vudBAc9+3q+s/ecaweQOgHpICgdBKQDH078phO/dOKTLnzSiQ8nXtOJHyfdR/ZJJz4ceHvf2/FhleHHjh+dADaC2AiiEet0kJngIjPeCsjnTUnjk3OzR/QjGSoAR7L/fQZwQEpZ0tOAZ4ArgSEDsHLybHY7qXkTSM2bMOj3Qb+fjuYmulqb6WxtoaulBW9HO50tzTRWHMbb2YmvswNfZwd+bze+7m78Xda4qd3lwu50YXc6sTmc2BwObHYHNocD3e5At9ux2e1oNht6z6bp1ma91xGahqbrve/dMRoxHg0t2/osNA0hBELoGEFJwOfDCHgJBgwCfpOgXxLoMuluMQkETIyASdBvYgSl9T5w9L0RlBhBEyMgkVKi6xqaXVivNq33VdOtfUITaLpA0wTagM8CoVmbpvV/j3CjaXmI1HxEOiD6HHMk5gf8SJ8P6fdZszf8XqTPj/R5rc3rO/q9/8ixPcf4fVYAlxJ6MsT1+1Uirf+Iftnj+rw3ejYfaEBMz9ZPn46YGCIL3fGN3nDmkXknGuDACmT9E7VKwIsUg98EPZGWCinp9lSw9fbnRhyAhxLJAJwD9F2tsQI489iDhBC3ArcC5OeH96kwZSCbw0FiRiaJGZnHP7gPIxjA7/US8HoJ+v0EfNarEfAT9PsJBvwYgYC1BYOYRtB6bxiYRhAzGCQY8GMaBqZhIE3z6KtpIs2+703rL0jPZ7ACqOzpRfZ+L639YP3lsOnWZsUpa79EgrSGSKS0Hk6R8kh5EmmAGZQY8kh5R489Uq91vvXnYH2WR2IePcX3vundT59jjhhqP33inwa4AJfACinuo9ci+1RwpF19Kz32tbdc2ae+Pu+P/devHCoBqAIgEBS0lJMZP3ze6JGIZAAe7N97A36+UspHgUfBGoKIYHuUk6DbrDFatycu2k1RIkxKaeV8DgSQR7Zg0NoCQWQwAIbRu6/3vWGAaVr7TBMZNMA0kIZpvZomGNYvW0wJ0rTOkfT08E3rl0zPd6ZpYhgBDCNI0Oz5pW4GMUwDwwxiShOz33sTUxpIKa3P0sCUVn2mEbB+ycsg0ggie/YHg0GCRpCAEcQwTKtsw8AwZW+Z1r8uLLvycpiZMDYCcAWQ1+dzLjB4eixFUU4ZQggrof4Ik+qPR0ceVKlu7aam1Yuzzcu8vMSwlR/JAPwxMFkIUQhUAp8HrotgfYqiKGElhCA51kFyrIOZ2aHnGwlVxAKwlDIohPgP4F9Y09CekFLujFR9iqIoY01EZyFLKVcBqyJZh6IoylgV/Yf6FUVRTlMqACuKokSJCsCKoihRogKwoihKlKgArCiKEiWnVDpKIUQ9EML63/2kAsMvJDb2qWsc+8b79cH4v8aTub4JUsq0Y3eeUgH4RAghNg6WZWg8Udc49o3364Pxf42RuD41BKEoihIlKgAriqJEyXgIwI9GuwGjQF3j2Dferw/G/zWG/frG/BiwoijKWDUeesCKoihjkgrAiqIoUTJmArAQ4lIhxF4hxAEhxH8O8r0QQjzQ8/02IcSCaLTzZIRwjdf3XNs2IcSHQoi50WjniTre9fU5brEQwhBCfGY02xcOoVyjEOJ8IcRWIcROIcS7o93GkxHC/6MJQoh/CiGKe67vpmi082QIIZ4QQtQJIXYM8X34Yk3v+lin8IaVT/ggMBFrDb5iYMYxx6wEXsNaCuksYH202x2BazwbSOp5f9lYusZQrq/PcW9jpTH9TLTbHYGfYSLWwrT5PZ/To93uMF/f94F7e96nAU2AI9ptH+F1LgMWADuG+D5ssWas9IB7V1iWUvqBIyss93Ul8Gdp+QhIFEJkjXZDT8Jxr1FK+aGUsrnn40dYyzyNFaH8DAG+CvwdqBvNxoVJKNd4HfAPKeVhACnlWLrOUK5PAnFCCAF4sAJwcHSbeXKklO9htXsoYYs1YyUAD7bCcs4JHHMqG2n7b8H6LTxWHPf6hBA5wFXAI6PYrnAK5Wc4BUgSQrwjhNgkhLhh1Fp38kK5vgeB6VjrP24H7pBSmqPTvFETtlgT0RUxwiiUFZZDWoX5FBZy+4UQy7EC8LkRbVF4hXJ99wF3SikNqwM15oRyjTZgIXAh1rrz64QQH0kp90W6cWEQyvWtALYCFwCTgDeEEO9LKdsi3LbRFLZYM1YCcCgrLI/1VZhDar8QYg7wGHCZlLJxlNoWDqFc3yLgmZ7gmwqsFEIEpZQvjkoLT16o/582SCk7gU4hxHvAXGAsBOBQru8m4B5pDZYeEEKUAtOADaPTxFERvlgT7QHvEAfFbUAJUMjRwf+ZxxxzOf0HxjdEu90RuMZ84ABwdrTbG4nrO+b4PzL2bsKF8jOcDrzVc2wMsAOYFe22h/H6HgZ+3PM+A2tF9NRot/0ErrWAoW/ChS3WjIkesBxihWUhxG093z+Cddd8JVaA6sL6TTxmhHiNdwEpwEM9vcSgHCPZp0K8vjEtlGuUUu4WQqwGtgEm8JiUctDpTqeaEH+GPwP+KITYjhWg7pRSjqkUlUKIp4HzgVQhRAXwI8AO4Y816lFkRVGUKBkrsyAURVHGHRWAFUVRokQFYEVRlChRAVhRFCVKVABWFEWJEhWAFUVRokQFYEVRlChRAViJGiFEwVA5V0/kHCHEh33ed/TdJ4RIFEL8v5Np7zF1fU0IsVsI8WS4yhykjmHz0ipjnwrASkT1JK8elf/PpJRnD7MvEQhbAO4pa6WU8vowlnmsPwKXRrB8JcpUAFZOWE9vdI8Q4k89KwM8L4SI6dm/WwjxELAZyBNCfFMIsaNn+3qfYmzHnt9T9os96Rp3CiFuPabqoc7pGKSNR/bdA0zqWYniV0KInwkh7uhz3C+EEF8b5PwB7RZCPIKVlPxlIcQ3jjl+thBibZ/PC4QQb4f6Z9qXPH5eWmWsi3bSC7WN3Q0rYYkEzun5/ATw7Z79JnBWz/6FWLlhY7GSdO8E5g91fs/75J5XN1bCmpTh6ux539GnbR3HvBbQJ7lKz+fNPe81rJUeUo65vkHb3fNdGYMkmekpqwbQez6vARYcc8z7WCkbj90uGuLPeNCkMGob+5vqASsnq1xKeaTH91eO5ig+JK3VAujZ94KUslNK2QH8A1h6nPO/JoQoxlr5Iw+YHEKdIZNSlgGNQoj5wCXAFjkwvedw7R6qXBMrUM8UQlwNHJZSbj7mmKVSynmDbG+O9DqUsW1MZENTTmnHZnM68rmzz77hsqsPOF8IcT5wEbBEStklhHgHcIVQ50g9BtwIZGL1pI91olnhPwLOwRonHjCGK4R4H4gb5LxvqyB8elE9YOVk5QshlvS8vxb4YJBj3gM+1TM+HIu17ND7w5yfADT3BN9pWDlXR1rnsdoZGPRewAqQi7FSLI6k3cP5CPg5Vu+58tgvVQ9YOUIFYOVk7Qb+TQixDUjGSsjdT88/wf+ItSrCeqwcuFuGOX811o22bVj5ZT86psjj1jlIGxqBtT03037Vs8+PNUb7nJTSGGG7h7MH8AH3hnDskHry0q4DpgohKoQQt5xMecqpR+UDVk6YEKIAeEVKOSvabTkRPdPjNgOflVLuD2O5DwIfSyn/FK4ylfFJ9YCV05IQYgbWigZvhSv4CiEmCSH2AG4VfJVQqB6woihKlKgesKIoSpSoAKwoihIlKgAriqJEiQrAiqIoUaICsKIoSpSoAKwoihIlKgAriqJEyf8HgBFCZ7GUpUIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline \n",
    "\n",
    "p = np.linspace(0.001, 1.0, num=100)\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(5, 4))\n",
    "\n",
    "for g in [0, 0.5, 1, 2, 5, 10]:\n",
    "    f = -(1 - p)**g * np.log(p)\n",
    "    \n",
    "    if g == 0.0:\n",
    "        ax.plot(p, f, linewidth=4)\n",
    "    elif 0.5 <= g < 5:\n",
    "        ax.plot(p, f, linewidth=2.5)\n",
    "    else:\n",
    "        ax.plot(p, f, linewidth=1)\n",
    "        \n",
    "ax.set_xlabel('probability of $y = 1$')\n",
    "ax.set_ylabel('Loss')\n",
    "ax.legend(['$\\gamma = {0}$'.format(g) for g in [0, 0.5, 1, 2, 5, 10]])\n",
    "\n",
    "fig.tight_layout()\n",
    "pngFile = './figures/CH05_F21_Kunapuli.png'\n",
    "plt.savefig(pngFile, dpi=300, pad_inches=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When $\\gamma=0$, the original cross-entropy loss is recovered. As $\\gamma$ increases, the part of the curve corresponding to \"well-classified\" examples becomes longer, reflecting the loss function's focus on poor classification.\n",
    "\n",
    "In order to use this focal loss to train gradient boosted decision trees, we have to provide ``LightGBM`` with two functions:\n",
    "* the actual loss function itself, which can be used for evaluation\n",
    "* the first derivative (gradient) and second derivative (Hessian), which will be used for learning\n",
    "\n",
    "**Listing 5.7**: Defining Custom Loss Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.misc import derivative\n",
    "\n",
    "def focal_loss(ytrue, ypred, gamma=2.0):   \n",
    "    p = 1 / (1 + np.exp(-ypred))\n",
    "    loss = -(1 - ytrue) * p**gamma * np.log(1 - p) - ytrue * (1 - p)**gamma * np.log(p)\n",
    "    return loss\n",
    "\n",
    "\n",
    "def focal_loss_metric(ytrue, ypred):\n",
    "    return 'focal_loss_metric', np.mean(focal_loss(ytrue, ypred)), False\n",
    "\n",
    "\n",
    "def focal_loss_objective(ytrue, ypred):\n",
    "    func = lambda z: focal_loss(ytrue, z)\n",
    "    grad = derivative(func, ypred, n=1, dx=1e-6)\n",
    "    hess = derivative(func, ypred, n=2, dx=1e-6)\n",
    "    return grad, hess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's focal_loss_metric: 0.141808\n",
      "[2]\tvalid_0's focal_loss_metric: 0.11333\n",
      "[3]\tvalid_0's focal_loss_metric: 0.0952922\n",
      "[4]\tvalid_0's focal_loss_metric: 0.0790233\n",
      "[5]\tvalid_0's focal_loss_metric: 0.0677255\n",
      "[6]\tvalid_0's focal_loss_metric: 0.0606197\n",
      "[7]\tvalid_0's focal_loss_metric: 0.0537766\n",
      "[8]\tvalid_0's focal_loss_metric: 0.0482069\n",
      "[9]\tvalid_0's focal_loss_metric: 0.0459177\n",
      "[10]\tvalid_0's focal_loss_metric: 0.0438938\n",
      "[11]\tvalid_0's focal_loss_metric: 0.0396912\n",
      "[12]\tvalid_0's focal_loss_metric: 0.0373091\n",
      "[13]\tvalid_0's focal_loss_metric: 0.0354938\n",
      "[14]\tvalid_0's focal_loss_metric: 0.0325854\n",
      "[15]\tvalid_0's focal_loss_metric: 0.0325117\n",
      "[16]\tvalid_0's focal_loss_metric: 0.0321897\n",
      "[17]\tvalid_0's focal_loss_metric: 0.03054\n",
      "[18]\tvalid_0's focal_loss_metric: 0.0303324\n",
      "[19]\tvalid_0's focal_loss_metric: 0.0283734\n",
      "[20]\tvalid_0's focal_loss_metric: 0.0272042\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9736842105263158"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm_focal_loss = LGBMClassifier(objective=focal_loss_objective, learning_rate=0.25, n_estimators=20, max_depth=1)\n",
    "gbm_focal_loss.fit(Xtrn, ytrn, eval_set=[(Xval, yval)], eval_metric=focal_loss_metric)\n",
    "accuracy_score(yval, gbm_focal_loss.predict(Xval))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.956140350877193"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbm_standard = LGBMClassifier(boosting_type='gbdt', learning_rate=0.25, n_estimators=20, max_depth=1)\n",
    "gbm_standard.fit(Xtrn, ytrn)\n",
    "accuracy_score(yval, gbm_standard.predict(Xval))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}