11_training_deep_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 11 – Training Deep Neural Networks**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 11._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/ageron/handson-ml2/blob/master/11_training_deep_neural_networks.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "tf.config.experimental.set_memory_growth(tf.config.experimental.list_physical_devices('GPU')[0], True)\n",
    "\n",
    "\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "%load_ext tensorboard\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanishing/Exploding Gradients Problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def logit(z):\n",
    "    return 1 / (1 + np.exp(-z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sigmoid_saturation_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3wUxfvA8c+kkA4hVOkgNUiRJkUhgqGIiDQFqaIiYEEBAUEQEFHp+EP5ioIgEUGkSJcIho4QMBGiEKWXUAIEU0m5+f2xR0y5kAQuuZTn/XrtK7nduZ3nNpd7bnZnZ5TWGiGEECKvsbN1AEIIIYQlkqCEEELkSZKghBBC5EmSoIQQQuRJkqCEEELkSZKghBBC5EmSoMQDUUoFKKUW2DoOyFosSqnjSqnJuRRSynqXKqU25UI9PkoprZQqmQt1DVFKnVdKmWxxTNPEMkgpFWXLGIT1KbkPSmREKVUKmAI8DTwERADHgU+01v7mMl5AgtY60maBmmUlFqXUceBHrfXkHIrBB/gVKKW1Dk+xvhjG/1uEFes6CyzQWs9Ksa4I4AVc1Tn4z62UKg5cA0YCPwKRWutcSRBKKQ300lr/mGKdC+Chtb6WGzGI3OFg6wBEnrYGcAVeBv4BSgNtgBJ3C2itb9omtPTyUixpaa1v51I98cCVXKiqMsbnxyatdVgu1HdPWutYINbWcQgr01rLIku6BfAENPBUJuUCML7F331cBtiA8WFxDngJo9U1OUUZDQwDfgJigFDgSaAC8DMQDQQBjdLU1R04BtwBLgATMJ8FyCCW0uY67sYyOG0sFl7Pw+bnXDHHcRR4Jk2ZIsB08z7vAKeBt4Aq5teWcllqfs5SjA9zgNeAq4BDmv2uAH7KShzm15qqLvN6H/Pjktk4bmeB94EvgX+Bi8C79zhGgyy8zirAZOC4hbJRKR5PNv8NegOngEhgfcp4zeUGpoj5aorjeDZNvWct1ZPiOP8DxJt/vppmuwaGAKvNx/g00M/W/3uy/LfINSiRkSjz8qxSyjkbz1uG8e26LdAV6Gd+nNb7wEqgARAIfA8sBr4AHgUuY3yoA6CUaozxQbIWqAeMA94D3rhHLEuB6sBTwHPAAIwP0ntxB7YCvubY1gBrlVK107zGARint+pgtDAjMD78e5jL1MU4LTrCQh0/YHwBeCrF63PDOF5+WYyjO0YimWqu5yFLLyYbx+0djITQCPgUmKGUamFpn8AqoKP592bmui9kUNaSKsALQDegPcbf+6MUMb+GkSy/AepjnGIOMW9uav75qrneu49TUUp1AxYA84BHgPnAF0qpLmmKTsL4ItDA/LqWKKUsvV+FLdg6Q8qSdxeMD9ubQBxwAJgFPJamTADmVgtQC+NbafMU2ysCSaRvQX2c4vEj5nUjU6zzIUVLAPgO2Jmm7snAxQxiqWl+fqsU2yunjSWLx+Eg8L759xrm/XbMoGyquFOsX4q5BWV+vA5YnuJxP+A24JyVOMyPzwKj71V/Fo/bWeD7NGX+TlmXhViamOupkma/WWlBxQHFUqybAPyT4vFFjOucGdWtgZ6Z1LMPWGLhb7D3Hu9DB4wWvbSi8sgiLSiRIa31GqAc0AXj23xL4KBSanwGT6kNmDBaRHf3cQGjNZTWHyl+v2r+eczCutLmn3UwPnRS2guUV0oVtbD/OuZYDqWI5VwGsSRTSrkppWYopf5USt0y9wxrAlQyF3nUvN9f77WfLPADnlNKuZof98XovBGXxTiyKqvH7Y80ZS7z37G3tnM69TW55LqUUqWB8sCOB6wjo9ftnWZd8uvWWicC18m51y2ySRKUuCetdZzW2l9rPVVr3RLjNNxkc2+xtFQ2dp2Qspp7rLv7HlUp1qUL8wFjSWkW0AuYiNEhpCFGkrv7eu93v2ltAhKBruYP5af47/ReVuLIqqwetwQL27L7+WAi/fFxtFDuXnVZ6/je3W9m66zxukUOkT+EyK4/MU6FWLou9RfGe6rx3RVKqQoYrTBr1Pt4mnWPY5yqstSt/G4sydcolFKVshDL48C3Wus1Wus/ME43PZxi+1Hzfp/M4Pnx5p/296pEa30Ho3t2X4zrMVeAXdmI425d96yH7B+3B3EdKKOUSplkGmZnB1rrq8AloN09iiWQ+ev+C8uv+8/sxCNsSxKUsEgpVUIptVMp1U8pVV8pVVUp1QsYA+zQWv+b9jla65MYvfD+p5RqrpRqiHGhO4aMv8Vn1WygjVJqslKqplKqLzAKmGGpsDmWbcCXSqkW5liWknlX5FCgm1KqkVKqHkarJjkZa63/xujk8LVSqof5uDyhlOpvLnIO47V2VkqVUkq536MuP6ADMBRYobU2ZTUOs7PAE0qp8ve4MTdbx+0BBWDcgzVeKfWwUuploOd97Ocj4G2l1DvmmBsqpUal2H4WaKeUKmu+H8uSmUB/pdTrSqkaSqk3Mb4M5MTrFjlEEpTISBTGRfkRGN/sQzC6Vq/A+MafkUEY3/YDMLqbf4dxQ2fcgwSjtT6KccqrB+abhc3LvUaOGAScAXYCG82xn82kqpHmePdgXHc7aP49pQHmfX0GnMBIfMXMcV4CPsD4kL2aSXy7MVoL3qQ+vZfVOCZhdEI5hdF6Sec+j9t90Vr/hXH7wBCMazu+GO+Z7O5nIfA6Rk+94xhfNOqmKDIKowV7Afg9g32sB97E6J34J8b7eLjWemN24xG2IyNJiBxl/mZ/Gehj7nQhhBBZIiNJCKtSSrUFPDB65JXGaEmEY3wLFkKILLPqKT6l1BtKqUCl1B2l1NJ7lBuolDqilPpXKXXR3J1WkmXB4AhMw0hQGzGu+bTWWkfbNCohRL5j1VN8SqnuGF1NOwAuWutBGZQbhnFu+TegFMa1itVa60+sFowQQoh8zaqtFq31WgClVBOMcdUyKrcwxcNLSqnvyLjbrhBCiEIor5xWa81/Y22lopQagtErCBcXl8YVK1bMzbiyxGQyYWcnHSKzQo5V5i5cuIDWmkqVsjtoROFky/dUkk7CXmV2S1bekVf//0JDQ8O11qXSrrd5glJKvYQxhMsrlrZrrRcBiwCaNGmiAwMDLRWzqYCAAHx8fGwdRr4gxypzPj4+REREEBQUZOtQ8oXcfE9FxUfRb20/pvhMoUHZBrlSpzXl1f8/pdQ5S+ttmkqVUs9h3JPRSaeY3E0IIfKa+KR4evzQg42hGzl32+LnqbAym7WglFIdga+AzlrrY5mVF0IIWzFpEwPXD2T7qe0sfnYxz9Z61tYhFQpWTVDmruIOGONk2ZvnEUo0jxKcslxbjBEGummtD6XfkxBC5A1aa0ZsHcHK4yv5pN0nDH50sK1DKjSsfYrvfYz7XsZhzG8TC7yvlKqklIoyD9YJxgjNxYAt5vVRSqmtVo5FCCEeWHxSPKE3QxnZfCRjWo2xdTiFirW7mU/GmJDMEvcU5aRLuRAizzNpE04OTmzqswl7O3tSD9Quclre628ohBB5wJo/1/D4kse5EXMDR3tH7JR8XOY2OeJCCJHGr2d+5cW1L6KUwsXRxdbhFFqSoIQQIoWjYUfpurIrNbxqsLHPRlwdXW0dUqElCUoIIcz+vvE3Hf06UtylOD/3+xkvFy9bh1SoSYISQggzBzsHHvZ6mO39tlO+aHlbh1Po2XyoIyGEsLXo+GhcHF2oWrwq+wfvl956eYS0oIQQhVpsQiwdv+vIaxtfA5DklIdIghJCFFqJpkRe+PEF9p3fh+/DvrYOR6Qhp/iEEIWS1ppXN77KxtCNfPH0Fzxf93lbhyTSkBaUEKJQmrBzAkuDlvJBmw8Y1nSYrcMRFkgLSghRKLWr2o74pHg+aPOBrUMRGZAEJYQoVM5GnKWKZxXaVWtHu2rtbB2OuAc5xSeEKDQ2hW6i5v/VZO1fa20disgCSVBCiEJh7/m99FrdiwZlG+BbTXrs5QeSoIQQBd6xq8fo8n0XKhWrxJYXt+Dh5GHrkEQWSIISQhRoEXERdPDrgKujK9v7baeUWylbhySySDpJCCEKNE9nT95v/T6tK7emsmdlW4cjskESlBCiQIq8E8mZiDPUL1Of4U2H2zoccR/kFJ8QosC5k3iHbqu64bPUh4i4CFuHI+6TtKCEEAVKkimJfuv6sePMDpY9twxPZ09bhyTuk7SghBAFhtaaN7a8wY9//sjs9rMZ0GCArUMSD0ASlBCiwFhxbAX/O/I/xrYay8gWI20djnhAcopPCFFgPF/3ee4k3eGlhi/ZOhRhBVZtQSml3lBKBSql7iillmZS9h2l1BWl1G2l1BKllJM1YxFCFB5b/97K1airONo7MvjRwTLpYAFh7VN8l4FpwJJ7FVJKdQDGAe2AKkA1YIqVYxFCFAKBNwPpurIrY34ZY+tQhJUprbX1d6rUNKCC1npQBttXAGe11uPNj9sB32mty95rvx4eHrpx48ap1j3//PMMHz6cmJgYnn766XTPGTRoEIMGDSI8PJyePXum2z5s2DBeeOEFLly4QP/+/dNtHzVqFF26dOHkyZO89tpr6ba///77ODg44Onpydtvv51u+/Tp02nZsiX79+9n/Pjx6bbPmzePhg0b8ssvvzBt2rR027/88ktq1arFxo0bmT17drrty5cvp2LFiqxatYqFCxem2/7jjz9SsmRJli5dytKlS9Nt37JlC66urnzxxRf88MMP6bYHBAQAMGvWLDZt2pRqm4uLC1u3bgXgww8/ZMeOHam2lyhRgjVr1gDw3nvvceDAASIiIvD0NHpVVahQAT8/PwDefvttgoKCUj2/Zs2aLFq0CIAhQ4YQGhqaanvDhg2ZN28eAP369ePixYuptrdo0YKPP/4YgB49enDjxo1U29u1a8fEiRMB6NSpE7Gxsam2P/PMM4wePRoAHx+fdMcmp957QUFBJCYm8v3332f63nvqqacICgoqtO+9w5cO03JRS5xinGj4e0McEo2rFpbeeykV1vfe3f8/a3zuWfO9t2vXriNa6yZpy9nqGlRd4KcUj4OBMkqpElrrVH9JpdQQYAiAo6MjERGp72kIDQ0lICCAuLi4dNsATpw4QUBAALdv37a4PSQkhICAAK5du2Zx+7Fjx/Dw8OD8+fMWtwcHB1OrVi3++ecfi9uPHj1KfHw8x48ft7g9MDCQiIgIgoODLW7/7bffCAsL49ixYxa3HzhwgFOnThESEmJx+759+yhWrBgnTpywuH337t04OzsTGhpqcfvdD4lTp06l2x4bG5u8/cyZM+m2m0ym5O13j19SUlJyOUdHx+TtFy9eTPf8y5cvJ2+/fPlyuu0XL15M3n716tV028+fP5+8/fr16/z777+ptp85cyZ5+82bN7lz506q7adOnUrebunY5NR7LzExEa11lt57Dg4Ohfa99+2Wb3nz9zdxTnKm0u5KRN2JSt5u6b2XUmF97939/3vQz72goGASEpwICTnP1atFSUpyxWRyRmsXTCYnvvoqkp9+OsGZM/GEhnZBaydMJmOb1k4MH+5KkSLXuHatKhcufAK0SFcH2K4FdQp4XWu9zfzYEYgHqmqtz2a03yZNmujAwECrx/ugAgICLH7LEenJscqcj48PERER6b7Vi9Q6+nXk9yu/M6fuHPp26mvrcPKFgIAA2rTxITYWbt60vNy6Bf/+C5GRxpLy95SLdVOHylMtqCigaIrHd3+PtEEsQoh8yK+7H2GRYdz460bmhQu4uDi4etVYrlwxlpS/h4cbyefKlRZERkKaBtt9cXEBD4//FldXY52l5V7bXFygQwfLddgqQYUADYC7J54bAFfTnt4TQoiUouOjmX1gNuMeH0dJ15KUdC1JwF8Btg4rR8XFwcWLcP586uXCBePnlStg4SxcBozO0kWKQIkS4OWVfileHIoWTZ180j728ACHB8ge8fHxfPfdd/Ts2R+He+zIqglKKeVg3qc9YK+UcgYStdaJaYp+CyxVSn0HhAHvA0utGYsQomBJSEqg1+pe/HzqZ1pXbo1PFR9bh2QVWsONG/D336mX06eNBHT1aub7cHCAMmWgbFnLP0uVMhLSyZMH6Ny5Ba6uYKue+GfOnKFLly6EhITQrl07KlWqlGFZa7eg3gc+SPG4HzBFKbUE+BPw1lqf11pvU0rNAH4FXIA1aZ4nhBDJTNrE4A2D2frPVr585st8mZy0NhLO8eNw7JjxMzTUSEb3agHZ20OFClCpUvqlYkUoV85o9dhl4aahW7fu4OZmvdeUXatXr2bw4MHExMTg4uKS6f1qVk1QWuvJwOQMNrunKTsHmGPN+oUQBY/WmtHbR+P3hx/TnpzGkMZDbB1SpmJi4PffjeVuMjp+3OhwYImHB9SokXp5+GGoXBkeeshIUvlZXFwcr7/+OitXriQmJgYApRR2mWRVGepICJGnXfz3Ikt+X8Jbzd5i/BPp76mxtYQECAmBw4fh0CHj5/HjkJSUvmypUlCvHjzyiLHUrg01a0Lp0rY75ZbTQkND6dy5M5cuXUp1v5fWOndbUEIIYW0Vi1Xk99d+p7Jn5TwxhFFkJOzbB7t2wZ49cPQopLnPFjs7qF8fGjc2ft5NSmXK2CZmW1m+fDlDhw4lNjYWS7c0SQtKCJEv/XTiJ0JvhPJuq3epWryqzeKIjjaS0c6dxs+jR8FkSl3m4YehaVNo1sz4+eij2PRaj61FR0fz6quv8tNPPyWf0ktLWlBCiHxp97ndvPDjCzQs25C3HnsLJ4fcG0taa+O60bZt8PPPsHcvxMf/t93eHh57DNq0gdatoXlzo4ecMBw/fpxnnnmGq1evEhcXd8+ykqCEEPlK8JVgunzfhWrFq7H5xc25kpzu3DFaSOvWwaZNEBb23zaljJaRr6+RlFq0AHf3jPdVmK1Zs4Z+/fplmpjAaEHJKT4hRL5x+tZpOvh1oKhTUX7u9zMlXHOuaRIVBVu3Gklp8+bUPeweesgY3aBDByMxSQspa4oWLYqnpydRUVFERUVlWl4SlBAi39h/YT8mbWJ7v+1ULFbR6vtPSDBO2/n5wU8/GaM03FW/PnTrBs89Bw0aFNxedTnJ19eX8+fPs2zZMsaPH090dLRcgxJC5G93P6z61e9Hl5pdKOZczIr7ht9+M5LSypXGqA13tWxpJKVu3YyODuLBOTo68sorr/D333/z2Wef3bOsJCghRJ4WlxhHjx968GazN+lYvaPVktONG7BsGSxaBCdP/re+bl3o3x/69DFGYxDWd+PGDRYsWJDqWlSRIkVwdHQkOjo6eV1mp/isPaOuEEJkWaIpkT5r+rDl7y3cjL35wPvTGvbvhwEDoHx5GDXKSE4PPQSjR0NQkNFDb+xYSU45afr06SSluVPZzs6O9957D09PT1xdXUlKSsq0BSUJSghhE1prhm0axvoT65nfcT4v1nvxvvcVFwdff21cO2rVCpYvN7qGd+wI69cbY+DNnCnXlnLD9evXWbhwYapJGB0dHenXrx8TJkzg8uXLTJ06lbp16+LkdO8emnKKTwhhE+/vfJ+vf/+a9594n7cee+u+9nHrFvj5VaJ37/9G/S5dGl5+GV59Fara7v7eQmvatGmY0tzJbG9vz5QpUwBwcXFh1KhRjBo1KtN9SYISQuQ6rTXXoq8xpNEQpj45NdvPP3sW5s6FxYshOroaYLSORo+G55835jsSue/q1at89dVXqVpPRYoU4aWXXqJcuXLZ3p8kKCFErkpISsDR3pFFXRZh0qZsja937hxMmwZLl0KieZa5Jk1uMn26F089JafvbG3q1Knprj3Z29szadKk+9qfXIMSQuSarX9vpe4XdTl96zRKKeztsjaPxKVLMHy4MQ3F118bY+H17Wt0epg58w98fSU52VpYWBjffPMN8SnGhXJycuLll1+mbNmy97VPSVBCiFxx4MIBevzQA/ci7pR0LZml51y5AiNGGPcoLVxotJr69oW//jLua2rQIIeDFlk2efJkiz33Jk6ceN/7lFN8QogcF3IthM4rOlPOoxxb+26lqFPRe5aPjTWuMX38sTEkEUCvXjB5Mnh753y8InsuXrzIt99+m671NHToUEqXLn3f+5UEJYTIUedvn6eDXwecHJzY3n87ZdwznhRJa/jhB+M+pXPnjHVduhjXnerXz6WARbZNmjTJ4rWn8eMfbIJJOcUnhMhR7kXcaVC2AT/3+5lqxatlWO7wYXjiCejd20hO9erBL7/Ahg2SnPKy8+fP8/3335OQkJC8ztnZmeHDh1OyZNZO5WZEWlBCiBwRFR+Fg50DXi5ebH5xc4blbt0yWkxffWU8LlXKaDG9/LIx95LI2yZOnJjhqBEPShKUEMLq4pPi6b6qOxrNz/1+xk6lP1mjNaxaBW+/bdxk6+ho/D5hAhSz3lixIgedPXuWH374IV3racSIEXh5eT3w/iVBCSGsyqRNDFw/EP/T/ix5donF5HTmjNFtfNs24/Hjj8OXX0oHiPxmwoQJJN69Ic3M3t6eMWPGWGX/cg1KCGE1WmtGbB3ByuMr+fSpT3np0ZdSbU9KglmzjBHFt20DT09jtPFduyQ55TenTp1i7dq1qRKUi4sL77zzDp6enlapw6oJSinlpZRap5SKVkqdU0pZHP1RGaYppS4ppW4rpQKUUnWtGYsQIvfN2j+LBYcXMKrFKN5t+W6qbadPG1Omv/uu0Y28d2/jfqZXX4VMZl0QedB7772X6tQeGNeeRo8ebbU6rH2K73MgHigDNAQ2K6WCtdYhacr1AgYDjwPngGnAcqCRleMRQuSip2s8zdXoq8zwnZE8hJHWxugP77wD0dHG1Bdffw1PP23jYMV9i46OZu3atak6R7i4uDBmzBiKWfECotW+tyil3IAewEStdZTWei+wAehvoXhVYK/W+rTWOgnwA6SBL0Q+9df1v9BaU7d0XWa1n5V83enqVXj2WRgyxEhOzz9vzMckySl/c3NzY/fu3Tz22GO4ubkBxrWnd955x6r1WLMFVRNI0lqHplgXDLSxUHYl8IJSqiZwBhgIbLO0U6XUEGAIQJkyZQgICLBiyNYRFRWVJ+PKi+RYZS4iIoKkpKR8c5yO3jrKuGPjeK3aa/So0CN5/b59JZg5sxa3bxfB3T2BESP+pl27axw7Zt365T2VddY+Vp988gnBwcF8+eWXPPnkkxw5csRq+waMi5rWWIAngCtp1r0KBFgoWwSYD2ggESNJVc2sjsaNG+u86Ndff7V1CPmGHKvMtWnTRjdo0MDWYWRJ4KVA7T7dXT/yxSP6ZsxNrbXW8fFajxyptXFyT+t27bS+cCHnYpD3VNbl1WMFBGoLn/nWvDQZBaQdYKsoEGmh7AdAU6Ai4AxMAXYqpVytGI8QIgf9feNvOn3XiRIuJdjWdxvFXYpz/jy0bg1z5oCDgzGL7fbtUKGCraMV+ZE1E1Qo4KCUqpFiXQMgbQeJu+tXaa0vaq0TtdZLgeLIdSgh8oX4pHieXvE0Gs32/tspX7Q8mzfDo4/CwYNQsSLs3m1MICg99MT9stpbR2sdDawFpiql3JRSrYCuGL3z0joM9FJKlVFK2Sml+gOOwD/WikcIkXOK2Bdhlu8stvbdSrViNRk3Dp55Bm7eNDpA/P47tGhh6yhFfmft7zbDARfgGvA9MExrHaKUqqSUilJKVTKX+xSjA0UQEAG8A/TQWkdYOR4hhBXFJMSw6+wuALrW7ko15yZ06gSffmqMm/fJJ7BxI5QoYeNARYFg1fugtNY3gecsrD8PuKd4HAe8bl6EEPlAQlICL/z4Aj//8zP/vPUPkRcr0bUrnDoFpUvD6tXG9SeRt/j4+PDII4+wYMECW4eSbXJ2WAiRKa01r258lU2hm5jfcT7BuyvRvLmRnBo1gsDAgpWcrl+/zvDhw6lSpQpOTk6UKVOGdu3a4e/vn6XnBwQEoJQiPDw8hyP9z9KlS3F3d0+3fu3atXz88ce5Foc1yWCxQohMjf1lLMuClzG5zRRu+Q/j9feNTuS9e8PixeBawPrf9ujRg5iYGBYvXkz16tW5du0au3bt4saNG7keS3x8PEWKFLnv51tjVHFbkRaUEOKefjn9CzP3z+S1BiM48eVEJkww1k+fDitWFLzkFBERwZ49e/jkk09o164dlStXpmnTpowePZrevXsD4OfnR9OmTfHw8KB06dL06tWLS5cuAcYUFE8++SQApUqVQinFoEGDAON02xtvvJGqvkGDBvHMM88kP/bx8WHYsGGMHj2aUqVK0apVKwDmzJlD/fr1cXNzo3z58rzyyitERBiX7QMCAnjppZeIjo5GKYVSismTJ1uss0qVKkybNo3XXnuNokWLUqFCBWbOnJkqptDQUNq0aYOzszO1atViy5YtuLu7s3TpUusc5CySBCWEuKd2VduxxPcnjs+ay8qVCnd3+OkneO89MA+3V6C4u7vj7u7Ohg0biIuLs1gmPj6eKVOmEBwczKZNmwgPD6dPnz4AVKxYkTVr1gAQEhJCWFgY8+fPz1YMfn5+aK3Zs2cP3377LWAMxDpv3jxCQkJYsWIFhw4d4s033wSgZcuWzJs3D1dXV8LCwggLC7vnoK1z586lXr16HD16lLFjxzJmzBgOHDgAgMlkolu3bjg4OHDw4EGWLl3KlClTuHPnTrZegzXIKT4hhEXb/tlG5WKVcY6qw6eDn+XkSShfHrZsKdhTsDs4OLB06VJeffVVFi1axKOPPkqrVq3o1asXjz32GACDBw9OLl+tWjUWLlxInTp1uHjxIhUqVEg+rVa6dOn7mva8atWqzJ49O9W6t99+O/n3KlWqMGPGDLp27cqyZcsoUqQIxYoVQylF2bJlM91/+/btk1tVb775Jp999hk7duygRYsW+Pv7c/LkSbZv30758uUBI6HdbcnlJmlBCSHS2Xt+L91WdWPwwi9o3hxOnjSS0sGDBTs53dWjRw8uX77Mxo0b6dSpE/v376d58+ZMnz4dgKNHj9K1a1cqV66Mh4cHTZo0AeD8+fNWqb9x48bp1u3cuRNfX18qVKiAh4cH3bt3Jz4+nitXrmR7//XT/BHLlSvHtWvXADhx4gTlypVLTk4ATZs2xc4Gd1xLghJCpHLs6jGeWfEMXhcG8Menn3HtGjz1lDEyRGEassjZ2RlfX18mTZrE/v37efnll5k8eTK3b9+mQ0YyfZ0AACAASURBVIcOuLq6snz5cg4fPsw289TA8fHx99ynnZ3d3fFIk6WdUwlIHiH8rnPnztG5c2fq1KnD6tWrOXLkCEuWLMlSnZY4OjqmeqyUwmQyAUaPTZVHzt1KghJCJDsbcZYOfh3gyBCufPU/YmIUAwfC5s1gxWl+8iVvb28SExMJCgoiPDyc6dOn07p1a2rXrp3c+rjrbq+7lPMlgdFpIiwsLNW64ODgTOsODAwkPj6euXPn0qJFC2rWrMnly5fT1Zm2vvtRp04dLl26lGr/gYGByQksN0mCEkIkm7prKhG/DOH2jzMwmRSTJsE338AD9HLOd27cuEHbtm3x8/Pjjz/+4MyZM6xevZoZM2bQrl07vL29cXJyYsGCBZw+fZrNmzczceLEVPuoXLkySik2b97M9evXiYqKAqBt27Zs3bqVDRs2cPLkSUaOHMmFCxcyjalGjRqYTCbmzZvHmTNn+P7775k3b16qMlWqVCEuLg5/f3/Cw8OJiYm5r9fv6+tLrVq1GDhwIMHBwRw8eJCRI0fi4OCQ6y0rSVBCCMC4r6nE/kXEbpuMUvDFFzBlSsHsqXcv7u7uNG/enPnz59OmTRvq1q3L+PHjefHFF1m1ahWlSpVi2bJlrF+/Hm9vb6ZMmcKcOXNS7aN8+fJMmTKFCRMmUKZMmeQOCYMHD05eWrVqhbu7O926dcs0pvr16zN//nzmzJmDt7c3X3/9NbNmzUpVpmXLlgwdOpQ+ffpQqlQpZsyYcV+v387OjnXr1nHnzh2aNWvGwIEDmTBhAkopnJ2d72uf983SHBx5dZH5oPI/OVaZy+35oOIS4vTILe/ql16J06C1vb3W332Xa9U/MHlPZd39HqugoCAN6MDAQOsGZEYG80FJN3MhCrEkUxIvrh7I2uld4bgTzs7GmHop7hsVhdC6detwc3OjRo0anD17lpEjR9KgQQMaNWqUq3FIghKikNJa89q6t1n7QT/4+xk8PIyRyNu0sXVkwtYiIyMZO3YsFy5coHjx4vj4+DB37txcvwYlCUqIQmrCto9YPOZZOO1LiRKwbRuYb+cRhdyAAQMYMGCArcOQBCVEYXT5xm3mvNUWTrekTBnNjh2KunVtHZUQqUmCEqKQiYqCPt2Lceefljz0kGbnTkXt2raOSoj0pJu5EIXIT8E7qfnYKXbvhnLlICBAkpPIu6QFJUQhsfOvI3R/1hXT+YcpX95EQIAd1avbOiohMiYJSohCIPBMKB06aEwXmlO+YiK7AxyoVs3WUQlxb5KghCjgTl6+RKt2t0m80JTyFRPYu9uRKlVsHZUQmZNrUEIUYLGx0LunK/FnmlLmoXj27JLkJPIPSVBCFFBxcZru3SHoQHHKlNXsDihC1aq2jkqIrJNTfEIUQDFxCVR7/AhXjzSnZEnYuUNRs6atoxIie6zaglJKeSml1imlopVS55RSL96jbDWl1CalVKRSKlwpdX9D7wohUolPMFGn3VGuHmmOa9E4fvkFvL1tHZUQ2WftU3yfA/FAGaAvsFAple7+dKVUEcAf2AmUBSoAflaORYhCJzFR06DjUc7vfwwntzh27XCmQQNbRyXE/bFaglJKuQE9gIla6yit9V5gA9DfQvFBwGWt9RytdbTWOk5r/Ye1YhGiMDKZoGW3YE7sbIKj8x12/OwkY+uJfM2a16BqAkla69AU64IBS2MjNwfOKqW2Ak2B48CbWutjaQsqpYYAQwDKlClDQECAFUO2jqioqDwZV14kxypzERERJCUlZes4aQ2ff16dw5saYucYx6cf/0lCwr8UhkMt76msy2/HypoJyh24nWbdbcDDQtkKwJPAs8AOYATwk1KqttY6PmVBrfUiYBFAkyZNtI+PjxVDto6AgADyYlx5kRyrzHl6ehIREZGt4/TBlATWrHHE0RE2bnSiQ4fcnbfHluQ9lXX57VhZ8xpUFFA0zbqiQKSFsrHAXq31VnNCmgWUAOpYMR4hCoWR00KZOtkRpTQrVkCHDoVsjnZRYFkzQYUCDkqpGinWNQBCLJT9A9BWrFuIQunTL88yd5IxoN6sz6Lp2dPGAQlhRVZLUFrraGAtMFUp5aaUagV0BZZbKO4HNFdKPaWUsgfeBsKBv6wVjxAF3Tc/Xmbc6+VA2zFmUgQj33C3dUhCWJW1u5kPB1yAa8D3wDCtdYhSqpJSKkopVQlAa30S6Af8D7iFkcieTXv9SQhh2ZZfb/JyX09IKsKgYTf5ZLKnrUMSwuqsOpKE1vom8JyF9ecxOlGkXLcWo8UlhMiGkBDo16M4Ol7xdM/rLF5QCiWXnUQBJGPxCZGPnPznDk/5mrh1S/Hss/DT96Wwk/9iUUDJW1uIfOJSWCKNn7jBlTA7nmhtYuVKcJDRNEUBJglKiHwgIkJTv9Vloq+Uo0LNa2zcYIeLi62jEiJnSYISIo+LjYX6bc5y80wlvMqHc2RPaYoVs3VUQuQ8SVBC5GGJidCi4zku/FEVV69bBO4pQenSto5KiNwhCUqIPMpkgldegeDdlXH2iOFAQFGqVpXueqLwkAQlRB6kNQwYdo1ly8DVFX7d7kr9eva2DkuIXCV9gITIgy5G9eOPRaWxc0hk7VoHmje3dURC5D5pQQmRx5wNb8/N06NBmVj4dTQdOtg6IiFsQxKUEHnIwmXXORcyDoCpM28yZKB01xOFlyQoIfKI7ds1r7/sCdhRovI8Jo4qaeuQhLApSVBC5AG//Qbduyt0kiOlqvlRvthSW4ckhM1JJwkhbCzoj3ie6qCJjnaif384d24xt9POTQ0sWbKE69evU7duXerUqUOVKlWwt5eefaLgkgQlhA2dPpNEyycjib1dgieeus3ixcXw9bU8l+fBgwdZsmQJbm5uJCUlER8fT/ny5albty5NmjRJTlw1atTAyckpl1+JENYnCUoIG7lyRdPo8RvE3ixNlfoX+XlDBRwdMy4/depU/Pz8+Pfff5PXnT17lrNnz7J161bc3NwAiImJoXTp0tSuXZvGjRvz5ptvUqlSpZx+OUJYnVyDEsIGbt+GR5+4wu3LpSn98CV+31Uh08Ffy5Yty7Bhw3B2dk63zWQyERkZSWRkJElJSYSFhfHrr78ye/Zsrl69mkOvQoicJQlKiFwWEwNtO0Zx5Z+HKPrQFYL2PoRnFifEff/997N83cnV1ZUxY8bQtGnTB4hWCNuRBCVELkpIgF694OhBd0qWjePI3hI8VDbr/4bFixdnzJgxuGTS3LKzs6N69epMmzbtQUMWwmYkQQmRS0wm6NjjClu2QIkSsHunM9Wr3eOiUwZGjRpFkSJF7lnGycmJEiVKEBkZeb/hCmFzkqCEyAVaQ+/B19i5sSx2TtFs2WqiTp3725ebmxuTJ09O7hRhSWxsLPv27aNmzZrs3r37PqMWwrYkQQmRC15/9warl5UGhzusXBNLs6YP9q83bNgwXF1d71kmPj6e8PBwOnbsyNixY0lISHigOoXIbZKghMhhH3x8m4WzS4BdIp8vCadX5wcfwsjJyYlPP/00XSvK0rWp2NhYFixYQOPGjTl9+vQD1y1EbrFqglJKeSml1imlopVS55RSL2bhOTuVUlopJfdkiQLnm29g6nhjwNfJcy4wvH95q+17wIABeHl5JT92cXHhvffew93dPV1Pv5iYGEJCQqhfvz5+fn5Wi0GInGTtFtTnQDxQBugLLFRK1c2osFKqL3KzsCig1q0zZsQFeG/aVT4YUdWq+7e3t2fu3Lm4ubnh6urKuHHjmDhxIsePH6devXrpTgGaTCaio6N57bXX6NWrV6obfoXIi6yWoJRSbkAPYKLWOkprvRfYAPTPoHwx4ANgjLViECKv2LAxkZ7PJ2IywaRJMH1CmRypp3v37lSsWJFHHnmECRMmAFC5cmUOHz7MyJEjLZ7yi4mJYdOmTdSqVYtDhw7lSFxCWIPS2vK4X9nekVKPAvu11i4p1o0G2mitu1go/znwD7AOOAM4aq0TLZQbAgwBKFOmTOOVK1daJV5rioqKwt3d3dZh5AuF4VgdDizGuPfqYkosQuNOe5j5bhJKZf35b7/9NklJSfzf//1flsrfuHEDJycni8f12LFjTJo0iejoaIudJJycnOjTpw/9+vXLtwPPFob3lLXk1WP15JNPHtFaN0m3QWttlQV4AriSZt2rQICFsk2AIIzTe1UADThkVkfjxo11XvTrr7/aOoR8o6Afq127tHZwuqNB66ZdD2uTKfv7aNOmjW7QoIHVYrp165Z+9tlntaurqzb/r6VaXF1dddOmTfWFCxesVmduKujvKWvKq8cKCNQWPvOteQ0qCiiaZl1RINWdgkopO+ALYIS20GISIr86eBB8O8aTeKcIdXwPcGBN42y1nHKKp6cn69ev5/PPP8fNzQ07u9T/9jExMRw9ehRvb2/WrFljoyiFSM+aCSoUcFBK1UixrgEQkqZcUYwW1Cql1BXgsHn9RaXUE1aMR4hcc+QIdOyoiY8tQqXH9xK85THs7fNAdjJTSjFo0CCCgoKoVatWug4USUlJREZGMmDAAAYMGEB0dLSNIhXiP1ZLUFrraGAtMFUp5aaUagV0BZanKXobKAc0NC9Pm9c3Bn6zVjxC5JY//oD27eH2bUXXbgn86d8UR4e8eYth9erVCQoKYujQoRl2oFi9ejV16tQhKCjIBhEK8R9r/xcNB1yAa8D3wDCtdYhSqpJSKkopVcl8yvHK3QW4bn7uVa11vJXjESJH/fUXtGkbz82b8HTnRH5Y6Yibc96eLLBIkSLMnj2bTZs24eXllW5cv7i4OC5cuEDLli2ZOXMmJpPJRpGKws6qCUprfVNr/ZzW2k1rXUlrvcK8/rzW2l1rfd7Cc85qrZVcjxL5TWgotHkygYgbRXCtvZcvv40gkzFc85S2bdsSGhqKj4+PxXH9YmNjmTJlCm3atOHKlSs2iFAUdnnzPIS4Lz4+Przxxhu2DqNQOHECHm+dyPWrjhR5eD9HdlSigteDD2GU20qUKMG2bduYMWMGrq6uqDS9OqKjozl48CC1a9dm8+bNNopSFFaFPkFdv36d4cOHU6VKFZycnChTpgzt2rXD398/S88PCAjgySefJDw8PIcj/c/SpUst3suwdu1aPv7441yLo7AKCYHWbZK4ftUBh2q72evvSe1y+XdKdaUUw4cP5/Dhw1SrVi3dtanExERu377N888/z9ChQ4mLi7NRpKKwKfQJqkePHhw6dIjFixcTGhrKpk2b6NSpEzdu3Mj1WOLjH+wSnJeXFx4eHlaKRlhy7Bg8+SRcv2aPS829bN/qRNOq3rYOyyq8vb05fvw4AwcOtDhSekxMDN9++y2PPPIIf/75pw0iFIWOpZuj8upi7Rt1b926pQHt7++fYZnly5frJk2aaHd3d12qVCnds2dPffHiRa211mfOnEl30+PAgQO11sbNlq+//nqqfQ0cOFB37tw5+XGbNm300KFD9ahRo3TJkiV1kyZNtNZaz549W9erV0+7urrqcuXK6ZdfflnfunVLa23caJe2zg8++MBinZUrV9YffvihHjJkiPbw8NDly5fXM2bMSBXTyZMndevWrbWTk5OuWbOm3rx5s3Zzc9PffPPNfR3TzOTVGwWzIihI6xIlTBq07tBB69uR8TlSj7Vv1L0fmzdv1sWKFdMODg7p3m9KKe3q6qoXLFigTfdzJ7KV5ef3VG7Lq8eKXLhRN99xd3fH3d2dDRs2ZHjaIj4+nilTphAcHMymTZsIDw+nT58+AFSsWDH5xsaQkBDCwsKYP39+tmLw8/NDa82ePXv49ttvAWO67nnz5hESEsKKFSs4dOgQb775JgAtW7Zk3rx5uLq6EhYWRlhYGKNHj85w/3PnzqVevXocPXqUsWPHMmbMGA4cOAAYg4d269YNBwcHDh48yNKlS5kyZQp37tzJ1msoDI4ehbZtNTduKKo1O8H69VDUPfuz4eYXTz/9NCdPnqRFixbpOlBorYmJiWHMmDG0b98+V09vi0LGUtbKq0tODHX0448/6uLFi2snJyfdvHlzPWrUKH3w4MEMy//1118aSB4W5m6L5vr166nKZbUFVa9evUxj3Lp1qy5SpIhOSkrSWmv9zTffaDc3t3TlLLWgevfunapM9erV9Ycffqi11nrbtm3a3t4+uUWotdb79u3TgLSgUti7V+tixYyWEzV/0jMC5udofXmhBXVXUlKSnjlzZobDJDk6OmovLy+9Y8cOm8WYH99TtpJXjxXSgrKsR48eXL58mY0bN9KpUyf2799P8+bNmT59OgBHjx6la9euVK5cGQ8PD5o0McYzPH8+XY/5+9K4ceN063bu3Imvry8VKlTAw8OD7t27Ex8ff19dfevXr5/qcbly5bh27RoAJ06coFy5cpQv/98cRU2bNk03FE5h9vPP4Otr3IRLnTW8O+8Q77Z5y9Zh5Ro7OztGjx7Nvn37qFixIs7Ozqm2JyQkcPPmTZ555hneeeedB76OKkRK8kkEODs74+vry6RJk9i/fz8vv/wykydP5vbt23To0AFXV1eWL1/O4cOH2bZtG5B5hwY7O7u7A+MmszSadNrTJ+fOnaNz587UqVOH1atXc+TIEZYsWZKlOi1xdEx9GkoplXzjpdY6Xbdi8Z8ff4QuXSA2Fmi4hJc++plPO35o67BsomHDhpw4cYLnn3/eYgeK2NhYFi1aRMOGDfn7779tEKEoiCRBWeDt7U1iYiJBQUGEh4czffp0WrduTe3atZNbH3fdvQs/KSkp1fpSpUoRFhaWal1wcHCmdQcGBhIfH8/cuXNp0aIFNWvW5PLly+nqTFvf/ahTpw6XLl1Ktf/AwEAZOQBYvBheeAESEqBTv5P0em87i7p+UagTuqurK8uWLWP58uV4eHhYnLX35MmTjBgxwkYRioKmUCeoGzdu0LZtW/z8/Pjjjz84c+YMq1evZsaMGbRr1w5vb2+cnJxYsGABp0+fZvPmzUycODHVPipXroxSis2bN3P9+nWioqIA4y79rVu3smHDBk6ePMnIkSO5cOFCpjHVqFEDk8nEvHnzOHPmDN9//z3z5s1LVaZKlSrExcXh7+9PeHg4MTEx9/X6fX19qVWrFgMHDiQ4OJiDBw8ycuRIHBwcCvUH8Zw5xky4JhNMmQKbv63Fql7f42Ankz+DMUnin3/+SaNGjdK1ppydnZk7d66NIhMFTaFOUO7u7jRv3pz58+fTpk0b6taty/jx43nxxRdZtWoVpUqVYtmyZaxfvx5vb2+mTJnCnDlzUu2jfPnyDBo0iAkTJlCmTJnkkRwGDx6cvLRq1Qp3d3e6deuWaUz169dn/vz5zJkzB29vb77++mtmzZqVqkzLli0ZOnQoffr0oVSpUsyYMeO+Xr+dnR3r1q3jzp07NGvWjIEDBzJhwgSUUumuNRQGJhO89x6MGmU8dukyjsa9N6MUhTphW1KhQgUOHDjAuHHjkm/sdXNzY968edSqVcvG0YkCw1LPiby6yISFOS8oKEgDOjAwMEf2n1ePVVyc1i++qDVobW9v0m7PD9fV5lfTYZFhuR5LXurFlxW//fabLlu2rO7cubNN7ovKq++pvCivHisy6MUn5ywKuXXr1uHm5kaNGjU4e/YsI0eOpEGDBjRq1MjWoeWaiAjo1g0CAsDN3YRLn4HY1/Rne799lHUva+vw8rxmzZpx9uxZ7OzspKUprEoSVCEXGRnJ2LFjuXDhAsWLF8fHx4e5c+cWmg+a8+fh6aeN8fXKlDXhOrAXNzx/YVe/XTzs9bCtw8s3nJzy9hQjIn+SBFXI3Z1BtTAKCoLOneHyZahTB7ZsUXx9ug6+1d6iYdmGtg5PiEJPEpQolDZuhBdfhKgoaN3axIJvL1GlckWmVZlm69CEEGaFuhefKHy0hunToWtXIzn1eVFTbvhgfFc35WbsTVuHV+hVqVIlXa9VUXhJC0oUGjExMHgwrFoFSsFHH2muPvoOnx1axvS20/Fy8bJ1iIXCoEGDCA8PZ9OmTem2HT582OLsvqJwKhQtqHnz5vHSSy9x7NgxW4cibOT8eXj8cSM5eXjATz+BeuITPjs0n7cfe5txj4+zdYgCYwQWS0Mp5TYZUzBvKPAJKi4ujokTJ7J8+XIee+wxGjVqxJo1a9KNkycKrr17oWlT+P13ePhhOHgQEquvY/zO8fSr34/ZHWYXml6LeV3aU3xKKRYtWkSvXr1wc3OjWrVq+Pn5pXrO9evX6d27N8WLF6d48eJ07tw51XiAp06domvXrpQtWxY3NzcaNWqUrvVWpUoVJk+ezODBg/H09KRv3745+0JFlhT4BLVq1SrAGCsvNjaW33//nd69exMbG2vjyEROM5lg5kzw8YFr1+Cpp+DQIfD2hvYPt2eqz1SWPLsEO1Xg/w3ytalTp9K1a1eCg4N54YUXGDx4MOfOnQOM8f9GjhyJs7Mzu3bt4sCBAzz00EM89dRTyUOARUVF0alTJ/z9/QkODqZHjx50796dEydOpKpnzpw51K5dm8DAwOTZDIRtFfj/zE8++SR5fDwwvpE9++yzeeI0gsg5N28aHSHGjIGkJGP4oq1b4XRcIJF3InEr4sbENhNxtC+4kw4WFP3796dfv35Ur16dDz/8EAcHB/bs2QPAypUr0VrzzTffUL9+fWrXrs2XX35JVFRUciupQYMGDB06lHr16lG9enUmTJhAo0aN+PHHH1PV06ZNG8aMGUP16tWpUaNGrr9OkV6BTlCHDx9ON2+Tq6srY8aMsVFEIjf89hs8+ihs2gSensb1plmz4Nj132m7rC1DNw+1dYgiG1LOaebg4ECpUqWSZxU4cuQIYWFheHh4JM+QXaxYMW7dusWpU6cAiI6OZsyYMXh7e1O8eHHc3d0JDAxM99lwd643kXdYtRefUsoLWAy0B8KB97TWKyyUGwi8BdQA/gVWAOO11onWjGfmzJnppnIvX748zZo1s2Y1Io/QGj77DN5915gmo2lT+OEHqFIF/rn5Dx2/64insyefPvWprUMV2XCvOc1MJhPVq1dn8+bN6Z7n5WX0yhw9ejTbtm1j1qxZ1KhRA1dXVwYMGJCuI4T0Hsx7rN3N/HMgHigDNAQ2K6WCtdYhacq5Am8DvwGlgA3AaOATawUSHh7Oxo0bU81t5O7uztixY+WCeAF05YoxRcbdz6kRI2DGDChSBMIiw+jg14EkUxLbB22nQtEKtg1WWE2jRo1Yvnw5JUuWxNPT02KZvXv3MmDAAHr06AEYHadOnTpFzZo1czNUcR+sdopPKeUG9AAmaq2jtNZ7MRJP/7RltdYLtdZ7tNbxWutLwHdAK2vFArBo0SKLiahPnz7WrEbkAWvWwCOPGMnJ09OYCXfePCM5Abyy8RWuRl1lS98t1C5Z27bBCgD+/fdfgoKCUi1nz57N9n769u2Ll5cXXbt2ZdeuXZw5c4bdu3czatSo5J58NWvWZN26dRw9epRjx47Rr1+/dGdWRN5kzRZUTSBJax2aYl0w0CYLz20NpG1lAaCUGgIMAShTpgwBAQGZ7iwpKYkZM2ak6qnn4OBAhw4d+O2337IQTvZERUVlKS5h3WMVFWXPZ5/VwN/fGHG8SZObjBlzghIl4klZRX+v/vi6+hLzdwwBf1un7pwUERFBUlJSgX1PXblyhT179vDoo4+mWt+6devk1k3K1x4SEkLJkiWTH6ct89FHH7FixQqee+45oqOjKVGiBA0bNuTPP//k0qVL9OrVi5kzZybPy9azZ0+8vb25cuVK8j4s1VsQ5bvPKktzcNzPAjwBXEmz7lUgIJPnvQRcBEpmVkdW54PasGGD9vDw0EDy4uzsrE+fPp31CUqyIa/OsZIXWetY7dihdcWKxvxNLi5aL1igdcqpiBKSEvTXR77WSaYkq9SXm/LbfFC2Jv9/WZdXjxUZzAdlzV58UUDRNOuKApEZPUEp9RzGdadOWutwawXy8ccfExmZutrHHnuMqlWrWqsKYSM3bxrXmtq1gwsXoFkz4wbc1183hi8C40vXaxtf45WNr+B/yt+2AQsh7ps1E1Qo4KCUSnkDQQMyPnXXEfgK6KK1ttoYRKGhoQQFBaVa5+7uzrhxMpRNfqY1fPcd1K4Nixcb15emToV9+yDtDOPjd4xnSdASJrWeRIfqHWwTsBDigVntGpTWOloptRaYqpR6BaMXX1egZdqySqm2GB0jummtD1krBjDG3UtISEi1zt3dnfbt21uzGpGLTp2C4cNh+3bjcZs28L//GckqrTkH5vDJvk8Y2ngok30m52qcQgjrsvaNusMBF+Aa8D0wTGsdopSqpJSKUkpVMpebCBQDtpjXRymltj5o5dHR0SxbtozExP9up3J1dWXkyJHY2RXoe5ILpNhYmDbN6KG3fTsUL260nn791XJyuvTvJSbsnEBP754seHqB3E4gRD5n1fugtNY3gecsrD8PuKd4/KQ1671r+fLl6T6UTCYTr7zySk5UJ3KI1sao42PHGqOQA/TtC3PmQOnSGT+vfNHy7HlpD/VK18Pezj53ghVC5JgC06zQWjNjxgyio6OT19nZ2dGjRw+KFy9uw8hEdhw6ZEyL0aePkZzq14cdO8DPL+PktP/CfpYHLwegSbkmODk45WLEQoicUmAmLNy3b1/y+Fx3OTs7M3r0aBtFJLLjzBmYNMlIRGAko48+gpdeAvt7NIaOXztO5xWdKe1Wml51e+Hs4Jw7AQshclyBSVCffvppqtYTwMMPP0zDhg1tFJHIigsXjES0eDEkJhq98955B8aPh6Jpb1pI41zEOTr4dcDFwYVtfbdJchKigCkQCSosLAx//9T3u0jX8rwtLAymT4dFiyA+HuzsoH9/mDIFsnK72vXo67T3a09MQgy7B+2manG5x02IgqZAJKiFCxemW2dvb0/Pnj1tEI24l2vXnHjnHaObeFyccXPtCy/ABx9AnTpZ38/6E+s5f/s8/v39qVemXs4FLISwmXyfJF9KUwAADwtJREFUoBISEvi///s/7ty5k7zOycmJoUOHUuTuaKHC5oKDjdltV658jKQkY1337jB5MtS7j/zyauNXaf9weyp7VrZqnEKIvCPf9+Jbv359qvue7nrjjTdsEI1ISWvw94f27aFhQ2MkCK0VffrA0aPGKOTZSU5JpiSGbRrGoUvGvd2SnIQo2PJVgkpKSsLPzy/VFO5pp3QH8PHxoUIFmfPHVm7dMiYOrFvXSE7+/uDmBm+/Dd999xsrVhgz3maH1poR20bwvyP/Y9/5fTkTuBAiT8lXp/hiY2MZOHAgTk5O9O/fn06dOvHXX3+lKnN3UkKRu7Q27mH68ktYudIYBQLgoYfgzTdh6FBjJIiAgPubh+fD3R/y+eHPebflu7zT4h0rRi6EyKvyVYKyt7fHzc2NyMhIlixZwrfffptu3D0vLy98fHxsE2AhdOWKkZCWLYOUY/T6+hpJqUsXSDNjd7YtPLyQDwI+YGCDgTJduxCFSL5LUHevNyUmJqa79uTm5sYbb7whY7DlsKgoWL/euKnW3x9MJmN9iRLGjbVDhkCNGvfeR1ZprfE/7c8zNZ/hqy5fyd9WiEIkXyUoBwcH4uPjM9yelJTExIkT+e233xgzZgzNmjXLxegKtqgo2LoV1q6FDRvg/9u7++Cq6juP4+/vTQISSKSBMQhIoWPUAWx5CEqXQVCEwNqpMgXXWdDS0WJhnFXGtsqKdZd1nG21bLXjsKXAooBlnBHZzuoItUNAnK4lrMGHtSKrouADFQhPeSAP3/3jJOSBPFySG845uZ/XzJl77rm/3Hw5nNzv/Z3zO99feXmwPSsr6CXNmxc8XpTCe2XdHTPj+bnPU11bTVZGF7tiIhIrsRokkUgk2v0GXVlZSVVVFZs3b2by5MksWLDgwgXXA331FaxdGySegQPh1luD03nl5TBpEqxcGdxwu2ULzJ2b2uRU8lkJk/9jMp+f/JzMRCZ9svqk7s1FJBZi1YOC4DTe8ePH221jZmdP90nyamuhpAS2boVXXoE33mg8fWcWJKXZs4P7l7pzcuJ9R/Yxa+Ms+vXqh+Pd94tEJNJil6BycnLaTVC9e/dm8ODBFBcXM2zYsDbbSTDy7qOPoLg4SEqvvhpMqd4gKysY7DB7Ntx8Mwwa1P0xHTpxiBnrZ2AY2+ZvY3DO4O7/pSISSbFLUBdffDEHDx5s9bXs7GzGjh3Lyy+/TG5HlUbTkDu8/z7s2AE7dwaPhw41bzNiBMycCUVFcP31HRdsTaWjFUcp2lDE0YqjFC8opmBAikZaiEgsxS5B5eXltbo9OzubOXPmsHr1arK6Oq65h/jyS9i9O7g/affuYDlypHmbvDyYPBluvDFISpdfHpzOC0NFdQW9Mnqx5bYtjLt0XDhBiEhkxC5BDRgw4Jxtffr04aGHHmLp0qVpOQzZPegJvf02vPVWYzJqmI22qfx8mDIFrrsueBw5MqgkHqaauhoMY0juEEoWlpCwWI3dEZFuErsEdUmLaVWzs7NZt24dc+fODSmiC8c96AG9916QjN55p/GxrOzc9v36QWEhTJgQLNdcA8OGhddDak2d13Hn7++korqCTXM2KTmJyFmxS1CXXnopEAw5z8nJYevWrVx77bUhR5VaZWXwwQeNy759jeutJSIITtVdfXWwjB8fJKMrr2x/NtooeOAPD/Ds3mdZPnW5kpOINBO7BJWXl0cikWDo0KHs2LGD4cOHhx3Seamuhs8+C06/tbWcONH2z/frFySehmQ0enTwOGhQtHpGyXj89cd54k9PcM+Ee1h23bKwwxGRiIldgho1ahTTp09n06ZN9O/fP+xwgOD+oSNH4PDhoDbdF18EAxRaWz98uPHeorZkZweDFQoKzl3y8+OXiFqzrnQdP331p9w2+jaenPVkWl47FJH2xS5BTZs2jWnTpqX0Pd2D2V1PngyWEycaH48dC+4NOno0SEIN602XsrIpeJL3k5rBkCHBtaCmy2WXNa7n5fWMJNSegrwCbh11K8/c8oxO7YlIq1KaoMwsD1gDzAC+Apa6+3NttF0CPAD0AV4AFrl7VWttG9TUwIEDwVQOrS3l5W2/dvp0YwJqmYhOnuTsLK+d/JfTvz9ccklwqm3QoKCn0/SxYT0/v+vVvePs2JljAEwaNolJwyaFHI2IRFmqe1BPA2eAfGAM8JKZ7XX3d5s2MrMi4EHgBuAz4EXgn+u3tWnvXuiuS069egU3pebkNC65uUFvpuUyYEDz56WlxUybNrV7AutB3vryLe7YfQe/HPBLFo5fGHY4IhJx5smem+rojcz6AseA0e6+r37beuCQuz/You1zwMfu/o/1z6cBG9293WI6ZmO9d++tJBJV9csZEokqMjIqm6w3fa2y/nmwnplZTkZGBRkZ5WRknCYzs2G9nETi3Gnjk1VWVhaZ62FRVXFRBaXjSvE6Z9yb47ioKoWVZXuY0tJSampqKCwsDDuUWNDfX/Kiuq927Nixx93POeBT2YO6AqhtSE719gJTWmk7CvjPFu3yzWyAuzerdWBmC4GFAFlZWVx11YwuB+oenC6s6XxOaqa2tpaytsZ/C9W9q9k/YT+1VsuInSOoLK+kks7NrJsOampqcHcdU0nS31/y4ravUpmg+gEtq7geB3KSaNuwngM0S1DuvgpYBVBYWOglJSUpCTaViouLNYtvG6prq5m4ZiKZX2VSfHsxVdOrtK86MHXqVMrKyihtOkWxtEl/f8mL6r5qaxRvKhPUKaBladFc4GQSbRvWW2srMZaVkcWPxv+IoblD+fZl36b4/4rDDklEYiKV43v3AZlm1rQE9beAd1tp+279a03bfdny9J7EV21dLe8cfgeAH47/IbMKZoUckYjETcoSlLufBjYDy82sr5lNAm4G1rfS/FngTjMbaWZfA5YB61IVi4TL3Vn80mKu+e01fFz2cdjhiEhMpfoOycUE9zUdBn5HcG/Tu2Y2zMxOmdkwAHd/BfgFsB04UL88kuJYJCQ/2/4zVv3PKpZMXMLw/sPDDkdEYiql90G5+1Hglla2f0IwMKLpthXAilT+fgnfU288xaOvPcpdY+/i0RseDTscEYkx1ZiRlNl5YCf3vnIvs6+azcrvrFR9PRHpktjV4pPomnTZJH5V9CvuLrybzIQOLRHpGvWgpMv2fLaHgycOkpHI4N6J93JRpqpEiEjXKUFJl7z31/co2lDEHS/eEXYoItLDKEFJp316/FOKNhSRmchk9XdXhx2OiPQwulAgnXKk/AhFG4o4XnWcHQt28I2vfSPskESkh1GCkk75yR9+wofHPmTr/K2MGTQm7HBEpAdSgpJOWVG0gtu/eTtThrdWrF5EpOt0DUqSVud1PPnfT1JRXUH/i/pz/Yjrww5JRHowJShJiruz5JUl3Lf1Pl5474WwwxGRNKAEJUl57LXHeOrPT7Fk4hLmXT0v7HBEJA0oQUmHVu1ZxbLty5j/zfk8MeMJlTASkQtCCUradaLqBA9vf5hZl89i7XfXkjAdMiJyYWgUn7Qrt3cuu36wi8E5g8nKyAo7HBFJI/o6LK168/M3eey1x3B3CgYU0LdX37BDEpE0owQl59h/dD8zN87kN3t+w7HKY2GHIyJpSglKmvn85OfMWD+DOq9j2/xt5PXJCzskEUlTugYlZ5VVljFz40wOnz7M9u9v58qBV4YdkoikMSUoOev1T15n/9H9bPm7LUwYMiHscEQkzSlByVk3XXETH/7Dh+T3yw87FBERXYNKd+7O4pcWs+UvWwCUnEQkMpSg0tzSPy5lZclKSr8oDTsUEZFmlKDS2Io/reDnr/+cRYWLeGTKI2GHIyLSTEoSlJnlmdmLZnbazA6Y2d+30/b7ZrbHzE6Y2UEz+4WZ6VrYBbZ+73ru33Y/c0bO4dezfq36eiISOanqQT0NnAHygXnASjMb1UbbbOA+YCBwLTAN+HGK4pAk7f1yLzeMuIENszeQkcgIOxwRkXN0uediZn2B7wGj3f0UsMvMfg/cDjzYsr27r2zy9JCZbQQ0890FUud1JCzB49Mf50ztGXpn9g47JBGRVqXi1NoVQK2772uybS+Q7Fzg1wHvtvWimS0EFtY/PWVm73cqyu41EPgq7CBiQvsqOQPNTPspOTqmkhfVffX11jamIkH1A4632HYcyOnoB83sB0AhcFdbbdx9FbCqKwF2NzMrcffCsOOIA+2r5Gg/JU/7Knlx21cdXoMys2Iz8zaWXcApILfFj+UCJzt431uAfwVmuXsUM7qIiISowx6Uu09t7/X6a1CZZlbg7h/Ub/4W7Z+2mwn8FrjJ3d9OPlwREUkXXR7F5+6ngc3AcjPra2aTgJuB9a21N7MbgI3A99z9z139/RER6VOQEaN9lRztp+RpXyUvVvvK3L3rb2KWB6wFpgNHgAfd/bn614YB/wuMdPdPzGw7MBmobPIWr7n7rC4HIiIiPUZKEpSIiEiqqdSRiIhEkhKUiIhEkhJUiplZgZlVmtmGsGOJIjPrbWZr6ms2njSzN81M1x/rnU9dy3Sm46hz4vb5pASVek8Du8MOIsIygU8JKo1cDDwMPG9mw0OMKUrOp65lOtNx1Dmx+nxSgkohM7sNKAP+GHYsUeXup939n9z9Y3evc/f/Aj4CxocdW9ia1LV82N1PufsuoKGupTSh4+j8xfHzSQkqRcwsF1gO3B92LHFiZvkE9RzbvLE7jbRV11I9qA7oOGpfXD+flKBS51+ANe7+adiBxIWZZRHctP2Mu/8l7HgioNN1LdOZjqOkxPLzSQkqCR3VIzSzMcCNwL+FHWvYkqjd2NAuQVBt5AxwT2gBR0un6lqmMx1HHYvz55Nmsk1CEvUI7wOGA5/Uz0zbD8gws5HuPq7bA4yQjvYVgAU7aQ3BQIC/dffq7o4rJvZxnnUt05mOo6RNJaafT6okkQJmlk3zb74/JjggFrn7X0MJKsLM7N+BMcCN9ZNcSj0z2wQ4wRQ0Y4CXgb9xdyWpFnQcJSfOn0/qQaWAu5cD5Q3PzewUUBn1//wwmNnXgbuBKuCL+m90AHe7+8bQAouOxQR1LQ8T1LVcpOR0Lh1HyYvz55N6UCIiEkkaJCEiIpGkBCUiIpGkBCUiIpGkBCUiIpGkBCUiIpGkBCUiIpGkBCUiIpGkBCUiIpH0/6WYcxybgbCEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [1, 1], 'k--')\n",
    "plt.plot([0, 0], [-0.2, 1.2], 'k-')\n",
    "plt.plot([-5, 5], [-3/4, 7/4], 'g--')\n",
    "plt.plot(z, logit(z), \"b-\", linewidth=2)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.grid(True)\n",
    "plt.title(\"Sigmoid activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "\n",
    "save_fig(\"sigmoid_saturation_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Xavier and He Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Constant',\n",
       " 'GlorotNormal',\n",
       " 'GlorotUniform',\n",
       " 'Identity',\n",
       " 'Initializer',\n",
       " 'Ones',\n",
       " 'Orthogonal',\n",
       " 'RandomNormal',\n",
       " 'RandomUniform',\n",
       " 'TruncatedNormal',\n",
       " 'VarianceScaling',\n",
       " 'Zeros',\n",
       " 'constant',\n",
       " 'deserialize',\n",
       " 'get',\n",
       " 'glorot_normal',\n",
       " 'glorot_uniform',\n",
       " 'he_normal',\n",
       " 'he_uniform',\n",
       " 'identity',\n",
       " 'lecun_normal',\n",
       " 'lecun_uniform',\n",
       " 'ones',\n",
       " 'orthogonal',\n",
       " 'serialize',\n",
       " 'zeros']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[name for name in dir(keras.initializers) if not name.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7fc228455278>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=\"he_normal\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7fc25bba6e10>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',\n",
    "                                          distribution='uniform')\n",
    "keras.layers.Dense(10, activation=\"relu\", kernel_initializer=init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonsaturating Activation Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leaky ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def leaky_relu(z, alpha=0.01):\n",
    "    return np.maximum(alpha*z, z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure leaky_relu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de3hU1b3/8fcXAkIIBDhovFBEVEQRuRitl4rxUu9WBVQoVSlqUA9q+1NU1CqKF05FT1FUBFEsUgEFQcFyrNSgqK1GwVZaUEFQqSAKCYQQAsn6/bEGHUIuM5NM9lw+r+eZJ3tmdmZ/ZmdnvrP3Xnstc84hIiKSaJoEHUBERKQ6KlAiIpKQVKBERCQhqUCJiEhCUoESEZGEpAIlIiIJSQVK6mRmBWY2PugcqcDM8szMmVmHRljWajO7uRGW083M3jOzMjNbHe/lRZDHmdmAoHNI/alAJTkzm2Jm84LOEa1Q0XOhW7mZrTSzB81sryhfZ4iZldSxnD2Ka12/1xBqKBDvAvsB3zfgckaZ2SfVPHUM8ERDLacW9wGlQLfQMhtFLdv+fsCrjZVD4icj6ACS1p4Fbgea4z/Yng09PjKwRHHmnCsH1jXSsjY0xnKAQ4C5zrnVjbS8WjnnGmX9SvxpDyrFmVm2mU00s2/NbIuZLTKz3LDn/8vMXjCzr81sm5ktM7Nf1/Gap5lZkZkNM7O+ZrbDzPatMs/9ZvaPOuKVOufWOee+dM7NAv4CnFHldQ4ws+lmtil0m29mh0a5GmJiZmPMbEVovaw2s9+bWYsq85xrZn8PzfO9mb1qZi3MrAA4EHho155iaP4fDvGF/jbbzOz8Kq95Rmid7lNXDjMbAtwNdA/bIx0Sem63PTgz62RmL4e2gy1mNtvMOoY9P8rMPjGzgaE92i1mNqe2w5Gh99UTuCu07FFm1jk0nVt13l2H3sLm6W9mfzGzUjP7l5n9vMrvdDOzV8ys2MxKQocSe5jZKOAK4Nyw951XdTmh+z3M7I3Q+tsY2vPKDnt+ipnNM7MbzWxtaDt71swya3rf0jhUoFKYmRkwHzgAOA/oDbwF/NXM9gvN1gL4KPR8d2Ac8JSZnVbDa/YHXgbynXNPOefeAlYCl4fN0yR0f3IUWXsCJwI7wh7LBN4EyoCTgeOBb4A3GunDYyswFDgcuA4YCNwRlu8sYC6+sB4NnAIswv9f9QO+Bu7FH3Lajyqcc8XAPGBwlacGA687576NIMcM4GFgRdhyZlRdVmhbmAPkAKeGsu4PzAk9t0tn4FLgIvyXhd7A/TWsH0LLWxHKsB8wtpZ5q3M/8Ci+yH0ATDezrFDm/YHFgAN+DvQBHgeahpYzE3gj7H2/W837zgQWACXAsaH3dQLwTJVZTwKOBE7nx/d/Y5TvRRqac063JL4BU4B5NTx3Kv4fs2WVx5cCt9TymtOBp8PuFwDjgXygGDijyvw3A/8Ou382sB34r1qWUQCUh/Jtx38IVQD9w+YZCnwGWNhjTfHnby4J3R8ClNSxnPHVPF7r79XwWtcAn4fdfweYXsv8q4GbqzyWF3qvHUL3L8Cfv2kdut8S2AwMiiLHKOCT2paP/4CvADqHPd8FqAROD3udMiA7bJ47wpdVQ55PgFFh9zuH3mNulfkcMKDKPMPCnj8g9NjPQvfvB9YAzaPZ9qss5+rQNtu6mr/BIWGv8xWQETbPJOCNWP4ndWu4m/agUtvRQCawIXR4pMR8w4AjgYMBzKypmd1hZv8IHaIqwX/771TltS7Af3s9yzn3epXnngO6mNkJoftDgTnOuboaAswAeuH3jGYCk5w/1Bee/yBgS1j2YqDdrvzxZGYDzGyxma0LLft/2X299AYW1nMxr+EL1EWh+78ADL9nFmmOSBwO/MeFnSdyzq0C/gMcETbfGuf37Hb5D7BPlMuKRvhh4P+Efu5aXm9gsfPn7WJ1OPAP59yWsMfexRfm8Pf9L+fczipZ4vm+JQJqJJHamgDr8Ycvqtoc+nkzcBP+cMY/8Xs0D7DnP+c/8N86rzSzv7nQ10zwJ+PN7BVgqJmtwH/Ink/dip1znwOY2a+AZWY2xDk3JSz/Uvwhrao2RvD64N9ndjWPt8UXu2qZ2XH4Pcl7gN8CRfj3Fe0hrFo553aY2Yv4w3p/DP2c7ZwrbeAchv/7VRsjbHpHNc9F+0W2MmyZfsKsWQ3z/rA855wLHW3ctTyr9jei05jvWxqYClRq+wh/zqEy9G25Oj8DXnXOTYUfzlV0xX8QhvsCuB5/yGyimeWHFyn8IZGXgFX4ovhGNEFDH9QPAA+a2czQB/RHwCDgO+dc1TyRWgGcY2ZWJW+f0HM1ORFY65wbvesBMzuwyjxLgNPw77065fhDknV5HlhkZkcAZwHnRpkjkuX8CzjAzDrv2osysy7481D/iiBjNHa1Hgw/79Yrhtf5CPiVmTWvYS8q0vc91Mxah+1FnYAvPv+OIZM0In1DSA1tzKxXlVtnfJF4B5hrZmeb2UFmdryZ3WNmu/aqPgVOM7OfmVk3/Lmmg6pbSKjInYL/EJ1Y5eT6X/Dnhu4GnnXOVVbzEnX5E/6b6/DQ/Wn4YjfXzE4O5e9rZg/b7i35mlTz/o8MPfck/lzLY2bW08wOM7Pf4gtfbXshn+I/0AebWRczuzb0O+HuBy42s/vM7Agz625mvw1rwLEaOMl8S8QaW8I5597Bn2v5E/Ad8Ncoc6wGDjSzPuZbB1Z3LdkbwMfANDM72nwLu2n4IvDXauaPmXNuG/A34NbQOjmB2PY8nwCygJlmdoyZHWJmg8xsV7FbDRwZ+pt2qGEvbRq+kckfzbfm6ws8hd9L/TyGTNKIVKBSw0n4b/Pht7GhPYZz8B9Ak/B7DDOBw/jxeP99wPvAn/Et/Lbi/6mr5ZxbiT/JfBa+tZ+FHnf465ia8eP1TFEJfUseD9wS+sZbCvTF75W9CCzHn+9qB2wK+9WW1bz/gtBrrgq9xqHA66H3OhC42Dn3Wi1ZXgUeAv6AP7z5c+CuKvO8hj93dHZomYvwBXxXcb4L+Am+lWNd1yRNw7dke8E5VxFNDmAW/lzWwtByqhawXX+fC0PPF+BbR64DLqyyZ9lQhoZ+foAvCHdG+wLOubX4v11zfN4l+L34XeeKJuH3ggrx7+vEal6jFDgTaIP/288F3gvLJwnM4rNtSjoysyfxLaN+XufMIiJ10DkoqTfzFz0ejb/26ZKA44hIilCBkoYwF38R5GTn3Pygw4hIatAhPhERSUhqJCEiIgkpbof4OnTo4Dp37hyvl6+XrVu30qpVq6BjJC2tv9isWLGCiooKjjjiiLpnlj1ou4tdTevu22/hq6/ADLp1g8yAusf98MMPv3PO7V318bgVqM6dO1NYWBivl6+XgoIC8vLygo6RtLT+YpOXl0dRUVHC/l8kOm13satu3S1cCGee6aenT4dLAmzeZGZrqntch/hERNLMqlW+IFVUwMiRwRan2qhAiYikkZISuPBC2LgRzj0XRo+u+3eCogIlIpImnIMhQ+Cf/4TDDoNp06BpJL1FBkQFSkQkTdx/P8yaBW3awNy5kF1dP/8JRAVKRCQNzJ0Lv/udb7H3wgt+DyrRRVWgzOxQMyszs+fjFUhERBrW6tWZ/OpXfvqBB+Ccc4LNE6lo96Aex/dOLCIiSWDTJrjzziMpKYFLL4Vbbw06UeQiLlBmNhA/iF19h7gWEZFGUFEBAwfC2rWZ9OoFzzzjD/Eli4gu1DWzNsC9+NFDr6xlvnwgHyAnJ4eCgoIGiNjwSkpKEjZbMtD6i01RUREVFRVadzHSdhe9CRO68PrrnWjTZju33voR77+/PehIUYm0J4nR+J6qv7Jayq9zbiIwESA3N9cl6lXfuiK9frT+YtO2bVuKioq07mKk7S4606bBjBmQkQH33PMvBg48PuhIUauzQIWGVz4d6B3/OCIiUl8ffghXXeWnx42DI44oDjZQjCLZg8oDOgNfhvaesoCmZnaEc65P/KKJiEi01q/3PUWUlcHVV8O118KiRUGnik0kBWoiMD3s/s34gnVtPAKJiEhsysuhf3/4+ms44QQYPz65GkVUVWeBcs6VAqW77ptZCVDmnNsQz2AiIhKdG26Ad96BAw7wPUY0bx50ovqJergN59yoOOQQEZF6mDABnnoK9toL5syBffcNOlH9qasjEZEk9/bbcP31fnrSJMjNDTZPQ1GBEhFJYl9+6c877dwJN90El10WdKKGowIlIpKkSkt9i70NG+DnP4cxY4JO1LBUoEREkpBz/lqnJUvg4IP9sO0ZUbcqSGwqUCIiSeihh/ywGVlZfiiN9u2DTtTwVKBERJLMggVw221+eupU6N492DzxogIlIpJEPv3U91DuHIwa5c9BpSoVKBGRJLF5M1xwARQXw0UX+RFyU5kKlIhIEqishMGDYflyOPJIeO45aJLin+Ap/vZERFLDXXfBvHnQrp3vKaJ166ATxZ8KlIhIgnvxRbj/fr/HNHOmb1aeDlSgREQS2Mcfw5AhfnrsWDj99EDjNCoVKBGRBPXdd76VXmkpXH45/OY3QSdqXCpQIiIJaMcOuOQSWL0ajjnG91SezGM7xUIFSkQkAd10E7z5ph824+WXoUWLoBM1PhUoEZEE88wz8NhjfsDB2bP9AITpSAVKRCSB/O1vcO21fvqJJ+D444PNEyQVKBGRBPGf/0C/flBeDsOHw5VXBp0oWCpQIiIJoKzMd1/0zTeQlwePPBJ0ouCpQImIBMw5uOYaeP99OPBAfzFus2ZBpwqeCpSISMAefdT3rZeZ6bsx2nvvoBMlBhUoEZEALVzom5QDPPss9OoVbJ5EogIlIhKQVav8xbgVFXD77X5afqQCJSISgJISP7bTxo1w7rkwenTQiRKPCpSISCOrrPQdwH7yCRx2GEyblvpjO8VCq0REpJHdfz/MmgXZ2TB3rv8pe1KBEhFpRHPn+sEHzeBPf/J7UFI9FSgRkUaybBn86ld++sEH4Zxzgs2T6FSgREQawaZNfmynkhIYOBBuuSXoRIlPBUpEJM527vRF6fPPoXdvmDw5/cZ2ioUKlIhInI0cCa+/Dh06+LGdMjODTpQcVKBEROJo2jQYOxYyMuCll3xfexIZFSgRkTgpLISrrvLTjz4KJ58cbJ5kowIlIhIH69f74TPKyuDqq31v5RIdFSgRkQZWXg79+8PXX8OJJ8L48WoUEQsVKBGRBnb99fDOO3DAAf68U/PmQSdKTipQIiINaMIEmDgRWrTwYzvtu2/QiZKXCpSISAN56y2/9wQwaRLk5gabJ9mpQImINIAvv4QBA/xFuTfd9GOXRhK7iAqUmT1vZt+Y2WYz+9TMrop3MBGRZFFa6rsx2rABzjgDxowJOlFqiHQP6kGgs3OuDfAL4D4zOzp+sUREkoNzcOWVsGQJHHwwTJ/uL8qV+ouoQDnnljnntu+6G7odHLdUIiJJ4qGHfFHKyvJDabRrF3Si1BFxnTezJ4AhQEtgCfBaNfPkA/kAOTk5FBQUNEjIhlZSUpKw2ZKB1l9sioqKqKio0LqLUSJud3//e3tGjuwBGLfe+k82bPieBIsIJOa6i4Q55yKf2awpcDyQB/yPc25HTfPm5ua6wsLCegeMh4KCAvLy8oKOkbS0/mKTl5dHUVERS5cuDTpKUkq07e7TT+HYY6G4GO65xw9CmKgSbd1VZWYfOuf2aPMYVSs+51yFc24x0BG4tqHCiYgkk+JiuOAC/7NfP7jzzqATpaZYm5lnoHNQIpKGKit9E/Lly+HII+G556CJLtiJizpXq5ntY2YDzSzLzJqa2ZnAIOCv8Y8nIpJY7roL5s2D9u19o4isrKATpa5IGkk4/OG8CfiCtgb4jXNubjyDiYgkmhdfhPvv93tMM2ZAly5BJ0ptdRYo59wGQKOYiEha+/hjGDLETz/8MJx+eqBx0oKOnIqI1OG773yjiNJSuOIKuPHGoBOlBxUoEZFa7NgBF18Ma9b4ZuUTJmhsp8aiAiUiUoubboKCAj9sxuzZfhgNaRwqUCIiNXjmGXjsMT/g4OzZfgBCaTwqUCIi1XjvPbg21B3Bk0/C8ccHmycdqUCJiFSxdq3vIaK8HIYPh6FDg06UnlSgRETClJX54rRuHeTlwSOPBJ0ofalAiYiEOAfXXAPvvw8HHugvzG3WLOhU6UsFSkQkZNw437deZqbvxqhDh6ATpTcVKBER4I034Oab/fSUKdCzZ6BxBBUoERFWrYJLL4WKCrj9dn9hrgRPBUpE0lpJie/GaONGOO88GD066ESyiwqUiKStykq4/HL45BM47DB4/nmN7ZRI9KcQkbR1333w8suQne0bRWRnB51IwqlAiUhamjsX7r7bd/z6wgt+D0oSiwqUiKSdZcv8sO0ADz4IZ58dbB6pngqUiKSVjRt9o4iSEhg4EG65JehEUhMVKBFJGzt3wqBBsHIl9O4NkydrbKdEpgIlImlj5Eh4/XXYe2+YM8f3GCGJSwVKRNLC88/D2LGQkQEvvQSdOgWdSOqiAiUiKa+wEK66yk8/+ij07RtsHomMCpSIpLR16+Cii2D7dsjP972VS3JQgRKRlFVeDgMGwNdfw4kn+uHb1SgieahAiUhKcs6PhvvOO9CxI8yaBc2bB51KoqECJSIpacIEmDQJWrTw3Rnl5ASdSKKlAiUiKeett+CGG/z0pEmQmxtsHomNCpSIpJQ1a/x5p507/QCEu7o0kuSjAiUiKaO01LfY27ABzjgDxowJOpHUhwqUiKQE5+DKK2HJEjjkEJg+HZo2DTqV1IcKlIikhN//3helrCzfjVG7dkEnkvpSgRKRpPfaa76fPfBdGnXvHmweaRgqUCKS1FasgF/+0h/iu/deP5SGpAYVKBFJWsXFviAVF0O/fnDHHUEnkoakAiUiSamiAgYP9ntQRx4Jzz0HTfSJllL05xSRpHTXXTB/PrRvD3Pn+sYRklpUoEQk6cycCQ884JuRz5wJXboEnUjiQQVKRJLKxx/Dr3/tp8eOhdNOCzaPxI8KlIgkje++840iSkvhiivgxhuDTiTxpAIlIklh507j4ot9X3vHHut7K9fYTqmtzgJlZnuZ2WQzW2NmW8xsiZmd3RjhRER2eeKJgykogH339cNntGgRdCKJt0j2oDKAr4CTgWzgd8BMM+scv1giIj+aPBlefrkjzZvD7Nmw//5BJ5LGkFHXDM65rcCosIfmmdkXwNHA6vjEEhHx3nsPrr3WTz/5JBx/fLB5pPHUWaCqMrMcoCuwrJrn8oF8gJycHAoKCuqbLy5KSkoSNlsy0PqLTVFRERUVFVp3UdiwoTnXXHM0O3bsxXnnfUGXLmvQ6otesv7PRlWgzKwZMA14zjm3vOrzzrmJwESA3Nxcl5eX1xAZG1xBQQGJmi0ZaP3Fpm3bthQVFWndRaisDPr2hY0b4ZRT4MYbv9S6i1Gy/s9G3IrPzJoAU4FyYHjcEolI2nMOhg2DDz6Azp39xbgZGS7oWNLIItqDMjMDJgM5wDnOuR1xTSUiaW3cOPjjHyEz04/t1KFD0IkkCJEe4nsSOBw43Tm3LY55RCTNvfEG3HSTn54yBXr2DDSOBCiS66AOBIYBvYB1ZlYSug2OezoRSSsrV8Ill0BlpR864+KLg04kQYqkmfkaQNdri0hclZTAhRfCpk1w3nl+8EFJb+rqSEQCV1kJl18On3wC3br5Yds1tpNoExCRwN13n+++KDvbj+2UnR10IkkEKlAiEqg5c+Duu33Hr9OnQ9euQSeSRKECJSKBWbYMLrvMT48ZA2edFWweSSwqUCISiI0b/dhOJSUwaBCMGBF0Ikk0KlAi0uh27oSBA32z8t694emnNbaT7EkFSkQa3W23wV/+Anvv7c9BZWYGnUgSkQqUiDSqqVPh4YchIwNmzYJOnYJOJIlKBUpEGk1hIVx9tZ9+7DE46aRg80hiU4ESkUaxbp3vKWL7dsjPh2uuCTqRJDoVKBGJu+3boX9/WLsWTjzR7z2J1EUFSkTiyjm4/np4913o2NGfd2rePOhUkgxUoEQkriZMgEmToEUL351RTk7QiSRZqECJSNwsWgQ33OCnn34acnODzSPJRQVKROJizRoYMMBflHvzzTBYI8hJlFSgRKTBlZb6FnvffQdnnOH72ROJlgqUiDQo52DoUFi6FA45xPdQ3rRp0KkkGalAiUiD+v3vYcYMyMryYzu1axd0IklWKlAi0mBeew1GjvTT06bBEUcEm0eSmwqUiDSIFSv8sBnOwb33wi9+EXQiSXYqUCJSb8XFfmynzZt9jxF33BF0IkkFKlAiUi8VFb4J+YoV0KMHTJkCTfTJIg1Am5GI1Mtdd8H8+dC+vR/bKSsr6ESSKlSgRCRmM2fCAw/4ZuQzZ0KXLkEnklSiAiUiMVm6FH79az/98MNw2mnB5pHUowIlIlHbsMH3FFFaCkOG/NjfnkhDUoESkajs2AEXX+z72jv2WHjySTALOpWkIhUoEYnK//t/vpfy/fbzw2e0aBF0IklVKlAiErHJk2H8eD/g4OzZsP/+QSeSVKYCJSIRefdduPZaPz1hAhx3XLB5JPWpQIlInb7+Gvr18+efbrjhx9Z7IvGkAiUitSor88Vp/Xo45RQYOzboRJIuVKBEpEbOQX4+fPABdO7sL8Zt1izoVJIuVKBEpEZ/+ANMnQqZmX5spw4dgk4k6UQFSkSq9cYbcPPNfnrKFDjqqEDjSBpSgRKRPaxcCZdcApWVfuiMiy8OOpGkIxUoEdnNli1+bKdNm+D88/3ggyJBUIESkR9UVsIVV8CyZXD44fD88xrbSYIT0aZnZsPNrNDMtpvZlDhnEpGAjB7tuy/KzvZjO7VpE3QiSWcZEc73H+A+4EygZfziiEhQ5syBUaP8HtP06dC1a9CJJN1FVKCcc7MBzCwX6BjXRCLS6JYtg8su89MPPghnnRVsHhHQOSiRtLdxo28UUVICgwbBiBFBJxLxIj3EFxEzywfyAXJycigoKGjIl28wJSUlCZstGWj9xaaoqIiKioqEWncVFcZtt/Vg5cr2HHroFi6/fAmLFlUGHata2u5il6zrrkELlHNuIjARIDc31+Xl5TXkyzeYgoICEjVbMtD6i03btm0pKipKqHV3001QWAh77w1vvNGaTp36Bh2pRtruYpes606H+ETS1NSp8MgjkJEBs2ZBp05BJxLZXUR7UGaWEZq3KdDUzFoAO51zO+MZTkTi44MP4Oqr/fRjj8FJJwWbR6Q6ke5B3QlsA24DfhWavjNeoUQkftatg4sugu3bYdgwuOaaoBOJVC/SZuajgFFxTSIicbd9O/TvD2vXws9+Bo8+GnQikZrpHJRImnAOhg/3Q7d37AgvvQTNmwedSqRmKlAiaeLJJ+Hpp6FFC99rRE5O0IlEaqcCJZIGFi2CG2/005Mnw9FHB5tHJBIqUCIpbs0aGDAAdu70vUT88pdBJxKJjAqUSAorLYULL4TvvoMzz/T97IkkCxUokRTlHAwdCkuXwiGHwAsvQNOmQacSiZwKlEiK+p//gRkzICsL5s6Fdu2CTiQSHRUokRQ0fz7cfrufnjYNjjgi2DwisVCBaiR5eXkMHz486BiSBlas8A0hnPMj5P7iF0EnEomNClTIkCFDOO+884KOIVIvxcV+bKfNm32PEXfcEXQikdipQImkiIoKGDzY70H16AFTpoBZ0KlEYqcCFYHi4mLy8/PZZ599aN26NSeffDKFhYU/PP/9998zaNAgOnbsSMuWLenevTvPPvtsra+5cOFC2rZty1NPPRXv+JImfvc7f+6pfXvfKCIrK+hEIvWjAlUH5xznnnsua9euZd68eSxZsoS+ffty6qmn8s033wBQVlZGnz59mDdvHsuWLePGG29k2LBhLFy4sNrXnDVrFhdddBETJ05k2LBhjfl2JEXNmOGvcWraFGbOhIMOCjqRSP016Ii6qejNN99k6dKlbNiwgZYtWwIwevRoXn31VaZOncott9zCAQccwIgRI374nfz8fP7617/ywgsvcNppp+32ehMnTmTEiBG89NJLnHHGGY36XiQ1LV0Kv/61n37kEaiyyYkkLRWoOnz44YeUlpay99577/Z4WVkZK1euBKCiooIxY8YwY8YM1q5dy/bt2ykvL99jiOW5c+fy1FNP8dZbb3H88cc31luQFLZhg28UsW0bDBkC118fdCKRhqMCVYfKykpycnJ4++2393iuTZs2AIwdO5aHH36YcePG0aNHD7Kysrj99tv59ttvd5v/qKOOwsyYPHkyxx13HKYz2FIPO3bAxRfDl1/CT3/qeyvXJiWpRAWqDn369GH9+vU0adKELl26VDvP4sWLOf/887nssssAf97q008/pW3btrvNd9BBB/HYY4+Rl5dHfn4+EydOVJGSmP32t76X8v32g9mz/TAaIqlEjSTCbN68maVLl+52O+SQQzjxxBO54IIL+POf/8wXX3zBe++9x9133/3DXlXXrl1ZuHAhixcvZvny5QwfPpwvvvii2mV06dKFN998kwULFpCfn49zrjHfoqSIp5+Gxx/3Aw7Ong377x90IpGGpwIV5u2336Z379673UaMGMFrr73GqaeeytVXX81hhx3GJZdcwooVK9g/9Klw5513cuyxx3L22WfTt29fWrVqxeDBg2tczsEHH0xBQQELFixg2LBhKlISlXffheuu89MTJsBxxwWbRyRedIgvZMqUKUyZMqXG58eNG8e4ceOqfa5du3bMnj271tcvKCjY7f7BBx/MV199FW1MSXNffw39+vnzTzfc8GPrPZFUpD0okSSxbRtcdBGsXw+nngpjxwadSCS+VKBEkoBzMGwYFBZC587+wtxmzYJOJRJfKlAiSeAPf4CpUyEz03dj1KFD0IlE4i/lC1RhYSGzZs0KOoZIzP7yF7j5Zj/93HNw1FHB5hFpLCnbSKKyspIxY8Zw3333AdCxY0d++tOfBpxKJDorV8Kll0JlJdx5JwwYEHQikcaTkgVq3bp19O/fn6VLl7Jt2zYALrjgAlasWEF2dnbA6UQis2WL78Zo0yY4/3y4556gE4k0rpQ7xLdgwQK6devG+++/T2lp6Q+PFxUVMWTIkOCCiUShshIuvxyWLYPDD4fnn/RoRTIAAApOSURBVIcmKfffKlK7lNnky8vLueGGG+jXrx/FxcXs3Llzt+ebNGnCypUrdVGsJIXRo2HOHGjb1jeKCHX7KJJWUqJAffbZZ/Ts2ZOnn376h0N64Vq2bMlVV11FYWGh+r6ThPfyyzBqlN9jeuEFOPTQoBOJBCPpz0E999xzXHfddWzbtm2PvaOMjAxatWrF9OnTOeusswJKKBK5Tz7xh/YAxowBbbaSzpK2QG3ZsoUhQ4awYMGC3c417ZKZmUmvXr2YNWsW++67bwAJRaKzcaNvFFFSAoMG/di0XCRdJeUhvsLCQrp168b8+fOrLU4tW7bkjjvu4O2331ZxkqSwcycMHAirVkGfPr63ch2NlnSXVHtQlZWVPPTQQ9xzzz3Vnmvaa6+9aNeuHa+88grHHHNMAAlFYnPrrf6C3H328eegMjODTiQSvKQpUOvXr2fAgAF89NFH1RanzMxMzjzzTKZMmfLDSLciyeCPf4RHHoGMDHjpJejUKehEIokhKQrU//3f/zFw4EC2bt3Kjh07dnvOzGjZsiWPP/44V1xxhVrpSVL54APIz/fT48fDSScFm0ckkSR0gSovL2fEiBFMmjSpxubjnTp14pVXXqFr164BJBSJ3bp1fviM7dt9T+XDhgWdSCSxBNpIoqysjMLCwmqfW7lyJb1796712qahQ4fy8ccfqzhJ0tm+Hfr3h7Vr4Wc/g0cfDTqRSOIJtECNGTOG4447jo8++mi3x6dOnUrPnj1Zvnz5Hq30MjIyaNOmDS+++CLjx49nr732aszIIvXmHPz3f/uh23/yE3/eqXnzoFOJJJ7ADvFt2rSJsWPHUlFRwfnnn8+KFSsAuPLKK5k3b16N1zb17NmTWbNmsd9++zV2ZJEG8cQTMHkytGjhW+zl5ASdSCQxRbQHZWbtzexlM9tqZmvM7Jf1XfADDzxARUUFABs3bqRfv35069aNV155pcZrm0aOHMnixYtVnCRplZRk8Jvf+OnJk+Hoo4PNI5LIIt2DehwoB3KAXsB8M/vYObcsloV+++23PP7445SVlQH+XNTixYtrvLapbdu2zJ07V+M5SVIrKoLVq1tRUQEjRsAv6/01TyS1WV29e5tZK2ATcKRz7tPQY1OBtc6522r6vdatW7uja/h6+Nlnn/HNN9/U2bN4kyZNaNeuHd26dSMjo+GORhYVFdG2bdsGe710o/W3p8pK3xtETbetW+Hbb5cC0L59L448Uj1FREvbXewSfd0tWrToQ+dcbtXHI/nU7wpU7CpOIR8DJ1ed0czygXyAZs2aUVRUtMeLlZeXR1SczIz999+f9u3bU1JSEkHMyFVUVFSbTSKTiuuvstKoqIj9FqlmzSrp2LGI4uI4vpkUlYrbXWNJ1nUXSYHKAqr+OxUDravO6JybCEwEyM3NddU1IR8yZAiff/75HhfchmvXrh3vvfcehx12WATxoldQUEBeXl5cXjsdJNr6q6iAzZv9IbSiIigu/nG6uvtVHysu9ntA9dGiBWRn+/Gbwm+7HmvXDmbPzqO8vIilS5c2zBtPM4m23SWTRF93NXWwEEmBKgGq9h3UBtgSbYhVq1YxY8aMWosT+HNSq1evjluBksSyY0f0RSX8sc2b65+hVavqC0tN98Mfy872BaouCxZAeXn9s4qki0gK1KdAhpkd6pz7LPRYTyDqBhK33XZbncUJYNu2bQwcOJDly5eToza4Ca+sLPa9l6IiqKbRZtSys2svIrUVmjZtoFmz+mcQkYZVZ4Fyzm01s9nAvWZ2Fb4V3wXACdEs6N///jevvvrqD03L67JlyxaGDRvGnDlzolmMRMk5XyAi3VspKoIvv+xDZeWP97dvr1+GJk0i31up7n7r1tC0acOsDxFJHJE2jbsOeAb4FvgeuDbaJuYjRoygvJrjG02bNqVVq1ZUVFRQXl5Ox44dOeqoozj22GM57bTTollEWqqshC1boj8sFn4/wu8MYXY/4tusmT/HEs1hsfBbq1Zq0SYie4qoQDnnNgIXxrqQf/7zn8yfP5+srCyccz8Uoh49enDMMcfQo0cPunfvzkEHHUTTNPsqvHPn7if4oz1UVlzs94Lqo2XL6A6LrVr1Eaec0ueH+y1aqMCISMNrlK6O2rRpwwMPPMDhhx9O9+7d6dKlS8oUovLy6PZWqt5viBb0rVvHfv4lOzv6fuAKCjZz+OH1zy0iUptGKVAHHnggI0eObIxFRcW53U/wx1Joqun8IipmuxeOSA+L7XqsTRs/0J2ISKpJ6o825/weSLSHxb755li2b/f3I2hUWKuMjOibJYffsrJ8IwEREdldoAWqsrJ+51+KimK9wDLzh6nmzf0J/liuf2nbFjIzdf5FRCQe4lag1q+Hu+6qvdA01AWW0R4WW7Hi75x55k8jvsBSREQaX9wK1Ndfw+jRdc/Xpk3s51+ys2O7wHLbtm0ag0dEJMHFrUDtsw9cd13thUYXWIqISE3iVqB+8hO4++54vbqIiKQ6tR8TEZGEpAIlIiIJSQVKREQSkgqUiIgkJBUoERFJSCpQIiKSkFSgREQkIalAiYhIQlKBEhGRhKQCJSIiCclcfccLr+mFzTYAa+Ly4vXXAfgu6BBJTOsvdlp3sdO6i12ir7sDnXN7V30wbgUqkZlZoXMuN+gcyUrrL3Zad7HTuotdsq47HeITEZGEpAIlIiIJKV0L1MSgAyQ5rb/Yad3FTusudkm57tLyHJSIiCS+dN2DEhGRBKcCJSIiCUkFSkREEpIKFGBmh5pZmZk9H3SWZGBme5nZZDNbY2ZbzGyJmZ0ddK5EZmbtzexlM9saWm+/DDpTMtC21jCS9TNOBcp7HPgg6BBJJAP4CjgZyAZ+B8w0s84BZkp0jwPlQA4wGHjSzLoHGykpaFtrGEn5GZf2BcrMBgJFwMKgsyQL59xW59wo59xq51ylc24e8AVwdNDZEpGZtQL6A79zzpU45xYDrwCXBZss8Wlbq79k/oxL6wJlZm2Ae4Gbgs6SzMwsB+gKLAs6S4LqClQ45z4Ne+xjQHtQUdK2Fp1k/4xL6wIFjAYmO+e+CjpIsjKzZsA04Dnn3PKg8ySoLKC4ymPFQOsAsiQtbWsxSerPuJQtUGZWYGauhttiM+sFnA78b9BZE01d6y5svibAVPy5leGBBU58JUCbKo+1AbYEkCUpaVuLXip8xmUEHSBenHN5tT1vZr8BOgNfmhn4b7lNzewI51yfuAdMYHWtOwDzK20y/qT/Oc65HfHOlcQ+BTLM7FDn3Gehx3qiw1QR0bYWszyS/DMubbs6MrNMdv9WezP+j3mtc25DIKGSiJlNAHoBpzvnSoLOk+jMbDrggKvw6+014ATnnIpUHbStxSYVPuNSdg+qLs65UqB0130zKwHKkuUPFyQzOxAYBmwH1oW+nQEMc85NCyxYYrsOeAb4Fvge/yGh4lQHbWuxS4XPuLTdgxIRkcSWso0kREQkualAiYhIQlKBEhGRhKQCJSIiCUkFSkREEpIKlIiIJCQVKBERSUgqUCIikpD+PxXjvHObzzSZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, leaky_relu(z, 0.05), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([0, 0], [-0.5, 4.2], 'k-')\n",
    "plt.grid(True)\n",
    "props = dict(facecolor='black', shrink=0.1)\n",
    "plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props, fontsize=14, ha=\"center\")\n",
    "plt.title(\"Leaky ReLU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.5, 4.2])\n",
    "\n",
    "save_fig(\"leaky_relu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['deserialize',\n",
       " 'elu',\n",
       " 'exponential',\n",
       " 'get',\n",
       " 'hard_sigmoid',\n",
       " 'linear',\n",
       " 'relu',\n",
       " 'selu',\n",
       " 'serialize',\n",
       " 'sigmoid',\n",
       " 'softmax',\n",
       " 'softplus',\n",
       " 'softsign',\n",
       " 'tanh']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.activations) if not m.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[m for m in dir(keras.layers) if \"relu\" in m.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a neural network on Fashion MNIST using the Leaky ReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full / 255.0\n",
    "X_test = X_test / 255.0\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.LeakyReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 1.2810 - accuracy: 0.6205 - val_loss: 0.8869 - val_accuracy: 0.7160\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.7952 - accuracy: 0.7369 - val_loss: 0.7132 - val_accuracy: 0.7626\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.6817 - accuracy: 0.7726 - val_loss: 0.6385 - val_accuracy: 0.7894\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 2s 37us/sample - loss: 0.6219 - accuracy: 0.7942 - val_loss: 0.5931 - val_accuracy: 0.8016\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.5830 - accuracy: 0.8074 - val_loss: 0.5607 - val_accuracy: 0.8170\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.5552 - accuracy: 0.8172 - val_loss: 0.5355 - val_accuracy: 0.8238\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5339 - accuracy: 0.8226 - val_loss: 0.5166 - val_accuracy: 0.8298\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 2s 43us/sample - loss: 0.5173 - accuracy: 0.8262 - val_loss: 0.5043 - val_accuracy: 0.8356\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.5039 - accuracy: 0.8306 - val_loss: 0.4889 - val_accuracy: 0.8384\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.4923 - accuracy: 0.8333 - val_loss: 0.4816 - val_accuracy: 0.8394\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try PReLU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(100, kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.PReLU(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 1.3452 - accuracy: 0.6203 - val_loss: 0.9241 - val_accuracy: 0.7170\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.8196 - accuracy: 0.7364 - val_loss: 0.7314 - val_accuracy: 0.7600\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.6970 - accuracy: 0.7701 - val_loss: 0.6517 - val_accuracy: 0.7880\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.6333 - accuracy: 0.7914 - val_loss: 0.6032 - val_accuracy: 0.8050\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.5916 - accuracy: 0.8049 - val_loss: 0.5689 - val_accuracy: 0.8162\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5619 - accuracy: 0.8143 - val_loss: 0.5416 - val_accuracy: 0.8222\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5391 - accuracy: 0.8208 - val_loss: 0.5213 - val_accuracy: 0.8300\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.5214 - accuracy: 0.8258 - val_loss: 0.5075 - val_accuracy: 0.8348\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 2s 42us/sample - loss: 0.5070 - accuracy: 0.8287 - val_loss: 0.4917 - val_accuracy: 0.8380\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.4946 - accuracy: 0.8322 - val_loss: 0.4839 - val_accuracy: 0.8378\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elu(z, alpha=1):\n",
    "    return np.where(z < 0, alpha * (np.exp(z) - 1), z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure elu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZgU1d328e8PBtlBEB0XRIwK0RAhYZInatSJ4VEgGI0a3CMaA4HwKlETlRd9fA2PRoMJRgXFaIiAC+IKsri2iBKVZQiggCCyiLI3MGzDzJz3j9ODQ8/aTM1U9fT9ua6+pqequ+rXZ2r67qo6fcqcc4iIiERNg7ALEBERKY8CSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUpB0zG2tmU+rRehqY2WNmttnMnJnl1vY6K6mlTl5zYl1tzGy9mZ1QF+tLlZlNMrObwq4jk5lGkqjfzGwscE05sz50zv0oMb+dc65PBc+PAYucc4OTpvcDHnbOtQi04OqtuzV+242n03oqWX8f4EUgF/gc2OKcK6jNdSbWGyPpddfVa06s6y/4be/a2l5XOes+C7gF6A4cDVzrnBub9JjvAu8CxzvnttV1jQJZYRcgdeJN4OqkabX+Blhb6urNog7flE4EvnLOfVBH66tQXb1mM2sGXA+cXxfrK0cLYBHwVOJWhnNuoZl9DlwFPFKHtUmCDvFlhr3Oua+Tbltqe6Vm1tPM3jOzrWa2xcxmmNnJpeabmd1sZp+Z2V4zW2tm9ybmjQXOBn6XOOzlzKxjyTwzm2JmAxKHiLKS1vu0mb1SnTqqs55Sy2lsZiMT69xjZv82sx+Xmh8zs1Fmdo+ZbTKzDWY2wswq/D9LrP9vQIfEur8otayHkx9bUk911nUw7Zvqaz7Y1w30BoqB98tpk+5m9paZ7Taz5WZ2lpn1NbMyjz1YzrmpzrmhzrlJiToq8ipweVDrldQooKQ2NQdGAj/EH77aBkw2s0MS8+8B7gDuBb4D/BJYk5h3IzAb+CdwVOJWMq/EROBQoEfJBDNrDlwAjK9mHdVZT4n7gUuB64DvAQuB6WZ2VKnHXAkUAqcDg4EhiedU5EbgbmBtYt0/qOSxyapaV03bF6r3mqtTS7Izgbku6RyDmf0AeA94BzgV+Dfw/4D/m3gtJD1+qJnlV3E7s5I6qvIR8EMza1qDZchB0iG+zNDTzPKTpj3inLu1NlfqnHuh9O9mdi2wHf8Pnwf8HhjinHsy8ZDl+DdNnHPbzKwA2OWc+7qC5W81s6n4N8fpicm/wL9RTq5OHc65WVWtJ/Gc5sBA4Hrn3GuJab8FzgF+BwxLPPQT59ydifvLzOw3wE+BZyp4DdvMbAdQVNn6K1DhusysBQfRvmZ2MK855dcNHAd8Vc70B4DJzrnhifU9jf9bznTOvV3O4x/Ff1CpzJdVzK/MOqAR/jzVihosRw6CAiozzAT6J02ri5PgJwB/Av4LOBy/x94A6IA/B9YYeKuGqxkPjDWzZs65XfiwmuSc21PNOqrrBPwb1f7DTM65IjObDZxS6nH/SXreOuCIFNaTisrWdQo1b9/qvuaqailPU2B96QlmdiR+z+onpSYX4P9WZfaeEvVsAWrzcPXuxE/tQYVAAZUZdjnnlh/kc7cDrcuZfij+UFllJuM/vQ5I/CwEPgEOAayS56ViSmK5F5jZW/jDfeemUEd1ldRbXrfX0tP2lTPvYA6lF1O2jRol/V7ZuoJo3+q+5qpqKc8moE3StJLzkx+XmtYZWOqcm1VugWZDgaGVrAegl3PuvSoeU5G2iZ8bD/L5UgMKKKnKUqC3mVnS+YLvJ+aVy8wOw7/h/M45905i2vf5Zpv7BNiLPwz0WQWLKQAaVlacc26vmU3C7zm1A77Gdw2ubh3VWg/+8FgB8GN8V3DMrCFwGvB0Fc89GBvx54VK6wp8Uc3nB9G+tfma5wP9kqYdig+24sS6WuLPPVV26LO2D/F1AdY559ZX+UgJnAIqMzROHD4prcg5V/KpsJWZdUuaH3fOfQGMxp/0fsjMHgf24HtgXY7vjFCRrfhPyb8xszXAMcBf8HsvOOd2mNmDwL1mthd/GPIwoLtzbnRiGV/gz1d1BPLx3w8qr8fVeHxX+uOBp5MeU2kd1V2Pc26nmY0G/mxmm4CV+HM82cCoStrhYL0NjDSzn+M/CAwAjqWaAXWw7Zu0jNp8zTOA+8zsMOfc5sS0PPxe2+1mNgH/d/oKONHMTnLOlQnagz3ElzhHd2Li1wb4XpTd8H/71aUeeibfnN+UOqZefJmhB/4fvfRtfqn5ZyZ+L30bAeCc+xw4CzgJeB3fq+ky4JfOuakVrTDxBn8pvifWIvz3SO7Af6ovcTtwX2L6p8ALQPtS80fgP8F/gt+jqOic0Uz8p+RTOLD3XnXrqO56bsV/Wv8n/s30VKCnc668k/019WSp2/v4AHkpxWUE0b618pqdcwv5ZlsqmbYSv8c0EFgA7MBvu4uAoL8jlsM323pTfE/B+fgelQCYWRN8p5vHA163VJNGkhCRUJhZT+BB4BTnXFHY9SQzs98BFzjnks9pSh3RHpSIhMI5Nx2/R9u+qseGZB/wf8IuIpNpD0pERCJJe1AiIhJJCigREYmk0LuZt2vXznXs2DHsMsrYuXMnzZs3D7uMtKN2S83SpUspKirilFOSB2aQyqTTduYcLF8O27fDIYfAt78NjZK/cl0Hotxmc+fO3eScOzx5eugB1bFjR+bMmRN2GWXEYjFyc3PDLiPtqN1Sk5ubSzwej+T/QJSly3ZWXAxXXAHz5sERR8CsWXDSSeHUEuU2M7NV5U3XIT4RkVrgHNx4Izz3HLRsCdOmhRdO6UoBJSJSC4YPh4cf9of1XnkFvv/9sCtKPwooEZGAPfoo3HknNGgAzzwDP/lJ1c+RsgINKDMbb2Zfmdl2M1tmZtcHuXwRkaibNAkGDfL3R4+Giy4Kt550FvQe1L1AR+dcK+DnwHAz6x7wOkREIumtt+DKK/35p+HDoX/yVdgkJYEGlHNusXOuZBBOl7idEOQ6RESiaO5cuPBCKCiAG26AoVVdpUqqFHg3czMbhb/OS1P86MBlRrw2s/4krvCanZ1NLBYLuoway8/Pj2RdUad2S008HqeoqEhtlqKobWdr1jTlhhu+R37+IZxzznouuOBT3n236ufVpai1WXXUylh8pS5qlgvc55xLvtrmfjk5OS6K3wGJ8ncGokztlpqS70Hl5eWFXUpaidJ2tm4dnH46rFoF550Hr77qe+5FTZTaLJmZzXXO5SRPr5VefM65osQlmtvjr+0iIlLvbN3qQ2nVKviv/4IXXohmOKWr2u5mnoXOQYlIPbRrF5x/PixaBCefDK+9BhEdSShtBRZQZnaEmV1mZi3MrKGZnYe/LPjbQa1DRCQK9u2Dvn3h/fehfXuYMQMOOyzsquqfIDtJOPzhvEfxwbcKGOKceyXAdYiIhKq4GK6/3u8xtW0Lr78Oxx4bdlX1U2AB5ZzbCJwd1PJERKLo1lvhqaegWTOYOtUf3pPaoaGORESq6S9/gREjICsLXnzRd4yQ2qOAEhGphn/+E/74R3//qad87z2pXQooEZEqvPoq/OY3/v6DD8Lll4dbT6ZQQImIVOK99+DSS6GoCIYN88MYSd1QQImIVOA///Hfddqzxw/8evfdYVeUWRRQIiLlWLnSn2fats1fMmPUKDALu6rMooASEUmyfj2cey58/bW/2OCECdCwYdhVZR4FlIhIKdu3Q69esHw5fO978PLL0KRJ2FVlJgWUiEjCnj3+mk7z58OJJ8K0adCqVdhVZS4FlIgIvpfelVfCO+/AkUf6IYyys8OuKrMpoEQk4zkHgwb50SFat/aDvx5/fNhViQJKRDLenXfCmDH+XNPkyXDqqWFXJKCAEpEM9/e/w/DhvpfexIlw5plhVyQlFFAikrGefhpuvNHf/8c//JdyJToUUCKSkaZPh2uu8ffvvx/69Qu1HCmHAkpEMs6HH8LFF0NhIdxyC/zhD2FXJOVRQIlIRvn0U+jdG3bt8ntQ990XdkVSEQWUiGSMNWv8EEZbtkCfPvD449BA74KRpT+NiGSEzZt9OK1dC2ecAc89B40ahV2VVEYBJSL1Xn4+/OxnsGQJdOniv+vUrFnYVUlVFFAiUq8VFMAll/iOEccd50eJaNMm7KqkOhRQIlJvFRf77uMzZsDhh/vx9Y4+OuyqpLoUUCJSLzkHQ4bAM89AixZ+ZPJOncKuSlKhgBKReumee+Chh+CQQ+CVV6B797ArklQpoESk3hkzBoYN85donzABzjkn7IrkYCigRKReeeEFGDjQ3x81yneQkPSkgBKReuOdd+CKK3zniLvvht/+NuyKpCYUUCJSL8ybBxdc4LuVDx7sD/FJelNAiUja++wz6NkTduyAyy6DBx/0558kvSmgRCStrVvnhzDauNH//Ne/NL5efaE/o4ikrXjc7zl98QX88Ie+g8Qhh4RdlQRFASUiaWn3bn8F3IULoXNneO01/4VcqT8CCygza2xmT5jZKjPbYWbzzaxXUMsXESlRVGRceinMmgXHHOOHMGrXLuyqJGhB7kFlAWuAs4HWwB3ARDPrGOA6RCTDOQcjRnRi8mQ/6Ovrr0OHDmFXJbUhK6gFOed2AneVmjTFzFYC3YEvglqPiGS2226D6dOPolkzf1jvlFPCrkhqS62dgzKzbKATsLi21iEimWXECLj/fmjYsJhJk+C008KuSGpTYHtQpZlZI2AC8C/n3JJy5vcH+gNkZ2cTi8Vqo4wayc/Pj2RdUad2S008HqeoqEhtVg3Tp2dz330nAzBkyAKaNt2Gmq360vF/05xzwS7QrAHwNNAKuMA5t6+yx+fk5Lg5c+YEWkMQYrEYubm5YZeRdtRuqcnNzSUej5OXlxd2KZE2ZQpceCEUFcHIkdC1q7azVEX5f9PM5jrncpKnB3qIz8wMeALIBi6uKpxERKoyaxb88pc+nIYOhRtvDLsiqStBH+IbDZwM9HDO7Q542SKSYRYu9N912rMHrr8ehg8PuyKpS0F+D+o4YADQDfjazPITtyuDWoeIZI4vvoDzzvOjRfziFzB6tMbXyzRBdjNfBWjzEZEa27DBj6v31Vdw9tnw9NOQVStduiTKNNSRiETK9u3Qq5cfobxbN3+59iZNwq5KwqCAEpHI2LvXH86bNw9OOAGmT4fWrcOuSsKigBKRSCgqgquugrffhiOP9EMYZWeHXZWESQElIqFzDn73O5g0CVq18ntO3/pW2FVJ2BRQIhK6u+6Cxx6Dxo1h8mTo2jXsiiQKFFAiEqqHH4a77/ZXwX3uOTjrrLArkqhQQIlIaJ59Fm64wd9//HG44IJw65FoUUCJSChefx1+9St//unPf4brrgu7IokaBZSI1LmPPoKLLoJ9++Cmm+CPfwy7IokiBZSI1KklS6B3b9i5E66+Gv7yFw1hJOVTQIlInVm71g9htHmzD6knnvCdI0TKo01DROrE5s0+nNasgdNPh+efh0aNwq5KokwBJSK1budO6NMHPv0UvvMd/12nZs3CrkqiTgElIrVq3z645BL497+hQweYMQPatg27KkkHCigRqTXFxXDttX7oonbtfNfyY44JuypJFwooEakVzvku5BMmQIsWMG0adO4cdlWSThRQIlIr/vxnePBB3xHipZcgJyfsiiTdKKBEJHD/+AcMHeq/3zR+PPToEXZFko4UUCISqJdeggED/P1HHoG+fcOtR9KXAkpEAhOLweWX+84Rd90FAweGXZGkMwWUiARi/nz4+c/9ZdsHDYI77wy7Ikl3CigRqbHly6FnT9ixwx/S+/vfNb6e1JwCSkRq5Kuv4LzzYMMG3xniqaegYcOwq5L6QAElIgctHodeveDzz3038hdf9JdtFwmCAkpEDsru3f4KuAsWQKdOMHUqtGwZdlVSnyigRCRlhYW+t97MmXD00X4Io8MPD7sqqW8UUCKSEuf895xeeQXatPHhdNxxYVcl9ZECSkRSMnQoPPkkNG0KU6b4y2eI1AYFlIhU21//6sfYa9gQJk3yFx4UqS0KKBGplnHj4Oab/f2xY/0l20VqkwJKRKr02mv+uk4Af/sbXHVVuPVIZlBAiUilPvgAfvlLKCqC22+HIUPCrkgyhQJKRCq0aBH87Gf+O0+//jX87/+GXZFkkkADyswGm9kcM9trZmODXLaI1K1Vq/wQRvE4XHghPPqoxteTupUV8PLWAcOB84CmAS9bROrIxo1w7rmwbh2cfTY88wxkBf1uIVKFQDc559yLAGaWA7QPctkiUjd27PA99JYtg65d/RdymzQJuyrJRKF8JjKz/kB/gOzsbGKxWBhlVCo/Pz+SdUWd2i018XicoqKiyLRZQYFx++2nMm9eG44+ejd33jmf+fMLwi6rDG1nqUvHNgsloJxzY4AxADk5OS43NzeMMioVi8WIYl1Rp3ZLzaGHHko8Ho9EmxUV+fH15s2D7GyYObMpJ5wQzW/iajtLXTq2mXrxiQjOwQ03wPPPQ6tWMH06nHBC2FVJplNAiQh33w2jRvlrOb36KnTrFnZFIgEf4jOzrMQyGwINzawJUOicKwxyPSISnFGj4K67oEEDePZZ32tPJAqC3oMaBuwGbgOuStwfFvA6RCQgEyfC4MH+/pgx/vtOIlERdDfzu4C7glymiNSON9/0Y+o5B/fe60eKEIkSnYMSyUAff+z3lvbtg9//Hm69NeyKRMpSQIlkmKVL/Rdxd+70e1AjRmgII4kmBZRIBvnySz+E0aZN0KuXvzJuA70LSERp0xTJEFu2+MFfV6+G007z33lq1CjsqkQqpoASyQC7dkGfPrB4MZxyCkyZAs2bh12VSOUUUCL13L59/oKDs2dDhw4wYwa0bRt2VSJVU0CJ1GPFxXDddTB1KrRrB6+/Du11nQFJEwookXrKObjlFhg/3h/OmzoVOncOuyqR6lNAidRT998Pf/ub7wjx0kvwgx+EXZFIahRQIvXQE0/Abbf57zeNHw///d9hVySSOgWUSD3z8svQv7+///DD0LdvuPWIHCwFlEg9MnMmXHaZ7xzxP/8DgwaFXZHIwVNAidQTCxbA+efD3r0wcKAPKJF0poASqQc+/9yPErF9u//O00MPaXw9SX8KKJE09/XXfny99evhpz+FceOgYcOwqxKpOQWUSBrbts0P+rpiBXTv7ruTN24cdlUiwVBAiaSpPXvgggsgLw86dYJp06Bly7CrEgmOAkokDRUWwuWXw7vvwtFH+/H1Dj887KpEgqWAEkkzzvleei+/DIce6sOpY8ewqxIJngJKJM0MGwb/+Ac0beovm9GlS9gVidQOBZRIGhk5Eu65x/fSe/55OOOMsCsSqT0KKJE0MWEC/P73/v6TT8LPfhZuPSK1TQElkgamTYN+/fz9Bx6AX/0q1HJE6oQCSiTiZs+Giy/2PfduvRVuuinsikTqhgJKJMIWL/aH8nbv9lfGvffesCsSqTsKKJGIWr3aj6+3dSv8/Ofw2GMaX08yiwJKJII2bfLj6335JZx5Jjz7LGRlhV2VSN1SQIlETH4+9O4NS5fCqafCq6/67zyJZBoFlEiEFBTARRfBxx/D8cfD9Ol+tAiRTKSAEomI4mLfffyNN+CII+D11+Goo8KuSiQ8CiiRCHAObrwRnnvOj0g+fTqceGLYVYmESwElEgHDh8PDD8Mhh/hzTt/7XtgViYQv0IAys7Zm9pKZ7TSzVWZ2RZDLF6mPNm9uzJ13QoMG8MwzkJsbdkUi0RB0x9VHgAIgG+gGvGZmC5xziwNej0i9sHEjrF3ru+g9+qjvICEinjnnglmQWXNgK9DFObcsMW0c8KVz7raKnteyZUvXvXv3QGoIUjwe51B1n0qZ2q36tmyBhQvzADj++G506BByQWlE21nqotxm77777lznXE7y9CD3oDoBRSXhlLAAODv5gWbWH+gP0KhRI+LxeIBlBKOoqCiSdUWd2q168vOz+Pzz5gBkZRXTqlUcNVv1aTtLXTq2WZAB1QLYljRtG9Ay+YHOuTHAGICcnBw3Z86cAMsIRiwWI1cnA1KmdqvanDlwzjm+595RR+VyxBFx8vLywi4rrWg7S12U28wqGMMryE4S+UCrpGmtgB0BrkMkreXlQc+esGMHXHYZnHRS2BWJRFeQAbUMyDKz0v9yXQF1kBABPvoIfvIT2LwZ+vSBp57S4K8ilQksoJxzO4EXgbvNrLmZnQFcAIwLah0i6WrWLOjRA+JxuPBCmDQJGjUKuyqRaAv6i7qDgKbABuAZYKC6mEume/ttf9mMksN6EydC48ZhVyUSfYF+D8o5twW4MMhliqSz55+Hq6+GvXvhmmvgiSegYcOwqxJJDxrqSKQWOAcjRkDfvj6cBg2CJ59UOImkQgElErDCQhg8GP7wB//7fff5cfYa6L9NJCW6RqdIgLZtgyuvhNde8wO/PvUUXHpp2FWJpCcFlEhAFi3yY+l99hm0bQsvv+wv1y4iB0cHHUQCMHEi/OhHPpy6dvVXxFU4idSMAkqkBvbuhZtu8ofxdu70h/c++AC+9a2wKxNJfzrEJ3KQPv0UrrjCD1+UlQV//avvHKHRIUSCoYASSZFz8Nhjfs9p926/tzRhgj/EJyLB0SE+kRSsXu3H0Rs40IfTNdfA/PkKJ5HaoIASqYbiYnjkEfjOd2DqVGjd2l+efexYaJU8hr+IBEKH+ESqsHgx/Pa3fsBX8F3JH34Yjjoq3LpE6jvtQYlUIB6HIUN8t/FZs+DII/0o5C+8oHASqQsKKJEkRUXw+OP+YoIPPug7RQwcCJ98AhdfHHZ1IplDh/hEEpzzoz8MG+bDCODss31Ide0abm0imUh7UJLxnIO33vI98S66yIdTx47w7LPwzjsKJ5GwaA9KMpZzMG0a3HMPvP++n5adDXfcAb/5jR/sVUTCo4CSjFNYCC++CPfe60eBAD+46803w403QvPm4dYnIp4CSjLGli2+88Mjj8CaNX7akUfCLbfAgAHQokW49YnIgRRQUq85B//+t7/U+tNP+9EfADp18l3Ir70WmjQJt0YRKZ8CSuqlr7+GceP8ZdaXLPlmes+ecMMNcN55usKtSNQpoKTe2LnTd3p46ik/HFFRkZ+enQ2/+hX8+tfQuXO4NYpI9SmgJK3t2OEvrz5pkg+lkkN4WVlw4YVw3XV+r6lRo3DrFJHUKaAk7Xz5JcyYAa++CtOn+4sGlvjRj6BvX3/hwCOOCK9GEak5BZRE3t69/ntK06f728KF38wz85dWv+QS/yXb9u3Dq1NEgqWAksjZuxc+/hhmzvS3WbP8+aUSzZvDOedAr17+MJ4GbhWpnxRQErr1630gffghvPee7xZe+rAdQJcu/lxSz57w4x9D48bh1CoidUcBJXVqwwZ/iG7OHB9KH3/sr1KbrEsXOOssfzvzTDj66LqvVUTCpYCSWrFzp7/Q38KFsGiR/7lwoQ+oZC1aQPfu8IMf+L2jH/8YDjus7msWkWhRQMlB27sXVq6Ezz7zt+XL4aOPTmXzZli1yo/ikKxlS7931K0b/PCH/ta5MzRsWPf1i0i0KaCkXM7Btm1+zLo1a/xhuNL3V63yP4uLk5/ZFvDfQ/r2t+G73/W3Ll38z+OO8z3vRESqooDKMIWFsHGj75iwYYP/WXLbsMEPEbR2rQ+f/PzKl9WgARx/vL/y7EknwYknwu7d/+Gii07l+ON1uQoRqRkFVBpyzo+YsH07bN3qR+neurXq+5s2webN5R96K0+zZtChAxx7rL8l3y8vhGKxLRpOSEQCEUhAmdlgoB/wXeAZ51y/IJabroqLfYDs2XPgz8qm5ef7244dlf8suZU9tFY9ZnD44X6Uhezsb26lf2/f3odQmzY6HCci4QlqD2odMBw4D2iayhP37oVly/zAnqVvxcVlpwU1vbAQ9u2DgoIDf5a+/+WXJ/PQQ2WnV3S/oOCbwNm3L6BWrUSTJr7DQdu2PkhKbpX93q6dv2Vpv1lE0oC56h7vqc7CzIYD7VPZgzJr6aB70tS+wCBgF9C7nGf1S9w2AZeUM38gcCmwBri6nPk3A+cDS4EB5cwfBvQA8oAh5cy/Bzgd+AAYWs78kTRt2o2GDd+koGA4DRpwwK1Ll8c47LDObN06mc8+e4AGDXwvtpLb9dePo0OHY8nLe4433hh9wLyGDWHSpEkceWQ7xo4dy9ixY8usferUqTRr1oxRo0YxceLEMvNjsRgAI0aMYMqUKQfMa9q0KdOmTQPgT3/6E2+99dYB8w877DBeeOEFAG6//XZmz559wPxGjRrxxhtvADBkyBDySi5Zm9CpUyfGjBkDQP/+/Vm2bNkB87t168bIkSMBuOqqq1i7du0B80877TTuvfdeAC6++GI2b958wPyf/vSn3HHHHQD06tWL3SWjxyb06dOHW265BYDc3FyS9e3bl0GDBrFr1y569y677fXr149+/fqxadMmLrmk7LY3cOBALr30UtasWcPVV5fd9m6++WbOP/98li5dyoABA8jLy6OwsJCcnBwAhg0bRo8ePcjLy2PIkLLb3j333MPpp5/OBx98wNChZbe9kSNH0q1bN958802GDx9eZv5jjz1G586dmTx5Mg888ECZ+ePGjePYY4/lueeeY/To0WXmT5o0iXbtwt/2rrzySr788ssD5rdv357x48cD2vbK2/bOPfdchg4dun/bSxbmtvfuu+/Odc7lJD8nlM/SZtYf6O9/a84hhxQnDiU5zKBlyz20abMD2MnatYWJ53xzuKldu3yyszdTVLSZZcv27Z9esoxjj41zzDFfsXfvehYsKMDMlZoP3/72Rjp2XMXOnWuZPXsPZm7/8s0cZ5yxivbt55Gfv5IZM3bun17ymF/8YgmdOjVi5cpPePHF7funN2jgaNAABg+ew0knxZk7dwHjxsXLvP4BAz6kQ4ev+OCDhezYUXb+CSfM5ogjVrB06WIgvn/Pr8SHH75P69atWbJkCfF42efPnDmTJk2asGzZsnLnl7xJrFixosz83bt375+/cuXKMvOLi4v3z1+9enWZ+W3atNk/f+3atWXmr1u3bv/8devWlZm/du3a/fPXr08UMHwAAAYFSURBVF9fZv7q1av3z9+4cSPbt28/YP7KlSv3z9+yZQt7k4akWLFixf755bXNsmXLiMVi7Nmzp9z5S5YsIRaLsW3btnLnL168mFgsxoYNG8qdv3DhQlq2bLm/7QoLC3HO7X/sggULyMrKYvny5eU+f968eRQUFLBo0aJy58+ZM4d4PM6CBQvKnf/hhx/y1VdfsXDhwnLnz549mxUrVrB48eJy57//fjS2vYKCgjLzGzVqpG2vkm1vz549xGKxcv9vIfxtrzyh70Hl5OS4OXPmBFZDUGKxWLmfcqRyarfU5ObmEo/Hy3zal8ppO0tdlNvMzMrdg6rymqJmFjMzV8FtVu2UKyIima7KQ3zOudw6qENEROQAQXUzz0osqyHQ0MyaAIXOucIgli8iIpmnykN81TQM2A3cBlyVuD8soGWLiEgGCmQPyjl3F3BXEMsSERGB4PagREREAqWAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSSFFAiIhJJCigREYkkBZSIiESSAkpERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJIUUCIiEkkKKBERiSQFlIiIRJICSkREIkkBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiqcYBZWaNzewJM1tlZjvMbL6Z9QqiOBERyVxB7EFlAWuAs4HWwB3ARDPrGMCyRUQkQ2XVdAHOuZ3AXaUmTTGzlUB34IuaLl9ERDJTjQMqmZllA52AxZU8pj/QHyA7O5tYLBZ0GTWWn58fybqiTu2Wmng8TlFRkdosRdrOUpeObWbOueAWZtYImAascM4NqM5zcnJy3Jw5cwKrISixWIzc3Nywy0g7arfU5ObmEo/HycvLC7uUtKLtLHVRbjMzm+ucy0meXuU5KDOLmZmr4Dar1OMaAOOAAmBwoNWLiEjGqfIQn3Mut6rHmJkBTwDZQG/n3L6alyYiIpksqHNQo4GTgR7Oud0BLVNERDJYEN+DOg4YAHQDvjaz/MTtyhpXJyIiGSuIbuarAAugFhERkf001JGIiESSAkpERCIp0O9BHVQBZhuBVaEWUb52wKawi0hDarfUqc1SpzZLXZTb7Djn3OHJE0MPqKgysznlfXFMKqd2S53aLHVqs9SlY5vpEJ+IiESSAkpERCJJAVWxMWEXkKbUbqlTm6VObZa6tGsznYMSEZFI0h6UiIhEkgJKREQiSQElIiKRpICqJjM7ycz2mNn4sGuJMjNrbGZPmNkqM9thZvPNrFfYdUWRmbU1s5fMbGeiva4Iu6Yo07ZVM+n4HqaAqr5HgI/DLiINZAFrgLOB1sAdwEQz6xhiTVH1CP4Cn9nAlcBoM/tOuCVFmratmkm79zAFVDWY2WVAHHgr7Fqizjm30zl3l3PuC+dcsXNuCrAS6B52bVFiZs2Bi4E7nHP5zrlZwKvA1eFWFl3atg5eur6HKaCqYGatgLuBm8OuJR2ZWTbQCVgcdi0R0wkocs4tKzVtAaA9qGrStlU96fwepoCq2p+AJ5xza8IuJN2YWSNgAvAv59ySsOuJmBbAtqRp24CWIdSSdrRtpSRt38MyOqDMLGZmroLbLDPrBvQA/hZ2rVFRVZuVelwDYBz+HMvg0AqOrnygVdK0VsCOEGpJK9q2qi/d38NqfEXddOacy61svpkNAToCq80M/KfehmZ2inPu+7VeYARV1WYA5hvrCfzJ/97OuX21XVcaWgZkmdlJzrnPEtO6osNVldK2lbJc0vg9TEMdVcLMmnHgp9xb8H/sgc65jaEUlQbM7FGgG9DDOZcfdj1RZWbPAg64Ht9eU4HTnXMKqQpo20pNur+HZfQeVFWcc7uAXSW/m1k+sCcd/rBhMbPjgAHAXuDrxKc2gAHOuQmhFRZNg4AngQ3AZvybhsKpAtq2Upfu72HagxIRkUjK6E4SIiISXQooERGJJAWUiIhEkgJKREQiSQElIiKRpIASEZFIUkCJiEgkKaBERCSS/j84MWScDDTxeAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, elu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1, -1], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(r\"ELU activation function ($\\alpha=1$)\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"elu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing ELU in TensorFlow is trivial, just specify the activation function when building each layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7fc208500a90>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"elu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SELU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This activation function was proposed in this [great paper](https://arxiv.org/pdf/1706.02515.pdf) by Günter Klambauer, Thomas Unterthiner and Andreas Mayr, published in June 2017. During training, a neural network composed exclusively of a stack of dense layers using the SELU activation function and LeCun initialization will self-normalize: the output of each layer will tend to preserve the same mean and variance during training, which solves the vanishing/exploding gradients problem. As a result, this activation function outperforms the other activation functions very significantly for such neural nets, so you should really try it out. Unfortunately, the self-normalizing property of the SELU activation function is easily broken: you cannot use ℓ<sub>1</sub> or ℓ<sub>2</sub> regularization, regular dropout, max-norm, skip connections or other non-sequential topologies (so recurrent neural networks won't self-normalize). However, in practice it works quite well with sequential CNNs. If you break self-normalization, SELU will not necessarily outperform other activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import erfc\n",
    "\n",
    "# alpha and scale to self normalize with mean 0 and standard deviation 1\n",
    "# (see equation 14 in the paper):\n",
    "alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)\n",
    "scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) * (2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def selu(z, scale=scale_0_1, alpha=alpha_0_1):\n",
    "    return scale * elu(z, alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure selu_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwV5b3H8c+PEJBNooCpCMqtioobYq69qK2xUhcEN6youFCrUCxWLLhRUCoISqlFqyAolgqooNSFRb1qG68WtUKhWFRwAcSdIAHCEkjy3D+ekxJOFnKSSWbOOd/36zUvDmcmM78zDOebmXnmecw5h4iISNQ0CrsAERGRyiigREQkkhRQIiISSQooERGJJAWUiIhEkgJKREQiSQElUgNmtsbMhjXAdkaZ2b8bYDuNzGyKmW0wM2dmufW9zb3UM93M5odZg0SPAkoSYmbtzGxS7Au7yMy+MbPXzOwn5ZbJi33pxU9PlVvGmdnFVWyjv5kVVjGvyp8LQjUB8d/ApAC30yn2WXLiZk0ATgtqO9XoCfwM6A0cCCxqgG1iZrmxz902btaNwBUNUYMkj8ZhFyBJZy7QHPg58DFwAP4LtU3ccn8Chse9t73eq6snzrn1DbSdQqDScA7YYcBXzrkGCaa9cc5tCrsGiR6dQUmNmVkW8EPgNufca865tc65d51zE5xzT8Utvs0593XcVO9fQmZ2tpm9YWYbzew7M3vZzI6KW6a9mc2KXd7aZmbLzOx0M+sP3AkcXe6sr3/sZ/5zic/MnjSzuXHrbGRm68zsphrWsTr257ux7eTFfm6PM7jYekfG1l1kZu+Z2fnl5pedifUxs1din+f98me0leyj6cAfgINjP7sm9n6emT0Yv2z5S2+xZSaZ2Vgzyzezb81sgpk1KrdMk9j8tbGaPzWzX5lZJ+BvscXWx7Y9vYrtNDWzibEz9B1m9raZnVpuftmZ2Blm9k7scy82s25VfW5JPgooSUTZb/fnmdk+YRdThRbAROAkIBfYBMwzsyYAZtYCeB3oBFwIHAvcFfvZ2cDvgZX4y14Hxt6LNxM4NxbYZU6LLf9kTeqIvQ9wduznLqri89wI3AzcGqv1WeAvZtY1brm7gQeA44F3gafMrGU167wL+Dy27f+uYrmq9AOKgZOBwcAQoG+5+X8GrgJ+DRyFP9suANYBfWLLHB3b9o1VbGN8bJ3XACcA7wEvmdmBccuNA24DugEbgFlmZgl+Hokq55wmTTWe8F8w3wE7gLfw90x+ELdMHrCT3YFWNl1fbhkHXFzFNvoDhVXMq/Lnqli+BVACnBr7+3XAFqBtFcuPAv5dyftrgGGx142Bb4Gfl5v/KPByAnV0in2WnOq2D3wB3FHJ/p0Zt56B5eYfFHvv1GrqGQasqWS9D8a9Nx2YH7fMW3HLvAI8Gnt9eGzbZ1ex3dzY/LZVbSe2r3YCV5WbnwF8AoyJW89Z5ZY5JfZeh7D/n2gKZtIZlCTEOTcXaI+/uf4i/rfot80s/n7TbKBr3DSrvuszs0PN7Akz+8TMNgPf4K8UHBxb5ARguXMuv7bbcM4V4z9fv9g2m+KDe2YCddTks+yL39d/j5v1JtAl7r3l5V5/GfvzgJpuK0HL4/7+ZbltnQCUsvtSXm0cCmRS7nM750rwvxCF+bmlgamRhCTMObcD/1vzK8BdZvYoMMrMJjjndsYW2+Sc+7iWm9gMNDOzTOfcrrI3y11Sq+5e1jz8WcfA2J/FwPtA2aW1oC7/zAQWmdlBwA9i6382gToSUdmQA/Hv/Wc/Oedc7CpXor+AllJx/2RWstyuuL+7ctsKYv+WrSOhz11unn7xThH6h5QgvI//ZSeo+1Ir8cfmCXHvdys3vwIza4O/5zHWOfeqc+4DoBV7/iL2T+C4Spo5l9mJv5xULefcO/hLTpfhz6Sec74FXk3rKAvyKrflnNuMPys4NW7Wqfh9HrT1+PtC5R2f4Dr+if+3O72K+Xv93PjWoTsp97nNLAPoTv18bokonUFJjcW+eJ8GHsNfWtkC5AC3AK/FvlDLNDez78WtYqdz7rtyf+9Uyc3+T51zK8zsf4FHzezX+CDoDNwPzHHOfVZFiRuBfOA6M1uHvxfzO/zZS5kn8DfVnzOz2/ENBY4Ftjjn/oa/13RIrDXYZ7H3i6rY3izgWnY3uEikjm/xze7PirWi2+Eqb+X4O/xZ6kfAEvyzQj8ETqyiprr4KzDRzM7D/xIwEOiI3yc14pz7yMzm4P/tbsQHVgegk3NuBrAWf6ZzrpnNA7aXBXu5dWw1s8nAPWaWj2/xeBOQTYDPokkSCPsmmKbkmYCmwFh8K7GNwDbgI+A+YP9yy+Xhv4TipzfLLVPZfAf0is3PwgfSx7HtrALuBVrupcYfA//GN+L4N3AWvoFG/3LLdMDfQyqIrXspkFvuMz4T+3yu7Oco10ii3HoOjS3zDdC4FnVciw/BEiAv9t4o9mwk0QgYiW8BtxPfmu2CcvM7UXlji2obk1B5I4lM4CF8uObjW/pNp2Ijib01pGiKb4X3BVCE/wVjcLn5I4Gv8JcUp1ezjomxfVsEvE25Rh9U0tiiqn2hKXkni/3DioiIRIruQYmISCQpoEREJJIUUCIiEkkKKBERiaTQm5m3bdvWderUKewyKti6dSstWrQIu4yko/2WmJUrV1JSUkKXLvEdJEh1onqcFRbCqlXgHHToANnZYVe0W1T3GcCSJUvynXPt4t8PPaA6derE4sWLwy6jgry8PHJzc8MuI+lovyUmNzeXgoKCSP4fiLIoHmerVkH37j6cBg+GBx6AKHVbG8V9VsbM1lb2vi7xiYjU0fr10LMnfPcd9OoFEydGK5ySlQJKRKQOtm+H88+HTz6Bbt3gySchY6+dZUlNKKBERGqptBSuvhreegs6doT586FlVaNwScIUUCIitTR8ODz9NLRqBQsWwIHxXe1KnQQaUGY208y+MrPNZrbKzK4Ncv0iIlHxyCNw773+ct4zz8Cxx4ZdUeoJ+gxqHL7X4n2B84AxZlYfvS6LiITm5Zdh0CD/+uGH4cwzw60nVQUaUM65FW730ARlvVMfGuQ2RETCtHw5/PSnUFICt98O1+o6Ub0J/DkoM5sE9Aea4YcxWFjJMgOAAQDZ2dnk5eUFXUadFRYWRrKuqNN+S0xBQQElJSXaZwkK6zjLz2/C9dd3Y8uWfTj99G/p0eN9kuWfLhn/b9bLcBvlRr/MBe515YbtjpeTk+Oi+JBilB9qizLtt8SUPai7bNmysEtJKmEcZ4WF8KMfwdKlcMop8OqrsE9QY0g3gCj/3zSzJc65nPj366UVn3OuxDn3Jn5guEH1sQ0RkYZSXAyXXurD6bDD4LnnkiucklV9NzNvjO5BiUgScw5uvNE3I2/TBhYuhLZtw64qPQQWUGZ2gJldamYtzSzDzM4CLgP+GtQ2REQa2h/+AJMmQZMm/szp8MPDrih9BNlIwuEv5z2MD761wBDn3PMBbkNEpME8+ywMG+Zf//nPcOqp4daTbgILKOfceuC0oNYnIhKmd96Bfv38Jb6xY/09KGlY6upIRCTO6tXQu7fvCPbnP4fbbgu7ovSkgBIRKWfjRj90xvr18JOfwOTJGjojLAooEZGYnTuhTx/48EM45hjfEWxmZthVpS8FlIgI/l7TddfB3/4G3/ueb1beunXYVaU3BZSICHDXXfD449C8uR/X6eCDw65IFFAikvZmzIBRo6BRI3jqKThRYzBEggJKRNJaXp5vqQcwcaJvvSfRoIASkbT1wQdw4YWwaxcMGQI33BB2RVKeAkpE0tK338K550JBAZx/PkyYEHZFEk8BJSJpZ/t2OO88/0BuTg7MmuWHbpdoUUCJSFopLYUrr/RdGR1yCMybBy1ahF2VVEYBJSJp5dZbYe5c/4zTggX+mSeJJgWUiKSNyZP9vabGjX1IHX102BVJdRRQIpIWFi6EwYP960cegTPOCLce2TsFlIikvGXLoG9ff/9pxAjo3z/siqQmFFAiktI+/9w3Jy8shMsv910aSXJQQIlIytq82YfTl1/Cj34Ejz2moTOSiQJKRFJScbG/rLd8OXTu7Idvb9o07KokEQooEUk5zvkGES+9BG3b+gYS++8fdlWSKAWUiKScCRNgyhR/xvTCC3DooWFXJLWhgBKRlPL003DLLf71jBnQvXu49UjtKaBEJGW89Zbvxgjg3nvhpz8Ntx6pGwWUiKSETz7xHcAWFcGAAXDzzWFXJHWlgBKRpPfdd9CzJ+Tnw9lnw0MPqTl5KlBAiUhSKyqCCy6AVavguONg9mzf154kPwWUiCQt5+Caa+CNN6B9e987+b77hl2VBEUBJSJJ68474Ykn/HhO8+dDhw5hVyRBUkCJSFKaPh1Gj4ZGjWDOHDjhhLArkqApoEQk6SxZksV11/nXDz7oG0hI6lFAiUhSef99uPPOYyguhqFDYdCgsCuS+qKAEpGk8fXX/mxp69bG9OkD48eHXZHUJwWUiCSFrVuhd29YuxaOOmozM2b4+0+SugL75zWzpmY2zczWmtkWM1tqZucEtX4RSV8lJdCvHyxeDP/1X3D33e/RrFnYVUl9C/L3j8bAOuA0oDUwEphjZp0C3IaIpKFhw+D55yEryz/rtN9+u8IuSRpAYAHlnNvqnBvlnFvjnCt1zs0HVgMnBrUNEUk/Dz4IEydCZqYfdPCoo8KuSBpKvXUIYmbZQGdgRSXzBgADALKzs8nLy6uvMmqtsLAwknVFnfZbYgoKCigpKdE+q8KiRW0YOfIYwLj55g+Ab8jL03FWG8m4z8w5F/xKzTKBF4FPnHMDq1s2JyfHLV68OPAa6iovL4/c3Nywy0g62m+Jyc3NpaCggGXLloVdSuQsWQI/+hFs2wajRvleI8roOEtclPeZmS1xzuXEvx94GxgzawTMAHYCg4Nev4ikvs8+g169fDhddRXccUfYFUkYAr3EZ2YGTAOygZ7OOd3JFJGEbNoE557rn3k6/XR45BENnZGugr4HNRk4CujhnNse8LpFJMXt2uVHwf33v+HII2HuXGjSJOyqJCxBPgd1CDAQ6Ap8bWaFsalfUNsQkdTlnO+26JVX4IADYOFC2G+/sKuSMAV2BuWcWwvoRFxEauWee2DaNNhnH3jhBf9ArqQ3dRQiIqF76ikYPtzfa5o1C37wg7ArkihQQIlIqN58E/r3968nTICLLgq1HIkQBZSIhOajj+D886GoCK6/Hm66KeyKJEoUUCISivx8P3TGd9/5P++/X83JZU8KKBFpcDt2wAUXwMcf+6HaZ8+GxvXW8ZokKwWUiDSo0lL42c/g73+HDh1g/nxo2TLsqiSKFFAi0qBGjPCt9lq18kNntG8fdkUSVQooEWkwjz4K48ZBRgY8/TQcd1zYFUmUKaBEpEG88gr84hf+9aRJcNZZ4dYj0aeAEpF69957cPHFfuj2W2+FAQPCrkiSgQJKROrVl1/63sk3b4ZLLoGxY8OuSJKFAkpE6k1hIfTuDevWQffuMH06NNK3jtSQDhURqRclJXD55fDPf8Khh8Lzz0OzZmFXJclEASUigXMOhgyBefNg//390Bnt2oVdlSQbBZSIBO7+++HBB/1gg889B507h12RJCMFlIgE6rnn4Ne/9q//9Cf44Q/DrUeSlwJKRALz7rv+vpNzMGaMfy1SWwooEQnEmjW+xd727XDNNX4AQpG6UECJSJ0VFPghM775Bs44Ax5+WENnSN0poESkTnbuhD594IMPoEsXeOYZyMwMuypJBQooEak152DgQPjrX+F73/PNybOywq5KUoUCSkRq7e67fe8QzZv7Z54OOSTsiiSVKKBEpFZmzYKRI/29pieegJycsCuSVKOAEpGE/d//+ZZ6AH/4A5x/frj1SGpSQIlIQlauhAsu8I0jfvUruPHGsCuSVKWAEpEaW7/eNyffuBHOOw/uuy/siiSVKaBEpEa2b/eh9OmncOKJ/r5TRkbYVUkqU0CJyF6VlsJVV8Hbb8PBB/sWey1ahF2VpDoFlIjs1e23+wdw990XFiyAAw8MuyJJBwooEanWlCkwfjw0bgxz58Ixx4RdkaQLBZSIVOmll+CXv/Svp0yBHj3CrUfSiwJKRCr1r3/BT3/qh27/zW92P/ck0lAUUCJSwRdfwLnnQmEhXHYZjB4ddkWSjgINKDMbbGaLzazIzKYHuW4RaRhbtkCvXj6kTj3Vj4qroTMkDI0DXt+XwBjgLKBZwOsWkXpWXAx9+8KyZXD44X749qZNw65K0lWgAeWc+wuAmeUAHYJct4jUL+d810Uvvght2vihM9q0CbsqSWdBn0HViJkNAAYAZGdnk5eXF0YZ1SosLIxkXVGn/ZaYgoICSkpKIrHP5szpwOTJh5GZWcqoUcv4/PPNfP552FVVTsdZ4pJxn4USUM65qcBUgJycHJebmxtGGdXKy8sjinVFnfZbYrKysigoKAh9n82d64dpB5g5sxGXXNIt1Hr2RsdZ4pJxn6kVn0iae/ttuOIKf4lv3Di45JKwKxLxFFAiaezTT30HsDt2wHXXwa23hl2RyG6BXuIzs8axdWYAGWa2D1DsnCsOcjsiUncbN/pnndavhzPPhIceUnNyiZagz6BGANuB24ArYq9HBLwNEamjoiK46CL48EM49lh4+mnIzAy7KpE9Bd3MfBQwKsh1ikiwnPOX8/LyfK/kCxb4XspFokb3oETSzG9/CzNm+PGc5s+Hjh3DrkikcgookTTy+OM+oBo1gtmzoVu0W5NLmlNAiaSJv/0Nrr3Wv37gAd9AQiTKFFAiaeCDD+DCC2HXLrjppt1jPIlEmQJKJMV98w307AmbNvmQ+t3vwq5IpGYUUCIpbNs2/yDumjVw0kkwcyZkZIRdlUjNKKBEUlRJie/C6B//gE6d4IUXoHnzsKsSqTkFlEiKuuUWePZZaN3aP+uUnR12RSKJUUCJpKBJk+C++3zvEH/5C3TpEnZFIolTQImkmAUL4IYb/OtHHoEf/zjcekRqSwElkkKWLvVDtpeWwh13wNVXh12RSO0poERSxLp1/uHbrVt944hRo8KuSKRuFFAiKWDzZh9OX30Fp50Gjz6qoTMk+SmgRJLcrl1+FNz33oMjjvAt95o2DbsqkbpTQIkkMed8t0Uvvwzt2sHChbDffmFXJRIMBZRIEhs/3rfU22cf/yDu978fdkUiwVFAiSSpOXPgttv8vaaZM+F//ifsikSCpYASSUKLFsFVV/nX48dDnz7h1iNSHxRQIknm44/h/POhqAh+8QsYOjTsikTqhwJKJIls2OCHzsjPh3POgT/+Uc3JJXUpoESSRFGRH8/po4/g+OP9kO2NG4ddlUj9UUCJJAHn4Jpr4I034KCDfH97rVqFXZVI/VJAiSSBO+6AJ56Ali19OB10UNgVidQ/BZRIxD32GIwZ40fCnTPHX94TSQcKKJEIe/VVGDjQv37oId8wQiRdKKBEImrFCv98U3Ex3Hzz7qASSRcKKJEI+vpr35x882a4+GK4556wKxJpeAookYjZuhV69YLPPvPdFz3+ODTS/1RJQzrsRSKkpAQuvxyWLPEdvz7/PDRrFnZVIuFQQIlEyNChvlfy/fbzQ2cccEDYFYmERwElEhEPPAD33w+ZmX7QwSOOCLsikXApoEQi4IUXYMgQ//qxx/yw7SLpLtCAMrP9zexZM9tqZmvN7PIg1y+SirZty+Cyy3x3RnfdBVdcEXZFItEQdFeTDwE7gWygK7DAzP7lnFsR8HZEUkJREaxe3YLiYujfH0aMCLsikegw51wwKzJrAWwEjnHOrYq9NwP4wjl3W1U/16pVK3fiiScGUkOQCgoKyMrKCruMpKP9lpi//30ZxcWQldWV447T0Bk1peMscVHeZ6+//voS51xO/PtBnkF1BkrKwinmX0CFq+lmNgAYAJCZmUlBQUGAZQSjpKQkknVFnfZbzW3c2ITiYv+6ffvNbNpUGm5BSUTHWeKScZ8FGVAtgU1x720CKgwK4JybCkwFyMnJcYsXLw6wjGDk5eWRm5sbdhlJR/utZvLz4cgjAXLp0GEbK1b8I+ySkoqOs8RFeZ9ZFZcOgmwkUQjsG/fevsCWALchkhJGj/aj42ZlQZs2O8MuRySSggyoVUBjMzu83HvHA2ogIVLOp5/C5Mn+ftNhh4VdjUh0BRZQzrmtwF+Au8yshZmdApwPzAhqGyKp4De/gV274MoroUWLsKsRia6gH9S9HmgGfAs8CQxSE3OR3f7xD3jqKWja1F/mE5GqBfoclHPuO+CCINcpkipKS+FXv/KvhwyBgw8Otx6RqFNXRyINZMYMeOcdOPBAf5lPRKqngBJpAJs3w623+tfjx0OrCg9fiEg8BZRIAxg9Gr75Brp3h379wq5GJDkooETq2YcfwsSJvln5H/+o7oxEakoBJVKPnIObboLiYrj2Wohgt5MikaWAEqlHc+bASy9B69Zw991hVyOSXBRQIvUkPx9uuMG//t3voF27cOsRSTYKKJF6ctNNsH49nH66v7wnIolRQInUgxdfhJkzYZ99YOpUNYwQqQ0FlEjAtmyBgQP969Gj1SGsSG0poEQCdvvtsG6db7E3ZEjY1YgkLwWUSIBeegkeeggaN4Zp0/yfIlI7CiiRgHz7LfTv71//9rdw/PGhliOS9BRQIgFwDn7+c9+d0Wmn7e53T0RqTwElEoBJk2D+fD+E+4wZkJERdkUiyU8BJVJHK1bAsGH+9SOPQMeO4dYjkioUUCJ1UFgIffvCjh1wzTVw8cVhVySSOhRQIrVUdt9pxQo48ki4//6wKxJJLQookVqaMMF3BtuqFTz3HLRsGXZFIqlFASVSC6++Crfd5l8//jgccUS49YikIgWUSILWroVLL4XSUvjNb+CCC8KuSCQ1KaBEErB5M5x3HmzYAGef7R/IFZH6oYASqaFdu3wrveXLoXNnmDVLzzuJ1CcFlEgNOOd7KH/lFT/w4Isvwv77h12VSGpTQInUwOjR8Kc/QbNmvseI738/7IpEUp8CSmQvpk2DO++ERo3gqafgpJPCrkgkPSigRKrxxBNw3XX+9QMP+AYSItIwFFAiVXjmGbjqKn//6e674Ze/DLsikfSigBKpxLx5cNllUFICI0fC8OFhVySSfhRQInEWLPDNyYuLfS/letZJJBwKKJFynnzS9wyxcycMHgzjx4NZ2FWJpCcFlEjMww9Dv37+zOmWW3yjCIWTSHgUUCLAPffAoEG+QcS4cXDvvQonkbAFElBmNtjMFptZkZlND2KdIg2huBhuuAFuv90H0qRJu3spF5FwNQ5oPV8CY4CzgGYBrVOkXm3a5EfDffllaNIE/vxn30u5iERDIAHlnPsLgJnlAB2CWKdIfVq9Gnr1gvff933rPfssnHJK2FWJSHlBnUElxMwGAAMAsrOzycvLC6OMahUWFkayrqhLhv22ZEkWY8Z0oaCgCYccspVx495j164dhFF2QUEBJSUlkd9nUZMMx1nUJOM+CyWgnHNTgakAOTk5Ljc3N4wyqpWXl0cU64q6KO+30lIYOxbuuMM3hjjrLJg9uwWtW/9PaDVlZWVRUFAQ2X0WVVE+zqIqGffZXhtJmFmembkqpjcbokiRusrPh3PP9b1CgA+pBQugdetw6xKRqu31DMo5l9sAdYjUm7/+Fa6+Gj7/3I/hNGuWHw1XRKItqGbmjc1sHyADyDCzfcwslMuHImW2b4ebboIzzvDh9IMfwNKlCieRZBHUg7ojgO3AbcAVsdcjAlq3SMKWLIETT4SJE/2w7KNGwRtvwMEHh12ZiNRUUM3MRwGjgliXSF0UFvowmjjR90R+5JEwYwbk5IRdmYgkSl0dScqYNw+6dIHf/9630hsyBP75T4WTSLLSfSJJeh9/7IfFeP55//du3WDqVH+JT0SSl86gJGlt3Ai//rU/a3r+eWjZ0l/ae+cdhZNIKtAZlCSd7dthyhQYMwY2bPCdvP7sZ/7v7duHXZ2IBEUBJUmjqAimTYO774Yvv/TvnXYa3Hefv6wnIqlFASWRt20bTJ/ux2j67DP/XteucNddvsNXjdskkpoUUBJZGzb48ZkeeMB3VQRw9NHw29/ChRdCI91BFUlpCiiJnKVL/fDrM2f6syfwTcVvvdUHU0ZGuPWJSMNQQEkkbNsGc+b4YHrnnd3vn3023HIL5ObqUp5IulFASWic8w/SzpjhR7MtKPDvZ2X5zl0HDoSjjgq3RhEJjwJKGtyqVfDkk/DEE/51mZNOgl/8wg/D3rx5ePWJSDQooKRBfPSRf5h29mxYvHj3+wcc4APp6qv1cK2I7EkBJfWiuBgWLfL9482bBytX7p7XqhVcdBFcfjn8+MfQWEehiFRCXw0SmNWrYcGCA5kyBf73f+G773bPy8qCnj19K7xzz4VmzcKrU0SSgwJKam3dOj/G0muv+VFr16wBOOI/8w8/HHr39tMpp0BmZliVikgyUkBJjRQV+eeT3nrLT4sWwRdf7LnMfvvBMcesp2/fdvToAUccUfm6RERqQgElFWzbBsuX+0Aqm5Yvh50791yudWvo3t3fRzrjDDj+eHjjjRXk5uaGUreIpBYFVBrbvt03XvjgAz+9/76fVq6E0tKKy3fp4gOpbDrySHU3JCL1RwGV4nbs8PeGVq+GTz/104cf+kBas8Y/LBsvIwOOPRZOOMFP3br5s6PWrRu6ehFJZwqoJFZa6jtR/eKLPafPPtsdRvH3icrLyPANGY46as+pSxe1shOR8CmgIsY53+XP+vV7Tvn58O23ewbRl1/Crl3Vry8jAw4+GL7//d1TWSgddhg0adIwn0tEJFEKqHpQWgpbtvigKSiATZt2v65s2rjRDy1RFkTFxTXf1n77wUEH7Tl17Lg7jDp21IOwIpKc0uaryznfCm3HDt9keseO3VNlf1+6NJuPP/Z/37oVCgtr/uf27XWrtVUraNeu8ql9+91B1L69+qwTkdQVekB99RWMHOnPGnbtqvrP6uZVtUxZIJWFTmLq1o12ixb+7CYra+9T69bQtq2f2rWDpk3rtGkRkZRgrrJmXA1ZgLVyEN9L6CXA9cA2oGclP9U/NuUDF1cyfxDQF1gHXFluW75ZdIsWQ2ndujeNGq1k/fqBNErq7AAAAAY/SURBVGrEHlOXLiNo2vRYWrb8inffHUJGhn8/I8NPl146lhNOOJm1axfx+OPDK8y///6JdOvWlVdffZUxY8ZUqG7KlCkcccQRzJs3j9///vcV5s+YMYOOHTsye/ZsJk+eXGH+M888Q9u2bZk+fTrTp0+vMH/hwoU0b96cSZMmMWfOnArz8/LyAJgwYQLz58/fY16zZs148cUXARg9ejSvvfbaHvPbtGnD3LlzAbj99tt566239pifmZnJK6+8AsCQIUNYtmzZHvM7d+7M1KlTARgwYACryndnDnTt2pWJEycCcMUVV/D555/vMb979+6MGzcOgD59+rBhw4Y95p9xxhmMHDkSgHPOOYftcaezvXr1YtiwYQCVPq91ySWXcP3117Nt2zZ69qx47PXv35/+/fuTn5/PxRdXPPYGDRpE3759WbduHVdeeWWF+UOHDqV3796sXLmSgQMHsmzZMoqLi8nJyQFgxIgR9OjRg2XLljFkyJAKPz927FhOPvlkFi1axPDhwyvMnzhxIl27pv6x169fP76IawHUoUMHZs6cCejYq+zYO/PMMxk+fPh/jr14YR57r7/++hLnXE78z4R+BtWkib9UZbZ76tbNP/hZWuqH+y4/zwzOPNP351ZYCKNGVZx/xRVwwQX+ns6NN+4OnjJDh/rud1au9GMOxRsxAho3/oCsrCwq+Xfi7LPh5JN9bwrPPVdxvp4NEhGpu9DPoHJyctzi8uMvREReXp56RKgF7bfE5ObmUlBQUOG3famejrPERXmfmVmlZ1D6XV9ERCJJASUiIpGkgBIRkUhSQImISCQpoEREJJLqHFBm1tTMppnZWjPbYmZLzeycIIoTEZH0FcQZVGP8E7GnAa2BkcAcM+sUwLpFRCRN1flBXefcVmBUubfmm9lqfPcQa+q6fhERSU+B9yRhZtlAZ2BFNcsMAAYAZGdn/6f7kygpLCyMZF1Rp/2WmIKCAkpKSrTPEqTjLHHJuM8C7UnCzDKBF4FPnHOVdCJUkXqSSC3ab4lRTxK1o+MscVHeZ7XuScLM8szMVTG9WW65RsAMYCcwONDqRUQk7ez1Ep9zLndvy5iZAdOAbKCnc24v47yKiIhUL6h7UJPxAyj1cM7Vcbg+ERGRYJ6DOgQYCHQFvjazwtjUr87ViYhI2gqimflawAKoRURE5D/U1ZGIiESSAkpERCIp9BF1zWw9sDbUIirXFsgPu4gkpP2WOO2zxGmfJS7K++wQ51y7+DdDD6ioMrPFlT04JtXTfkuc9lnitM8Sl4z7TJf4REQkkhRQIiISSQqoqk0Nu4Akpf2WOO2zxGmfJS7p9pnuQYmISCTpDEpERCJJASUiIpGkgBIRkUhSQNWQmR1uZjvMbGbYtUSZmTU1s2lmttbMtpjZUjM7J+y6osjM9jezZ81sa2x/XR52TVGmY6tukvE7TAFVcw8B74ZdRBJoDKwDTgNaAyOBOWbWKcSaouoh/ACf2UA/YLKZHR1uSZGmY6tuku47TAFVA2Z2KVAAvBZ2LVHnnNvqnBvlnFvjnCt1zs0HVgMnhl1blJhZC6APMNI5V+icexN4Abgy3MqiS8dW7SXrd5gCai/MbF/gLmBo2LUkIzPLBjoDK8KuJWI6AyXOuVXl3vsXoDOoGtKxVTPJ/B2mgNq70cA059y6sAtJNmaWCcwC/uyc+zDseiKmJbAp7r1NQKsQakk6OrYSkrTfYWkdUGaWZ2auiulNM+sK9AD+EHatUbG3fVZuuUbADPw9lsGhFRxdhcC+ce/tC2wJoZakomOr5pL9O6zOI+omM+dcbnXzzWwI0An4zMzA/9abYWZdnHPd6r3ACNrbPgMwv7Om4W/+93TO7arvupLQKqCxmR3unPso9t7x6HJVtXRsJSyXJP4OU1dH1TCz5uz5W+4w/D/2IOfc+lCKSgJm9jDQFejhnCsMu56oMrOnAAdci99fC4GTnXMKqSro2EpMsn+HpfUZ1N4457YB28r+bmaFwI5k+IcNi5kdAgwEioCvY7+1AQx0zs0KrbBouh54DPgW2ID/0lA4VUHHVuKS/TtMZ1AiIhJJad1IQkREoksBJSIikaSAEhGRSFJAiYhIJCmgREQkkhRQIiISSQooERGJJAWUiIhE0v8DZM0V3ORMMOYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(z, selu(z), \"b-\", linewidth=2)\n",
    "plt.plot([-5, 5], [0, 0], 'k-')\n",
    "plt.plot([-5, 5], [-1.758, -1.758], 'k--')\n",
    "plt.plot([0, 0], [-2.2, 3.2], 'k-')\n",
    "plt.grid(True)\n",
    "plt.title(\"SELU activation function\", fontsize=14)\n",
    "plt.axis([-5, 5, -2.2, 3.2])\n",
    "\n",
    "save_fig(\"selu_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the SELU hyperparameters (`scale` and `alpha`) are tuned in such a way that the mean output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation 1 across all layers, avoiding the exploding/vanishing gradients problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer 0: mean -0.00, std deviation 1.00\n",
      "Layer 100: mean 0.02, std deviation 0.96\n",
      "Layer 200: mean 0.01, std deviation 0.90\n",
      "Layer 300: mean -0.02, std deviation 0.92\n",
      "Layer 400: mean 0.05, std deviation 0.89\n",
      "Layer 500: mean 0.01, std deviation 0.93\n",
      "Layer 600: mean 0.02, std deviation 0.92\n",
      "Layer 700: mean -0.02, std deviation 0.90\n",
      "Layer 800: mean 0.05, std deviation 0.83\n",
      "Layer 900: mean 0.02, std deviation 1.00\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "Z = np.random.normal(size=(500, 100)) # standardized inputs\n",
    "for layer in range(1000):\n",
    "    W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun initialization\n",
    "    Z = selu(np.dot(Z, W))\n",
    "    means = np.mean(Z, axis=0).mean()\n",
    "    stds = np.std(Z, axis=0).mean()\n",
    "    if layer % 100 == 0:\n",
    "        print(\"Layer {}: mean {:.2f}, std deviation {:.2f}\".format(layer, means, stds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using SELU is easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x7fc228758668>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.layers.Dense(10, activation=\"selu\",\n",
    "                   kernel_initializer=\"lecun_normal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"selu\",\n",
    "                             kernel_initializer=\"lecun_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"selu\",\n",
    "                                 kernel_initializer=\"lecun_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pixel_means = X_train.mean(axis=0, keepdims=True)\n",
    "pixel_stds = X_train.std(axis=0, keepdims=True)\n",
    "X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
    "X_valid_scaled = (X_valid - pixel_means) / pixel_stds\n",
    "X_test_scaled = (X_test - pixel_means) / pixel_stds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "55000/55000 [==============================] - 13s 238us/sample - loss: 1.1277 - accuracy: 0.5573 - val_loss: 0.8152 - val_accuracy: 0.6700\n",
      "Epoch 2/5\n",
      "55000/55000 [==============================] - 11s 198us/sample - loss: 0.6935 - accuracy: 0.7383 - val_loss: 0.5806 - val_accuracy: 0.7928\n",
      "Epoch 3/5\n",
      "55000/55000 [==============================] - 11s 196us/sample - loss: 0.5871 - accuracy: 0.7865 - val_loss: 0.6876 - val_accuracy: 0.7462\n",
      "Epoch 4/5\n",
      "55000/55000 [==============================] - 11s 199us/sample - loss: 0.5281 - accuracy: 0.8134 - val_loss: 0.5236 - val_accuracy: 0.8230\n",
      "Epoch 5/5\n",
      "55000/55000 [==============================] - 11s 201us/sample - loss: 0.4824 - accuracy: 0.8327 - val_loss: 0.5201 - val_accuracy: 0.8312\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look at what happens if we try to use the ReLU activation function instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "for layer in range(99):\n",
    "    model.add(keras.layers.Dense(100, activation=\"relu\", kernel_initializer=\"he_normal\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "55000/55000 [==============================] - 12s 213us/sample - loss: 1.7518 - accuracy: 0.2797 - val_loss: 1.2328 - val_accuracy: 0.4720\n",
      "Epoch 2/5\n",
      "55000/55000 [==============================] - 10s 177us/sample - loss: 1.1922 - accuracy: 0.4982 - val_loss: 1.0247 - val_accuracy: 0.5354\n",
      "Epoch 3/5\n",
      "55000/55000 [==============================] - 10s 178us/sample - loss: 0.9390 - accuracy: 0.6180 - val_loss: 1.0809 - val_accuracy: 0.5118\n",
      "Epoch 4/5\n",
      "55000/55000 [==============================] - 10s 178us/sample - loss: 0.7787 - accuracy: 0.6937 - val_loss: 0.7067 - val_accuracy: 0.7344\n",
      "Epoch 5/5\n",
      "55000/55000 [==============================] - 10s 180us/sample - loss: 0.7465 - accuracy: 0.7122 - val_loss: 0.9720 - val_accuracy: 0.5702\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not great at all, we suffered from the vanishing/exploding gradients problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_4 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_212 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "dense_213 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_214 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 271,346\n",
      "Trainable params: 268,978\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('batch_normalization/gamma:0', True),\n",
       " ('batch_normalization/beta:0', True),\n",
       " ('batch_normalization/moving_mean:0', False),\n",
       " ('batch_normalization/moving_variance:0', False)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1 = model.layers[1]\n",
    "[(var.name, var.trainable) for var in bn1.variables]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'cond/Identity' type=Identity>,\n",
       " <tf.Operation 'cond_1/Identity' type=Identity>]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bn1.updates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 3s 63us/sample - loss: 0.8760 - accuracy: 0.7122 - val_loss: 0.5509 - val_accuracy: 0.8224\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.5737 - accuracy: 0.8039 - val_loss: 0.4723 - val_accuracy: 0.8460\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.5143 - accuracy: 0.8231 - val_loss: 0.4376 - val_accuracy: 0.8570\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4826 - accuracy: 0.8333 - val_loss: 0.4135 - val_accuracy: 0.8638\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.4571 - accuracy: 0.8415 - val_loss: 0.3990 - val_accuracy: 0.8654\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 3s 53us/sample - loss: 0.4432 - accuracy: 0.8456 - val_loss: 0.3870 - val_accuracy: 0.8710\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 3s 56us/sample - loss: 0.4255 - accuracy: 0.8515 - val_loss: 0.3782 - val_accuracy: 0.8698\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.4150 - accuracy: 0.8536 - val_loss: 0.3708 - val_accuracy: 0.8758\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.4016 - accuracy: 0.8596 - val_loss: 0.3634 - val_accuracy: 0.8750\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.3915 - accuracy: 0.8629 - val_loss: 0.3601 - val_accuracy: 0.8758\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes applying BN before the activation function works better (there's a debate on this topic). Moreover, the layer before a `BatchNormalization` layer does not need to have bias terms, since the `BatchNormalization` layer some as well, it would be a waste of parameters, so you can set `use_bias=False` when creating those layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dense(300, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(100, use_bias=False),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Activation(\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "55000/55000 [==============================] - 4s 64us/sample - loss: 0.8656 - accuracy: 0.7094 - val_loss: 0.5650 - val_accuracy: 0.8098\n",
      "Epoch 2/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.5766 - accuracy: 0.8018 - val_loss: 0.4834 - val_accuracy: 0.8358\n",
      "Epoch 3/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.5184 - accuracy: 0.8216 - val_loss: 0.4461 - val_accuracy: 0.8470\n",
      "Epoch 4/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4852 - accuracy: 0.8314 - val_loss: 0.4226 - val_accuracy: 0.8558\n",
      "Epoch 5/10\n",
      "55000/55000 [==============================] - 3s 54us/sample - loss: 0.4579 - accuracy: 0.8399 - val_loss: 0.4086 - val_accuracy: 0.8604\n",
      "Epoch 6/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4406 - accuracy: 0.8457 - val_loss: 0.3974 - val_accuracy: 0.8640\n",
      "Epoch 7/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4263 - accuracy: 0.8498 - val_loss: 0.3883 - val_accuracy: 0.8676\n",
      "Epoch 8/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4152 - accuracy: 0.8530 - val_loss: 0.3803 - val_accuracy: 0.8682\n",
      "Epoch 9/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.4032 - accuracy: 0.8564 - val_loss: 0.3738 - val_accuracy: 0.8718\n",
      "Epoch 10/10\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.3937 - accuracy: 0.8623 - val_loss: 0.3690 - val_accuracy: 0.8732\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Clipping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Keras optimizers accept `clipnorm` or `clipvalue` arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipvalue=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(clipnorm=1.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Pretrained Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reusing a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the fashion MNIST training set in two:\n",
    "* `X_train_A`: all images of all items except for sandals and shirts (classes 5 and 6).\n",
    "* `X_train_B`: a much smaller training set of just the first 200 images of sandals or shirts.\n",
    "\n",
    "The validation set and the test set are also split this way, but without restricting the number of images.\n",
    "\n",
    "We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B, since classes in set A (sneakers, ankle boots, coats, t-shirts, etc.) are somewhat similar to classes in set B (sandals and shirts). However, since we are using `Dense` layers, only patterns that occur at the same location can be reused (in contrast, convolutional layers will transfer much better, since learned patterns can be detected anywhere on the image, as we will see in the CNN chapter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_dataset(X, y):\n",
    "    y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts\n",
    "    y_A = y[~y_5_or_6]\n",
    "    y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7\n",
    "    y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task: is it a shirt (class 6)?\n",
    "    return ((X[~y_5_or_6], y_A),\n",
    "            (X[y_5_or_6], y_B))\n",
    "\n",
    "(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)\n",
    "(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)\n",
    "(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)\n",
    "X_train_B = X_train_B[:200]\n",
    "y_train_B = y_train_B[:200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(43986, 28, 28)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 28, 28)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,\n",
       "       1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_A[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,\n",
       "       0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_B[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.Sequential()\n",
    "model_A.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_A.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_A.add(keras.layers.Dense(8, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 43986 samples, validate on 4014 samples\n",
      "Epoch 1/20\n",
      "43986/43986 [==============================] - 2s 49us/sample - loss: 0.5902 - accuracy: 0.8131 - val_loss: 0.3784 - val_accuracy: 0.8692\n",
      "Epoch 2/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.3517 - accuracy: 0.8784 - val_loss: 0.3369 - val_accuracy: 0.8832\n",
      "Epoch 3/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.3163 - accuracy: 0.8896 - val_loss: 0.3017 - val_accuracy: 0.8959\n",
      "Epoch 4/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2969 - accuracy: 0.8972 - val_loss: 0.2912 - val_accuracy: 0.9028\n",
      "Epoch 5/20\n",
      "43986/43986 [==============================] - 2s 45us/sample - loss: 0.2831 - accuracy: 0.9027 - val_loss: 0.2816 - val_accuracy: 0.9018\n",
      "Epoch 6/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2725 - accuracy: 0.9066 - val_loss: 0.2737 - val_accuracy: 0.9071\n",
      "Epoch 7/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2644 - accuracy: 0.9095 - val_loss: 0.2652 - val_accuracy: 0.9091\n",
      "Epoch 8/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2577 - accuracy: 0.9119 - val_loss: 0.2580 - val_accuracy: 0.9126\n",
      "Epoch 9/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2516 - accuracy: 0.9137 - val_loss: 0.2582 - val_accuracy: 0.9136\n",
      "Epoch 10/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2466 - accuracy: 0.9153 - val_loss: 0.2522 - val_accuracy: 0.9153\n",
      "Epoch 11/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2420 - accuracy: 0.9178 - val_loss: 0.2489 - val_accuracy: 0.9168\n",
      "Epoch 12/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2381 - accuracy: 0.9192 - val_loss: 0.2457 - val_accuracy: 0.9175\n",
      "Epoch 13/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2347 - accuracy: 0.9197 - val_loss: 0.2449 - val_accuracy: 0.9198\n",
      "Epoch 14/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2312 - accuracy: 0.9206 - val_loss: 0.2431 - val_accuracy: 0.9170\n",
      "Epoch 15/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2282 - accuracy: 0.9219 - val_loss: 0.2430 - val_accuracy: 0.9180\n",
      "Epoch 16/20\n",
      "43986/43986 [==============================] - 2s 44us/sample - loss: 0.2255 - accuracy: 0.9229 - val_loss: 0.2411 - val_accuracy: 0.9150\n",
      "Epoch 17/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2228 - accuracy: 0.9229 - val_loss: 0.2370 - val_accuracy: 0.9178\n",
      "Epoch 18/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2202 - accuracy: 0.9242 - val_loss: 0.2433 - val_accuracy: 0.9170\n",
      "Epoch 19/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2177 - accuracy: 0.9252 - val_loss: 0.2605 - val_accuracy: 0.9043\n",
      "Epoch 20/20\n",
      "43986/43986 [==============================] - 2s 43us/sample - loss: 0.2158 - accuracy: 0.9265 - val_loss: 0.2328 - val_accuracy: 0.9205\n"
     ]
    }
   ],
   "source": [
    "history = model_A.fit(X_train_A, y_train_A, epochs=20,\n",
    "                    validation_data=(X_valid_A, y_valid_A))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A.save(\"my_model_A.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B = keras.models.Sequential()\n",
    "model_B.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "for n_hidden in (300, 100, 50, 50, 50):\n",
    "    model_B.add(keras.layers.Dense(n_hidden, activation=\"selu\"))\n",
    "model_B.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_B.compile(loss=\"binary_crossentropy\",\n",
    "                optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/20\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.9509 - accuracy: 0.4800 - val_loss: 0.6533 - val_accuracy: 0.5568\n",
      "Epoch 2/20\n",
      "200/200 [==============================] - 0s 202us/sample - loss: 0.5837 - accuracy: 0.7100 - val_loss: 0.4825 - val_accuracy: 0.8479\n",
      "Epoch 3/20\n",
      "200/200 [==============================] - 0s 206us/sample - loss: 0.4527 - accuracy: 0.8750 - val_loss: 0.4097 - val_accuracy: 0.8945\n",
      "Epoch 4/20\n",
      "200/200 [==============================] - 0s 219us/sample - loss: 0.3869 - accuracy: 0.9050 - val_loss: 0.3630 - val_accuracy: 0.9209\n",
      "Epoch 5/20\n",
      "200/200 [==============================] - 0s 224us/sample - loss: 0.3404 - accuracy: 0.9300 - val_loss: 0.3302 - val_accuracy: 0.9280\n",
      "Epoch 6/20\n",
      "200/200 [==============================] - 0s 225us/sample - loss: 0.3073 - accuracy: 0.9350 - val_loss: 0.3026 - val_accuracy: 0.9381\n",
      "Epoch 7/20\n",
      "200/200 [==============================] - 0s 228us/sample - loss: 0.2797 - accuracy: 0.9400 - val_loss: 0.2790 - val_accuracy: 0.9452\n",
      "Epoch 8/20\n",
      "200/200 [==============================] - 0s 221us/sample - loss: 0.2554 - accuracy: 0.9450 - val_loss: 0.2595 - val_accuracy: 0.9473\n",
      "Epoch 9/20\n",
      "200/200 [==============================] - 0s 232us/sample - loss: 0.2355 - accuracy: 0.9600 - val_loss: 0.2439 - val_accuracy: 0.9493\n",
      "Epoch 10/20\n",
      "200/200 [==============================] - 0s 220us/sample - loss: 0.2187 - accuracy: 0.9650 - val_loss: 0.2293 - val_accuracy: 0.9523\n",
      "Epoch 11/20\n",
      "200/200 [==============================] - 0s 210us/sample - loss: 0.2041 - accuracy: 0.9650 - val_loss: 0.2162 - val_accuracy: 0.9544\n",
      "Epoch 12/20\n",
      "200/200 [==============================] - 0s 223us/sample - loss: 0.1906 - accuracy: 0.9650 - val_loss: 0.2049 - val_accuracy: 0.9574\n",
      "Epoch 13/20\n",
      "200/200 [==============================] - 0s 211us/sample - loss: 0.1791 - accuracy: 0.9700 - val_loss: 0.1946 - val_accuracy: 0.9594\n",
      "Epoch 14/20\n",
      "200/200 [==============================] - 0s 214us/sample - loss: 0.1686 - accuracy: 0.9750 - val_loss: 0.1856 - val_accuracy: 0.9615\n",
      "Epoch 15/20\n",
      "200/200 [==============================] - 0s 203us/sample - loss: 0.1591 - accuracy: 0.9750 - val_loss: 0.1765 - val_accuracy: 0.9655\n",
      "Epoch 16/20\n",
      "200/200 [==============================] - 0s 222us/sample - loss: 0.1502 - accuracy: 0.9900 - val_loss: 0.1695 - val_accuracy: 0.9655\n",
      "Epoch 17/20\n",
      "200/200 [==============================] - 0s 235us/sample - loss: 0.1424 - accuracy: 0.9900 - val_loss: 0.1624 - val_accuracy: 0.9686\n",
      "Epoch 18/20\n",
      "200/200 [==============================] - 0s 224us/sample - loss: 0.1351 - accuracy: 0.9900 - val_loss: 0.1567 - val_accuracy: 0.9686\n",
      "Epoch 19/20\n",
      "200/200 [==============================] - 0s 207us/sample - loss: 0.1290 - accuracy: 0.9900 - val_loss: 0.1513 - val_accuracy: 0.9696\n",
      "Epoch 20/20\n",
      "200/200 [==============================] - 0s 213us/sample - loss: 0.1229 - accuracy: 0.9900 - val_loss: 0.1450 - val_accuracy: 0.9696\n"
     ]
    }
   ],
   "source": [
    "history = model_B.fit(X_train_B, y_train_B, epochs=20,\n",
    "                      validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_5 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 784)               3136      \n",
      "_________________________________________________________________\n",
      "dense_215 (Dense)            (None, 300)               235200    \n",
      "_________________________________________________________________\n",
      "batch_normalization_4 (Batch (None, 300)               1200      \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_216 (Dense)            (None, 100)               30000     \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 100)               400       \n",
      "_________________________________________________________________\n",
      "dense_217 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 270,946\n",
      "Trainable params: 268,578\n",
      "Non-trainable params: 2,368\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A = keras.models.load_model(\"my_model_A.h5\")\n",
    "model_B_on_A = keras.models.Sequential(model_A.layers[:-1])\n",
    "model_B_on_A.add(keras.layers.Dense(1, activation=\"sigmoid\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_A_clone = keras.models.clone_model(model_A)\n",
    "model_A_clone.set_weights(model_A.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = False\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/4\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.5619 - accuracy: 0.6650 - val_loss: 0.5669 - val_accuracy: 0.6531\n",
      "Epoch 2/4\n",
      "200/200 [==============================] - 0s 208us/sample - loss: 0.5249 - accuracy: 0.7200 - val_loss: 0.5337 - val_accuracy: 0.6957\n",
      "Epoch 3/4\n",
      "200/200 [==============================] - 0s 200us/sample - loss: 0.4923 - accuracy: 0.7400 - val_loss: 0.5039 - val_accuracy: 0.7211\n",
      "Epoch 4/4\n",
      "200/200 [==============================] - 0s 214us/sample - loss: 0.4630 - accuracy: 0.7550 - val_loss: 0.4773 - val_accuracy: 0.7383\n",
      "Train on 200 samples, validate on 986 samples\n",
      "Epoch 1/16\n",
      "200/200 [==============================] - 0s 2ms/sample - loss: 0.3864 - accuracy: 0.8200 - val_loss: 0.3357 - val_accuracy: 0.8661\n",
      "Epoch 2/16\n",
      "200/200 [==============================] - 0s 207us/sample - loss: 0.2701 - accuracy: 0.9350 - val_loss: 0.2608 - val_accuracy: 0.9249\n",
      "Epoch 3/16\n",
      "200/200 [==============================] - 0s 226us/sample - loss: 0.2082 - accuracy: 0.9650 - val_loss: 0.2150 - val_accuracy: 0.9503\n",
      "Epoch 4/16\n",
      "200/200 [==============================] - 0s 212us/sample - loss: 0.1695 - accuracy: 0.9800 - val_loss: 0.1840 - val_accuracy: 0.9625\n",
      "Epoch 5/16\n",
      "200/200 [==============================] - 0s 226us/sample - loss: 0.1428 - accuracy: 0.9800 - val_loss: 0.1602 - val_accuracy: 0.9706\n",
      "Epoch 6/16\n",
      "200/200 [==============================] - 0s 236us/sample - loss: 0.1221 - accuracy: 0.9850 - val_loss: 0.1424 - val_accuracy: 0.9797\n",
      "Epoch 7/16\n",
      "200/200 [==============================] - 0s 218us/sample - loss: 0.1067 - accuracy: 0.9950 - val_loss: 0.1293 - val_accuracy: 0.9828\n",
      "Epoch 8/16\n",
      "200/200 [==============================] - 0s 229us/sample - loss: 0.0952 - accuracy: 0.9950 - val_loss: 0.1186 - val_accuracy: 0.9848\n",
      "Epoch 9/16\n",
      "200/200 [==============================] - 0s 224us/sample - loss: 0.0858 - accuracy: 0.9950 - val_loss: 0.1099 - val_accuracy: 0.9848\n",
      "Epoch 10/16\n",
      "200/200 [==============================] - 0s 241us/sample - loss: 0.0781 - accuracy: 1.0000 - val_loss: 0.1026 - val_accuracy: 0.9878\n",
      "Epoch 11/16\n",
      "200/200 [==============================] - 0s 234us/sample - loss: 0.0719 - accuracy: 1.0000 - val_loss: 0.0964 - val_accuracy: 0.9888\n",
      "Epoch 12/16\n",
      "200/200 [==============================] - 0s 222us/sample - loss: 0.0664 - accuracy: 1.0000 - val_loss: 0.0906 - val_accuracy: 0.9888\n",
      "Epoch 13/16\n",
      "200/200 [==============================] - 0s 228us/sample - loss: 0.0614 - accuracy: 1.0000 - val_loss: 0.0862 - val_accuracy: 0.9899\n",
      "Epoch 14/16\n",
      "200/200 [==============================] - 0s 225us/sample - loss: 0.0575 - accuracy: 1.0000 - val_loss: 0.0818 - val_accuracy: 0.9899\n",
      "Epoch 15/16\n",
      "200/200 [==============================] - 0s 219us/sample - loss: 0.0537 - accuracy: 1.0000 - val_loss: 0.0782 - val_accuracy: 0.9899\n",
      "Epoch 16/16\n",
      "200/200 [==============================] - 0s 221us/sample - loss: 0.0505 - accuracy: 1.0000 - val_loss: 0.0752 - val_accuracy: 0.9899\n"
     ]
    }
   ],
   "source": [
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,\n",
    "                           validation_data=(X_valid_B, y_valid_B))\n",
    "\n",
    "for layer in model_B_on_A.layers[:-1]:\n",
    "    layer.trainable = True\n",
    "\n",
    "model_B_on_A.compile(loss=\"binary_crossentropy\",\n",
    "                     optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "                     metrics=[\"accuracy\"])\n",
    "history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,\n",
    "                           validation_data=(X_valid_B, y_valid_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what's the final verdict?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 28us/sample - loss: 0.1426 - accuracy: 0.9695\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.14263125681877137, 0.9695]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 25us/sample - loss: 0.0697 - accuracy: 0.9925\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.06966363421082497, 0.9925]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_B_on_A.evaluate(X_test_B, y_test_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We got quite a bit of transfer: the error rate dropped by a factor of 4!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.933333333333337"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(100 - 96.95) / (100 - 99.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faster Optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Momentum optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nesterov Accelerated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adagrad(lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.RMSprop(lr=0.001, rho=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adamax Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nadam Optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=0.001, beta_1=0.9, beta_2=0.999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Power Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 / (1 + steps / s)**c```\n",
    "* Keras uses `c=1` and `s = 1 / decay`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, decay=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 2s 45us/sample - loss: 0.4842 - accuracy: 0.8319 - val_loss: 0.4167 - val_accuracy: 0.8572\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.3789 - accuracy: 0.8646 - val_loss: 0.3781 - val_accuracy: 0.8678\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3461 - accuracy: 0.8768 - val_loss: 0.3690 - val_accuracy: 0.8706\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3252 - accuracy: 0.8836 - val_loss: 0.3532 - val_accuracy: 0.8770\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3091 - accuracy: 0.8896 - val_loss: 0.3454 - val_accuracy: 0.8814\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2970 - accuracy: 0.8938 - val_loss: 0.3431 - val_accuracy: 0.8798\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2862 - accuracy: 0.8975 - val_loss: 0.3377 - val_accuracy: 0.8850\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2774 - accuracy: 0.9010 - val_loss: 0.3324 - val_accuracy: 0.8850\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2696 - accuracy: 0.9036 - val_loss: 0.3297 - val_accuracy: 0.8884\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2625 - accuracy: 0.9054 - val_loss: 0.3309 - val_accuracy: 0.8852\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2564 - accuracy: 0.9083 - val_loss: 0.3255 - val_accuracy: 0.8894\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2508 - accuracy: 0.9101 - val_loss: 0.3242 - val_accuracy: 0.8900\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2458 - accuracy: 0.9119 - val_loss: 0.3239 - val_accuracy: 0.8894\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2413 - accuracy: 0.9145 - val_loss: 0.3248 - val_accuracy: 0.8902\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2371 - accuracy: 0.9161 - val_loss: 0.3201 - val_accuracy: 0.8900\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2328 - accuracy: 0.9171 - val_loss: 0.3223 - val_accuracy: 0.8894\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2295 - accuracy: 0.9187 - val_loss: 0.3181 - val_accuracy: 0.8908\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2254 - accuracy: 0.9190 - val_loss: 0.3220 - val_accuracy: 0.8890\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2224 - accuracy: 0.9216 - val_loss: 0.3205 - val_accuracy: 0.8884\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 2s 42us/sample - loss: 0.2194 - accuracy: 0.9234 - val_loss: 0.3178 - val_accuracy: 0.8920\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 2s 42us/sample - loss: 0.2166 - accuracy: 0.9237 - val_loss: 0.3174 - val_accuracy: 0.8918\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2135 - accuracy: 0.9252 - val_loss: 0.3185 - val_accuracy: 0.8888\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2110 - accuracy: 0.9253 - val_loss: 0.3174 - val_accuracy: 0.8924\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 2s 38us/sample - loss: 0.2084 - accuracy: 0.9270 - val_loss: 0.3193 - val_accuracy: 0.8894\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2063 - accuracy: 0.9274 - val_loss: 0.3179 - val_accuracy: 0.8934\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxU1f3/8dcnCdkIJGwCiexiZFFEWkHRr9Sl2GorVWtrrVWr9Vdbv21tq8VWKy6trVpbbW0rX6UuVavWtaKiiKjgSlVkX1RA2ddAIIGQfH5/3BscJpNkApmZJPN+Ph7zyMy9Z8793GvMh3PPueeYuyMiItLcMlIdgIiItE1KMCIikhBKMCIikhBKMCIikhBKMCIikhBKMCIikhBKMCKtiJktM7OfJ6DeM82sSc8smNn5ZlZe32cRJRhpVczsHjPz8FVlZh+Z2S1m1j7VscXDzC4ys/fMrNzMyszsAzO7IdVxNZOHgf6pDkJajqxUByCyD6YC5wLtgGOBu4D2wCWpDKqWmWW7+64Y278L3A5cBrwEZANDgKOSG2FiuHsFUJHqOKTlUAtGWqOd7r7G3T9x9weBB4BxtTvN7H/M7C0zqzSztWb2RzPLDvd9ycy2mVlW+Hlg2Br6W8T3f2NmL0Z8Hmxmk8PvrTOzh8ysR8T+e8zsGTP7hZl9CnxaT9xfBR539zvdfam7z3f3R939p5GFzOyUMP4KM9toZv8xs9yIIrlmdqeZbTWzT83s8qjvF5rZxDDWbWb2ipl9LqrMd8xsuZntMLNngO5R+yeY2dyobQ3eAotxy2yCmc01s2+a2YdhLE+aWdeIMlnhf5/N4euPZvY3M5te33Gk9VCCkbaggqA1g5mVAM8B7wHDgQuBs4Ebw7KvAblA7R/cMcAG4AsR9Y0Bpof19QReBeYCRwInAgXA02YW+f/PccBhwMnACfXEuQY40szqvY1kZicDTwEvAiPCuF5h7/9XLwPmAEcAvwduMrOjwu8bMBkoAU4Nr8GrwLTwXDCzkcA9wETgcOA/wHX1xbSf+gLfAL4GfDGM5zcR+38OnA9cBIwiOM9vJSgWSTZ310uvVvMi+MP4TMTnIwkSxMPh598AS4GMiDLnAzuB/PDzW8CV4fsHgGsIklRPIB/YBYwO918HvBQVQyfAgSMjYloP5DQSe0/gjfC7S4B/At8B2kWUmQn8q4E6lgEPRW1bAlwVvj8eKAfyosq8D1wRvn8QeDFq/13Bn4M9nycAc6PKnA+UN+HzBKASKIzY9itgacTn1cD4iM8GLASmp/p3Ta/9f6kFI63RyWEneSXBH+xXgf8N9w0C3nD3mojyMwj6Ow4KP08naKVA0PJ4Dng73DYaqAo/Q9CK+J/weOXhLaBPwn0DIo4x1913NhS0u69296OAQ4E/EfwxvRN428zyw2LDCfpnGvJB1OdVwAER8eYD66NiHhoR7yCC6xYp+nNzWe7uZbFiNbNCoAefXesgw8E7CYpFkkyd/NIavQpcTJAIVrl7VcQ+I2ghxFK7fTrwQzMbDHQA/htu+wJBS+T1iDozCG45xRoavDbi/fZ4g3f3uQS33O4ws2MIbtudRdASikdV1Gfns1toGWFcx8b43tbwp8VxjJoY5drFGV+khmKN3CZtkBKMtEY73H1pPfvmA2eZWUZEK+YYgtteH4afXwNygCuAGe5eHXYqTwTWAc9G1PcuwR//5VGJrLnMD38WhD/fI+jD+b99rO9dgg77Gnf/qIFjjoraFv15PdDdzCxsVUDQX9Ns3L3MzNYQ3OZ8Gfb0IX2eoL9KWjndIpO25q9AMfBXMxtkZqcAvwP+4u47ANy9nOAP8bcJ/7AR3CLqBYwk7OAP3QEUAg+b2Ugz629mJ4ajtDo0JbBwdNTVZjbazPqY2SjgPmAH8EJY7DfA183shnD02hAzuyziFlpjphL04zwVjpjrZ2ZHmdm1ZlbbqrkdONHMrgxH0X2PoBM+0nSgM/BLMxtgZhcCZzblfON0G3CFmX3NzEqBPxD0ValV0wYowUib4u4rgS8R9GW8D0wCHgJ+GVX0ZSCTMJm4eyXwJsFggMg+gVUE/TI1wPPAPIKkszN8NcWLBAnsEWAx8ES4/SR3Xxwe71mCP/ZfImjNvEJw666mTm0xhK2NLwPTCFpBi8LjlRL0f+DubxKMrruEoD/ndIIO+ch6FoT7Lw7LnAT8tonnG49bgPuBfxBcfwiuS2UCjiVJZp+1fkVEUs/M3gVmuvv/NlpYWjT1wYhIyphZH2AsQUsti6DFNCz8Ka1cUm+RmVlnM3vCzLaHTxHX+0BVeN95jQXzNU0ys5yIfZea2Swz22lm98T47glmtjB8Svnl8JdYRFqeGoJngd4muEU2CviSu89KaVTSLJJ6i8zMHiJIahcSjEiZDBzt7vOiyo0l6Pw8nuC+8RPAm+4+Ptx/OsEv5liCB8rOj/huV4LRQhcRPKF8PXCsu0ePkhERkQRKWoKxYLbbzcDQ2g5NM7sfWFmbOCLKPggsc/dfhp9PAB5w9x5R5W4ADoxKMBcD57v70RHH3QAMd/eFiTo/ERHZWzL7YA4GqmuTS2g2wZPU0YYQzMcUWa67mXVx942NHGdIWB4Ad99uZh+G2/dKMGEyuhggI6/jiKzCA/bs69tRA+wAampqyMjQtYim6xKbrktdbf2aLF68eIO7d4u1L5kJpgAoi9pWRvAkdWNla993ABpLMAUED4k1ehx3n0jwcB05PQd6z/P+BEBJUR4zxx/fyGHSw/Tp0xkzZkyqw2hxdF1i03Wpq61fEzNbXt++ZKbVcqBj1LaOwLY4yta+j1V2f45TR05WBpePLY2nqIiINCCZCWYxkGVmAyO2DSN4cC3avHBfZLm1cdweq/PdsA9mQD3H2YsBQ4s7Mm54SRyHERGRhiQtwbj7duBx4Doza29mo4HTCJ7ijXYfcGE4VUYn4CoiJgIMFynKJXgSO9PMci1cQIpgxNlQMzsjLPNr4IPGOvj7dszggtH9mP1pGWu36iFiEZH9leyepx8AeQQTCj4EXOLu88ysdziteG8Ad38euIlgOo/l4euaiHquIli/YzzBfFIV4TbcfT1wBsGcTpsJpub4ZjzBnX90X6rduf+Nem8piohInJL6JL+7byJiaduI7Sv4bDbZ2m23ArfWU88EouZOito/FTikqfH17pLPSYO688Bby7n0+IPIbZfZ1CpERCTUdsfO7aPvHtOPzTuqePK9lakORUSkVVOCiTKyX2cG9+zIpJkfo4lARUT2nRJMFDPju8f0Y/HacmYujWfQmoiIxKIEE8NXhvWka0E2k2Z+nOpQRERaLSWYGHKyMvn2qD5MW7iOj9aXpzocEZFWSQmmHueM7EN2Zgb3vL4s1aGIiLRKSjD16NYhh68eXsyjsz6lbEdVqsMREWl1lGAacMHovlRUVfPwrBWpDkVEpNVRgmnAkOJCRvXvzL2vL2d3dU2qwxERaVWUYBrx3dH9WLmlghfmr011KCIirYoSTCNOGNSd3p3zmTRDQ5ZFRJpCCaYRmRnG+Uf3Zdbyzcz+ZEuqwxERaTWUYOJw1ud70SEni3/owUsRkbgpwcShICeLsz7fi2c+WK21YkRE4qQEE6fzj+5LjdaKERGJmxJMnHp1zuekwcFaMZVV1akOR0SkxVOCaYLvjg7WinlCa8WIiDRKCaYJjuzXmSHFHZk0Q2vFiIg0RgmmCcyM747ux5J15cxYuiHV4YiItGhKME106rCedC3I0YOXIiKNUIJpopysTM4d1YeXF63nQ60VIyJSLyWYfXDOqN7BWjEzl6U6FBGRFisr1QG0Rl0Lcji8VyH/fHM5/3xzOcVFeVw+tpRxw0tSHZqISIuhBLMPnnxvJbM/LaN2HNnKLRVc+fgcACUZEZGQbpHtg5unLGLn7r3Xh6moqubmKYtSFJGISMujBLMPVm2paNJ2EZF0pASzD4qL8pq0XUQkHSnB7IPLx5aS1y5zr22ZGcblY0tTFJGISMujTv59UNuRf/OURazaUkF+Tibbd1YzoFtBiiMTEWk5lGD20bjhJXsSzdbKKo6/ZTrXPD2Xxy45GjNLcXQiIqmnW2TNoGNuO644+RDeXbFFMy2LiISUYJrJmUccyLBeRdz43ELKd+5OdTgiIimnBNNMMjKMa786hPXbdvLnl5akOhwRkZRTgmlGh/cq4usjDmTSzI81EaaIpL2kJhgz62xmT5jZdjNbbmbfaqDsZWa2xszKzGySmeXEW4+ZnWVmC8xsm5nNN7NxiTyvSFecfAi5WZlc95/5WpRMRNJaslswdwC7gO7AOcDfzGxIdCEzGwuMB04A+gL9gWvjqcfMSoB/Aj8FOgKXAw+a2QGJOaW9deuQw49PHMgri9fz0oJ1yTikiEiLlLQEY2btgTOAq9293N1nAE8D58Yofh5wt7vPc/fNwPXA+XHWcyCwxd2f88BkYDswIIGnt3fwR/floAMKuO6Z+VRWVSfrsCIiLUoyn4M5GKh298UR22YDx8UoOwR4KqpcdzPrAvRupJ5ZwAIz+yowGfgKsBP4IPogZnYxcDFAt27dmD59+j6cVmxf613NzbMq+dV9L/GVAdnNVm+ylZeXN+t1aSt0XWLTdakrna9JMhNMAVAWta0M6BBH2dr3HRqrx92rzew+4EEgl+BW2tfdfXv0Qdx9IjARoLS01MeMGdOE02nYGGBOxX95dvF6fnbGyFY7T9n06dNpzuvSVui6xKbrUlc6X5Nk9sGUE/SJROoIbIujbO37bY3VY2YnAjcR/I3PJmjZ3GVmh+9H7PvkV6cMosadG59bmOxDi4ikXDITzGIgy8wGRmwbBsyLUXZeuC+y3Fp33xhHPYcDr7r7LHevcfd3gLeAE5vpPOLWq3M+3z9uAP+ZvYo3P9qY7MOLiKRU0hJMeIvqceA6M2tvZqOB04D7YxS/D7jQzAabWSfgKuCeOOt5Bzi2tsViZsOBY4nRB5MM3z9uACVFeUx4eh67q2sa/4KISBuR7GHKPwDygHXAQ8Al7j7PzHqbWbmZ9QZw9+cJbnO9DCwPX9c0Vk/43VeACcC/zWwb8BjwW3d/IQnnV0dediZXnTKIhWu28eDbK1IRgohISiR1NmV33wTUeejR3VcQdN5HbrsVuLUp9UTs/wvwl/0KthmdPLQHRw/owh9eWMyphxXTuX3rHVUmIhIvTRWTBGbGhK8OoXznbm55YVGqwxERSQolmCQ5uHsHvnNUHx56ewVzV0aPshYRaXu04FgS/eTEg3nknU84/a+vU1VdQ3FRHpePLd2zcJmISFuiBJNELy9cx67qGqqqg0kwV26p4MrH5wAoyYhIm6NbZEl085RFe5JLrYqqam6eon4ZEWl7lGCSaNWWiiZtFxFpzZRgkqi++cha6zxlIiINUYJJosvHlpLXLrPO9tOPUP+LiLQ9SjBJNG54CTeefiglRXkY0LMwl67t2/HIrE/YUL4z1eGJiDQrjSJLsnHDS/YaMTZ/1Va+9teZ/ORf73Pvd48kM8NSGJ2ISPNRCybFBhd35LrThjBj6QZuf2lJqsMREWk2cScYM+tuZj83s7+ZWddw22gz65e48NLDWZ/rxelHlHD7tCW8tmR9qsMREWkWcSUYMxsBLALOAS7kswW/TgJ+k5jQ0oeZccO4oQw8oICf/Ot91pRVpjokEZH9Fm8L5hbgNncfTrC+fa0pwOhmjyoN5Wdn8ddzRlBRVc2lD75LldaOEZFWLt4EMwK4N8b21UD35gsnvR10QAE3nn4os5Zv5hY93S8irVy8CaYC6BRj+yEEi35JMznt8BK+Pao3d776ES/OX5vqcERE9lm8CeYp4Bozywk/u5n1BX5PsGKkNKOrTx3MoSWF/OyR9/lk045UhyMisk/iTTA/BzoD64F8YAawFNgCXJWY0NJXTlYmd3zrCBz4wQPvsnN3dapDEhFpsrgSjLtvdfdjCJYp/gVwG3Cyux/n7tsTGWC66t0lnz98fRhzVpZxwzMLUh2OiEiTxTtM+TtmluPu09z9Fne/yd2nmlm2mX0n0UGmqy8O6cHF/9Of+99cztOzV6U6HBGRJon3Ftk/gMIY2zuE+yRBLh9byuf6dOJnj7zPkb+ZSr/xkxn9u2k8+d7KVIcmItKgeBOMAR5je29AC8wnULvMDE4d1pOqamfdtp04n62EqSQjIi1Zg5NdmtkcgsTiwCtmtjtidybQB3g2ceEJwP+9+nGdbbUrYWqpZRFpqRqbTfnf4c+hwGSgPGLfLmAZGqaccFoJU0RaowYTjLtfC2Bmy4CH3V2TZKVAcVEeK2MkE62EKSItWbzDlO9Vckmd+lbCHNkv1uQKIiItQ7zDlLPN7FozW2xmlWZWHflKdJDpLnolzOKiXIYWd+Tx91bx8DsrUh2eiEhM8a5oeT3wDeBG4I/A5UBf4JvA1QmJTPYSvRLmzt3VfO++/zL+8TnktsvktMPV2S8iLUu8w5TPAr7v7ncC1cBT7v4j4BqCNWEkyXKyMrnz2yP4fN/O/PSR2bwwb02qQxIR2Uu8CaY7MD98Xw4Uhe+fB77Y3EFJfPKyM5l0/uc5tKSQSx98j1cXazVMEWk54k0wK4Di8P1SYGz4/iiCqfwlRQpysrj3giMZcEABF98/i7c+2pjqkEREgPgTzBPACeH724Brzexj4B7grgTEJU1QmN+O+y88kpKiPC68dxbvf7Il1SGJiMQ9TPlKd/9N+P7fwDHAn4HT3f1XCYxP4tS1IIcHLhpFp/btOG/S2yxYvTXVIYlImou3BbMXd3/L3W9192fMrH283zOzzmb2hJltN7PlZvatBspeZmZrzKzMzCZFLHbWaD1mlm9mfzWzDeH3X92X82xtehTm8uBFo8jPzuTcu99i6bryxr8kIpIg+5RgAMws18wuB+pOlFW/OwimmOkOnAP8zcyGxKh7LDCe4LZcX6A/cG0T6plIsEDaoPDnZU2IsVXr1Tmff140EoBv3/WWVsQUkZRpbLLLbIKhyF8EqoCb3P3JcA2Y3xFMgvnHeA4UtnTOAIa6ezkww8yeBs4lSCaRzgPudvd54XevBx4AxjdWj5mVAl8FDnT32vtE/40nxrZiQLcC7r9wJN+c+Can3TGD7MwM1m7dSXFRHpePLdUEmSKSFI09aDkB+CHwIjAaeNTM/o+gZXEl8KC7V8V5rIOBandfHLFtNnBcjLJDgKeiynU3sy4ESwQ0VM9IYDnBQIRzgdXABHevMymnmV0MXAzQrVs3pk+fHueptA7HlzhPLP1sAuyVWyq44tH3mb9gPkcXt4urjvLy8jZ3XZqDrktsui51pfM1aSzBnAWc7+5PmNkw4D2gEzDE3Xc3/NU6Cqi7dkwZwaJljZWtfd8hjnoOJJj9+TGCodVHAZPNbL6777X2sLtPJLidRmlpqY8ZM6YJp9Py/erNacDe/5l21cDkFZn88ltj4qpj+vTptLXr0hx0XWLTdakrna9JY30wvYB3ANx9NkG/x+/3IblA8IBmx6htHYFtcZStfb8tjnoqCG7n3eDuu9z9FeBl0vCBUE3zLyKp1FiCaQfsjPhcxb6vYLkYyDKzgRHbhgHzYpSdF+6LLLfW3TfGUc8H+xhfm1PfdP6d2mcnORIRSUfxjCK70cxuN7PbgWxgQu3niO2NcvftwOPAdWbW3sxGA6cB98cofh9woZkNNrNOwFUED3XGU8+rBDMPXGlmWeH+McCUeOJsS2JN828Gm7bvYtKMj3GPtQq2iEjzaKwP5lVgQMTn1wk62SM15a/UD4BJwDpgI3CJu88zs94Ec50NdvcV7v68md1EcGsrj6A/5ZrG6gFw9yozO41ghoHxBB3+33H3hU2Is02oHS1285RFrNpSQXFRHj8+4SBeWriO656Zz7KN2/n1qYPJytzn0eoiIvVqbEXLMc15MHffBIyLsX0FQed95LZbgVubUk/E/nkEnftpL3qaf4AzR/Ti988v5M5XP2LFph38+ezhdMiNb1SZiEi89E/XNJSRYVz55UHcePqhvLZkA1//+xsxl2QWEdkfSjBp7Owje3PvBUeycksFp/1lJrM1SaaINCMlmDR3zMCuPH7J0eS2y+AbE9/g+bmrUx2SiLQRSjDCwO4dePKHoxnUsyOXPPAud77yoUaYich+a2wUmaSJrgU5PPS9Ufz80dnc+NxCpi9ax/JNO1i1pZKSN6dpDjMRabK4Ekw4jDgWByrdXWv1tgG57TK5/ZvD2bW7mhfmr9uzfeWWCq58fA6AkoyIxC3eW2TLCKblj34tA9aY2WYzu9XM1CJq5TIyjHmr6s7eU1FVzc1TFqUgIhFpreJNCGcDNwF/B94Kt40kmIl4AlBE8LT9NvZ+IFJaIc1hJiLNId4Ecwlwmbs/HrFtmpktAn7s7seZ2TqCRcGUYFq54qK8mM/FZGdlsLqsgp6Fsec4ExGJFO8tspHAnBjb5wKfD9+/QTBVvrRyseYwa5dp1NQ4J//pNZ6do6HMItK4eBPMcsKFuaJ8j2BiSYBuwKbmCEpSa9zwEm48/VBKwtmYS4ryuPnMYbzw0+Po2yWfHzzwLpc/OpvynfuyaoOIpIt4b5H9DHjMzL5MsD6ME7RcBhAsX0z4+ZFmj1BSonYOs+jFkv59ydHcNnUJf52+lLeXbeJP3zic4b07pS5QEWmx4mrBuPtkYCDwNMHiXkXh+1J3fzYs81d3/2miApWWoV1mBj8fW8q/Lj6K3dXOmX9/g9tfWsLu6ppUhyYiLUzcw4rd/RPgygTGIq3Ikf0689xPjuXqJ+dy64uLeXXxev74jcPp1Tk/1aGJSAsRd4Ixs3zgcOAAolo+UaPLJE10zG3Hbd8czhdKD+DqJ+fypdte47ThPZm+cD2rtlRSXJSnGQBE0li8T/KfCDwEdImx24HMGNslTYwbXsKIPp34zqS3eODNT/Zs1wwAIukt3lFktwGTgQPdPSPqpeQi9Oqcz87ddfthNAOASPqK9xZZX+Cr7r4qgbFIK7d6S2XM7ZoBQCQ9xduCmQmUJjIQaf2Ki2I/4e/Ab59doOdmRNJMvAnm78AtZnaRmY00syMiX4kMUFqPWDMA5LbLYGS/Tkx89SNO+MN0np69SmvNiKSJeG+R/Tv8OTHGPnXyC/BZR/7NUxaxakvFXqPI3l2xmV8/NZcfPfQeD721gmtPG8LB3TukOGIRSaR4E0y/hEYhbUbtDADRjujdiad+eAwPvr2CW6Ys4su3vcYFo/vy4xMPpiBHqzyItEVx/Z/t7ssTHYi0fZkZxrmj+nDKoT256fmF3DXjY556fxW/OmUQNTXOLS8srtPyEZHWq94EY2anA/9x96rwfb30oKU0Ref22fzujMP45pG9+fVTc/nxv94nw6Am7JrR8zMibUNDLZh/Az2AdXzWBxOL+mBknxzeq4gnfjCaEde/yJaKqr321T4/owQj0nrVm2DcPSPWe5HmlJlhlEUll1p6fkakdVPikJSr7/kZgFtfWFRvAhKRlq0pk132Ao4l9mSXtzZzXJJGLh9bypWPz6GiqnrPtpysDAb16MDt05Zyz+vL+N6x/bngmH4acSbSisQ72eU5wCRgN7CeoN+llgNKMLLPGnp+Zt6qMv744hL+8OJiJs38mP933AC+c1Qf8rOVaERaunj/L70O+ANwtbtXN1ZYpKnqe35mSHEhd533OWZ/soU/Tl3M755byF2vfcQlYw7inJG9eX7umpiJSURSL94E0x24S8lFUmVYryLuueBI/rt8E7e+uJjrn5nPbVMXUVFVQ1V10KDW8GaRliXeTv5ngZGJDEQkHiP6dOaBi0bx0PdGUVnle5JLLS0PINJyxNuCeRH4vZkNAeYAew3r0YOWkmxHDehCVXXd9WdAw5tFWop4E8yd4c9fxtinBy0lJYqL8lgZI5k48LNHZnPhMf0YXNwx+YGJCBDnLbIYq1ju04qWZtbZzJ4ws+1mttzMvtVA2cvMbI2ZlZnZJDPLaWo9ZnaNmXm45LO0MbGWB8jJyuDYgV15bu5qvnz7a5w98U2mzl9LTY2WCBBJtkZbMGbWDpgBfMfd9/fm9h3ALoJBA4cDk81strvPizrmWGA8cDywCngCuDbcFlc9ZjYAOBNYvZ8xSwvV0PDmsooq/vX2Cu59fRkX3TeL/l3bc8Hovpwx4kDys7N48r2VGn0mkmCNJphwsst+7P3sS5OZWXvgDGCou5cDM8zsaeBcPksctc4D7q5NGGZ2PfAAML4J9fwF+AXw1/2JW1q2+oY3F+a14/8dN4DvHtOP5+au4e4ZH3P1U/O45YXFfK5vJ2Yu2UDl7qAPR6PPRBIj3j6Ye4HvAZfvx7EOBqrdfXHEttnAcTHKDgGeiirX3cy6AL0bq8fMvg7scvdnzazegMzsYuBigG7dujF9+vQmnVA6KC8vb/XXpSPwk8HO0uJcpiyr4qUF6+qUqaiq5vqnZlNUtiSuOtvCdUkEXZe60vmaxJtg2gPnmNlJwH+B7ZE73f1HcdRRAJRFbSsDYi1rGF229n2HxuoxswLgt8AXGwvI3ScSrtJZWlrqY8aMaewraWf69Om0levyBYJ/JfUbPzlmc3xTpcd9rm3pujQnXZe60vmaxPsczCDgXWAz0B84NOI1NM46ygn+MRmpI7AtjrK177fFUc+1wP3u/nGccUmaqW9yTQd++MC7vLJ4PdUaFCCy3+Jd0fILzXCsxUCWmQ1099r7EMOAeTHKzgv3PRJRbq27bzSzykbqOQE40Mx+EH7uBjxiZr939983w3lIK1ff5Jqj+nXm9Q83MHnOaooLcznzc734+ogD6dU5P4XRirReSZsx0N23m9njwHVmdhHB6K/TgKNjFL8PuMfMHiAYBXYVcE+c9ZwAtIuo6x3gp8BzzX5S0io1NPps5+5qps5fx8OzPuHP05bw52lLGD2gK2d9vhdfHNx9z9xnK7dUUPLmNI0+E2lAU6br/wJwNkEne3bkPnc/Ps5qfkAwK/M6YCNwibvPM7PewHxgsLuvcPfnzewm4GUgD3gMuKaxesJYNkbFXQ1sDkeciQD1jz7LycrklMN6csphPVm5pYJHZ33Co7M+5UcPvUdeuwx2Vfue22cafSbSsLj6YMzsfIIWQAdgDMGU/Z2AIwgSQ1zcfbIfppoAABPWSURBVJO7j3P39u7e290fDLevcPcCd18RUfZWd+/u7h3d/QJ339lYPfUcs6+7T403RpFaJUV5/OTEg3ntii9w/4VHAtTpm9HcZyL1i7eT/+fApe5+NsE8ZFe6+3DgnwSd7iJtVkaGcezAblRWxZ77bOWWCu567SNWl2kONJFI8SaY/kBtK2AnwVBhCB5mPL+ZYxJpkeobfdYu07hh8gKOunEaX//769z7+jLWbatMcnQiLU+8fTAb+ex5lZUEQ5M/ALoQ9JGItHmxRp/ltcvkxtMPZVivIp6ZvYpnPljNNU/P49r/zGNU/y6celgxXxrag1cWr9fUNJJ24k0wrxE8uDiHYOjw7eFDlycQTOUv0uZFjj5buaWCkqhE8b8nDOR/TxjI4rXb9iSbXz4xh189MQczqO2+0eAASRfxJphLgdzw/Y3AbmA0QbK5IQFxibRItaPPGno6++DuHfjpF0u57KSDmb96K9+4803Kd+7eq0xFVTU3PrdACUbatHgftNwU8b4G0AOLIo0wM4YUF7I9KrnUWrt1Jyfe+gonDurOiYMOYHjvTmRm1D93nkhr05TnYLoTzFg8ALja3TeY2WhglaZlEalffQujFeZl0aNjLne99hF/f+VDOrfP5gulB3DS4AM4dmA32udoWQFp3eJKMGY2AngJ+JhgpuObgQ3ASQSzJNe7cJhIuqtvcMC1Xx3KuOElbK2s4tXF65k6fy1TF6zlsXc/JTszg37d2vPR+nKqqvVgp7RO8bZgbgFuc/drzCxycsopwAXNH5ZI29HQ1DQAHXPbcephxZx6WDG7q2uYtXwzLy1Yy6SZy2I+2Pn75xcqwUirEG+CGQFcGGP7aoJVJUWkAfVNTRMtKzODUf27MKp/F+56Lfad59VllZw98U2OGdiVYwd2ZUhxofpupEWKN8FUEEwNE+0QgvnARKSZ1dd3U5CTxZaKKm6esoibpyyiKL8dowcEyeaYgV05sFO++m6kRYg3wTwFXBOuFAngZtaXYDTZYwmISyTt1dd3c8O4oO9m/badzFy6gdeWbGDG0vVMnrMagK4F2WzeUaVJOSXl4k0wPweeJZjkMh+YQXBr7HWCqfRFpJk11nfTrUPOnltv7s7SdeW8umQDNz2/MGbfzXX/mcexA7vSpSAn6eci6Sne52C2AseY2fEEMyhnAO9qlmKRxIq378bMGNi9AwO7d+CGZ2JPcL5pRxUjbpjKQQcUcGS/zozs15mR/brQozB3TxndWpPm1KQFx9x9GjCt9rOZ9QFudvezmjswEdk39fXddC3I4cJj+vH2xxv5z/urePCtYHWM3p3zObJfZ9plGo+/u5Kdu4NZo3VrTfbX/q5oWQSc0RyBiEjzqK/v5qpTBjFueAmXjBlAdY2zYPVW3vp4E299tJGXFqxl846qOnVVVFVzk4ZFyz5K2pLJIpIcjfXdAGRmGENLChlaUsiFx/SjpsYZ8Mtn8Rj1rSqrZNwdMzmidyeG9y5ieO8iSoryMPtsaHTtrTUtJS2RlGBE2qB4+25qZWRYg8OiszMzePDt5UyaGTyb061DDkf0LmJ4706UV1Zx14yP9yzIpltrUksJRkSAxodFV1XXsGjNNt5bsZl3V2zhvRWbmTJvbcy6dGtNoJEEY2ZPN/L9js0Yi4ikUGO31tplZuy5rXbuUcF3Nm3fxRHXx14SalVZJV/58wyGlhRyaPgq7dGB7KzPFtLVqLW2rbEWzMY49msmZZE2oqm31jq3z6akgVtrHfOymPzBKh56Oxixlp2ZQWmPDgwtKWR3TQ1Pv79Ko9basAYTjLtrIksRaVBjt9bcnRWbdjBnZRlzVpYxd2UZkz9YxdbKuuvkVFRVc/0z8xt9IFQtn9ZBfTAisl8aW0razOjTpT19urTn1MOKAXB3+l8Ze9Taxu27GHHDVA7okMMhPTsyqGcHBvXoyKCeHenfrT2TP1i9V0JTy6flUoIRkf0Wz1LSkczqH7XWtSCb7x83gAWrt7Fg9Vb+8eFGdlUHt9GyMzOocWd3jKlwbp6ySAmmhVGCEZGUqP+B0MF7JYqq6ho+Wr+dBau3smDNVu585aOY9a3cUsH4xz4Ipsw5oICDu3ege8ecmM/r6NZacijBiEhKxPNAKASj10p7dKC0RwfGUcIzs1fHbPlkZ2bwwvy1/OudT/Zs65CbtSfZVFZV8+ycNXtaQ7q1lnhKMCKSMk0dtQb1t3xuPP1Qxg0vYWP5ThavLWfJum0sWVvO4rXbeGH+WjZt31Wnroqqaq5+ai5ZmUb/rgX079ae3HaZdcqp5bNvlGBEpFVprOXTpSCHowpyOGpAl72+12/85JiDCrZV7ubSB98DwAyKC/Po3609A7oVMKBbe1aXVTJpxsdUajh1kynBiEirsy8tn/oGFRQX5nLXeZ/nw/XlfLR+Ox9tKOfD9eU8MusTduyqjlFT0PKZ8PQ8ehbm0rdrew7osHdfD2h+NlCCEZE0Ud+ttStOPoTBxR0ZXLz3xCTuztqtOxl140sx69tSUcU3Jr65p54+XfLp0yWfvl3bs3n7Lp58fxW70rzVowQjImkh3kEFtcyMHoW59c5U0L1jDjefOYzlG7ezbOMOlm3YztJ15by8cP2egQSRKqqq+dUTc9hQvpMDO+XTu3M+vTrn0SG33V7l2lJ/jxKMiKSN5hxUcOWXBvE/B3cDuu1VvrrGOaiepQ+276rmhskL9trWKb8dvTvnc2DnfCqrqnl18XqqqoNvN6Xl0xITkxKMiEgDmtryyWxg6YOSolwm/+hYVmzawSebKoKfm3fwyaYdzFtZxrKNO+p8p6Kqmiv+/QHTFq6jpFMeJUV5lHTK48DwZ352Fk++t7JFzm6Q1ARjZp2Bu4EvAhuAK939wXrKXgb8AsgDHgMucfedjdVjZqOA64ERQDUwHfiRu69O3JmJSFvW1JZPfa2ey8ceQlF+NkX52Rx2YFGd79U30m1XdQ3vf7KFZ+esrjOLQef22WyrrNrT6qlVUVXN755byFeHFZORsfcAhEiJbPkkuwVzB7AL6A4cDkw2s9nuPi+ykJmNBcYDxwOrgCeAa8NtjdXTCZgITAF2A38B/gGcnNhTExEJNDY/W33qb/nk8eoVX6C6xlm3rZKVmytYuaWCT8OfD761ImZ9a7ZWUnr1c/QozKVnYR7Fhbn0LAp/FuaxcO1W/jJt6T4tFlebmLJ7HDSivjJJSzBm1h44Axjq7uXAjHC9mXP5LHHUOg+4uzbxmNn1wAPA+Mbqcffnoo77F+CVBJ6aiEgdTZ2fDRpq+ZQCwe23noV59CzM43MR33tl0fqYiakwrx1nH9mb1WUVrN5Syazlm1k7Z3Wd1k6kiqpqrnpyLpt37KJHx1y6F+bSo2MuB3TIISszWMsn+pZcfcy9/gM1JzMbDrzu7nkR234OHOfuX4kqOxv4rbs/HH7uCqwHugK9460n3PcT4JvuPirGvouBiwG6des24pFHHtn/E21jysvLKSgoSHUYLY6uS2y6LnU19Zq8vqqKxxZXsbHS6ZJrnHFwO44ubtfod+6Zu4tdEYPXsjPg/KHZdb5b487WXc6mSue6NyrjjsuAjjlGpxxjZXkNYaOH1ff+hJ2rl8S8B5fMW2QFQFnUtjKgQxxla993aEo9ZnYY8GvgtFgBuftEgttplJaWerz/ykgnTfnXVzrRdYlN16Wupl6TMcAvm3iMMcDgfehLuXvBtHoHIzx96TGs2VrJ2q2VrCnbGbwvq2TN1kqWLV4fV1zJTDDl1F1iuSOwLY6yte+3xVuPmR0EPAf82N1f28eYRURaheYcgn352EPoUpBDl4IchhQX1vne6N/FTkzRMhot0XwWA1lmNjBi2zBgXoyy88J9keXWuvvGeOoxsz7AVOB6d7+/meIXEWlTxg0v4cbTD6WkKA8jGExQO2loQy4fW0pejElBoyWtBePu283sceA6M7uIYPTXacDRMYrfB9xjZg8Aq4GrgHviqcfMSoBpwB3u/vfEnpWISOu2Ly2fyFFyDT3/kcwWDMAPCJ5rWQc8RPBsyzwz621m5WbWG8DdnwduAl4GloevaxqrJ9x3EdAfuCass9zMypNwbiIiaWPc8BJmjj+eXWuW/re+Mkl9DsbdNwHjYmxfQdB5H7ntVuDWptQT7ruW4JkZERFJoWS3YEREJE0owYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIowYiISEIkNcGYWWcze8LMtpvZcjP7VgNlLzOzNWZWZmaTzCwn3nrM7AQzW2hmO8zsZTPrk8jzEhGRupLdgrkD2AV0B84B/mZmQ6ILmdlYYDxwAtAX6A9cG089ZtYVeBy4GugMzAIeTszpiIhIfZKWYMysPXAGcLW7l7v7DOBp4NwYxc8D7nb3ee6+GbgeOD/Oek4H5rn7o+5eCUwAhpnZIYk7OxERiZaVxGMdDFS7++KIbbOB42KUHQI8FVWuu5l1AXo3Us+Q8DMA7r7dzD4Mty+MPIiZXQxcHH7caWZzm3xWbV9XYEOqg2iBdF1i03Wpq61fk3q7IJKZYAqAsqhtZUCHOMrWvu8QRz0FwPp4juPuE4GJAGY2y90/1/AppB9dl9h0XWLTdakrna9JMvtgyoGOUds6AtviKFv7flsc9TTlOCIikiDJTDCLgSwzGxixbRgwL0bZeeG+yHJr3X1jHPXs9d2wz2ZAPccREZEESVqCcfftBKO7rjOz9mY2GjgNuD9G8fuAC81ssJl1Aq4C7omznieAoWZ2hpnlAr8GPnD3hdEHiTJx/86wzdJ1iU3XJTZdl7rS9pqYuyfvYGadgUnAScBGYLy7P2hmvYH5wGB3XxGW/SnwCyAPeAz4vrvvbKieiOOcCPyFoPPpLeB8d1+WlJMUEREgyQlGRETSh6aKERGRhFCCERGRhEj7BNOU+dHSiZlNN7NKMysPX4tSHVMqmNmlZjbLzHaa2T1R+9Jyzrv6romZ9TUzj/idKTezq1MYalKZWY6Z3R3+HdlmZu+Z2Zci9qfd70vaJxjinB8tTV3q7gXhqzTVwaTIKuAGgkEle6T5nHcxr0mEoojfm+uTGFeqZQGfEMwqUkjwu/FImHjT8vclmU/ytzgR85oNdfdyYIaZ1c5rNj6lwUmL4O6PA5jZ54ADI3btmfMu3D8B2GBmh8QxJL5Va+CapLXwEYoJEZueMbOPgRFAF9Lw9yXdWzD1zY+mFkzgRjPbYGYzzWxMqoNpYerMeQfUznmX7pab2adm9o/wX+5pycy6E/yNmUea/r6ke4Jpyvxo6eYXBMsklBA8KPYfMxuQ2pBaFP3u1LUB+DzB82cjCK7FAymNKEXMrB3Bud8btlDS8vcl3ROM5i2rh7u/5e7b3H2nu98LzAS+nOq4WhD97kQJl8+Y5e673X0tcCnwRTOLvk5tmpllEMwssovgGkCa/r6ke4Jpyvxo6c4BS3UQLYjmvGtc7VPcafN7Y2YG3E0waOgMd68Kd6Xl70taJ5gmzo+WNsysyMzGmlmumWWZ2TnA/wBTUh1bsoXnnwtkApm114R9n/Ou1avvmpjZSDMrNbOMcO2m24Hp7h59a6gt+xswCPiKu1dEbE/P3xd3T+sXwZDBJ4HtwArgW6mOKdUvoBvwDkHzfQvwJnBSquNK0bWYQPAv8cjXhHDfiQSL2FUA04G+qY43ldcEOBv4OPx/aTXBpLU9Uh1vEq9Ln/BaVBLcEqt9nZOuvy+ai0xERBIirW+RiYhI4ijBiIhIQijBiIhIQijBiIhIQijBiIhIQijBiIhIQijBiLRR4dosZ6Y6DklfSjAiCWBm94R/4KNfb6Y6NpFkSev1YEQSbCrB2kKRdqUiEJFUUAtGJHF2uvuaqNcm2HP76lIzmxwuobvczL4d+WUzO9TMpppZhZltCltFhVFlzjOzOeHyxWujl3UGOpvZo+GS4B9FH0MkkZRgRFLnWuBp4HCCNXfuC1eJxMzygecJ5rI6EvgacDQRyxSb2f8D7gT+ARxGsJxC9Oy8vwaeIpjJ92FgUjqsBS8tg+YiE0mAsCXxbYKJDyPd4e6/MDMH7nL370V8Zyqwxt2/bWbfA24BDnT3beH+McDLwEB3X2pmnwL/dPeYy3uHx/idu18Zfs4CtgIXu/s/m/F0RWJSH4xI4rwKXBy1bUvE+zei9r0BnBK+H0QwnXvkglSvAzXAYDPbSrDa6EuNxPBB7Rt3321m64ED4gtfZP8owYgkzg53X7qP3zU+W7ArWlMWf6uK+uzo1rgkiX7RRFJnVIzPC8L384FhZha5ZvvRBP/PLvBgSeKVwAkJj1JkH6kFI5I4OWbWI2pbtbuvD9+fbmbvECw+dSZBshgZ7nuAYBDAfWb2a6ATQYf+4xGtot8AfzSztcBkIB84wd3/kKgTEmkKJRiRxDmRYGXHSCuBA8P3E4AzCJYWXg9c4O7vALj7DjMbC/wJeJtgsMBTwI9rK3L3v5nZLuBnwO+BTcCziToZkabSKDKRFAhHeH3d3f+d6lhEEkV9MCIikhBKMCIikhC6RSYiIgmhFoyIiCSEEoyIiCSEEoyIiCSEEoyIiCSEEoyIiCTE/wcssPuOFC6eQgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learning_rate = 0.01\n",
    "decay = 1e-4\n",
    "batch_size = 32\n",
    "n_steps_per_epoch = len(X_train) // batch_size\n",
    "epochs = np.arange(n_epochs)\n",
    "lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)\n",
    "\n",
    "plt.plot(epochs, lrs,  \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Power Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exponential Scheduling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```lr = lr0 * 0.1**(epoch / s)```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch):\n",
    "    return 0.01 * 0.1**(epoch / 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay(lr0, s):\n",
    "    def exponential_decay_fn(epoch):\n",
    "        return lr0 * 0.1**(epoch / s)\n",
    "    return exponential_decay_fn\n",
    "\n",
    "exponential_decay_fn = exponential_decay(lr0=0.01, s=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.8503 - accuracy: 0.7562 - val_loss: 0.7115 - val_accuracy: 0.7578\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 5s 95us/sample - loss: 0.7107 - accuracy: 0.7861 - val_loss: 0.7508 - val_accuracy: 0.7642\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 5s 88us/sample - loss: 0.6246 - accuracy: 0.8084 - val_loss: 0.6204 - val_accuracy: 0.7968\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 5s 91us/sample - loss: 0.5598 - accuracy: 0.8278 - val_loss: 0.6224 - val_accuracy: 0.8446\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 5s 87us/sample - loss: 0.5242 - accuracy: 0.8365 - val_loss: 0.5441 - val_accuracy: 0.8338\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 5s 84us/sample - loss: 0.4533 - accuracy: 0.8541 - val_loss: 0.5239 - val_accuracy: 0.8466\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.4378 - accuracy: 0.8613 - val_loss: 0.5651 - val_accuracy: 0.8346\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.3915 - accuracy: 0.8705 - val_loss: 0.5245 - val_accuracy: 0.8422\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 5s 85us/sample - loss: 0.3693 - accuracy: 0.8772 - val_loss: 0.4801 - val_accuracy: 0.8674\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 5s 89us/sample - loss: 0.3403 - accuracy: 0.8855 - val_loss: 0.4553 - val_accuracy: 0.8672\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 5s 92us/sample - loss: 0.3194 - accuracy: 0.8933 - val_loss: 0.4427 - val_accuracy: 0.8794\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 4s 82us/sample - loss: 0.2960 - accuracy: 0.8991 - val_loss: 0.4337 - val_accuracy: 0.8674\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.2744 - accuracy: 0.9069 - val_loss: 0.4876 - val_accuracy: 0.8762\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2600 - accuracy: 0.9104 - val_loss: 0.4965 - val_accuracy: 0.8852\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.2398 - accuracy: 0.9175 - val_loss: 0.5139 - val_accuracy: 0.8812\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.2229 - accuracy: 0.9237 - val_loss: 0.5102 - val_accuracy: 0.8812\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.2100 - accuracy: 0.9281 - val_loss: 0.4916 - val_accuracy: 0.8824\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.1989 - accuracy: 0.9321 - val_loss: 0.5105 - val_accuracy: 0.8826\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 5s 90us/sample - loss: 0.1861 - accuracy: 0.9362 - val_loss: 0.5424 - val_accuracy: 0.8794\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.1717 - accuracy: 0.9410 - val_loss: 0.5630 - val_accuracy: 0.8854\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 5s 86us/sample - loss: 0.1638 - accuracy: 0.9450 - val_loss: 0.5584 - val_accuracy: 0.8824\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 4s 80us/sample - loss: 0.1545 - accuracy: 0.9487 - val_loss: 0.5977 - val_accuracy: 0.8860\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 5s 87us/sample - loss: 0.1426 - accuracy: 0.9520 - val_loss: 0.6271 - val_accuracy: 0.8836\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 5s 90us/sample - loss: 0.1364 - accuracy: 0.9546 - val_loss: 0.6106 - val_accuracy: 0.8826\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 5s 91us/sample - loss: 0.1323 - accuracy: 0.9567 - val_loss: 0.6387 - val_accuracy: 0.8850\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxU1fnH8c+TBEgI+2KAKCCiIKCIGyq1omixrVVcalutlVbl1/qzrbbuPze07lVr61KtW913pIpiRYyKC4IgCCigssm+SCAhgRCe3x/3BsfJJLkBZiZkvu/Xa16ZuffMmecehjy595x7jrk7IiIiO1pWugMQEZHGSQlGRESSQglGRESSQglGRESSQglGRESSQglGRESSQglGJAXMbLiZldTzPUVmdleyYgo/Y76ZXZiEek8xs3rdAxHfRtvSZtKwKMFIUpnZI2bmCR4fpju2ZAmP75S4zc8APZLwWWeb2VQzKzGzYjObbmZ/2dGfkyZJaTNJnZx0ByAZYRxwRty2TekIJF3cvQwo25F1mtlvgL8DFwBvAk2BvsChO/Jz0iUZbSappTMYSYWN7r4s7rEGwMyOMLMKMxtcVdjMfmtm68ysR/i6yMz+aWZ3mtk34eNWM8uKeU9bM/t3uK/MzMaZWd+Y/cPDv/KHmNkMMys1s7fMbPfYQM3sJ2b2sZmVm9k8M7vezJrG7J9vZleY2X1hjF+b2UWx+8Onz4VnMvNjPz+m3B5mNtrMloWxTDGz4+rZrscDL7r7fe7+hbvPcvfn3P1Pccf0YzObGLbLajN72cxyY4rk1nQ84ftbm9n9ZrbCzNab2dtmdmBcmV+Z2QIz22BmrwAFcfuvMbMZcdtqvQSWoM2uCf/tfm5mX4axvGRmHWLK5JjZHTHfkzvM7F4zK6q7OWVHU4KRtHL3t4FbgcfMrJ2Z9QZuA37v7l/FFD2d4Pt6KPA/wAjg/Jj9jwADgROAg4ENwFgzy4sp0wy4DPhNWE8b4J9VO81sKPAEcBfBmcBvgFOAG+LCvgD4FNgfuBm4xcyqzhoOCn+eA3SOeR2vBfAacAzQH3gBeDE8/qiWAQdXJeJEzOxYYDTwBnAAcCTwNt/9v1/j8ZiZAWOAQuA4YADwDjDezDqHZQYStP/9wH7Ay8C19TiO+ugO/Aw4EfhBGM/1MfsvBIYDZwOHEBznaUmKReri7nrokbQHwS+ezUBJ3OPmmDJNgEnAi8AU4Jm4OoqAOYDFbLsC+Dp8vifgwPdj9rcGioGzw9fDwzK9YsqcTnCpLit8/Q5wZdxnDwvjtfD1fOCpuDJzgStiXjtwSlyZ4UBJHW31YVw9RcBdtZTvDHwQft5c4HHgV0CTmDLvAU/XUketxwMcFR5/XlyZT4CLw+dPAm/E7X8g+PWy9fU1wIza2iTC62uAcqB1zLb/A76Ieb0UuDTmtQGfA0Xp/r+QiQ+dwUgqvEPwl23s49aqne5eQfBX5nHALgRnKPE+9PA3RugDoNDMWgF7A1vCbVV1FhP8Vd4n5j0b3X12zOslBMmtTfj6AOD/wktpJeHlmSeBfKBTzPumx8W2JIw7MjPLN7NbzGxWeCmnBDgQ6Bq1Dndf6u6HAvsAfyP4ZXof8JGZNQ+LDSDon6lNbcdzANAcWBnXLv2APcIyexPT9qH41zvKgvDftlqsZtaa4N/po6qd4XdmUpJikTqok19SYYO7f1FHmarLGW2AjsDaetRvteyLTUqba9iXFfNzJPBcgnpWxjyvSFBPff9Y+ytwLMElnbkEl/QeJeiorxd3nwHMAO42s+8B7wKnEpw9RlHb8WQBy4HDE7xvXfiztvavsiVBuSYR44sVpe01RXwDoTMYSTsz607Q7/G/BH0FT5hZ/B8/A8P+gCqHAEvcfR0wi2/7Z6rqbEXwl/2seoQyBejtQYd5/CM+OdWmAsiuo8z3gEfd/QV3nw58zbdnBNuj6nhbhD+nAkO2o74pBB32WxK0yYqYzzwk7n3xr1cCBXH/hvttR1zVhGc2ywj64ICtfUg19YNJkukMRlKhmZl1ittW6e4rzSyboO/gbXe/z8yeJ7i0dTVwZUz5LsDfzOwegsRxEfAXAHefa2ajgfvMbATB2c/1BH9hP1mPOK8FXjGzBcCzBGc8/YCD3f3ietQzHxhiZm8TXJb7JkGZOcCJYdwVBMebm6BcjczsXoJLROMJElRngr6pDcB/w2LXAy+b2RcEbWEEneP3ufuGCB8zjqAfZ7SZXUzQn9GJ4OxrnLu/SzBU+n0zuwx4HhhM0AkfqwhoB1xuZk+HZeLvFdoR7gQuNrM5BInvfwjaZWkSPkvqoDMYSYWjCf6Dxz6mhvsuB3oCZwG4+2rgTODS8HJPlScIzgomAv8CHgTuiNn/a4Jr7/8JfzYHjvXgXopI3P114McEI60+Ch+XAgujHyoAfw7rWMS3xxnvT8AKgstZrxF08L9bz895g2Dk3LMECWtUuP0Yd58D4O6vEvyy/2EYy9thbFuifEDYh/EjgiT2L2B2+Hm9CJIb7v4hwb/f7wj6c04i6JCPreezcP+IsMwxVB+dtyP8FXgMeJigTSFol/IkfJbUoWpkjEiDFd7DMMPdz0t3LLLzMbMpwHvu/vt0x5JpdIlMRBoNM+sGDCU4U8shOGPqH/6UFFOCEZHGZAvBvUC3EnQBzAJ+6O6T0xpVhtIlMhERSQp18ouISFLoElmoTZs23rNnz3SH0eCUlpaSn5+f7jAaHLVLYmqX6hp7m3z88cer3L1jon1KMKGCggImT9Zl2nhFRUUMHjw43WE0OGqXxNQu1TX2NgnvG0tIl8hERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQplGBERCQpUppgzKydmY0ys1IzW2Bmp9VS9gIzW2ZmxWb2kJk1i9l3nplNNrONZvZIgvcOMbPPzWyDmb1lZt3qim3+ui0Mumk8L01dvM3HJyIi30r1GczdwCagADgduNfM+sYXMrOhwKXAEKA70AMYGVNkCfAX4KEE7+0AvAhcCbQDJgPPRAlu8doyLnvxUyUZEZEdIGUJxszygZOBK929xN0nAP8BzkhQ/EzgQXef6e7fANcBw6t2uvuL7v4SsDrBe08CZrr7c+5eDlwD9Dez3lHiLKuo5NbXZ9fjyEREJJFULpm8F1Dp7nNitk0DjkhQti8wOq5cgZm1d/dESSX+vdOqXrh7qZl9GW7/PLagmY0ARgA07dRz6/bFa8soKiqq63gyQklJidoiAbVLYmqX6jK5TVKZYFoAxXHbioGWEcpWPW9J4rOW+PeujPI57n4/cD9As857etX2wjZ5jXoN7fpo7OuJbyu1S2Jql+oyuU1S2QdTArSK29YKWB+hbNXzRGW353OqMeD8o/eMUlRERGqRygQzB8gxs9jf3v2BmQnKzgz3xZZbHuHyWLX3hn0/e9TwOd/RoUVTHPhyZWmEjxERkdqkLMG4eynB6K5rzSzfzAYBJwCPJSj+KHCWmfUxs7bAFcAjVTvNLMfMcoFsINvMcs2s6nLfKKCfmZ0clrkKmO7un1OL7q2ymHzFMfzswN3417tf8enX8VfzRESkPlI9TPlcIA9YATwF/M7dZ5pZVzMrMbOuAO4+FrgFeAtYED6ujqnnCqCMYCjzL8PnV4TvXUkwWu164BtgIPDzqAFe/uO9aZ/flItfmE5F5ZbtOVYRkYyW0gTj7mvcfZi757t7V3d/Mty+0N1buPvCmLK3u3uBu7dy91+7+8aYfde4u8U9ronZP87de7t7nrsPdvf5UWNsndeE64b147Ol67jv7S93zIGLiGQgTRWTwNC+nfjxPp35+5tf8MWKSGMDREQkjhJMDa45vi/Nm2Vz8fPTqdzidb9BRES+QwmmBh1bNuOq4/owZeFaHvtgfrrDERHZ6SjB1OLEAYUcsVdHbnl9NovWbEh3OCIiOxUlmFqYGdef2A8DLh/1Ke66VCYiEpUSTB12bducS37Ym3fnruKFKZplWUQkKiWYCH45sBsHdmvLda/MYsX68nSHIyKyU1CCiSAry7j5lH0pq6jk6tF1zjgjIiIowUS2R8cW/HHInrw2Yxmvfbo03eGIiDR4SjD1MOL7PejTuRVXjp5J8YaKdIcjItKgKcHUQ5PsLG45ZV++2bCJv4yZle5wREQaNCWYeupX2Jr/+X4Pnvv4a96dG7+umYiIVEnlipaNxh+G7Mlzkxcx/OFJbNnidGmTx0VDezFsQGG6QxMRaTCUYLbB2BnLKC7fvHWOssVry7jsxU8BlGREREK6RLYNbn19Nps2f3etmLKKSm59fXaaIhIRaXiUYLbBkrVl9douIpKJlGC2QZc2eTVsz01xJCIiDZcSzDa4aGgv8ppkV9s+sEf7NEQjItIwKcFsg2EDCrnxpH0obJOHAYVtctm7U0temb6UWUvWpTs8EZEGQaPIttGwAYXfGTG2umQjx975Lr9/agov//57NG+qphWRzKYzmB2kfYtm/O1n+/HVqlKufVl3+YuIKMHsQIN6duB3R+zB05MW8fK0JekOR0QkrZRgdrALjtmLAV3bcPmLn2qZZRHJaEowO1iT7Cz+/vMBAPzh6alUVG6p4x0iIo2TEkwS7NauOTectA9TF67lb+PmpDscEZG0UIJJkp/078KpB+7KPUVf8v4Xq9IdjohIyinBJNE1x/dl9w75nP/MJ6wu2ZjucEREUkoJJomaN83hH78YwNoNFVz0/HTcPd0hiYikjBJMkvXt0prLf9Sb8Z+v4JH356c7HBGRlFGCSYEzD+vO0Xvvwo2vfs6MxcXpDkdEJCVSmmDMrJ2ZjTKzUjNbYGan1VL2AjNbZmbFZvaQmTWLWo+ZnWpmn5nZejObZWbDknlcdTEzbjmlP23zm/CHp6ZSunFzOsMREUmJVJ/B3A1sAgqA04F7zaxvfCEzGwpcCgwBugM9gJFR6jGzQuBx4E9AK+Ai4Ekz2yU5hxRNu/ym3BFOJXPQ9ePY/dIxDLppPC9NXZzOsEREkiZlCcbM8oGTgSvdvcTdJwD/Ac5IUPxM4EF3n+nu3wDXAcMj1rMrsNbdX/PAGKAU2COJhxfJinUbyckyNmyqxPl2qWUlGRFpjFI55e9eQKW7x955OA04IkHZvsDouHIFZtYe6FpHPZOBz8zseGAM8BNgIzA9/kPMbAQwAqBjx44UFRVtw2FFd13RBjZv+e5IsrKKSq4bPY02xXOT+tnbqqSkJOntsjNSuySmdqkuk9sklQmmBRDfw10MtIxQtup5y7rqcfdKM3sUeBLIJbiU9lN3L43/EHe/H7gfoFevXj548OB6HE79rRk7JvH2cifZn72tioqKGmxs6aR2SUztUl0mt0nkS2RmVmBmF5rZvWbWIdw2yMx2j1hFCUGfSKxWwPoIZauer6+rHjM7GrgFGAw0JTizecDM9osYZ9LUtNRyZy21LCKNUKQEY2YHALMJOtTP4ttf8McA10f8rDlAjpntGbOtPzAzQdmZ4b7YcsvdfXWEevYD3nH3ye6+xd0nAROBoyPGmTQ1LbXcuyDRSZyIyM4t6hnMX4E73X0AQX9GldeBQVEqCC9RvQhca2b5ZjYIOAF4LEHxR4GzzKyPmbUFrgAeiVjPJODwqjMWMxsAHE6CPphUq77Uch6D9mjP+NkrGTX163SHJyKyQ0XtgzmA4Mwl3lKCocJRnQs8BKwAVgO/c/eZZtYVmAX0cfeF7j7WzG4B3gLygBeAq+uqB8Dd3zaza4DnzawAWAnc4O7/rUecSRO/1HJF5RbOeHAil7zwKT06tKD/bm3SGJ2IyI4TNcGUAW0TbO9N8Es+EndfA1S76dHdFxJ03sduux24vT71xOy/C7gralzp1CQ7i3tOP4Dj75rAiMcm8/J532OXVuqTEZGdX9RLZKOBq2Pupncz6w7cTHB2IduhXX5T/vWrA1lfvpkRj31MeUVlukMSEdluURPMhUA7gstNzYEJwBfAWoL+EdlOe3duxe2n9ueTRWv5v1EzNPOyiOz0Il0ic/d1wPfM7Chgf4LENMXdxyUzuExzbL/O/HHIntz55lz27tySsw/vke6QRES2WaQEY2a/Ap5x9/HA+JjtTYGfu/ujSYov4/xxyJ7MXraeG179jL0KWvL9vTqmOyQRkW0S9RLZw0DrBNtbhvtkB8nKMm47tT97FbTkvCenMG9VtQkIRER2ClETjAGJOgW6Un3aFtlO+c1y+NevDiQ7yzjn0cmsL69Id0giIvVWa4Ixs0/NbDpBcnnbzKbHPGYC7wLqh0mC3do1557TD2D+qlLOf/oTKreo019Edi519cE8H/7sRzAzcUnMvk3AfDRMOWkO3aM9V/+kD1eOnslt/53Nxcf2TndIIiKR1Zpg3H0kgJnNJ+jkL09FUPKtXx7SjVlL13NP0Zc89dFC1m6ooEubPC4a2us7MwKIiDQ0kfpg3P3fSi7pYWYc0LUNWQbfbKjQQmUistOIOptyUzMbaWZzzKzczCpjH8kOMtPdMW4u8V0wZRWV3Pr67PQEJCISQdRRZNcRLGN8G7CFYJ37uwkmmjw3OaFJlSVry+q1XUSkIYiaYE4Ffuvu9wGVwGh3/wPBDMfHJCs4CdS0UFmn1poUU0QarqgJpoBgOn0IRpJVzSk/FvjBjg5Kvqumhcpa5TVh0+YtaYhIRKRuURPMQqBL+PwLYGj4/FCCqfwliRItVPbzg3Zj9rL1XPjcNLboHhkRaYCirgczChgCfAjcCTxlZucAhcCtSYpNYsQvVAbQrX0+N4/9nLbNm3DN8X0xszRFJyJSXdTZlC+Lef68mS0iWCp5jru/kqzgpHa/PaIHa0o38q9359G+RTP+MGTPdIckIrJV1DOY73D3icBEADPLd3fNyJgGZsZlP9yb1aWbuP2NObTNb8oZh3RLd1giIkD0PphqzCzXzC4C5u3AeKSesrKMm0/elyG9d+Gq0TMYM31pukMSEQHqnuyyqZldb2aTzOx9MxsWbv8V8BVwPnBHCuKUWjTJzuKu0/bnwG5tOf+ZqUyYuyrdIYmI1HkGcw1wHrAA2B14zszuAf4PuAzo7u43JjVCiSSvaTYPnHkQe3RswYjHJjNt0dp0hyQiGa6uBHMqMNzdTwGOBbKBtkDfcH4yLVTSgLTOa8KjvzmYdvlNGf7wR3yxoqTuN4mIJEldCWY3YBKAu08jmKL/ZnffnOzAZNvs0iqXx88aSHaW8asHJ7K0WLcpiUh61DWKrAmwMeZ1BVrBssHr3iGfR359MD+//0NOuOs9srOMZcXlmuZfRFIqyjDlG81sQ/i8KXCNmX0nyYTzkkkD0q+wNWce1o273/py67aqaf4BJRkRSbq6Esw7wB4xr98HusaV0TwlDdRLU5dU21Y1zb8SjIgkW10rWg5OURySBJrmX0TSaZtvtJSGT9P8i0g6KcE0YjVN85+bk8X6co0wF5HkSmmCMbN2ZjbKzErNbIGZnVZL2QvMbJmZFZvZQ2bWLGo9ZtbczO4xs1Xh+99J5nE1VImm+T/z0G4s+qaMXz74EcUblGREJHm2abLL7XA3wb00BcB+wBgzm+buM2MLmdlQ4FLgKGAJwXIBI8NtUeq5n+DY9gbWhGUyUqJp/gf17MD/PjmF0x74kMfPGkjb/KZpik5EGrOUncGYWT5wMnClu5e4+wTgP8AZCYqfCTzo7jPd/RvgOmB4lHrMrBdwPDDC3Ve6e6W7f5zkw9up/KBvJ+7/1YHMXVHCL/71IatKNtb9JhGReop0BmNm8UOTqzhQ7u4rI1SzF1Dp7nNitk0DjkhQti8wOq5cgZm1JxgmXVs9AwnmThtpZmcAS4Fr3P2F+A8xsxHACICOHTtSVFQU4TAaBwP+uF9T7pyynuPveJOLD8qlTW71vzdKSkoyql2iUrskpnapLpPbJOolsvnUcr+Lma0DHgYurmUamRZUnwWgGGgZoWzV85YR6tkV6Ae8QLDM86EEl9BmuftnsW9y9/sJLqfRq1cvHzx4cA2hN06DgQP2X81vHpnEnTOyePKcgXRu/d2RZ0VFRWRau0ShdklM7VJdJrdJ1EtkvwC+Bq4AjgkfVwALgd8QzLp8BnBlLXWUAK3itrUC1kcoW/V8fYR6ygimtPmLu29y97eBt4Af1BJbxjqkR3seO+tgVq3fyKn3fcCiNRvqfpOISARRE8zvgAvc/UZ3Hx8+bgT+DPzG3e8E/kCQiGoyB8gxs9h1ffsDMxOUnRnuiy233N1XR6hnesRjktAB3drx+NkDKd5Qwc/u+4D5q7RAqYhsv6gJZiDwaYLtM4CDwucfEFyeSihcVvlF4FozyzezQcAJwGMJij8KnGVmfcysLcHZ0iMR63mH4MzqMjPLCfcPBl6PeKwZqf9ubXhqxCGUVVTys/s/0FT/IrLdoiaYBYSd4XHOIfhlDtCRYEhwbc4F8oAVwFPA79x9ppl1NbOSqsEE7j4WuIXg0taC8HF1XfWE760gSDg/Iuib+RfwK3f/POKxZqy+XVrz9IhDqdziDLt7AgdfP47hY0sZdNN4Xpq6ON3hichOJmon/5+BF8zsRwTrwzjBmcseBEOGCV8/W1sl7r4GGJZg+0KCzvvYbbcDt9ennpj9Mwk696WeenVqyTmH9+DG1z6nZGMloFmYRWTbRDqDcfcxwJ4E95u0AtqEz3u5+6thmXvc/U/JClRS59EPFlTbVjULs4hIVJHv5Hf3RcBlSYxFGgjNwiwiO0LkBGNmzQmmXNmFuDMfd39xB8cladSlTR6LEySTFrk5uDtmloaoRGRnE+kSmZkdTdDRPoFgBNfzMY/nkhadpEWiWZizzVhfvpkLn5vOps1b0hSZiOxMoo4iuxMYA+zq7llxj+rzwctOLXYWZghmYf7rT/flgqP34oUpX3PmQx9RXKaZmEWkdlEvkXUHjnf36mvwSqNUNQtz/DQXu7XL45IXpnPKve/z0PCD2K1d8/QFKSINWtQzmPeAXskMRHYOJ+2/K4/+ZiDL15Vz4j3vM/3rtekOSUQaqKgJ5p/AX83sbDMbaGb7xz6SGaA0PIfu0Z4Xzz2M3CZZ/Oy+D3lj1vJ0hyQiDVDUBPM80Jtg5uEPgMkxj0nJCU0asp67tGTUuYPYq6AFIx6bzMPvzUt3SCLSwETtg9k9qVHITqljy2Y8PeJQ/vj0VEa+PIuFazZwxY/7kJ2lYcwiEjHBuHv1W7tFgLym2dz7ywO4fsxnPPTePCbNW8Oa0k0sLS6nS5s8LhraS9PLiGSoGhOMmZ0EvOzuFeHzGulGy8yWnWVc9ZM+rN2wkRenfjvQUHOYiWS22s5gngc6EcxY/Hwt5RzQvTDCxHnfVNtWNYeZEoxI5qkxwbh7VqLnIjXRHGYiEkuJQ3aYLuGd//Fym2RTtqkyxdGISLpFTjBmtpuZnWZm55vZn2IfyQxQdh6J5jDLyTLKKio58Z73mKelmEUySqRRZGZ2OvAQsBlYSdDvUsWpYWEwySxV/Sy3vj6bJWvLto4ia5vflPOfnsrx/5jArT/dl2P7dU5zpCKSClHvg7kWuA240t11rUNqVDWHWbxX/nA45z4xhd8+PoUR3+/BxUN7kZOtK7QijVnU/+EFwANKLrKtCtvk8ez/HMIZh3Tj/ne+4rQHJrJiXXm6wxKRJIqaYF4FBiYzEGn8muVkc92wfvztZ/vx6dfF/PgfE5j41ep0hyUiSRL1EtkbwM1m1hf4FPjOYiC60VLqY9iAQvbu3IrfPf4xpz0wkUuO7cU5h/fQSpkijUzUBHNf+PPyBPt0o6XUW69OLRl93iAufn46N7z6Oa9MW8LKkk0s0xQzIo1GpEtkCVax1IqWst1a5jbhntP3Z9h+XZi+eB1Li8txvp1i5qWpi9MdoohshzoTjJk1MbOJZqYFx2SHMzMmza95ihkR2XnVmWDcvYJgun6vq6zIttAUMyKNU9RRZP8GzklmIJK5appiBuDZyYtw1982IjujqAkmHxhhZp+Y2YNm9vfYRzIDlMYv0RQzzXKy6NEhn4ufn845j37MyvUb0xSdiGyrqKPI9gamhM97xO3Tn5eyXWqaYub4/l146L153PL6bI792ztcf+I+HNuvU5qjFZGooq5oeWSyA5HMVtMUM2cf3oMj9urIBc9+wm8f/5iT99+Vq4/vQ6vcJmmIUkTqQ5NBSYO3Z0FLRp07iD8c1ZOXPlnMsXe8w/tfrEp3WCJSh/pM13+kmd1vZmPNbHzsox51tDOzUWZWamYLzOy0WspeYGbLzKzYzB4ys2b1rcfMrjYzN7Ojo8YoDVOT7Cz+9INePP/bQ8ltks1pD0xk5MszeW7yIgbdNJ7dLx3DoJvG694ZkQYkUoIxs+HAa0BLYDDBlP1tgf2BWfX4vLuBTQSTZ54O3BtOPxP/eUOBS4EhQHeCfp+R9anHzPYATgGW1iM+aeAGdG3LmD8czpmHduPh9+Zz8fPTWby2TDdoijRAUc9gLgTOc/dfEMxDdpm7DwAeB0qiVGBm+cDJBFP+l7j7BOA/wBkJip8JPOjuM939G+A6YHg967kLuIQgEUkjktc0m5En9KN9ftNqI0x0g6ZIwxF1FFkPYFz4fCPQInx+F1BEcLZRl72ASnefE7NtGnBEgrJ9gdFx5QrMrD3Qta56zOynwCZ3f7W2CRTNbAQwAqBjx44UFRVFOIzMUlJS0mDbZXVp4r8dFq8tS3rMDbld0kntUl0mt0nUBLOa4PIYwGKgHzAdaA/UfJfcd7UAiuO2FcfUW1vZquct66rHzFoANwA/qCsgd78fuB+gV69ePnjw4LreknGKiopoqO1S+OF4Fie42z+vSRZ7738IBa1yk/bZDbld0kntUl0mt0nUS2Tv8u0v7GeBv5vZw8BTBFP5R1ECtIrb1gpYH6Fs1fP1EeoZCTzm7vMixiU7qUQ3aOZkGZs2b2HIbW/z0IR5bK7ckqboRCRqgjmPIJkA3AjcSnD28ixwdsQ65gA5ZrZnzLb+wMwEZWeG+2LLLXf31RHqGQL8IRyBtgzYDXjWzC6JGKfsJIYNKOTGk/ahsE0eRrBq5l9/2p/xFw7mgG5tufaVWZxw93tMXR/I+dEAABWKSURBVFh9Mk0RSb6oN1quiXm+Bbi5vh/k7qVm9iJwrZmdDewHnAAclqD4o8AjZvYEwSiwK4BHItYzBIi9C28S8CeCUXDSyNR0g+Yjvz6I12YsY+TLMznp3vf5xcFduWRob1o31w2aIqlSn/tgCszsQjO718w6hNsGmdnu9fi8cwn6bFYQnBH9zt1nmllXMysxs64A7j4WuAV4C1gQPq6uq57wvavdfVnVA6gEvnH3SKPdpHEwM360T2fe/PNgfjNod57+aCFH3VbECx9/rckzRVIk0hmMmR0AvAnMIxjhdSuwCjiGYHRYjTdMxgrPhIYl2L6Qb0emVW27Hbi9PvXUULZ7lHLSOLVolsOVx/XhpP0LueKlGfz5uWk89/EijuzVkUc/WPiduc+0gqbIjhX1DOavwJ3hvS+x09q+Dgza4VGJ7GB9u7Tmhd8exg0n7sO0RWu58bXZukFTJMmiJpgDCNaEibeU4G56kQYvK8s4bWBXWjdvWm2fbtAU2fGiJpgygqlh4vUm6AcR2WksLy5PuH3x2jL1z4jsQFETzGjg6pgJJ93MuhOMJnshCXGJJE1tK2ie8eBHzFgcfx+viGyL+sxF1o5gksvmwATgC4I76K9ITmgiyZHoBs3cJlmcNKALM5cUc9w/JnDBM5/w9Tcb0hShSOMQ9T6YdcD3zOwoghmUs4Ap7j6u9neKNDw1raA5bEAh68or+GfRlzw4YR5jpi9l+KDu/O/gnrp/RmQbRJ2LDAB3Hw9sXf/FzLoBt7r7qTs6MJFkqukGzVa5Tbj42N6ccWg3bv/vHP717lc8M2kR5x3ZkzMO7cbYGcu49fVgBFrhh+M1vFmkFvVKMAm0IZg6X6RR6dw6j1t/2p/ffG93bh77Ode/+hl3v/UFpZs2U1EZDASoGt4MKMmIJKAlk0VqsXfnVjzy64N54uyB30kuVTS8WaRmSjAiEQzq2YHNlYmHMC9JsGSAiCjBiERW0/Dm7Czj1U+XsmWL7qERiVVrH4yZ/aeO98evyyLSaF00tBeXvfgpZRWVW7c1yTba5DXh3Cem0HOXFpx3ZE+O27czOdn6202krk7+1RH2a2EvyQixw5sXry2jMBze/JP+XRjz6VLuGj+X85/5hL+Nm8O5R/bkxAGFNFGikQxWa4Jx91+nKhCRnUHV8Ob4ZXCP79+F4/bpzH9nLecf4+dy8fPTuXPcXH43eA9+euCuvPbpsoT33Yg0Zts7TFlEQllZxrH9OjG0bwFvzV7B39/8gitemsEtYz+nrKJSw5sl4+j8XWQHMzOO6l3AqHMP4/GzBlJesUXDmyUjKcGIJImZ8b09O1BRuSXhfg1vlsZOCUYkyWoa3uzA6Q98yLhZyzXEWRolJRiRJKtp9ubj9u3EVytLOfvRyRx5WxEPTZjH+vKKNEUpsuOpk18kyWqbvbmicguvz1zGQxPmce0rs7j9jTn89MBdGX5Yd7q1z+elqYs1+kx2WkowIilQ0+zNTbKzOG7fLhy3bxemLVrLw+/N4/EPF/DI+/Pp07klc5eXsinsw9HoM9nZ6BKZSAPRf7c2/O3nA3jvkqP4/ZE9+Wzp+q3JpYpGn8nORAlGpIHZpVUuf/pBL7yGfv8la8vwmnaKNCBKMCINVG2jz478axF3v/UFy9eVpzYokXpQghFpoGoafXb6wN0oaJXLra/P5rCbxnPWI5N4feayGu+3EUkXdfKLNFC1jT4DmL+qlGcnL+L5j7/mzc9X0KFFM07ev5BTD9qNT78u1ugzSTslGJEGrKbRZwDdO+Rz8bG9+dMxe1E0eyXPTF7EAxPmcd87X5FlUHXvpkafSbroEpnITi4nO4uj+xTwr18dyAeXHUWr3BziJwYoq6jklrGfpydAyVhKMCKNyC4tc1lfvjnhviXF5fz52WkUzV6h/hpJiZQmGDNrZ2ajzKzUzBaY2Wm1lL3AzJaZWbGZPWRmzaLUY2aHmNkbZrbGzFaa2XNm1jnZxybSUNQ0+qx502z+O2sZwx+exMHXj+PyUZ/y4VerqYw53Xlp6mIG3TSe3S8dw6CbxvPS1MWpClsaoVT3wdwNbAIKgP2AMWY2zd1nxhYys6HApcBRwBJgFDAy3FZXPW2B+4HXgc3AXcDDwLHJPTSRhiHR0s55TbK54cR9+OE+nXhnzipenraEUVMW8+TEhRS0asaP9+lCq7wc7nv7S8oqNHOA7BgpSzBmlg+cDPRz9xJggpn9BziDbxNHlTOBB6sSj5ldBzwBXFpXPe7+Wtzn3gW8ncRDE2lQ6hp9dkyfAo7pU8CGTZt587MVvDxtCY9/uKDarAHw7cwBSjCyLSxVdwSb2QDgfXfPi9l2IXCEu/8kruw04AZ3fyZ83QFYCXQAukatJ9x3PvBzdz8kwb4RwAiAjh07HvDss89u/4E2MiUlJbRo0SLdYTQ4ja1dSiuc/31zQ437H/hBc3KyrM56Glu77AiNvU2OPPLIj939wET7UnmJrAVQHLetGGgZoWzV85b1qcfM9gWuAk5IFJC7309wOY1evXp57BrrEohfe14CjbFdbvh4PItrWATt/Lc3cUSvjvygTwGD99qF1s2bJCzXGNtle2Vym6QywZQAreK2tQLWRyhb9Xx91HrMrCfwGvBHd393G2MWyRiJ+m5ym2Txy0O6UbpxM2/MWsGY6UvJyTIO3r0dx/Qp4Oi9C9itXfOtywosXltG4YfjdWOnAKlNMHOAHDPb093nhtv6AzMTlJ0Z7ns2ptxyd19tZuV11WNm3YBxwHXu/lgSjkWk0amr7+b6Yc4nX69l3KzlvDFrOSNfnsXIl2fRuVUzVpZsYnM4Gk2DA6RKyhKMu5ea2YvAtWZ2NsHorxOAwxIUfxR4xMyeAJYCVwCPRKnHzAqB8cDd7v7P5B6VSONS28wBWVnG/l3bsn/Xtlx8bG/mrypl3GfLuWXs7K3JpUpZRSU3vfa5EkyGS/WNlucCecAK4Cngd+4+08y6mlmJmXUFcPexwC3AW8CC8HF1XfWE+84GegBXh3WWmFlJCo5NJKN075DP2Yf3qPGmzWXryjn2b+9w46ufMWHuKspjLr1V0X03jVtK74Nx9zXAsATbFxJ03sduux24vT71hPtGEtwzIyIp0KVNXsLBAa1yc2iX35SH35vPfe98RbOcLAb2aM/39+zA4Xt2ZNaSYi4fNWNrn48urTU+muxSRLZLTTd2XntCP4YNKGTDps1MnLeGd+es4p25K/nLmM+Az74zIWcV3XfTuCjBiMh2iR0csHhtGYVxgwOaN83hyF67cGSvXYBgRc4Jc1dx8QvTE9a3eG3Z1npk56YEIyLbrWpwQJR7Prq0yePUg3bjzjfn1njfzaCbxlPYJo+BPdoxcPd2DNy9Pd3aN8csuNmzali01rtp2JRgRCQtEl9ay+K8o/Ykv2k2E+et4e3ZK3lxStDxX9CqGQfv3p5mOVm8PG0JGzdrzrSGTglGRNKirvtuhg/aHXfny5UlTJy3holfrWHivNUsX7exWl1lFZXcPFbDohsaJRgRSZva7rsBMDN67tKSnru05PSB3XB3elz2KolmUFxaXM7QO95hQNc27LdbGwZ0bUvPXVqQHTOHmi6tpZYSjIjsNMysxmHRLXNz6Nwml9dmLOPpSYsAaNEsh313bc2Arm3YWLGFxycuoFzLEaSMEoyI7FRqGhZ9XTgs2t2Zt6qUTxatZerCtUxd9A3/fPur7yysVqWsopKbdGktaZRgRGSnUlffjZnRo2MLenRswUn77wpA2aZK+lw1NuGltWXF5Rxyw5v07dKKvoWt6Rf+7NI6V6PWtpMSjIjsdOrqu4mX1zS7xktrrfNyOHSP9sxYXMxbs1dsvfmzbfMm9CtsTZNs4925q6io1GSe9aUEIyIZoaZLayOP77c1UZRtquSzZeuYubiYGYvXMXNp8DNeWUUlV42eQdv8pvQqaElBq2Zbz3aqaAkDJRgRyRB1XVqD4EynasboKrtfOibhpbV15Zs586GPAGid14ReBS3Zq1MLehW0ZPm6ch6YMC/jBxQowYhIxqjvpTWoeTLPzq1zueNn+zFn+Xo+X7aeOcvWM/qTJawv35ywnrKKSv4yZhaH9Gif8IynSmPq71GCERGpRU2X1i45tjeH9GjPIT3ab93u7ixbV86hN45PWNeqkk0ccuOb5DfNpkfHFuzRMT/82YIeHfOZsbiYq0bPbDQzTCvBiIjUIsqltSpmRufWeRTWcNbToUVT/jhkT75cWcqXK0uYNP8bXvpkSa2fX1ZRyQ2vfsYP9+lEs5zsGss1xDMfJRgRkTrU99JaTWc9V/y4T7V6NmzazLxVpXy1spTfPzU1YX0r1m+k95Vj6dI6j27tm9OtfXO6tssPfzZnxuK1jHz5swZ35qMEIyKyg9W1hEGs5k1z6NulNX27tOam1z5PeObTtnkTzji0OwtXl7JgzQb+O3M5q0s31RpDWUUl170yi312bU1hmzxymyQ++0nmmY8SjIhIEtRnCYMqNZ35XP2TvtV+6a8vr2DB6g0sXLOBc5+YkrC+1aWbGHLb2wB0aNGMwrZ57No2j13b5FHYNo9Fazbw6AcLtmlm6qrE1LRTzwNqKqMEIyLSQNSnv6dlbnAjaL/C1rX2+Vz+o71Z/E2wiNvX35Qxa8k63pi1nE1hUolXVlHJ5aM+ZcHqDXRunUun1rlbf7bMbQIEySU+ESaiBCMi0oBsy1Dq+vT5AGzZ4qwq2cjAG95MeI/Phk2V3DFuTrXtLZrl0Kl1LovWbNh61lMbJRgRkZ1cfc58ALKyjF1a5dZ4j09hmzzGX3gEK9ZtZNm6cpYWl7OsuCz8Wc4XK0oixaUEIyLSCOzIM5+LhvaiWU42u7Vrzm7tmld736Cbxte43HWsrHpFIyIijcawAYXceNI+FLbJwwjOXG48aZ86E9VFQ3uRV8OotFg6gxERyWDbcuYTe0luaS3ldAYjIiL1NmxAIe9dehSbln3xcU1llGBERCQplGBERCQplGBERCQplGBERCQplGBERCQpUppgzKydmY0ys1IzW2Bmp9VS9gIzW2ZmxWb2kJk1i1qPmQ0xs8/NbIOZvWVm3ZJ5XCIiUl2qz2DuBjYBBcDpwL1m1je+kJkNBS4FhgDdgR7AyCj1mFkH4EXgSqAdMBl4JjmHIyIiNUlZgjGzfOBk4Ep3L3H3CcB/gDMSFD8TeNDdZ7r7N8B1wPCI9ZwEzHT359y9HLgG6G9mvZN3dCIiEi+Vd/LvBVS6e+wUndOAIxKU7QuMjitXYGbtga511NM3fA2Au5ea2Zfh9s9jP8TMRgAjwpcbzWxGvY+q8esArEp3EA2Q2iUxtUt1jb1NauyCSGWCaQEUx20rBlpGKFv1vGWEeloAK6N8jrvfD9wPYGaT3f3A2g8h86hdElO7JKZ2qS6T2ySVfTAlQKu4ba2A9RHKVj1fH6Ge+nyOiIgkSSoTzBwgx8z2jNnWH5iZoOzMcF9sueXuvjpCPd95b9hns0cNnyMiIkmSsgTj7qUEo7uuNbN8MxsEnAA8lqD4o8BZZtbHzNoCVwCPRKxnFNDPzE42s1zgKmC6u38e/yFx7t++I2y01C6JqV0SU7tUl7FtYu6JFsxM0oeZtQMeAo4BVgOXuvuTZtYVmAX0cfeFYdk/AZcAecALwG/dfWNt9cR8ztHAXQSdTxOB4e4+PyUHKSIiQIoTjIiIZA5NFSMiIkmhBCMiIkmR8QmmPvOjZRIzKzKzcjMrCR+z0x1TOpjZeWY22cw2mtkjcfsycs67mtrEzLqbmcd8Z0rM7Mo0hppSZtbMzB4Mf4+sN7OpZvbDmP0Z933J+ARDxPnRMtR57t4ifPRKdzBpsgT4C8Ggkq0yfM67hG0So03M9+a6FMaVbjnAIoJZRVoTfDeeDRNvRn5fUnknf4MTM69ZP3cvASaYWdW8ZpemNThpENz9RQAzOxDYNWbX1jnvwv3XAKvMrHeEIfE7tVraJKOFt1BcE7PpFTObBxwAtCcDvy+ZfgZT0/xoOoMJ3Ghmq8zsPTMbnO5gGphqc94BVXPeZboFZva1mT0c/uWekcysgOB3zEwy9PuS6QmmPvOjZZpLCJZJKCS4UexlM9sjvSE1KPruVLcKOIjg/rMDCNriibRGlCZm1oTg2P8dnqFk5Pcl0xOM5i2rgbtPdPf17r7R3f8NvAf8KN1xNSD67sQJl8+Y7O6b3X05cB7wAzOLb6dGzcyyCGYW2UTQBpCh35dMTzD1mR8t0zlg6Q6iAdGcd3Wruos7Y743ZmbAgwSDhk5294pwV0Z+XzI6wdRzfrSMYWZtzGyomeWaWY6ZnQ58H3g93bGlWnj8uUA2kF3VJmz7nHc7vZraxMwGmlkvM8sK1276O1Dk7vGXhhqze4G9gZ+4e1nM9sz8vrh7Rj8Ihgy+BJQCC4HT0h1Tuh9AR2ASwen7WuBD4Jh0x5WmtriG4C/x2Mc14b6jCRaxKwOKgO7pjjedbQL8ApgX/l9aSjBpbad0x5vCdukWtkU5wSWxqsfpmfp90VxkIiKSFBl9iUxERJJHCUZERJJCCUZERJJCCUZERJJCCUZERJJCCUZERJJCCUakkQrXZjkl3XFI5lKCEUkCM3sk/AUf//gw3bGJpEpGrwcjkmTjCNYWirUpHYGIpIPOYESSZ6O7L4t7rIGtl6/OM7Mx4RK6C8zsl7FvNrN9zGycmZWZ2ZrwrKh1XJkzzezTcPni5fHLOgPtzOy5cEnwr+I/QySZlGBE0mck8B9gP4I1dx4NV4nEzJoDYwnmsjoYOBE4jJhlis3sf4D7gIeBfQmWU4ifnfcqYDTBTL7PAA9lwlrw0jBoLjKRJAjPJH5JMPFhrLvd/RIzc+ABdz8n5j3jgGXu/kszOwf4K7Cru68P9w8G3gL2dPcvzOxr4HF3T7i8d/gZN7n7ZeHrHGAdMMLdH9+BhyuSkPpgRJLnHWBE3La1Mc8/iNv3AfDj8PneBNO5xy5I9T6wBehjZusIVht9s44Yplc9cffNZrYS2CVa+CLbRwlGJHk2uPsX2/he49sFu+LVZ/G3irjXji6NS4roiyaSPockeP1Z+HwW0N/MYtdsP4zg/+xnHixJvBgYkvQoRbaRzmBEkqeZmXWK21bp7ivD5yeZ2SSCxadOIUgWA8N9TxAMAnjUzK4C2hJ06L8Yc1Z0PXCHmS0HxgDNgSHufluyDkikPpRgRJLnaIKVHWMtBnYNn18DnEywtPBK4NfuPgnA3TeY2VDgb8BHBIMFRgN/rKrI3e81s03An4GbgTXAq8k6GJH60igykTQIR3j91N2fT3csIsmiPhgREUkKJRgREUkKXSITEZGk0BmMiIgkhRKMiIgkhRKMiIgkhRKMiIgkhRKMiIgkxf8DT/J7NZoOCisAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The schedule function can take the current learning rate as a second argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exponential_decay_fn(epoch, lr):\n",
    "    return lr * 0.1**(1 / 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to update the learning rate at each iteration rather than at each epoch, you must write your own callback class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 6s 106us/sample - loss: 0.7950 - accuracy: 0.7678 - val_loss: 0.7626 - val_accuracy: 0.7202\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 6s 103us/sample - loss: 0.6238 - accuracy: 0.8097 - val_loss: 0.5616 - val_accuracy: 0.8304\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 6s 108us/sample - loss: 0.5716 - accuracy: 0.8217 - val_loss: 0.6381 - val_accuracy: 0.8120\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 6s 106us/sample - loss: 0.5121 - accuracy: 0.8391 - val_loss: 0.6855 - val_accuracy: 0.8114\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 6s 103us/sample - loss: 0.4785 - accuracy: 0.8495 - val_loss: 0.5208 - val_accuracy: 0.8556\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.4188 - accuracy: 0.8659 - val_loss: 0.5239 - val_accuracy: 0.8542\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.3950 - accuracy: 0.8711 - val_loss: 0.4905 - val_accuracy: 0.8576\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.3658 - accuracy: 0.8799 - val_loss: 0.4733 - val_accuracy: 0.8646\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 6s 109us/sample - loss: 0.3451 - accuracy: 0.8855 - val_loss: 0.4483 - val_accuracy: 0.8688\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.3192 - accuracy: 0.8933 - val_loss: 0.4850 - val_accuracy: 0.8810\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 6s 113us/sample - loss: 0.3002 - accuracy: 0.8986 - val_loss: 0.4633 - val_accuracy: 0.8776\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 6s 115us/sample - loss: 0.2756 - accuracy: 0.9051 - val_loss: 0.4408 - val_accuracy: 0.8822\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 6s 113us/sample - loss: 0.2590 - accuracy: 0.9116 - val_loss: 0.4711 - val_accuracy: 0.8836\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 6s 117us/sample - loss: 0.2500 - accuracy: 0.9178 - val_loss: 0.4527 - val_accuracy: 0.8886\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 7s 122us/sample - loss: 0.2213 - accuracy: 0.9236 - val_loss: 0.4510 - val_accuracy: 0.8814\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.2052 - accuracy: 0.9289 - val_loss: 0.4838 - val_accuracy: 0.8856\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 6s 104us/sample - loss: 0.1892 - accuracy: 0.9343 - val_loss: 0.4641 - val_accuracy: 0.8874\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 6s 109us/sample - loss: 0.1797 - accuracy: 0.9382 - val_loss: 0.4913 - val_accuracy: 0.8932\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 6s 111us/sample - loss: 0.1676 - accuracy: 0.9418 - val_loss: 0.4841 - val_accuracy: 0.8882\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.1548 - accuracy: 0.9471 - val_loss: 0.5274 - val_accuracy: 0.8916\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 6s 116us/sample - loss: 0.1460 - accuracy: 0.9501 - val_loss: 0.5155 - val_accuracy: 0.8922\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 7s 118us/sample - loss: 0.1356 - accuracy: 0.9533 - val_loss: 0.5479 - val_accuracy: 0.8912\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 7s 119us/sample - loss: 0.1270 - accuracy: 0.9570 - val_loss: 0.5588 - val_accuracy: 0.8912\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 6s 108us/sample - loss: 0.1202 - accuracy: 0.9597 - val_loss: 0.6076 - val_accuracy: 0.8948\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 6s 107us/sample - loss: 0.1145 - accuracy: 0.9620 - val_loss: 0.6109 - val_accuracy: 0.8952\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialDecay(keras.callbacks.Callback):\n",
    "    def __init__(self, s=40000):\n",
    "        super().__init__()\n",
    "        self.s = s\n",
    "\n",
    "    def on_batch_begin(self, batch, logs=None):\n",
    "        # Note: the `batch` argument is reset at each epoch\n",
    "        lr = K.get_value(self.model.optimizer.lr)\n",
    "        K.set_value(self.model.optimizer.lr, lr * 0.1**(1 / s))\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        logs = logs or {}\n",
    "        logs['lr'] = K.get_value(self.model.optimizer.lr)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "lr0 = 0.01\n",
    "optimizer = keras.optimizers.Nadam(lr=lr0)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "exp_decay = ExponentialDecay(s)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[exp_decay])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = n_epochs * len(X_train) // 32\n",
    "steps = np.arange(n_steps)\n",
    "lrs = lr0 * 0.1**(steps / s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwU9fnA8c+TgyTkgpAQbsKN3AioiHKI1qNavK1aFZWiVm2r1aotKmq1rWf9eeBRFW9FkcNbqYByiICc4ZIjnAkEArlIQiDP74+Z4LJukk3I7ibkeb9e82J35jvfeebL7j6Zme98R1QVY4wxpraFhToAY4wxxyZLMMYYYwLCEowxxpiAsARjjDEmICzBGGOMCQhLMMYYYwLCEoyp90RktIgUVHOdWSLybKBicreRISJ3BKDei0WkWvcXeLdRTdrsaIjIeBF5NVjb87F9FZGLQ7DdKttZRG4RkenBiimYLMHUYyIy0f3ieE/fhzq2QKngh+J9oGMAtjVGRJaISIGI5IrIchH5R21vJ0QC0ma+iEhz4HagXredmyRXBqDql4GBInJqAOoOqYhQB2CO2gzgKq95B0IRSKioahFQVJt1ish1wP8BtwH/AxoBPYHBtbmdUAlEm1ViDPCDqm4M9IZEJFJVSwO9ndqkqiUi8g7wR+C7UMdTm+wIpv4rUdUsrykHQESGiUipiAwvLywiN4pInoh0dN/PEpEXRORpEdnrTo+JSJjHOk1F5HV3WZGIzBCRnh7LR7t/5Y8UkZUiUigiM0Wkg2egInKeiCwWkWIR2SQiD4tII4/lGSIyTkRedGPcJiJ3ei53X37gHslkeG7fo1wnEZkmIlluLD+KyLnVbNffAB+p6ouqul5VV6nqB6p6u9c+/VpEFrjtskdEPhaRaI8i0RXtj7t+ooi8JCK7RCRfRGaLyECvMleLyGYR2S8inwCpXst/8Zd1VadmfLTZePf/7rcissGNZaqIJHuUiRCRpzw+J0+JyAQRmVVFW14BHHEKyM/PXSMR+bfbboUislBEzvRYPtz9HJwjIj+IyAHgTCrWQkQ+ddtxs4j8ziumf4nIWvf/MkNEHi3/vxSR0cD9QE/5+UzBaHdZgtsOme5ne7WIXOZVd6XfDbd9fiMijatoy/pFVW2qpxMwEfikijKPAFuBJKA7UAhc47F8FpAPPOMuvxTIBW73KDMNWAMMBXrjfBm2AjHu8tFAKc7R1AlAH2AJ8KVHHWcCecC1QCdgBLAWeNyjTAawB7gF6AzcCigw2F2e4r4fA7QAUjy2X+BRT1/gRjfWzsDfcY7qunvt97OVtNsLwDqgYyVlzgIO4pz66eHu9x1AYz/3R4A5wKduu3UGHnLbqaVb5kSgzN2HrsANbp3qEcd4YKVXbN5tUtX78UABMMXdj8HAZuBFjzJ3A3uBi4BuwNPuZ2VWJW2U5MZ/stf8WVT9uXsb+B7nc9fRbccDQF93+XC3PVcAv3LLpFQQh7rtdoPbjn934xroUeZeYAiQBpwDbAEecpfFAI/jfA9auFOM+384F1jlfh46AmcDF/j73XDLNQYOASND/btSq79RoQ7ApqP4z3MSzEH3h8Fz+rdHmUhgIfAR8CPwvlcds3B+SMVj3jhgm/u6i/vlHOqxPNH9MRjjvh/tlunmUeZK98cgzH3/LXCv17bPd+MV930G8K5XmZ+AcR7vFbjYq8xoPH4sK2ir773qmUXlCaYlMN/d3k/AW8DVQKRHmbnAe5XUUen+AKe5+x/jVWYp8Ff39TvA117L/0tgEkwxkOgx7+/Aeo/3mcDdHu8F5wd3ViVt0M9tww7V/Nx1wkkA7bzWmwo8774e7tZ9kR/fFQVe9po3A3irknVu9Np/X+18hhvncRXUMZoqvhse83OA66val/o02Smy+u9bnC+x5/RY+UJ1zkdfAZwLNMf5C87b9+p+wl3zgdYikgAch/MFmu9RZy7OX409PNYpUdW1Hu934CS3Ju77AcDf3VNpBe7pmXeAWJy/Bsst94pthxu330Qk1j29sco99VIADATa+VuHqmaq6mCco6D/4PyYvgj84HEaoz/O9ZnKVLY/A3D+cs32apdeOD+w4LT/fK86vN/Xls3u/+0vYhWRRJz/px/KF7qfmYVV1Bnj/lvsY1lln7vjcdp8lVfb/Jqf26bcoipi8Kzf+/3hz7A4vfPmuKdWC4CnqPoz0x/IVNXVlZSp6rtRroif2+uYYBf567/9qrq+ijIn4Vxva4JzmmlfNeqXSpZ5/jgcrGBZmMe/DwAf+Kgn2+O19wVapfrXCh/HOV1xB84Rw37gDZwL9dWiqiuBlcBzInIKzkXYS3GOHv1R2f6EATsBX72H8tx/K2v/cmU+ykX6GZ8nf9q+usOv73b/bYpzBOSvMHdbg3zE5d05obCaMf2CiJwEvIfzGb0N5zvyG5zPUqWr+lF9Vd+Nckkc+V2o9+wI5hgnImnAs8DNwNfA2yLi/YfFiSLi+UU5Cdihqnk455bD8Og95f6F2dtd5q8fca6BrPcxeX8BK1MKhFdR5hTgDVWdrKrLgW388q/emijf3zj33yXAyKOo70ecC/ZlPtpkl8c2T/Jaz/t9NpDq9X/Y7yji+gX3yCYL5zoCAO72BlWx6gacZNnDx7LKPndLcH68W/hom+013A1f7Vh+5DEE2K6qD6nqQlX9CWjvVf4Av/zs/Qi0FJHjahgT4HRMAaLd+o4ZdgRT/0WJSAuveYdUNVtEwnGuHcxW1RdF5EOcU1v341zQLNcK+I+IPI+TOO7EvWdBVX8SkWnAiyIyFucvu4dxfjTeqUacDwKfiMhmYBLOX3W9gBNU9a/VqCcDGCkis3FOPez1UWYdcIEbdynO/kb7KFchEZmAcyrjG5wE1RLnGsF+4Cu32MPAxyKyHqctBOdi84uqut+PzczAuY4zTUT+ys8XkM8CZqjqdzhdpeeJyD3AhzjXHS7wqmcWzl+/fxOR99wygbip8GngryKyDifx3YDTLhUemahqmYjMwEn6H3otruxzt05E3gYmishfcH54k3D2baOqflSD+C8UkYU47XUxzh8HJ7rL1uGcnrsS59TZmcDlXutnAO1F5HicDgD5OKdIFwCTReQ2t57OQKyqTq1GbKe6+/VTDfarzrIjmPrvdJwvuOe0xF32N5wP+/UAqroHuAa42z3dU+5tnL/MFuDc9PUKzvnnctfinHuf7v7bGDhLnXsp/KKqX+KcPx/h1vEDTq+kLf7vKgB/cevYys/76e12YBfO6azPcS7wV/f+gq9xfnwm4fxoTHHnn6Gq6wBU9TOcH/uz3Vhmu7GV+bMB9/rDOThJ7GWcXnWTcHpo7XDLfI/z/3cTzvWcC3EuNnvWs9pdPtYtcwZO78Ha9jjwJvAaTpuC0y6+rq94egm4zP2Dx5M/n7vXgEdxku8nOD3KNtcw/vE4PeCW47TXtaq6EEBVP8a5dvkffm7D+7zWnwx8hpNUsoHLVbUM5/9/Ls4fc6txEnF1T8dejtMGx5Ty3jumgXLvYVipqreEOhZT/4jIj8BcVb21inLzcXp/vem+n4V97gAQkV44SaurVyeLes9OkRlj/CIi7XFOHc3G+e0Yi3PP0Vg/Vr8Bp8eV+aVWwNXHWnIBSzDGGP+V4dwL9BjO6fVVwNmqWmU3YbezhXeXbQOo6ldVl6qf7BSZMcaYgLCL/MYYYwLCTpG5mjRpop07dw51GD4VFhYSGxsb6jB8sthqxmKrGYutZgIZ2+LFi3eraorPhaEeq6auTF27dtW6aubMmaEOoUIWW81YbDVjsdVMIGMDFqmNRWaMMSaYLMEYY4wJCEswxhhjAsISjDHGmICwBGOMMSYgLMEYY4wJCEswxhhjAsISjDHGmICwBGOMMSYgLMEYY4wJCEswxhhjAsISjDHGmICwBGOMMSYgLMEYY4wJCEswxhhjAiKoCUZEkkRkiogUishmEbmikrK3iUiWiOSKyKsiEuWx7BYRWSQiJSIy0ce6I0VkjYjsF5GZItI+QLtkjDGmAsE+gnkOOACkAlcCE0Skp3chETkTuBsYCaQBHYEHPIrsAP4BvOpj3WTgI+BeIAlYBLxfVWBlWr0dMcYYU7mgJRgRiQUuAu5V1QJVnQNMB67yUfwa4BVVTVfVvcBDwOjyhar6kapOBfb4WPdCIF1VP1DVYmA80FdEulcW39b8MhZs9FWdMcaYmhDniZdB2JBIf2CeqsZ4zLsDGKaq53mVXQY8oqrvu++TgWwgWVX3eJT7B9BGVUd7zHsaaKSqN3nMWwncr6qTvbYzFhgL0KhF5wGDbn6a8SfHEBEmtbXbtaKgoIC4uLhQh+GTxVYzFlvNWGw1E8jYRowYsVhVB/paFhGQLfoWB+R6zcsF4v0oW/46Ht9HLd7rZvuzHVV9CXgJILZVF91WoGREtmfMqR2r2ERwzZo1i+HDh4c6DJ8stpqx2GrGYquZUMUWzGswBUCC17wEIN+PsuWvfZU9mu0clhTjHLX8Z8ZP7Mwr9mMzxhhjKhPMBLMOiBCRLh7z+gLpPsqmu8s8y+30PD1WiSPWda/9dKpgO4c1jhDO6JFKQclBHv50tR+bMcYYU5mgJRhVLcTp3fWgiMSKyBBgFPCmj+JvANeLSA8RaQqMAyaWLxSRCBGJBsKBcBGJFpHy031TgF4icpFb5j5guaquqSrG+87tQXRkGNOX7WDeht1HsbfGGGOC3U35D0AMsAt4F7hJVdNFpJ2IFIhIOwBV/QJ4FJgJbHan+z3qGQcU4XRl/p37epy7bjZOb7WHgb3AicBv/QmubVJjbhnRGYD7pqVTeqjsqHbWGGMasmBe5EdVc4DzfczfgnNx3nPek8CTFdQzHqf7cUXbmQFU2i25Ir8f2pEPF29j/a4CXpu7ibFDO9WkGmOMafBsqBgvURHhPDCqF+Bc8M/MLQpxRMYYUz9ZgvFhWNcUzu7Vgv0HDvGPT+yCvzHG1IQlmAqMO7cHMZHhfLoik5lrdoU6HGOMqXcswVSgdZMYbj+jKwDjpq5k/4GDIY7IGGPqF0swlbh2SBo9WyWwfV8RT329LtThGGNMvWIJphIR4WH868I+hAm8MmcTK7d7j3RjjDGmIpZgqtC7TSLXDelAmcLdHy3noN0bY4wxfrEE44fbzuhK6yYxrNyex8R5GaEOxxhj6gVLMH6IjYrgH+c798Y88dU6tubsD3FExhhT91mC8dOI7s05t09LikoPMW7qSoL1HB1jjKmvLMFUw33n9SAhOoLZ67KZvmxHqMMxxpg6zRJMNTSPj+Zv5xwHwPjp6ewuKAlxRMYYU3dZgqmmywa15ZTOyezdX8p901aGOhxjjKmzLMFUk4jwzwt7E9sonM9WZPHp8sxQh2SMMXWSJZgaaJvUmHvcU2X3TVvJHjtVZowxv2AJpoauOKEdgzs2Y0/hAe6fXunTmI0xpkGyBFNDYWHCoxf3oXGjcD5ZnskXK+1UmTHGeLIEcxTaJjXm7rOdB2eOm7qSvYUHQhyRMcbUHZZgjtLvTmzPiR2S2F1wgPEf26kyY4wpZwnmKJWfKouODGPa0h18tsJOlRljDFiCqRXtm8UevgHzb1NWsCuvOMQRGWNM6FmCqSVXndSeoV1T2Le/lDs/XG5jlRljGjxLMLVERHjs4j40aRzJ7HXZvLVgS6hDMsaYkLIEU4tSE6J55ILeADz86So2ZBeEOCJjjAkdSzC17JzeLbmwf2uKS8u4/f2llNoTMI0xDZQlmAAYP6onrZvEsGxbLs9+sz7U4RhjTEhYggmAhOhIHr+kLyLw7Mz1LNmyN9QhGWNM0FmCCZDBnZox5pQOHCpT/vz+UvKLS0MdkjHGBJUlmAD6y6+60aNlApv37Oe+aXaXvzGmYbEEE0DRkeH83+X9iYkMZ8qS7UxevC3UIRljTNAENcGISJKITBGRQhHZLCJXVFL2NhHJEpFcEXlVRKL8rUdELhWR1SKSLyKrROT8QO5XZTo3j+OB3/QE4N5pK9loXZeNMQ1EsI9gngMOAKnAlcAEEenpXUhEzgTuBkYCaUBH4AF/6hGR1sBbwO1AAnAn8I6INA/MLlXtkoFtOK9vK/YfOMSt7y6h5OChUIVijDFBE7QEIyKxwEXAvapaoKpzgOnAVT6KXwO8oqrpqroXeAgY7Wc9bYB9qvq5Oj4FCoFOAdy9SokID1/Qi7ZJMaTvyOPfn68NVSjGGBM0Eqwxs0SkPzBPVWM85t0BDFPV87zKLgMeUdX33ffJQDaQDLSrrB4RCQe+AZ4APgXOA54Fuqlqodd2xgJjAVJSUgZMmjSplvf6SBv3HeLhBcUcUvjz8VH0ax7h13oFBQXExcUFNLaasthqxmKrGYutZgIZ24gRIxar6kCfC1U1KBNwKpDlNe/3wCwfZTcAZ3m8jwQU53RZlfUA1wMFwEFgP/DrquLr2rWrBsOEWeu1/V2faP8Hv9LMfUV+rTNz5szABnUULLaasdhqxmKrmUDGBizSCn5X/T5FJiKpInKHiExwjygQkSEi0sHPKgpwrol4SgDy/Shb/jq/qnpE5HTgUWA40AgYBvxXRPr5GWdAjT21I6d2SSan8AC3vvujDSVjjDlm+ZVgRGQAsBbngvr1/PwDfwbwsJ/bWgdEiEgXj3l9AV83iKS7yzzL7VTVPX7U0w/4VlUXqWqZqi4EFgCn+xlnQIWFCU9d1o/UhCgWZuzlsS/teowx5tjk7xHM48DTqtofKPGY/yUwxJ8K1Ln+8RHwoIjEisgQYBTwpo/ibwDXi0gPEWkKjAMm+lnPQuDU8iMW99rPqcByP/c14JLjonj2iuMJDxNe+nYjX6zMCnVIxhhT6/xNMAOA133Mz8TpKuyvPwAxwC7gXeAmVU0XkXYiUiAi7QBU9Quc01wzgc3udH9V9bjrzgbGAx+KSD4wGafDwFfViDPgBqUlcc/Z3QG484NlZOwurGINY4ypX/zrxgRFQFMf87vj/Mj7RVVzgF/c9KiqW4A4r3lPAk9Wpx6P5c/i9Byr064/pQMLM3L4Mn0nN739I1P+cDLRkeGhDssYY2qFv0cw04D7Pe6mVxFJA/6Nc4RgakBEeOySvqQ1a8zqzDzut/HKjDHHEH8TzB1AEs69KI2BOcB6YB/O9RFTQwnRkTx/5QCiIsJ4f9FWJi3aGuqQjDGmVviVYFQ1T1VPwTktdRfwNM59KsPU6+ZFU309WiXw0Pm9ALh36kpWbs8NcUTGGHP0/O2mfLWIRKnqN6r6uKo+qqozRKSRiFwd6CAbgksHtuW3g9pScrCMsW8sYndBSdUrGWNMHebvKbLXgEQf8+PdZaYWPDCqJ/3bNWFHbjE3v203YRpj6jd/E4zgDNXirR1g53NqSVREOC/8bgDN46NYsCmHf3yyKtQhGWNMjVXaTVlEVuAkFgVmi8hBj8XhQHvgs8CF1/CkJkTzwlUD+O2L3/P6/M30bJVIyJ4zYIwxR6Gq+2A+dP/thTMysefTsg4AGVg35Vp3fLumPHR+T+6avIJxU1fy10GNGB7qoIwxppoqTTCq+gCAiGQA76tqcTCCMnDZoHak78jjjfmbeXZJCb85rZjmCdGhDssYY/zmbzfl1y25BN+95/bghA5J7CtRbnhrMcWl9iRMY0z94W835UYi8oCIrBORYhE55DkFOsiGKjI8jOevPJ6kaGHJln389cPl5c+7McaYOs/fXmQP4TzG+AmgDOc5988Be3AGnjQBkhwXxW0DooltFM70ZTt4+n8/hTokY4zxi78J5lLgRlV9ETgETFPVP+KMcHxGoIIzjrbxYTxzRX/CBP4z4yemLd0e6pCMMaZK/iaYVKD8powCoIn7+gvgV7UdlPml07qnMu7XPQC488PlLN6cE+KIjDGmcv4mmC1AK/f1euBM9/VgnKH8TRBcOySN353UjgMHyxj7xmK25uwPdUjGGFMhfxPMFGCk+/pp4AER2YTzlMn/BiAu44OIcP95PTm1SzJ7Cg9w3cSF5BWXhjosY4zxyd9uyveo6sPu6w+BU4BngAtV9e8BjM94iQwP49krjqdz8zh+2lXAzW//yIGDNmaZMabu8fcI5giqukBVn1TVT0QktraDMpVLjInk1WsG0Sy2Ed/9tJu7J1v3ZWNM3VOjBAMgItEiciewqRbjMX5q16wxr107iMaNwvloyXYe+3JtqEMyxpgjVJpg3BssHxaRhSIyT0TOd+dfDWwE/gw8FYQ4jQ992jThuSuPJzxMeH7WBt6YnxHqkIwx5rCqjmDGA7cAm4EOwAci8jzwd+AeIE1V/xnQCE2lRnRrzr8u7A3A/dPT+WJlVogjMsYYR1UJ5lJgtKpeDJyFM0R/U6CnOz6ZdWGqAy4Z2JY7ftUVVfjje0tYmGH3yBhjQq+qBNMWWAigqstwhuj/t6oerHQtE3Q3j+jMlSc698iMeX0RP+3MD3VIxpgGrqoEEwl4Phy+FHuCZZ0kIjw4qhdn9Eglt6iUq175wW7ENMaEVFUPHAP4p4iU/1I1AsaLyBFJxh2XzIRYeJjwf7/tzzWv/sAPGTlc9coCJt04mObx9hwZY0zwVXUE8y3QCejtTvOAdh7ve+M87dLUETGNwvnv6IH0ap1Axp79XP3KD+zbfyDUYRljGqCqnmg5PEhxmFqUEB3J69eewKUvzmdNVj6jX1vI22NOJDbKnwNWY4ypHTW+0dLUbc3ionhrzIm0bhLD0q37GPvmInsipjEmqCzBHMNaJsbw9pgTSY6LYu76Pdz67hIOHrJxy4wxwRHUBCMiSSIyRUQKRWSziFxRSdnbRCRLRHJF5FURifK3HhFpLCLPi8hud/1vA7lfdVlacixvjTmBxJhIvl61k798sIxDZTZumTEm8IJ9BPMczr00qcCVwAQR6eldSETOBO7GeURAGtAReKAa9bwEJAHHuf/eVts7Up90b5HAa9cOIrZRONOW7uBOSzLGmCAIWoJxR12+CLhXVQtUdQ4wHbjKR/FrgFdUNV1V9wIPAaP9qUdEugG/AcaqaraqHlLVxQHevTrv+HZNmXjdCYcHx7xr8nLKLMkYYwJI/BnmXUTaVbBIgWJVzfajjv7APFWN8Zh3BzBMVc/zKrsMeERV33ffJwPZQDJON+kK63EH4rwTmIGTdDKB8ao62UdMY4GxACkpKQMmTZpU1W6EREFBAXFxcbVS19qcQzyxuJgDh2BomwhG92xEmEidiK22WWw1Y7HVTEONbcSIEYtVdaDPhapa5QSUAYcqmfYCTwIRldRxKpDlNe/3wCwfZTcAZ3m8j8RJZmlV1QP8zS07HufG0GFAAXBcZfvYtWtXratmzpxZq/XNW79bu437TNvf9YnePXm5HjpUVuO6aju22mSx1YzFVjMNNTZgkVbwu+rvKbLLgW3AOOAMdxoHbAGuc3/MrwLuraSOAiDBa14C4GvQLO+y5a/z/ainCGdIm3+o6gFVnQ3MBH5VSWwNyuBOzXjlmkFERYTx7g9buG/6SntgmTGm1vmbYG4CblPVf6rqN+70T+AvwHWq+jTwR5xEVJF1QISIdPGY1xdI91E23V3mWW6nqu7xo57lfu5TgzakczIvXz2QRhFhvPX9Fu6dttKuyRhjapW/CeZEYIWP+SuBQe7r+UCbiipQ1ULgI+BBEYkVkSHAKOBNH8XfAK4XkR4i0hTnaGmin/V8i3NkdY+IRLjLhwNf+rmvDcbQrim8dNWAw0nmrsnLrXeZMabW+JtgNuNeDPfye5wfc4AUoKoHkfwBiAF2Ae8CN6lquoi0E5GC8s4EqvoF8CjOqa3N7nR/VfW465biJJxzcEZ+fhm4WlXX+LmvDcrwbs159ZpBxESG88Hibfz5/aWU2s2Yxpha4O/gVH8BJovIOTjPh1GcI5dOOF2Gcd9X2g1LVXOA833M3wLEec17EqfjgN/1eCxPBwZXFov52Sldknn9uhO4buJCPl62g5LSQzxzRX+iIsJDHZoxph7z6whGVT8FuuDcb5IANHFfd1PVz9wyz6vq7YEK1ATWCR2SeGvMiSRER/DVqp2MfWOxjV1mjDkqft9oqapbVfUeVb1QVS9Q1b+5Rx7mGNGvbRPeHXsSSbGNmL0um2tfW0hhiT281BhTM34nGHd8r5NF5HwRudBzCmSAJrh6tkrk/bEn0Tw+ivkb93Dlfxewt9CeJ2OMqT6/EoyInI5zoX0OTg+uDz2mDwIWnQmJLqnxTLph8OGh/i95cT479hWFOixjTD3j7xHM08CnQBtVDfOa7ErwMSgtOZbJN51M19Q41u8q4KIJ81i/y9c9scYY45u/CSYNeEhVdwQwFlPHtEiMZtINgxnQvimZucVc/MJ8lmzZG+qwjDH1hL8JZi7QLZCBmLqpSeNGvHX9iYzs3px9+0u54uUFzFq7K9RhGWPqAX8TzAvA4yIyRkROFJHjPadABmhCL6ZROC9cNYCLjm9DUekhxry+iKlLtoc6LGNMHefvjZYfuv++5GOZAnYd5hgXGR7G45f0ITmuES9+u5E/v7+U7fuK6IENLWOM8c3fBNMhoFGYekFEuOec40iJj+Lhz1bz2JdrGdomglOGlhEZHuyHoxpj6jp/7+TfXNkU6CBN3TLm1I5MuHIA0ZFhfLvtINe+tpC84tJQh2WMqWMqPIJxb6D8WFVLq7qZUlU/qvXITJ12Vq8WvJc4mKtfnsuc9bu5eMI8Xh09iDZNG4c6NGNMHVHZKbIPgRY4IxZ/WEk5uwbTQPVr24R7T4rhxTXhrNtZwAXPz+PVawbRu01iqEMzxtQBFZ4ic2+i3OXxuqLJkksDltI4jMk3nszgjs3Izi/h0hfn8/mKzFCHZYypA+zKrDlqiY0jef26Ew53Y77p7R956ut19oRMYxo4f3uRISJtgVOB5nglJvfZLaYBaxThdGPu1iKOf32+hqf/9xNrs/J54tK+xEb5/TEzxhxD/Prmi8iVwKvAQSAbjrj5QangwWCmYRERxg7tRJfUeP747hK+SM8iY0IhL189kLZJdvHfmIbG31NkDwJPAAmqmqaqHTymjgGMz9RDI7o1Z+rNQ+iYHMuarHxGPTeX7zfuCXVYxpgg8zfBpAL/VVV7xKHxS6eUOKbcPIRhXVPIKTzA7/67gDfmZ6Bq12WMaSj8TTCfAYfjM7EAAB9TSURBVCcGMhBz7EmMieTV0YMYO7QjB8uU+6alc9v7S9l/wJ6SaUxD4O/V16+Bf4tIT2AFcMRt23ajpalIeJjwt3OOo1frRO6evJypS3ewKjOPCb8bQKeUuFCHZ4wJIH8TzIvuv3/zscxutDRV+k3fVhzXIp4b31rMup0FjHp2Lo9d3Ieze7cMdWjGmADxdywyu9HSHLUuqfFMu+UUft2nJQUlB7np7R95+NNVlB4qC3VoxpgAqDLBiEikiCwQEXvgmDlqcVERPHt5f+47twcRYcLL323iypcXkJlbFOrQjDG1rMoEo6qlOMP1W/cfUytEhOtO6cB7Y08iNSGKHzJyOOfp75ixameoQzPG1CJ/e5G9Dvw+kIGYhmdgWhKf/vFUhndLYe/+Usa8sYjx09MpOWi94Y05Fvh7kT8WuFJEzgAWA4WeC1X1j7UdmGkYkuOiePWaQbwyZxOPfrmGifMy+GFTDs9c0d96mRlTz/l7BHMc8COwF+gI9PaYegUmNNNQhIUJvx/akck3nUz7Zo1ZlZnHec/M4cPF2+zGTGPqMb+OYFR1RKADMaZPmyZ8cuspjJu6kmlLd3DHB8v4dl02D43qRWLjyFCHZ4ypJhuu39Qp8dGR/Oeyfjx2cR9iIsOZvmwHZ/7nW+b8tDvUoRljqsnvBCMiI0TkJRH5QkS+8ZyqUUeSiEwRkUIR2SwiV1RS9jYRyRKRXBF5VUSiqluPiNwvIioip/sbowk9EeGSgW357E+n0r9dE7LyivndKwsYPz2dogPWAcCY+sKvBCMio4HPgXhgOM6Q/U2B44FV1djec8ABnMEzrwQmuMPPeG/vTOBuYCSQhnPd54Hq1CMinYCLAXu8Yj3VITmWD24YzJ1ndiMiTJg4L4NfP/Mdy7buC3Voxhg/+HsEcwdwi6pejjMO2T2q2h94CyjwpwIRiQUuAu5V1QJVnQNMB67yUfwa4BVVTVfVvcBDwOhq1vMscBdOIjL1VER4GDeP6MzUm4fQpXkcG7MLuXDCPJ76ep2NAGBMHSf+9NIRkf1AD1XNEJHdwGmqulxEugOzVLWFH3X0B+apaozHvDuAYap6nlfZZcAjqvq++z4Z56gpGWhXVT0icgnwO1UdJSIZwBhVneEjprHAWICUlJQBkyZNqrItQqGgoIC4uLrZZTeYsR04pEz+6QBfZRxEgXbxYVzfuxHtE3yPVmTtVjMWW8001NhGjBixWFUH+lrm730we3BOjwFsx+mavBxoBsRUtJKXOCDXa16uR72VlS1/HV9VPSISBzwC/KqqgFT1JeAlgG7duunw4cOrWiUkZs2ahcXm+NVImL9hD3d+uIwte4t48PsSbhjakT+O7EJ05JGJxtqtZiy2mrHYfsnfU2Tf8fMP9iTg/0TkNeBdnKH8/VEAJHjNSwDy/Shb/jrfj3oeAN5U1U1+xmXqmcGdmvHln4dy7ZA0ylR5ftYGfv1/37F4c06oQzPGePA3wdyCk0wA/gk8hnP0MgkY42cd64AIEeniMa8vkO6jbLq7zLPcTlXd40c9I4E/uj3QsoC2wCQRucvPOE09EBsVwf3n9eTDGwfTKSWWDdmFXPzCfMZPT6ewxB5oZkxd4O+Nljker8uAf1d3Q6paKCIfAQ+KyBigHzAKONlH8TeAiSLyNk4vsHHARD/rGQl43pW3ELgdpxecOcYMaO+MZ/bMNz/xwuyNTJyXwYzVO3no/F5IqIMzpoGrzn0wqSJyh4hMcC+6IyJDRKRDNbb3B5xrNrtwjohuUtV0EWknIgUi0g5AVb8AHgVmApvd6f6q6nHX3aOqWeUTcAjYq6p+9XYz9U90ZDh3ntmdaTcPoUfLBLbtLeLa1xby7JJiewyAMSHk1xGMiAwA/gdsAnrinCLbDZwBdAUqvGHSk3skdL6P+VtwLt57znsSeLI69VRQNs2fcqb+69U6kWm3DGHi3AyemrGORTsPcfoTs7ntjK6MPjmNiHAbuMKYYPL3G/c48LR770uJx/wvgSG1HpUxNRQZHsbvh3Zkxu3DGJAaTuGBQ/zj09Wc+8wcFm/eG+rwjGlQ/E0wA3CeCeMtE+duemPqlFZNYri1fzSvjh5Im6YxrMnK56IJ87h78nJyCu3eW2OCwd8EU4QzNIy37jjXQYypk07rnsrXtw3j5hGdiAwX3lu4leGPzeS1uZtsJABjAszfBDMNuN9jwEkVkTSc3mSTAxCXMbUmppHTCeDzP53KqV2SySs+yAMfr+Lsp79j9rrsUIdnzDGrOmORJeEM19IYmAOsx7mDflxgQjOmdnVuHs8b153Ay1cPJK1ZY9bvKuCaV3/g+okL2ZhtnQyNqW3+3geTB5wiIqfhjKAcBvzoa3wvY+oyEeGMHqkM7ZrMxLkZPPPNev63Zhff/pTN6JPTuHVkFxKi7eFmxtSGavXbVNVvVPVxVX1UVWeISHsRqZsjRBpTiaiIcG4Y1olv7hjGpQPbcLBMefm7TQx7dCavzNlEyUF77owxR+tobwxogjN0vjH1UvP4aB69uC/Tbz6FE9KS2Lu/lIc+WcXIJ2Yzdcl2ysqqHm3cGOOb3XlmDNC7TSLv33ASr1wzkK6pcWzbW8Sf31/Kec/O4bufrCOAMTVhCcYYl4gw8rhUPv/TUB69qA8tEqJJ35HHVa/8wFWvLGDldu+nRBhjKmMJxhgv4WHCpYPaMvOO4fz1rG7ER0fw3U+7OfeZOdz45mLWZOWFOkRj6oVKe5GJyPQq1vd+Losxx4yYRuH8YXhnLh/UjudmrufN7zfzRXoWX6Rn8es+Lbnt9C50bu7reXnGGKi6m/IeP5bbg73MMa1pbCPGnduD3w/tyIRZG3hnwRY+XZ7JZysyGdW3FX86vSsdkmNDHaYxdU6lCUZVrw1WIMbUdakJ0Yz/TU/GDu3I87PW8/7CrUxduoOPl2dyQf/W3DKiM2mWaIw5zK7BGFNNrZrE8I/ze/PNX4bz20FtAfhw8TZOe2IWt767hNWZdo3GGLAEY0yNtU1qzL8u6sM3fxnGZQPbEh4mfLxsB2c//R3XTVzI4s05VVdizDHMEowxR6l9s1j+fXEfZt85gmuHpBEdGcY3a3Zx0YT5XPbifL5dl42q3bBpGh5LMMbUklZNYrj/vJ7Mves0bj2tM/HRESzYlMPVr/7Ab56dy7Sl2+0RAaZBsQRjTC1rFhfFX37VjXl3n8ZdZ3UnOa4RK7bn8qf3ljL00Zm8OHsDhaV2RGOOfX6NpmyMqb746EhuGt6Ja4ek8dGP23llzkY2ZBfyz8/XEB0Oi4rTufbkDrRr1jjUoRoTEHYEY0yARUeGc8WJ7fj6tmG8NnoQQzo3o/gQvDY3g+GPz+SmtxazKCPHrtOYY44dwRgTJGFhwojuzRnRvTlvTP8fy4qTmb5sO5+vzOLzlVn0aJnAVYPbM6pfKxo3sq+mqf/sCMaYEGiXEM4Tl/Zl7l2nccuIziTFNmJVZh73fLSCEx/5Hw98nM4Ge8qmqecswRgTQs0TornjzG7Mv+c0nrqsL8e3a0J+8UFem5vByCdm87v/LuCLlVkctN5nph6y43Bj6oCoiHAu6N+GC/q3YeX2XN76fjNTl25nzvrdzFm/m5aJ0VwyoA2XDGxL2yTrFGDqBzuCMaaO6dU6kX9d1IcFfzud+87tQcfkWDJzi/m/b9Zz6qMzufK/3zNt6XaKS+2xzqZusyMYY+qoxJhIrjulA6NPTuP7TXuYtHArn6/MYu76Pcxdv4fEmEjO79eKSwe1pWerxFCHa8wvWIIxpo4LCxNO7pTMyZ2SeWB/KdOXbef9RVtZuT2P1+dv5vX5m+nVOoFLBrTl3D4taRYXFeqQjQEswRhTryQ2juSqwWlcNTiN9B25TFq4lSlLtrNyex4rt6fz0CerGNo1hQv6t+b041KJaRQe6pBNAxbUazAikiQiU0SkUEQ2i8gVlZS9TUSyRCRXRF4VkSh/6hGRk0TkaxHJEZFsEflARFoGet+MCbaerRJ5YFQvfvj76Tz9236M6JaCAt+s2cWt7y5h0MMzuOODZcxdv5tDZXYTpwm+YB/BPAccAFKBfsCnIrJMVdM9C4nImcDdwGnADmAK8IA7r6p6mgIvAV8CB4FngdeAswK7a8aERnRkOKP6tWZUv9bsLijhk2U7mLJ0B8u27uPDxdv4cPE2UhOiGNWvNef1aUWv1gmISKjDNg1A0BKMiMQCFwG9VLUAmCMi04Gr+DlxlLsGeKU88YjIQ8DbwN1V1aOqn3tt91lgdgB3zZg6IzkuitFDOjB6SAc2ZhcwdekOpi7Zzpac/bz07UZe+nYjbZNiOKd3S37duyW9WydasjEBI8Ea/0hE+gPzVDXGY94dwDBVPc+r7DLgEVV9332fDGQDyUA7f+txl/0Z+K2qnuRj2VhgLEBKSsqASZMmHf2OBkBBQQFxcXGhDsMni61mghmbqrJhXxnzMw+yaOchckt+/s6nxAiDWkQwqEU4aQlhiIi1Ww011NhGjBixWFUH+loWzFNkcUCu17xcIN6PsuWv46tTj4j0Ae4DRvkKSFVfwjmdRrdu3XT48OGV7kCozJo1C4ut+iy2n40AxgCHypRFGTl8tiKTz1ZmkZ1fwmebSvlsU6lzZNOrBSmlh7hu6DDCwurekY39n9ZMqGILZoIpABK85iUA+X6ULX+d7289ItIZ+Bz4k6p+V8OYjTmmhIcJJ3Zsxokdm3HfeT2PSDZbc4p48duNALyQ/j9OP645Z/RIZUjnZKIjrTeaqb5gJph1QISIdFHVn9x5fYF0H2XT3WWTPMrtVNU9IlJcVT0i0h6YATykqm8GYF+Mqfd8JZvPV2bx8Y+b2V1QwnsLt/Lewq3ERIZzapdkzuiRysjjUkmKbRTq0E09EbQEo6qFIvIR8KCIjMHp/TUKONlH8TeAiSLyNpAJjAMm+lOPiLQGvgGeU9UXArtXxhwbPJPNsPhdpHYbwNerdjJj9U5WbM/lq1U7+WrVTsIEBrZP4rTjmjO8WwrdUuOtk4CpULC7Kf8BeBXYBewBblLVdBFpB6wCeqjqFlX9QkQeBWYCMcBk4P6q6nGXjQE6AveLyOF1VLVuXn0zpo4REXq0SqBHqwT+dHoXMnOLmOEmmO837uGHjBx+yMjhX5+voWViNMO6pjCsawpDuiSTEB0Z6vBNHRLUBKOqOcD5PuZvwbl47znvSeDJ6tTjLnsA554ZY0wtaJkYc3j0gPziUmavy2bW2mxmr8smM7f48Km0iDDh+PZNGdY1heHdUujR0u63aehsqBhjjN/ioyM5t08rzu3TirIyZVVmHrPXZTN7bTaLt+zlh005/LAph8e+XEvz+CiGdE7m5E7NGNI5mVZNYqregDmmWIIxxtRIWJjQq3UivVoncvOIzuQWlTJv/W5mrc1m1rpd7MwrYcqS7UxZsh2AtGaNOblzMkM6JTO4UzPrLNAAWIIxxtSKxJhIzu7dkrN7t0RVWbezgLnrdzNvw24WbMwhY89+MvZs4Z0FWwA4rmUCQzo14+TOzRiYlmTXb45BlmCMMbVOROjWIp5uLeK57pQOHDxUxortuczbsIe563ezaPNeVmfmsTozj//O2USYOAlnUFoSJ3RIYlBaEinx9tiB+s4SjDEm4CLCw+jfrin92zXl5hGdKS49xI+b9zJ3w27mbdjDim25pO/II31HHhPnZQDQITmWQWlNDyeddvao6HrHEowxJuiiI8M5uXMyJ3dOBqDowCGWbN3Lwk17WZiRw49b9rJpdyGbdhcyadE2AJrHR9E+9iDrwzfSv10TerZKtBEG6jhLMMaYkItpFH74qZ0ApYfKWLUjj4UZTq+0RZv3siu/hF35sPDT1QBEhDn36/Rv24R+7ZrQv21T2jdrbF2j6xBLMMaYOicyPIy+bZvQt20Txpza0RkROruAd776nqLGqSzZso+1O/NZvi2X5dtyeX3+ZgCaNo6kX9sm9G/XlL5tm9C7daL1VgshSzDGmDpPROjcPJ6hbSIZPrwPAAUlB1m+dR9Ltu5jyZZ9LN26l90FB5i5NpuZa7MPr9u6SQw9WyXQu3Uivdok0qtVonUgCBJLMMaYeikuKuKI6ziqyra9RSzZuo8fN+9lxfZcVu3IY/u+IrbvK+KrVTsPr9siIZperRPo1TrRSTytE2keH2Wn12qZJRhjzDFBRGib1Ji2SY35Td9WgPP8m43ZBazckcuKbXms3OEknay8YrLyipmxetfh9ZNiG9G9RTzdWyQ4/7aMp0vzeGIaWUeCmrIEY4w5ZoWHCV1S4+mSGs8F/Z15ZWVKxp5CVmx3ukY7XaRzySk8wLwNe5i3Yc/h9cME0prF0r2lk3i6tYjnuBYJtGkaUycfyFbXWIIxxjQoYWFCx5Q4OqbEMapfa8A5vZaZW8yarDxWZ+azNiufNVl5bMguZONuZ/psRdbhOmIbhdO5eRydmsfRuXkcnVPiyCks4+ChMiLCw0K1a3WOJRhjTIMnIrRqEkOrJjGc1j318PySg4dYv6vATTj5rM7MY01WPtn5JSzblsuybUc+vf2+eV/SITn2cPLp4iagDsmxDfKeHUswxhhTgaiIcHq2SqRnq8Qj5ucUHmD9rgLW7yrgp135rN9VQPrWPeQUl7F2Zz5rdx75JPgwgTZNG5OWHEuHZs6/ac1iSUuOpU3TGCKP0aMeSzDGGFNNSbGNOKGDM4RNuVmzZjFo8ClsyC7gp50FrM92EtCGXQVsztnPFnf61quu8DChbdMY2jeLpUNyLGnNGtM+OZYOzZzkU59PuVmCMcaYWhIbFUGfNk3o06bJEfNLDh5iy579zojSuwvZtKeQjN2FbN6znx25Re5I0/uZvS77iPUiwoSWTaJp27SxMyXF0Mb9t23TxiTHRdXpzgaWYIwxJsCiIsIP92bzVlx6iC05+9m0u5DNewrZtNtJQhl7CsnMLWZrThFbc4pwng7vXW8YrZs6yaZN0xinm7b7unXTGJrFNgrpvT2WYIwxJoSiI8PpmhpP1wqSz7a9RWzdu59te4vYlrP/8OutOfvZu7+UjdmFbMwu9Fl3o/AwWiRGE6PFTNu5lJaJ0bRsEkOrxGhaJsbQMjGaJo0jA5aELMEYY0wdFR3pdIfu3DzO5/L84lIn8bgJxzP5ZOYWk1tUypac/QCs3bvdZx0xkeFu4nGSTqvEaFITo0mNj6Z5QhSpCdE0i21Uo2tBlmCMMaaeio+O5LiWkRzXMsHn8sKSg2TmFvPFtwto3r4rmbnFZOYWsSO3mMx9RWTmFlNQcvDwvT4VCRNoFhdFakLU4cTTPD6a1IToSuOzBGOMMceo2KgIOjePo1dyOMMHtfVZJr+4lMzcYna4CSdzXxE780rYlV98+N/dBQfIzi8hO7+EleT5vX1LMMYY04DFR0cSHx3p8xpQudJDZewuKGFnXgk784qdZ/PkFbMzr5jHKqnbEowxxphKRYaHuZ0CYn6xrLIEU3/v4DHGGFOnWYIxxhgTEJZgjDHGBIQlGGOMMQFhCcYYY0xAWIIxxhgTEEFNMCKSJCJTRKRQRDaLyBWVlL1NRLJEJFdEXhWRKH/rEZGRIrJGRPaLyEwRaR/I/TLGGPNLwT6CeQ44AKQCVwITRKSndyERORO4GxgJpAEdgQf8qUdEkoGPgHuBJGAR8H5gdscYY0xFgpZgRCQWuAi4V1ULVHUOMB24ykfxa4BXVDVdVfcCDwGj/aznQiBdVT9Q1WJgPNBXRLoHbu+MMcZ4C+ad/F2BQ6q6zmPeMmCYj7I9gWle5VJFpBnQrop6errvAVDVQhHZ4M5f47kRERkLjHXflojIymrvVXAkA7tDHUQFLLaasdhqxmKrmUDGVuEliGAmmDgg12teLuBrABzvsuWv4/2oJw7IrmT5Yar6EvASgIgsUtWBle9CaFhsNWOx1YzFVjMW2y8F8xpMAeA9pnQCkO9H2fLX+X7UU53tGGOMCZBgJph1QISIdPGY1xdI91E23V3mWW6nqu7xo54j1nWv2XSqYDvGGGMCJGgJRlULcXp3PSgisSIyBBgFvOmj+BvA9SLSQ0SaAuOAiX7WMwXoJSIXiUg0cB+wXFXXeG/Ey0tHt4cBZbHVjMVWMxZbzVhsXkRVg7cxkSTgVeAMYA9wt6q+IyLtgFVAD1Xd4pa9HbgLiAEmAzeqakll9Xhs53TgWZyLTwuA0aqaEZSdNMYYAwQ5wRhjjGk4bKgYY4wxAWEJxhhjTGCoaoOecIaTmQIUApuBKwK8vVlAMU536gJgrceykTg3g+4HZgLtPZYJ8G+ca057gEdxT3G6y9Pcdfa7dZzuRyy34AylUwJM9FoWsFiAK9y2LgSmAkn+xubWrR7tV4AzqkNQYgOigFfcMvnAEuDsutBulcUW6nZzy7wFZAJ5OL1Bx9SFdqsstrrQbh5lu+D8drxVV9qtyt+Y6q5wrE3AuzhjlcUBp+DclNkzgNub5fnF8pif7G77EiAa51HX33ssvwFYC7QBWuN0irjRY/l84EmcThEXAfuAlCpiuRA4H5jAkT/iAYsFZ0SFfGCo2+bvAO9VI7byL3xEBfsU0NiAWJzhh9JwzgCc666TFup2qyK2kLabR7ko93V3IAsYEOp2qyK2kLebR11fAd/hJpi60G5V/t4F6oe0Pkw4X8gDQFePeW8C/wrgNmfhO8GMBeZ5xVYEdHffzwPGeiy/vvzDhDMMTwkQ77H8O88PUxUx/YMjf8QDFgvwCPCOx7JO7v9BvJ+xVfWFD1psHuWWu1/QOtNuPmKrU+0GdMM5Yri0rrWbV2x1ot2A3wKTcP6AKE8wdardfE0N/RpMReOj/WKE51r2TxHZLSJzRWS4O+8XY6gB5WOo/WK5V5w9gY2qml/B8uoKZCzedW/ATfLVjHGziGwTkdfcEbR9xh7o2EQk1V2e7mP9kLabV2zlQtpuIvK8iJSfkskEPvOxfkjarYLYyoWs3UQkAXgQ+ItXyHWi3SrT0BNMdcZHqy134Tx+oDXOzU8fi0gnP2LxNT5bnIiIH+tWVyBjOdpYdwODcO5xGuCu93YlsQcsNhGJdLf9ujo38taZdvMRW51oN1X9gzv/VJwbpktqUH8wY6sL7fYQzujyW73m14l2q0xDTzBBH7dMVReoar6qlqjq68Bc4Bw/YvE1PluBOsevtb0fgYzlqGJV5xENi1T1oKruxOkM8Cv3r7ygxSYiYTinUw+4Mfizfshiqyvt5sZySJ3HbLQBbqpB/UGLLdTtJiL9gNOBp3yEW2farSINPcFUZ3y0QFGc3h5VjaHma3w2z2UdRSS+guXVFchYvOvuiNP7yfM0ZXVoeVXBis39C/AVnAfeXaSqpRWsH/R2qyQ2b0FvNx8i+Ll96trnrTw2b8Fut+E414G2iEgWcAdwkYj86GP9utBuR6rOBZtjcQLew+lJFgsMIYC9yIAmwJk4PT4icJ7GWYhzUTHF3fZF7vJ/c2SPkBuB1Tin1lq5HwDPHiHfA4+7616Af73IItzy/8T5i7c8roDFgnNuNw/nNEQsTvdQX716KortRLe9woBmOD0AZwY5thfceuK85teFdqsotpC2G9Ac50J1HBCO8z0oxBlHMKTtVkVsoW63xkALj+lx4EO3zUL+eavyNy8QP6T1acK5D2aq+4HaQgDvg3E/EAtxDjP3uf/BZ3gsPx3nAmMRTm+zNI9lgtOPPcedfPVpn+Wuuxb/7oMZj/MXmec0PtCx4PSv3+K2+TR835fgMzbgcmCTu24mzsCoLYIVG865eOXIe5kKgCtD3W6VxVYH2i0FmI3zuc8DVgC/D8Zn/2hiC3W7VfC9eKsutJs/k41FZowxJiAa+jUYY4wxAWIJxhhjTEBYgjHGGBMQlmCMMcYEhCUYY4wxAWEJxhhjTEBYgjHmGCMio0WkINRxGGMJxpgAEZGJIqIe024R+UREulejjvEisjKQcRoTKJZgjAmsGUBLd/oVzsOdpoQ0ImOCxBKMMYFVoqpZ7vQjzqi43UUkBkBE/iUia0WkSEQyRORREYl2l40G7gd6ehwFjXaXJYjIBBHJFJFiEVktIpd5blhERorIShEpFJGZItIhmDtuTESoAzCmoXBHrr0MWKGqRe7sQuA6YDvQA2ewyhLgXpyBFXvhPPp4uFs+1x0x+XOgKXAtzgi33XAGLSwXBdzj1l0MvO7WfWZg9s6YX7IEY0xgneVxwT0W2Irz/B8AVPUhj7IZIvIIzpDs96pqkbvuQVXNKi8kImcAg3FG/V7tzt7otd0I4GZVXeuu8zjwmoiEqWpZLe6fMRWyU2TGBNa3QD93OhH4BvhKRNoCiMjFIjJHRLLcZPIU0K6KOvsDmR7JxZeS8uTi2gFE4jwywpigsARjTGDtV9X17vQDcD3OkwHHishJOM8j+hI4DydxjMNJBJWRKpYDHPR6Xz5sun3nTdDYKTJjgkuBMpwHSQ0BtnueJhOR9l7lD+A8BMvTj0BLETmuiqMYY0LKEowxgRUlIi3c101xnukeB3wMxAOtReRKYD7OBfjLvdbPANqLyPE4D3/KB/4HLAAmi8htOBf5OwOxqjo1sLtjjP/scNmYwDod50mImThJYRBwiarOUtWPgceA/wDLgTOA+7zWnwx8hpNUsoHL3Yv0ZwNzcR5luxp4GmgU8L0xphrsiZbGGGMCwo5gjDHGBIQlGGOMMQFhCcYYY0xAWIIxxhgTEJZgjDHGBIQlGGOMMQFhCcYYY0xAWIIxxhgTEP8PkjY6Srt64p8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(steps, lrs, \"-\", linewidth=2)\n",
    "plt.axis([0, n_steps - 1, 0, lr0 * 1.1])\n",
    "plt.xlabel(\"Batch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Exponential Scheduling (per batch)\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Piecewise Constant Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant_fn(epoch):\n",
    "    if epoch < 5:\n",
    "        return 0.01\n",
    "    elif epoch < 15:\n",
    "        return 0.005\n",
    "    else:\n",
    "        return 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "def piecewise_constant(boundaries, values):\n",
    "    boundaries = np.array([0] + boundaries)\n",
    "    values = np.array(values)\n",
    "    def piecewise_constant_fn(epoch):\n",
    "        return values[np.argmax(boundaries > epoch) - 1]\n",
    "    return piecewise_constant_fn\n",
    "\n",
    "piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.8429 - accuracy: 0.7634 - val_loss: 0.9667 - val_accuracy: 0.6432\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 5s 97us/sample - loss: 0.8500 - accuracy: 0.7677 - val_loss: 0.6907 - val_accuracy: 0.8070\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 5s 97us/sample - loss: 0.7974 - accuracy: 0.7831 - val_loss: 0.8228 - val_accuracy: 0.7608\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 5s 98us/sample - loss: 0.8931 - accuracy: 0.7344 - val_loss: 1.1275 - val_accuracy: 0.7434\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.9565 - accuracy: 0.7025 - val_loss: 0.8436 - val_accuracy: 0.7664\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 6s 104us/sample - loss: 0.6047 - accuracy: 0.7910 - val_loss: 0.7323 - val_accuracy: 0.8054\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 5s 99us/sample - loss: 0.5715 - accuracy: 0.8027 - val_loss: 0.6902 - val_accuracy: 0.7888\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.5593 - accuracy: 0.8068 - val_loss: 0.6892 - val_accuracy: 0.7762\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 6s 101us/sample - loss: 0.5603 - accuracy: 0.8135 - val_loss: 0.6659 - val_accuracy: 0.7780\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.5320 - accuracy: 0.8303 - val_loss: 0.6034 - val_accuracy: 0.8246\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.4994 - accuracy: 0.8472 - val_loss: 0.5903 - val_accuracy: 0.8458\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 6s 104us/sample - loss: 0.5150 - accuracy: 0.8429 - val_loss: 0.6375 - val_accuracy: 0.8274\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 6s 103us/sample - loss: 0.4800 - accuracy: 0.8591 - val_loss: 0.6542 - val_accuracy: 0.8454\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.4637 - accuracy: 0.8595 - val_loss: 0.6974 - val_accuracy: 0.8538\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.5155 - accuracy: 0.8531 - val_loss: 0.6640 - val_accuracy: 0.8426\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.3436 - accuracy: 0.8896 - val_loss: 0.5793 - val_accuracy: 0.8584\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 6s 102us/sample - loss: 0.3130 - accuracy: 0.8979 - val_loss: 0.5593 - val_accuracy: 0.8624\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 6s 110us/sample - loss: 0.2968 - accuracy: 0.9020 - val_loss: 0.5435 - val_accuracy: 0.8666\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 6s 109us/sample - loss: 0.2859 - accuracy: 0.9075 - val_loss: 0.5807 - val_accuracy: 0.8648\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 6s 109us/sample - loss: 0.2758 - accuracy: 0.9109 - val_loss: 0.5802 - val_accuracy: 0.8698\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 6s 108us/sample - loss: 0.2670 - accuracy: 0.9143 - val_loss: 0.5445 - val_accuracy: 0.8688\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.2580 - accuracy: 0.9173 - val_loss: 0.5952 - val_accuracy: 0.8708\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 6s 105us/sample - loss: 0.2524 - accuracy: 0.9193 - val_loss: 0.5608 - val_accuracy: 0.8700\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 6s 106us/sample - loss: 0.2449 - accuracy: 0.9216 - val_loss: 0.5656 - val_accuracy: 0.8746\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 6s 109us/sample - loss: 0.2409 - accuracy: 0.9228 - val_loss: 0.6174 - val_accuracy: 0.8714\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEeCAYAAAC30gOQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de5xdVX338c83VzKThJBJSCCaGa4TCC0gKtVoSQVMtU8LBWtbEKFV6YP1qaJSoQ+XAPWK5ak+UBQVA4pWkXBRFB4RRopXbgUMkIiSAAm5ktvkfvk9f6w94XByzsyezOxzMnO+79frvHJm77XXXnvN5PzO2mvttRQRmJmZ9bch9S6AmZkNTg4wZmZWCAcYMzMrhAOMmZkVwgHGzMwK4QBjZmaFcICxiiS1SQpJr693WaqR1CHpmnqXw/KRdI6kzoLy/o2k2b08ZqGkj1f72frOAaZBSZqTBZCQtE3S7yV9XlJzluQF4ADgv+tYzJ6cBlxU5AmUvF/SLyStl7RO0qOS/lnS2CLPXVaOwj788uYt6SBJ35T0oqQtkpZIukvSsUWUqw7eAPxHvQsxmAyrdwGsru4FzgKGA28Fvgo0A+dFxA5gaR3L1qOIeLkGp/kGcDrwKeDDwHJgOvCh7P2cGpSh7iQNB34M/A54N7AYmAKcDIyvY9H6TUSsqHcZBp2I8KsBX6QPxh+UbfsK8FL2vg0I4PUl+48E7gLWkz5cvw1MLsvjbOBJYAuwDJhTsm9f4Prs2PXAT8vyXwr8dcnPP8vSDct+Piwr05Ts5w7gmpL0pwFPAJuAl7P8J5Xs/3PgEWAz8BzwSWBEN3X07ux8p1XZPy77dwhwCanVtyW7/lNK0nXV5emkD+mNwFPAySVphgNfBJZkebwAfKbkOqP0lW1vyX4HL2bXPA/4u7IydpC+lX8KWJnV/eeBId3lXeFaj8n2H9rD39VY4Drgpayen+76nQLnAJ3AicBvgA3A/cBBZXl0+3sC9gfuyK55EfD3WX6zS9IE8K6yfBcCH+/FzwGcC9ySlfX3wHvK8jweeDQr62PAO7PjZtb7//je8PItMiu1ifRBtxtJBwAPkP4jvxE4CRgN3ClpSJbmH4AvA18H/pD0n21etk+k4DQF+B/AsVl+92V5QwoIf5KlbwJeT/qw7eoHmgk8GxGLK5RvMvCfwI3AEcAfk1ofXftnATcD15BaIH8PvIv0wVvNmcCCiJhbaWdErMnefhi4APgE8AfAbcBcSceUHfJJUhA5GngI+E9Jo7N9/wT8JfA3pED618D8bN9ppCByBem2ZVd97UP6cPsf2TV9AfiypBMrXMd24M2kltdHsvy7y7vcCmAncLqkinc+st/xj4ATgL8jfSH5KLC1JNlI0m3NvwfeBIwDvlSSR57f0xzgUNLf4KnAe0lBvAiXkoLZ0cB3gBsktWZlHQ38AHgGOA74Z+CqgsoxMNU7wvlVnxdlLRhS0FgJfCf7uY2SFgzpA+gnZXnsl6V5Y/bzi2Tfuiuc722kb6+jyrb/N/DP2fvzgPnZ+5NJ335vBC7Ktt0MfKXk2A6yFgzwuqwsrVXO/wBwSdm2U7MyqcoxTwF35KjLxcClZds6gG+W1eU/lOyfkm17S/bzF4GfdFOWhZR8u+6mLP8JfLWsHL8oS/PjsjR58/5H0jf5TtKXgSuB6SX7TyYFoSOqHH9Ods3tJdvOJAWgrhZVt78n4PAsjxkl+1uBHRTTgvl0yc/DSK3P92Q//wOppTyqJM0ZuAWz6+UWTGP7U0mdkjYDvyD95/5fVdIeB/xxlr4zGw30QrbvEEn7kz40f9LN8U3AirI8jgIOydJ0AIdLOpDUWrk/2zYz239C9nMlj5P6lH4j6VZJ50maWHb+/1127m+R+pwmV8lTVba/kiB19B9Iup1X6kHSN/hST5S8X5L9u3/27xzSbagFkq6V9GddLcNuzj1U0v+W9ISkVdk1nQZM7ea8Xefen16KiGtJdXUG6fpOAf5b0llZkmNJt1if7iabLRExv+TnJaRW87js555+T0eQgtivS8q1iFfqs7/tqruI2E5qyXXV3TTgNxGxqST9rwoqx4DkTv7G9gDpHvM2YElEbOsm7RDSLa5Ko42WkYJHd4Zk6d5aYd86gIh4WtIyUkCZCfw76VbS/5V0JCmAdVTKPCJ2SHo78EfA24H3AZ+WdEJEPJ6d/3LS/fRy1Tp3F5A+0PKoNC15+bZd9RsRke4opS95EfGopDbgT0mtvRuBxyWdHBE7q5zz48DHSLfoniR9y/8UuweP8t9rsIcjSCNiPXAn6dboxcA9pJbMN8gRkEm36srLQkl5evo95TlHV77laSve/u1Bd3UnKv/eLeMA09g2RsSzOdM+Sur0XlQlEK2XtJjUgfvjKsdPAnZGxO+7Oc9PgT8j9bv8NCKWS1pJur9dsf+lS6R7FL8AfiHpClL/z1+TWjePAtN6cb2Qvjn/p6TTokI/jKRxEbFG0hLgLcB9JbvfQrrFllv24X0LcIukOcAvSX0NC0i3kYaWHfIW4PsR8Y2sPF23kNbQO5XyzlPekPQM6fYkpDo+QNIRPbRiutPt70nS06QP+DcAP8+2TSW1IkutoKQ/SdIkqvcv7amngfdKGlXSinljP59jQPMtMsvrWtIosO9IOl7SwZJOknS9pDFZmk8CH5F0vqTDJR0j6WPZvntJt5HukPSO7JmKN0m6XFJpq6aDFBR+GxHLs20/Bd5D9dtjSPojSRdLekP2gfMXwGt55UP+CuAMSVdIOkrSNEnvkvS5bq75u6Q+jZslXZLl3SrpTyXdReobgNSx+3FJf5td9xWkltq/dVehZeX/aHb8EZIOJd2GWkfq14LUP/BWSVMkTci2LQBOlPQWSdNIHeMH5T1niUp5l5fvGEl3ZHV2pKRDJb2P1Al/W5bsJ6RbRLdKmpX9jk+WdGqlPKvo9veU3V67mzSY4U3ZQIo5pAEqpe4D/lHS65We05lDGunVn24m9f18JauTk4B/yfa5ZYMDjOUUEUuAGaT733eTWgfXkkZ5bcnSXEfqCP4AabTZ3aSRQF2ti3eS/uN/hTRC6rtAO6++f34/6dt0Rw/byq3NyvcD4LekD/crI+Kb2fnvIbWM/oR0//7XwIXA891cc5A+6D9MGql1P+lW1KdJQe/WLOkXSUHmc9l1/yVwekT05iHV9aSRaL8mfYs/BnhHRGzM9l9KCpi/45Vbev+apf8R6XbnBtKHXm9Vyrvci6RhupeSWlb/Tbo993myfrvsVt47SF8kvkn6hv8FYETeguT8PZ1DGr58H/B9UktzYVlWH8vK2wF8j/SM13L6UUR0koZUTycNUb4KmJ3t7u9gNiApG/lgZmZ9JOkUUotu/4hYWe/y1Jv7YMzM9pCks0ktpRdIIyL/ndQv1vDBBRxgzMz6YhJp1NsBpJko7iI9cGv4FpmZmRXEnfxmZlYI3yLLjBs3Lg499NB6F2Ovs2HDBpqbm3tO2GBcL5W5XnY32OvkkUceWRkREyvtc4DJTJo0iYcffrjexdjrdHR0MHPmzHoXY6/jeqnM9bK7wV4nkhZV2+dbZGZmVggHGDMzK4QDjJmZFcIBxszMCuEAY2ZmhXCAMTOzQjjAmJlZIRxgzMysEA4wZmZWCAcYMzMrhAOMmZkVwgHGzMwK4QBjZmaFcIAxM7NCOMCYmVkhahpgJI2XdJukDZIWSTqjm7TnS1oqaa2kGySNLNn3IUkPS9oiaU6FY0+U9IykjZLul9TaU9kWrtvJjM/cx+2PLc51Lbc/tpgZn7mPgy68a1AeZ2bWV7VuwVwLbAUmAWcC10maXp5I0izgQuBEoA04GLi8JMkS4F+BGyocOwGYC1wCjAceBr6Tp3CL12ziorlP9vghfPtji7lo7pMsXrOJGITHmZn1h5qtaCmpGTgdOCoiOoEHJd0JnEUKJqXOBr4WEfOyY68Ebu5KFxFzs+2vB15TduxpwLyIuCVLMxtYKWlaRDzTUzk3bdvBv9z2JA8+u7Jqmh8++RKbtu0YsMdddc98Tj12StXjzMz6Qy2XTD4c2BERC0q2PQ6cUCHtdOCOsnSTJLVExKoezjM9Sw9ARGyQ9Lts+6sCjKRzgXMBRkw+dNf2jVt3cP+86t/yN26NKtsHxnGL12yio6Oj6nGlOjs7c6dtJK6Xylwvu2vkOqllgBkNrC3bthYYkyNt1/sxQE8BZjSwIs95IuJ64HqAkQcctuvTeMq4UfzswrdVPcGMz9zH4jWbdts+kI7Lu0b4YF9PfE+5Xipzveyukeukln0wncDYsm1jgfU50na9r5S2L+fZzajhQ7lgVnu3aS6Y1c6o4UMH7XFmZv2hlgFmATBM0mEl244G5lVIOy/bV5puWY7bY7sdm/X9HFLlPK8yZdwoPn3aH/TYP3HqsVP49Gl/wJRxo9AAOO6AffcBYOw+w3IdZ2bWH2p2iyzrC5kLXCHp/cAxwCnAmyskvwmYI+lm4CXgYmBO105Jw0hlHwoMlbQPsD0itgO3AVdJOh24C7gUeKKnDv62sUO6vd1U7tRjp+zRB3W9jjv+U/fy1sMmOriYWc3UepjyB4FRwHLg28B5ETFP0lRJnZKmAkTE3cDngPuBRdnrspJ8LgY2kUaVvSd7f3F27ArSaLVPAquB44G/Kf7S9m6tLc0sWrWh3sUwswZSy05+IuJl4NQK258ndc6XbrsauLpKPrOB2d2c515gWh+KOugc1NLMT55ZXu9imFkD8VQxDaJ1QhMrO7fQuWV7vYtiZg3CAaZBtLU0A/g2mZnVjANMg2htaQJg0aqNdS6JmTUKB5gG0Zq1YBa6BWNmNeIA0yBGjxzGhNEjWbTSLRgzqw0HmAbS1tLkFoyZ1YwDTANpbWl2gDGzmnGAaSAHTWhi2botbNzqocpmVjwHmAbS1dH//MvuhzGz4jnANJCuZ2EWuqPfzGrAAaaBTN31LIz7YcyseA4wDWTfUcMZ3zyChX7Y0sxqwAGmwbS2NLkFY2Y14QDTYNpamj1djJnVhANMg2lraWbJ2k1s3raj3kUxs0HOAabBtE1oIgJe8FBlMyuYA0yDeWXSSwcYMyuWA0yDafNQZTOrEQeYBjOuaQT7jhruOcnMrHAOMA2oraXJI8nMrHAOMA3IsyqbWS04wDSgtpYmFq/exNbtO+tdFDMbxBxgGlDbhGZ2Bryw2rfJzKw4DjANqGuoskeSmVmRHGAaUNdQZU/bb2ZFcoBpQOObRzBm5DC3YMysUA4wDUgSrROa/DS/mRXKAaZBtbY0uwVjZoVygGlQbS1NvLh6E9t2eKiymRXDAaZBtbU0s31nsHj1pnoXxcwGqZoGGEnjJd0maYOkRZLO6Cbt+ZKWSlor6QZJI/PmI+ndkp6WtF7SU5JOLfK6BqK2CV2zKvs2mZkVo9YtmGuBrcAk4EzgOknTyxNJmgVcCJwItAEHA5fnyUfSFOCbwEeBscAFwLck7V/MJQ1MrbtmVXZHv5kVo2YBRlIzcDpwSUR0RsSDwJ3AWRWSnw18LSLmRcRq4ErgnJz5vAZYExE/iuQuYANwSIGXN+BMHD2SphFD3YIxs8IMq+G5Dgd2RMSCkm2PAydUSDsduKMs3SRJLcDUHvJ5GHha0l8AdwF/DmwBnig/iaRzgXMBJk6cSEdHxx5c1sDVMjJ4dMELdHSsqJqms7Oz4eolD9dLZa6X3TVyndQywIwG1pZtWwuMyZG26/2YnvKJiB2SbgK+BexDupX2VxGx21f1iLgeuB6gvb09Zs6c2YvLGfiOevER5i9bT3fX3dHR0e3+RuV6qcz1srtGrpPct8gkTZL0cUnXSZqQbZsh6aCcWXSS+kRKjQXW50jb9X59T/lIOgn4HDATGEFq2XxV0jE5y9kwWluaeeHljezYGfUuipkNQrkCjKTjgPmkDvX38coH/MnAJ3OeawEwTNJhJduOBuZVSDsv21eabllErMqRzzHAAxHxcETsjIiHgF8BJ+UsZ8Noa2li245gyRoPVTaz/pe3BfN54AsRcSypP6PLPcCMPBlkt6jmAldIapY0AzgF+EaF5DcB75N0pKT9gIuBOTnzeQh4a1eLRdKxwFup0AfT6DxU2cyKlDfAHAfcWGH7S6Shwnl9EBgFLAe+DZwXEfMkTZXUKWkqQETcTbrNdT+wKHtd1lM+2bE/BWYD35O0HrgV+FRE/L9elLMhtLV0BRgPVTaz/pe3k38TsF+F7dNIH/K5RMTLwG4PPUbE86TO+9JtVwNX9yafkv3XANfkLVej2n/MSPYZPoRFK92CMbP+l7cFcwdwWcnT9CGpDfgsqYVgA9CQIaJ1fLNbMGZWiLwB5uPAeGAF0AQ8CDwLrCH1j9gA1drS5FmVzawQuW6RRcQ64C2S3ga8jhSYHo2Ie4ssnBWvbUIzHQtWsHNnMGSI6l0cMxtEcgUYSe8FvhMR9wH3lWwfAfxNRNxUUPmsYK0tTWzdvpOl6zZz4LhR9S6OmQ0ieW+RfR3Yt8L2Mdk+G6AO6hpJ5o5+M+tneQOMgEqPe09l92lbbABpneChymZWjG5vkUl6khRYAvippO0lu4cCrcAPiyueFe2AsfswYtgQd/SbWb/rqQ/me9m/R5FmJu4s2bcVWIiHKQ9oQ4aIqeOb/DS/mfW7bgNMRFwOIGkhqZN/cy0KZbXV1tLkhcfMrN/l6oOJiBsdXAav1pZmFq7aQIRnVTaz/pN3NuURki6XtEDSZkk7Sl9FF9KK1dbSxOZtO1m+fkvPic3Mcso7iuxK0jLG/wbsJK1zfy2wijTxpA1grdlQ5ec8VNnM+lHeAPNu4H9GxJeBHcAdEfFPpBmOTy6qcFYbB2VDlT2SzMz6U94AMwl4KnvfCYzL3t8NvL2/C2W1dcC++zB8qPwsjJn1q7wB5nngwOz9s8Cs7P2bSFP52wA2bOgQXrufJ700s/6VN8DcBpyYvf8CcLmk50irTH61gHJZjbW2NLFwpVswZtZ/8s6mfFHJ++9JeoG0VPKCiPhBUYWz2mltaebXz71MRCB5VmUz67u8K1q+SkT8CvgVgKTmiPC9lQGuraWJDVt3sLJzKxPHjOz5ADOzHuS9RbYbSftIugB4rh/LY3XS6pFkZtbPug0w2QOWn5T0kKSfSzo12/5e4PfAR4D/U4NyWsEO8rMwZtbPerpFNhv4R+DHpD6XWyR9hdThfxHwrYjYVmgJrSam7DeKoUPkOcnMrN/0FGDeDZwTEbdJOhp4DNgPmB4R27s/1AaS4UOH8Jr9RnlWZTPrNz31wbwWeAggIh4nTdH/WQeXwam1pdktGDPrNz0FmOFA6QyI2/AKloNWW0uTZ1U2s36TZ5jypyV1fa0dAcyW9Kogk81LZgNca0sz6zdvZ/XGbYxvHlHv4pjZANdTgHkAOKTk558DU8vS+OvuINHW0gTAwlUbHGDMrM96WtFyZo3KYXuBrmn7F67cwOum7lfn0pjZQLfHD1ra4PPa8aMYIjyrspn1CwcY22XksKEcOG6Un+Y3s35R0wAjabyk2yRtkLRI0hndpD1f0lJJayXdIGlk3nwkNUn6D0krs+MfKPK6BpO2lma3YMysX9S6BXMt6VmaScCZwHWSppcnkjQLuJA0Y0AbcDBweS/yuR4YDxyR/Xt+f1/IYNXa4nVhzKx/1CzASGoGTgcuiYjOiHgQuBM4q0Lys4GvRcS8iFgNXAmckycfSe3AXwDnRsSKiNgREY8UfHmDRltLM2s2bmPNxq31LoqZDXC5puuXVD40uUsAmyNiRY5sDgd2RMSCkm2PAydUSDsduKMs3SRJLaRh0t3lczywiLQo2lnAS8DsiLi1/CSSzgXOBZg4cSIdHR05LmNwW78sTdIw98f/xcH7DqWzs9P1UoHrpTLXy+4auU7yrgezkG6ed5G0Dvg68M/dTCMzmt1nAVgLjMmRtuv9mBz5vAY4CriVtMzzm4C7JD0VEU+XHhQR15Nup9He3h4zZ86sUvTGceCy9XzxsQdoaZ3GzGOm0NHRgetld66Xylwvu2vkOskbYP4W+BzwJbKFxkgthXNJMy6PAy4G1gOXVcmjExhbtm1sdkxPabver8+RzybSlDb/mgW7n0q6H3g78DTWranjm5Dw8slm1md5A8x5wPkRMbdk232S5gMfjogTJC0ndcRXCzALgGGSDouI32bbjgbmVUg7L9v33ZJ0yyJilaTNPeTzRM5rsgr2GT6UA8bu445+M+uzvJ38xwNPVtj+G+AN2ftfkG5PVZQtqzwXuEJSs6QZwCnANyokvwl4n6QjJe1Hah3NyZnPA8DzwEWShmX7ZwL35LzWhtfa0uxp+82sz/IGmEVkneFlPkD6MAeYCLzcQz4fBEYBy4FvA+dFxDxJUyV1dg0miIi7Sbfk7s/OvYhXt4wq5pMdu40UcN5J6pv5CvDeiHgm57U2vLYJTZ6238z6LO8tso8Bt0p6J2l9mCC1XA4hDRkm+/m7lQ9PIuJl4NQK258ndd6XbrsauLo3+ZTsn0fq3Lc90NrSzKoNW1m32YuVmtmeyxVgIuIuSYeRWg7tgEjPnnwpCw5ExH8UVkqrqa5ZlZ93K8bM+iBvC4aIeAG4qMCy2F5i16zKqza8ullpZtYLuQOMpCbgGGB/yvpuykaX2QDX2rUuzMoNHOXpUM1sD+V9kv8kUmd6S4XdAQztz0JZfTWNGMaksSNZuGojR02sd2nMbKDK+/30C8BdwGsiYkjZy8FlEGptafazMGbWJ3kDTBtwZUQsKbAsthdpa2nytP1m1id5A8zPSKPHrEG0tjSzYv0WNm+vOgWdmVm38nbyfwn4vKQDSU/0v+oBiYh4tL8LZvXVlo0kW75xZ51LYmYDVd4A873s3+sr7HMn/yDUNZJs2Ua3YMxsz+QNMAcVWgrb67RNcAvGzPom75P8i4ouiO1d7n1qGUMEtyzYxs8/cx8XzGrn1GOn9Hjc7Y8t5qp75rNkzSYOHDdq0B63eM0mpvzS9WLWnaoBRtJpwPcjYlv2vio/aDm43P7YYi6a+yQ7s7tji9ds4qK5aTLt7j5suo7btG2Hj2ug48yqUUTle+ySdgKTI2J59r6aGAzPwrS3t8f8+fPrXYy9wozP3MfiNZt22z58qDjywH2rHvfUkrVs27H735OPG9jHTRk3ip9d+Laqx5Vq5NUbqxnsdSLpkYh4faV9VVswETGk0nsb/JZUCC4A23YE40YNr3pcpQ8nHzfwj6v292DWk9xzkVnjOHDcqIotmCnjRnHj37+x6nHVWj4+bmAfd+C4UVWPMetO7paJpNdKOkPSRyR9tPRVZAGt9i6Y1c6o4a++6zlq+FAumNX9s7Y+rjGPM6sm72SXZwI3ANuBFaRnX7oEVRYGs4Gpq0N312ipnKOJSo/rzSikgXjcYK6X2XfOY82mbUwaO5KL3nGEO/htj1Xt5H9VIul3wHeASyJiR+GlqgN38lc22Dso99RgrpfHnl/NX/7Hz/nyWccxa/rkXh07mOtlTw32Oumukz/vLbJJwFcHa3Axs1ccPmkMAPOXrq9zSWygyxtgfggcX2RBzGzv0DxyGFPHNznAWJ/lHUX2Y+CzkqZTebJLP2hpNohMmzyGZ5auq3cxbIDLG2C+nP37LxX2ebJLs0Fm2uQx3Pv0MjZv28E+w/3f2/ZMrltkFVax9IqWZoNY++Sx7Ax4dnlnvYtiA1iPAUbScEm/kuTB8GYNon1y6uh/xv0w1gc9BpiI2Eaart8Lg5g1iLaWJkYMG8J898NYH+QdRXYj8IEiC2Jme49hQ4dw2P6j3YKxPsnbyd8MnCnpZOARYEPpzoj4p/4umJnVV/vkMTz425X1LoYNYHkDzBHAo9n7g8v2+daZ2SB0xOSxzH10MS9v2Mr45hH1Lo4NQHlXtPyTogtiZnuXVzr61/HmQybUuTQ2EHmdFzOraNpkTxljfdOb6fr/RNL1ku6WdF/pqxd5jJd0m6QNkhZJOqObtOdLWippraQbJI3sbT6SLpMUkk7KW0YzSyaOGcl+TcMdYGyP5Qowks4BfgSMAWaSpuzfD3gd8FQvznctsJU0eeaZwHXZ9DPl55sFXAicCLSR+n0u700+kg4B3gW81IvymVlGEu2Tx3gkme2xvC2YjwMfioi/Jc1DdlFEHAt8E8j1qK+kZuB00pT/nRHxIHAncFaF5GcDX4uIeRGxGrgSOKeX+VwDfIIUiMxsD0ybPJYFy9azc6fH8ljv5R1FdjBwb/Z+CzA6e38N0EFqbfTkcGBHRCwo2fY4cEKFtNOBO8rSTZLUAkztKR9JfwVsjYgfSqpaIEnnAucCTJw4kY6OjhyX0Vg6OztdLxU0Sr1o7TY2bt3B9+6+n/2bev4+2ij10huNXCd5A8wq0u0xgMXAUcATQAuQd8Hu0cDasm1rS/LtLm3X+zE95SNpNPAp4O09FSgirgeuh7Tg2GBeFGhPDfbFkvZUo9TLvs+v5uvzfs6+rUcyM8fiY41SL73RyHWS9xbZf/HKB/Z3gS9K+jrwbdJU/nl0AmPLto0FKt3gLU/b9X59jnwuB74REc/lLJeZVeHFx6wv8gaYD5GCCcCngatIrZfvAu/PmccCYJikw0q2HQ3Mq5B2XravNN2yiFiVI58TgX/KRqAtBV4LfFfSJ3KW08wyXnzM+iLvg5Yvl7zfCXy2tyeKiA2S5gJXSHo/cAxwCvDmCslvAuZIupk0CuxiYE7OfE4Ehpfk9RDwUdIoODPrpXYvPmZ7qDfPwUyS9HFJ10makG2bIemgXpzvg6Q+m+WkFtF5ETFP0lRJnZKmAkTE3cDngPuBRdnrsp7yyY5dFRFLu17ADmB1RHhhC7M9MG3yGBau2sjmbTvqXRQbYHK1YCQdB/wEeI40wusqYCVwMml0WNUHJktlLaFTK2x/nldGpnVtuxq4ujf5VEnbliedmVU2bfJYduwMnl3eyVFT9q13cWwAyduC+TzwhezZly0l2+8BZvR7qcxsr9HuKWNsD+UNMMeR1oQp9xLpaXozG6S6Fh9zP4z1Vt4As4k0NUy5aaR+EDMbpLz4mO2pvAHmDuCykgknQ1IbaTTZrQWUy8z2Iu2Tx/gWmfVab+YiG0+a5LIJeBB4lvQE/cXFFM3M9hbTJo9h+fotrN7gqf0sv7zPwZfLpUwAAAykSURBVKwD3iLpbaQZlIcAj0bEvd0faWaDwbTJafKMZ5au502HtNS5NDZQ5J2LDICIuA/Ytf6LpFbgqoh4d38XzMz2Hq8sPrbOAcZy6+uKluNIU+eb2SDWtfiYO/qtN7xkspn1yIuP2Z5wgDGzXLz4mPWWA4yZ5dI+eQwbt+7gxdWb6l0UGyC67eSXdGcPx5evy2Jmg1RXR/8zS9cxtaWpzqWxgaCnUWSrcuz3wl5mDaB08bG351jd0qzbABMRf1ergpjZ3q1r8TF39Fte7oMxs9y8+Jj1hgOMmeXmxcesNxxgzCy39sljdi0+ZtYTBxgzy61rTjLPrGx5OMCYWW5di4/NX+YAYz1zgDGz3LoWH3v6JXf0W88cYMysV7z4mOXlAGNmveLFxywvBxgz65X2ksXHzLrjAGNmvXJEyeJjZt1xgDGzXulafMwjyawnDjBm1itdi489/ZIDjHXPAcbMes2Lj1keDjBm1mtefMzycIAxs15rL1l8zKyamgYYSeMl3SZpg6RFks7oJu35kpZKWivpBkkj8+Qj6Y8k/VjSy5JWSLpF0gFFX5tZI2kvWXzMrJpat2CuBbYCk4AzgeskTS9PJGkWcCFwItAGHAxcnjOf/YDrs+NagfXA1/v/Uswa167FxzySzLpRswAjqRk4HbgkIjoj4kHgTuCsCsnPBr4WEfMiYjVwJXBOnnwi4kcRcUtErIuIjcA1wIyCL8+s4bRPHsMznpPMutHtksn97HBgR0QsKNn2OHBChbTTgTvK0k2S1AJM7UU+AH8MzKu0Q9K5wLkAEydOpKOjI8dlNJbOzk7XSwWuF9hny1aeW7mN//eT+xkxVIDrpZJGrpNaBpjRwNqybWuBMTnSdr0f05t8JP0hcClwSqUCRcT1pNtptLe3x8yZM7u9gEbU0dGB62V3rhfoHL+E7//uMQ6c9jqOmrIv4HqppJHrpJZ9MJ3A2LJtY0l9JD2l7Xq/Pm8+kg4FfgR8OCL+aw/LbGZVTJvsjn7rXi0DzAJgmKTDSrYdTeXbV/OyfaXplkXEqjz5SGoF7gWujIhv9FP5zaxEW0uzFx+zbtUswETEBmAucIWkZkkzSLeuKgWAm4D3STpS0n7AxcCcPPlImgLcB1wbEV8q+LLMGlbX4mOeVdmqqfUw5Q8Co4DlwLeB8yJinqSpkjolTQWIiLuBzwH3A4uy12U95ZPtez9pWPNlWZ6dkjprcG1mDccjyaw7tezkJyJeBk6tsP15Uud96bargat7k0+273Je/cyMmRVk2uQxzH10Mas3bGW/5hH1Lo7tZTxVjJntMS8+Zt1xgDGzPTbNi49ZNxxgzGyP7T9mJOO8+JhV4QBjZntMEtMmj/EtMqvIAcbM+mTa5LEsWOrFx2x3DjBm1iftk8ewwYuPWQUOMGbWJ158zKpxgDGzPjnci49ZFQ4wZtYno0cO47XjR3nxMduNA4yZ9dm0yWPdgrHd1HSqGDMbnETw7PJOzrkbpvzyPi6Y1c6px07p8bjbH1vMVffMZ8maTRw4btSgOq7rmMVrNg3qOhkx+dDjqqVxgDGzPrn9scXcP3/Frp8Xr9nERXOfBOj2g+r2xxZz0dwn2bRtx6A7biCUsT+Pq0YRHrsOaUXL+fPn17sYe51GXo2vO66XV8z4zH0sXrP7EOWRw4Zw/MEtVY/71e9XsWX7zkF53EAoY38d99KNH2HLS79VpXRuwZhZnyypEFwAtmzfybpN26oeV+mDbbAcNxDKWMRx5RxgzKxPDhw3qmILZsq4Udz+jzOqHlet5TMYjhsIZSziuHIeRWZmfXLBrHZGDR/6qm2jhg/lglntDXvcQChjfx9XiVswZtYnXZ3Bu0ZM5RyJVHpcb0YwDYTjGqlOXuomnTv5M+7kr8yd2ZW5XipzvexusNeJpEci4vWV9vkWmZmZFcIBxszMCuEAY2ZmhXCAMTOzQjjAmJlZIRxgzMysEA4wZmZWCAcYMzMrhAOMmZkVwgHGzMwK4QBjZmaFcIAxM7NC1DTASBov6TZJGyQtknRGN2nPl7RU0lpJN0gamTcfSSdKekbSRkn3S2ot8rrMzGx3tW7BXAtsBSYBZwLXSZpenkjSLOBC4ESgDTgYuDxPPpImAHOBS4DxwMPAd4q5HDMzq6ZmAUZSM3A6cElEdEbEg8CdwFkVkp8NfC0i5kXEauBK4Jyc+ZwGzIuIWyJiMzAbOFrStOKuzszMytVywbHDgR0RsaBk2+PACRXSTgfuKEs3SVILMLWHfKZnPwMQERsk/S7b/kzpSSSdC5yb/bhF0m96fVWD3wRgZb0LsRdyvVTmetndYK+Tql0QtQwwo4G1ZdvWAmNypO16PyZHPqOBFXnOExHXA9cDSHq42qI5jcz1UpnrpTLXy+4auU5q2QfTCYwt2zYWWJ8jbdf79Tny6c15zMysILUMMAuAYZIOK9l2NDCvQtp52b7SdMsiYlWOfF51bNZnc0iV85iZWUFqFmAiYgNpdNcVkpolzQBOAb5RIflNwPskHSlpP+BiYE7OfG4DjpJ0uqR9gEuBJyLimfKTlLm+b1c4aLleKnO9VOZ62V3D1okionYnk8YDNwAnA6uACyPiW5KmAk8BR0bE81najwKfAEYBtwL/MyK2dJdPyXlOAq4hdT79CjgnIhbW5CLNzAyocYAxM7PG4alizMysEA4wZmZWiIYPML2ZH62RSOqQtFlSZ/aaX+8y1YOkD0l6WNIWSXPK9jXknHfV6kRSm6Qo+ZvplHRJHYtaU5JGSvpa9jmyXtJjkt5Rsr/h/l4aPsCQc360BvWhiBidvdrrXZg6WQL8K2lQyS4NPuddxTopMa7k7+bKGpar3oYBL5BmFdmX9Lfx3SzwNuTfSy2f5N/rlMxrdlREdAIPSuqa1+zCuhbO9goRMRdA0uuB15Ts2jXnXbZ/NrBS0rQcQ+IHtG7qpKFlj1DMLtn0A0nPAccBLTTg30ujt2CqzY/mFkzyaUkrJf1M0sx6F2Yvs9ucd0DXnHeNbpGkFyV9Pfvm3pAkTSJ9xsyjQf9eGj3A9GZ+tEbzCdIyCVNID4p9X9Ih9S3SXsV/O7tbCbyB9PzZcaS6uLmuJaoTScNJ135j1kJpyL+XRg8wnresioj4VUSsj4gtEXEj8DPgnfUu117EfztlsuUzHo6I7RGxDPgQ8HZJ5fU0qEkaQppZZCupDqBB/14aPcD0Zn60RheA6l2IvYjnvOtZ11PcDfN3I0nA10iDhk6PiG3Zrob8e2noANPL+dEahqRxkmZJ2kfSMElnAn8M3FPvstVadv37AEOBoV11wp7PeTfgVasTScdLapc0JFu76YtAR0SU3xoazK4DjgD+PCI2lWxvzL+XiGjoF2nI4O3ABuB54Ix6l6neL2Ai8BCp+b4G+CVwcr3LVae6mE36Jl76mp3tO4m0iN0moANoq3d561knwN8Cz2X/l14iTVo7ud7lrWG9tGZ1sZl0S6zrdWaj/r14LjIzMytEQ98iMzOz4jjAmJlZIRxgzMysEA4wZmZWCAcYMzMrhAOMmZkVwgHGbJDK1mZ5V73LYY3LAcasAJLmZB/w5a9f1rtsZrXS0OvBmBXsXtLaQqW21qMgZvXgFoxZcbZExNKy18uw6/bVhyTdlS2hu0jSe0oPlvQHku6VtEnSy1mraN+yNGdLejJbvnhZ+bLOwHhJt2RLgv++/BxmRXKAMaufy4E7gWNIa+7clK0SiaQm4G7SXFZvBP4SeDMlyxRL+gfgy8DXgT8kLadQPjvvpcAdpJl8vwPc0AhrwdvewXORmRUga0m8hzTxYalrI+ITkgL4akR8oOSYe4GlEfEeSR8APg+8JiLWZ/tnAvcDh0XEs5JeBL4ZERWX987O8ZmIuCj7eRiwDjg3Ir7Zj5drVpH7YMyK8wBwbtm2NSXvf1G27xfAn2XvjyBN5166INXPgZ3AkZLWkVYb/UkPZXii601EbJe0Atg/X/HN+sYBxqw4GyPi2T08VryyYFe53iz+tq3s58C3xq1G/IdmVj9/VOHnp7P3TwFHSypds/3NpP+zT0dakngxcGLhpTTbQ27BmBVnpKTJZdt2RMSK7P1pkh4iLT71LlKwOD7bdzNpEMBNki4F9iN16M8taRV9Evg/kpYBdwFNwIkR8W9FXZBZbzjAmBXnJNLKjqUWA6/J3s8GTictLbwC+LuIeAggIjZKmgX8O/Br0mCBO4APd2UUEddJ2gp8DPgs8DLww6Iuxqy3PIrMrA6yEV5/FRHfq3dZzIriPhgzMyuEA4yZmRXCt8jMzKwQbsGYmVkhHGDMzKwQDjBmZlYIBxgzMyuEA4yZmRXi/wMNSxyGNXnrWQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.epoch], \"o-\")\n",
    "plt.axis([0, n_epochs - 1, 0, 0.011])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.title(\"Piecewise Constant Scheduling\", fontsize=14)\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 2s 44us/sample - loss: 0.5950 - accuracy: 0.8063 - val_loss: 0.5177 - val_accuracy: 0.8426\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.5091 - accuracy: 0.8373 - val_loss: 0.4850 - val_accuracy: 0.8534\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.5029 - accuracy: 0.8437 - val_loss: 0.4905 - val_accuracy: 0.8400\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5372 - accuracy: 0.8436 - val_loss: 0.6500 - val_accuracy: 0.8378\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5189 - accuracy: 0.8497 - val_loss: 0.4873 - val_accuracy: 0.8618\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.5190 - accuracy: 0.8515 - val_loss: 0.8216 - val_accuracy: 0.8420\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.5411 - accuracy: 0.8513 - val_loss: 0.7188 - val_accuracy: 0.8286\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3137 - accuracy: 0.8926 - val_loss: 0.4186 - val_accuracy: 0.8738\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2553 - accuracy: 0.9067 - val_loss: 0.3912 - val_accuracy: 0.8888\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2292 - accuracy: 0.9150 - val_loss: 0.3965 - val_accuracy: 0.8840\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2207 - accuracy: 0.9185 - val_loss: 0.4154 - val_accuracy: 0.8744\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2049 - accuracy: 0.9242 - val_loss: 0.4514 - val_accuracy: 0.8810\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.1987 - accuracy: 0.9266 - val_loss: 0.4412 - val_accuracy: 0.8794\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.1880 - accuracy: 0.9310 - val_loss: 0.4852 - val_accuracy: 0.8892\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.1321 - accuracy: 0.9477 - val_loss: 0.4251 - val_accuracy: 0.8934\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.1148 - accuracy: 0.9540 - val_loss: 0.4347 - val_accuracy: 0.8890\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.1063 - accuracy: 0.9576 - val_loss: 0.4490 - val_accuracy: 0.8906\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.1003 - accuracy: 0.9603 - val_loss: 0.4748 - val_accuracy: 0.8940\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.0948 - accuracy: 0.9626 - val_loss: 0.4804 - val_accuracy: 0.8926\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 2s 43us/sample - loss: 0.0730 - accuracy: 0.9719 - val_loss: 0.4814 - val_accuracy: 0.8912\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.0673 - accuracy: 0.9741 - val_loss: 0.5017 - val_accuracy: 0.8900\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.0633 - accuracy: 0.9759 - val_loss: 0.5200 - val_accuracy: 0.8930\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.0603 - accuracy: 0.9781 - val_loss: 0.5209 - val_accuracy: 0.8914\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.0573 - accuracy: 0.9785 - val_loss: 0.5416 - val_accuracy: 0.8934\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.0488 - accuracy: 0.9830 - val_loss: 0.5411 - val_accuracy: 0.8940\n"
     ]
    }
   ],
   "source": [
    "lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.02, momentum=0.9)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[lr_scheduler])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcwAAAEeCAYAAAAHLSWiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOydeXxVxfXAvycEAiSIbLJpAlFQQQV3RREUKlhta9W2KrbaqrjUam1tq61WFLU/qbXVal3qrqhY0bovIERF1NYaEXHBDVBJwqZAwp6c3x9zL7l5ecu9yVuScL6fz3zevTNzZ+YNj5w7Z845I6qKYRiGYRjJycv1AAzDMAyjNWAC0zAMwzBCYALTMAzDMEJgAtMwDMMwQmAC0zAMwzBCYALTMAzDMEJgAtPY5hCRE0TE/KnSjIiMFhEVkZ65HothZAITmEaLRETu8f74qohsEZElInKLiHTL9djShfcdn05SvigwB+tF5EMR+Y2ISDbHGRhPWWA8G0VkoYj8XkTaNbPNm9I5TsPIFCYwjZbMTKAvMAA4A/gO8I9cDigHXImbg92B64BrgIk5HM/d3nh2BW4ErgIuyuF4DCNrmMA0WjIbVbVSVb9U1ReBacCRwQoi0lVEbheRZSKyVkReFpH9Yur8REQWi8g6b0XXO6Z8koi8F5N3mohUx+QdLSJvequ9lSLylIh09Mo6iMi1IvKliNSIyH9FZFwa5mCtNweLVPUO4N3YOYhFRApE5G8iUiUiG0TkDRE5NFDuq07HeN9nnYi8JSL7hBjPusB4bgJeAo5NMI4eIvKQNyfrRWSBiPw0UH4PMAr4eWDlOsArGyIiz3j/psu8dvoEnt1fRF4UkRUiskZE5ojIwTH9q4icEJO3SERMwBtNwgSm0SoQkVJgPLA5kCfAM0B/4Bhgb+AVYJaI9PXqHAjcA9wODAeewq3aovY/HngCmAHsCxwOvEz9/6G7cX/8Twb2BO4FnhKRYVH7StC/iMho3Epzc4rqU4AfAT/Dzcl84Hl/TgL8CbgY2AdYCUxtgrp3PdA+QVlH4G3cv81Q4AbgNhEZ45VfALxO/aq1L/CFN85XgPeAA4CxQBHwpIj4890FuB8Y6dV5B3jW9k+NjKKqliy1uIQTcluAatwfZfXShYE6R3jlnWKefQf4rXf9IDAjpvwO99Pfej8JeC+mzmlAdeD+NeDhBGPdGagDimPy/w38I8V3fDpJ+SJgo/cdN3nffz0wIskzhV7dnwTy2gGfAld596O9tsYF6hzi5e2YpO0y4CbvOg/3ArMRuDam3Z5J2ngYuCNem4G8K4GXYvK6eW0fkKBdASqAUwJ5CpwQZ04vyvXv21LrTLbCNFoyr+BWhQcAfweexe2b+ewLdAaWi0i1n4A9cEIM3Irs9Zh2Y+/DsDdO/RiPfXB/sN+PGcfRgXE0letxczAKmA1coapzk9TfGbfie83PUNVa3HceElP33cD1Uu9zhxTjmeh9tw3Ak8ADwBXxKopIOxH5g4i866mwq4HjgOIUfewLHBYzl18Evh8isoOI3OYZHq0G1npjT9W2YTSZ/FwPwDCSsE5VP/GuzxeR2cBluBUhuFVOFU4tF8sa7zOMirEuTr1EasZ45OFWM/vTWF26PkI78VjpzcEnInI88LGIvKmqsxPU979HPLeZ2LzNccpSvURPwwnIjcBSTxgn4iLg1zjV63zcSvkaUgvlPJyqPd5eY5X3eS9uL/pC6lfiLwEdAnWV5v27GkYDTGAarYkrgOdE5HZVXYrbH+sN1KnqZwmeeR84KCYv9n450FtERFV9wTE8pk45MAb4Z5w+ynF/mPskEWTNRlW/9lww/ioiewfGGuQTnEr2UOAzcCs94GCcerq5rA68xKTiUOApVb3fG4cAg4FvAnU24VTGQd4GfggsVtVE+7WHAuer6jNe271xe6BBlgfzEtQxjNCYStZoNahqGbAAuNTLmolTPT4hIkeJyEAROVhErhARf9V5IzBWRC4RkUEicibw/Zimy4DuwO9FZGcROR04IabO1cAPROQqz4JzqIhcKCKdVXUhMBW4R1xQhFIR2U9ELhKR41J8re1EZHhMGpCk/s04l44fJJijGuAW4P9E5Nsisrt335vsu+QsBMaIyKEishtwEzAwps4i4AARGSAiPT2jnpuBrsA0ETnQm8+x4qyhuwTaPsX7t9gftze6KabtWTgL3P1EZG/cnvGGTHxRY9vABKbR2rgeOF1ESrwV1rdxfxj/CXwEPIITKEsBVPUN4HTgHNye3XHUq3Tx6nzglU/06nwLpzoM1nkWJ2iPwq0oX8ZZytZ5VX6Ks/acAnwIPA0cBixO8X1Geu0F03WJKqvqcpx16KSAxWgsv8PNw904A6i9gPGqWpFiLOnmKuA/wHO4/ega3ItFkOtwgu593Iqw2NMeHIKb2+dxL0k349SuG73nfoaznP0fTljehRO+QX6NW2WXAY/ijL2Wpem7GdsgEl+rYxiGYRhGEFthGoZhGEYITGAahmEYRghMYBqGYRhGCExgGoZhGEYIzA8zJHl5edqpU6dcD6PFUVdXR16evXfFYvPSGJuT+LT1eVm3bp2qapv4giYwQ9KhQwdqampyPYwWR1lZGaNHj871MFocNi+NsTmJT1ufFxFpbrSrFkObkPqGYRiGkWlMYBqGYRhGCExgGoZhGEYITGAahmEYRghMYBqGYRhGCLIqMEXoLsLjItSIsFiEk5PUvVCEShFWi3CXCAVefoEId3rPrxWhXISjYp4dI8KHIqwTYbYIJYEyEeFaEVZ6aYpI6jMTN25sx4ABMDU2dHQCpk6FAQMgL482+9z+O1Ygh1/O/jtVZrw/wzCMnKOqWUugD4FOAy0CPRR0NejQOPXGgVaBDgXtBloG+n9eWSHoJNABoHmgx4CuBR3glff02v0BaEfQP4O+EWj7LNCPQHcE7Q/6PujZqcfeWUG1c2fVBx7QpDzwgKsH9aktPncz5+gW8vQmzs1of62R2bNn53oILQ6bk/i09XkBajSLciaTKWunlYhQCHwN7KHKQi/vfuArVS6OqfsgsEiV33v3Y4CpqvRJ0Pa7wBWqTBdhInCaKiMC/a4A9lblQxHmAveocrtXfjpwpmqjQ4Vj+ihUdzoRdO0K55+fuO6NN8Lq1Y3z29JznVZXsIgBFLCJdXSilM/Y0LVPk/orKYFFixI/1xpp6751TcHmJD5tfV5EZJ2qFuZ6HOkgmwJzb2CuKp0CeRcBo1T5TkzdecA1qkzz7nvizsrrqcrKmLq9cWcODvcE4g1AB1XOCdR5D7jcE6irgSNVedMr2w+YrUoXYvCE70R3V7ivLzBBkSRKXDel8Sq0nedu5uecwy0IsIEO3MkZnMdNTepPRJk16+XED7ZCqqurKSoqyvUwWhQ2J/Fp6/Ny+OGHtxmBmU117EjQypi8M0HL4tT9FHR84L69p8IbEFOvPehM0NsCeXf66ttA3mugp3nXtaC7BcoGeW1L8vF33qpGLCmJo3cIUFKiDdSObe25/fov1XV0bPBQDZ10vx0rMtJfa6Stq9mags1JfNr6vNCGVLLZNPqpBraLydsOWBuirn+9ta4IebiT5zcB50XoJ17b1aqEWmp37gxXX528ztVXu3pt9bmpu00mj9oGeXnU8sCukzPSn2EYRksgmwJzIZAvwqBA3jBgQZy6C7yyYL0q9dSxnlXrnUBv4HhVNid61tvD3DnQT7y2442hESUlcPvtMGFC8noTJrh6JSUg0vaeG7zydQoaTDl0ZBO7rpwbqr8unvK7uDhcf4ZhGC2CbC5nQR/2LGULQQ9JYiU7HrQSdIhnJTsrqGYFvRX0DdCiOM/28to93rOSvTbGSvZs0A88C9l+oAvCWMkWFBSkVD1sUzz2mCro+h12UC0tVa2rC/3o3//uVLEVyTW4rZq2rmZrCjYn8Wnr84KpZJvMuUAnYBnwEHCOKgtEKBahWoRiJ8R5HpgCzMYZ9CwGLgfwfCrPAoYDld5z1SJM8J5dDhwPXI2zyj0QODEwhtuAp4D5wHvAM16eEYXKSvcxfjx89hl89FHoR0tL3ednn2ViYIZhGJkhq8d7qbIKODZO/hKgKCbveuD6OHUXE9+0M1hnJrBbgjIFfuslo6lUVQFOYA647z54+mnYLe6UNyIoMEeMyNQADcMw0ouFxjOaRlUV9OjBhr59Ya+9nMAMyYAB7tNWmIZhACDSHZHHEalBZDEi8aPAiQgiVyHyFSKrESlDZGi2hmkC02gaVVXQx4sjccwxMGcOfPNNqEc7doT+/U1gGoaxlZtxHg+9gQnALQkE4Q+AnwEjge7A6zhviaxgAtNoGpWV0Lu3uz7mGKithRdeCP14aSl8/nmGxmYYRutBpBBnd3IZqtWozgGeBH4cp/ZAYA6qn6FaCzwADMnWUE1gGk2jqqpeYB5wAPTsGUktO3CgrTANY1ugJ+Qj8lYgTYypMhioRXVhIG8eEG+F+TCwCyKDEWkPnAo8n5mRNyarRj9GGyIoMNu1g6OOgmefdSvNdu1SPl5aCvffDxs2OBWtYRhtkxWwBdX9klQpAmKjTK+GxuFKgQrgVeAjoBb4AjgiHeMMg60wjehUV0NNTf0eJji17MqV8OaboZooLXWB8RYvztAYDcNoLUSJAnc5sD+wE9ARuAKYhUjnOHXTjglMIzqeS8nWFSbAkUe6lWVItaz5YhqG4bEQp7YNEwVuGDAN1S9R3YLqPUA3srSPaQLTiE48gbn99jByJDzzTKgmTGAahgGAag3wGHAlIoWIHAJ8j/jWr/8FfoBIb0TyEPkx0B74JBtDNYFpRMcXmH1ijic95hh4911YsiRlE336uL1LE5iGYRAnChyqCxApRqQakWKv3rU4g6B3gG+AC4HjUQ3n09ZMTGAa0fHC4jVYYQIcfbT7DLHKFHGrTBOYhmGgugrVY1EtRLUY1Qe9/CWoFqG6xLvfgOrPUe2L6nao7oNq1qxkTWAa0fFXmL16NczfdVfYeedIalnzxTQMo7VgAtOIjhcWj/btG+aLuFXmSy/BunUpm/F9MTXUSaSGYRi5xQSmEZ1gWLxYjjnGOVfOmpWymdJSWLvWeaMYhmG0dExgGtEJhsWL5bDDoKgolFrWLGUNw2hNmMA0ohOM8hNLQQF861vOHzOFrtUEpmEYrQkTmEZ0kglMcGrZL790LiZJGDjQfZrANAyjNZBVgSlCdxEeF6FGhMUixD/zzNW9UIRKEVaLcJcIBYGy80R4S4SNItwT89wEEaoDaZ0IKsK+XvkkETbH1CnN2Jdua8QLixfLt7/tPlOoZQsLndw1gWkYRmsg2yvMRmeeiTSOSC/COOBiYAwwACjFxQz0WQpcBdwV+6wqU1Up8hPOIfYz4O1AtWnBOqrYn+ywxIvyE0ufPrDffqHC5JkvpmEYrYWsCUwRtp55pkq1KsnOPDsVuFOVBap8DUwGTvMLVXlMlX8DYewrTwXuU8WcF9JBGIEJTi37xhuwfHnSauaLaRhGayGbK8zBQK0qYc48G+qVBev1FqFHlA5FKAEOA+6LKfqOCKtEWCDCOVHa3OaJIjBV4fnkQTgGDnSR9DZvTtP4DMMwMkQ2z8OMcuZZbF3/ugvhVpU+PwFeVSW4hnkEuB2oAg4EpovwjSoPxT4swkRgIkB+vlBWVhah67ZJv1dfZTAw97PP2LRmDdXV1fHnpa6Og7t3Z/Vdd/H+TjslbG/Tpj7U1e3GI4+8Qf/+GzI27myTcF62YWxO4mPz0nrIpsCMcuZZbF3/Ol7dZPwEuCaYocr7gdu5ItwAnACNBaYqt+OEKx07qo4ePTpi920Q7z/2iO99D9q3p6ysjITz8v3vs8O//sUOhxzSOCqQhwhMmQI77HAQbWl6k87LNorNSXxsXloP2VTJLgTyRQhz5tkCryxYr0o1/OpShEOAfsCjKaoqIGHb3eZJFBYvHkcfDWvWwGuvJaxivpiGYbQWsiYwVdl65pkIhZ5AS3Tm2X3A6SIMEaEbcCnUu4+IkC9CR6Ad0E6EjiKNVsunAtNVG65KRfieCN1EEBEOAM4HnkjT12z7pPLBDDJ2LHTokNRatl8/V8UEpmEYLZ1su5U0OvNMlQUiFHv+kMUAqjwPTAFmA4u9dHmgnUuB9TjXk1O860v9Qk+Y/hC4N84YTsQdNroWJ5ivVY1bz4hHsjiysXTpAqNHJxWY7drBgAEmMA3DaPlkcw8TVVYBx8bJX4Iz9AnmXQ9cn6CdScCkJP1sALZPUHZS6AEbjamshAMPDF//6KPhggvgk09gl13iVjFfTMMwWgMWGs+IRhSVLIQ6VNp8MQ3DaA2YwDTCU1PjUhSBufPOsPvuSdWyAwfC11+7ZBiG0VIxgWmExw9aEHYP0+foo+Hll93hl3HwLWVtlWkYRkvGBKYRnspK9xllhQku6s/mzTBjRtxicy0xDKM1YALTCE/YsHixjBgB22+fUC1rx3wZhtEaMIFphKepKtn27WHcOHj2Waira1TctauLhWAC0zCMlowJTCM8vsDs1Sv6s8cc457/3//iFptriWEYLR0TmEZ4KivDh8WLZfx4Fzg2gVrWBKZhGC0dE5hGeKL6YAbp2RMOPjihP2ZpKSxeDLW1zRifYRhGBjGBaYQnSli8eBxzjFPJLl3aqGjgQNiyBb78shnjMwzDyCAmMI3wVFY2fYUJ9VF/nn22UZG5lhiG0dIxgWmEpzkqWYA994SddoLp02HUqHq/TkxgGobR8jGBaYSjKWHxYhFxatmZM2HOHJg8eWvRTju5k0tMYBqG0VIxgWmEo6k+mLEcfLDbrKyrg7vv3rrKzM+HkhITmIZhtFxMYBrhaGpYvFhefbX+ura2wSrTXEsMw2jJmMA0wtHUsHhBKirg/vvr7zdtarDKNIFpGEbGEdkBkdMQiXCwr8MEphGOdAjMyZMbh8YLrDJLS2HFioSHmhiGYURH5FlEfuldFwJvATcCcxCZEKWprApMEbqL8LgINSIsFuHkJHUvFKFShNUi3CVCQaDsPBHeEmGjCPfEPDdABBWhOpAuC5SLCNeKsNJLU0SQjHzhtoQvMHfYoeltvP66W1UG2bQJ5s4F6oOw2zFfhmGkkf2AWd71cUAN0As4C/htlIayvcK8GdgE9AYmALeIMDS2kgjjgIuBMcAAoBS4IlBlKXAVcFeSvrZXpchLkwP5E4FjgWHAXsAxuIkzktGcsHg+5eWgCr/4hTu9RNWl8nLAXEsMw8gI2wH+8fRHAo+juhGYCewSpaHQAlOE3iJcJMItIvT08g4RYWDI5wuB44HLVKlWZQ7wJPDjONVPBe5UZYEqXwOTgdP8QlUeU+XfwMqw449p+y+qfKnKV8Bfgm0bCWiuD2aQvn3hm29g3boG2SYwDcPIAEuAgxHpDIzDCUqAbsC6hE/FIT9MJRH2BV4CPgeGAn8GVgDfAgZDYtVqgMFArSoLA3nzgFFx6g4Fnoip11uEHqqhheRiERSYAfxGlRWBtufFtN1olQsgwkTcipT8fKGsrCxk122PvRcupK6ggHkxc1BdXR15XnqvXs3uwBuPP86G/v235qtCYeGhvPpqFfvs83HzB51DmjIvbR2bk/jYvAAi3YE7cSvAFcAlqD4Yp96twCmBnPbAJlS7JGn9b8ADwBqgAijz8g8D3os0TlVNmUBng17hXa8FLfWuDwZdHLKNkaCVMXlngpbFqfsp6PjAfXtPfzcgpt5VoPfE5BWB7geaD9ob9FHQFwLltaC7Be4HeW1LsvEXFBToNs3OO6uedFKj7NmzZ0dv68UXnTL2lVcaFe29t+pRRzVhfC2MJs1LG8fmJD5tfV6AGk0lI+AhhWkKRQqHKqxWGBriuXsU7gpR70CFHyh0CeR9V2FkymcDKdQKE9gXOD1OfgVuPzIM1ThdcpDtgHg2kbF1/euU9pOqVOOsoACqRDgPqBBhO1XWJGi72s2fkZDmxpEN0q+f+4wThL20FN6L9s5nGEZrxlmuHg/sgWo1znrV3667OMRzx6TsQ/VN4M3As+1QfTLqUMPuYa7H6Xtj2Q1YFrKNhUC+CIMCecOABXHqLvDKgvWqIqhjg/iC0LeEjdd2vDEYPukIixckhcBctKix94lhGK2TnpCPyFuBNDGmymCgFtXY7bq4W2UBjgeWA68krSVyLiLHBe5vAzYgsgCRQYkfbExYgfkEcHnAtUNFGABcC0wP04AqNcBjwJUiFIpwCPA94P441e8DThdhiAjdgEuh3n1EhHwROgLtgHYidBRxq2URDhRhVxHyROiB87cpU2V1oO1fidBfhH7Ar4NtG3FIhw9mkO23h44dEwrMjRtdjAPDMFo/K2ALqvsF0u0xVYpg699nn9VAsn1JcAac93l7a8m4EN9AVGQkzkPjVOB9nNFnaMIKzIuA7jhp3hmYA3wCfIMTZmE5F+iEW5U+BJyjygIRij1/yWIAVZ4HpgCzgcVeujzQzqW4Ve/FuA3g9YFxlALP49S37wEbgZMCz94GPAXM98qf8fKMRKQrjqyPiFtlJjgXE8xS1jC2IaJs1zlEdsIZjN4Xov0dAf8vyneARz2DosuBEVEGGmoP09v7O1SEI4B9cIL2bdWt5rmhUGUVzgcyNn8J7i0jmHc9cH2CdiYBkxKUPYQTxonGoDhn1UgOq9s06YojGySBwAy6lowcmb7uDMNosSzEqW0Hoeqbx6faKvsJMBfVMK/Wa3GBCr7AeXb4q8pNQMcoAw3rVvITYJoqs6iPmIAIHYATVUNJeaO1km6VLDiBOW9eo+ySErcAtRWmYWwjqNYg8hhwJSJnAMNx23XJVn8/wW0JhmEGcBsib+P2S5/z8ocAi6IMNaxK9m6ga5z8Ll6Z0ZZJR1i8WBKsMDt0cGdjmsA0jG2KRtt1qC5ApBiRakSKt9YUORinZv1XyLZ/jvOc2BH4Iaq+8ej+wLQogwzrViIQ1+2imMabtUZbo6qq+WHxYunb10VZX7sWujTc27dTSwxjG0M17nYdqo2261B9HSiM0PY3wDlx8i9rXDk5SQWmCPNxglKBl0XYEihuB5QAz0bt1GhlpNMH08d3LamoiCswn7VflWEY6UKkA3AiTg2ruP3RR1DdlPS5GFKtMB/1PvfAWZNWB8o24fS/odxKjFZMOuPI+gR9MQcPblBUWupk9Lp10Llzers1DGMbQ2Q3nOdEd+oNiX4OTEZkPKofhW0qqcBUdSeEiLAIZ/SzoUkDNlo3VVVwwAHpbTNF8AJwAQyGDElvt4ZhbHPcgHMhPMVTz4LI9sBUr2x82IZCGf2ocq8Jy22YTK8wYzBfTMMw0sihwMVbhSX4+5qXeGWhCSUwReggwhUiLBRhgwi1wRSlQ6OVUVMD1dXpF5hdukBhYUpfTMMwjGaykcaBEcB5eUTawwzrVjIZ7xxJoA74De4w6JU4c2CjrZIJH0xIGu2nVy8nS01gGoaRBp4BbkfkQETESwcBt+KivoUmrMD8IXC2KrcBtcATqpyPCy30rSgdGq2MdIfFC9KvX9ygsSLmWmIYRto4Hxde9XVgg5dewxmt/jJKQ2H9MHvjAtWCs5Td3rt+nvDRFozWSKZWmOAE5n//G7eotBQ++ST9XRqGsY2h+jVwtGctuzsursD7qH4YtamwK8wlgGelwSfAOO/6YFzgc6Otkok4sj59+zqVbJzDBkpL4fPP4xYZhmFER/VDVB9H9TFUP0RkF0TmRmkirMB8HBjjXd8AXCHC57hjse6I0qHRyshEWDyffv2cs+WaNY2KSktd0bKwp60ahmFEoxA4MMoDYU8ruSRw/agIXwCHAAtVeTrSEI3WRSbC4vkEXUu6NgxVHLSUzcTi1jAMIyphV5gNUOVNVa5X5WmRCDH9jNZHJsLi+ZgvpmEYrYgmCUwAETqK8Bvg8zSOx2hpZCJogU8SgTlggPs0gWkYRkshqcD0AhZcLcJ/RZgr4qLJe+djfoYzyf1r2M5E6C7C4yLUiLBYhJOT1L1QhEoRVotwlwgFgbLzRHhLhI0i3BPz3EEizBBhlQjLRfiXCH0D5ZNE2CxCdSCVhv0OLZqKChg1qt5QJx1kUmD29f5Z4gjMTp2cPDWBaRhGkxApR+TthCni0V6Qeg9zEi5I7QzcnuW/RPgnzgDoEuBBVTZH6O9mXGSF3rhDQp8RYZ5qw5O1RRgHXAwcASzFGR1d4eXh5V2Fs9btFNNHN+B24AVgC3AT7szOYLzAaaqcEmHcrYPJk2HOHPd5883pabOqKjM+mABFRbDddnEFJpgvpmEYzSLt9jWpBOYPgdNUeVyEYUA5TiANVW1w1FdKvL3O44E9VKkG5ojwJPBj6gWhz6nAnb4gFWEyLlDuxQCqPObl74c7FHQrqltP0/b7vQl4OcpYWyUVFXDHHVBXB3ffDZdd1nxBl6mweEESRPsBJzBnzcpc14ZhtGGacN5lKlIJzJ2A/7q+mSfCJuDaqMLSYzBQq8rCQN48YFScukOBJ2Lq9Rahhyor49RPxmHQcAULfEeEVUAFcJMqt8R7UISJwESA/HyhrKwsYtfZY9Bf/0q/zZsRoG7zZirOPpuPfxkpiEUjOi5dykHAh19/TWWC715dXd2seRnWqRN5H35IeZw28vJK+OqrAbz44it06NC6HDKbOy9tEZuT+Ni8tCJUNWECrQPtFbhfCzow2TNJ2hoJWhmTdyZoWZy6n4KOD9y3dy7sOiCm3lWg9yTpcy/QVaAjA3lDQPuBtgMdAVoBelKq8RcUFGiLZelS1YICVW+SFFQ7dVKtqGheu3PnuraeeSZhldmzZzevj1NOUR04MG7Rffe57j/6qHld5IJmz0sbxOYkPm19XoAabYLMaIkpjB/mn0RY5113ACaJsLqh0OX8EO1U0zhi/HbA2hB1/et4deMiwi7Ac8AFqrwaGOv7gWpzRbgBOAF4KGzbLY7Jk6E25tCY2trm72VmMo6sTzDaj0iDoqBrScwZ04ZhGFknlcB8Bdg5cD8XKI6pE1ZXthDIF2GQKh97ecNorC7FyxsGPBKoVxVWHStCCTATmKzK/SmqKy62YOvl9ddhS4yWfNMmmBsp6lNjMhkWz6dfP9i4Eb7+Grp3b1Bkx3wZhtGSSCowVRmdro5UqRHhMeBKEc7AWcl+DxgRpyWIhqoAACAASURBVPp9wD0iTMXtM14K9e4jIuTjxt4OaCdCR2CLKltE6A/MAm5W5dbYhkX4Hu5F4Btgf1wk+9+n63vmhPJyOPZYmDcPvvgCLr4Yrrqq+e1mMiyeT9AXM0Zg9ukDHTuawDQMo2UQ9rSSdHEucBewDHeW5jmqLBChGHcayhBVlqjyvAhTgNk4t5HpuKPEfC6NuT8F53YyCTgDKAUuF6mvo0qRd3miN4YC4EucEdO96f6iWae8HEaMcK4a77yTnjYzGRbPJygw99ijQVFenlPLmsA0DKNZiPQDDgV2IDb+gOqNYZvJqsBUZRW44Acx+Utgq0Dz864Hrk/QziSccIxXdgVOeCYaw0mhB9xaWLUKliyBn/8c2rWDdFncZTJogU+SaD9gvpiGYTQTkROp11CuoOE2ogKhBWaTQ+MZLQh/RTl8uEtffQUrVjS/3UzGkfVJEu0H6gWmti6vEsMwWg5X4YRiF1R3RHWnQIq1yUmKCcy2QHm5+9x7bycwwe1nNpdsrDA7dYJu3ZIKzLVrYWVU71vDMAxHH+BWVKNEpYuLCcy2QHk59O8PvXrBsGEuLx37mJkMixckRbQfcIdJG4ZhNIHncQaezSbUHqZnlBMPBTaosjwdgzGayDvvuNUlOKHZv3/zBWY2wuL59OvnQvvFIeiLuX9afvKGYWxjPAdMQWR3YD7ExD9XfTJsQ2GNfhaRxN9ShDW4AOe/1aaFzTOayvr18OGH8P3v1+cNG9Z8lazvUpItgTl7dtwiOxfTMIxm8k/v849xyhTnnhiKsALzJGAKcCvwppd3IC7O6iRge5yrx1oaunsYmWb+fBfVx19hgtvHfPFF2LDBOTI2hWwKzL593Qqzrs75kgQoKnJuoCYwDcNoImnziwsrMM8BLlTvlBCPWSJ8hAs9N0qEZTh3DhOY2SRo8OMzfLiL/PP++7DPPk1rNxth8Xz69YPNm51lT69ejYrNtcQwjCajWpu6UjjCGv0ciNP9xvIe9ZuprxNz1JaRBd55B7p2hQED6vN8S9nm7GNmWyUL5otpGEZmEBmHyCxEKhGpQOQlRI6M2kxYgbkY75irGM4ElnjXvYBVUQdgNJPycicgg4HLd94ZCgubt4/px5HNZFg8nxACc8kStwg1DMOIhMhPcYdJf4XTgE7ChVx9GpHTojQVViX7a2C6CN/GnY+puJXlzrhDofHuH4n/uJERamvh3XfhrLMa5uflwV57NX+F2b17ZsPi+YQQmHV1TmjuvHPcKoZhGIm4BLgI1RsCebch8pZXdk/YhkKtMFV5BhgEPIk7amt773pXVZ716vxDlV+F7dhIAx995Kxkg/uXPsOHO4HZ1BA52fLBhPp+zBfTMIz0UwI8Eyf/aa8sNKFjyaryBU4aGy0FfwWZSGDecgssWlTvmxGFbET58SkogJ49EwpMcy0xDKMZfAGMAT6JyR/rlYUmtMAUoTPuSK5G0d5jrGeNbFFe7oTNbrs1Lgsa/jRFYFZWZjdSQJLgBf37O82wCUzDMJrA9cCNiAzHnemsuJNLTgN+GaWhsJF+xgIPAT3iFEdy/DTSSHm5OxIr3j7jHnu4vcx58xoGNQhLNleYkDQ8Xrt2zgjYBKZhtFFEugN3AkfiThS5BNUHE9QtxQVTHwVsBO5C9bcJ21b9ByLLcbY4J3u5HwATUJ0eZZhhrWRvwOmAd1QlLyaZsMwFqk5gxlPHAnTuDIMHN83wZ906FxYvW3uYkFRggrmWGEYb52ZgE9AbmADcgsjQRrVEOgAzgFm4oOo7Ag+kbF31X6gehGpXLx0UVVhCeIE5AJisSuK/aEZ2+fJLdw5mIoEJ9YY/UcmmD6ZP375ODVwb38fYBKZhtFFECnHeFpehWo3qHJxR6Y/j1D4NWIrq9ajWoLoB1XezNdSwAvM1YNdMDsSISLwIP7EMHw6LF8PXX0dr2/fBzLZKtrYWlseP419a6r5G1K9iGEZu6Qn5iLwVSLE+/YOBWlQXBvLmAY1XmHAQsAiR5xBZgUgZIns2qiWyCpGe3vXX3n38FIGwAvNW4DoRzhDhQBH2CaawnYnQXYTHRagRYbHIVn1yvLoXilApwmoR7hKhIFB2nghvibBRpLEPjQhjRPhQhHUizBapNx0WQUS4VoSVXpoigsS20eIpL3fBCvZs/FvZin/U17sRX8ByscJM4Yv55Zfus0cPt585dWq4ZqdOdfXz8rL73BFHjMp4f9n+bobRFFbAFlT3C6TbY6oUAatj8lYDXeI0tyNwIm4Psx9uq/AJT1Ub5De42Ob+dbIUmrBWso96n7FfFKIZ/QT11MOBZ0SYp8qCYCURxgEXA0cAS4HHcXFqL/aqLMWdoj0O6BTzbE/gMeAM4ClgMjAN92YCLmLRscAwb+wzgM9wLwWth/Jyt0dZVJS4TtBSdtSo8G1nM46sT1BgxsS/nToVbrvNXau6RfOZZ7qDpY8/noRMnw6/+pVzVYVsPycZ7S+d322i974/YULi5wwjg1Tj/PuDbEe9wAuyHpiD6nMAiFyHO/hjd9yq1KF6Z+D6jrSNVFVTJtCSZClkG4Wgm0AHB/LuB/2/OHUfBL0mcD8GtDJOvatA74nJmwg6N6bf9aC7efdzQScGyk8HfSPV+AsKCrRFUVKieuKJqev17q162mnR2p40SRVUN21KWXX27NnR2k7EF1+4Pm+7rVFRSYkrspS5VFKSnn/GZKTtt9LGaOvzAtRosr+vUKiwSWFQIO8+hUayQWGywqzAvSisVhiWpP2FCt3j5G+vsDDp2GJSqBWmKovTIJsHA7WqxOqp4y19hgJPxNTrLUIPVVam6GcogTcNVWpE+NTL/zC2nMS6ckSYiBdDNz9fKCsrS9F1dshfs4ZDFy/m03Hj+CLFmPYqLqb9nDn8L8LYB739Njtstx2vvfZayrrV1dVpmRfZsoXDRFg8dy6LBg9uULZkySiIqzVXzj//44Rt3njjoDb7XLr7WrJEKSt7OeFz6SBdv5W2xjY/L6o1iDwGXInIGTjt4/eAEXFqPwD8GpGxwGzgfJwbygdJetiF+NrUAiJG+kkoSUGPA20fuE6Ywkhm0JGxq0TQM0HL4tT9FHR84L699yY8IKZevBXmncSsWkFfAz3Nu67FW21694O8tiXZ+FvUCvOll9yy4IUXUtf93e9U27dX3bgxfPvHHac6ZEioqml9O+7dW/XMMxtlJ1phploVteXnsj3GdNDWV1JNpa3PC6lWmG61113h3wo1CksUTvbyixWqFYoDdY9T+ERhjUKZwtAEbX7XS3UKPw7cf1fh+wo3KXyUcmyBlMzo51GgW+A6UfpXSNkcRU8dW9e/jlc3aj/x2q5289tKCGMh6zNsmDvm48MPw7ef7aAFPgmi/Vx9tXMrDdK5s8tPRlt+LttjNIyMoroK1WNRLUS1GD9ogeoSVItQXRKo+xiqu6C6HaqjUV2QoNV/e0mBewP3/8YdFHIUEY1+EgpMdUEJlgWuE6WwBj8LgXwRBgXyhgHxvuwCryxYryqEOrbRsyIU4k5VWRCvPMkYWi7vvOPixcU5bLkRTTkbs7IydwIzjpXshAlw++1QUuIMg0tK3H0qI5XcPqcZ7S9d3w3gd78zgx+jTdIe6IAzEu3n3buk2h7VnVF9MlKLUZajzU2gD4M+hDPEOQR0NWij5TToeNBK0CGg3UBnBdWsoPmgHUH/hDMc6gia75X18to93su/loBRD+jZoB+A9gftB7oA9OxUY29RKtmhQ1WPOSZc3S1bVDt1Ur3wwvDtFxWpXnBBqKppVSedeaZqnz7pay+HtAY127JlTh179dXZ6a81zEkuaOvzQhiVbCtJUYKv7wSMJH7w9etDNnMucBewDFgJnKPKAhGKgfeBIaosUeV5EabgNnU7AdNxB3/6XBpzfwrO7WSSKstFOB64CbdB/CbOb8fnNqAUmO/d3+HltQ7Wr3fq1bDxYdu1c76aYVeYuQiL59O3r1MHb9kC+aF/mkYT6dXLKSBmzoTf/z7XozGMDCLSFeeGWIxbddajek3YZsIGX5+AE3RbgOXQYL9PIZzAVGUVzgcyNn8Jznk1mHd9onZVmYQ7NTtRPzOBOEd4gLdX+VsvtT7mz3cRccLsX/oMG+Yc8FTr9XCJyEXQAp9+/dwYq6qcytnIOGPGwN//7t6TYvc2DaNNILI/8BxQB3QHKnBxaDfgjvcKLTDDRvq5EvgLsJ0qA1QZGEilkQZvNI9kZ2AmYvhwF3fWD5eTjFyExfNJEe3HSD9jx8KmTRDCg8gwWit/AR7GBcxZj3NlLAb+h5NtoQkrMHsDd6gSPzK2kT3Ky6FrVxfTLCxRDH9yvcIEE5hZZORIdzrczJm5HolhZIy9gL97Riy1QAGqFTgtY0YE5rPAgZGGaGSG8nInAFOpVoPsuaerH0Vg5mIP0wRm1ikshBEjTGAabZrN1G8jVuFWlwBrcLFpQxPWsmIGcK0IQ3HGMpuDhao8FqVTo4nU1rpA6medFe25Ll1gl13cYdKp8AXmDjtEH19z2WEHFxHcBGZWGTMGLr8cVq50we0No41RDuyHc218GRdRqCfu+LBIJ1OEXWHehpPEvwceommBC4zmsnChs5KNsn/pM2xYuBVmZSV07+70dNmmXTu3so0TvMDIHGPHOlur2bNzPRLDyAiX4laW/vUa4J84w59Iq49QAjNNgQuM5hIlwk8sw4fDp5/CmjXJ6+Uqyo9PguAFRubYf3+nhDC1rNEmUf0Pqi9518tQ/RaqnVEdjmoItVs9KQWmCO1FeFPEDpDOOeXlUFAAu8X1mEmOb/iT6mzMqqrc7F/6mMDMOvn5cPjhJjANIxUpBaYqm4GB0IpirbZVysthjz2api4NaylrK8xtkjFjnAJi0aJcj8Qw0oDIx4gsDJUiEHYP817gzOijNtKGqhN2TVHHghNEPXqkNvzJVRxZn759Yfly5xxoZI2xY93nSy/ldhyGkSbuAO700kM418ivqLe9+dLLeyhKo2GtZAuBCSJ8C+fsWRMsVOX8KJ0aTeDLL50ZY1MFpohbZSZbYfph8XK9wgQnuIuLk9c10sbuu7t3lZkz4fTTcz0aw2gmqtduvRa5G7gO1ckN6ohcijunOTRhBebuwNvedWxkH1PVZoPmGPz4DB8ON92UOFZrLn0wfYK+mCYws4aIU8u+8ALU1TnvHsNoIxwH7BsnfxpuARiaUAJTlcOjNGpkgPJy91dtzz2b3sbw4bBxI3z0EQwd2rg8l1F+fCx4Qc4YOxYeeMCFKx42LHV9w2glrAcOAz6JyR8JrIvSkB0J0Vp45x0YPBiKilLXTYRv+DNvXnyBmcs4sj4mMHPGmDHu86WXTGAabYobgH8gsg/whpd3EPAz4KooDYVWvIhwuAi3i/C8CLOCKUqHRhMpL2+eOhZg112hQ4fE+5gtYYXZs6dTF5vAzDo77ug8lsy9xGhERQWMGlX/Uh3huV2hY2YGFRLVP+GE4/7AP7y0P3BGlKO9IKTAFOE03PEoXYDRuCO+ugH74M6xNDLJqlWweHHzBWb79s4tJZXAzEVYPJ+8PGd9YtF+csKYMfDyy2akbMQweTLMmeM+Iz5XFGFhljFUH0T1QFS389KBqD4YtZmwKtmLgPNUuUOEtcAlqnwmwk1AddROjYj4As5XqTaH4cPhqafin41ZVeXC4nXoEP/ZbGG+mDlj7Fi4+WZ44w047LBcj8ZoEcydC7fd5qzBbrkFnnkGOnZ0mqD27Rt+Bq9ra2HGjFyPPq2ElfylgK+o2Uj9Yc83AaeF7UyE7iI8LkKNCItFODlJ3QtFqBRhtQh3iVAQph0RJohQHUjrRFARZyUlwiQRNsfUadlneqbDQtZn+HDn5xhvBZdrH0wfE5g5Y/Rot8g3f0wDVScgDzvMCUtwL9kdOri/I7vu6izZe/VythX5+bB5swu/WVUFb71V/1y2EVnlBVgHka+9+/gpAmFXmCtx6lhwzp974KK89wA6RejvZmATzmF0OPCMCPNUWRCsJMI44GLgCGAp8DhwhZeXtB1VpgJTA22dBlxGvVsMwDRVTokw7tzyzjvQv7/7YTYX35pj3rx6AxufXIfF8+nXz+kFjayz/fYutuzMmXDFFbkejZEzvvjCOeTOmNHQx6iuzvmEv/JK8r8VFRVQmtN1yG+Atd71RelqNOwK81XgSO/6EeBGEe7GRUkIteYWoRA4HrhMlWpV5gBP4o5YieVU4E5PAH4NTMZbyUZsx2/rPtVW7C+aDoMfH19gxtvHzHVYPJ++fd2+7YYNuR7JNsmYMfDmm6nj9BttEFW4915n6zB3rltdxvps19am3sucPDl3q0sA1TtR3Ri4TpwiEHaFeR71lk5/ArYAh+CEZ1iz3MFArSrB2H3zgFFx6g4Fnoip11uEHrjDP0O1I0IJzv/mZzFF3xFhFVAB3KTKLfEGLMJEYCJAfr5QVlaW+NtliLyNGxn5wQcs3mcfFqWp/wP79mXtiy/y/sEHN8g/dOlSKjdv5pMI/VRXV6d9XvqsWcNuwBuPP86Gvn3T2na2yMS8ZIuePbentnY4N900nxEjVqat3dY8J5mkpcxL+1Wr2PX66+n52mt8s9defPi73zH08svpEmsBtmkTa198kf8lGfO+M2Y0fq4toKpZSaAjQStj8s4ELYtT91PQ8YH79u7VRwdEbOey2HzQIaD9QNuBjgCtAD0p1fgLCgo0J7z5piqoTp+evja//33VwYMb5tXUuH6uuSZSU7Nnz07fuHyef96NZc6c9LedJTIyL1li/XrVTp1UL7ggve225jnJJC1iXh59VLVnT9WCAtXrrlPdsiVtTQM1miU5szXB1wqrQqUI7YYOXCBCb5zac2ecOnSFCIcAS1X5PEQT1cB2MXnbUa9nTlbXv14bsZ2fAA38bFQbuMHMFeEG4AQiBuHNGr7qNF0qWXAb9v/+N9TUQGGhy2sJYfF8LHhBTunYEUaONH/MbYKvv4bzzoMHH4R994X77oMhQ3I9qnSQtn3LIKEEpmdh+hLwOU5d+mdgBfAtnKo1obVrgIVAvgiDVPnYyxsGDQ1+PBZ4ZY8E6lWpslKEDWHa8YR5P1xk+mQoICnq5I7ycmeJMWBA+tocNswt2OfPh4MOcnktIWiBjwnMnDNmDPzud852o5VqxY14VFTAiSfCtGnub8sZZ8CyZc7C65JLmnZ0YEsk4t5kWMIa/VwH3KDK3ji3Ep8XcHuZKVGlBngMuFKEQk+gfQ+4P071+4DTRRgiQjfgUuCeiO2cCkxXbbjyFOF7InQTQUQ4ADifhvulLYvycrcijPWZbA7xzsZsSQLT9wU1gZkz7LivFk5TI+/4AQiOOAK+/W3o1s1ZeP3xj21HWGaQsCrZfYF4h/5U4Fw7wnIucBewDOeqco4qC0QoxkUMGqLKElWeF2EKMBvntjIduDxVO36hCB2BH+KsaWM50Xu2AHcm2rWq3Jtq4AM3bXI/zmyqLGtr4d134ayz0ttucbFbtQYFZkuII+sj4laZFu0nZwwf7t5bXnoJTmk9DljbDr7gmzQJpkxxFuUbNrjDFfzr2LzKSrjjDme9+sEHcO65cP31UFCQsrtWjUh7nEviSTij0YaRWVRDR2oJKzDX40LhxbIbTmiFQpVVwLFx8pdQHwzBz7seuD5KO4HyDcD2CcpOCjveIIWq7kd6881NebxpLFwI69end/8S4p+N2RLC4gWx4AU5JS/PqWVnzowfFMrIIS+8ALfe6v5hbrvNpaj4q8m2LiwdVwITgGtx2tI/AAOBHwB/jNJQWJXsE8DlgWg7KsIAbwDTo3TYqrn77ugqkOaQzgg/sQwb5vYwa2vdfUsJi+djAjPnjBnjfNQXLkxd18gCq1a5VeH48U5YArRrByNGuBf5O+9057M9+ig8/bR723n1Vfjvf2HWrIbCcfPm7P89yx0/As5C9WacS+RjqJ6LC4YT6ejKKLFkn8UFXe8MzMGpYufi9he3DTZscCqQW2/NTn/l5e5Hvttu6W97+HBYtw4++cSFuGopYfF8+vVrc3EoWxv+PubMme4nYuSIujq45x5nhbVypROS/otuba37OzF9evLtonPPrReyPn4AgmxqzXJDH+qNQqup1z4+C1wdpaFQK0xV1qhyKE4N+jvc+WLjVTnMM8LZNlB1ewDZWvmUl7uIG5nYjI81/GkpUX58+vaF1aud64uRE0pLnXG2Gf7kkPJyOPRQF6Zu8GA44QQnMIOEibzz+uuNj6DZtMlF82n7fAH4tt6f4rw7AA4AIoUTi3TsiiqzVLlOlSmqzBShRGSr68e2QW1tQ5VIplB1wiwT6lhwvlbt2zcUmC3BB9PHdy0xw5+cIeJWmbNm1S9ojCzxzTfwi1/Afvs5LdA99zj16scfN03wlZd7sV9ikr/t07Z5knoh+XdgMiIfA/cCd0dpqLnnlG1PfEvUts38+XDllZnt48svnfolUwKzQwcnNOfNc/ctbYVpvpgtgjFj3EL/f//L9Ui2EVRd8IBdd4V//APOOQc++ghOPdVZYm3bgq9pqP4G1au862m4fct/Aj9C9eJkj8aS+4M9WwnvFRS4H2ZdHfz0p24v88YbM9dhJg1+fIYNcyvMdetg7VoTmEYjjjjCfVrUnwxRUcHwCy5wNgTvvuuCnZ96Kgwc6Ix1brrJ+Uoa0REZEzdfdQ6qU1D9d9QmTWBGRQRuvx2+/3244AJnlZYJystdX3vumZn2we1jVlS4FTOYwDQascMO7r3KBGaGmDyZrvPnw7hxsM8+zj/yjjucinWffXI9uuwh0h2RxxGpQWQxIvGjx4mchkgtItWBNDpBqzMQ+QyRPyDSPx3DNIHZFPLzXezFI46A006Dp55Kfx/vvOM2+YuKUtdtKr7hz4svus+WtIfZtSt06mR7mC2AMWPgtdecIsJII0uXwh13IKpudXnyyU79evrpDc+g3DYInnE8AbgFkaEJ6r6OalEglSWoNxQXFe4XwCJEnkHkWETaJaifkqT/KiI8mSzhrGW3TTp2dAHM99kHfvCD9B94nM4zMBPhn435wgvusyWtMP1oP7bCzDljxzq7ktdey/VI2hBVVc76dfNmd9++PXTpAj165HZcuUBk6xnHqFajmuqM43CofoDqRcCOOF9MBf4FfIXItYhEdpZK9RqzMkX6HBf3ddukSxd47jnYeWf4znfg7bfT0+6qVbB4ceYFZvfusNNO8MYb7r4lCUwwgdlCGDnS/T03tWwaUHWBz3ffHT4PHPLUhgMJ9IR8RN4KpIkxVQYDtajGnnGcaIW5NyIrEFmIyGWIJI8noLoF1cdQPQYoAW4EjgPeR+SVKN8laUeq/DRKY9skPXo4leYhh7h9iFdfbX6gAd/xrbi4+eNLxfDh8MUX7rqlhMXz6dfPrP9aAEVFcPDBJjCbzbJlLoDA9OnQq5d7C/FXmNBmAwmsgC2o7pekShGwOiZvNdAlTt1XgD2AxTiBOg0XvedPoQajuhSRf+COg5xEyMNDfLY5RXlG6N/fRaXJy4Mjj4QlS5rX3t/+5j79vcVM4u9jtqSweD62wmwxjBnj3l1Wrsz1SFopjzzi3Lieegr+9Cf32w4KS9iWAgnEEv6MY9XPUP0c1TpU5+PixJ4QqheRsYg8CCzFhcV7GEgmyBthAjNdDBrk9gLXrHFCc/ny6G1s3OiEpK8iffjhzKtofIG5fn3LUwf17QvV1c7lxcgpY8c6beLs2bkeSStj2TJn4/CjH7nQSW+/DRdf7Iz6PB/Kstmzt3V/yoU4te2gQF6is5JjSX6esUgxIpcj8jnwIu6M5IlAP1R/jmqkCTeBmU6GD3dBj5cscdGAFi5MfGZdVZUTjn/+szs/ac89ne5r3Djn6wnhQl6lY8zgBGam+4qKuZZkjojnKe6/v9uyN7VsBB55BIYOhSefdKvKuXPdvdEQ1a1nHCNSiEjis5JFjkKkt3e9G3AZic4zFpkBfAachVtNDkZ1NKoPoBopJJ6PCcx0c+ihbo/i3Xdh9Gh3Zt0vf+n8NX/zGycQ+/Rxadw4+O1vnYVtSYnb3wjGjd20KfOGAMETDFqa0YEJzMzhn6cY8iWpfXv3czaBmYDgC8jy5fWrygED6leV+WHPutgmORd39vEy4CHgHFQXeCvEakR8g44xwLuI1OCCpz8GXJOgzfU4456dUL0E1U+aO0gTmJngqKNcFKCKCrdanDYNfvxj+PvfYcUKV/7Xv7ognStWOKObp592exqxBw9mepV59dX1/5GzsaKNggnMzFBR4Y6CqquDu+4K/ZI0Zgx8+iksWpTZ4eWUiCvvrfgvIKee6vYqn3wSrrnGBT23VWVqVFeheiyqhagWo/qgl7/E87Vc4t1fhGpvr14pqn9EdXOCNr+L6pOopi0SclYFpgjdRXhchBoRFosQP5qDq3uhCJUirBbhrsBZnEnbEWGACCpCdSBdFigXEa4VYaWXpogk0YE3lfnz6wVRfr5721y71gXlvPtut+o8/PCGflfZPlGgosKNZcuW+r5a0irTBGZm+P3v639nmzaFfkna4CmxBg50C6epU8N1N3Wqq3/EEaOa9FxeXtP6a8pz9w+eTO0rc7hv0OTGz9XVuegNq1bBV1+5oOjvvedcy/wXkBdfdEaAb78Nl1xiq8q2hqpmLYE+BDoNtAj0UNDVoEPj1BsHWgU6FLQbaBno/4VpB3SAt4Oen2AMZ4F+BLojaH/Q90HPTjX2goICDc3SpaodOzYMj9ypk2pFRfg2ssE556h26NBwnB06qJ57bugmZs+enbnxqaoWFaleeGFm+8gAGZ+XpvLVV6p5eQ3/zQsKUv42H3hAtXPnho917uzyW/VzdXWqn32mZb98XP+a/2vdgpubWkQXs5Ou69pHtWtXN0fxw543TO3aqZ59dvJBxtBifytpAqjRLMqZTKasvf6I4Edz2EOVamCOFy3ox0BsxPhTgTtVnZWUCJOBqcDFEduJx6nA+aplUwAAGMRJREFUX1T50mv7L8CZQPpOhZ48ud5wx6cl+li1hjPyzLUkvZxySuPf5ubNKX+bf/hD49B469a5cMrJjv761a9y81wfKniYE/kR06ha14cLLnANdPtyPt2WzKPbF+/S7Qv32WH9GkYBh+FMLgEUYR2deLx2PCf/rJOL7NWpk0vB6/Xr3TFc/v+j2lq49164/PKWFWrSSAviXgCy0JGwNzBXlU6BvIuAUap8J6buPOAaVaZ59z2B5UBPoDhZOyIMwEUgWor7/c8AfqPKCq/uauBIVd707vcDZqs2dpIVYSLOBJn8/E77zpjxXKjvuu+ZZ9Llk8b7y2t32YX//fOfodpoLVRXV1OUwXi3wy68EKmr450bWlcUxkzPS1MoqKzkoJNPdrFLY6gpLua/996b8NkjjhiFavp3LjLFrUzkTO7gP+zPEkoYxjwG8TF5nkhcQxfeZS/mMYx5DONL+jOdE+gUOE94HZ3YmU95aPZHCfsZ9Ne/0vfZZ8nztzWAuvx8Ko4+mo9/+ctQY22Jv5V0cvjhh69T1cJcjyMdZFPBHiWaQ2xd/7pLiHZWAPsD7wA9cEF9pwLjkrRdJIKo0uAviSq3A7cDdOyoOnr06MTfLsjHH8fN7gKEbKHVUFZWRuh5aQpDhsCbb2a2jwyQ8XmJSl2ds9opLHQW3AMHuvzly2HgQApHjEg63uJiF60xln79XHCrRIwcGV9BkMnnei/9H2dyB3koB/Ef+lLB+x33ZYeJJ7Fxt2Fs2n0YW3YcQL+8PPoBRwGv7nkusq7hyjuPWq4tuorRo5NohS68sN4GwH9uyxb6L15M/5D//i3ut2IkJJsCM3w0h8Z1/eu1qdrx1LRveflVIpwHVIiwnSprErRdHSssjRZC377uL6dqYwtiIzx/+xuUlTnjFF9YggvR9otfwLXXwqWXJrTovPpqmDixoZq0c2eYMsX54ydiypTsPvfPc97msMsOQ7z/zhvowPPtvkPRHTfTbULi57bv9TodFzfcnujIJo7pkWJ7YtsMNLDNkk0r2YVAvghhojks8MqC9apUWRmxHajflvD/2sZrO0xECSMX9Ovn9olWxyoVjNAsWOAsY7/7XXf4eSwXXeRWnldckbCJCRPcMbAlJe69paTE3U9IIoQaP6dNfC5kf088wbevOZRC1m39z96RTZyedzcTxiS3/O6+qJypDygDSpQ8cemA/ZXui0wgGgGyaWEE+rBn4VoIekgSK9nxoJWgQzwr2VkxVrIJ2wE9EHRX0DzQHp417ezAs2eDfuBZyPYDXZB2K9ltiIxb+D30kLM+XLAgs/2kmRZj+bhxo+rw4aq9eqlWVSWu94c/uHl+992MDSVjc1JXp3rddaoi7ns20/JbVfXyy92j//lPZoYcpMX8VjIEbchKNtuBCxpFc1BlgQjFnr9ksRPiPA//3979h1lVVgsc/y4ZYPg1EIjKjxjDEBUTTRQfTMT0RlKmiRY/VLLEwigSNCUvOImgmdnV5woXM5QRRSqhQEnBC0QohVikIUiIgMLAhUhkhhwYWPePdcbZnDkzc2bmzN7nx/o8z37g7F/nPfvZc9Z53/2+7+IBYDk2K/024O66zhPb1hN4EWui/TtQDgwPHDsTWAS8Gdv+QmydS0c+FrNxfvxjm7f0F7+oPRvN+PFQUABFRaEVLSUOH4YxY6yWPHSoNeGnoOf3+PFw/PFWMXeuUqijalXZB1yVYP12rDNOcN1DwEP1OU9s21wsiNZUBgV+GFtcuvOA2XCvvgr332/NsFdeWfu+HTvaZBr33GMBtnKO4XT2wQfwta9ZpqCJE+Hee222ghQoKLBgOX68Zdu79NKUnNZlOJ8az6W3Ll3s35KSaMuRaUpL4YYbrHtrZbq4utx6K7RvX+uzzLTx7rswYIClT5k1y6ahS1GwrDRmjOVXnzjR2nad84Dp0lubNvYl7jXM+rntNtiyxQbRF8R3Kq9Bhw5Wpfrtb21qt3S1ejX0729TOC5ZkrgjUwrk51sL9WuvwYIFTfIWLsN4wHTpL1dm+2noxN/xFi+GmTNhwgQYOLB+x44bZ4EzXZ9lPvuszcFcUGCB85JLmvTtbrgBTjvNRtzEDbd0OcgDpkt/uRIw65lyK6G9e+Fb34Izz2zYedq3t9rpokVWtYpa5Y+IkhL7PMOHw/nnW5L13r2b/O3z8uzR6IYN8FT17Iwux3jAdOkvFwLmli3w+OM2I09DM8ao2oO3f/7Tvt3z8xtWlu99zzoBpUMts/JHxKBBMHmypclbutS6sIbk6quhXz+7HB81KO2wyxYeMF36C872k42OHoUvfMGGSIBN1HD55fYcsT6f+emn4Te/sU47jenlWlBgtczFi+HPf274eRqrpMQ69Bw9Cps2WbL12bOPTXoeAhG47z7Yvh3+J3UpGlwG8oDp0l/XrjaWbt++qEvSNG67zTIzB61bB+eeC7162fiGdetqD57vvQdjx1rP0R+mYMTU2LFWi7v77rr3bQqbN9tYjvJye52XZz1/I5oe8bLLrDhTp1paW5ebPGC69JfNYzGfeQZ+/vPqQyJatLBmyJ49bQLVc86x3ieTJlly8srgWVJiHXuGD7deKcXF0KxZ48vVrh3cfju89FK46d7WrYNhw+DUU+3BYaWKisiTm0+bZo+IH0o4OtzlAg+YLv1la8BcvRq++U0bOhOfo/LQIRuYv2SJBcWZM21Q4LRpcNZZNkl6UZGNnVy1Cl55xb7JTzkldeX77ndtcvamrmWqwsqV1gx9zjnWFNy3r/1oCKrMKRuR88+355k/+5kleXG5xwOmS3/ZGDC3boWrroLu3S1v1rGzn9pSmQmjc2dL3fHyyxY8p0+HE0+0Z5Xz5tm+xx0HV1xR61vWW5s2cMcd9r6rVqX23GA/EhYtgs99znrC/uUv9oNg+3bbnobJze+9F8rK7Jmmyz0eMF36y7bZfj780ILboUPw/PPQqVPyx55wgvWEXb4cRo2qan6tHP+QamPGWHBubC2zpISzx42zJtWKCuug1LevZVDZuRMefdR+REycaONA//rX2n9EROT00+2yT59eFddd7vCA6dJffr4Nc8iGGmZFhT2j27DBerSedlrDzlNSYrXLI0fs9aFDTfOMr3VruPNOWLYM/vCHhp9nyhTav/mmzf3aqxdcd50FwKeesh6wt9wCrVqlrtxNqKjIip4JMwi61PKA6TJDtozFvO02+P3vrYrSmBm9p0yp/tyzqZ7xffvbVsufPLn+Q3uOHLF55R57DFGFP/7RatQLF8Ibb1jgbN489WVuQj16WHx/8knYuDHq0rgwecB0mSEbAuaMGfDww9ZR5+abG3eu1avDe8bXqpU1la5caU3BdfnoI3jhBbjpJgu0V19dVRPOy7PeM1dckfLJ0sP0ox9Z5fs//zPqkrgwZe4d63JLpgfMpUttBp0vfQl++tPGny/sZ3yjR0O3bhY4E813u38/zJ1rTa6dO8OXvwy/+pWNCw3WICsqrGoW4fCQVOjc2abqfe659JhB0IXDA6bLDF262HO7+GbITLBhA1x7LZxxhgWVVIyTDFt+vlWr1qyxZtUpUyzozZxpw0E6d4YRI6wWOmKENTvv2WM/dOInG4h4eEiqjB9vrcueZDp3hJpA2rkG69rVaid791pP0Uyxd6/VtvLzbQhFu3ZRl6jhhgyxf1Vtjrjp0+11z57w/e/DV78KF1xw7A+CMJuOQ1ZQAHfd5Ummc0moNUwROoqwQIQyEbaJMKKWfW8VYZcI+0WYJULLZM4jwgUiLBVhnwh7RPi1CF0C24tEOCxCaWDp2XSf2qVE5VjMyy/PnOa88nJ7frdjh+WYLCyMukSN88ADVcHw6FE47zzruLN5Mzz4IFx4YfXac6DpeMXy5WkzPCRVxoyxDtxDhtgj2ZNPthEzyXj6adv/85+/uEHHNfT9wjouK6lqaAvoXNB5oG1BPwe6H7RPgv0Gg+4G7QP6CdAVoPcncx7Qy0GvBS0AbQ06C/TFwLFFoHPqW/aWLVuqq2758uXhvNHq1fZVK6J6yy3hvGcjLF+2THXUKCvz3LlRF6fxdu5Uzc8/9olpq1aqJSVJnyK0eyVEc+aotmhx7GVp3Vq1uFi1oqLmpbjY9svU4+bMSf4aAWUaYpxpyiXMYNkG9BDoqYF1TwUDYWD9M6DTAq8vBd1V3/PEtn0W9EDgtQfMFArtS3DNmgZ/UYdu5049eNJJVtaioqhLkxpjxlSPDC1a1OvHSzYGzMLCYy9JriyFhclfo2wKmGE+wzwVOKLKpsC6vwEXJ9i3D/C7uP1OFKET0KMe5wEYCKyPW3eFCPuAEuC/VZmR6EARbgZuBsjLE1asWFHDW+Su0tLSUK5Lr4ceoisgwNHDhyn5znf4xw9+0OTv2xCfuf12Ou3axcFu3VgzcCBkwX1z7tKltEvwLPLAkiW8nuTnC+teCdP27Rdjd2U85cYbt9Z43BNPnJzRx23frqxY0YiJLDJVWJEZ9KLKWmJg3WjQFQn2fQf0i4HXzWO/bE6u53nOAt0HelFg3RmgXUGbgQ4ALQEdXlf5vYaZWCi1hkTNgS1bpl8ts6JCderUqjKme004ZLlUw6yrBpbpx3XvXvtxQWRRDTPMTj+lQEHcugIgUXa5+H0r/38g2fOI8Gng98A4Vf5YuV6Vt1TZqcoRVV4FHgauqedncWFKNKtNeXnjB/+n0rp1NubwrruqhlFkyfAJV7OpU20Cg6DWrW19th4Hdmvv2FH7sVkprMhM1bPHXoF1xdT8DHNq4PXnqf4Ms8bzgBaCbgX9ThLlugN0fl37eQ0zsVBqDWefnfhnLtSv90FTOHBAdcIE1WbNVDt1Um3e/NjyeS3zY9lYw1S1W7Cw0PqjFRYmf0tWHXe0gcc19P0ad9xdd6m2a6f6qU+pvvNO3ceTRTXMcN8MfRbr4doG9EJq7iX7RdBdsebTT4AuiwuINZ4HtFusSff2GspwZeycAno+6A7QUXWV3QNmYpF9Cf7rX6oXX2y38IMPRlOGhQtVP/lJK8Po0ao33tjojjHZLFsDZmNl4nVZs0a1Y0fVLl1U16+vfd9sCphhz/RzC9AK+D9gLjBGlfUi9IiNh+wBoMqLwAPAcmBbbLm7rvPEtt0E9ATuDo61DBw7DNiMNeEWAz9RZXbTfFzXZDp0gBdfhGuusQnNJ0wIbxag99+38ZVf+YqNXl+1Ch57zMYWZukgfeeCzjvPkteowsCB8PrrUZcoJFFH7ExZvIaZWOS/jisqVMeOVQXVESNUy8ub9r0efli1bVvrhHTffTW+X+TXJQ35NUksk6/LP/5hzbQFBaorVybeh2RqmNBRYYFCmcI2hRFJHLMs1pKTV+e+GVrDdC61mjWDRx6BadPgmWdscvMDifqRNUBJSdVE46+/Dv37w7hxNqPN+vWWJ7JFi9S8l3MZ6NOftgaWLl1g8GB46aUGn+pR4BBwIjASmIFInxr3FhlJBFO7esB0mU/Esmg88YSlnxo0CHbvbvx5p0yxb4PBgy0l1Y4d8OyzNrF4T59N0TmA7t1tzv3evS1r2/z59TyBSBtgKDAJ1VJUVwELgetr2L899ojuh40pd0N4wHTZ4xvfsMTEGzfaEI/Nmxt+rvXr4fHH7bnoG2/A9ddb1pGvf7169g3nctwJJ9hv1X79LDFPcXHVtuMhD5G1gSV+PNipwBFU4yejqamGOQ2YAYQ+qbRnK3HZZcgQWLbMmmYHDIDFiy2P47BhMG8enHTSsfuXlcFbb8Hf/w5vvln1b3CC9+bNoU0b62jknEuoQwdYsgSuugpGjbI/wxUrYC+tW6Bl/Wo5tC2wP27dfqB6ah+RfsCFwDige4qKnjQPmC779O8Pr7xiTamDBsEll1jT6q232l9zMDhu2WLdBsBScPXpAxddBAsWWDoxgMOHrbl30qTqAdc597G2beH55+0x/+zkxx4kN6mNyHHAdGAcqhVRtPR4k6zLTr1723COwkL7Cz561J4/DhtmHYTefhs++1koKoLnnoNNm6C0FNauheOPt1xGQT5rj3NJyc+3NLD1sAlrtu0VWNeX6nOAFwD9gHmI7AJei61/H5GLGljcevEapsteXbtaQuO337aAl5dn4ydnz7a/6ppkcdJj58Lw3nv12Fm1DJH5wD2I3AScDVwJDIjbcz/QNfD6k8Aa4FxgTyOKmzSvYbrsVVJiQ02OHLHXFRWwaBF88EHtxwWSHh+zZEnSY+eaWo8e9T6k2mQ0qK5HpAcipYj0iA2G3PXxUhUkd6N6qKYTp5IHTJe9Ek3a7k2rzjW5miZtr5HqPlSvQrUNqj1QfSa2fjuqbVHdnuCYragKqhUpKnadPGC67OVNq85FYuRImy2ysDDqkqSWB0yXvbxp1bnIjBwJW7cCHDwYcVFSxgOmc845lwQPmM4551wSPGA655xzSfCA6ZxzziXBA6ZzzjmXBNHKeTRdrUTkKPDvqMuRhvKA0MZBZRC/LtX5NUks269LK1XNisqZT42XvL+oam0z7uckEVnr16U6vy7V+TVJzK9L5siKqO+cc841NQ+YzjnnXBI8YCbvsagLkKb8uiTm16U6vyaJ+XXJEN7pxznnnEuC1zCdc865JHjAdM4555LgAdM555xLggfMOohIRxFZICJlIrJNREZEXaZ0ICIrROQjESmNLW9HXaawichYEVkrIuUi8mTctktFZKOIHBSR5SKSZZkBa1bTdRGRk0VEA/dMqYhMirCooRGRliLyy9h3yAER+auIXB7YnrP3SybxgFm3R4FDwInASGCGiPSJtkhpY6yqto0tvaMuTAR2AvcCs4IrReR4YD4wCegIrAXmhV666CS8LgEdAvfNlBDLFaU84D3gYqA9dm/8KvYjItfvl4zhM/3UQkTaAEOBM1W1FFglIguB64E7Iy2ci5yqzgcQkX5A98Cmq4H1qvrr2PYiYK+InKaqG0MvaMhquS45S1XLgKLAqudF5F3gXKATOXy/ZBKvYdbuVOCIqm4KrPsb4DVMc5+I7BWRV0RkUNSFSSN9sPsE+PjL8h38vqm0TUTeF5EnYrWrnCMiJ2LfL+vx+yVjeMCsXVtgf9y6/UC7CMqSbu4AegLdsIHXi0TklGiLlDb8vklsL3AeUIjVrNoBT0daogiISHPsc8+O1SD9fskQHjBrVwoUxK0rAA5EUJa0oqp/VtUDqlquqrOBV4AhUZcrTfh9k4CqlqrqWlWtUNXdwFjgCyISf62ylogcBzyF9YsYG1vt90uG8IBZu01Anoj0CqzrizWjuGMpIFEXIk2sx+4T4ONn4afg9028ymnGcuK+EREBfol1IByqqodjm/x+yRAeMGsRe5YwH7hHRNqIyIXAldgvxJwlIh1EZLCI5ItInoiMBAYCL0VdtjDFPns+0AxoVnk9gAXAmSIyNLZ9MvBGrnTgqOm6iEh/EektIseJSCfgEWCFqsY3R2arGcDpwBWqGsytm9P3S0ZRVV9qWbBu3r8FyoDtwIioyxT1AnQGXsOajD4A/gT8R9TliuA6FGG1pOBSFNt2GbARSzq+Ajg56vJGfV2A4cC7sb+lEqAYOCnq8oZ0TQpj1+EjrAm2chmZ6/dLJi0++bpzzjmXBG+Sdc4555LgAdM555xLggdM55xzLgkeMJ1zzrkkeMB0zjnnkuAB0znnnEuCB0znclQsN+U1UZfDuUzhAdO5CIjIk7GAFb/8KeqyOecS83yYzkXnZSy3atChKArinKub1zCdi065qu6KW/bBx82lY0XkBRE5KCLbROS64MEi8hkReVlE/i0i+2K11vZx+4wSkTdFpFxEdovIk3Fl6CgivxaRMhHZEv8ezrkqHjCdS18/BhYCZ2M5R4tFpB+AiLQGXsTmIz0f+CowAJhVebCIfBuYCTwBnIWlX4vPgDEZ+B2WLWMeMEtECpvuIzmXuXwuWeciEKvpXYdNxh30qKreISIKPK6qowPHvAzsUtXrRGQ08CDQXVUPxLYPApYDvVR1s4i8D8xR1TtrKIMC96vqxNjrPOBD4GZVnZPCj+tcVvBnmM5FZyVwc9y6DwL/Xx23bTXwpdj/T8dSQAWTDL8KHAXOEJEPgW7A/9ZRhjcq/6OqFSKyBzghueI7l1s8YDoXnYOqurmBxwpVCZjj1SeZ9+G414o/qnEuIf/DcC59XZDg9YbY/98C+opIu8D2Adjf9AZV3Q3sAC5t8lI6lyO8hulcdFqKyElx646o6p7Y/68WkdewhMLXYMGvf2zb01inoGIRmQx8AuvgMz9Qa50K/FxEdgMvAK2BS1X1Z031gZzLZh4wnYvOZUBJ3LodQPfY/4uAocAjwB7gRlV9DUBVD4rIYOC/gDVY56HfAeMqT6SqM0TkEDAB+AmwD1jcVB/GuWznvWSdS0OxHqzXqupvoi6Lc874M0znnHMuCR4wnXPOuSR4k6xzzjmXBK9hOuecc0nwgOmcc84lwQOmc845lwQPmM4551wSPGA655xzSfh/rqZA8UVQ8jkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.epoch, history.history[\"lr\"], \"bo-\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Learning Rate\", color='b')\n",
    "plt.tick_params('y', colors='b')\n",
    "plt.gca().set_xlim(0, n_epochs - 1)\n",
    "plt.grid(True)\n",
    "\n",
    "ax2 = plt.gca().twinx()\n",
    "ax2.plot(history.epoch, history.history[\"val_loss\"], \"r^-\")\n",
    "ax2.set_ylabel('Validation Loss', color='r')\n",
    "ax2.tick_params('y', colors='r')\n",
    "\n",
    "plt.title(\"Reduce LR on Plateau\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf.keras schedulers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 2s 44us/sample - loss: 0.4872 - accuracy: 0.8296 - val_loss: 0.4141 - val_accuracy: 0.8548\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3829 - accuracy: 0.8643 - val_loss: 0.3773 - val_accuracy: 0.8704\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3495 - accuracy: 0.8763 - val_loss: 0.3696 - val_accuracy: 0.8730\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3274 - accuracy: 0.8831 - val_loss: 0.3545 - val_accuracy: 0.8760\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.3102 - accuracy: 0.8899 - val_loss: 0.3460 - val_accuracy: 0.8784\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2971 - accuracy: 0.8945 - val_loss: 0.3415 - val_accuracy: 0.8796\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2858 - accuracy: 0.8985 - val_loss: 0.3353 - val_accuracy: 0.8834\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2767 - accuracy: 0.9018 - val_loss: 0.3321 - val_accuracy: 0.8854\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2685 - accuracy: 0.9043 - val_loss: 0.3281 - val_accuracy: 0.8862\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2612 - accuracy: 0.9075 - val_loss: 0.3304 - val_accuracy: 0.8832\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2554 - accuracy: 0.9097 - val_loss: 0.3261 - val_accuracy: 0.8868\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2502 - accuracy: 0.9115 - val_loss: 0.3246 - val_accuracy: 0.8876\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2456 - accuracy: 0.9133 - val_loss: 0.3243 - val_accuracy: 0.8870\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2416 - accuracy: 0.9141 - val_loss: 0.3238 - val_accuracy: 0.8862\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2380 - accuracy: 0.9170 - val_loss: 0.3197 - val_accuracy: 0.8876\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2346 - accuracy: 0.9169 - val_loss: 0.3207 - val_accuracy: 0.8866\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2321 - accuracy: 0.9186 - val_loss: 0.3182 - val_accuracy: 0.8878\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2291 - accuracy: 0.9191 - val_loss: 0.3206 - val_accuracy: 0.8884\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2271 - accuracy: 0.9201 - val_loss: 0.3194 - val_accuracy: 0.8876\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2252 - accuracy: 0.9215 - val_loss: 0.3178 - val_accuracy: 0.8880\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2234 - accuracy: 0.9218 - val_loss: 0.3171 - val_accuracy: 0.8904\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 2s 41us/sample - loss: 0.2218 - accuracy: 0.9230 - val_loss: 0.3171 - val_accuracy: 0.8884\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 2s 40us/sample - loss: 0.2204 - accuracy: 0.9227 - val_loss: 0.3168 - val_accuracy: 0.8882\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2191 - accuracy: 0.9240 - val_loss: 0.3173 - val_accuracy: 0.8900\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 2s 39us/sample - loss: 0.2182 - accuracy: 0.9239 - val_loss: 0.3166 - val_accuracy: 0.8892\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)\n",
    "learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)\n",
    "optimizer = keras.optimizers.SGD(learning_rate)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 25\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For piecewise constant scheduling, try this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(\n",
    "    boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],\n",
    "    values=[0.01, 0.005, 0.001])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1Cycle scheduling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend\n",
    "\n",
    "class ExponentialLearningRate(keras.callbacks.Callback):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "        self.rates = []\n",
    "        self.losses = []\n",
    "    def on_batch_end(self, batch, logs):\n",
    "        self.rates.append(K.get_value(self.model.optimizer.lr))\n",
    "        self.losses.append(logs[\"loss\"])\n",
    "        K.set_value(self.model.optimizer.lr, self.model.optimizer.lr * self.factor)\n",
    "\n",
    "def find_learning_rate(model, X, y, epochs=1, batch_size=32, min_rate=10**-5, max_rate=10):\n",
    "    init_weights = model.get_weights()\n",
    "    iterations = len(X) // batch_size * epochs\n",
    "    factor = np.exp(np.log(max_rate / min_rate) / iterations)\n",
    "    init_lr = K.get_value(model.optimizer.lr)\n",
    "    K.set_value(model.optimizer.lr, min_rate)\n",
    "    exp_lr = ExponentialLearningRate(factor)\n",
    "    history = model.fit(X, y, epochs=epochs, batch_size=batch_size,\n",
    "                        callbacks=[exp_lr])\n",
    "    K.set_value(model.optimizer.lr, init_lr)\n",
    "    model.set_weights(init_weights)\n",
    "    return exp_lr.rates, exp_lr.losses\n",
    "\n",
    "def plot_lr_vs_loss(rates, losses):\n",
    "    plt.plot(rates, losses)\n",
    "    plt.gca().set_xscale('log')\n",
    "    plt.hlines(min(losses), min(rates), max(rates))\n",
    "    plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 2])\n",
    "    plt.xlabel(\"Learning rate\")\n",
    "    plt.ylabel(\"Loss\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples\n",
      "55000/55000 [==============================] - 1s 22us/sample - loss: nan - accuracy: 0.3879\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAERCAYAAABcuFHLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO29ebwkZXn3/b16Ofuc2WcYhhmGVRQQxAFJEBGNgPK4L48LbjExbtHo6/PEGBfcInHhSXjfqMGIPO64G0WNuGtcYBBRAUFBBoZhhmHWs/dS9/tH1V19V3V1n+5zuvtU9Vzfz6c/011VXX1X95n7V9d6izEGRVEURVksuaUegKIoitIfqKAoiqIoHUEFRVEURekIKiiKoihKR1BBURRFUTqCCoqiKIrSEQpLPYBOsmbNGrNly5alHobSJlXPcOv9h8LXx64ZRUQYKubIiSzhyJRucnCmzD37pjl+3RjDxXzL77vt/kOMDxfZuGK4i6M7vLjxxhsfNMasXex5+kpQtmzZwrZt25Z6GEqbzJSqPPRt3w5ff/AFZ/CqT/+Ks09ax1UvOXMJR6Z0kzdc82u+fNN9vPPZp/HMRx7V8vse+a7ruOiUI3jP00/t4ugOL0RkeyfOoy4vZckp5KNWyNRcBYAbt+9fiuEoPWKu6gEwWGxvGjKglmtKUUFRlpxCLjo5TJeqAJSDCUfpT+bK/u87kG9vGvKMQfUknaigKEuOxGaHycBCUUHpb+Yq/o3DQKFNC8WohZJWVFCU1HDsmlGg5vIqVw0zpWooMEp/UaoELq9C6wF58C0UJZ2ooCip4Oa3XcDVLz0LqAkKwEPf9m2e+aGfLdWwlC5SCizQeAxtXgzq8kopfZXlpWSX5SNFKp4/wUzELJLbd08sxZCULmMtlHYNDg3Kpxe1UJTUUAx86VMxQTl27ehSDEfpMnOhoLSnKJ4xqJykExUUJTXYbJ+puWpk+84DM2x507X84PYHlmJYSpewForXroViIJdTSUkjKihKarCCEg/CzwbppZ/42d29HpLSRWyWl0EtlH5BBUVJDbmcUMhJncvLEk8vVrLNYmIoqijpRAVFSRXFfK6xoPR4LEp3mVugoKB1KKlFBUVJFQOFXMO6E3cOuWfvNP/xk7t6NCqlG9RiKOry6hc0bVhJFcV8jv3TpQZ7a9PI8//jF+zYP8Ozt25i+XCxN4NTOkoliMYvwEBRCyWlqIWipIrBQo5qg7Qfdw45OFMG2k85VdLHgiwU1ZNUooKipIpirGp6dKDWliNpDqm0m3OqpI8FpA2rnqQTFRQlVcQbBV54yhHh88hdaTAJaQPJ7NNO2rC1SDXjL52ooCipoui0Mr/zn57EGZtXhq/FuS+1U1ClqhZK1vHmuSf48q92cPsuv/2O9Y6pnqQTDcorqcJaKPmckM9JxGJxJxF7p1pSCyXzzHdL8IbP3wzA3ZddHB6rQfl0ohaKkiqshWKr5gcbCIpFXV7ZxE2maCcob49VOUknKihKqrACYoPzrgtMEqYRdXllE9eybCfJS11e6UYFRUkVoYUSLLrkLg/r3snaZ+ryyia2Pxu0l/rtaVA+1aigKKliIHR5BRaK4/JKcm+phZJN3N9yIb+g6kk6UUFRUoUVEBuMdy2UkiMeRtOGM423wBiKLWgdG9R8ojSigqKkChs7CQWlULsVLVVq66TY2gV1eWUTV0Oa6UncHXbnnkkAjl0z1o1hKYtEBUVJFSuGBwAo5KyFUquULye4t9TllU1atVDibXju3DMF6CqeaUUFRUkVW9aMADXXRtGxUMoJmUHq8somrXbMqcbE5q49k4wM5DlifKgLo1IWiwqKkiq2rPbvPHcenAFiMZRKvXiooGQTz2vNQonvumvPFMesGdUlgFOKCoqSKqyg2InErZT//a4J3vWNW/39wbYkN5iSTqqe4SM/upOZUrXlGErc5bX70CxHrhju0giVxdJTQRGR14jINhGZE5GrW3zP90XEiIimdRwGHLki6spwLRSAj/30T1Qcq0QtlOzw1Zvu47Jv/Z5/+e4dEVdWM0GJWy9Vz1BQ6yS19HqS3gm8G7gQmPc2Q0RegPYbO6woBAIyXAwKGwv19zy7J+ZCE6WigpIZZoMsvUOzlZaD8vHGkZ4x6u5KMT2drI0xXwYQka3AUc2OFZHlwNuBFwE/7/7olLTwo//1WIaDdVCK+XpB2bFv2kkbVpdXVrCtc4wxkXTgZr9gPChvdD35VJPmu/9/Aj4M7Gp2kIi8HHg5wObNm3swLKXbHL26lhKaZKHcu38mfK4WSnawOmBMNMurWeuVOpeXMaiBkl5SGZQPLJhzgP93vmONMVcaY7YaY7auXbu2+4NTekqSv3zH/mlNG84g9pc0mGhftmYxlFhQ3jOGvFooqSV1giIiOeBDwOuMMZWlHo+ytCQ1Adyxf8ZpDqkur6xgXVXGRGMjzWpS4vs8TxtDppnUCQowDmwFrhGRXcANwfYdInLu0g1LSQu7D82G6aTq8kovb/7Kb/nijTtqGwId8Ey8c3STSnlTb6Goyyu99DptuCAiQ0AeyIvIUEI68EHgSOD04PGkYPsjgV/2bLBKavirRx/DplW1pMC9k6Xwubq80stnfnkPb/zCzeFr1+Xl6kRTCyXJ5aWKklp6baG8BZgB3gRcEjx/i4hsFpFJEdlsfHbZB7AneO9uY0ypwXmVPuYt/+NhnH3M6vD13qm58LkWNmaHMDsrZqE0C6LEg/KeUZdXmul12vClwKUNdie2DzXG3I2u+HnYM132axiWDRXUQskojp7E6lAavydeKe956vJKM2mMoShKHTMlX1COGB+i4kwyKijZoZY2bNpIG649r3pGXV4pRwVFyQSveuxxrBwpcv5J6yLbtX19drCFjZ6h5cJG15IpVz2qntHCxhSjgqJkgq1bVnHT2y5g08poxx5dYCs7RF1ete3N04ajgmKMLv+bZlRQlEwxPlyMvFYLJZ3Es7OgFkz3TLywsbUFtspVo4WNKUcFRckU40NRQdEYSjopx7s6Qi2YHq9DaZo2XHteqXp+6xWNoaQWFRQlU4wPRxMT1eWVTuLZWeA0h4zVoTQrbIy4vDwTpA13bpxKZ1FBUTLFsiF1eWWBpPqgaHPIFtOGXUGpeBh1eaUaFRQlU7gur0cevVJdXikl2ULx8erShhufx42vVDzN8ko7KihKplgeBOXXjw8yNlig3Oz2VlkyknqsNbZQmgXla89LFV+INIaSXlRQlEwxPJDnyhc+km++9lyKeaFUUQsljSQJvdUNQ/PMLpdqQhGr6kl6SfMCW4qSyAUnHwHAYCFPKVhWVkkX1YQYit1ijIm2r29iZbrCM1exgqKKklbUQlEyy2Axx2xZLZQ0kpQ2bF1bcZdXq0sAzwU3D9p6Jb2ooCiZZbCQDyeZVrjpnv0cnC53cUSKJSko77q8opXyrfXysu5NNVDSiwqKklmGijnm2rBQnv6hn3Hhv/y4iyNSLEnZdzULxUR7ebW4Hoq6vNKPCoqSWYaKeWZbtFDsBLbr0Gw3h6QENKsP8kzUldXU5eUluLxUUFKLCoqSWYYKecpVk+heieO2vN++d6qbw1KIft+W0EKBNtrXO4JSVpdX2lFBUTLLUNH/850tz2+luKLz1Zt28sPbH+jauJTkOpQwhtKOy8vZ+cPb/cVb1eWVXlRQlMwyWGhdUFyf/v/57h285OM3dG1cSnJQ3gsFpfXCRvues45Zxbdv2QVolleaUUFRMstQMQ/UgrXN0J5fvSW5sNG6vKJ1KK3EUM7YvDLcpnqSXlRQlMxiBaWRhfKVm3bwq3v2A8k+faV7NHd5tWOh+PsG8jUVEXV5pRatlFcySy2GkmyhvP6amwG4+7KLqSQU2indI0nAbZt6z0Tb1zczUaygFPO1e191eaUXtVCUzDJY8C2Ue/ZN8eav/DZS5BjPHFKXV29xv2/7WywkhmINnWKhNlWpnqQXFRQlswwGFsq7vnEbn/nlPXznlt3hvnhcRV1evcW1CO1X3zhtuPF5kiwUzfJKLyooSmaxMRTL/ulS+PzQTLTFSpJPX+keroVixcVNG251gS1bKe/GUFRQ0osKipJZhgKXl42lHHD6dB2MCUrSCoKttlBX2se1UGymVpjlZaLffbMlgG1F/YDr8tJZK7XoT6NkFuvysu6siIUyGxWUpLqIJJFROoPrYrTPw/b1tOPy8v9Vl1c2UEFRMot1ee2b9IVk18Fan65DM5XIsUnt1DXzq3u4Li+7NornWCqR9vXN0oY9jaFkiZ4Kioi8RkS2iciciFzd5LgXi8iNInJIRHaIyPtERFOclQhDgRtkYs4Xj52uoAQWinWHqYXSW5pZKJ6pWR75nDQtbNSgfLbotYWyE3g3cNU8x40AfwesAR4FPB54Y3eHpmSNeFB+54GZ8LkNyg8HxyS1U9dAffdwv1sr5mHaMDWrJC8yT9qwjaHURCSvfpXU0tO7fmPMlwFEZCtwVJPjPuy8vE9EPg2c3+XhKRljsBCdWSZm64PyVnSsC2ZkIM90ya9X0VTi7hG1UGyWV73LK5+TltKGB/K1mwetlE8vWdH6xwC3LPUglHRRyOcoOFVus2WPLW+6liu+9wcOzfpuMLvX3umODtbuoVRQukckhhLGTgj/dV1eTdOGw6C8pg1ngdQLioi8FNgKfKDB/pcHcZlte/bs6e3glCUn7vYCuPy6O0KXVylwvViX1zJXUNTl1TXchIdaDMVpDulYKM16r1gxcivl1eWVXlL904jI04DLgCcaYx5MOsYYc6UxZqsxZuvatWt7O0BlyUkSFICf3bkXqK1Dbie1saGaoGhQvnu41l9dDMXUrJV8TmiWbGdCl1dtqlKXV3pJraCIyEXAR4EnG2N+u9TjUdLJEcsHE7ffs2+a0zetCC2UUFAiLi+1ULqFa/1Z95cVEc/U0oH9LK8Wenlpllcm6HXacEFEhoA8kBeRoaR0YBF5HPBp4JnGmOt7OUYlW2xcMQzAypFiZPvx68Z49PFrQivETnARQVELpWuUE2IoXiQo7+/zs7wan8dWyhecGIquKZ9eem2hvAWYAd4EXBI8f4uIbBaRSRHZHBz3VmA58M1g+6SIfKvHY1UywFErRwBYNToQ2f6BZ59GMZ+j6vlrzlvxcAUlKZVY6Qxu3U85luUFtJzlZYwhJ0SSL7TbcHrpddrwpcClDXaPOcdpirDSEhuWDwFRv/pn//psTt+0gp8HcZRy1UuMoWiWV/dI7uVF+K8xBhEQaV4pX/UMOZHIGigaQ0kvqY2hKEorrBzxLRN31caRAT9QbxsKzlW8cIJTC6U3RLoNV2NBeQxV4wuFSPMlgD0DuVxUUHSBrfSigqJkmtVjvqC4gdrRwUBQAr97ueqFk9rZx65mWWClaAyleyRledVWbAyEQvzfrWkvr8DllVeXVyZQQVEyzTnHr+Gl52zhvc84Ndw2POALhrVQSo6FcsbRK/nkyx4FaJZXN3GtP/s9uwttecYgIgjN10Opeoa8CIWc275eFSWtqKAomaaYz/H2J5/McWvDEBwjQW2KTTUtV70w66iQkzDA26gOxRjDe791G398YKKbQ+9rqgkWig2ieJ6/pnxexLdQmpzHM8Z3eYlWymcBFRSlLxh2ChyHYzGUUsWrVVznc2EKaiOX1wMTc/z7j+7ixVfd0M0h9zVJ3YbtpqoxeJ7vykKarynv2aB8Xl1eWUAFRekLhgZqf8q2aaS1UEpVL6xD8VNQ7cJczV1eJQ3aLxjPM2H/rboYimdjKL6F0sxE8YwfP1ELJRuooCh9QVJrDtdCKQcTnIiEE918QXldIXjhVDwT/iZxC6UWQyGIoTRfArg+KK+CklZUUJS+IKk2YSCMofjFjdYyKeSbWyi1gLIqykLxjGEwcENWw8JGgtfGL1jMBTGUZhZKQh2KrimfXvSnUfqWiIVS9cJgfHGeoHy895TSPpWqCV2Pte+z1oLFurxkvhiKMeRzEombaOuV9KKCovQtbpZXpWrCYHxooTSIkYQLQvVgjP1K1TiCElsCuOqZsL5E5snyqnpWeLRSPguooCh9i3V5+ZXyhnzo8gpiKA0KIMqxO2qlfaqeCeuB7BICtsOwLyiEdSjNvmffNRbdplle6UUFRelb7Drk5SDLywbji7labCWJ0EXTgzH2K1XPMBqkb9u2ONH1UKyF0ty1aFu0uGjrlfTS0+aQitJNvvuG8yLZXnYdcluHUnN52SyvepfXe669la/ffD+gMZTFUPUM48P+kgKzZetCDCwUY11eflC+aZZXUCnvolle6UUFRekbjl83Fnltg/ITs2V2HZqtZXnZoHzM5bV/qsRHf/Kn8LW6vBaOTRsu5oXZim+huFleblC+2bdsTH2rFdWT9KIuL6VvWTM2wEAhx6Vfv5Wf3bk39L1LkIZaqXpMzVXCFivX3bY78n6Vk4XjeYZCThgq5EOXVzTLy7avb542XLUV9Q7q8kovKihK31LI5zhxfc1qeWBirrYvJ1Q8w0uvvoG/uPzHGGM4NFOOnmABinLrzkNMzlUWOuS+oeJ55HPCYDEfurw8x0Ix1kJh/rThuItLXV7pRQVF6WvcppETs7WJvpjPUa56XP+nfYCfCRYXgnb1xPMMz/jwf/PpX2xf8HjTynW37mauUp3/wADbMmWomKtZKE77emt5zGdsqKBkCxUUpa9pNPkU8hLpiDtdqjJdik6Y7cZQSlWP2bLHodny/AdniOv/tI+//sQ23v/t21t+j7VQhov5uiwvwFlgq4WgfC4uKO2NX+kdKihKX/O3jzues7asqtteyOUiacNTcxWmFmmh2JYtjdKRs8reSd9VuGP/TMvv8TxrobgxlNr+StVDAgulaesVUy8gaqGkl0UJiogMi8hfiMjRnRqQonSSY9eO8flX/BlAWBcBUMxLJG14ulStF5Q2dcEKiS3k6xdsAajbQn7+93jkxbq8bC+v2hf6X7fsDmIozS0Ua+m46AJb6aWttGERuRq43hjzIREZAK4HTgZKIvJ0Y8y3ujBGRVk0v7n0AtxpqJCXyKqCU6UKU3GXV5s2ij1fv60EaV2DhTYm8qrnC9BQMR/GpuK6EaYNN/maSxWPIWetG/99LQ9D6THtWigXAr8Inj8FWAYcAVwaPBQllYwPFVk2VAxfrxgeYP90LdYx04aFMlOq8t5v3sZMTICsZVKu9I/Lq+qZ8LraacpYDSyUwYKb5RX9XqSFSvm5ihf2BLOoyyu9tFvYuBJ4IHh+EfAlY8wDIvI54B87OjJF6SJHLB/irj2T4eupuSQLJcq+qRIPTs5x3a27+fcf38X4cJFXn398uN+6hsp9tDDXYz/wA+7d58dO2qn/qATB9KFijrkwyyuKrZSv0vj7mit7YYGqRetQ0ku7Fsou4BQRyeNbK98Nto8B/ZXaovQ1G5YPceeeqfB1UgwlPgM+4fIfccH/+XEYZI4v0GWFpJ9WerRiAu1N5GFhYyTLK/p95XIE7esbn2euUmWwEHV5qYGSXtoVlKuAa4DfAVXge8H2RwG/7+C4FKWrbFg+HHk9XaoyPVdhbLBmtMdjKHunSkBtYozPr6HLq48ExaWdYLhrocxWagtsbVo1zOsef0L4WpCm6dnq8soWbbm8jDHvFJFbgM3AF4wxpWBXBfjnTg9OUbrFhuVDkdfTQVB+xUhx3kp3qxfxCTYMyvdZ2rClnaC8XRgr3nolJxIG2StVM28vr1LFY7AYc3mpoKSWtptDGmO+lLDt/3ZmOIrSG46ICcrUnO/y2rxqJKy3aHTjbO+o4/NamDbcrxZKGxN5zULxBcUEqzQKhBZH2fOCwsbG5/EtFHV5ZYW2XF4i8hwRucB5/TYR2SEi/yUiG1p4/2tEZJuIzAUpyM2Ofb2I7BKRgyJylYgMtjNWRWnGkY7Lq5ATDsyUqHiGFSO1TLCq8dc+PzRbjtSWhHUZkmyh9KvLq9UYihf06rIuL8/4YmsgYqGUq57vNmzq8qrWubx0xcb00m4M5VL7RETOAN4MXAEUgQ+28P6dwLvxYzENEZELgTcBjwe2AMcC72hzrIrSkI0rh3nh2Udz7WsfzbKhQtg4csXIQHiMCXpOPfzS73DJx34Zbrc9reJ37KU+rZS3tOryqpqa4FrxmK1U/diTOBZKxQTNIRucxzOUq6Yuy0tJL+26vI4GbEOfpwNfNca8T0S+A/zXfG82xnwZQES2Akc1OfTFwMeMMbcEx78L+DS+yCjKosnnhHc97RQARgYK7AkEZaVjoUBNHGwTSagtGBWPodjYSb9aKK0G5atOZX0oKOUqBB2GbUykVPXIiTQsILVWYdzlpaSXdqV/Fr+YEXzrwaYNH3S2d4KTgZud1zcD60VkdQc/Q1EAGB3M82DQr2r5cFRQkuIhbpD5pnv2c/9BP+bSr728LC1bKF69hTJX9vw1UKgJRLni9/NyGwvMlqthc82aoKiFkhXa/aV+AnxQRN4KbAW+GWw/Ebi3g+Mawxcpi31eJ1oi8vIgLrNtz549HRyCcrgwPFBgf5ASPD4Ut1DqBWWuUqs3efqHfsazPvzzyLH9aqG0ShhjCmIoADPlargGihWIUtUPyrvy+/QP/YyHX/odoOZajGd5Keml3V/qNUAJeBbwCmPMzmD7E2nB5dUGk8C489o+n4gfaIy50hiz1Rizde3atR0cgnK4MFLMczBYXGt0MOoFttZI0rY7dvl/jvcd8C2Ufq9DqTZLx3LwnN5fVqAnZsvhKo1uUN6PydfOe9v9h8Lnc+ryyhzt1qHsAJ6csP3vOjYin1uA04DPB69PA3YbY/Z2+HMUJcxEAhgbiv6XiPfrgpqg3HTvAQCOGPdTkK2rq9xn3YYt1RbbL7sWis2a2z9VxuBnaFkLxa4r3+i01kLRoHx2aLsOBUBEHgc8DL8m6VZjzA9afF8h+Mw8kBeRIaBijIlXkn0CuFpEPg3cD7wFuHohY1WU+XDvgJfFLJS/+sS2uuMn5/yJbvveaQDWL7eCYl1h/RlD8Vq1UGyWVy7HimE/a+7ATBljYyiOC8tvvZJ8Xpv8oDGU7NBu+/qNwFeAR+KnAAMcKSLbgKc7LrBGvAV4u/P6EuAdInIVcCvwMGPMPcaYb4vI+4AfAMPAl2LvU5SO4U5wcQvFiobL5Fy0bd1sYMX0a/t6S6uevJqFAssDC+XAdMm3SHJRAc/FYiguNiFCBSU7tPtLXYHfw+t4Y8wmY8wm4IRg2xXzvdkYc6kxRmKPSwMRGTPG3OMce7kxZr0xZtwY81JjzFybY1WUlnAnrNGB+e+xJmejBrVt1dLvLq9mC2FFjvNqFsqywQI5gYOBhZKTWqAegAYWijGGubLGULJGu4LyBODVxpg/2Q3GmLuA1wb7FCVzuBPW2GALghLr9TURpLn2e9pwq0F510LJ5YTlw0UOTJed1iu171sgbOblutSqnqnL8nrN+cdz/LqxRV+H0j06ZUv25y2ZcljgWiju3fNTTz8y8fi4YEzOVTDGRNrXxzvoPvfKn/Oiq67v1JCXhFaD8lXHQgG/+8CBmfqgPERdXlOlmlCXqybM8hrI+8e/8cKH8N03nLfIq1C6SbuC8j3gChHZZDeIyGbgX4Hvd3JgitIr3BiKm1F0znFrWnq/Z/w6C7cIshK7m//FXfv48R3ZrpNqNShfjfU6WzFS5MB0yQ/KS1TA3aD8IceVWKp6oaAMaR1KZmj3l3otMALcJSLbReRu4E78wPnfdnhsitITXBeM+7yQb70J4eRsJbL0bz+2sG/d5RUsGRxU1q8IXF62sLGQj1kowWkPzdSSHcpVT1uvZJB261DuBc4QkScAJ+G7QG8F/ghcDjyn4yNUlC7j3jG7FkoxH73fGhnIMx2rSxkdyDNVqjIxV4kUNJaqHsP010TYqsvLJrnZVi0rRgb4455Jlg0ViEu03xwysFBighLGUDTLKzMsqA7FGHMdcJ19LSKnAc/s1KAUpZfYCauQk0iL9rigjA0W6gRlzbJBpvZOMzEbFZR+rJZv1eVVZ6GMFHngkJ+kuWE8ulKmuBaK4/IqVzTLK4uo9CuHPYNBK5B4RfZAIXo/bVuGuKwZ85fpmZytRIL1/SgorXrxaoWN/vf3vLM2M5DP+evTx0wUkVrrFddCKVWrtaC8WiiZYUEWiqL0E9ZCsRPXUSuHOXXj8joLJcn1snrUrwSfnCtHLZRKf8RQclJbr6RlC6UaFZQT1y/jYUeO88s/7cMagM89cxOnbFzOb3YcCLO8DrqCUjHaeiWDqKAohz3WpWIF46d//zgAfnFXtHVc0sS2dplvoRyaibm8+qRavpDPhcHxVgsbqzELBWruQwlMlMue+XAA/v6LB8PzHpguhceXqx6e52eFtbpSpLL0tCQoIvKf8xwyPs9+RUktcQvF0oqFcuQKPyawb7rUlzGUQk6w03yrWV5Vr15QbMZcLvYV5nK1FYAPxILy1aCyXskOrVoo83X53Qv8aZ5jFCWV2DqUePB3IB+PqSS7vAYLOfZNlSJNIaMpxNkVF9coadVCqSQJSqAk9QIhoUtt/7QbQ/GCbsTtj1lZOloSFGPMS7s9EEVZKqyQxAWkGAvKJ7lehop51owN8uDkXKSHl1vkOJfh3l5uqnC766HkxXV5JSuDOL1Xoi4vg6cWSubQaJdy2NOqy0vqqij8Ku7VYwPsnfRdXnb+i9SkOIISb8mSdowxPPm0Izl+3VjLWV6JFko+2UJxg/4HZ8phkkO54oWFkEp2UEFRDntqLq+YiysuKAlz22Axz6rRAfZOzVGqeowF3YrLDSyUVu/y00LVMxy9aoTxoULr66EkCIq1UOLfoSChyO6fLoVJDuWqR9Uz6vLKGCooymFP6PKaz0JxZsPxYN2UwUKO1aOD7JssMTFbYdOqEQD2TtbcN66FkrVOxP4aJn7BZ7vdhguuoDSIoeQkbDbMgelyKCh+DEVdXllDBUU57LGWSTwoH/f7u3fLy4K10v0YygAPTpWYmC1z0oZlAOzYX1uYq1StOs+zE0+xlkZOfCFoufVKQtqwzfKqC8mL4HmGStVjYrbiWCjGd3mpiZIpVFCUw56aoET/OxRiFsqJ65eFz62bZqiQZ/XYAKWKx94p32WzZmzQrwoPsEvZQtRaSbH8JHEAACAASURBVDuhMIhvoSy0sBGcOpSYxSGBhWKLGtctqy2n7Fsoi7oEpceooCiHPQ1brziC8pFLzuB/XfiQ8LW9Wx8q5lg16t9VGwPjQ0U2rRpmxwHXQslmfYq9xtDltYjCRuv+So6h1FKG62MoqihZQgVFOexpZKG4Lq/HnLiWYj5HPiccvXokXF99qOhbKJaxwQJHrRxhx/6ahTJXbk1Qtt29j4v+5cfMxBpQLhVWP3Ii5KR1CyW5sNHGUKLH5oJeXrNl/5ptbKpU8etQ4haNkm5UUJTDnkJOyEm9hZJUmHfbOy/iutefF7qDBgo51gQWCsCyoQJHrRzmvv0z4cTaqoVy264Jfr9rgt2HZhd/UR2g6sRQ2rFQktKGwywv6l1enql91vCAby36MRRDXmeoTKE/l3LYIyKsGh1g5chA3XaLnRAHCjkGCjk2rRwOX7sWyrKhImvGBql4hsmgHftc2QnKN2kaaSvq3aVwlxI3uJ4ToVVvXVJhoxVkQ/T6/SWATShWw0UrKJrllUW0OaSiAF965Z+zemyw4f646+VjLzmTG7fvZ3yoGIm1LBsqhOJjG0S2aqHYu/SpuXS4vGx/SxEhn2tnPRSbNlz7XmyWV93lxyyUgUIOERtD0cLGrKEWiqIAR68eZWyw9furNWODXHjyEYAfR7HvHRsshBOpzXZqNYZia1RSZ6G06fKq2gW28vUWXjXWhdlfAthEMsOK+RylqheuQa9kBxUURekA1u01PlQM78ateLgWivv8izfuYN9UrQDSTrZTc+kQFDfLq52g/P7pMoWcMDpQq+uxIhuv6yzkhIpnQgulkMsxkM9Rrvi9vLR1fbZQQVGUDmB7UC0bKoQuMCsobgzFWiHb907xxi/czOs+d1O4z7qKptPi8rKCIu2lDT84McfqsYHEGFRclAbyOYwhXEzLt1AkiKGoyytrqKAoSgew8ZexoUJooVSSsrwq1grxJ9A9E3PhvkqbLq9uN5q03qmcCHlpvfXK3qlSuDSyxaYNV2IuL5tZNx2kShcCl5cNyqueZAsVFEXpAGvGBhgu5inmc6F7J3R5VepjKEm1GpUwKN+aoDz13/6bM9/z3cUPvgG1LC/f7dWqy+vBybl6QclZCyV6rBUUW3vjxlA0yyt7aJaXonSAFzzqaE7duAKouXfCoHzCOik2A8xt71JLG27N5fWbHQcXOermWNETa6G04fI6ft1YZFuxgYVit8+Ua4IyUMgxV/HwvGjqsZJ+emqhiMgqEfmKiEyJyHYReX6D40RE3i0i94nIQRH5oYic3MuxKko7nLJxOc9/1Gag3r0zXXLrUKJWS6GJhbJ971SXR90cqx95EXK51upQjDE8OFVibZ3LK8jyimlSkstr44phtu+dUpdXBum1y+vfgBKwHngB8OEGQvFs4C+Bc4FVwM+BT/ZqkIqyGKyFYosY79g9EbqAbFB+1rkjr1Q9v+OuV4uv/OfNOznv/T/kx3fsSfyM2XL3A/e1LC/f7dXKEsATcxVKFS/B5RVkecUslMHQ5eWLaD4nnLJxObfvmmC24qnLK2P0TFBEZBR4JvBWY8ykMeanwH8CL0w4/Bjgp8aYu4wxVeBTwMN6NVZFWQyue8cYw607D3H6Jt8dZmMotgNxMS889gM/5In/+pPQxTRdqnDzvQcAuH3XROJn3H+w++1ZIlleLQbl7Towa5ZFuw4UGxQ22oy4aSeGcsrGccpVw+/vP0ROo7yZopc/14lA1Rhzh7PtZiDJQvkccLyInCgiReDFwLeTTioiLxeRbSKybc+e5Ls5Rekl1o31wo9dz7M+8nP2TpU4fdNywBUUfwI1Bnbsn+H23ROh9TI5VwndTY1u0HcemEne0UFq66FIy0H5//7jgwBsWD4c2W7dgHELJQzKOxbbqRv97+qBiTmNoWSMXgblx4B4FPEgsCzh2PuBnwC3A1XgXuBxSSc1xlwJXAmwdevWbC2Hp6Sem992QcvBaIu70uON2/cDcPqmlUAtKG8F5db7D4XH1iyUal3PK4vnGd76td/1JLZg9SOfay0o73mG933795x97CrO2rIqsq+YsxZKrA4lluVVyOVYPlwriNRuw9mil4IyCYzHto0DSTb924EzgU3ALuAS4PsicrIxZjrheEXpCstHim2/p5CvnwSPXTsKQDmIq9g78gPBOiBQs16apQ3vOjTLp395T/g63nK/k7jdhnM5mTeGUvY8Ds1WOPeEtXUrLRYL1kKpL2yEqIXiplJroXy26KXL6w6gICInONtOA25JOPY04BpjzA5jTMUYczWwEo2jKBmgEHP85wTWLRskn5OwItxdxdFi04unSq7LKzqjxmMnhS7OuG4MxW+90vx467KLL50MtXFWGlgobgzFDcRrUD5b9ExQjDFTwJeBd4rIqIicAzyV5OytG4Bni8h6EcmJyAuBIvDHXo1XURbKQGwRj9VjgxTyOUaK+fBOfCYhS+vAtB/QdluvxKdTd616qJ+gO0m09Uot6+t39x1ky5uu5Xf3RT3Yto4mLqhQcwM2dHk1slDURMkUvc6heBUwDDwAfBZ4pTHmFhHZLCKTIrI5OO6f8QP2vwYOAK8HnmmMOdDj8SpK28RdXsuCTsSjg4VQLOYSBMU2ipxs4vK6LxaMb7UdSiOm5ircuWcycZ89dS5HJMvru7ftBuA7t+6OHG/jQ8UEN1ytfX10vIN1MRSJBOJVT7JFTyvljTH7gKclbL8HP2hvX88Crw4eipIp4oJi77JHBvNMBvUWSRaKFZS5ihfGU+Jxi/v2RwWl4pmgzfvCZt4XXXU9N27fz92XXVy3L+LyClunmFpadCwH2HYGKCaogLVa4tczkPcD8NNOHUoupy6vrKJZ3orSYYoxl4+94x4dKDAdWB9JhYn7nQC9jSmUY6XlcQsFFmel2Cy0pEaTbtqwvYaqMQ3jIVYEiwnr9hbzzWMobi8vqMVcVFCyhQqKonSYuIViJ8nRwXzYp2smISjvYt1ebmNJgN2H5uqObSWO8tM/PMjP79zbcP+O/TNsedO13HD3vnCbmzacc9J+7fW4i4UdmC5x2/1+wmZSlluh1RhKICD28zSGki1UUBSlw8Tv0J/2iCOBwEKxLi+nv5ddS8Xl4IxvrZSqUUtmJqG1fSuCcsnHfsnzPvqLhvt/dqdfkPipX2wPt9WaQ9ZE8RHvvM5xedU+98M/upNXfOpGoD4pAeavQ5kuVcP0ZHCERfUkU6igKEqHcQXlvc84lb8+91gARgYL4TooNn0YqOt7BbA/iKfELZSkdON4LGMh2FO4ni/rBss7Lq+ZctVZ76X2uYdmau66QoKgNLRQgu1zFS+SHZZXl1cmUUFRlA7jpr2uGRsMA+ajA/mwaNG1UFaMFMM7cZsRtn+6gaBU6mMvnUgdtinBJmFbLhYoD2MojoViG2FCgzqUBlle7rFu6Ml+nFoo2UIFRVG6yICTQjs6WAiD7a4wDBbzjAZCMj7sV+bbjK9SLCifFMyvxHvCLwAbgHezsMK0YRFcjbACVvEMh2bLzFWqkVUpE4Py4Zry0bGKSPgdqYWSfXSBLUXpIm5rlNGBfFAFbyIWykBeGC7mmZitsHy4yH0HZsLJ3LVQjDHJLq95SthbyQKzAXY328tzWq+42WZWwMpVj4df+h3O2rKKVU4cKElQrIWS1L1lMJ+jVPEilp0KSjZRC0VRuohroYwMFjDGj0O4qzgOFHIMD/j1GMuHo73D3Dv/uZj7ayywauYTjEZr1LviYbPKXG2qLQEs7D5Ua/lixccKy/V374uMMzHLq4nvqmah1NefaPv6bKE/l6J0ETfjybq1puZigpLPMVRoICiOa8y6u+w5raCUq4Zf3rWXK773h0RxmZxNFhTX2rGxHbfLcdWpQ9k9UUtXtuLhxm7cFOJ4HQ74rq1nnLGRT77srLp9VlByCRaKdhvOFiooitJFhopRlxf4VeFu65WBQo6hYN/4cNQL7bq8rADYDshjQzUL5cUfv57Lr7uD4978TT7yozsj52jUymVituwcU1ufxeLGUB66obbKxGzgrmtkPRULySJw+XNO59wT1tZtb2ah6Hoo2UIFRVG6iG0tAjAy0MBCKeQYDoRnoJCLiJA7adviv5VWUEILxePIFbUFra654d7IGCYaWCgTjtBYC8WLCEptCeC/ecxxXHL25sg4pp33uxZKUnPIZliLy42hWLeZZnllCxUURekig444WAGYKlUiQjGQzzNc9IWnkMuFx0Ft/RSoubxWDA9Ezlf1DA/bUFtq6Li1YVs8oJmFUi8oVc9jrlLl2H+4ls8E667kxe8AbM9rBcV9fzTJoE1BSbBQaoWNqihZQgVFUbqIO7mODPqisX+qFHEt+VaJFRQJLRmAuarr8vIn7eUxC6XieZHJ+NBsmQcmakH0RjEUd/tk2GPMY/9UGc/AT4PlfG0cwxYnWtfbQaeY8QEnxpIUlG9GUgwlpzGUTKKCoihdxM3yWhtUxN8bdAy2RX2+yysQlHyONWO1FFw3hlLn8gpiKJWqoewZNq0a5nEnreP6P+3jrPd8L7Q6JudqE7+LLZ70j/GPnS5XI7EVqLmibPsUOw5XUGzdjH9dC3N5JVkobZ5KWWL051KULuLWoWxYPkQ+J+H6I4NBZtegE5Qv5IRj1tRcVm6W11xgGawcibq8Kp6hUvUYKRbYsHwoPN66pOy/cVfUg5M1q8KKz2ypyqGYRWPn+dBCCRtc1hdZQnKlfDNsynTeib3ktA4lk6igKEoXcftaFfI5Niwf4o8PTAava5OmFZ5CXsL15yEalI+7vJYNuYJiKOSF1U5fMGuZWOvDjedA1E1ls7xmEiwUO6lboWgkJJZ2LZRhx91nsadQl1e2UEFRlB5y1Mph7gosFDvx5qT2vJATjl49Eh4fCcoH1sqmlf7+9eO+NVL1PMqeqXOXWctkv+OOctkzUW+h+IISs1DCNUqireYb0W4MxQpKLjEo39aplCVGW68oSg/ZtHKEX9zlrzliXVAiUTfRUStrghJJGy75z8/csorvvuG80GIpV32XVzEnEevAWiZ3PTgF1FfUu4JiRWK2VC8o+TAo7/+b1P7FJamwsRmuu88StrFXRckUaqEoSg/ZtKomFmF2k0h491+qGh6y3i8iXLtsMFbY6E/6Q8Ucx68bC8Wj6ri8VjiV9tbquGuPLyjxrsR7JuYiMRdo5PLy/7UTflKDysjxbYqAtVDymjaceVRQFKWHrFtWi3FYq0REakvkVj2GB/LcfdnFPOOMjVFBqVhBiU7A5apH2fMo5nNcdMoRvOMpJwO+y2u6VAmXDfbigjJZLygVz7BvOuoiC11e+ehyvZ1ixAblpd5CUT3JFiooitJDbKov1OImQm2ydivOB/M5SlUvbOJoXU02gF901hgpV31BERGefJq/QuTkXCW0To5fNxaxUKqeYe/kHBuW1yrsLXsOzUV6ioVBeWuhJKzJshisQLqt7XVN+WyiMRRF6QLfef1jODBdX//hVsEnBeXdNvErgvTgA9NlVo4OMFuuMlTMhZlP1kKpVAOXl7N2Pfgur3v2TQNw3NpR/vjAJJ5nyOWEn9+5F88QSQCw3L57gmVDhbDOJF9X2NhZQbEuL9cay2sMJZOohaIoXeDE9cs465hVdduXDdXu/G1QPpdzXF5O/3ibxbU7qHr3BaXWG8zGXSqOhQJ+fctAIcfEXCUUhbWBq81aKf/87d+zZfUIz3rkUeH5zjtxLQP5HLfsPBQZpwSzRKtB+XaxdSiudWYtEzVQsoUKiqL0kGWuy6tQi6GE4uBYKOvHfRHYvneaqmfYN1WKBN1ry+p6VDwTSdcdGywwOVsJ13pfFVg7NtNr+94pzjtxbcRiOmrlME97xJF148yHLq/adOEWbLrdABaCtVBcQdEFtrKJCoqi9BB3oj5hnZ/NtWF8KBQDN03YWih/88kbeeMXbmbngRk2rqzFPKyL661fu4Xte6cjXX7HBgtMzVU4NFumkJNaq3vjWzOHZiusHB2IuJSGinn+6txjARh3xpmLpQ0DkRUaXVFaCNZCcV1eOa1DySQqKIrSQ9zJ91XnH8cnX3YWf/Gw9Zy6cTkAj3HWC1nrZIR95ab72HlgNhJEL8Qq0otxC2WuwqGZCuPDxbCtSbVqwtjO6tGByDmGi3lOXL+MV5x3HBedsiHcbo0E9/y2/QvUYjYLJTmGEvyrFkqm0KC8ovSQUaeT8HAxHy449dAN4/zm0gsYd2IXbrwEYNeh2ci6J/FldeMur4nZCiMDZcaHCuGxVWPCppArRwci57CWwpueeBIAb/zCzUDN/eRaQCtH62NBCyW0UBx3n67YmE16aqGIyCoR+YqITInIdhF5fpNjjxWRb4jIhIg8KCLv6+VYFaUbuEV/tjmkxRWTRmxcUasbqRMU1+U1VGCq5Lu8xoeL4ed+9vp72HXQD/KvGhmIiNBwMdnSSHJ5uce227srTlIMRdAYShbptcvr34ASsB54AfBhETk5fpCIDADXAd8HjgCOAj7Vw3EqStdppSuvLfqzuBZKPKXWPd/68SG2753mwHSZ5cPFUHze/1+38+5rbwWsheK4vGKf9Y6nnMzoQD6MY7jC4b6v3d5dcZJiKIGeaAwlY/RMUERkFHgm8FZjzKQx5qfAfwIvTDj8JcBOY8zlxpgpY8ysMeY3vRqrovSCVtw5P/nf53P9Pz6ei0/1YxquoMTf78ZDth69konZCr++9wDjQ8WI+Pwh6Ha8KhaUj4vXi/98C7e886LaAlsJS/RC5ywUNyHB0m4bF2Vp6aWFciJQNcbc4Wy7GaizUICzgbtF5FuBu+uHInJqT0apKCli9dgg65YN8cHnnMbHX3Jm3fK+Lu7EfuaWWg3M+HAhEty2BeluYB0au7wsrlXifla7zSDj2M91m1fmtJdXJumloIwBB2PbDgLLEo49CngucAVwJHAt8LXAFRZBRF4uIttEZNuePXs6PGRFSQdDxTznn7Su6TFF525+06phjgjSjseHinVuqWWDhbr6kbjLK457jnwDa+X4dY0FrxFJn2vPqAZKtuiloEwC47Ft48BEwrEzwE+NMd8yxpSADwCrgYfGDzTGXGmM2WqM2bp27dr4bkU5bHBdXiLCYx/i/3/I5aTuTt8u0uUSd3nVn991cwnve9bDefOTTgo/9+MvPZPrXv+Ytsc9mFAYKWEMRRUlS/QybfgOoCAiJxhj/hBsOw24JeHY3wDn9GxkitJDvvrqc7g/6ADcSeJB/gtOXs/nbriXcsWrywiLpyQ32hY5fy4alH/O1k2AX6D54zv2cPKR4wtK8xURROC1jzshYV/bp1OWkJ5ZKMaYKeDLwDtFZFREzgGeCnwy4fBPAWeLyF+ISB74O+BB4LZejVdRusXpm1bwxFM3zH9gC3zplX8ePo+LxvkPWcflzzmNv33cCQkZYfX/9UcGmt9f+pZO8FmOeJ1/0jruvuxi1i0bavDO+fnTey/m9U84MXxtz67NIbNFr9OGXwUMAw8AnwVeaYy5RUQ2i8ikiGwGMMbcDlwCfATYjy88TwncX4qiBDzy6JUcFbRjiVfOiwjPOOMolo8U6ybmgYRU3/mC8u5nxMWr04gG5TNJTyvljTH7gKclbL8HP2jvbvsyvkWjKEoTrFg0q2upE5SEuMV8QXnwA/8l6sWr02hQPptoLy9FyThJrVEaHWNJcnm1Y6EUuz3TB6fX1ivZQgVFUTJOPqE1St0xLQhKK23oazGU3kwdGkPJFiooipJxai6vxv+d49bLQqvb7Xoti62On49aL6+ufozSYVRQFCXjhGu+N5nk47sGCgubqSfmKsDCChjbQYUkm6igKErGsa6u5i6vzlgolocftXxR758PDZ1kExUURck4ocurWVBe5o+htINdTbLbGDP/MUp60AW2FCXjdCoo39Jn5aSuqWQ3sDEU1ZNsoYKiKBnHikWzu/m42Lj9s655+dlUWzQFfnvpBT0pNrQfoRZKtlBBUZSMYwXFbf8eJy4CbhHko45d3fJnzdeepVNoDCWbaAxFUTKOFZSKV79AlSXeKqXbab+dwqjTK1Nk469KUZSGWEHxmviHOhVD6R3zu/GU9JH2vypFUebhZY8+BoBTNjZO5W2ll1eaCGMoSzsMpU00hqIoGefcE9Zy92UXNz2m3uWV7iBFODo1UTJFum9TFEXpCLmMubzUQskm6f6rUhSlI8QtlNS7vDSGkknS/VelKEpHyKyFooqSKdL9V6UoSkeos1DSLihLPQBlQaT7r0pRlI6QvbRhH7VPskU2/qoURVkU9c0h020D2JUa1eOVLVRQFOUwoM5CSXlQ3qJ6ki2y8VelKMqiiK/NnvoYSroNKKUB6f6rUhSlY3z8JWcyPuTXMqc9bfgxJ6wF4BGbVyzxSJR20Ep5RTlMOP+kdawbH+LQ7GTqg/Lnn7SO37/rIoaK+aUeitIG6f6rUhSlK6Q9KA+omGQQFRRFOYywhYJpt1CUbKJ/VYpyGJJLv4GiZBAVFEU5jNA0XKWb9FVQ/vbbb+exj33sUg9DUVLLjtP+EoZX86IXvZiB2X1LPRylz5B+ar4mIhPA7Ys8zXLg4CKPS9o337b4fvva3b4GeLCFsTWjV9fX7HWj5726vnavLWn7Ulxft367pO3tXl+W/jaTtvXz9bUytzzEGLOshbE1xxjTNw9gWwfOceVij0vaN9+2+H77OnZMZq6v2esmz3tyfe1eW1qur1u/XSeuL0t/m4fb9fVqbjHGaAwlga934LikffNti+//eoPti6VX19fsdbPrXiytnK/da0vavhTX163fLml7P11fu3+v/XZ9vZpb+s7ltc0Ys3Wpx9Et9PqyTT9fXz9fG+j1tUq/WShXLvUAuoxeX7bp5+vr52sDvb6W6CsLRVEURVk6+s1CURRFUZYIFRRFURSlIxx2giIiW0Rkj4j8MHisXeoxdRoReZ6I7FnqcXQaEVkvIj8TkR+JyPdFZMNSj6mTiMificjPg+v7rIgUl3pMnURElovI9SIyKSKnLPV4OoGIvEdEfiIiXxSRkaUeTydZyO912AlKwI+MMY8NHn018YpIDngWcO9Sj6ULPAg82hhzHvAJ4GVLPJ5Osx14XHB9dwFPXeLxdJpp4GLgi0s9kE4QTLLHGWPOBb4L/OUSD6nTtP17Ha6Cck5wV/FPEl/KLvs8H/8PwFvqgXQaY0zVGGOvaxlwy1KOp9MYY3YaY2aClxX67Dc0xpT77AbuXOBbwfNvAY9ewrF0nIX8XqkWFBF5jYhsE5E5Ebk6tm+ViHxFRKZEZLuIPL/F094PHA88BlgHPKOzo26NblybiOSB5wDXdGHIbdGl3w4ROV1Efgm8BvhVh4fdMt26vuD9xwBPBL7RwSG3RTevL20s4lpXUmtdchBY1aMht0Uvf8u0N4fcCbwbuBAYju37N6AErAdOB64VkZuNMbeIyBEkm2nPMsbsAuYAROTLwNnAl7o0/mZ0/NqCc33eGOOlwPDqym9njPk18CgReQ7wD8ArunYFzenK9YnIOPB/gRcaY0rdG/68dOv/XhpZ0LUC+/H7YRH8m9Zumwu9vvbpRP+Wbj+CL+Nq5/Vo8CWc6Gz7JHBZC+cad56/F3hRH13bPwPfAb6Nf8d0RZ/9doPO8wuBy/vs+grAtfhxlCW9rm5cn3P81cApS31ti71W4FTgM8HzlwN/u9TX0I3fsp3fK9UuryacCFSNMXc4224GTm7hveeJyI0i8hNgI/CZbgxwESz42owxf2+MucAYcxHwB2PMa7s1yEWwmN/uDBH5sYj8APg74P3dGOAiWcz1PQ94FPC2IAPxf3ZjgItkMdeHiHwTuAD4qIi8pPPD6yhNr9UY81tgezCXXAhc1fshLop5f8t2f6+0u7waMUZ9u+aD+IHaphhjvk4XmqJ1kAVfm4tJb9+hxfx2P8ePfaWZxVzfJ/HvENPMov4+jTFP6viIuse812qM+YeejqiztHJ9bf1eWbVQJoHx2LZxYGIJxtJp+vnaQK8v6/T79bn0+7V2/PqyKih3AAUROcHZdhr9kUbaz9cGen1Zp9+vz6Xfr7Xj15dqQRGRgogMAXkgLyJDIlIwxkwBXwbeKSKjInIOfhFY2t0FIf18baDXh15fZuj3a+3p9S115sE8WQmXAib2uDTYtwr4KjAF3AM8f6nHq9em16fXl71Hv19rL69P29criqIoHSHVLi9FURQlO6igKIqiKB1BBUVRFEXpCCooiqIoSkdQQVEURVE6ggqKoiiK0hFUUBRFUZSOoIKiKB1ERIyIPGupx6EoS4EKipIpRORqEVmylQxbYAMp7mYtIpeKyO+WehxKf6KCoijzICIDrR5r/FUl57o5niTaGaOidAsVFKWvEJHlInKliDwgIhMi8iMR2ersXy0inxWRHSIyIyK3iMhLY+f4oYh8WEQ+ICJ7gP8OthsRebmIfCFYg/suEbkk9t7Q5SUiW4LXzxSR60RkWkRuFZEnxN5zsYjcLiKzwQJizw3et6XJdd4dWBtXicgB4NPB9suCc80Ex7wvaAxIsEDS24GTg/Mbu2jSfN+borSCCorSN4iI4C+huxH4H8AjgB8D3xeRDcFhQ8Cvgv0nA/8K/LuIPD52uksAAc4FXuRsfxvwNfw239cAV4nI0fMM7T3AFcF7bgA+JyJjwZg343d8vTbYfwXwvhYv+Q3A74GtwJuDbVPAXwIPBV4FPBf4x2DfNcAHgdvxXXMbgGta/N4UZX6WuhOmPvTRzgN/fetvNNj3OPxFg4Zj238N/O8m5/wc8B/O6x8Cv0k4zgDvdV4XgGngktgxzwqebwle/42zf2Ow7dHB6/cCt4HfqDXY9ubgmC1Nxnw38PUWvq9XAH90Xl8K/K4T35s+9BF/ZHUJYEVJ4pHACLDHv+kOGQKOAxCRPPAm4H/iT+6DwAC+iLjc2OAzfmOfGGMqgUts3Tzj+o3zfGfwr33PScANFNxuqwAAAepJREFUxhi37fcv5zmfZVt8Q+Bu+zvgePwlXvPBoxnzfm+K0goqKEo/kQN247up4hwK/n0j8P8ArwN+i39n/k/Ui8JUg88ox14b5ncdh+8xxphg0rbvkeAcCyEyRhE5G9/aegfweuAA8BTgA/Ocp5XvTVHmRQVF6Sd+BawHPGPMXQ2OeTS+q+iTEMZdTsSffJeC2/BXyXM5a4HnOge4zxjzLrshIb5Tot5iaeV7U5R50aC8kkXGReT02GML8F38jKyvicgTReQYEfkzEXmHiNi77zuAx4vIo0XkJOD/A45Zkqvw+QhwXJBR9hAReQbwN8G+di2XO4CNIvICETlWRF4JPC92zN3A0SJyhoisEZFBWvveFGVeVFCULHIucFPs8YEgDvEk4PvAR/GzmT4PPIRa7OLdwPXAt/AzmaYIUm6XAmPMduCZ+K6pm/FdVe8Ids+2ea6vA+8H/gU/bvME/Kw0ly8B3wS+B+wBntfi96Yo86JLACtKyhCR1wHvBFYaY7ylHo+itIrGUBRliRGRV+PXp+wBzgbeClytYqJkDRUURVl6jsevPVkN7MCPq7xzSUekKAtAXV6KoihKR9CgvKIoitIRVFAURVGUjqCCoiiKonQEFRRFURSlI6igKIqiKB1BBUVRFEXpCP8/ddeeybimGuoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "class OneCycleScheduler(keras.callbacks.Callback):\n",
    "    def __init__(self, iterations, max_rate, start_rate=None,\n",
    "                 last_iterations=None, last_rate=None):\n",
    "        self.iterations = iterations\n",
    "        self.max_rate = max_rate\n",
    "        self.start_rate = start_rate or max_rate / 10\n",
    "        self.last_iterations = last_iterations or iterations // 10 + 1\n",
    "        self.half_iteration = (iterations - self.last_iterations) // 2\n",
    "        self.last_rate = last_rate or self.start_rate / 1000\n",
    "        self.iteration = 0\n",
    "    def _interpolate(self, iter1, iter2, rate1, rate2):\n",
    "        return ((rate2 - rate1) * (self.iteration - iter1)\n",
    "                / (iter2 - iter1) + rate1)\n",
    "    def on_batch_begin(self, batch, logs):\n",
    "        if self.iteration < self.half_iteration:\n",
    "            rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)\n",
    "        elif self.iteration < 2 * self.half_iteration:\n",
    "            rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n",
    "                                     self.max_rate, self.start_rate)\n",
    "        else:\n",
    "            rate = self._interpolate(2 * self.half_iteration, self.iterations,\n",
    "                                     self.start_rate, self.last_rate)\n",
    "            rate = max(rate, self.last_rate)\n",
    "        self.iteration += 1\n",
    "        K.set_value(self.model.optimizer.lr, rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/25\n",
      "55000/55000 [==============================] - 1s 22us/sample - loss: 0.6576 - accuracy: 0.7743 - val_loss: 0.4901 - val_accuracy: 0.8300\n",
      "Epoch 2/25\n",
      "55000/55000 [==============================] - 1s 18us/sample - loss: 0.4587 - accuracy: 0.8387 - val_loss: 0.4316 - val_accuracy: 0.8490\n",
      "Epoch 3/25\n",
      "55000/55000 [==============================] - 1s 22us/sample - loss: 0.4119 - accuracy: 0.8560 - val_loss: 0.4117 - val_accuracy: 0.8580\n",
      "Epoch 4/25\n",
      "55000/55000 [==============================] - 1s 23us/sample - loss: 0.3842 - accuracy: 0.8657 - val_loss: 0.3920 - val_accuracy: 0.8638\n",
      "Epoch 5/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.3636 - accuracy: 0.8708 - val_loss: 0.3739 - val_accuracy: 0.8710\n",
      "Epoch 6/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.3460 - accuracy: 0.8767 - val_loss: 0.3742 - val_accuracy: 0.8690\n",
      "Epoch 7/25\n",
      "55000/55000 [==============================] - 1s 22us/sample - loss: 0.3312 - accuracy: 0.8818 - val_loss: 0.3760 - val_accuracy: 0.8656\n",
      "Epoch 8/25\n",
      "55000/55000 [==============================] - 1s 19us/sample - loss: 0.3194 - accuracy: 0.8846 - val_loss: 0.3583 - val_accuracy: 0.8756\n",
      "Epoch 9/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.3056 - accuracy: 0.8902 - val_loss: 0.3474 - val_accuracy: 0.8820\n",
      "Epoch 10/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.2943 - accuracy: 0.8937 - val_loss: 0.3993 - val_accuracy: 0.8562\n",
      "Epoch 11/25\n",
      "55000/55000 [==============================] - 1s 20us/sample - loss: 0.2845 - accuracy: 0.8957 - val_loss: 0.3446 - val_accuracy: 0.8820\n",
      "Epoch 12/25\n",
      "55000/55000 [==============================] - 1s 20us/sample - loss: 0.2720 - accuracy: 0.9020 - val_loss: 0.3348 - val_accuracy: 0.8808\n",
      "Epoch 13/25\n",
      "55000/55000 [==============================] - 1s 20us/sample - loss: 0.2536 - accuracy: 0.9094 - val_loss: 0.3386 - val_accuracy: 0.8822\n",
      "Epoch 14/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.2420 - accuracy: 0.9125 - val_loss: 0.3313 - val_accuracy: 0.8858\n",
      "Epoch 15/25\n",
      "55000/55000 [==============================] - 1s 19us/sample - loss: 0.2288 - accuracy: 0.9174 - val_loss: 0.3241 - val_accuracy: 0.8840\n",
      "Epoch 16/25\n",
      "55000/55000 [==============================] - 1s 18us/sample - loss: 0.2169 - accuracy: 0.9222 - val_loss: 0.3342 - val_accuracy: 0.8846\n",
      "Epoch 17/25\n",
      "55000/55000 [==============================] - 1s 18us/sample - loss: 0.2067 - accuracy: 0.9264 - val_loss: 0.3208 - val_accuracy: 0.8874\n",
      "Epoch 18/25\n",
      "55000/55000 [==============================] - 1s 20us/sample - loss: 0.1977 - accuracy: 0.9301 - val_loss: 0.3186 - val_accuracy: 0.8888\n",
      "Epoch 19/25\n",
      "55000/55000 [==============================] - 1s 19us/sample - loss: 0.1892 - accuracy: 0.9329 - val_loss: 0.3278 - val_accuracy: 0.8848\n",
      "Epoch 20/25\n",
      "55000/55000 [==============================] - 1s 19us/sample - loss: 0.1818 - accuracy: 0.9375 - val_loss: 0.3195 - val_accuracy: 0.8894\n",
      "Epoch 21/25\n",
      "55000/55000 [==============================] - 1s 20us/sample - loss: 0.1756 - accuracy: 0.9395 - val_loss: 0.3163 - val_accuracy: 0.8948\n",
      "Epoch 22/25\n",
      "55000/55000 [==============================] - 1s 21us/sample - loss: 0.1701 - accuracy: 0.9416 - val_loss: 0.3177 - val_accuracy: 0.8920\n",
      "Epoch 23/25\n",
      "55000/55000 [==============================] - 1s 22us/sample - loss: 0.1657 - accuracy: 0.9441 - val_loss: 0.3168 - val_accuracy: 0.8944\n",
      "Epoch 24/25\n",
      "55000/55000 [==============================] - 1s 19us/sample - loss: 0.1629 - accuracy: 0.9454 - val_loss: 0.3167 - val_accuracy: 0.8946\n",
      "Epoch 25/25\n",
      "55000/55000 [==============================] - 1s 18us/sample - loss: 0.1611 - accuracy: 0.9465 - val_loss: 0.3170 - val_accuracy: 0.8934\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 25\n",
    "onecycle = OneCycleScheduler(len(X_train) // batch_size * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Avoiding Overfitting Through Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $\\ell_1$ and $\\ell_2$ regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "# or l1(0.1) for ℓ1 regularization with a factor or 0.1\n",
    "# or l1_l2(0.1, 0.01) for both ℓ1 and ℓ2 regularization, with factors 0.1 and 0.01 respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 7s 130us/sample - loss: 1.5735 - accuracy: 0.8126 - val_loss: 0.7327 - val_accuracy: 0.8222\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 6s 115us/sample - loss: 0.7186 - accuracy: 0.8260 - val_loss: 0.6929 - val_accuracy: 0.8338\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(100, activation=\"elu\",\n",
    "                       kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01)),\n",
    "    keras.layers.Dense(10, activation=\"softmax\",\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 7s 133us/sample - loss: 1.6006 - accuracy: 0.8129 - val_loss: 0.7374 - val_accuracy: 0.8236\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 7s 128us/sample - loss: 0.7179 - accuracy: 0.8265 - val_loss: 0.6905 - val_accuracy: 0.8356\n"
     ]
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "RegularizedDense = partial(keras.layers.Dense,\n",
    "                           activation=\"elu\",\n",
    "                           kernel_initializer=\"he_normal\",\n",
    "                           kernel_regularizer=keras.regularizers.l2(0.01))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    RegularizedDense(300),\n",
    "    RegularizedDense(100),\n",
    "    RegularizedDense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 8s 139us/sample - loss: 0.5856 - accuracy: 0.7992 - val_loss: 0.3908 - val_accuracy: 0.8570\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 6s 117us/sample - loss: 0.4260 - accuracy: 0.8443 - val_loss: 0.3389 - val_accuracy: 0.8730\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    keras.layers.Dropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alpha Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      "55000/55000 [==============================] - 3s 55us/sample - loss: 0.6616 - accuracy: 0.7616 - val_loss: 0.6682 - val_accuracy: 0.8258\n",
      "Epoch 2/20\n",
      "55000/55000 [==============================] - 3s 49us/sample - loss: 0.5526 - accuracy: 0.7969 - val_loss: 0.5835 - val_accuracy: 0.8382\n",
      "Epoch 3/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.5259 - accuracy: 0.8060 - val_loss: 0.5312 - val_accuracy: 0.8512\n",
      "Epoch 4/20\n",
      "55000/55000 [==============================] - 3s 49us/sample - loss: 0.5076 - accuracy: 0.8111 - val_loss: 0.4969 - val_accuracy: 0.8606\n",
      "Epoch 5/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4929 - accuracy: 0.8175 - val_loss: 0.4875 - val_accuracy: 0.8610\n",
      "Epoch 6/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.4844 - accuracy: 0.8190 - val_loss: 0.5141 - val_accuracy: 0.8552\n",
      "Epoch 7/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.4716 - accuracy: 0.8258 - val_loss: 0.4474 - val_accuracy: 0.8660\n",
      "Epoch 8/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4643 - accuracy: 0.8272 - val_loss: 0.4535 - val_accuracy: 0.8596\n",
      "Epoch 9/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.4574 - accuracy: 0.8302 - val_loss: 0.4602 - val_accuracy: 0.8652\n",
      "Epoch 10/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.4500 - accuracy: 0.8345 - val_loss: 0.4225 - val_accuracy: 0.8652\n",
      "Epoch 11/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4484 - accuracy: 0.8334 - val_loss: 0.4461 - val_accuracy: 0.8674\n",
      "Epoch 12/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4466 - accuracy: 0.8341 - val_loss: 0.4461 - val_accuracy: 0.8688\n",
      "Epoch 13/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4402 - accuracy: 0.8377 - val_loss: 0.4570 - val_accuracy: 0.8730\n",
      "Epoch 14/20\n",
      "55000/55000 [==============================] - 3s 48us/sample - loss: 0.4336 - accuracy: 0.8390 - val_loss: 0.4867 - val_accuracy: 0.8708\n",
      "Epoch 15/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4324 - accuracy: 0.8404 - val_loss: 0.4342 - val_accuracy: 0.8752\n",
      "Epoch 16/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4292 - accuracy: 0.8395 - val_loss: 0.4934 - val_accuracy: 0.8602\n",
      "Epoch 17/20\n",
      "55000/55000 [==============================] - 3s 49us/sample - loss: 0.4294 - accuracy: 0.8390 - val_loss: 0.4297 - val_accuracy: 0.8802\n",
      "Epoch 18/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4230 - accuracy: 0.8435 - val_loss: 0.4425 - val_accuracy: 0.8754\n",
      "Epoch 19/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4242 - accuracy: 0.8425 - val_loss: 0.4083 - val_accuracy: 0.8742\n",
      "Epoch 20/20\n",
      "55000/55000 [==============================] - 3s 47us/sample - loss: 0.4201 - accuracy: 0.8450 - val_loss: 0.4256 - val_accuracy: 0.8766\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(300, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\"),\n",
    "    keras.layers.AlphaDropout(rate=0.2),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])\n",
    "n_epochs = 20\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 25us/sample - loss: 0.4736 - accuracy: 0.8648\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.47358015085458754, 0.8648]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "55000/55000 [==============================] - 1s 22us/sample - loss: 0.3532 - accuracy: 0.8859\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.35321958436965945, 0.8859091]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples\n",
      "55000/55000 [==============================] - 2s 44us/sample - loss: 0.4186 - accuracy: 0.8451\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MC Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer flatten_18 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_probas = np.stack([model(X_test_scaled, training=True)\n",
    "                     for sample in range(100)])\n",
    "y_proba = y_probas.mean(axis=0)\n",
    "y_std = y_probas.std(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(model.predict(X_test_scaled[:1]), 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.05, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.17, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.01, 0.  , 0.8 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.03, 0.05, 0.9 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.04, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.05, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.02, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.61, 0.  , 0.01, 0.  , 0.38]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.02, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.03, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.33, 0.  , 0.04, 0.  , 0.63]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.2 , 0.  , 0.06, 0.  , 0.74]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.42, 0.  , 0.09, 0.  , 0.49]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.05, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.16, 0.  , 0.03, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.  , 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.09, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.11, 0.  , 0.03, 0.  , 0.87]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.64, 0.  , 0.13, 0.  , 0.22]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.42, 0.  , 0.2 , 0.  , 0.39]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.2 , 0.  , 0.78]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.02, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.17, 0.  , 0.15, 0.  , 0.67]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.01, 0.01, 0.05, 0.08, 0.1 , 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.02, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.68, 0.  , 0.06, 0.  , 0.26]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.01, 0.12, 0.  , 0.15, 0.06, 0.66]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  , 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.35, 0.  , 0.01, 0.  , 0.65]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.24, 0.  , 0.28, 0.  , 0.48]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.03, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.53, 0.  , 0.4 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.03, 0.  , 0.93]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.09, 0.  , 0.89]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.36, 0.  , 0.  , 0.  , 0.63]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.02, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.27, 0.  , 0.04, 0.  , 0.68]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.05, 0.  , 0.01, 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.11, 0.01, 0.86]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.03, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.25, 0.  , 0.75]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.9 , 0.  , 0.01, 0.  , 0.09]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.14, 0.  , 0.04, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.01, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.47, 0.  , 0.  , 0.  , 0.52]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.15, 0.  , 0.08, 0.  , 0.77]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.04, 0.  , 0.92]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.03, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.22, 0.  , 0.04, 0.  , 0.74]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.1 , 0.  , 0.86]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.08, 0.  , 0.03, 0.  , 0.89]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.02, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.1 , 0.  , 0.07, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.16, 0.01, 0.79]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.62, 0.  , 0.04, 0.  , 0.35]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.04, 0.04, 0.91]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.29, 0.  , 0.04, 0.  , 0.67]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.33, 0.  , 0.04, 0.  , 0.63]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.  , 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.27, 0.  , 0.72]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.03, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.  , 0.  , 0.99]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.2 , 0.  , 0.76]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.04, 0.  , 0.01, 0.  , 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.3 , 0.  , 0.68]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.09, 0.  , 0.9 ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.96, 0.  , 0.02, 0.  , 0.02]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.13, 0.  , 0.04, 0.  , 0.84]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.97]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.16, 0.  , 0.06, 0.  , 0.78]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.18, 0.  , 0.01, 0.  , 0.81]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.31, 0.  , 0.25, 0.01, 0.43]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.11, 0.  , 0.83]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.07, 0.  , 0.08, 0.  , 0.85]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.03, 0.  , 0.96]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.02, 0.  , 0.02, 0.01, 0.95]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.  , 0.  , 0.94]],\n",
       "\n",
       "       [[0.  , 0.  , 0.  , 0.  , 0.  , 0.41, 0.  , 0.02, 0.  , 0.56]]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_probas[:, :1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.06, 0.  , 0.81]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(y_proba[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.19, 0.01, 0.08, 0.01, 0.21]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_std = y_probas.std(axis=0)\n",
    "np.round(y_std[:1], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8606"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = np.sum(y_pred == y_test) / len(y_test)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCDropout(keras.layers.Dropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)\n",
    "\n",
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_20\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_18 (Flatten)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout (MCAlphaDro (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_262 (Dense)            (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_1 (MCAlphaD (None, 300)               0         \n",
      "_________________________________________________________________\n",
      "dense_263 (Dense)            (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "mc_alpha_dropout_2 (MCAlphaD (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_264 (Dense)            (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "mc_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)\n",
    "mc_model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=optimizer, metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model.set_weights(model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use the model with MC Dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.12, 0.  , 0.07, 0.01, 0.79]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in range(100)], axis=0), 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Max norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(100, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                           kernel_constraint=keras.constraints.max_norm(1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/2\n",
      "55000/55000 [==============================] - 6s 114us/sample - loss: 0.4734 - accuracy: 0.8364 - val_loss: 0.3999 - val_accuracy: 0.8614\n",
      "Epoch 2/2\n",
      "55000/55000 [==============================] - 6s 100us/sample - loss: 0.3583 - accuracy: 0.8685 - val_loss: 0.3494 - val_accuracy: 0.8746\n"
     ]
    }
   ],
   "source": [
    "MaxNormDense = partial(keras.layers.Dense,\n",
    "                       activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       kernel_constraint=keras.constraints.max_norm(1.))\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    MaxNormDense(300),\n",
    "    MaxNormDense(100),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"nadam\", metrics=[\"accuracy\"])\n",
    "n_epochs = 2\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs,\n",
    "                    validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. to 7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See appendix A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Deep Learning on CIFAR10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a.\n",
    "*Exercise: Build a DNN with 20 hidden layers of 100 neurons each (that's too many, but it's the point of this exercise). Use He initialization and the ELU activation function.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 activation=\"elu\",\n",
    "                                 kernel_initializer=\"he_normal\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b.\n",
    "*Exercise: Using Nadam optimization and early stopping, train the network on the CIFAR10 dataset. You can load it with `keras.datasets.cifar10.load_data()`. The dataset is composed of 60,000 32 × 32–pixel color images (50,000 for training, 10,000 for testing) with 10 classes, so you'll need a softmax output layer with 10 neurons. Remember to search for the right learning rate each time you change the model's architecture or hyperparameters.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add the output layer to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use a Nadam optimizer with a learning rate of 5e-5. I tried learning rates 1e-5, 3e-5, 1e-4, 3e-4, 1e-3, 3e-3 and 1e-2, and I compared their learning curves for 10 epochs each (using the TensorBoard callback, below). The learning rates 3e-5 and 1e-4 were pretty good, so I tried 5e-5, which turned out to be slightly better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.Nadam(lr=5e-5)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the CIFAR10 dataset. We also want to use early stopping, so we need a validation set. Let's use the first 5,000 images of the original training set as the validation set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.cifar10.load_data()\n",
    "\n",
    "X_train = X_train_full[5000:]\n",
    "y_train = y_train_full[5000:]\n",
    "X_valid = X_train_full[:5000]\n",
    "y_valid = y_train_full[:5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create the callbacks we need and train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-71944bb8fa193bc4\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-71944bb8fa193bc4\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          url.port = 6006;\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "  "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_cifar10_logs --port=6006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/100\n",
      "45000/45000 [==============================] - 9s 208us/sample - loss: 4.0892 - accuracy: 0.1672 - val_loss: 2.1601 - val_accuracy: 0.2020\n",
      "Epoch 2/100\n",
      "45000/45000 [==============================] - 8s 175us/sample - loss: 2.0558 - accuracy: 0.2469 - val_loss: 1.9845 - val_accuracy: 0.2808\n",
      "Epoch 3/100\n",
      "45000/45000 [==============================] - 7s 161us/sample - loss: 1.9306 - accuracy: 0.2942 - val_loss: 1.9089 - val_accuracy: 0.2934\n",
      "Epoch 4/100\n",
      "45000/45000 [==============================] - 7s 157us/sample - loss: 1.8437 - accuracy: 0.3332 - val_loss: 1.7957 - val_accuracy: 0.3510\n",
      "Epoch 5/100\n",
      "45000/45000 [==============================] - 7s 162us/sample - loss: 1.7829 - accuracy: 0.3560 - val_loss: 1.7779 - val_accuracy: 0.3582\n",
      "Epoch 6/100\n",
      "45000/45000 [==============================] - 7s 154us/sample - loss: 1.7374 - accuracy: 0.3702 - val_loss: 1.8059 - val_accuracy: 0.3540\n",
      "Epoch 7/100\n",
      "45000/45000 [==============================] - 8s 171us/sample - loss: 1.6938 - accuracy: 0.3881 - val_loss: 1.7060 - val_accuracy: 0.3796\n",
      "Epoch 8/100\n",
      "45000/45000 [==============================] - 8s 174us/sample - loss: 1.6624 - accuracy: 0.4014 - val_loss: 1.7063 - val_accuracy: 0.3806\n",
      "Epoch 9/100\n",
      "45000/45000 [==============================] - 8s 180us/sample - loss: 1.6328 - accuracy: 0.4108 - val_loss: 1.6685 - val_accuracy: 0.4108\n",
      "Epoch 10/100\n",
      "45000/45000 [==============================] - 8s 176us/sample - loss: 1.6090 - accuracy: 0.4192 - val_loss: 1.6384 - val_accuracy: 0.4046\n",
      "Epoch 11/100\n",
      "45000/45000 [==============================] - 8s 172us/sample - loss: 1.5826 - accuracy: 0.4304 - val_loss: 1.6340 - val_accuracy: 0.4084\n",
      "Epoch 12/100\n",
      "45000/45000 [==============================] - 9s 194us/sample - loss: 1.5638 - accuracy: 0.4374 - val_loss: 1.6190 - val_accuracy: 0.4134\n",
      "Epoch 13/100\n",
      "45000/45000 [==============================] - 9s 197us/sample - loss: 1.5454 - accuracy: 0.4459 - val_loss: 1.5927 - val_accuracy: 0.4326\n",
      "Epoch 14/100\n",
      "45000/45000 [==============================] - 9s 196us/sample - loss: 1.5266 - accuracy: 0.4523 - val_loss: 1.6527 - val_accuracy: 0.4210\n",
      "Epoch 15/100\n",
      "45000/45000 [==============================] - 9s 203us/sample - loss: 1.5140 - accuracy: 0.4534 - val_loss: 1.5902 - val_accuracy: 0.4352\n",
      "Epoch 16/100\n",
      "45000/45000 [==============================] - 9s 196us/sample - loss: 1.4965 - accuracy: 0.4622 - val_loss: 1.6270 - val_accuracy: 0.4146\n",
      "Epoch 17/100\n",
      "45000/45000 [==============================] - 9s 199us/sample - loss: 1.4846 - accuracy: 0.4677 - val_loss: 1.6368 - val_accuracy: 0.4066\n",
      "<<49 more lines>>\n",
      "45000/45000 [==============================] - 10s 214us/sample - loss: 1.2420 - accuracy: 0.5531 - val_loss: 1.5280 - val_accuracy: 0.4594\n",
      "Epoch 43/100\n",
      "45000/45000 [==============================] - 10s 223us/sample - loss: 1.2347 - accuracy: 0.5564 - val_loss: 1.5307 - val_accuracy: 0.4692\n",
      "Epoch 44/100\n",
      "45000/45000 [==============================] - 10s 222us/sample - loss: 1.2271 - accuracy: 0.5579 - val_loss: 1.5747 - val_accuracy: 0.4626\n",
      "Epoch 45/100\n",
      "45000/45000 [==============================] - 10s 217us/sample - loss: 1.2224 - accuracy: 0.5587 - val_loss: 1.5249 - val_accuracy: 0.4678\n",
      "Epoch 46/100\n",
      "45000/45000 [==============================] - 10s 217us/sample - loss: 1.2108 - accuracy: 0.5636 - val_loss: 1.5235 - val_accuracy: 0.4786\n",
      "Epoch 47/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 1.2049 - accuracy: 0.5655 - val_loss: 1.5776 - val_accuracy: 0.4594\n",
      "Epoch 48/100\n",
      "45000/45000 [==============================] - 10s 211us/sample - loss: 1.1984 - accuracy: 0.5687 - val_loss: 1.5269 - val_accuracy: 0.4690\n",
      "Epoch 49/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 1.1893 - accuracy: 0.5736 - val_loss: 1.5449 - val_accuracy: 0.4682\n",
      "Epoch 50/100\n",
      "45000/45000 [==============================] - 10s 213us/sample - loss: 1.1850 - accuracy: 0.5741 - val_loss: 1.5345 - val_accuracy: 0.4800\n",
      "Epoch 51/100\n",
      "45000/45000 [==============================] - 9s 205us/sample - loss: 1.1772 - accuracy: 0.5749 - val_loss: 1.5430 - val_accuracy: 0.4660\n",
      "Epoch 52/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 1.1701 - accuracy: 0.5777 - val_loss: 1.5470 - val_accuracy: 0.4684\n",
      "Epoch 53/100\n",
      "45000/45000 [==============================] - 10s 217us/sample - loss: 1.1628 - accuracy: 0.5802 - val_loss: 1.6219 - val_accuracy: 0.4622\n",
      "Epoch 54/100\n",
      "45000/45000 [==============================] - 10s 228us/sample - loss: 1.1582 - accuracy: 0.5820 - val_loss: 1.5455 - val_accuracy: 0.4778\n",
      "Epoch 55/100\n",
      "45000/45000 [==============================] - 10s 222us/sample - loss: 1.1499 - accuracy: 0.5866 - val_loss: 1.6274 - val_accuracy: 0.4534\n",
      "Epoch 56/100\n",
      "45000/45000 [==============================] - 10s 222us/sample - loss: 1.1451 - accuracy: 0.5893 - val_loss: 1.5577 - val_accuracy: 0.4736\n",
      "Epoch 57/100\n",
      "45000/45000 [==============================] - 10s 223us/sample - loss: 1.1406 - accuracy: 0.5903 - val_loss: 1.6167 - val_accuracy: 0.4610\n",
      "Epoch 58/100\n",
      "45000/45000 [==============================] - 10s 231us/sample - loss: 1.1372 - accuracy: 0.5896 - val_loss: 1.6059 - val_accuracy: 0.4564\n",
      "Epoch 59/100\n",
      "45000/45000 [==============================] - 10s 229us/sample - loss: 1.1271 - accuracy: 0.5938 - val_loss: 1.5618 - val_accuracy: 0.4660\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fb37881c5c0>"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000/5000 [==============================] - 0s 65us/sample - loss: 1.5099 - accuracy: 0.4736\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.5099372177124024, 0.4736]"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model with the lowest validation loss gets about 47% accuracy on the validation set. It took 39 epochs to reach the lowest validation loss, with roughly 10 seconds per epoch on my laptop (without a GPU). Let's see if we can improve performance using Batch Normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c.\n",
    "*Exercise: Now try adding Batch Normalization and compare the learning curves: Is it converging faster than before? Does it produce a better model? How does it affect training speed?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below is very similar to the code above, with a few changes:\n",
    "\n",
    "* I added a BN layer after every Dense layer (before the activation function), except for the output layer. I also added a BN layer before the first hidden layer.\n",
    "* I changed the learning rate to 5e-4. I experimented with 1e-5, 3e-5, 5e-5, 1e-4, 3e-4, 5e-4, 1e-3 and 3e-3, and I chose the one with the best validation performance after 20 epochs.\n",
    "* I renamed the run directories to run_bn_* and the model file name to my_cifar10_bn_model.h5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/100\n",
      "45000/45000 [==============================] - 21s 466us/sample - loss: 1.8365 - accuracy: 0.3390 - val_loss: 1.6330 - val_accuracy: 0.4174\n",
      "Epoch 2/100\n",
      "45000/45000 [==============================] - 16s 352us/sample - loss: 1.6623 - accuracy: 0.4063 - val_loss: 1.5967 - val_accuracy: 0.4204\n",
      "Epoch 3/100\n",
      "45000/45000 [==============================] - 16s 355us/sample - loss: 1.5946 - accuracy: 0.4314 - val_loss: 1.5225 - val_accuracy: 0.4602\n",
      "Epoch 4/100\n",
      "45000/45000 [==============================] - 17s 367us/sample - loss: 1.5417 - accuracy: 0.4551 - val_loss: 1.4680 - val_accuracy: 0.4756\n",
      "Epoch 5/100\n",
      "45000/45000 [==============================] - 17s 367us/sample - loss: 1.5013 - accuracy: 0.4678 - val_loss: 1.4378 - val_accuracy: 0.4862\n",
      "Epoch 6/100\n",
      "45000/45000 [==============================] - 16s 361us/sample - loss: 1.4637 - accuracy: 0.4797 - val_loss: 1.4221 - val_accuracy: 0.4982\n",
      "Epoch 7/100\n",
      "45000/45000 [==============================] - 16s 355us/sample - loss: 1.4361 - accuracy: 0.4921 - val_loss: 1.4133 - val_accuracy: 0.4968\n",
      "Epoch 8/100\n",
      "45000/45000 [==============================] - 15s 326us/sample - loss: 1.4078 - accuracy: 0.4998 - val_loss: 1.3916 - val_accuracy: 0.5040\n",
      "Epoch 9/100\n",
      "45000/45000 [==============================] - 14s 315us/sample - loss: 1.3811 - accuracy: 0.5104 - val_loss: 1.3695 - val_accuracy: 0.5116\n",
      "Epoch 10/100\n",
      "45000/45000 [==============================] - 14s 318us/sample - loss: 1.3571 - accuracy: 0.5205 - val_loss: 1.3701 - val_accuracy: 0.5112\n",
      "Epoch 11/100\n",
      "45000/45000 [==============================] - 15s 329us/sample - loss: 1.3367 - accuracy: 0.5246 - val_loss: 1.3549 - val_accuracy: 0.5196\n",
      "Epoch 12/100\n",
      "45000/45000 [==============================] - 14s 316us/sample - loss: 1.3158 - accuracy: 0.5322 - val_loss: 1.4038 - val_accuracy: 0.5048\n",
      "Epoch 13/100\n",
      "45000/45000 [==============================] - 15s 328us/sample - loss: 1.3028 - accuracy: 0.5392 - val_loss: 1.3453 - val_accuracy: 0.5242\n",
      "Epoch 14/100\n",
      "45000/45000 [==============================] - 15s 331us/sample - loss: 1.2798 - accuracy: 0.5460 - val_loss: 1.3427 - val_accuracy: 0.5218\n",
      "Epoch 15/100\n",
      "45000/45000 [==============================] - 15s 327us/sample - loss: 1.2642 - accuracy: 0.5502 - val_loss: 1.3802 - val_accuracy: 0.5072\n",
      "Epoch 16/100\n",
      "45000/45000 [==============================] - 15s 336us/sample - loss: 1.2497 - accuracy: 0.5592 - val_loss: 1.3870 - val_accuracy: 0.5154\n",
      "Epoch 17/100\n",
      "45000/45000 [==============================] - 15s 332us/sample - loss: 1.2339 - accuracy: 0.5645 - val_loss: 1.3270 - val_accuracy: 0.5366\n",
      "Epoch 18/100\n",
      "45000/45000 [==============================] - 15s 331us/sample - loss: 1.2223 - accuracy: 0.5688 - val_loss: 1.3054 - val_accuracy: 0.5506\n",
      "Epoch 19/100\n",
      "45000/45000 [==============================] - 15s 339us/sample - loss: 1.2015 - accuracy: 0.5750 - val_loss: 1.3134 - val_accuracy: 0.5462\n",
      "Epoch 20/100\n",
      "45000/45000 [==============================] - 15s 326us/sample - loss: 1.1884 - accuracy: 0.5796 - val_loss: 1.3459 - val_accuracy: 0.5252\n",
      "Epoch 21/100\n",
      "45000/45000 [==============================] - 17s 370us/sample - loss: 1.1767 - accuracy: 0.5876 - val_loss: 1.3404 - val_accuracy: 0.5392\n",
      "Epoch 22/100\n",
      "45000/45000 [==============================] - 16s 366us/sample - loss: 1.1679 - accuracy: 0.5872 - val_loss: 1.3600 - val_accuracy: 0.5332\n",
      "Epoch 23/100\n",
      "45000/45000 [==============================] - 15s 337us/sample - loss: 1.1513 - accuracy: 0.5954 - val_loss: 1.3148 - val_accuracy: 0.5498\n",
      "Epoch 24/100\n",
      "45000/45000 [==============================] - 16s 346us/sample - loss: 1.1345 - accuracy: 0.6033 - val_loss: 1.3290 - val_accuracy: 0.5368\n",
      "Epoch 25/100\n",
      "45000/45000 [==============================] - 16s 350us/sample - loss: 1.1252 - accuracy: 0.6025 - val_loss: 1.3350 - val_accuracy: 0.5434\n",
      "Epoch 26/100\n",
      "45000/45000 [==============================] - 15s 341us/sample - loss: 1.1192 - accuracy: 0.6070 - val_loss: 1.3423 - val_accuracy: 0.5364\n",
      "Epoch 27/100\n",
      "45000/45000 [==============================] - 15s 342us/sample - loss: 1.1028 - accuracy: 0.6093 - val_loss: 1.3511 - val_accuracy: 0.5358\n",
      "Epoch 28/100\n",
      "45000/45000 [==============================] - 15s 332us/sample - loss: 1.0907 - accuracy: 0.6158 - val_loss: 1.3706 - val_accuracy: 0.5350\n",
      "Epoch 29/100\n",
      "45000/45000 [==============================] - 16s 345us/sample - loss: 1.0785 - accuracy: 0.6197 - val_loss: 1.3356 - val_accuracy: 0.5398\n",
      "Epoch 30/100\n",
      "45000/45000 [==============================] - 16s 352us/sample - loss: 1.0718 - accuracy: 0.6198 - val_loss: 1.3529 - val_accuracy: 0.5446\n",
      "Epoch 31/100\n",
      "45000/45000 [==============================] - 15s 333us/sample - loss: 1.0629 - accuracy: 0.6259 - val_loss: 1.3590 - val_accuracy: 0.5434\n",
      "Epoch 32/100\n",
      "45000/45000 [==============================] - 15s 331us/sample - loss: 1.0504 - accuracy: 0.6292 - val_loss: 1.3448 - val_accuracy: 0.5388\n",
      "Epoch 33/100\n",
      "45000/45000 [==============================] - 15s 325us/sample - loss: 1.0420 - accuracy: 0.6318 - val_loss: 1.3790 - val_accuracy: 0.5350\n",
      "Epoch 34/100\n",
      "45000/45000 [==============================] - 16s 346us/sample - loss: 1.0304 - accuracy: 0.6362 - val_loss: 1.3621 - val_accuracy: 0.5430\n",
      "Epoch 35/100\n",
      "45000/45000 [==============================] - 16s 356us/sample - loss: 1.0280 - accuracy: 0.6362 - val_loss: 1.3673 - val_accuracy: 0.5366\n",
      "Epoch 36/100\n",
      "45000/45000 [==============================] - 16s 354us/sample - loss: 1.0100 - accuracy: 0.6439 - val_loss: 1.3659 - val_accuracy: 0.5420\n",
      "Epoch 37/100\n",
      "45000/45000 [==============================] - 15s 329us/sample - loss: 1.0060 - accuracy: 0.6473 - val_loss: 1.3773 - val_accuracy: 0.5398\n",
      "Epoch 38/100\n",
      "45000/45000 [==============================] - 15s 332us/sample - loss: 0.9966 - accuracy: 0.6496 - val_loss: 1.3946 - val_accuracy: 0.5340\n",
      "5000/5000 [==============================] - 1s 157us/sample - loss: 1.3054 - accuracy: 0.5506\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.305354326057434, 0.5506]"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "model.add(keras.layers.BatchNormalization())\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100, kernel_initializer=\"he_normal\"))\n",
    "    model.add(keras.layers.BatchNormalization())\n",
    "    model.add(keras.layers.Activation(\"elu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_bn_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_bn_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_bn_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* *Is the model converging faster than before?* Much faster! The previous model took 39 epochs to reach the lowest validation loss, while the new model with BN took 18 epochs. That's more than twice as fast as the previous model. The BN layers stabilized training and allowed us to use a much larger learning rate, so convergence was faster.\n",
    "* *Does BN produce a better model?* Yes! The final model is also much better, with 55% accuracy instead of 47%. It's still not a very good model, but at least it's much better than before (a Convolutional Neural Network would do much better, but that's a different topic, see chapter 14).\n",
    "* *How does BN affect training speed?* Although the model converged twice as fast, each epoch took about 16s instead of 10s, because of the extra computations required by the BN layers. So overall, although the number of epochs was reduced by 50%, the training time (wall time) was shortened by 30%. Which is still pretty significant!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### d.\n",
    "*Exercise: Try replacing Batch Normalization with SELU, and make the necessary adjustements to ensure the network self-normalizes (i.e., standardize the input features, use LeCun normal initialization, make sure the DNN contains only a sequence of dense layers, etc.).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/100\n",
      "45000/45000 [==============================] - 12s 268us/sample - loss: 1.9310 - accuracy: 0.3055 - val_loss: 1.7814 - val_accuracy: 0.3566\n",
      "Epoch 2/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 1.7077 - accuracy: 0.3935 - val_loss: 1.9245 - val_accuracy: 0.3654\n",
      "Epoch 3/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 1.6116 - accuracy: 0.4314 - val_loss: 1.6612 - val_accuracy: 0.4316\n",
      "Epoch 4/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 1.5385 - accuracy: 0.4590 - val_loss: 1.5879 - val_accuracy: 0.4414\n",
      "Epoch 5/100\n",
      "45000/45000 [==============================] - 11s 234us/sample - loss: 1.4911 - accuracy: 0.4746 - val_loss: 1.5470 - val_accuracy: 0.4626\n",
      "Epoch 6/100\n",
      "45000/45000 [==============================] - 9s 205us/sample - loss: 1.4419 - accuracy: 0.4964 - val_loss: 1.5535 - val_accuracy: 0.4662\n",
      "Epoch 7/100\n",
      "45000/45000 [==============================] - 9s 192us/sample - loss: 1.3981 - accuracy: 0.5112 - val_loss: 1.5027 - val_accuracy: 0.4772\n",
      "Epoch 8/100\n",
      "45000/45000 [==============================] - 9s 196us/sample - loss: 1.3598 - accuracy: 0.5258 - val_loss: 1.5501 - val_accuracy: 0.4762\n",
      "Epoch 9/100\n",
      "45000/45000 [==============================] - 9s 207us/sample - loss: 1.3212 - accuracy: 0.5420 - val_loss: 1.5219 - val_accuracy: 0.4858\n",
      "Epoch 10/100\n",
      "45000/45000 [==============================] - 10s 232us/sample - loss: 1.2949 - accuracy: 0.5514 - val_loss: 1.4786 - val_accuracy: 0.4968\n",
      "Epoch 11/100\n",
      "45000/45000 [==============================] - 10s 228us/sample - loss: 1.2686 - accuracy: 0.5620 - val_loss: 1.4878 - val_accuracy: 0.4994\n",
      "Epoch 12/100\n",
      "45000/45000 [==============================] - 9s 210us/sample - loss: 1.2334 - accuracy: 0.5747 - val_loss: 1.5364 - val_accuracy: 0.4926\n",
      "Epoch 13/100\n",
      "45000/45000 [==============================] - 10s 215us/sample - loss: 1.2144 - accuracy: 0.5812 - val_loss: 1.4626 - val_accuracy: 0.5140\n",
      "Epoch 14/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 1.1909 - accuracy: 0.5906 - val_loss: 1.4844 - val_accuracy: 0.5078\n",
      "Epoch 15/100\n",
      "45000/45000 [==============================] - 10s 221us/sample - loss: 1.1606 - accuracy: 0.5990 - val_loss: 1.5233 - val_accuracy: 0.4972\n",
      "Epoch 16/100\n",
      "45000/45000 [==============================] - 10s 221us/sample - loss: 1.1447 - accuracy: 0.6077 - val_loss: 1.4782 - val_accuracy: 0.5060\n",
      "Epoch 17/100\n",
      "45000/45000 [==============================] - 10s 221us/sample - loss: 1.1154 - accuracy: 0.6176 - val_loss: 1.4666 - val_accuracy: 0.5162\n",
      "Epoch 18/100\n",
      "45000/45000 [==============================] - 10s 212us/sample - loss: 1.0949 - accuracy: 0.6258 - val_loss: 1.4978 - val_accuracy: 0.5108\n",
      "Epoch 19/100\n",
      "45000/45000 [==============================] - 9s 207us/sample - loss: 1.0778 - accuracy: 0.6321 - val_loss: 1.5461 - val_accuracy: 0.5130\n",
      "Epoch 20/100\n",
      "45000/45000 [==============================] - 9s 210us/sample - loss: 1.0552 - accuracy: 0.6406 - val_loss: 1.5072 - val_accuracy: 0.5190\n",
      "Epoch 21/100\n",
      "45000/45000 [==============================] - 9s 207us/sample - loss: 1.0361 - accuracy: 0.6492 - val_loss: 1.4997 - val_accuracy: 0.5126\n",
      "Epoch 22/100\n",
      "45000/45000 [==============================] - 9s 209us/sample - loss: 1.0229 - accuracy: 0.6517 - val_loss: 1.5663 - val_accuracy: 0.5066\n",
      "Epoch 23/100\n",
      "45000/45000 [==============================] - 9s 202us/sample - loss: 1.0040 - accuracy: 0.6560 - val_loss: 1.5809 - val_accuracy: 0.5098\n",
      "Epoch 24/100\n",
      "45000/45000 [==============================] - 9s 200us/sample - loss: 0.9848 - accuracy: 0.6656 - val_loss: 1.5104 - val_accuracy: 0.5122\n",
      "Epoch 25/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 0.9657 - accuracy: 0.6743 - val_loss: 1.6135 - val_accuracy: 0.5196\n",
      "Epoch 26/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 0.9530 - accuracy: 0.6764 - val_loss: 1.6536 - val_accuracy: 0.5070\n",
      "Epoch 27/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 0.9296 - accuracy: 0.6857 - val_loss: 1.6331 - val_accuracy: 0.5162\n",
      "Epoch 28/100\n",
      "45000/45000 [==============================] - 9s 204us/sample - loss: 0.9220 - accuracy: 0.6887 - val_loss: 1.5864 - val_accuracy: 0.5104\n",
      "Epoch 29/100\n",
      "45000/45000 [==============================] - 9s 201us/sample - loss: 0.9126 - accuracy: 0.6907 - val_loss: 1.6421 - val_accuracy: 0.5106\n",
      "Epoch 30/100\n",
      "45000/45000 [==============================] - 9s 201us/sample - loss: 0.9011 - accuracy: 0.6963 - val_loss: 1.6751 - val_accuracy: 0.5088\n",
      "Epoch 31/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 1.4447 - accuracy: 0.5958 - val_loss: 1.5772 - val_accuracy: 0.4806\n",
      "Epoch 32/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 1.1083 - accuracy: 0.6158 - val_loss: 1.6008 - val_accuracy: 0.4988\n",
      "Epoch 33/100\n",
      "45000/45000 [==============================] - 9s 207us/sample - loss: 1.0031 - accuracy: 0.6533 - val_loss: 1.6117 - val_accuracy: 0.5120\n",
      "5000/5000 [==============================] - 0s 66us/sample - loss: 1.4626 - accuracy: 0.5140\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.462584439086914, 0.514]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=7e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_selu_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_selu_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000/5000 [==============================] - 0s 74us/sample - loss: 1.4626 - accuracy: 0.5140\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.462584439086914, 0.514]"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_selu_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get 51.4% accuracy, which is better than the original model, but not quite as good as the model using batch normalization. Moreover, it took 13 epochs to reach the best model, which is much faster than both the original model and the BN model, plus each epoch took only 10 seconds, just like the original model. So it's by far the fastest model to train (both in terms of epochs and wall time)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### e.\n",
    "*Exercise: Try regularizing the model with alpha dropout. Then, without retraining your model, see if you can achieve better accuracy using MC Dropout.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/100\n",
      "45000/45000 [==============================] - 12s 263us/sample - loss: 1.8763 - accuracy: 0.3330 - val_loss: 1.7595 - val_accuracy: 0.3668\n",
      "Epoch 2/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 1.6527 - accuracy: 0.4148 - val_loss: 1.7666 - val_accuracy: 0.3808\n",
      "Epoch 3/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 1.5682 - accuracy: 0.4439 - val_loss: 1.6393 - val_accuracy: 0.4490\n",
      "Epoch 4/100\n",
      "45000/45000 [==============================] - 10s 211us/sample - loss: 1.5030 - accuracy: 0.4698 - val_loss: 1.6028 - val_accuracy: 0.4466\n",
      "Epoch 5/100\n",
      "45000/45000 [==============================] - 9s 209us/sample - loss: 1.4430 - accuracy: 0.4913 - val_loss: 1.5394 - val_accuracy: 0.4562\n",
      "Epoch 6/100\n",
      "45000/45000 [==============================] - 10s 215us/sample - loss: 1.4005 - accuracy: 0.5084 - val_loss: 1.5408 - val_accuracy: 0.4818\n",
      "Epoch 7/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 1.3541 - accuracy: 0.5298 - val_loss: 1.5236 - val_accuracy: 0.4866\n",
      "Epoch 8/100\n",
      "45000/45000 [==============================] - 10s 214us/sample - loss: 1.3189 - accuracy: 0.5405 - val_loss: 1.5174 - val_accuracy: 0.4926\n",
      "Epoch 9/100\n",
      "45000/45000 [==============================] - 10s 212us/sample - loss: 1.2800 - accuracy: 0.5570 - val_loss: 1.5722 - val_accuracy: 0.4998\n",
      "Epoch 10/100\n",
      "45000/45000 [==============================] - 10s 214us/sample - loss: 1.2512 - accuracy: 0.5656 - val_loss: 1.4974 - val_accuracy: 0.5082\n",
      "Epoch 11/100\n",
      "45000/45000 [==============================] - 9s 203us/sample - loss: 1.2141 - accuracy: 0.5802 - val_loss: 1.6123 - val_accuracy: 0.4916\n",
      "Epoch 12/100\n",
      "45000/45000 [==============================] - 9s 201us/sample - loss: 1.1856 - accuracy: 0.5893 - val_loss: 1.5449 - val_accuracy: 0.5016\n",
      "Epoch 13/100\n",
      "45000/45000 [==============================] - 9s 204us/sample - loss: 1.1602 - accuracy: 0.5978 - val_loss: 1.6241 - val_accuracy: 0.5056\n",
      "Epoch 14/100\n",
      "45000/45000 [==============================] - 9s 199us/sample - loss: 1.1290 - accuracy: 0.6118 - val_loss: 1.6085 - val_accuracy: 0.4936\n",
      "Epoch 15/100\n",
      "45000/45000 [==============================] - 9s 198us/sample - loss: 1.1050 - accuracy: 0.6176 - val_loss: 1.6951 - val_accuracy: 0.4860\n",
      "Epoch 16/100\n",
      "45000/45000 [==============================] - 9s 201us/sample - loss: 1.0786 - accuracy: 0.6293 - val_loss: 1.5806 - val_accuracy: 0.5044\n",
      "Epoch 17/100\n",
      "45000/45000 [==============================] - 10s 212us/sample - loss: 1.0629 - accuracy: 0.6362 - val_loss: 1.5932 - val_accuracy: 0.4970\n",
      "Epoch 18/100\n",
      "45000/45000 [==============================] - 10s 215us/sample - loss: 1.0330 - accuracy: 0.6458 - val_loss: 1.5968 - val_accuracy: 0.5080\n",
      "Epoch 19/100\n",
      "45000/45000 [==============================] - 9s 195us/sample - loss: 1.0104 - accuracy: 0.6488 - val_loss: 1.6166 - val_accuracy: 0.5152\n",
      "Epoch 20/100\n",
      "45000/45000 [==============================] - 9s 206us/sample - loss: 0.9896 - accuracy: 0.6629 - val_loss: 1.6174 - val_accuracy: 0.5154\n",
      "Epoch 21/100\n",
      "45000/45000 [==============================] - 9s 211us/sample - loss: 0.9741 - accuracy: 0.6650 - val_loss: 1.7201 - val_accuracy: 0.5040\n",
      "Epoch 22/100\n",
      "45000/45000 [==============================] - 10s 220us/sample - loss: 0.9475 - accuracy: 0.6769 - val_loss: 1.7498 - val_accuracy: 0.5176\n",
      "Epoch 23/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 0.9346 - accuracy: 0.6780 - val_loss: 1.7491 - val_accuracy: 0.5020\n",
      "Epoch 24/100\n",
      "45000/45000 [==============================] - 10s 223us/sample - loss: 1.1878 - accuracy: 0.6792 - val_loss: 1.6664 - val_accuracy: 0.4906\n",
      "Epoch 25/100\n",
      "45000/45000 [==============================] - 10s 219us/sample - loss: 0.9851 - accuracy: 0.6646 - val_loss: 1.7358 - val_accuracy: 0.5086\n",
      "Epoch 26/100\n",
      "45000/45000 [==============================] - 10s 220us/sample - loss: 0.9053 - accuracy: 0.6911 - val_loss: 1.8361 - val_accuracy: 0.5094\n",
      "Epoch 27/100\n",
      "45000/45000 [==============================] - 10s 215us/sample - loss: 0.8681 - accuracy: 0.7048 - val_loss: 1.8487 - val_accuracy: 0.5036\n",
      "Epoch 28/100\n",
      "45000/45000 [==============================] - 10s 220us/sample - loss: 0.8460 - accuracy: 0.7132 - val_loss: 1.8516 - val_accuracy: 0.5068\n",
      "Epoch 29/100\n",
      "45000/45000 [==============================] - 10s 223us/sample - loss: 0.8258 - accuracy: 0.7208 - val_loss: 1.9383 - val_accuracy: 0.5094\n",
      "Epoch 30/100\n",
      "45000/45000 [==============================] - 10s 216us/sample - loss: 0.8106 - accuracy: 0.7248 - val_loss: 2.0527 - val_accuracy: 0.4974\n",
      "5000/5000 [==============================] - 0s 71us/sample - loss: 1.4974 - accuracy: 0.5082\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.4974345008850098, 0.5082]"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.Nadam(lr=5e-4)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_alpha_dropout_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_alpha_dropout_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]\n",
    "\n",
    "X_means = X_train.mean(axis=0)\n",
    "X_stds = X_train.std(axis=0)\n",
    "X_train_scaled = (X_train - X_means) / X_stds\n",
    "X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=100,\n",
    "          validation_data=(X_valid_scaled, y_valid),\n",
    "          callbacks=callbacks)\n",
    "\n",
    "model = keras.models.load_model(\"my_cifar10_alpha_dropout_model.h5\")\n",
    "model.evaluate(X_valid_scaled, y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model reaches 50.8% accuracy on the validation set. That's very slightly worse than without dropout (51.4%). With an extensive hyperparameter search, it might be possible to do better (I tried dropout rates of 5%, 10%, 20% and 40%, and learning rates 1e-4, 3e-4, 5e-4, and 1e-3), but probably not much better in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use MC Dropout now. We will need the `MCAlphaDropout` class we used earlier, so let's just copy it here for convenience:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCAlphaDropout(keras.layers.AlphaDropout):\n",
    "    def call(self, inputs):\n",
    "        return super().call(inputs, training=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a new model, identical to the one we just trained (with the same weights), but with `MCAlphaDropout` dropout layers instead of `AlphaDropout` layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "mc_model = keras.models.Sequential([\n",
    "    MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout) else layer\n",
    "    for layer in model.layers\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then let's add a couple utility functions. The first will run the model many times (10 by default) and it will return the mean predicted class probabilities. The second will use these mean probabilities to predict the most likely class for each instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mc_dropout_predict_probas(mc_model, X, n_samples=10):\n",
    "    Y_probas = [mc_model.predict(X) for sample in range(n_samples)]\n",
    "    return np.mean(Y_probas, axis=0)\n",
    "\n",
    "def mc_dropout_predict_classes(mc_model, X, n_samples=10):\n",
    "    Y_probas = mc_dropout_predict_probas(mc_model, X, n_samples)\n",
    "    return np.argmax(Y_probas, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make predictions for all the instances in the validation set, and compute the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5094"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "y_pred = mc_dropout_predict_classes(mc_model, X_valid_scaled)\n",
    "accuracy = np.mean(y_pred == y_valid[:, 0])\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only get virtually no accuracy improvement in this case (from 50.8% to 50.9%).\n",
    "\n",
    "So the best model we got in this exercise is the Batch Normalization model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### f.\n",
    "*Exercise: Retrain your model using 1cycle scheduling and see if it improves training speed and model accuracy.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(lr=1e-3)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples\n",
      "45000/45000 [==============================] - 3s 60us/sample - loss: nan - accuracy: 0.1403\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1e-05, 9.999868, 2.0895472, 3.482099260602679]"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOydeZwkdX33P986+px79r5hF1huhFUIhyhHuBQPFIWYxxh98IgawpMHI4qSaNQQJQlGSVB8REWFKIogqCiXCILLzS67sCx7zh6zu3P39FX1e/6o+lX/qrq6p3ump6/9vl+vee10dVX1r2Zm61Pfm4QQYBiGYZgwtEYvgGEYhmleWCQYhmGYkrBIMAzDMCVhkWAYhmFKwiLBMAzDlIRFgmEYhimJ0egF1JI5c+aIFStWNHoZDMNUwGTOwqa94yACjlzQhfW7RrGwO4Y5HdFGL+2g46mnntonhJgb9l5bicSKFSuwdu3aRi+DYZgKeHHnCN7y9UcRMzVcdeGRuPaudbj2LUfhg6cf0uilHXQQ0dZS77G7iWGYhmC7hbzpnI1r71oHAKBGLogJhUWCYZiGYIc0eyBWiaaDRYJhmIZgc0ugloBFgmGYhhDWN44NieaDRYJhmIYQZkgQ+5uaDhYJhmEaQlhMgmk+WCQYhmkIYTEJNiSaj7qKBBH9gIh2EdEoEb1MRB+q4JgHiEgQUVvVdDDMwU6oSDRgHUx56m1JfBnACiFEF4CLAXyRiE4qtTMR/QXarOCPYRgHTm5qDeoqEkKIdUKIjHzpfq0M25eIugF8HsDVdVoewzB1JDQFlv1NTUfdYxJE9E0iSgHYAGAXgHtL7PolADcB2F2vtTEMUz9CNaL+y2CmoO4iIYT4GIBOAGcAuBNAJrgPEa0BcBqAr091PiK6gojWEtHawcHBWi+XYZhZggPXrUFDspuEEJYQ4lEASwB8VH2PiDQA3wTwt0KIfAXnulkIsUYIsWbu3NAmhgzDNCEck2gNGp0Ca6A4JtEFYA2A24loN4A/udt3ENEZ9VwcwzCzR3h2E5sSzUbdMoeIaB6AswDcA2ASwDkALgNweWDXEQCLlNdLATwJ4CQA7E9imDaBG/y1BvVMLxVwXEv/BceC2QrgSiHEXUS0DMB6AEcJIbZBCVYTUcz9dk8l7ieGYVoDbvDXGtRNJIQQgwDOLPHeNgAdJd7bAk56YJi2g7ObWoNGxyQYhjlICe0CyyrRdLBIMAzTELjBX2vAIsEwTEPg7KbWgEWCYZiGEN6Wo/7rYMrDIsEwTEPg5KbWgEWCYZiGIMCtwlsBFgmGYRqCbRdv4/GlzQeLBMMwDYGHDrUGLBIMwzQEjkm0BiwSDMM0BG4V3hqwSDAM0xDCDAkWieaDRYJhmIbADf5aAxYJhmEaQlhbjrzFwtFssEgwDNMQwhr85bmhU9PRdiJx//o9+PbvNzd6GQzDTIEdIgg5K6R4gmko9Rw6VBf+9/fWAgA+dMahDV4JwzDlYHdTa9B2lgTDMK1BmBzkw8qwmYbCIsEwTEMIi0nk2JJoOlgkGIZpCGEpsByTaD5YJBiGaQhhMQmLs5uajrYViTBTlmGY5iHckuD/t81GXUWCiH5ARLuIaJSIXiaiD5XY7/1E9JS73w4iup6IqsrE4icShmluwp7j2N3UfNTbkvgygBVCiC4AFwP4IhGdFLJfAsCVAOYAOBnA2QD+vpoPKvVE8qsXd2HPaLqaUzEMMwuEFtOxSDQddRUJIcQ6IURGvnS/Vobsd5MQ4vdCiKwQYieA2wCcNtX5VeMhmy/+Y0vnLHzkB0/j/d95cnoXwDBMzQgz9nPsAWg66h6TIKJvElEKwAYAuwDcW8FhbwSwbqqdVB9nNuSJZGQyBwDYzZYEwzScsJgEWxLNR91FQgjxMQCdAM4AcCeATLn9iegDANYA+GqJ968gorVEtHbfvv3e9jDf5lAqCwBIRtqu0JxhWo4wo6EvGa3/QpiyNCS7SQhhCSEeBbAEwEdL7UdEbwfwFQAXCCH2lTjXzUKINUKINT29vd72MJEYTjmWRDyiz2j9DMPMHDUmcejcJL727uPxd+ce1sAVMWE0OgXWQEhMAgCI6HwA3wLwViHEC5WczOduColJSJFIuCIxmbXw6TtfwM7hySqXzTDMTFH/v+pEuOSkJYga/ADXbNRNJIhoHhG9l4g6iEgnovMAXAbggZB9z4ITrL5ECFFxlFlt+xIek3DcTXHT+UNcv2sEP3pyG87/90equpZGkrdsvLhzpNHLYJgZo4YkdI1H0jUr9bQkBBzX0g4AQ3BiDFcKIe4iomVENE5Ey9x9rwXQDeBed/s4Ed031QeoTyZhKbBBd1Mm5wjJWDqP9QOj076wevLV37yMt3z9Uby8Z6zRS2GYGaHGJDSeW9q01C2CK4QYBHBmife2AehQXr95Op9h+UQixN3kZjcZmqONqazlvbdjKIWjFnVN52PryjPbhgAA+8ezwPwGL4ZhZoAak2BLonlpdEyipqhPJrkyMQnpiprMFURiLJ2f3cXVCFlJzv+pmFZHtfw1/ntuWtpLJBSVyIRmNzkxiWzeEYdJxZIYTedmeXW1QVpLLBJMq6M+1On859y0tJdIqO6mcpZEvnUtCZstCaZN4MB1a9BmIlH4PjRwPel3N8mYBBEwOtkaloQcFM9dbplWx+du4sB109JmIlE+cD3iuZv8lsTcjmjLWBIyJpHnHjdMi8OB69agrURCfbgOLaabDLibsnnETR1dcbNkTOLxV/fje49vqfVSp40nEtx3n2lxbHY3tQTtJRIQiJnOJQWL6YQQnuWgWhKJiI6umFFSJH7y1A58/YFNs7jq6pCBa56XwbQ6NlsSLUFbdboTwqmmTuds5CwbQxNZaBqhO24ia9mepaHGJGKuJXFgIus7177xDDYPTiBr2cgoAe5GIwPXOZu7ZTKtjT+7iUWiWWkrSwIAEm6H15xl42O3PY1r7nTaPmUU95P8Pu1aEp0xsyhw/b3Ht+Ivb3kC2byFdIjrqlImsxYe3Lh32scHkbEIi91NTMvDdRKtQFuJhBCFlhs5S2DD7lEMjDjN+2QLjmRE90QilbUQd91NwcD1RCaPTN5GKmshm7d9NRgv7hzB9gOpitb0d7c/iw/8vz9hx5B//+0HUnhpV/WtQGwOXDNtgmoMsyXRvLSZSBRiEgcmshhK5bybf9p1GXXGTGTzthOjyFqIm64lkc75si1kdpQ8Xo1x/J87nsMN979c0Zr+sGmfuzb/9i/f9xL+7vZnq75Gjkkw7QLHJFqD9hIJOH2ZIrqGVwfHAQDj6Tx2Dk9icNyZbdQVl+4oJ5Adj+joihvIWcLnkpLB7TE3oJ1W4hKpXB7jmcpSZsfc/YIpuYNjGa+4rxoKKbAck2BaG1+DPxaJpqXtAtemTjB1wubBCQDOTf60rxS6kXfFTACOZTCZtZDodSwJwCmoi7ltxKXlIMVAFZC8JUJTbMsRLO5zrJzpi8T2Ayn88IltuPzkZVMcwTDNia9OgjWiaWkrSwJwzFbT0LDdjQFMZP2ZSZ0xRxc/+7MX8MrecSe7yd02qsQlCpaE310FODf8SkRCdQkFLYnhVA4TWatqt5Hc/85nduKan71QsUXDMM0GN/hrDdpKJAQETN1xN5XqWiGthp8/OwDAmVLXHXe2yQaAQEEkZOsO1ZKwbDt0qFGQXSOFiXdqoFkI4Q1AqvYmL0VCNidspvRchqkG9b8oB66bl/YSCQEYGsHUS1+WjElI4qaOeZ0xAMDesYy3Pfjkr1oS5dxNagB86/6Uckxh/4ms5bmfqhUJKTZh4sUwrQRXXLcG7SUSAHRNQ8RwLssI+cOTloQkHjGwoNsRid0jaW970FJQb8Y52w4VifUDozjuut/gF885Vsqe0fDzqRZLtXEJaaLL6nEWCaZV4eym1qC9REIIL3ANAKvmdRTtI2MSkripozdhIqJrvpt6Lu/3V6mWhGWLopu+ZQs8vnk/AOCprc70uMmA9VHYvyAM41U2FpTuJilSmTy7m5jWhBv8tQZtJRKA88cmLYmwcaRdAUsinbNARJjXFfWJRHBoUdotxhNC+ALXn77zeZzwT/fjR09u846f3+VYJupQIzVlVRWJarvPBuPccl0M00ocmMj6/na5VXjz0lYi4aTAal5M4qiFxSIRtCRkcHlBVwy7fZZE0N3k3PDlk3wmbyOds/CjJ7cDAJ7fMeyJRG8iAsAvElnFMhmeVNxNVcQkwmZIZHIWbnroVbzjm3+o+DwM02hO/ML9eGBDoV0NWxLNS3vVScBNgdUrtyQuPHYhAGB+dwzrBwptMoIxCfnUk/fcPRb2K00Bt+xPQf6Z520bD788iFQu3JIY8lkSlcckwjKqMnkb//KrDRWfg2GaERaJ5qWuIkFEPwBwNoAkgN0ArhdCfLvEvn8H4FMA4gB+CuCjQohM2L4SGZOIuCJx5IIQkVCym7Z85SLv+wVdMTy4YS+EECCiouwmaUl4ImHZ2OdmQ/UlI9iybwIJt2/Uj57cjpd2jaIzWvgsNSYxogSuq4lJhAWpfQF1yy6b2cUwzQaR4wFgd1PzUu87ypcBrBBCdAG4GMAXieik4E5EdB6Af4AjKCsAHArgH6c6ubQkIoaG+V1R9CRM3xOKRkAyGq6L87uiSGUtz/0TzF6SloTsvprN29g/4YjESct7sXcsgy1uyqts5qe6knKWPyYRN3VoVF1MIiyjSg1c12q63p1P78BHvv9UTc7FMOUwNecWxM82zUtdfzVCiHWKNSDcr5Uhu74fwC3u/kMAvgDgr6b+AKd301mr5+E9r18GIvLFIKKG7lkZQWSwee+os7zgDVnejOUcB1sAe9x91yzv9e2rPhXJhoNqW46hVA69CRMdUWPKOokDE1mc/++P4NXB8XBLQgn+jVQxp9uyBe5Yu91XvyF5ausQHn55sOJzMUw1hGU1cTFd81J3/SaibxJRCsAGALsA3Buy29EAnlNePwdgPhH1lzu30+CP8L5TluOqcw8H4A9UR81CDUXQBSr3m8gUd30FlJiEcrMfGHaC3mtWBEWi8L2MgagxiZHJLLoTEXTGTHz3sS246o5nsXN4EmHc+8IubNg9hm//fnNodbUqHNWIxFNbh3D1T57HWjddN3jOTN4KDZQzzExRM/RkLRO35Whe6i4SQoiPAegEcAaAOwGExRk6AIwor+X3ncEdiegKIlpLRGttIWAELIVzjpzvPenHDN0TiaDbKW66IpENdzfJG7R6sx8YTiMZ0XH0om4cNq8Dn7nwSGjkD8JJ8QmzJKR1cufTO/GFu9eH/BgKLqSumFkicF0QjuDgpHJIMUyXEB5b8MwKZnZQ+5XpOlsSzU5DPIFCCEsI8SiAJQA+GrLLOAA16iy/Hws5181CiDVCiDVAcZX15996NP7mrFUAHEsiqjvB5VMO9RslyaizffuBFG7/07aQwHW4JdHfEUXM1HH/VWfif7/xUBiBvlFdbl+ofKD4ridhYt+4E8BeOTeJ1/ZNhPwY4M3e7owZPteSuq6oK3zVWBJSHILdaYGCIHI1NzMbqJXWbEk0P41OgTUQHpNYB+B4AHe4r48HsEcIsX/KE4b0HE647b+jhobuhIk7P3YqVi/wGyUyM+mHT27Hc9uHi86RDrEkdo1MYk5HxLefqZHv6Vy6m1TRGZnMoTsewYdOPwTrBkZxxIJO/PSpHaHXI1Nku+JmyZhE1NCQydvViYTM1iqRVgs419xRItDPMNPFLxKlW+gwzUHdLAkimkdE7yWiDiLS3QymywA8ELL79wB8kIiOIqJeAJ8F8N1KPifsj03OvY4ajhCcuKzX2xbcZ4/Sv0nFsyRs1ZJIo78j6v98XfPVRwTdTUIIDLvups++5Sj86IpTsKgnhrFM3rMaVEYnHbeQqWsls5vkyNaw40shrZJciEtJurDYkmBmA5+7SQauWSSalnq6mwQc19IOAEMAvgrgSiHEXUS0jIjGiWgZAAghfgXgegAPAtjqfn2+kg8JxiSAwtxrmWkURtIVCTnBLohnSSjumaxlF1sSOvncTcmIAV0jzwIZz+SRtwV6EoWivsU9CQDAzqHi4LW0JCxbhPZpyuRtkFvGNx13UzlLgtuQM7OBOlRRWv5cJ9G81M2XIIQYBHBmife2wQlWq9tuAHBDtZ8TZklIkZCWRBhyn1JDgOSNMxirkC04Cp/vF6J4RIepkycusm9Tj3Lcoh4n/XZgeBJHBlqJyEFIli1C+zRl8pYnHtUErtMhMRbvnDnpbmJLgqk9VlgKLFsSTUvblbCEupuUmEQpIobmdY8Nkozo3pN3UESCrceDMZGYqcPUNC8zyROJuGJJ9MYBFFJqVWRFds6yMZ4pBLElmZztCZh0TVWCF7gOmZVdcDexJcHUHvX/UDJiwNTJG/zFNB9tJxJ6WXdTaUsCcNqGh6EGjYPZQMEhRsG2GImIDkO1JNzmfqolMScZRUTXsGN4Eufc8DBuefQ17z3V3TSesdxjC/+hMvnCbIvq3E3u9ZRp9cExCWY2UAPXPQkT9/3tG3HRcQsbuCKmHG0nEmaIJRE1NGhU3pIASrfs6IwZodlNQHHDwKA1Ejd1GLrmHSeb+/UqN3pNI8zvjmLPSBqb9o7jC/cUaiakuylvC8+qUJ+6Utm8F0yvRiQ8F1U6j9ue2OornFOzmxim1qiWBBFh1bwO7jnWxLRdfmOYb5OIkIgYiJYJXAOFNNgg3XET+92ahmCBWVfATA6LSUR0zbNAZHO/7oT/uIRpYCLrvynbtvDadjiWhNPzSY2tjCr9mqrJbpKWxA33vwwAmNMRxXlHLwDAdRLM7KKKBIcimp+2k+9STySXvWEpzl49v+yxwbRYyfyumNcWPBjo7YoF3U1hlkShq2whJuEPeMdMrcgSUG/6eVcwOmKGTwjVpn7VNPgLZi6pmVXsbmJmE9XdxFlNzc9BYUkAwGcuOmrKY4OWhGzAt7gnjpHJHJ7dPowXdvgL7YosCb3YkjC0Qkxi+1AKiUihPYgkauhF2UmqaFi2jbF0Hp1RwydE8pjuuImxdM5rdT4V6UBQ+oArgkIIToFlZhWfu6mB62Aqo+1EolSGUiUEYxKeSLjZR2//RvH0t2BMIphd5aTAashZNh7auBf/89QOXHLikqLzRE0NI0N+kVCf5P2WREFgpLXR3xHByGAOmbw9ZYAeKLibEhEdqayFA64bTO0PlWZLgpkFVEuikgcaprG0nbtJ16Z/SfGgJREzQFRoIx5GcBxq0N0VNwsi8Z8PbMKyvgS++PZjis4TM3Wf5bBtf8rnPrIsJ3DdETV8QiT3meNWflfqcpKBaxmcPuDGXFRhYkuCmQ3UUiOOSTQ/bWdJhPVuqpRkiLspomveDTiM4FN78PM7ogYMnbBx9xgGRtL49AWrQ5/0Y6bzRC95478+iMPnF+oLpSWxLJlAWLnfXE8kcpjbWXq9EmlJSMtfWhJqE0GOSTCzgT9wzSrR7LSfSMzg0SQYuJ7TEUVnzPRuwJV9vmNJrJybxCfOOgxHL+qCqWkYcHtCvTPE1QSEp+e+vGfc+96yBcbSjrspLDW1320PUqklETyHjEmoBXRsSTCzgd/d1MCFMBXRhiIxfXdTMHD9ibNWwdAJczojJY4oRsZEklEDb3/dYmdNcltEL/mUX66vFFCwJDqjhpdOq2vkPZX1J53zTjXpThIUiT2jjoj53E1sSTCzgGpJlGqDwzQPbReTmJG7KRC4XtAdw9GLupGIGCVrKILImIRq0chtwUwolViZvlKA04hPBq7ludU23lLIxgK1Emu3HMCukeJ2H0EBGEvnkcrmi9xNti14Qh1TU9TeTSwSzU/7icQM3E3BthxqELpcXML3+a5IqcdK6yKYCaUSVuinXstENg/LFuiIml6arxo0l5bEaMDd9K7/ehzn/dsjRecOa963fzzrdzflLfzfnzyPD966tuS6GaZabEUYePph89OGIjH9S+rviPh8pGo6bbAleClM9/PVOgi5pmCfJ5VyloRGhSK8jlihTmKJm5qrrm9cEQn5nzEoHEB4vGE8k/dZGGPpPH69bjf+uHm/7z82w8wE1Xqw2UptetpPJGbgbrrw2IW4629Ow7tOcoLL6o3+TUfM8+27cm4SH37joSU/X7UkjAosibCMJ/mUlYwYXnpsZ7RQcb2iP+nt2x+SAivndYcRLKYDnD5Qqkj8cfN+jGfySGUt7AiZdcEw00F1N4W1qmeaixkFrokoDuA0AK8IIbbWZkkzYybuJlPXcNySHnz5nV24+vwjfD2SPnn2YehNRnDtz18EAPz2qjNDC4HCYhIRd1u5dsjlAtfxiO5ZEsmo4VkmyxWRSER0JCK6LyYRZkEAzpNc2Gzr8YzldZTVCN4MbgDYsHsUy/oTJdfIMJWiGg8WWxJNT1WWBBF9l4g+5n4fAfAkgN8A2EhEF8zC+qombDJdtZi6hnmdxQV0asyiVKWoFAfTCLEkyohEuYFIiYiOYbeOIRnVFUuicNOOGho6Y0agl1N4w79ScyJSmbz3npyTcdg8p1Zjw+6xkutjmGrg7KbWolpL4jwAN7rfXwygE8ACAH8N4DoA99VsZdNkNgeqT5WmChREKqKIlbypB5sBqpTrUJuIGJjIprzvpeionWQjhoaOqIGfPbMTfR0RXHz8opLpsMGgdVfMwGg6j4ms5fXS6Yw5Lq41K/qQydvYyCLB1Aifu4lFoump9rG7F8Be9/vzAfxUCLEXwI8BTN1Brw7MJCYxFaWGEqnIoLIqVtLPXzYFVjn3RccWBrAQ+QUkomveuVUhiugahHB6L9300Kv40r0vlbQkgjUSvUkn6D2hBK5l9fdRi7qwqCeGvWNpb/+9Y2lMZrnQjpkets+S4FqcZqdakdgN4Bgi0uFYFb91t3cAqHyYwSwym7NyK2mcJ+MFqrtJ3lArCVzHTA3f+IsTcfZqJ1Ae0TUvYwpwLAbZn8rwBcc1bN43AQBY3BPHUCrrjTMNNj0MioSckjeRLbibZAX2UQu73N5Thf/Yb/jn3+GDt/6pzE+BYUrjdzc1cCFMRVTrbvoOgNsBDACwAPzO3X4ygA01XNe0MWeQAjsVlYiEaRQ/5cun8nIpsLIth4xNSOshamg+4VMtCVN3pnpt2uu07/i39xyPnUOTeHVwAmu3HvAsiUggThPmbtI1wkQmX9RLZ/WCTkSUyXopN2PqsVf3l/9BMEwJbMGWRCtRlUgIIf6JiNYBWAbgf4QQMv0lD+Bfar246TCb7qZKYhJSpFR3U7WWBFAQi4ih+64pYmje64iu4c6PnYq9bkuNd7zOSd297hfrMJzKedlNZqAv1KuDjqgs7olj5/AkIrqGRETHRMZCRHc+9+NvXoVfPDfgZFPphFze+Y+9dzQz5c+AYcqhWg8cuG5+qn7sFkL8VAjxb0KIHcq2W4UQd5U7joiiRHQLEW0lojEieqZURhQ5fJGIdhLRCBE9RERHV7K+mRTTTUUlMQmvTkJ1N7nunVIztIFicShYFgFLwihYEoauoStmYtW8Tt+5uuImxtJ5r7YimGX45GsH0BE1cNySbmetuoZkxEAqm8dQKotERMffn3cEHrn6zd77crLe3jEWCWZmcFuO1qLaFNhLiejPldefI6IdRPRrIlpY7lg4Vst2AGcC6AZwLYA7iGhFyL7vhpMxdQaAPgCPA/h+JWucXUuiEpFwYxI+d5PzRF+u/5MUBykWspBPFQX5WsYkSg1Y6nED5HIkaTAG8eRrB3Di8l5PiCKGhmTUsSTWDYzgyIVdvv1NXUPOliLhWC3Btuq15OfP7MRIqilCXMwswG05WotqH7uvk98Q0YkAroGTEmsC+Fq5A4UQE0KI64QQW4QQthDiHgCvATgpZPdDADwqhNgshLAA/AAVZk/NZgpsRdlNMl6grOOQOU6tgZqyGkSKgxSikpaELyYR/uuTRXvbh5y02Uze9pr0DU1ksXHPGE4+pM8naMmogdF0DusGRnHs4m7/NSnupj2uuylRxiqaCZv2juHK25/F//3Jc7NyfqbxcJ1Ea1Ht//TlADa6378DwM+FENcT0W8A/LqaExHRfACHA1gX8vaPAbyHiA6HIyTvB/CrEue5AsAVABBZsGpWxyFWZUko7qYb3nM8XtgxElqg5527yM0kYxKaz4Vm6oSzVs/D7tE05pVoOy5FQm2lIcea/mnLAQDAGw7pww5XRCKG425aPzCKVNbCMQGRMJTAtbQkZkuMh1wLYnCc3VrtCrubWotqLYk0nAI6ADgbhRTYEWX7lBCRCeA2ALcKIcKyonYB+D0cQZqE4376u7BzCSFuFkKsEUKsme35JWGDgYKYIb2bumImTls1p+xxhcC1XywiesGSiOgaiAhL+xL41PmrSwpij2uxyDRWoDBx7k9bDiBiaDhuSbcnPhGdkIzq2O/uH7QkIrrmteuQgetK51ZUi3SNVWK1Ma2JzZZES1GtSPwewNeI6FoAawDc624/HE68YUqISIMTX8gC+HiJ3T4P4PUAlgKIAfhHAA8QUdnmQbM9VF2r4OnZmCJeUArVvQQUUmB92UwViBQQ3iNKNvR78rUDOGFpD6JK1lTE0LypfDFTw8q5Sd+xhkae71haEhOZ/KzMmZjIWO46WCTahYdfHsSv1+32XvssCe7d1PRUKxIfh3NzfxeAjwghBtztF6ACdxM5d/FbAMwHcIkQolR08ngAtwshdggh8kKI78Kp9i4bl1jW1/gGdGGWRCVoGiGia4h6lkTBovAqrKchEp1u7CCdszCRyePFgVG8YUWfc75ATAIAjlzYVdT/yjSc7Ka/+eHT+MMmpz7CFoWsrVoy6mZksSXRPnzrkc345oObvNc+S4K7wDY9Vd3J3Jv2W4UQxwshvqNsv1II8ckKTnETgCMBvFUIUa739J8AvJuI5hORRkR/CSc4vqnMMb4hPI3CDMluqpSoqRVZFGo2U7AorhRq+49TVvYDcAroXt4zBssWONZNfVUtCZmtFHQ1AU4QPmcJPLJxEABw4rIeAP7ZFbVieNJxeZXrZcW0Ftm8jawV7mLi7KbmZ1p3VSI6C85TvQCwXgjxYAXHLAfwYQAZALsV19CH4bix1gM4SgixDU5h3jwAzwJIwhGHS4QQw32BQkIAACAASURBVNNZbz0pzJOo3vW1uCeOJT3OIKGCu0lXOstWdk7VVXPqyn7cv34P0jnLq8yWnV0LrrGCJREMWsv3nc/X8N7jluKUQ/vx9LZnMZ7JY17R3jND1nbMZuU8U18ylu3V2QCAajxwTKL5qUokiGgxgJ/BSVuVrqZFRLQWwDsU91MR7ryJcne5DmXfNIC/cb9aCvXGWy13fuxU7zjV3aSHNPSrlMPcQrtM3samwXFEdM1zy0khixqa9/0xi0JEwrVqJjJ5r9ssUD54vW1/Cv/vsddwzYVHVvWzkCKR46Y+bUM27xcJn7uJYxJNT7WWxI1wejatEkK8BgBEdCicOoYb4cQq2prvfuD13k0yjOnGJAB4wWPA724qxCQq99MTOeNNZf3Fq4Pj2Lh7DCvmJLyYg1oncdqqfgwMp3HEguIkNfn5mbyNiK6hw3XrlXM3/etvNuLu5wbwZ4f248+PXlDxuuVwJXVCHtPa5CwbubxqSRSEIcGxp6anWpE4F8CbpEAAgBBiMxF9EoVmf21NcIxpkCW9CczpiOKQOTMLont1EroGvcrsJgB48brzoGvkuZg+fecLAIALjy3csNWA+Kp5nbju4vDOJ+rnRs3KLImF3U5NyIsDo1WJhLQkSg1GYlqPUjGJT52/GuceVWuHJVNrahXp5cc+lwXdMaz97DkzPo/aBVbezKNVWCcyxhCs7ZjbUSjAqzTIrhbzRXS9IpGQn/vCjurCSAWR4D+pdiGbt71iTKDgbvrg6YdU9eDDNIZqf0O/A3AjES2VG4hoGYD/APBALRd2sONvy+Hv51QNahD74uMX4YozV3qvpThMdV4z0IVWCtBXf70Rr7kzLILI9uhrtw5VFZz0RCLHItEuZAPuJvnnMJuzX5jaUe1d55MAEgA2u91ctwB4FUAcwCdqvLaDGn9bjulnTKmppF942zFY7GZPAUoK7BTnVS0NOUsbAAZG0vjOo6+FHiObGo6l817tQyUUYhLsbmoXcnnbN7RKxiRYI1qDaudJbAdwIhGdC2A1nGyl9XBSVG8AcGnNV3iQogau5X+wmVoSwQaDlQbZ1fcjhuZzYZU6VlZOA4Vq76mwbYHRNLub2o2MZSNrOU0miQi2LaDR7HdIYGrDtByCQoj7hRBfF0LcKIT4LZzW35fUdmkHN16dhD697CbvPGWExajQjRUcekREeObaczGvM1pyjnZKmYEdnIRXiols3pt9wSLRHgghvL5fsnDOEoJdTS0ER42alPmdMfzVqSvwpiPmzahOotwxlVoSkYC7CQB6kxF0u8ONwpDuJgB4cMNefPGe9VOuVW3zkZmFlh9M/VErqmWthGNJsEi0CiwSTYqmEa67+GismJOsOMAcBhHhtFX9uP6S44reiyjB8XKoloS6b2fMwFgm3JKYUCyJu58fwK2Pb5lyremscxOJmzoGxzO49L8fx7b9qSmPY5qXrGIRypkkls2WRCvBItECFCyJ6f3Huu1Dp+DS1y8t2n7qyjn4hwtWh/ZrUgnGJCRdZSyJyWwevW4MZGgii5wlpuwaKy2JnoSJnCXw5GsH8I0Hy7brYpocVSSyriVhCQGdLYmWoaLANRH9YopduqZ4n5kB1XaBrZSYqeMjSkpsKfzZTYW4SGfMxJYSKbATGQt9yQiGUjlvrsUnfvQMHn55EC9cd17oMVIkuuMmdo04LckTUa7IbWXUdhw+dxNbEi1DpdlN+yt4PzwXkpkx+iyJRKUE6yQknTGjbExicU8nXh2cwKi7zz3P7yr7OZPZgiUhKdcChWl+1ASEnGpJsEi0DBX9DxRCfGC2F8KUplDP0Jinap+7KTBxr7RIWOhNlp7pHYacStcTj3jb1H5WTOuRDbEkLBscuG4hOCbRAsyk4roWqJaEWpzXGTOQtWx86ifPY+v+gtspb9nI5G30JcNncJeqwFZjEpI4z5VoaXwxCTdwbdsC00jUYxoE/6pagJlUXNeC0paE85R/+9rt+OgPnva2p9ybfX+yYBGolJpoJ91NatEfjxtobcJiEhy4bi1YJFoA6b+dKlV1tjBKZDd1xgo38/W7Rr3sJXmz7yshEim3MeBHf/AU/vmXhfqJyRB3k9oYTmXdwAi+/8etVV0HU1/2jWewYdeY91r+Ljlw3VqwSLQAs5XdVCk+d5OS3dQV98cLXnJvCBOuCJQSCVlDsXH3mNfKHFBiEoolkSsxA/l/1u7AF+6eukCPaRzX/2oDrv7p895r6W7iwHVrwSLRAjQ8u0mb2pIAgMc3O0lwsiVHIqKHWj9SRCZzli+wWQhcF86bLyESqWweWcv2+byZ5mLnsH+MfSFwze6mVoJFogXweiw1KrtJHToUSIGVLO6J45ltQwAKIpGMGr4GgxKfSCg3+cmcBUMjrxU5UNrdJD9josxMC6axDI5lfK+9OgnB7qZWgkWiBWi0JWEo/6HVwHXSTU/tSZg4YVkPntnmDBiacPs2xSO6Nz5VRd7gJ7OWb2LZZNZG3PRbH0F308hkDo+/ut+Le5QbfMQ0llIiYbldYJnWgEWiBZA32niD5gEXJtiR7wlwQXcMa5b34huXn4jXLe3BzuFJ7B1NI+W2CU9GSlgS2TxsWyCTt4ssiVhE94nhzuFJvOe/H8f+8Qzylo0Tv3A/LvvWH7HPreJmkWhOcpaNoZS/r5d8IOA6idaibpVKRBQF8E0A5wDogzOD4hohxH0l9j8UwI0AzgSQAfAdIcTVdVpuU3H8kh7823uOxymH9jXk83WNoGtU1FHW1DX85KOnAlDGle4c8TrAJiJ6qLClMpY3YyIXiEnETd2X9nr3cwMAgDvW7sDC7phXYzE46rTtYHdTc7J/PFu0TU6nszlw3VLU05IwAGyHc9PvBnAtgDuIaEVwRyKKALgfzkjUBQCWAPhBvRbabGga4R2vW+JLRa03hkaIlrFkVsxJAgC2HUj5A9clLAnpLvJZEllHJMKK7XKWja1KR9jBcceVMcYi0ZQEXU2APybBItE61O2uI4SYEEJcJ4TYIoSwhRD3wOn3dFLI7n8FYEAIcYN7XFoI8XzIfkydiOha2dkU/ckI4qaO7QcmvZhEMmogFhJHSWUtryYizN100vJefOxNK32xkGzexv6Jwo1HxirYkmhO9o2XFgmL50m0FA17NCWi+QAOB7Au5O1TAGwhovuIaB8RPUREx5Y4zxVEtJaI1g4ODs7mkg9qDJ3KBs6JCEv74tg+lMJk1oJGjguqVHaTTHdV3U2TOQtxU4OuEa4+fzUWdMe893KWHerCYJFoTsIsCRmTYEuitWiISBCRCeA2ALcKITaE7LIEwHvhxCQWAfglgLtcN5QPIcTNQog1Qog1c+fOnc1lH9SYujZldtXS3gS2H0hhImMhETFARKHZTROZPCbdAUOqJZHOWT5RUduBZC0b+8YzSEb8ojOe4Ql2zcjgFJYE10m0DnUXCSLSAHwfQBbAx0vsNgngUSHEfUKILICvAugHcGR9VskEMXVtyrYgS/sS2DE0iVQ2j4R7Mw/Pbiq4mzJWcUxCUuxuymJ5f9J3rvESXWiZxrJvPFP0UOEFrm1A47zKlqGuvyoiIgC3AJgP4BIhRPjsS+B5ANzarYkwp3A3AcCS3jjGM3nsHJ70CuJiRkh2UzbviUTOsgs9n3IBkdDVegkb+8czWN6f8J1rIssi0YykMpY3mVCSs7ktRytSbz2/CY418FYhxGSZ/X4A4BQiOoeIdABXAtgH4KU6rJEJwZgicA04lgQAvLxnzLvZxyNhMQnLy24SAsjbzmjTdM5GLKK6mwo3ksmck3e/LCASXCfRnGTyftdhRNeQzlmwbMGB6xajnnUSywF8GE7Nw24q/JF8GMDvAawHcJQQYpsQYiMRvQ/AfwGYB+BpABe7riemAZi6VjYFFgDmdTrzI/aMZrDMFYxoaMV1IXANAId95j5cdOxCr05Corqbdo84zxRLeuLQNfLSZNnd1Jxk8rbPPWnqhJsf2YwnXzvAgesWo24iIYTYCqDcX0ZHYP87Adw5q4tiKubi4xehO15+0lxPoniiXGfU8N3UAWAsnS+aKfHLF3ZB16iku2nbAadGor8jimRE90aicnZTc1KUhGBoQNbCs9uHcfSiLg5ctxA8G5KpiI++aeWU+6g+aBm4vvzk5Th+aQ8++N21yFo2FnTFsGX/BMbSxeEoyxZY3Bv3XqvurT2jTrZMfzKCjqjhiQS7m5qToCWhZrFZPE+ipeAcA6ZmdMVMyAdEaUn0JSM447C5XnxhzYpepHM2ntsxEnqOVfMKBqURMomvvyOKDrf7bNzUWSSaFEckdHS4CQyyCh9w6yTYkmgZWCSYmqFp5LmkEoGAtcyMev0Kp//UE5sPhJ5j1VxFJELyJBf3xL3MqbmdUXY3zZBX9ozNyoS/dM5C1NDw2KfPwlOfPcf3nmVzTKKVYHcTU1N6ExEMp3JIRP0iIQvjVs3rQHfcDG3bMKcjil5lml1wpveCrhjikcLT6bzOKLYo/ZyY6nn3fz+O4VQO7339Ul/x4kzJ5G3ETB1dseI4li3A7qYWgi0JpqbI0aNy1oRE3oCihobjlnSHHnvYPF/uQlFDQ5n+KkViTgdbEjNFxgqGJmqbOJjJWyWLL8fSOYR4EpkmhUWCqSk9U7ibIoaG45f0hB67emGn77UZeNpc5PZyUt1NkzkLeYtHmE4X6R7cF9IXazrc9exOnP21hzCZtULTnwFgOJVjS6KFYJFgakqvmwabKLIkyP033JL49AWr8cmzDvNtCwau53Q4dRiquwlw2nww00OKxIEaWRIbd4/h1cEJDKdyiCrV9ivnFtqp5Ll3U0vBIsHUFFkrkSwRk4gYGo5fWmxJrFnR64tHAMXupjmuKPQmIojoGvpd0WCX0/SRIrF/IoPP/OwFXHTj72d0vnTOserytvBZEr/7P2/C9/76Dd5rrrhuHThwzdQUGZMITqTzRELXML8rhgVdMYymc15qZNws/lMMtgGRbqr3n7ocpx82B7vcKmxOg50+qrvptie2zfh8apFkNNC3q0spxgxr18I0J2xJMDVFFtTJuIFEjUkAwN+ctQqXv2GZ934whgEU2nK8+Yi5+NWVZ+DPVvYDcKyVk5b3ep/BIjF95M/wwERxttl0SPtEwn976YwV/iYOmePv5ss0LywSTE2R7qbgk2JEsSQA4C9PWY7LTy6IRNiTpeFlROlYvaCr6H0Zm6jU3fTU1iEOcgeQ7VLCBjpNh0klPhRsE6+mwwYz2ZjmhUWCqSmnr5qD952yDEct9N/UvcC18nSpth4PEwl5TKkW5VIkKmnyt2XfBC656TH89qW9U+57MCFFolbZTel8ZZbEKhaJloFjEkxN6U1G8MW3F0+aNQOWBBAQiZAOs7LiulSRV0cV7qadw078YmSSGwmryGlxqrtpJhXRqiURFAnVspjrJiEwzQ9bEkxdkBaEWkUtBcPUKVQIjCksiWQZd1Mmb+GtX38Uv3huAACwdywNoJB9wzh47iYlBTadm35KsXps2FRCCXF2U8vAIsHUhYg7tEi9Ocibf5gVARQEpVTlrkyzDbMkfvzkdrywcwS3PrYFADA45jwpz+QG2I7IaXFDMxSJ53cM47Kb/4iRyUJ337Df23FLunGZkrDAND/sbmLqQtj4U2lJBAvvJNLdVMqSiBo6IrqG8UzxTe3bj24GAC82UhCJ2lsS6wdGceTCzpZ8OrZs5+eRswrzPtL56n9GT752AI9v3g/1RxA2pOoXHz+9+kUyDYUtCaYudMVMX+ASAHSNQBSe/goUXFTlxqYmozoefnkQv3tpDwDnhr1tfwp7RhxRkL2J9kqRyNfWkvjdS3tw4Y2/x51P76zpeetF3hWHvF0QhulYEmNu8oBQJtOXsgCZ1oJ/i0xd+OibVuI7f/V63zYiQkTXSvquZe+mUpYEAHTEDLy0axRX3fEc8paNK29/Bv90zzpk3YCs/He23E3PbBsGAGwf8nejfWnXKP79ty/X9LNmg7zrbvJZEtP4GY2GDJEqF5NgWgcWCaYu9HdEceTC4lqHiK6VtCQMvby7CQBiblXvyGQOT20dwu6RNLYq7cOlJTFb7qYDKceX3xdoKfKrF3fj33/7im8iWzOSV8bKStI5G0II2LbATQ+9ivvX75nyPKOTxXEhtiTaA45JMA0lYmglWzSoTQFLMeCmtgLAvS/swmg6j7xd2JaRIuHOr8hM4yn5y/e+hCW9cfzln60oeu+AW1/QEagwl6mlOcsuK3IzZXAsAwGBeZ2xaR1v2cUilslZePs3H8NIKuvN63jyM2cjauj4yPefwt+fdzjmdsSwtC/uxWHCLAkWifaARYJpKBFDK5ndNFXgGih0gF3SG8e9L+4G4B+VmbVsZPIWhlPOTWw6MYn/fsQJgoeKhGtJBJ/IVZEYSeUwns1jcU+86PiZcs3PXkA2b+NWpXleNeStEEsib+G57cO+bbc+tgUTGQuPb96Pr/3mZTz26n589qIj8aEzDgUAjE6yu6ldqZvUE1GUiG4hoq1ENEZEzxDRBRUc9wARCSJiQWtD5nZGsbA7/ClY1klEK5iYdvqqOZ5LSSWbt3zVxOmcjXTOws2PvFqTFh2yxXYucC7p489aNv71Nxvw/u88OePPCmM4lcVwavoFgqXcTZKOqOFMEhzL4rYnnDGnMjj9m3UFN9RoSNU7WxLtQT1/iwaA7QDOBNAN4FoAdxDRilIHENFfgK2dtubWD7wBn7pgdeh7U7XlAIBfXXkGvvfXb8CS3uKn9J6EiWze9olHOmfhkZcH8aV7N+DpbcNFxwRRg7hh8QVZXxB8Ipeikc3b2DWcxi7FLVZLspbwBZ2rJUwo0zkLXW4m2pmHz0UyomNkMud9jizA2zOWxg+f2Ia8ZWMszN3ElkRbUDeREEJMCCGuE0JsEULYQoh7ALwG4KSw/YmoG8DnAVxdrzUy9ac3GZl2nQQArF7QhTcePheLQ0SiPxlB1rKxd9Sptu6Om0jnLK/ga3/InO0ganHYQMiNXrqbii2JQv3ByGQOE1lrVoLYubxd9NnVUMqSSOdsvPPExfjyJcciauq+mMOw29pk6/4UrvnZC/jFcwM+d5NMRGBLoj1o2G+RiOYDOBzAuhK7fAnATQB2121RTFPhteWowN20uCdRtK0/GXUsCVcMlvbFkc7ZBZGoYBqbjGUAwLYD/jTXVDbvuV6ygRu1tCxylu3dYIdnoW9UzrJDb/SVEtanaSKTR9aysaI/ia6YiYiueXUQgP9nAjjupzGl6n1ORxS6Ft5qhWk9GvJbJCITwG0AbhVCbAh5fw2A0wB8vYJzXUFEa4lo7eDgYO0XyzSMnriTVioHGZUjzJLoS0Z87qbFPXGk85bnP69kZKfq7w+KxJ7RgiWSy/tv1FnF3SRFaSRV7JKZKTnLnpGFkrNEUeKAtI48i8DUfO6koEgMpbK+IrqzVs/D+ccsmPaamOai7v5+ItIAfB9AFsDHS7z/TQB/K4TIT9XqQAhxM4CbAWDNmjXTf6Rimo6jFnXhFx8/DccuLp6JHWR+p/P0Kv3lUUNDMmogm7exdyyDvmQEHVETmZztuUbCRCKbt3H72u3oihl42wmLMaTcELcHREIVkLwd7m7KWgWRGKqhSPzsmR3YtHccOUsUfXY1WLaNmKlB9bzJOIt0AwYtiaDVtGPI74a7+IRFOHFZ77TXxDQXdRUJcu74twCYD+BCIUTY/5ouAGsA3O4KhHzM2UFE7xZCzGwIL9NSHLekeB52GIauYUFXDIPjGWTzNjpjBiKGhqzlWBLzOqOImRrSOcsTiTB309cfeAVff2ATIoaGt52w2GstTlRo7SEpd+OU7qaJTN7LFppJFlKQ+9fvwfM7RpDJ27DF1M9GqWw+NPaTt0TR9v0TYZZE6XbsUiScn69dMqWZaU3q7W66CcCRAN4qhCiV7jECYBGAE9yvC93tJwF4YtZXyLQsb149F39+1HwAQGfMRNTQkHHdTXM7o4gaOjL5QowgbGTnq4PjAApT1KRrZWlvoqjbrHrjLOVuUie+Bd00MyGVtZBxg9ZTBa63H0jhuOt+g2e2DXnbNu0dw+fuehFZyy4qZpSWhNwe0bUiEVTZ4bYkkXUgLBLtRT3rJJYD+DCcG/9uIhp3v/6CiJa53y8TDrvlFwAZaNgjhOCJMUxJvvj2Y/HP7sCjjqhrSUiR6FAtCefmHjayU743mXX+HZ7MwdQJ8zqjRXMrVD99qeymfYofp5aB60k3W6oSkdi6P4W8LfDK3nFv24MbBvG9x7cik3fcTSoyJpF0LYyoUbjpq/NAHv/0WVjaF8dO15JY3OskD3ARXXtRzxTYrUIIEkLEhBAdytdtQoht7vfbQo7b4h7H0+6ZKelw8/s7Y4b3BDw4lsHcrihipo68LbDftSBkTGLb/pQ3d0JaGamcBdsWGE7l0B2PoCNmlLQk4qZeFBeQ7qZBRSRqGZOYzKkiUd7dJMXpQImZEUWB66AloaSydrvJBKZOWNgdR1fM9DKbDp2TdLr6Rlkk2gkuVGPaCl0jdEYNz5IQwnH9zElGIeDcTGVswcnKEXjPzY9j10ga716zxNfyOp23MDKZRU/CRDJq+LKbbn1sC14dHAcR0Jswkc2HF9PtG5s9d1M6b3lZRbYtoJUYOSrFSRWJyTIT5OQ65VCnqE8kDOwbz3iNFaVbjgj4xFmr8KYj5nrbmPaARYJpOxb3xrGoJ+57Au6KG16zv7F0HrpGyFkCo+k8do0URpuqRWETGQtDEzn0xE10Rg2MuwIyOJbB53+xDkRAR8QRo6DLJ2vJsaCOIBka1TRwPZm1fGmnOdtGVAt/gh9xP3d/oD2JRHUnqSRMN7vJ93N0BCDmWhldcWefeZ1R9HdE8aYj5lV7KUyTw9UuTNvx/Q+ejL8/7whfEV5nzPSefgFgqVtXoT5dp7J5jKZzmN8V9V4PT+Y8S0LGJKRLSgjnpmnqxSKRD8QklvYlprQk9oymYYcUxqWyeTz6yj7ftslAN9uwRn2S4VRxoF49vlQX3rhXOV14v1uKhBvHkK8XzULzQqY5YJFg2o65nVHP3STpiBqIKgHaFXOSAPw3zuGU059oQZfTcDCVtTCSyjoxiaiBiaxV1FCvM2a4IuHMX9jsZkdJ0ZCFfEv7EhgO6ZQq2Teewclf+h2u//XGovfufHon3nfLE742IpNZv0gERWrj7jGMTOZw5Y+fwZ+2DrnXWli32jI9NsUMcX9MwhWJgLuJRaJ9YZFg2hafSMQMn+99Rb8jEqoLZu+Y43Za0C1FwrEkehOmNy/i5C/9Dv/5wCbvmM6YAdN1N93y6Gs462sP48WdI14wef94FjFTQ1fMKDvxTcZC7n5uoOg9uUYZW7BtUWRJqMHriUwe5/37I3jHN/6Anz874LX9VutC1Jbp6s9Ftb6kEETDRMI9ptMVifnTnGfBND8sEkzbot7cOqN+kTh0rrQkCjdO2WZjYbfzVDw0kUMqa6EnYXpZU5m8jRcHRgvnjZkwNULOsvH8zhEAwCt7x7wn+7wt0BkzvXTcUsj9gxlUQKHJoHRzZULOo1oSG/eMAQA275vw7eMLXGfD3U2Hze/wvtdCxsdKy0G6m2RWV3B+OdM+sEgwbYv6VNwRM3xulTcc0gfA/3S91xWJ+a67aWDEyf/vTkR8k+fU1uMFd5Pt7TOWzvtu2smIjqhRviBN3rSDtRhAQSSktZHKFu+jxiQ27h4L/YxU1vKsGTVwrf5cjpjfWXScFFuighhIwZVrYpFoX1gkmLYlGJOQT8xrlvdi9YIuxE0drylP23s8d5MTuN7ptgbviZtF40klBXeT8GYwOCJRuGknIm7NRt7GVbc/i0/f+XzReeTNO6yja0EknH+DribA3xYkTCSkYEpR9KXA+iyJ0iIRNTRPHOS/p62aAwA4+ZD+ouOY9oDln2lbVJFIRgwcs6gb17/rOLzluIUAnC6xahWynDuxoMtxNw0MO697EmbJKuLOmImInkZOaW8RtCQSEd1zN20aHC9qzQ2E3/gl0s0kn9qDQWvA32Bww+7RovcXdMew7UAKB8azTjdcX+C6cG2HuAF9lahRyHKSbib58zj3qPlY94/nIVlCRJnWh3+zTNuiupukf/3SNUu9bf0dEWzaU3jqlkV2MnC9y7MkIqE3dsCxJAzNcTfJDrTOFDdFJKKFZoOZXLjLKV1iO1CYHz2WzmE8k8dLIZaC2jvq5T3jRe/L2RxDbmZWukQxXVg6rBTbiGpJqALMAtHW8G+XaVvKTbQDHEvieeWpfI9rSczrjIKoMImu3DyLzpjpuZtkQHlwLAPVa5SM6IjoOixbIJXLI6yzt3rTDnZs9QLXk3n810Ov4j8f3FR0fM49qW2L0BboMgtJWiWqKMUjhZ+TqRGuOvdwX6sO1d0krYpStRVM+8EiwbQtlYiEyp7RDEydkIjoiJs6BkYK7qZgf6RlfQmctXoe3nzEXDyzbQg5y/ZqD3aP+hscx113E+BUcYuQ1t6qu2nfWBa7RkbwhkP6QEQ+S2IixNUEOGNM1fN0xQyMpvNY2hfHRccuwluOW4i3fP3RQgPDEu4mXSN88uzDfOeO+GISfncT0/5w4JppW6aasdzvisScjoJYdMVMEJH3JK9rhI6o4RWWSXoSJq67+Ggs6U3AdN1N0pLY7YqLJBkxvO6pY+kcJjLFN3rVknhw41685+Y/4onXDiBn2Z4wjKXzvpnbKlLEJtzMJ9mRdU5HFP9wwWov5bdgSRQ+Ty0yNEJGjvpjEsXuJqa94d8007ZE9PJPuzKT6KJjF3rulV5XOKQo9MQd0Qj2N1KznUyDfO6mfYEW5Imo7glWzhLIhowcVZ/sZZ3D/vGsr5fUaDmRcN1NKVeA5GyH3oRzPXFTh6E5VoltC1+thaFpMNyYixESewmLSUTZkjhoYJFg2pap3E1yLOplJy/zfOx97k1VWhLdSjzinScuxjtetxiAvy7AHwGi/AAAEmFJREFU1DXk8jYy+XBXUDJiFK0lWOuQVtxIMhYynsn5RGEsnfOJhop0N0lLYonbm6onLru0ErriJkbTuaJiPEMnL7Bt6MUiEQ1xN/FgoYMHjkkwbctUIvGO1y3G2UfOR3fc9G56Mkgtn6jlkzgA3HDpCXh62xB+9sxOdEQL4hHRNeTsYutAklBiEpKJrIWeROF1WjlWFuuNZyyMummvhkYYTedLioS0ilKu2HgikVBdaQZGJ/NF7UEMTYOpaUjDhqEV/8y8mISpe/ELjkkcPLAlwbQtU4kEERV1NZXBbJkh9P5TV/iOkU/mqiVh6H53UxCnmM5/Uw1WVk9mLc+F5YmE4l5a1BPHWDpXJiZh+85bcDcVxKzbtSSCNRmqJRGW6qtaEot64njLcQtx8qF9oetg2g+2JJi2RdZJrOhPTLFn4clYPnl/7dLjcWAiiwuPXejbT74fdDdZtvCe4oMko8WWRLBHUzpnoTNmIJu3varoiWze6zi7uCeOF3aOlM5usvyWxPL+JN52wiK88fC53j5dcROjk7kQS4K8gLUZ4m5SYxIRQ8N/Xn5i6BqY9oRFgmlbIoaGb1x+Itas6J1yX1nX0Jd0nrxPOTS8zUR33MSa5b04YWmPt83UZXprHnM6or651oDjvw+KxDu/+Rg+fOah2HFgEm87YREmcxbipo5kVPe6vY6l83jytQOImzqOWtSFxzfvL7n+oCXRGTPwH+99nW+frpiJgeHJIktC1wimVs6SKO4Gyxw8sEgwbc1Fxy2ceicUUkJVH34Yukb4yUdP9W2TFst4Jo+5ncUikYwa0Kj45vu7l/Zi095x/PKFXTjnyHmImToSEcMTidF0Dk9sPoA3r56Lhd2FVtxvPX4RDpmTxE0PbfIsCDnkSFoSYVXQXXGndiJY3W3qmmJJhKTAmtLdxHGIgxF+NGAYFESibwqRCEP688czeczrjBa9Hxa4BoBNbt+ovmQE6ZyNmKn5UmsffWUf9o1ncMExC7FybqGF99tPWISrzj3c13YkG6iTSIRURHfFHHdTJsSSKBeTkJ/DlsTBSd1+60QUJaJbiGgrEY0R0TNEdEGJfd9PRE8R0SgR7SCi64mIrR5m1pAumN5k6RYcpZBP3+PpPPo7ikUmETHK3mAX9cQcd1NE9xXtySD165b1+OY8yGC7KjyeJZGxoGsU+nldcROZvF0U/DY0gulmNZkh2U1e4NpkkTgYqedv3QCwHcCZALoBXAvgDiJaEbJvAsCVAOYAOBnA2QD+vi6rZA5KZGfV3mlYEvJJO2+L0Cf4REQPdeNI+pNRTGZlTKL4WWhuZ9TLVgIKIqG6f7yYRDaPREQHhbi3ZCvzvWN+d5ihawVLIiRwTUS45MQlOG3lnJLXwLQvdXs6F0JMALhO2XQPEb0G4CQAWwL73qS83ElEtwF482yvkTl4kemr0xEJ0yjcWMP89smoETonQpKzbKTzFqKmXlSn0Jswi87ZpVgScVPHZM4qZDdlLCQj4f+t5XG7R/1tQ9TsprCKa8DJ9mIOThpmPxLRfACHA1hXwe5vrHA/hpkWqxc4w3bkjbQa1Bt7mJunVExCWiCTOQvpEpaEnJKnorqbIoYGUye/JRENDzDLGpA/bNrnS3U1lOymUiLBHLw0RCSIyARwG4BbhRAbptj3AwDWAPhqifevIKK1RLR2cHCw9otlDgp+8KGTcfsVp5ScG1EO1ZUUZklEDc0XZJZcff4R6IwZmMxamMxZiJlaUSPBuUog/LYPnYxL1ywp9E8yNCczyW0wCDjZTaWm6B23pAcaAc/vGMHRi7q9azU0rWzgmjm4qXswmIg0AN8HkAXw8Sn2fTuArwA4RwixL2wfIcTNAG4GgDVr1pS26RmmDHM6opjTUZyZVAkR1d2kBHdvuPR4bNmfAhGFWhIfOuNQvLhzBM9sH0Y6ZztN+AJiorYzP23VHG9cqPO5GiI6IetWfANOnURYXARwLJBjF3fjuR0jOHFZL9YPjMKCgK6TKzYUGstgDm7qakmQ8xd4C4D5AC4RQoT3GHD2PR/AtwC8VQjxQp2WyDBV47ckCt+ffGg/rjr38KLtFx23ENdcuBqAM2tiImMViukCN/hyU98iugbTtSZUS6JUTAIATlnpFAmetLwX87ocUdTIcTOxFcGEUW9L4iYAR8KxDCZL7UREZ8FxR71DCPFkvRbHMNNBDXbLOdDpnO3z+6vups+95Sgv1hA3Da9pX9TUi6yAoGioRFyBMHUNeaVOIhEtnlMtufj4RXhs036curIfP77iFDzy8j4kIgYM9zwME6SedRLLAXwYwAkAdhPRuPv1F0S0zP1+mbv7tXDSZO9V9ruvXmtlmGpY2lfoDRU1NK/NOKEgEppGXlBYtSriEWf2NeCkqAYth0QZq6A/GUFvwnQbDBbqJMoJy9GLunH3J05HbzKCJb0JXH6y81/O1NmSYMKpZwrsVgDl/go7lH053ZVpGbqVjKioqeH6S47Dl+59qWg2dsTQkM9avuC2KgLdiQh0NyZwyqF9+OPmAyV7SAHA5956NHKWjctu/iNytkA6Z2HfeGZasRVD00Kb+zEMVzEzTA0gAoRw3ErnHDUf5xw1v2ifiKEhlbV8QWx1LkNvwoQlp+UdtwjfuPxE9Je54cugtqETcnkbG3aPIW8LHLO4q+r1G2xJMCVgkWCYGrCgK4ZdI+myYz0jbgaRejNWYxA98Yg33S4Z0csKhIqhaXjklUEc73amPcaduFcNbzthsVcrwjAqLBIMUwPmuSJhlnkaN3WtqNhOHQPakzBBZEIjZ8hQpfQlI1i/axT/8qsN6IgavhYelXLm4XNxpjJ7gmEknM7AMDVgvlv0dsAdEhRG1NCKLI14xC8SS3oTeOKac8rGIoLceNnr8KV3HAvAcWlxrQNTS9iSYJgacPX5qzEwMokzVpV+Go8YpS0JXSOvUnpuSLvxcvQlI7j85GWYzFk4Yj67jJjawiLBMDVg1bwO3POJM8ruI3stqciYRE/cnLEF8MHTD5nR8QwTBrubGKZOREJiEoXZ2tU3FmSYesAiwTB1wnE3+WMSniUxjRblDFMP2N3EMHXiwmMXesONJHHF3cQwzQiLBMPUifedsrxoW8J0/guyJcE0KyRE+3TX7uzsFCeddFKjl8EwFSNIx9aTr0LXrrXo2/pgo5fDHKQ8/PDDTwkh1oS911YiQURjADbO8DTdAEZmuF/Ye1NtC74vX6vb5wAInatRBfW6vnKvS31fr+ur9trCtjfi+mbrdxe2vdrra6W/zbBt7Xx9ldxblgshwvO3hRBt8wVgbQ3OcfNM9wt7b6ptwffl68A+LXN95V6X+b4u11fttTXL9c3W764W19dKf5sH2/VVcm8p98XZTcXcXYP9wt6balvw/btLbJ8p9bq+cq/LXfdMqeR81V5b2PZGXN9s/e7CtrfT9VX799pu1zeje0u7uZvWihJ+tXaAr6+1aefra+drA9r/+srRbpbEzY1ewCzD19fatPP1tfO1Ae1/fSVpK0uCYRiGqS3tZkkwDMMwNYRFgmEYhinJQScSRLSCiAaJ6CH3q+0mrRDRZUQ02Oh11Boimk9EjxHRw0T0ABEtbPSaagkR/RkRPe5e34+IqK16dRBRNxE9SUTjRHRMo9dTC4jon4no90T0EyJKNHo9s8FBJxIuDwsh3uR+tdXNlIg0AO8CsL3Ra5kF9gE4XQhxJoDvAfhgg9dTa7YCOMu9vs0A3tbg9dSaFICLAPyk0QupBa7QrRRCnAHgtwD+usFLmhUOVpE4zVX/L1H7jfG6HM5/QrvRC6k1QghLCCGvqxPAukaup9YIIQaEEJPuyzza7HcohMi12UPZGQDuc7+/D8DpDVzLrNHUIkFEHyeitUSUIaLvBt7rI6KfEdEEEW0lossrPO0uAKsAvBHAPADvrO2qK2M2ro2IdACXArh9FpZcFbP0uwMRnUBETwD4OICna7zsipmt63OPPwTABQDuqeGSq2I2r6/ZmMG19qLQ1mIEQF+dllxXmr0L7ACALwI4D0Bwuvs3AGQBzAdwAoBfEtFzQoh1RLQA4Sbtu4QQuwFkAICI7gRwCoCfztL6y1Hza3PPdYcQwm4CA2lWfndCiGcBnExElwL4NICPzNoVlGdWro+IugDcCuAvhRClB2bPPrP1f68Zmda1AhiC0/8I7r8H6rPcOjPTfiT1+ILzC/yu8joJ5xd3uLLt+wC+UsG5upTvvwzgf7XRtf0LgN8A+BWcJ5sb2+x3F1W+Pw/ADW12fQaAX8KJSzT0umbj+pT9vwvgmEZf20yvFcCxAH7ofn8FgE80+hpm46up3U1lOByAJYR4Wdn2HICjKzj2TCJ6ioh+D2AxgB/OxgJnwLSvTQjxKSHEnwshzgfwihDik7O1yBkwk9/diUT0CBE9COBKAP86GwucITO5vssAnAzgc27m3XtmY4EzZCbXByK6F8CfA/gWEf1V7ZdXU8peqxDiBQBb3XvJeQC+U/8lzj7N7m4qRQeKW+OOwAlmlkUIcTdq31Sulkz72lRE8/aZmcnv7nE4saRmZibX9304T6rNzIz+PoUQF9Z8RbPHlNcqhPh0XVfUAFrVkhgH0BXY1gVgrAFrqTXtfG0AX1+r0+7Xp3IwXWtJWlUkXgZgENFhyrbj0R4pke18bQBfX6vT7tencjBda0maWiSIyCCiGAAdgE5EMSIyhBATAO4E8E9ElCSi0+AUHjW7qe7RztcG8PWBr69lOJiudVo0OnI+RbbBdQBE4Os6970+AD8HMAFgG4DLG71evja+Pr6+1vs6mK51Ol/cKpxhGIYpSVO7mxiGYZjGwiLBMAzDlIRFgmEYhikJiwTDMAxTEhYJhmEYpiQsEgzDMExJWCQYhmGYkrBIMEwNISJBRO9q9DoYplawSDAtBRF9l4gaNrGtAhaiibsME9F1RPRio9fBtA4sEgwzBUQUqXRf4UzPy8zmesKoZo0MUw0sEkxbQUTdRHQzEe0lojEiepiI1ijv9xPRj4hoBxFNEtE6IvpA4BwPEdFNRPRVIhoE8Ad3uyCiK4jof9yZx5uJ6H2BYz13ExGtcF9fQkT3E1GKiNYT0bmBYy4ioo1ElHaHKr3XPW5Fmevc4loF3yGiYQC3udu/4p5r0t3nerd5HdwhP58HcLR7fiEH/0z1c2MOXlgkmLaBiAjO+M/FAN4C4HUAHgHwABEtdHeLAXjaff9oAP8B4L+J6OzA6d4HgACcAeB/Kds/B+AuOC2jbwfwHSJaPsXS/hnAje4xfwLwYyLqcNe8DE6n0V+6798I4PoKL/kqABsArAFwjbttAsBfAzgSwMcAvBfAZ9z3bgfwNQAb4bjFFgK4vcKfG3Ow0ugOg/zFX9V8wZmPfE+J986CMygmHtj+LICry5zzxwC+rbx+CMDzIfsJAF9WXhsAUgDeF9jnXe73K9zXH1beX+xuO919/WUALwFOs0132zXuPivKrHkLgLsr+Hl9BMAm5fV1AF6sxc+Nvw6Or1YdX8owYZwEIAFg0Hk49ogBWAkARKQD+AcA74Fzw44CiMARBpWnSnzG8/IbIUTedUfNm2JdzyvfD7j/ymNWA/iTEEJtx/zEFOeT/P/27p41iigK4/j/0cJOEAULC40JxlJSSMRYBQstUmgV/AAiKWLQ1mKj2JjCwkKwEewCFiLERv0AIkpIIaQQBS1CGhECIYHcFOdujBOvu0kzSeb5wbDsvNydvcWcuefcYT5WV+RU122gj3j95sG8/E/HfrPmcpCw/eQAsECkiKp+58+7wB1gHJgj7qAfsvVCv1T4jdXK90TntO3GMSmllC/E7WOU29iJv85R0iAxKmoBE8AvYASY6tBON/1mDeUgYfvJJ+A4sJZS+lrYZ4hI07yAjTrGGeKCWocvxNvONju/w7YuAj9TSvfbK/5RL1lh68iim36zhnLh2vaiw5LOVZZTwFtiJtIrSVck9Ui6IKklqX2XPA8MSxqSdBZ4AvTU8i/CU6A3z6Tql3QNuJm3bXeEMQ+ckHRD0mlJt4DRyj7fgJOSBiQdk3SI7vrNGspBwvaiS8DnyjKV8/pXgffAM2IWzzTQz59awAPgA/CGmMGzRJ4+WoeU0nfgOpEWmiXSRK28eXmbbb0GHgGPiTrIZWI21mYvgRngHbAIjHbZb9ZQfn2p2S4jaRyYBI6klNbqPh9rNtckzGomaYx4fmIRGATuAc8dIGw3cJAwq18f8WzEUeAHUaeYrPWMzDKnm8zMrMiFazMzK3KQMDOzIgcJMzMrcpAwM7MiBwkzMytykDAzs6J10RJvgwnjbZ0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1, batch_size=batch_size)\n",
    "plot_lr_vs_loss(rates, losses)\n",
    "plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 1.4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100,\n",
    "                                 kernel_initializer=\"lecun_normal\",\n",
    "                                 activation=\"selu\"))\n",
    "\n",
    "model.add(keras.layers.AlphaDropout(rate=0.1))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))\n",
    "\n",
    "optimizer = keras.optimizers.SGD(lr=1e-2)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 45000 samples, validate on 5000 samples\n",
      "Epoch 1/15\n",
      "45000/45000 [==============================] - 3s 69us/sample - loss: 2.0504 - accuracy: 0.2823 - val_loss: 1.7711 - val_accuracy: 0.3706\n",
      "Epoch 2/15\n",
      "45000/45000 [==============================] - 3s 57us/sample - loss: 1.7626 - accuracy: 0.3766 - val_loss: 1.7751 - val_accuracy: 0.3844\n",
      "Epoch 3/15\n",
      "45000/45000 [==============================] - 3s 59us/sample - loss: 1.6264 - accuracy: 0.4272 - val_loss: 1.6774 - val_accuracy: 0.4216\n",
      "Epoch 4/15\n",
      "45000/45000 [==============================] - 3s 57us/sample - loss: 1.5527 - accuracy: 0.4474 - val_loss: 1.6633 - val_accuracy: 0.4316\n",
      "Epoch 5/15\n",
      "45000/45000 [==============================] - 3s 59us/sample - loss: 1.4997 - accuracy: 0.4701 - val_loss: 1.5909 - val_accuracy: 0.4540\n",
      "Epoch 6/15\n",
      "45000/45000 [==============================] - 3s 60us/sample - loss: 1.4564 - accuracy: 0.4841 - val_loss: 1.5982 - val_accuracy: 0.4624\n",
      "Epoch 7/15\n",
      "45000/45000 [==============================] - 3s 56us/sample - loss: 1.4232 - accuracy: 0.4958 - val_loss: 1.6417 - val_accuracy: 0.4382\n",
      "Epoch 8/15\n",
      "45000/45000 [==============================] - 3s 58us/sample - loss: 1.3530 - accuracy: 0.5199 - val_loss: 1.5050 - val_accuracy: 0.4778\n",
      "Epoch 9/15\n",
      "45000/45000 [==============================] - 3s 57us/sample - loss: 1.2771 - accuracy: 0.5480 - val_loss: 1.5254 - val_accuracy: 0.4928\n",
      "Epoch 10/15\n",
      "45000/45000 [==============================] - 3s 56us/sample - loss: 1.2073 - accuracy: 0.5726 - val_loss: 1.5013 - val_accuracy: 0.5052\n",
      "Epoch 11/15\n",
      "45000/45000 [==============================] - 3s 57us/sample - loss: 1.1380 - accuracy: 0.5948 - val_loss: 1.4941 - val_accuracy: 0.5170\n",
      "Epoch 12/15\n",
      "45000/45000 [==============================] - 3s 56us/sample - loss: 1.0672 - accuracy: 0.6204 - val_loss: 1.5091 - val_accuracy: 0.5106\n",
      "Epoch 13/15\n",
      "45000/45000 [==============================] - 3s 56us/sample - loss: 0.9967 - accuracy: 0.6466 - val_loss: 1.5261 - val_accuracy: 0.5212\n",
      "Epoch 14/15\n",
      "45000/45000 [==============================] - 3s 58us/sample - loss: 0.9301 - accuracy: 0.6712 - val_loss: 1.5437 - val_accuracy: 0.5264\n",
      "Epoch 15/15\n",
      "45000/45000 [==============================] - 3s 59us/sample - loss: 0.8893 - accuracy: 0.6866 - val_loss: 1.5650 - val_accuracy: 0.5276\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 15\n",
    "onecycle = OneCycleScheduler(len(X_train_scaled) // batch_size * n_epochs, max_rate=0.05)\n",
    "history = model.fit(X_train_scaled, y_train, epochs=n_epochs, batch_size=batch_size,\n",
    "                    validation_data=(X_valid_scaled, y_valid),\n",
    "                    callbacks=[onecycle])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One cycle allowed us to train the model in just 15 epochs, each taking only 3 seconds (thanks to the larger batch size). This is over 3 times faster than the fastest model we trained so far. Moreover, we improved the model's performance (from 50.8% to 52.8%). The batch normalized model reaches a slightly better performance, but it's much slower to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "nav_menu": {
   "height": "360px",
   "width": "416px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}