t81_558_class_10_2_lstm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tdSnFaggnD2W"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_10_2_lstm.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gc4uQ-bYnD2Y"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 10: Time Series in PyTorch**  \n",
    "\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QFj4U9c8nD2Y"
   },
   "source": [
    "# Module 10 Material\n",
    "\n",
    "* Part 10.1: Time Series Data Encoding for Deep Learning, PyTorch** [[Video]](https://www.youtube.com/watch?v=CZi5Avp6p1s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_1_timeseries.ipynb)\n",
    "* **Part 10.2: LSTM-Based Time Series with PyTorch** [[Video]](https://www.youtube.com/watch?v=hIQLy5zCgH4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_2_lstm.ipynb)\n",
    "* Part 10.3: Transformer-Based Time Series with PyTorch [[Video]](https://www.youtube.com/watch?v=NGzQpphf_Vc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_3_transformer_timeseries.ipynb)\n",
    "* Part 10.4: Seasonality and Trend [[Video]](https://www.youtube.com/watch?v=HOkxoLaUF9s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_4_seasonal.ipynb)\n",
    "* Part 10.5: Predicting with Meta Prophet [[Video]](https://www.youtube.com/watch?v=MzjMVsz0GyA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_10_5_prophet.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5MtBKuaZnD2Z"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code checks that Google CoLab is and sets up the correct hardware settings for PyTorch.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nT6Wli6knD2Z",
    "outputId": "9819d7d3-66b8-4aba-cdf2-9af14f25ff0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n",
      "Using device: cuda\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_RH3-887nD2a"
   },
   "source": [
    "# Part 10.2: LSTM-Based Time Series with PyTorch\n",
    "\n",
    "So far, the neural networks that we’ve examined have always had forward connections. Neural networks of this type always begin with an input layer connected to the first hidden layer. Each hidden layer always connects to the next hidden layer. The final hidden layer always connects to the output layer. This manner of connection is why these networks are called “feedforward.”  Recurrent neural networks are not as rigid, as backward linkages are also allowed. A recurrent connection links a neuron in a layer to either a previous layer or the neuron itself. Most recurrent neural network architectures maintain the state in the recurrent connections. Feedforward neural networks don’t keep any state.\n",
    "\n",
    "## Understanding LSTM\n",
    "\n",
    "Long Short Term Memory (LSTM) layers are a type of recurrent unit that you often use with deep neural networks.[[Cite:hochreiter1997long]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.676.4320) For TensorFlow, you can think of LSTM as a layer type that you can combine with other layer types, such as dense. LSTM makes use of two transfer function types internally.  \n",
    "\n",
    "The first type of transfer function is the sigmoid.  This transfer function type is used form gates inside of the unit.  The sigmoid transfer function is given by the following equation:\n",
    "\n",
    "$ \\mbox{S}(t) = \\frac{1}{1 + e^{-t}} $\n",
    "\n",
    "The second type of transfer function is the hyperbolic tangent (tanh) function, which allows you to scale the output of the LSTM. This functionality is similar to how we have used other transfer functions in this course.  \n",
    "\n",
    "We provide the graphs for these functions here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 879
    },
    "id": "aK4KCNPEnD2a",
    "outputId": "bd0b7388-b2b3-490e-a2a7-8ecc4c1be184"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sigmoid\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9EUlEQVR4nO3deXxU9b3/8fdMkpkkZCMJSUgI+y6bgsTgikYpWlqrtRS94qVqq6VWjbc/xSpUvRWrVulVblHrdq+1or11qVB4AIKIIMimguz7loQQksk+ycz390fIQCSBTEhyZiav5+Mxj2TO+Z6Zz+E4k7fn+z3fYzPGGAEAAFjEbnUBAACgYyOMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsFW51Ac3h9Xp1+PBhxcbGymazWV0OAABoBmOMSktLlZ6eLru96fMfQRFGDh8+rMzMTKvLAAAALXDgwAF169atyfVBEUZiY2Ml1e1MXFycxdUAAIDmcLlcyszM9P0db0pQhJH6rpm4uDjCCAAAQeZsQywYwAoAACxFGAEAAJYijAAAAEsRRgAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALOV3GFm+fLkmTJig9PR02Ww2ffDBB2fdZtmyZbrgggvkdDrVt29fvfHGGy0oFQAAhCK/w0h5ebmGDx+u2bNnN6v9nj17dN1112ns2LHauHGj7rvvPt1xxx1auHCh38UCAIDQ4/e9acaPH6/x48c3u/2cOXPUq1cv/fGPf5QkDRo0SCtWrNDzzz+vcePG+fv2AAAgxLT5jfJWrVqlnJycBsvGjRun++67r8ltqqurVV1d7XvucrnaqjwAQIgwxqjWa1Tj8aqm1qjG6/X9Xuv1+tZ5vHXtPF6jWs+Jn16vvMbI45U8J9Z5jTmxzMhrJK8xMubk715T957eU9ZL9e1Ut151v5sTy8wp66W65zJGpu7HiWUN15+6XA2Wm0b+DU553e8s8z0/Ze2p626/pJcyE6P9+0dvJW0eRvLy8pSamtpgWWpqqlwulyorKxUVFXXaNjNnztRjjz3W1qUBANqZMUaVNR6VVNaotKpWrvqfVTUqr/aowl2rsupaVbjrfq9we1RV41Gl26PKGo+qaryqqvGoutar6hM/3bVeVXvqgkcjf5/RTD8YkR66YaQlpk2bptzcXN9zl8ulzMxMCysCADTF6zUqqnArr6RKeSVVKiit1rGyahWWVauwzK3CsmoVV9ToeIVbxZU1ctd62602R5hd4WE2hdttCg+z1/088XuY3aawE8/tNpvCw+p+htltCjvx026X7DbbiYcUZrfJduJ3u80mm+3keptNsql++cnndT/rXktquPzEklOW1S2s3+bk7/L9rlPbnbqzp7zeqW1Pa/eddfVS4yL9/edtNW0eRtLS0pSfn99gWX5+vuLi4ho9KyJJTqdTTqezrUsDADRTaVWNdh0t175j5TpQVKH9RRU6UFSpg8UVyi+pltvjX8AIt9sUGxmu2MgIxUWFK8YZrhhnhDo5w9TJGa5OjjBFO8IV7QhTlCNMkRFhioqo+xkZYZcz/ORPR7hdjnC7IsJscobVPa8PILbG/uoi4LR5GMnOztb8+fMbLFu0aJGys7Pb+q0BAH6qrvVoe16ZNh0u0dYjLu08WqadBWXKd1WfcTubTUqOcSotLlIpsU4lxziVHOtQcoxTSTFOJUY7lBAdoYToCHWOdijaEUZQgI/fYaSsrEw7d+70Pd+zZ482btyoxMREde/eXdOmTdOhQ4f0P//zP5Kku+66Sy+++KL+3//7f/rZz36mTz75RO+++67mzZvXensBAPCbMUb7jlVozd4irdt7XJsOl2h7fqlqPI0PvOgS61SvpE7KTIxW98RodU+KUrfO0eoaH6mU2Eg5wplHEy3jdxhZu3atxo4d63teP7bjtttu0xtvvKEjR45o//79vvW9evXSvHnzdP/99+tPf/qTunXrpr/85S9c1gsAFjhQVKFl2wq0avcxfbn3uI6Wnn7GIyE6QkPS4zU4PU59U2LUNyVGfbrEKD4qwoKK0RHYTGPXBgUYl8ul+Ph4lZSUKC4uzupyACBo1Hi8WrOnSEu3FmjZ9qPaWVDWYL0jzK5h3eI1qmeiRmQmaEhGnDISouhCQato7t/vgLyaBgDQcl6v0dp9x/XRV4c0/5s8FZW7fevC7DaN7N5Zl/VP1uheSRrWLV6REWEWVgsQRgAgZBwoqtBfV+/XRxsP6XBJlW95UieHrhyYorEDU3Rx32S6WxBwCCMAEMS8XqPPdxXqzZX7tGRrvm/Sr1hnuMYNSdMPhqdrTJ8khYcxuBSBizACAEGoxuPVP9Yf1EvLd2v30XLf8kv6JuvmrO66cmAK3S8IGoQRAAgitR6vPth4WP+1ZIf2F1VIkmKc4frxyG76t4t6qG9KjMUVAv4jjABAEDDG6KOvDmvW4h3aU1h3JiQ5xqG7Lu+jn47urhgnX+cIXvzXCwABbmueS49+sElf7j0uSUrs5NAvLuutW7N7KNrB1ziCH/8VA0CAKquu1axF2/X6yr3yeI2iIsI0dWwfTbm4lzpxJgQhhP+aASAALf42X498sEl5rrpLdL93XpqmTxis9ITGbzAKBDPCCAAEkKoaj2bO36I3V+2TJHVPjNZjPzhPYwemWFwZ0HYIIwAQIHYWlOmev23QliMuSdLtl/TSb8YN4BJdhDzCCAAEgHfXHtCMDzerssajpE4OPXvTcM6GoMMgjACAhTxeo/+c961e/3yvJOnivkl6/icjlBIXaW1hQDsijACARSrdHt03d4MWbs6XJD1wdX9NHdtXdjt3zEXHQhgBAAsUllXrjjfXauOBYjnC7PrjT4ZrwvB0q8sCLEEYAYB2tvtomf799S+1v6hC8VERemXyKI3ulWh1WYBlCCMA0I52Hy3TT176QoVl1cpMjNIbU0arTxfuJ4OOjTACAO1k/7EK3fzKahWWVWtQ1zj97+2jlRzjtLoswHKEEQBoB4eKKzXplS+U56pSv5QYvXX7aCURRABJkt3qAgAg1OW7qnTLK1/oUHGleiV30l/vyCKIAKcgjABAGyoqd+vmV77Q3mMV6tY5Sn+9I4s5RIDvIIwAQBup8Xh191vrtOtoubrGR+pvd17Eje6ARhBGAKCNPPbPzVq9p0gxznD9z89GKzMx2uqSgIBEGAGANvDWF/v01hf7ZbNJf/rpCPVLjbW6JCBgEUYAoJV9sfuYfvfRZknSf1wzQFcNSrW4IiCwEUYAoBUdKKrQL/+6XrVeownD0/XLK/pYXRIQ8AgjANBKqms9uuutdSoqd2tIRpyevnGYbDZuegecDWEEAFrJc4u2a/NhlxI7OfTyraMU5QizuiQgKBBGAKAVrN59TC8v3y1JmnnDUC7hBfxAGAGAc1RaVaPcd7+SMdJPRnXTuPPSrC4JCCqEEQA4R7/76FsdKq5UZmKUpk84z+pygKBDGAGAc/Cvb47o/9YflN0mPf+TEYpxcv9RwF+EEQBooQJXlR5+/xtJ0l2X99GonokWVwQEJ8IIALTQYx9/q+MVNTovPU735fS3uhwgaBFGAKAFVuwo1Lyvj8huk57+8TA5wvk6BVqKTw8A+Mld69WMjzZJkm69qIfOS4+3uCIguBFGAMBPr3++R7uOliupk0O51wywuhwg6BFGAMAPeSVV+tOSHZKkh8YPVHxUhMUVAcGPMAIAfvj9/C2qcHt0QfcE3XhBN6vLAUICYQQAmmnlrkL986vDstukx384RHY7N8EDWgNhBACaodbj1YwPN0uS/u2iHhqSwaBVoLUQRgCgGf6x4ZB2FJSpc3SEHriaQatAayKMAMBZVNd69KfFdYNW776ij+KjGbQKtCbCCACcxdwvD+hQcaVSYp2anN3T6nKAkEMYAYAzqHR79MInOyVJ91zZV5ERYRZXBIQewggAnMH/frFXR0urlZEQpYkXdre6HCAkEUYAoAmlVTX687JdkqR7c/px/xmgjfDJAoAmvLZir45X1Kh3l0664fwMq8sBQhZhBAAaUVzh1l8+2y1Juj+nv8LD+LoE2gqfLgBoxCuf7VZpda0GpsXquqFdrS4HCGmEEQD4jrLqWv3vqn2SpPty+jPtO9DGCCMA8B3vfnlArqpa9UrupGsGp1pdDhDyCCMAcIpaj1evrtgjSbrj0l6cFQHaAWEEAE7xr015OlRcqaRODt14QTerywE6BMIIAJxgjNHLy+uuoJmc3ZPZVoF2QhgBgBO+2F2kbw6VyBlu163ZPawuB+gwWhRGZs+erZ49eyoyMlJZWVlas2bNGdvPmjVLAwYMUFRUlDIzM3X//ferqqqqRQUDQFt55cS8IjeN6qbETg6LqwE6Dr/DyNy5c5Wbm6sZM2Zo/fr1Gj58uMaNG6eCgoJG27/99tt66KGHNGPGDG3ZskWvvvqq5s6dq4cffviciweA1rIjv1SfbC2QzSbdcUlvq8sBOhS/w8hzzz2nO++8U1OmTNHgwYM1Z84cRUdH67XXXmu0/cqVK3XxxRfr5ptvVs+ePXXNNddo0qRJZz2bAgDtqf6syLjBaeqZ3MniaoCOxa8w4na7tW7dOuXk5Jx8AbtdOTk5WrVqVaPbjBkzRuvWrfOFj927d2v+/Pm69tprz6FsAGg9R0ur9cGGw5KkOy/jrAjQ3sL9aVxYWCiPx6PU1IaTAKWmpmrr1q2NbnPzzTersLBQl1xyiYwxqq2t1V133XXGbprq6mpVV1f7nrtcLn/KBAC/vLv2gNwer0ZkJmhkj85WlwN0OG1+Nc2yZcv05JNP6r//+7+1fv16/eMf/9C8efP0xBNPNLnNzJkzFR8f73tkZma2dZkAOiiv1+idL/dLkm7J6m5xNUDH5NeZkeTkZIWFhSk/P7/B8vz8fKWlpTW6zaOPPqpbb71Vd9xxhyRp6NChKi8v189//nP99re/ld1+eh6aNm2acnNzfc9dLheBBECb+GxnoQ4UVSo2MlzfH5ZudTlAh+TXmRGHw6GRI0dqyZIlvmVer1dLlixRdnZ2o9tUVFScFjjCwuomEjLGNLqN0+lUXFxcgwcAtIW3V9fdEO/GC7opysEkZ4AV/DozIkm5ubm67bbbNGrUKI0ePVqzZs1SeXm5pkyZIkmaPHmyMjIyNHPmTEnShAkT9Nxzz+n8889XVlaWdu7cqUcffVQTJkzwhRIAsEK+q0qLt9RNS3AzXTSAZfwOIxMnTtTRo0c1ffp05eXlacSIEVqwYIFvUOv+/fsbnAl55JFHZLPZ9Mgjj+jQoUPq0qWLJkyYoN///vettxcA0ALvfnlAHq/RhT07q39qrNXlAB2WzTTVVxJAXC6X4uPjVVJSQpcNgFbh8Rpd9vRSHSqu1PMTh+tH53NTPKC1NffvN/emAdAhLd9+VIeKK5UQHaHxQ7paXQ7QoRFGAHRIf11ddznvjRd04+68gMUIIwA6nMPFlfpka90UBZNGM3AVsBphBECHM/fLA/IaKatXovqmxFhdDtDhEUYAdCher9H/rT8oict5gUBBGAHQoazdd1wHj1cqxhmucec1PnM0gPZFGAHQoby/4ZAkafyQNAauAgGCMAKgw6iq8Wje14clST+6IMPiagDUI4wA6DCWbi2Qq6pWXeMjdVGvJKvLAXACYQRAh1HfRfPDERmy220WVwOgHmEEQIdwvNytpdvqbor3o/PpogECCWEEQIcw75sjqvEYDe4apwFp3BQPCCSEEQAdQn0XzQ0MXAUCDmEEQMjbd6xc6/Ydl90m/WB4utXlAPgOwgiAkPfBhrrLeS/um6yUuEiLqwHwXYQRACHNGKP3N9RN/87AVSAwEUYAhLSvDpZo77EKRUWEMf07EKAIIwBCWv2MqzmDU9XJGW5xNQAaQxgBELKMMZr/TZ4k6bqhnBUBAhVhBEDI+vpgiQ4VVyraEaYrBqRYXQ6AJhBGAISs+ZuOSJLGDkzhDr1AACOMAAhJdV00dWHkuqFdLa4GwJkQRgCEpM2HXTpQVKnICLuuGNDF6nIAnAFhBEBIqj8rcuXAFEU7uIoGCGSEEQAh59QumvFD6KIBAh1hBEDI2XKkVHuPVcgZbteVA7mKBgh0hBEAIaf+rMgVA7ow0RkQBAgjAELKqV0013IVDRAUCCMAQsq2/FLtLiyXI9yuqwalWl0OgGYgjAAIKfXTv1/ev4ti6KIBggJhBEBIWbCp/ioa7kUDBAvCCICQse9YubbnlyncbtNVA+miAYIFYQRAyFj0bb4kaXSvRMVHR1hcDYDmIowACBmLt9SFkasHc1YECCaEEQAhobjCrS/3Hpck5XAVDRBUCCMAQsLSbQXyeI0GpsUqMzHa6nIA+IEwAiAkLP62QBJdNEAwIowACHrVtR4t21YXRuiiAYIPYQRA0Ptid5HK3R6lxDo1NCPe6nIA+IkwAiDoLT5xSW/O4FTZ7TaLqwHgL8IIgKBmjDl5SS9dNEBQIowACGqbD7t0pKRK0Y4wZfdJsrocAC1AGAEQ1OpnXb2sXxdFRoRZXA2AliCMAAhqi04ZLwIgOBFGAAStQ8WV+vaIS3abdOXAFKvLAdBChBEAQeuTEwNXR/borMRODourAdBShBEAQWvptqOSpCsH0kUDBDPCCICgVFXj0cpdhZKksQO7WFwNgHNBGAEQlL7YfUxVNV51jY/UgNRYq8sBcA4IIwCC0rITXTRXDEiRzcasq0AwI4wACEpLT9wYb+wAumiAYEcYARB09hSWa9+xCkWE2TSmb7LV5QA4R4QRAEFn6da6syKjeyUqxhlucTUAzhVhBEDQOdlFw0RnQCggjAAIKhXuWq3eXSSpbvAqgOBHGAEQVFbuPCa3x6vMxCj16dLJ6nIAtALCCICgcmoXDZf0AqGhRWFk9uzZ6tmzpyIjI5WVlaU1a9acsX1xcbGmTp2qrl27yul0qn///po/f36LCgbQcRljfPOLMF4ECB1+D0OfO3eucnNzNWfOHGVlZWnWrFkaN26ctm3bppSU078c3G63rr76aqWkpOjvf/+7MjIytG/fPiUkJLRG/QA6kB0FZTpUXClnuF0X9U6yuhwArcTvMPLcc8/pzjvv1JQpUyRJc+bM0bx58/Taa6/poYceOq39a6+9pqKiIq1cuVIRERGSpJ49e55b1QA6pGUnumgu6p2kKEeYxdUAaC1+ddO43W6tW7dOOTk5J1/AbldOTo5WrVrV6DYfffSRsrOzNXXqVKWmpmrIkCF68skn5fF4mnyf6upquVyuBg8AWLq1vouGWVeBUOJXGCksLJTH41FqasPbdaempiovL6/RbXbv3q2///3v8ng8mj9/vh599FH98Y9/1H/+5382+T4zZ85UfHy875GZmelPmQBCUFl1rb7cyyW9QChq86tpvF6vUlJS9PLLL2vkyJGaOHGifvvb32rOnDlNbjNt2jSVlJT4HgcOHGjrMgEEuFW7jqnWa9QjKVo9k7mkFwglfo0ZSU5OVlhYmPLz8xssz8/PV1paWqPbdO3aVREREQoLO9m/O2jQIOXl5cntdsvhcJy2jdPplNPp9Kc0ACFu+fa6LprL+tFFA4Qav86MOBwOjRw5UkuWLPEt83q9WrJkibKzsxvd5uKLL9bOnTvl9Xp9y7Zv366uXbs2GkQAoDHLd5wII/0JI0Co8bubJjc3V6+88orefPNNbdmyRXfffbfKy8t9V9dMnjxZ06ZN87W/++67VVRUpHvvvVfbt2/XvHnz9OSTT2rq1KmttxcAQtq+Y3V36Q2325Tdh0t6gVDj96W9EydO1NGjRzV9+nTl5eVpxIgRWrBggW9Q6/79+2W3n8w4mZmZWrhwoe6//34NGzZMGRkZuvfee/Xggw+23l4ACGn1XTQje3TmLr1ACLIZY4zVRZyNy+VSfHy8SkpKFBcXZ3U5ANrZHW+u1eIt+frNuAGaOrav1eUAaKbm/v3m3jQAApq71qtVuwolSZczXgQISYQRAAFt/f7jKnd7lNTJocFdOTMKhCLCCICAVj9e5NJ+ybLbuUsvEIoIIwACGpf0AqGPMAIgYBWWVWvTobp7U13KZGdAyCKMAAhYK3bUDVwd3DVOXWKZlRkIVYQRAAHLNwU8XTRASCOMAAhIXq/R8hNnRi7rn2xxNQDaEmEEQEDakudSYVm1oh1hGtUj0epyALQhwgiAgLR8e91ZkezeSXKE81UFhDI+4QAC0mc7Ts4vAiC0EUYABJxKt0dr9x6XJF3K4FUg5BFGAASc1XuOye3xKiMhSr2TO1ldDoA2RhgBEHDq5xe5tF+ybDamgAdCHWEEQMD57EQYuYTxIkCHQBgBEFDyXVXall8qm026uA9hBOgICCMAAkp9F83QjHh17uSwuBoA7YEwAiCgcEkv0PEQRgAEDK/XaMXOY5K4Sy/QkRBGAASMrXmlvingL+je2epyALQTwgiAgFHfRXMRU8ADHQqfdgABY8XOk/OLAOg4CCMAAkJVjUer9xRJIowAHQ1hBEBAWLOnSO5ar7rGR6pPlxirywHQjggjAALCqV00TAEPdCyEEQABYfn2usGrl3BJL9DhEEYAWK6gtEpb8+qngE+yuhwA7YwwAsByn5/oohmSHq+kGKfF1QBob4QRAJb7bDt36QU6MsIIAEsZY/QZ84sAHRphBICltuWX6mhptaIiwjSyB1PAAx0RYQSApeq7aLJ6J8oZHmZxNQCsQBgBYKmTXTRc0gt0VIQRAJapqvFo9e5jkhgvAnRkhBEAllm797iqa71KjXOqXwpTwAMdFWEEgGU+21k36+ql/bowBTzQgRFGAFimfvAqXTRAx0YYAWCJwrJqfXvEJUm6uC9hBOjICCMALFE/BfzgrnFKZgp4oEMjjACwxPL6Lpr+nBUBOjrCCIB2Z4zRihODVy9jfhGgwyOMAGh3OwrKlO+qljPczhTwAAgjANrf8u11Z0WyeicpMoIp4IGOjjACoN19tuPEeBGuogEgwgiAdlZV49HqPXVTwF/Wn/EiAAgjANrZl3uLVFXjVVpcpPqnMgU8AMIIgHb26bb6KeCTmQIegCTCCIB2tnzHiUt66aIBcAJhBEC7OVJSqe35ZbLZpEsYvArgBMIIgHZTf2O8Yd0S1LmTw+JqAAQKwgiAdvPpiS6ay7lLL4BTEEYAtAuP12jFiflFGC8C4FSEEQDt4uuDxSqprFFsZLhGZCZYXQ6AAEIYAdAu6u/Se3GfZIWH8dUD4CS+EQC0Cy7pBdAUwgiANldSWaONB4olSZf1Z/AqgIZaFEZmz56tnj17KjIyUllZWVqzZk2ztnvnnXdks9l0/fXXt+RtAQSplTsL5fEa9e7SSd06R1tdDoAA43cYmTt3rnJzczVjxgytX79ew4cP17hx41RQUHDG7fbu3av/+I//0KWXXtriYgEEJ18XTT+6aACczu8w8txzz+nOO+/UlClTNHjwYM2ZM0fR0dF67bXXmtzG4/Holltu0WOPPabevXufU8EAgosxxjd49XLGiwBohF9hxO12a926dcrJyTn5Ana7cnJytGrVqia3e/zxx5WSkqLbb7+9We9TXV0tl8vV4AEgOO06WqZDxZVyhNmV1TvR6nIABCC/wkhhYaE8Ho9SU1MbLE9NTVVeXl6j26xYsUKvvvqqXnnllWa/z8yZMxUfH+97ZGZm+lMmgACy7MRderN6JyraEW5xNQACUZteTVNaWqpbb71Vr7zyipKTmz+Cftq0aSopKfE9Dhw40IZVAmhLS7fVjSe7YkCKxZUACFR+/W9KcnKywsLClJ+f32B5fn6+0tLSTmu/a9cu7d27VxMmTPAt83q9dW8cHq5t27apT58+p23ndDrldDr9KQ1AACqrrtWaPUWSpLEDGC8CoHF+nRlxOBwaOXKklixZ4lvm9Xq1ZMkSZWdnn9Z+4MCB+uabb7Rx40bf4wc/+IHGjh2rjRs30v0ChLjPdxaqxmPUIylavZI7WV0OgADldwdubm6ubrvtNo0aNUqjR4/WrFmzVF5erilTpkiSJk+erIyMDM2cOVORkZEaMmRIg+0TEhIk6bTlAELPshNdNGMHpMhms1lcDYBA5XcYmThxoo4eParp06crLy9PI0aM0IIFC3yDWvfv3y+7nYldgY7OGOMbvHoFXTQAzsBmjDFWF3E2LpdL8fHxKikpUVxcnNXlAGiGrXkufW/WZ3KG2/XVjGsUGRFmdUkA2llz/35zCgNAm1i6te6syJg+SQQRAGdEGAHQJuov6R07kEt6AZwZYQRAqyuprNG6fcclSVf0J4wAODPCCIBWt2JH3V16+3TppO5J3KUXwJkRRgC0uqWnXNILAGdDGAHQqrzek5f0Ml4EQHMQRgC0qs2HXSosq1YnR5hG9exsdTkAggBhBECrqp91dUzfZDnDuaQXwNkRRgC0qiVbGS8CwD+EEQCtpqC0ShsPFEuSrhpEGAHQPIQRAK3mky11Z0WGd4tXalykxdUACBaEEQCtZtG3+ZKknEGpFlcCIJgQRgC0igp3rVbsLJQkXX0eYQRA8xFGALSKFTsKVV3rVbfOURqQGmt1OQCCCGEEQKtYvOVkF43NZrO4GgDBhDAC4Jx5vEZLTgxevWYwXTQA/EMYAXDONh44rmPlbsVGhuvCXolWlwMgyBBGAJyzRd+enOgsIoyvFQD+4VsDwDmrHy9yNV00AFqAMALgnOwpLNfOgjKF2226fEAXq8sBEIQIIwDOyeITE51d1DtJcZERFlcDIBgRRgCck0V00QA4R4QRAC12vNyttXuLJHFjPAAtRxgB0GKLtuTLa6RBXePUrXO01eUACFKEEQAt9q9vjkiSrh2SZnElAIIZYQRAi5RU1vhujDd+aFeLqwEQzAgjAFpk8bf5qvEYDUiNVd+UGKvLARDECCMAWmT+iS6a8UPpogFwbggjAPzmqqrRZzvqumiuo4sGwDkijADw25It+XJ7vOqbEqN+qbFWlwMgyBFGAPht/jd5kriKBkDrIIwA8EtpVY0+3X5UknTtMLpoAJw7wggAv3yytUDuWq96J3fSALpoALQCwggAv9RfRXPt0K6y2WwWVwMgFBBGADRbeXWtlm2r66Lhkl4ArYUwAqDZPtlaoOpar3omRWtw1zirywEQIggjAJrt5ERndNEAaD2EEQDN4qqq0ZKtBZKY6AxA6yKMAGiWBd/kyV1bN9HZeel00QBoPYQRAM3yjw0HJUk/Oj+DLhoArYowAuCsDhVX6ovdRZKk68/PsLgaAKGGMALgrD7ceEiSlNUrURkJURZXAyDUEEYAnJExRu+vrwsjN1zAWREArY8wAuCMNh92aUdBmRzhdo3nKhoAbYAwAuCM3t9Qd1bk6kGpiouMsLgaAKGIMAKgSbUerz7ceFhS3VU0ANAWCCMAmvT5rmMqLKtW5+gIXda/i9XlAAhRhBEATXp/fd3cIhOGp8sRztcFgLbBtwuARpVX12rh5nxJdNEAaFuEEQCN+temPFXWeNQruZNGZCZYXQ6AEEYYAdCot1fvkyT9eGQ3pn8H0KYIIwBOszXPpfX7ixVut+mmUd2sLgdAiCOMADjN26v3S5KuHpyqlNhIi6sBEOoIIwAaqHDX+qZ/vzmru8XVAOgICCMAGvj4qyMqra5Vj6RoXdwn2epyAHQAhBEADfx1TV0XzaTR3WW3M3AVQNtrURiZPXu2evbsqcjISGVlZWnNmjVNtn3llVd06aWXqnPnzurcubNycnLO2B6AdTYfLtFXB4oVEWbTj0cycBVA+/A7jMydO1e5ubmaMWOG1q9fr+HDh2vcuHEqKChotP2yZcs0adIkLV26VKtWrVJmZqauueYaHTp06JyLB9C66geujjsvTckxTourAdBR2Iwxxp8NsrKydOGFF+rFF1+UJHm9XmVmZuqee+7RQw89dNbtPR6POnfurBdffFGTJ09u1nu6XC7Fx8erpKREcXFx/pQLoJnKqmuV9fvFKnd79PadWRrDeBEA56i5f7/9OjPidru1bt065eTknHwBu105OTlatWpVs16joqJCNTU1SkxMbLJNdXW1XC5XgweAtvXRxsMqd3vUO7mTsnsnWV0OgA7ErzBSWFgoj8ej1NTUBstTU1OVl5fXrNd48MEHlZ6e3iDQfNfMmTMVHx/ve2RmZvpTJgA/GWP09pq6GVcnje7OjKsA2lW7Xk3z1FNP6Z133tH777+vyMimJ1KaNm2aSkpKfI8DBw60Y5VAx7N6T5E2HXLJGW7XjQxcBdDOwv1pnJycrLCwMOXn5zdYnp+fr7S0tDNu++yzz+qpp57S4sWLNWzYsDO2dTqdcjoZPAe0l5eX75Yk3TSqmxI7OSyuBkBH49eZEYfDoZEjR2rJkiW+ZV6vV0uWLFF2dnaT2z399NN64okntGDBAo0aNarl1QJodTvyS/XJ1gLZbNLtl/S2uhwAHZBfZ0YkKTc3V7fddptGjRql0aNHa9asWSovL9eUKVMkSZMnT1ZGRoZmzpwpSfrDH/6g6dOn6+2331bPnj19Y0tiYmIUExPTirsCoCX+8tkeSdI1g1PVK7mTxdUA6Ij8DiMTJ07U0aNHNX36dOXl5WnEiBFasGCBb1Dr/v37ZbefPOHy5z//WW63Wz/+8Y8bvM6MGTP0u9/97tyqB3BOClxVen9D3Zw/P7+sj8XVAOio/J5nxArMMwK0jWcWbtXspbs0skdn/d/dY6wuB0CIaZN5RgCEjvLqWr31Rd2Mq3deylgRANYhjAAd1LtrD6ikska9kjvp6sGpZ98AANoIYQTogGo9Xr26om7g6u2X9FIYd+cFYCHCCNABzd+Up4PHK5XYyaEbL2CSMwDWIowAHYzHa/SnxdslSbdl91SUI8ziigB0dIQRoIP5YMMh7TparoToCP3skp5WlwMAhBGgI3HXejVrSd1Zkbsu76PYyAiLKwIAwgjQoby79oAOFFUqOcapydk9rC4HACQRRoAOo6rGoxc+2SFJ+tXYPop2+D0BMwC0CcII0EG89cU+5buqlZEQpUlZ3a0uBwB8CCNAB1BeXas/L9slSfr1VX3lDOcKGgCBgzACdACvf75Hx8rd6pkUrRuYVwRAgCGMACHuWFm1Xlq+W5J0/9X9FRHGxx5AYOFbCQhxf1iwVaVVtRrcNU4ThqVbXQ4AnIYwAoSw9fuP6921ByVJT1x/nuzcgwZAACKMACHK4zWa/uEmSdKPR3bTyB6JFlcEAI0jjAAh6m9r9mvTIZdiI8P10PiBVpcDAE0ijAAhqKjcrWcWbpMkPXB1fyXHOC2uCACaRhgBQtDTC7aqpLJGA9Ni9W8XMe07gMBGGAFCzMYDxZq79oAk6YnrhyicS3kBBDi+pYAQUlXj0W/e+0rGSDdckKELezJoFUDgI4wAIeTpBdu0o6BMyTFO/fbaQVaXAwDNQhgBQsSKHYV67fM9kqRnfjxMSQxaBRAkCCNACCipqNF/vPeVJOnfLuqusQNTLK4IAJqPMAKEgEc+3KQ8V5V6JXfSw3TPAAgyhBEgyH248ZD++dVhhdlten7iCEU7wq0uCQD8QhgBgti+Y+V65IO6Kd/vubKvRmQmWFsQALQAYQQIUqVVNbrjzbUqrarV+d0TNHVsX6tLAoAWIYwAQcjrNbp/7kbtKChTSqxTc/5tpCKY3AxAkOLbCwhCzy3arsVbCuQIt+vlyaOUGhdpdUkA0GKEESDIfPz1Yb24dKck6akbhjJOBEDQI4wAQWTToRLffCJ3XtpLN1zQzeKKAODcEUaAILGzoEz//voaVdV4dXn/LnpoPPOJAAgNhBEgCOw7Vq5b/vKFCsvcGtw1Tv816XyF2W1WlwUArYIwAgS4g8crdPMrq5Xvqlb/1Bi9dUeW4qMirC4LAFoNYQQIYHklVbrlL6t1qLhSvZM76a07spTYyWF1WQDQqggjQIA6UlKpm//yhfYdq1BmYpT+emeWUmK5hBdA6OEmFkAA2ny4RD9740vlu6qVHh+pt++4SF3jo6wuCwDaBGEECDCfbj+qX761TuVuj/qlxOj1KReqW+doq8sCgDZDGAECyNwv9+vh9zfJ4zXK7p2kObeOZLAqgJBHGAECQI3Hq2cXbtNLy3dLkm44P0NP3ThMjnCGdQEIfYQRwGIHiir063c2aMP+YknSr6/sq/uv7i+bjXlEAHQMhBHAQh9/fVjT/u8blVbXKi4yXE/dOEzXDu1qdVkA0K4II4AFyqpr9Z8ff6t3vjwgSRrZo7P+9NMRDFQF0CERRoB2ZIzRvG+O6ImPv1W+q1o2m/SrsX1171X9FB7G+BAAHRNhBGgnu4+WacZHm/XZjkJJUs+kaD15w1CN6ZNscWUAYC3CCNDGiivcemn5br362R65PV45wu2aekVf/eLy3oqMCLO6PACwHGEEaCMlFTV6dcVuvfb5XpVV10qSLu/fRY//8Dz1SOpkcXUAEDgII0ArO17u1pur9urVFXtUWlUXQgZ1jVPu1f2VMyiFS3YB4DsII0Ar+fpgsf5n1T599NVhuWu9kqQBqbG6/+p+umZwmux2QggANIYwApyD0qoaLdiUp7+u3q+NB4p9y4dkxOkXl/XRdUO7EkIA4CwII4Cfqmo8+mRrgT7aeFifbCvwnQWJCLPp+8PSdWt2D52fmUB3DAA0E2EEaIZ8V5U+3XZUy7YXaPn2Qt+AVEnqmxKjH52foZ+MylSXWKeFVQJAcCKMAI0oqazR+n3HtXpPkZZvP6pvj7garM9IiNKE4en6wfB0Deoay1kQADgHhBF0eLUer3YdLdemQyX66mCx1uwp0rb8Uhlzso3NJg3rlqAr+nfR2IEpGpYRz1gQAGglhBF0GMYYHSmp0s6CMu0sKNOOgjJ9e8SlrUdcqj4x7uNUvZI7aVSPzsruk6TL+3dRUgxdMADQFloURmbPnq1nnnlGeXl5Gj58uF544QWNHj26yfbvvfeeHn30Ue3du1f9+vXTH/7wB1177bUtLhpojDFGrqpa5ZVU6XBJpQ4er9SBogrtP1ah/UUV2nesXOVuT6PbxjjDNTg9TkPS43Vhz84a2bOzUmIj23kPAKBj8juMzJ07V7m5uZozZ46ysrI0a9YsjRs3Ttu2bVNKSspp7VeuXKlJkyZp5syZ+v73v6+3335b119/vdavX68hQ4a0yk4gNBljVF3rVUlljY5XuFVcUaPiCreOV9SosLRax8rdOlpWrcLSah0trVaeq0oVTYSNeuF2m3okRatvSoz6psRoYFqchmTEq0diNN0uAGARmzGn9oyfXVZWli688EK9+OKLkiSv16vMzEzdc889euihh05rP3HiRJWXl+vjjz/2Lbvooos0YsQIzZkzp1nv6XK5FB8fr5KSEsXFxflTLtqAMUY1HqMaj/fEw8jt8cpde8rD41F1jVfVtV5V13pUVeNVVY1HlfUPd92j3O1RhbtW5dW1Kq/2qNxdK1dljUqralVaVSu35/Tuk7NJiI5QWlykunWOVmZilLonRqt7YrR6JEWre2InOcK5Oy4AtIfm/v3268yI2+3WunXrNG3aNN8yu92unJwcrVq1qtFtVq1apdzc3AbLxo0bpw8++KDJ96murlZ1dbXvucvlarLtuXh1xR4dKKo4a7v6vGZ8zxtpc2LtqetOb3+yTf0yI1P3vH656p6c3Nacsk7ymvr1ddt5Tf3PulfzNlh28qfXGHm9ksf3e11bj9fUPU4s8xjjW1brrVtW4/HK4zWq8Z5c157sNikh2qGEqAglREeoc7RDSTEOJcc4lRzjVFKMQ11inUqPj1JqXKSiHNx8DgCCiV9hpLCwUB6PR6mpqQ2Wp6amauvWrY1uk5eX12j7vLy8Jt9n5syZeuyxx/wprUXmfX1Y6/cXt/n7dASOMLsc4SceJ353htvljLArMjxMkRFhcobbFeUIU1REmO9ntCNcnZxh6uQMV7QjTDHOcMVGRig2MlxxUXU/YxzhdKEAQAgLyKtppk2b1uBsisvlUmZmZqu/z40ju2lMn+TTljc2ZcRpi05pZPvOItsprU8u+85zW8M29dvU/X5y2ant7b51tlPa2Xxt7TbJXv/cZlOYvf75yXX2E7+H2W2+tuF2u+z2up9hdinMbleYzaYwu03hYXXtIsJsCg+zK8JetzziROgIP/GceTYAAC3lVxhJTk5WWFiY8vPzGyzPz89XWlpao9ukpaX51V6SnE6nnM62v4zylqwebf4eAADgzPwayedwODRy5EgtWbLEt8zr9WrJkiXKzs5udJvs7OwG7SVp0aJFTbYHAAAdi9/dNLm5ubrttts0atQojR49WrNmzVJ5ebmmTJkiSZo8ebIyMjI0c+ZMSdK9996ryy+/XH/84x913XXX6Z133tHatWv18ssvt+6eAACAoOR3GJk4caKOHj2q6dOnKy8vTyNGjNCCBQt8g1T3798vu/3kCZcxY8bo7bff1iOPPKKHH35Y/fr10wcffMAcIwAAQFIL5hmxAvOMAAAQfJr795vZnwAAgKUIIwAAwFKEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApfyeDt4K9ZPEulwuiysBAADNVf93+2yTvQdFGCktLZUkZWZmWlwJAADwV2lpqeLj45tcHxT3pvF6vTp8+LBiY2Nls9la7XVdLpcyMzN14MCBkL3nDfsYGtjH0MA+hgb2sfmMMSotLVV6enqDm+h+V1CcGbHb7erWrVubvX5cXFzI/gdVj30MDexjaGAfQwP72DxnOiNSjwGsAADAUoQRAABgqQ4dRpxOp2bMmCGn02l1KW2GfQwN7GNoYB9DA/vY+oJiACsAAAhdHfrMCAAAsB5hBAAAWIowAgAALEUYAQAAlgr5MPL73/9eY8aMUXR0tBISEhpts3//fl133XWKjo5WSkqKfvOb36i2tvaMr1tUVKRbbrlFcXFxSkhI0O23366ysrI22AP/LFu2TDabrdHHl19+2eR2V1xxxWnt77rrrnas3D89e/Y8rd6nnnrqjNtUVVVp6tSpSkpKUkxMjG688Ubl5+e3U8X+2bt3r26//Xb16tVLUVFR6tOnj2bMmCG3233G7QL9OM6ePVs9e/ZUZGSksrKytGbNmjO2f++99zRw4EBFRkZq6NChmj9/fjtV6r+ZM2fqwgsvVGxsrFJSUnT99ddr27ZtZ9zmjTfeOO14RUZGtlPF/vvd7353Wr0DBw484zbBdAylxr9bbDabpk6d2mj7YDiGy5cv14QJE5Seni6bzaYPPvigwXpjjKZPn66uXbsqKipKOTk52rFjx1lf19/P85mEfBhxu9266aabdPfddze63uPx6LrrrpPb7dbKlSv15ptv6o033tD06dPP+Lq33HKLNm/erEWLFunjjz/W8uXL9fOf/7wtdsEvY8aM0ZEjRxo87rjjDvXq1UujRo0647Z33nlng+2efvrpdqq6ZR5//PEG9d5zzz1nbH///ffrn//8p9577z19+umnOnz4sG644YZ2qtY/W7duldfr1UsvvaTNmzfr+eef15w5c/Twww+fddtAPY5z585Vbm6uZsyYofXr12v48OEaN26cCgoKGm2/cuVKTZo0Sbfffrs2bNig66+/Xtdff702bdrUzpU3z6effqqpU6fqiy++0KJFi1RTU6NrrrlG5eXlZ9wuLi6uwfHat29fO1XcMuedd16DelesWNFk22A7hpL05ZdfNti/RYsWSZJuuummJrcJ9GNYXl6u4cOHa/bs2Y2uf/rpp/Vf//VfmjNnjlavXq1OnTpp3LhxqqqqavI1/f08n5XpIF5//XUTHx9/2vL58+cbu91u8vLyfMv+/Oc/m7i4OFNdXd3oa3377bdGkvnyyy99y/71r38Zm81mDh061Oq1nwu32226dOliHn/88TO2u/zyy829997bPkW1gh49epjnn3++2e2Li4tNRESEee+993zLtmzZYiSZVatWtUGFre/pp582vXr1OmObQD6Oo0ePNlOnTvU993g8Jj093cycObPR9j/5yU/Mdddd12BZVlaW+cUvftGmdbaWgoICI8l8+umnTbZp6nspUM2YMcMMHz682e2D/RgaY8y9995r+vTpY7xeb6Prg+0YSjLvv/++77nX6zVpaWnmmWee8S0rLi42TqfT/O1vf2vydfz9PJ9NyJ8ZOZtVq1Zp6NChSk1N9S0bN26cXC6XNm/e3OQ2CQkJDc405OTkyG63a/Xq1W1esz8++ugjHTt2TFOmTDlr27/+9a9KTk7WkCFDNG3aNFVUVLRDhS331FNPKSkpSeeff76eeeaZM3atrVu3TjU1NcrJyfEtGzhwoLp3765Vq1a1R7nnrKSkRImJiWdtF4jH0e12a926dQ3+/e12u3Jycpr891+1alWD9lLdZzOYjpeksx6zsrIy9ejRQ5mZmfrhD3/Y5PdOoNixY4fS09PVu3dv3XLLLdq/f3+TbYP9GLrdbr311lv62c9+dsabtAbbMTzVnj17lJeX1+A4xcfHKysrq8nj1JLP89kExY3y2lJeXl6DICLJ9zwvL6/JbVJSUhosCw8PV2JiYpPbWOXVV1/VuHHjznqjwZtvvlk9evRQenq6vv76az344IPatm2b/vGPf7RTpf759a9/rQsuuECJiYlauXKlpk2bpiNHjui5555rtH1eXp4cDsdp44ZSU1MD7pg1ZufOnXrhhRf07LPPnrFdoB7HwsJCeTyeRj9rW7dubXSbpj6bwXC8vF6v7rvvPl188cUaMmRIk+0GDBig1157TcOGDVNJSYmeffZZjRkzRps3b27Tm4O2VFZWlt544w0NGDBAR44c0WOPPaZLL71UmzZtUmxs7Gntg/kYStIHH3yg4uJi/fu//3uTbYLtGH5X/bHw5zi15PN8NkEZRh566CH94Q9/OGObLVu2nHVgVTBpyT4fPHhQCxcu1LvvvnvW1z91vMvQoUPVtWtXXXXVVdq1a5f69OnT8sL94M8+5ubm+pYNGzZMDodDv/jFLzRz5syAnqK5Jcfx0KFD+t73vqebbrpJd9555xm3DYTjCGnq1KnatGnTGcdTSFJ2drays7N9z8eMGaNBgwbppZde0hNPPNHWZfpt/Pjxvt+HDRumrKws9ejRQ++++65uv/12CytrG6+++qrGjx+v9PT0JtsE2zEMVEEZRh544IEzJlVJ6t27d7NeKy0t7bQRwPVXWKSlpTW5zXcH6dTW1qqoqKjJbc5VS/b59ddfV1JSkn7wgx/4/X5ZWVmS6v6PvL3+iJ3Lcc3KylJtba327t2rAQMGnLY+LS1NbrdbxcXFDc6O5Ofnt9kxa4y/+3j48GGNHTtWY8aM0csvv+z3+1lxHBuTnJyssLCw065eOtO/f1paml/tA8WvfvUr36B2f//POCIiQueff7527tzZRtW1roSEBPXv37/JeoP1GErSvn37tHjxYr/PKgbbMaw/Fvn5+eratatveX5+vkaMGNHoNi35PJ9Vi0aaBKGzDWDNz8/3LXvppZdMXFycqaqqavS16gewrl271rds4cKFATWA1ev1ml69epkHHnigRduvWLHCSDJfffVVK1fWNt566y1jt9tNUVFRo+vrB7D+/e9/9y3bunVrQA9gPXjwoOnXr5/56U9/ampra1v0GoF0HEePHm1+9atf+Z57PB6TkZFxxgGs3//+9xssy87ODtjBj16v10ydOtWkp6eb7du3t+g1amtrzYABA8z999/fytW1jdLSUtO5c2fzpz/9qdH1wXYMTzVjxgyTlpZmampq/Nou0I+hmhjA+uyzz/qWlZSUNGsAqz+f57PW1aKtgsi+ffvMhg0bzGOPPWZiYmLMhg0bzIYNG0xpaakxpu4/nCFDhphrrrnGbNy40SxYsMB06dLFTJs2zfcaq1evNgMGDDAHDx70Lfve975nzj//fLN69WqzYsUK069fPzNp0qR237+mLF682EgyW7ZsOW3dwYMHzYABA8zq1auNMcbs3LnTPP7442bt2rVmz5495sMPPzS9e/c2l112WXuX3SwrV640zz//vNm4caPZtWuXeeutt0yXLl3M5MmTfW2+u4/GGHPXXXeZ7t27m08++cSsXbvWZGdnm+zsbCt24awOHjxo+vbta6666ipz8OBBc+TIEd/j1DbBdBzfeecd43Q6zRtvvGG+/fZb8/Of/9wkJCT4rmS79dZbzUMPPeRr//nnn5vw8HDz7LPPmi1btpgZM2aYiIgI880331i1C2d09913m/j4eLNs2bIGx6uiosLX5rv7+Nhjj5mFCxeaXbt2mXXr1pmf/vSnJjIy0mzevNmKXTirBx54wCxbtszs2bPHfP755yYnJ8ckJyebgoICY0zwH8N6Ho/HdO/e3Tz44IOnrQvGY1haWur72yfJPPfcc2bDhg1m3759xhhjnnrqKZOQkGA+/PBD8/XXX5sf/vCHplevXqaystL3GldeeaV54YUXfM/P9nn2V8iHkdtuu81IOu2xdOlSX5u9e/ea8ePHm6ioKJOcnGweeOCBBml46dKlRpLZs2ePb9mxY8fMpEmTTExMjImLizNTpkzxBZxAMGnSJDNmzJhG1+3Zs6fBv8H+/fvNZZddZhITE43T6TR9+/Y1v/nNb0xJSUk7Vtx869atM1lZWSY+Pt5ERkaaQYMGmSeffLLBmazv7qMxxlRWVppf/vKXpnPnziY6Otr86Ec/avDHPZC8/vrrjf53e+rJzGA8ji+88ILp3r27cTgcZvTo0eaLL77wrbv88svNbbfd1qD9u+++a/r3728cDoc577zzzLx589q54uZr6ni9/vrrvjbf3cf77rvP9++Rmppqrr32WrN+/fr2L76ZJk6caLp27WocDofJyMgwEydONDt37vStD/ZjWG/hwoVGktm2bdtp64LxGNb/Dfvuo34/vF6vefTRR01qaqpxOp3mqquuOm3fe/ToYWbMmNFg2Zk+z/6yGWNMyzp4AAAAzl2Hn2cEAABYizACAAAsRRgBAACWIowAAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEv9fy1jpUlY2oqfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hyperbolic Tangent(tanh)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCyklEQVR4nO3deXxU5d3///dMlkkCJCFkh7AjiLIVSgy1aiU3QWkrvb0tWFqUG+FWQYvBhfRRQaQVF2qtlm/pwqI/tW51qdUbRRC9lQgYQAUBgQbCNgkQk8kCWWau3x8hIyNJSCCT2V7Px+M8yJxznTOfw2Emb865znUsxhgjAACAIGL1dQEAAADtjYADAACCDgEHAAAEHQIOAAAIOgQcAAAQdAg4AAAg6BBwAABA0CHgAACAoBPu6wJ8weVy6ciRI+rSpYssFouvywEAAK1gjFFFRYXS09NltbZ8jiYkA86RI0eUkZHh6zIAAMB5OHjwoHr06NFim5AMOF26dJHU8BcUGxvr42oAAEBrOBwOZWRkuH+PtyQkA07jZanY2FgCDgAAAaY13UvoZAwAAIIOAQcAAAQdAg4AAAg6BBwAABB0CDgAACDoEHAAAEDQIeAAAICgQ8ABAABBh4ADAACCjlcDzocffqgf/ehHSk9Pl8Vi0euvv37OddavX6/vfOc7stls6t+/v1atWnVWm6VLl6p3796KiopSZmamNm3a1P7FAwCAgOXVgFNVVaVhw4Zp6dKlrWpfWFioCRMm6Ac/+IG2bdumOXPm6JZbbtE777zjbvPiiy8qNzdXCxYs0JYtWzRs2DDl5OSopKTEW7sBAAACjMUYYzrkjSwWvfbaa5o4cWKzbe677z699dZb2r59u3ve5MmTVVZWptWrV0uSMjMz9d3vfld//OMfJUkul0sZGRm64447NG/evFbV4nA4FBcXp/Lycp5FBQBAgGjL72+/ethmfn6+srOzPebl5ORozpw5kqTa2loVFBQoLy/PvdxqtSo7O1v5+fnNbrempkY1NTXu1w6Ho30LB4AAVed06WSdUydrG6bqWqdq6p2qrXepzmlU63Sqtt6o3uVSvdOo3mVU73Sp3mXkMkZOV8PkMkYuI7mMkTGSy3XGa0k6vdyoYblpmCWj0y/UOO/s/3ObM5af+bolRh3yf3e0YGSvrvrh0HSfvb9fBRy73a6UlBSPeSkpKXI4HDp58qS+/vprOZ3OJtvs2rWr2e0uXrxYCxcu9ErNAOBPnC6jo+UnVVRaraNlp3S8skbHK2t0orJWx6tqVX6yThUn6+Q4Va+KU3WqqXf5umQEqZp6FwHH2/Ly8pSbm+t+7XA4lJGR4cOKAODCGGN0sPSkvjhcri8Ol2vnUYcOnKjS4bKTqnO2/eyF1SLFRIYrKsIqW3iYbOFWRZ6eIsKsCrdaGv4MsyjcalHY6clqOf2zxSKr1SKrRbJaLLJYLLJYGrZrUcN8i8UiSbKcnnf6pSyn5zUss8jSWJTlm/osZ744o73HvDbvdfPbwoUb1iPep+/vVwEnNTVVxcXFHvOKi4sVGxur6OhohYWFKSwsrMk2qampzW7XZrPJZrN5pWYA6CgHS6u1bleJ3t9doi0HvpbjVH2T7SLCLMroGqP0+GgldbGpW6dIJZ7+Mz4mUrFR4eoSFaHY6HB1sUUoOjJMEWEWdwABgoFfBZysrCy9/fbbHvPWrFmjrKwsSVJkZKRGjhyptWvXujsru1wurV27VrNnz+7ocgHA674qrtA/Cg5p7a4S7S2p9FgWGWbVoLQuurR7nC5Jj1W/pM7qmRCjlNgohVkJKwhtXg04lZWV2rt3r/t1YWGhtm3bpoSEBPXs2VN5eXk6fPiwnnnmGUnSrbfeqj/+8Y+699579d///d9at26dXnrpJb311lvubeTm5uqmm27SqFGjNHr0aD3xxBOqqqrStGnTvLkrANBhXC6jdbtKtGrDfn2097h7fpjVopG9uurqQcm6vH+iLkrposhwxmsFmuLVgPPpp5/qBz/4gft1Yz+Ym266SatWrdLRo0dVVFTkXt6nTx+99dZbuuuuu/SHP/xBPXr00N/+9jfl5OS420yaNEnHjh3T/PnzZbfbNXz4cK1evfqsjscAEGjqnS79fVOR/vZRoQ6cqJbU0IflPwan6IdD03XFgCTFxUT4uEogMHTYODj+hHFwAPib7YfLNe/Vz7X9cMMwFrFR4bpxdE/9/LJeykiI8XF1gH8I2HFwACDUVNfW64n39mj5R4Vyuoxio8KV+x8X6affzVBMJF/RwPni0wMAPvLp/lLd9dI2HSw9KUn64dA0zf/RYCV3ifJxZUDgI+AAgA+892WxZj2/RTX1LnWPj9aiiZfo6kH0JQTaCwEHADrYa1sP6e6XP5fTZZR9cYr+MHm4Otn4OgbaE58oAOhAKz8u1MI3v5Qk/ed3uuvR64cqPIxbvYH2RsABgA5gjNEf1u7RE+/tkSRN+15v3T9hsKwMyAd4BQEHADrAS58edIeb3P+4SHdc3Z9HIwBeRMABAC/bZXdo/hs7JDWEmzvHDvBxRUDw48IvAHhRVU29Zj3XcLfUlRclafYP+vu6JCAkEHAAwIvuf2O79h2rUkqsTY//dBh9boAOQsABAC95+dODenXLYVkt0pOTR6hbZ5uvSwJCBgEHALxgT3GFR7+bzL7dfFwREFoIOADQzowxuvvlz3SyzqnvD0jU7VfR7wboaAQcAGhn7+yw67ND5YqJDNPv6HcD+AQBBwDakdNltOTdryRJt1zehwdnAj5CwAGAdvT61sPaW1KpuOgI3XJFX1+XA4QsAg4AtJPaepd+/17D2Zvbruqn2KgIH1cEhC4CDgC0kxc2F+nQ1yeV3MWmm7J6+7ocIKQRcACgHVTX1uvJtXslSXeMHaDoyDAfVwSENgIOALSDpzcc0PHKGmUkRGvSqAxflwOEPAIOAFyg8pN1WvbBPknSXdkXKTKcr1bA1/gUAsAFevnTgyo/WacByZ113fDuvi4HgAg4AHBBjDF6peCQJGnqmN4KY1A/wC8QcADgAuw44tAue4Uiw6z68dB0X5cD4DQCDgBcgMazN/9xSYriYhj3BvAXBBwAOE819U69vu2wJOmGkT18XA2AMxFwAOA8rdtZorLqOqXE2vT9AUm+LgfAGQg4AHCeGi9P/WREDzoXA36GgAMA56Gk4pTWf3VMkvRfXJ4C/A4BBwDOw+tbD8vpMhrRM179kzv7uhwA30LAAYA2OnPsG87eAP6pQwLO0qVL1bt3b0VFRSkzM1ObNm1qtu1VV10li8Vy1jRhwgR3m5tvvvms5ePHj++IXQEAfXG4XF8VV8oWbtUPGfsG8Evh3n6DF198Ubm5uVq2bJkyMzP1xBNPKCcnR7t371ZycvJZ7V999VXV1ta6X584cULDhg3TDTfc4NFu/PjxWrlypfu1zWbz3k4AwBle/rTh7E3OJamKi2bsG8Afef0MzuOPP64ZM2Zo2rRpGjx4sJYtW6aYmBitWLGiyfYJCQlKTU11T2vWrFFMTMxZAcdms3m069q1q7d3BQDkdBm99cVRSdL1XJ4C/JZXA05tba0KCgqUnZ39zRtarcrOzlZ+fn6rtrF8+XJNnjxZnTp18pi/fv16JScna+DAgbrtttt04sSJZrdRU1Mjh8PhMQHA+dh+uFylVbXqbAvXmH7dfF0OgGZ4NeAcP35cTqdTKSkpHvNTUlJkt9vPuf6mTZu0fft23XLLLR7zx48fr2eeeUZr167VI488og8++EDXXHONnE5nk9tZvHix4uLi3FNGRsb57xSAkPbh6VvDx/Trpogw7tMA/JXX++BciOXLl2vIkCEaPXq0x/zJkye7fx4yZIiGDh2qfv36af369Ro7duxZ28nLy1Nubq77tcPhIOQAOC8f7mkIOFcOZORiwJ959b8fiYmJCgsLU3Fxscf84uJipaamtrhuVVWVXnjhBU2fPv2c79O3b18lJiZq7969TS632WyKjY31mACgrRyn6rSlqEySdAWPZgD8mlcDTmRkpEaOHKm1a9e657lcLq1du1ZZWVktrvvyyy+rpqZGP//5z8/5PocOHdKJEyeUlpZ2wTUDQHM27D0hp8uob2InZSTE+LocAC3w+gXk3Nxc/fWvf9XTTz+tnTt36rbbblNVVZWmTZsmSZo6dary8vLOWm/58uWaOHGiunXz7MRXWVmpe+65R5988on279+vtWvX6rrrrlP//v2Vk5Pj7d0BEMIaL09dcRFnbwB/5/U+OJMmTdKxY8c0f/582e12DR8+XKtXr3Z3PC4qKpLV6pmzdu/erY8++kjvvvvuWdsLCwvT559/rqefflplZWVKT0/XuHHjtGjRIsbCAeA1xhh9sLsx4CT6uBoA52IxxhhfF9HRHA6H4uLiVF5eTn8cAK2y71ilxv7uA0WGWbVtwX8oJtKv79EAglJbfn9zjyMAtELj7eGjencl3AABgIADAK3QGHCupP8NEBAIOABwDjX1Tn3y71JJdDAGAgUBBwDO4dP9X+tknVPJXWwalNrF1+UAaAUCDgCcQ+Plqe8PSJLFYvFxNQBag4ADAOfwwVfcHg4EGgIOALSg2HFKu+wVslgazuAACAwEHABowf/tOS5JGto9TgmdIn1cDYDWIuAAQAsKDnwtSbqsX7dztATgTwg4ANCCbQfLJEkjMuJ9WgeAtiHgAEAzqmvr9VVxhSRpeEZXH1cDoC0IOADQjO2HHXK6jFJibUqNi/J1OQDagIADAM3YdrCh/81wLk8BAYeAAwDN+OxguSQuTwGBiIADAM1o7GA8LCPOt4UAaDMCDgA0oaTilA6XnZTFIg3tEe/rcgC0EQEHAJrQeHnqouQu6mwL93E1ANqKgAMATWjsYMzlKSAwEXAAoAmN/W/oYAwEJgIOAHyLy2X0+elLVJzBAQITAQcAvuXfxytVUVOv6IgwDUzp4utyAJwHAg4AfMvWojJJ0pDucQoP42sSCER8cgHgWz47VCaJy1NAICPgAMC30MEYCHwEHAA4w6k6p3YdPf0E8Z7xvi0GwHkj4ADAGXYcKVe9yyixs03pPEEcCFgEHAA4Q2MH4+EZ8bJYLL4tBsB5I+AAwBka+9+M4PIUENAIOABwBvcTxHnAJhDQCDgAcFp5dZ0OfX1SkjSkB7eIA4GMgAMAp31V0nD3VPf4aMVFR/i4GgAXokMCztKlS9W7d29FRUUpMzNTmzZtarbtqlWrZLFYPKaoKM87GYwxmj9/vtLS0hQdHa3s7Gzt2bPH27sBIMjtsjcEnItSOvu4EgAXyusB58UXX1Rubq4WLFigLVu2aNiwYcrJyVFJSUmz68TGxuro0aPu6cCBAx7LH330UT355JNatmyZNm7cqE6dOiknJ0enTp3y9u4ACGJfNQacVJ4/BQQ6rwecxx9/XDNmzNC0adM0ePBgLVu2TDExMVqxYkWz61gsFqWmprqnlJQU9zJjjJ544gn9+te/1nXXXaehQ4fqmWee0ZEjR/T66697e3cABLHdxQ0BhwdsAoHPqwGntrZWBQUFys7O/uYNrVZlZ2crPz+/2fUqKyvVq1cvZWRk6LrrrtOOHTvcywoLC2W32z22GRcXp8zMzGa3WVNTI4fD4TEBwJmMMfqqMeBwBgcIeF4NOMePH5fT6fQ4AyNJKSkpstvtTa4zcOBArVixQm+88YaeffZZuVwujRkzRocOHZIk93pt2ebixYsVFxfnnjIyMi501wAEmWMVNSqrrpPVIvVLog8OEOj87i6qrKwsTZ06VcOHD9eVV16pV199VUlJSfrzn/983tvMy8tTeXm5ezp48GA7VgwgGDRenuqd2ElREWE+rgbAhfJqwElMTFRYWJiKi4s95hcXFys1NbVV24iIiNCIESO0d+9eSXKv15Zt2mw2xcbGekwAcKbddvrfAMHEqwEnMjJSI0eO1Nq1a93zXC6X1q5dq6ysrFZtw+l06osvvlBaWpokqU+fPkpNTfXYpsPh0MaNG1u9TQD4tt3uW8QJOEAwCPf2G+Tm5uqmm27SqFGjNHr0aD3xxBOqqqrStGnTJElTp05V9+7dtXjxYknSgw8+qMsuu0z9+/dXWVmZHnvsMR04cEC33HKLpIY7rObMmaPf/OY3GjBggPr06aP7779f6enpmjhxord3B0CQauxgPIgOxkBQ8HrAmTRpko4dO6b58+fLbrdr+PDhWr16tbuTcFFRkazWb04kff3115oxY4bsdru6du2qkSNHasOGDRo8eLC7zb333quqqirNnDlTZWVluvzyy7V69eqzBgQEgNZwuYy+Kq6UxBg4QLCwGGOMr4voaA6HQ3FxcSovL6c/DgAVnajWFY+9r8hwq75cmKPwML+7/wKA2vb7m08xgJDXeAdV/6TOhBsgSPBJBhDydtsbBv9kgD8geBBwAIS83af73xBwgOBBwAEQ8r5iDBwg6BBwAIS02nqX9h3jDiog2BBwAIS0/SeqVO8y6mwLV3ocQ00AwYKAAyCkfTOCcWdZLBYfVwOgvRBwAIS0xhGM6WAMBBcCDoCQtosOxkBQIuAACGmNZ3DoYAwEFwIOgJBVXVuvotJqSZzBAYINAQdAyNpbUiljpMTOkerW2ebrcgC0IwIOgJD1zR1UnL0Bgg0BB0DIcve/IeAAQYeAAyBkFR6vkiT1T+7s40oAtDcCDoCQ9e/TAadvYicfVwKgvRFwAISkeqdLRSca7qDqk0TAAYINAQdASDr09UnVu4yiI8KU0oVnUAHBhoADICQ19r/pndhJVivPoAKCDQEHQEii/w0Q3Ag4AEJS4fFKSVIfAg4QlAg4AEJS4yUqAg4QnAg4AEJS4bHTAYc7qICgRMABEHJO1jp1pPyUJPrgAMGKgAMg5DRenuoaE6H4mEgfVwPAGwg4AEIO/W+A4EfAARByvrmDimdQAcGKgAMg5LjHwKGDMRC0CDgAQg6XqIDgR8ABEHIIOEDwI+AACClfV9WqrLpOktS7GwEHCFYdEnCWLl2q3r17KyoqSpmZmdq0aVOzbf/617/q+9//vrp27aquXbsqOzv7rPY333yzLBaLxzR+/Hhv7waAINDY/yY9LkrRkWE+rgaAt3g94Lz44ovKzc3VggULtGXLFg0bNkw5OTkqKSlpsv369et144036v3331d+fr4yMjI0btw4HT582KPd+PHjdfToUff097//3du7AiAIuC9P0cEYCGpeDziPP/64ZsyYoWnTpmnw4MFatmyZYmJitGLFiibbP/fcc7r99ts1fPhwDRo0SH/729/kcrm0du1aj3Y2m02pqanuqWvXrt7eFQBBgIdsAqHBqwGntrZWBQUFys7O/uYNrVZlZ2crPz+/Vduorq5WXV2dEhISPOavX79eycnJGjhwoG677TadOHGi2W3U1NTI4XB4TABC0zcdjBkDBwhmXg04x48fl9PpVEpKisf8lJQU2e32Vm3jvvvuU3p6ukdIGj9+vJ555hmtXbtWjzzyiD744ANdc801cjqdTW5j8eLFiouLc08ZGRnnv1MAAtq/Tz9kk2dQAcEt3NcFtOThhx/WCy+8oPXr1ysqKso9f/Lkye6fhwwZoqFDh6pfv35av369xo4de9Z28vLylJub637tcDgIOUAIcrmM9p/gFnEgFHj1DE5iYqLCwsJUXFzsMb+4uFipqaktrrtkyRI9/PDDevfddzV06NAW2/bt21eJiYnau3dvk8ttNptiY2M9JgChx+44pVN1LoVbLerRNdrX5QDwIq8GnMjISI0cOdKjg3Bjh+GsrKxm13v00Ue1aNEirV69WqNGjTrn+xw6dEgnTpxQWlpau9QNIDg19r/p2S1G4WEMAwYEM69/wnNzc/XXv/5VTz/9tHbu3KnbbrtNVVVVmjZtmiRp6tSpysvLc7d/5JFHdP/992vFihXq3bu37Ha77Ha7Kisb7nyorKzUPffco08++UT79+/X2rVrdd1116l///7Kycnx9u4ACGDuZ1BxeQoIel7vgzNp0iQdO3ZM8+fPl91u1/Dhw7V69Wp3x+OioiJZrd/krD/96U+qra3Vf/3Xf3lsZ8GCBXrggQcUFhamzz//XE8//bTKysqUnp6ucePGadGiRbLZbN7eHQABrPAY/W+AUGExxhhfF9HRHA6H4uLiVF5eTn8cIIRMW7lJ7+8+pod+MkQ/y+zp63IAtFFbfn9zERpAyOAhm0DoIOAACAm19S4d/PqkJKkvj2kAgh4BB0BIOPR1tZwuo+iIMCV3ob8eEOwIOABCwoET1ZKkXt1iZLFYfFwNAG8j4AAICQdOj2Dcq1uMjysB0BEIOABCwoHSxjM49L8BQgEBB0BIKDp9iapnAmdwgFBAwAEQEr45g0PAAUIBAQdA0HO5jIoaA04Cl6iAUEDAARD07I5Tqq1veIp4enyUr8sB0AEIOACCXuMt4j26RvMUcSBE8EkHEPSKShtuEe/JHVRAyCDgAAh67kH+uIMKCBkEHABBjzuogNBDwAEQ9BgDBwg9BBwAQc0Yo/3uxzTQBwcIFQQcAEGtrLpOFafqJXEGBwglBBwAQa2x/01KrE3RkWE+rgZARyHgAAhq7qeIM4IxEFIIOACCmruDMXdQASGFgAMgqLlvEaf/DRBSCDgAghpncIDQRMABENQabxHvzS3iQEgh4AAIWidrnSqpqJHEKMZAqCHgAAhaRaf738RGhSs+JtLH1QDoSAQcAEHrACMYAyGLgAMgaDWewaGDMRB6CDgAgtaBE9wiDoQqAg6AoMUdVEDoIuAACFpcogJCFwEHQFCqd7p0+OuTkrhFHAhFHRJwli5dqt69eysqKkqZmZnatGlTi+1ffvllDRo0SFFRURoyZIjefvttj+XGGM2fP19paWmKjo5Wdna29uzZ481dABBgjpSdUr3LKDLcqpQuUb4uB0AH83rAefHFF5Wbm6sFCxZoy5YtGjZsmHJyclRSUtJk+w0bNujGG2/U9OnTtXXrVk2cOFETJ07U9u3b3W0effRRPfnkk1q2bJk2btyoTp06KScnR6dOnfL27gAIEAdKG/rf9EyIkdVq8XE1ADqaxRhjvPkGmZmZ+u53v6s//vGPkiSXy6WMjAzdcccdmjdv3lntJ02apKqqKv3rX/9yz7vssss0fPhwLVu2TMYYpaena+7cubr77rslSeXl5UpJSdGqVas0efLkc9bkcDgUFxen8vJyxcbGttOeAvAnz35yQL9+fbvGDkrW8pu/6+tyALSDtvz+9uoZnNraWhUUFCg7O/ubN7RalZ2drfz8/CbXyc/P92gvSTk5Oe72hYWFstvtHm3i4uKUmZnZ7DZramrkcDg8JgDBjUH+gNDm1YBz/PhxOZ1OpaSkeMxPSUmR3W5vch273d5i+8Y/27LNxYsXKy4uzj1lZGSc1/4ACBzuMXDoYAyEpJC4iyovL0/l5eXu6eDBg74uCYCXcYs4ENq8GnASExMVFham4uJij/nFxcVKTU1tcp3U1NQW2zf+2ZZt2mw2xcbGekwAgpcxxh1wGMUYCE1eDTiRkZEaOXKk1q5d657ncrm0du1aZWVlNblOVlaWR3tJWrNmjbt9nz59lJqa6tHG4XBo48aNzW4TQGg5Vlmj6lqnrBapR1cCDhCKwr39Brm5ubrppps0atQojR49Wk888YSqqqo0bdo0SdLUqVPVvXt3LV68WJL0y1/+UldeeaV+97vfacKECXrhhRf06aef6i9/+YskyWKxaM6cOfrNb36jAQMGqE+fPrr//vuVnp6uiRMnent3AASAotP9b9LiohUZHhJX4gF8i9cDzqRJk3Ts2DHNnz9fdrtdw4cP1+rVq92dhIuKimS1fvMFNGbMGD3//PP69a9/rV/96lcaMGCAXn/9dV166aXuNvfee6+qqqo0c+ZMlZWV6fLLL9fq1asVFcVgXgCk/acDTu9Ezt4Aocrr4+D4I8bBAYLb4+/u1pPr9urG0T21+D+H+LocAO3Eb8bBAQBfOFDKLeJAqCPgAAg67jFwuIMKCFkEHABBhzFwABBwAAQVx6k6lVbVSuIxDUAoI+AACCqNt4gndo5UZ5vXbxQF4KcIOACCSmP/m570vwFCGgEHQFA5UMpTxAEQcAAEmSLO4AAQAQdAkHHfIs4dVEBII+AACCpFDPIHQAQcAEGkpt6pI+UnJdEHBwh1BBwAQeNg6UkZI3WKDFO3TpG+LgeADxFwAASNotN3UPXs1kkWi8XH1QDwJQIOgKDBM6gANCLgAAga3EEFoBEBB0DQ4CGbABoRcAAEjf0nTo9inMAdVECoI+AACApOl9Gh0sZbxDmDA4Q6Ag6AoGB3nFKt06WIMIvS46N9XQ4AHyPgAAgKB05fnurRNUZhVm4RB0IdAQdAUOAhmwDORMABEBQO8AwqAGcg4AAICo2XqDiDA0Ai4AAIEo2D/PXmIZsARMABEASMMe4+OFyiAiARcAAEga+r61RRUy9JyuASFQARcAAEgcb+N6mxUYqKCPNxNQD8AQEHQMArPN4QcPok0v8GQAMCDoCA5w44SQQcAA0IOAAC3r9PB5y+nMEBcBoBB0DAKzzGJSoAnrwacEpLSzVlyhTFxsYqPj5e06dPV2VlZYvt77jjDg0cOFDR0dHq2bOn7rzzTpWXl3u0s1gsZ00vvPCCN3cFgJ8yxtAHB8BZwr258SlTpujo0aNas2aN6urqNG3aNM2cOVPPP/98k+2PHDmiI0eOaMmSJRo8eLAOHDigW2+9VUeOHNErr7zi0XblypUaP368+3V8fLw3dwWAnyp21OhknVNhVgu3iANw81rA2blzp1avXq3Nmzdr1KhRkqSnnnpK1157rZYsWaL09PSz1rn00kv1j3/8w/26X79++u1vf6uf//znqq+vV3j4N+XGx8crNTXVW+UDCBD/Pt5wVrhnQowiwrjqDqCB174N8vPzFR8f7w43kpSdnS2r1aqNGze2ejvl5eWKjY31CDeSNGvWLCUmJmr06NFasWKFjDHNbqOmpkYOh8NjAhAcuDwFoCleO4Njt9uVnJzs+Wbh4UpISJDdbm/VNo4fP65FixZp5syZHvMffPBBXX311YqJidG7776r22+/XZWVlbrzzjub3M7ixYu1cOHC89sRAH6NDsYAmtLmMzjz5s1rspPvmdOuXbsuuDCHw6EJEyZo8ODBeuCBBzyW3X///fre976nESNG6L777tO9996rxx57rNlt5eXlqby83D0dPHjwgusD4B84gwOgKW0+gzN37lzdfPPNLbbp27evUlNTVVJS4jG/vr5epaWl5+w7U1FRofHjx6tLly567bXXFBER0WL7zMxMLVq0SDU1NbLZbGctt9lsTc4HEPgKGQMHQBPaHHCSkpKUlJR0znZZWVkqKytTQUGBRo4cKUlat26dXC6XMjMzm13P4XAoJydHNptN//znPxUVFXXO99q2bZu6du1KiAFCTJ3TpaLShqeIM4oxgDN5rQ/OxRdfrPHjx2vGjBlatmyZ6urqNHv2bE2ePNl9B9Xhw4c1duxYPfPMMxo9erQcDofGjRun6upqPfvssx4dgpOSkhQWFqY333xTxcXFuuyyyxQVFaU1a9booYce0t133+2tXQHgpw59fVL1LqPoiDCldDn3f4YAhA6vjoPz3HPPafbs2Ro7dqysVquuv/56Pfnkk+7ldXV12r17t6qrG/4HtmXLFvcdVv379/fYVmFhoXr37q2IiAgtXbpUd911l4wx6t+/vx5//HHNmDHDm7sCwA8Vnr5FvHdiJ1mtFh9XA8CfWExL91cHKYfDobi4OPct6AAC09/+79/6zVs7NWFImpZO+Y6vywHgZW35/c2oWAACFndQAWgOAQdAwCLgAGgOAQdAwHIHHO6gAvAtBBwAAam6tl5Hy09JYgwcAGcj4AAISPuPN9x92TUmQvExkT6uBoC/IeAACEj0vwHQEgIOgIDUOAZOn8TOPq4EgD8i4AAISP9ufAYVHYwBNIGAAyAgcYkKQEsIOAACEgEHQEsIOAACztdVtSqrrpMk9e5GwAFwNgIOgIDT2P8mPS5K0ZFhPq4GgD8i4AAIOIxgDOBcCDgAAs43t4gTcAA0jYADIOB808GYMXAANI2AAyDg7CthDBwALSPgAAgotfUu7TvWcIlqYEoXH1cDwF8RcAAElMLjVap3GXWxhSstLsrX5QDwUwQcAAFld3GFJOmi1C6yWCw+rgaAvyLgAAgoX9lPBxwuTwFoAQEHQEBpPIMzMIU7qAA0j4ADIKB81RhwUmN9XAkAf0bAARAwqmvrVVRaLUm6iDM4AFpAwAEQMPYUV8oYKbGzTd0623xdDgA/RsABEDDc/W9SOXsDoGUEHAABgzuoALQWAQdAwPjmDioCDoCWEXAABIzd9sZLVAQcAC0j4AAICF9X1aqkokaSNIAzOADOgYADICA0jn/To2u0OtvCfVwNAH9HwAEQEL6i/w2ANvBqwCktLdWUKVMUGxur+Ph4TZ8+XZWVlS2uc9VVV8lisXhMt956q0eboqIiTZgwQTExMUpOTtY999yj+vp6b+4KAB878yGbAHAuXj3PO2XKFB09elRr1qxRXV2dpk2bppkzZ+r5559vcb0ZM2bowQcfdL+OiYlx/+x0OjVhwgSlpqZqw4YNOnr0qKZOnaqIiAg99NBDXtsXAL7l7mDMGRwAreC1gLNz506tXr1amzdv1qhRoyRJTz31lK699lotWbJE6enpza4bExOj1NTUJpe9++67+vLLL/Xee+8pJSVFw4cP16JFi3TffffpgQceUGRkpFf2B4DvGGO4gwpAm3jtElV+fr7i4+Pd4UaSsrOzZbVatXHjxhbXfe6555SYmKhLL71UeXl5qq6u9tjukCFDlJKS4p6Xk5Mjh8OhHTt2NLm9mpoaORwOjwlA4Ch21Mhxql5hVov6JnXydTkAAoDXzuDY7XYlJyd7vll4uBISEmS325td72c/+5l69eql9PR0ff7557rvvvu0e/duvfrqq+7tnhluJLlfN7fdxYsXa+HChReyOwB8qLH/TZ/ETrKFh/m4GgCBoM0BZ968eXrkkUdabLNz587zLmjmzJnun4cMGaK0tDSNHTtW+/btU79+/c5rm3l5ecrNzXW/djgcysjIOO8aAXSsr+h/A6CN2hxw5s6dq5tvvrnFNn379lVqaqpKSko85tfX16u0tLTZ/jVNyczMlCTt3btX/fr1U2pqqjZt2uTRpri4WJKa3a7NZpPNxpOHgUC1i2dQAWijNgecpKQkJSUlnbNdVlaWysrKVFBQoJEjR0qS1q1bJ5fL5Q4trbFt2zZJUlpamnu7v/3tb1VSUuK+BLZmzRrFxsZq8ODBbdwbAIHgK54iDqCNvNbJ+OKLL9b48eM1Y8YMbdq0SR9//LFmz56tyZMnu++gOnz4sAYNGuQ+I7Nv3z4tWrRIBQUF2r9/v/75z39q6tSpuuKKKzR06FBJ0rhx4zR48GD94he/0GeffaZ33nlHv/71rzVr1izO0gBByOky2lPSGHBifVwNgEDh1YH+nnvuOQ0aNEhjx47Vtddeq8svv1x/+ctf3Mvr6uq0e/du911SkZGReu+99zRu3DgNGjRIc+fO1fXXX68333zTvU5YWJj+9a9/KSwsTFlZWfr5z3+uqVOneoybAyB4HCyt1qk6l2zhVvVMiDn3CgAgyWKMMb4uoqM5HA7FxcWpvLxcsbH8jxDwZ6u3H9Wtz27RJemxeuvO7/u6HAA+1Jbf3zyLCoBf++xQuSRpSPc4H1cCIJAQcAD4tW1FZZKk4RnxPq0DQGAh4ADwW06X0eeHyiRJwwg4ANqAgAPAb+07VqmqWqdiIsMYAwdAmxBwAPitxstTQ7rHKcxq8W0xAAIKAQeA39p2+vIU/W8AtBUBB4DfooMxgPNFwAHgl07WOt1PER/eM963xQAIOAQcAH5p+5FyOV1GyV1sSo2N8nU5AAIMAQeAXzrz8pTFQgdjAG1DwAHgl7YdLJPE5SkA54eAA8AvuQNOj3if1gEgMBFwAPidYxU1Olx2UhaLNKQHz6AC0HYEHAB+p/HszYDkzuoSFeHbYgAEJAIOAL/z2emAM4zLUwDOEwEHgN+hgzGAC0XAAeBXXC6jz3hEA4ALRMAB4Ff+fbxKFafqFRVh1UCeIA7gPBFwAPiVxstTQ7rHKTyMrygA54dvDwB+pbGDMZenAFwIAg4Av9J4BmcYAQfABSDgAPAbZdW12nGkXJI0sldXH1cDIJARcAD4jY/2HpfLNAzwlxYX7etyAAQwAg4Av/HhV8ckSVdelOTjSgAEOgIOAL9gjNGHXx2XJF1BwAFwgQg4APzCnpJK2R2nZAu3anSfBF+XAyDAEXAA+IXGy1OZfbspKiLMx9UACHQEHAB+4YPTAeeKAYk+rgRAMCDgAPC5k7VObSwslUQHYwDtg4ADwOc2Fp5Qbb1LaXFR6p/c2dflAAgCBBwAPtd499SVFyXJYrH4uBoAwcCrAae0tFRTpkxRbGys4uPjNX36dFVWVjbbfv/+/bJYLE1OL7/8srtdU8tfeOEFb+4KAC/6cM/p/jdcngLQTsK9ufEpU6bo6NGjWrNmjerq6jRt2jTNnDlTzz//fJPtMzIydPToUY95f/nLX/TYY4/pmmuu8Zi/cuVKjR8/3v06Pj6+3esH4H2Hy05qb0mlrBbpe/3oYAygfXgt4OzcuVOrV6/W5s2bNWrUKEnSU089pWuvvVZLlixRenr6WeuEhYUpNTXVY95rr72mn/70p+rc2fO6fHx8/FltAQSextvDh2fEKy4mwsfVAAgWXrtElZ+fr/j4eHe4kaTs7GxZrVZt3LixVdsoKCjQtm3bNH369LOWzZo1S4mJiRo9erRWrFghY0yz26mpqZHD4fCYAPiHxoDD5SkA7clrZ3DsdruSk5M93yw8XAkJCbLb7a3axvLly3XxxRdrzJgxHvMffPBBXX311YqJidG7776r22+/XZWVlbrzzjub3M7ixYu1cOHC89sRAF5T73Tpo708ngFA+2vzGZx58+Y12xG4cdq1a9cFF3by5Ek9//zzTZ69uf/++/W9731PI0aM0H333ad7771Xjz32WLPbysvLU3l5uXs6ePDgBdcH4MJ9dqhMFafqFRcdoWE94n1dDoAg0uYzOHPnztXNN9/cYpu+ffsqNTVVJSUlHvPr6+tVWlraqr4zr7zyiqqrqzV16tRzts3MzNSiRYtUU1Mjm8121nKbzdbkfAC+9c6OYknS5QMSFWbl9nAA7afNAScpKUlJSec+lZyVlaWysjIVFBRo5MiRkqR169bJ5XIpMzPznOsvX75cP/7xj1v1Xtu2bVPXrl0JMUAAqXe69OqWw5KkHw87+6YDALgQXuuDc/HFF2v8+PGaMWOGli1bprq6Os2ePVuTJ09230F1+PBhjR07Vs8884xGjx7tXnfv3r368MMP9fbbb5+13TfffFPFxcW67LLLFBUVpTVr1uihhx7S3Xff7a1dAeAFH3x1TMcra9StU6SuHpR87hUAoA28Og7Oc889p9mzZ2vs2LGyWq26/vrr9eSTT7qX19XVaffu3aqurvZYb8WKFerRo4fGjRt31jYjIiK0dOlS3XXXXTLGqH///nr88cc1Y8YMb+4KgHb2SsEhSdJ1w7srIoxB1QG0L4tp6f7qIOVwOBQXF6fy8nLFxsb6uhwg5HxdVavRD72nOqfR23d+X4PT+RwCOLe2/P7mv00AOtwb2w6rzml0SXos4QaAVxBwAHS4V7Y0XJ76r5E9fFwJgGBFwAHQoXYedWj7YYciwiy6bnh3X5cDIEgRcAB0qH+c7lw8dlCKEjpF+rgaAMGKgAOgw9Q5XXp9W8PYN1yeAuBNBBwAHWb97mM6XlmrxM6RunIgz54C4D0EHAAd5pWChufA/WQEY98A8C6+YQB0iAMnqrR2Z8Pz6a7n8hQALyPgAOgQv1/zlepdRldelKRBqYx9A8C7CDgAvG6X3aE3PjsiSbonZ6CPqwEQCgg4ALzud+9+JWOkCUPSdGn3OF+XAyAEEHAAeNXWoq+15stiWS3SXf9xka/LARAiCDgAvOqxd3ZLahj3pn9yZx9XAyBUEHAAeM3He49rw74Tigyz6s6xA3xdDoAQQsAB4BXGGD16+uzNzzJ7qkfXGB9XBCCUEHAAeMU7O+z67GCZYiLDNOsH/X1dDoAQQ8AB0O7s5af0q9e2S5KmX95HSV1sPq4IQKgh4ABoV/VOl+78+1aVVtVqcFosZ28A+AQBB0C7euK9Pdq0v1SdIsO0dMp3FBUR5uuSAIQgAg6AdvN/e45p6fq9kqTF1w9Vn8ROPq4IQKgi4ABoFyWOU5rzwjYZI904uqd+PCzd1yUBCGEEHAAX7FSdU3f8fatOVNVqUGoXLfjRYF+XBCDEEXAAXJCKU3WatnKzNhaWKoZ+NwD8RLivCwAQuE5U1ujmlZv1xeFydbaF669TR6lfEo9jAOB7BBwA5+VI2Un9YvlG7TtWpYROkXp62mgN6cGTwgH4BwIOgDbbba/QtJWbdKT8lNLjovTM9EwepAnArxBwALRaTb1T/+/9ffp/6/eqzmnUN6mT/r/pmeoeH+3r0gDAAwEHQKts3l+qef/4XPuOVUmSrh6UrMf+a6i6deYxDAD8DwEHQIsOnKjSn9bv0wubD0qSEjvb9MCPB2vCkDRZLBYfVwcATSPgADiLMUYf7z2hlR8Xat3uEhnTMH/ydzOUd83FiouJ8G2BAHAOBBwAkhpCzZ6SSq3bVaJ/FBzSnpJK97KrBibp9qv6a3SfBB9WCACt57WB/n77299qzJgxiomJUXx8fKvWMcZo/vz5SktLU3R0tLKzs7Vnzx6PNqWlpZoyZYpiY2MVHx+v6dOnq7KyspktAmjJsYoavfdlse5/fbsuf+R9jfv9h3r4f3dpT0mlOkWG6aasXlo390qtmjaacAMgoHjtDE5tba1uuOEGZWVlafny5a1a59FHH9WTTz6pp59+Wn369NH999+vnJwcffnll4qKipIkTZkyRUePHtWaNWtUV1enadOmaebMmXr++ee9tStAwCuvrlNRabWKSqu1p6RC2w+Xa/thh+yOUx7tIsOtyurbTdkXJ+u6Ed0VG8WlKACByWJM49V171i1apXmzJmjsrKyFtsZY5Senq65c+fq7rvvliSVl5crJSVFq1at0uTJk7Vz504NHjxYmzdv1qhRoyRJq1ev1rXXXqtDhw4pPb11D/dzOByKi4tTeXm5YmNjL2j/gI7kchmdrHM2TLVOOU7VqeJUvSpO1ctxsk5fV9fqeGWtjlfW6HhljY5V1OhgabUcp+qb3J7FIvVN7KTL+nbT1YOSNaZfoqIjecwCAP/Ult/fftMHp7CwUHa7XdnZ2e55cXFxyszMVH5+viZPnqz8/HzFx8e7w40kZWdny2q1auPGjfrJT37S5LZrampUU1Pjfu1wOLyyDwUHSvWvz496ZdvBxLuRujXv71lAU+U0NjFnLDXmm7YNy03DvDPaGiO5zDc/G2PkMpLLNC4zcrqM+0+naQgt9S6X6p1G9Wf8XFvvUk29S7VOl2rrXTpV51RNveu89zuxs009E6LVO7GTLk2P05AecRqcFqtONr/5GgCAduM332x2u12SlJKS4jE/JSXFvcxutys5OdljeXh4uBISEtxtmrJ48WItXLiwnSs+2257pVZ+vN/r7wNIUlSEVV2iItQlKlxdoiIUGxWuuOgIJXa2KamLTYmdI5XY2aYeXWOUkRCtmEi/+bgDgNe16Rtv3rx5euSRR1pss3PnTg0aNOiCimpveXl5ys3Ndb92OBzKyMho9/e5JD1Ws37Qr923i29Y1PZxV5oaquWsWU00sjSxyCKL+7Xl9LIzx4KxWCSrpaFKq8XiXm49Pd9qbfg57PTPYRaLwqwNU0SYRWFWq8LDLIqwWhUZfnoKa/gzKsKq6IgwRUeGKSo8TFYrY9AAQHPaFHDmzp2rm2++ucU2ffv2Pa9CUlNTJUnFxcVKS0tzzy8uLtbw4cPdbUpKSjzWq6+vV2lpqXv9pthsNtls3h9tdVhGvIZlxHv9fQAAQMvaFHCSkpKUlJTklUL69Omj1NRUrV271h1oHA6HNm7cqNtuu02SlJWVpbKyMhUUFGjkyJGSpHXr1snlcikzM9MrdQEAgMDjtXFwioqKtG3bNhUVFcnpdGrbtm3atm2bx5g1gwYN0muvvSap4TT+nDlz9Jvf/Eb//Oc/9cUXX2jq1KlKT0/XxIkTJUkXX3yxxo8frxkzZmjTpk36+OOPNXv2bE2ePLnVd1ABAIDg57Veh/Pnz9fTTz/tfj1ixAhJ0vvvv6+rrrpKkrR7926Vl5e729x7772qqqrSzJkzVVZWpssvv1yrV692j4EjSc8995xmz56tsWPHymq16vrrr9eTTz7prd0AAAAByOvj4PgjxsEBACDwtOX3t9cuUQEAAPgKAQcAAAQdAg4AAAg6BBwAABB0CDgAACDoEHAAAEDQIeAAAICgQ8ABAABBh4ADAACCjtce1eDPGgdvdjgcPq4EAAC0VuPv7dY8hCEkA05FRYUkKSMjw8eVAACAtqqoqFBcXFyLbULyWVQul0tHjhxRly5dZLFY2nXbDodDGRkZOnjwYNA+54p9DHzBvn8S+xgs2Mfg0F77aIxRRUWF0tPTZbW23MsmJM/gWK1W9ejRw6vvERsbG7T/UBuxj4Ev2PdPYh+DBfsYHNpjH8915qYRnYwBAEDQIeAAAICgQ8BpZzabTQsWLJDNZvN1KV7DPga+YN8/iX0MFuxjcPDFPoZkJ2MAABDcOIMDAACCDgEHAAAEHQIOAAAIOgQcAAAQdAg4bfTb3/5WY8aMUUxMjOLj45tsU1RUpAkTJigmJkbJycm65557VF9f3+J2S0tLNWXKFMXGxio+Pl7Tp09XZWWlF/agbdavXy+LxdLktHnz5mbXu+qqq85qf+utt3Zg5W3Tu3fvs+p9+OGHW1zn1KlTmjVrlrp166bOnTvr+uuvV3FxcQdV3Db79+/X9OnT1adPH0VHR6tfv35asGCBamtrW1zP34/j0qVL1bt3b0VFRSkzM1ObNm1qsf3LL7+sQYMGKSoqSkOGDNHbb7/dQZW23eLFi/Xd735XXbp0UXJysiZOnKjdu3e3uM6qVavOOl5RUVEdVHHbPfDAA2fVO2jQoBbXCaRjKDX93WKxWDRr1qwm2wfCMfzwww/1ox/9SOnp6bJYLHr99dc9lhtjNH/+fKWlpSk6OlrZ2dnas2fPObfb1s/zuRBw2qi2tlY33HCDbrvttiaXO51OTZgwQbW1tdqwYYOefvpprVq1SvPnz29xu1OmTNGOHTu0Zs0a/etf/9KHH36omTNnemMX2mTMmDE6evSox3TLLbeoT58+GjVqVIvrzpgxw2O9Rx99tIOqPj8PPvigR7133HFHi+3vuusuvfnmm3r55Zf1wQcf6MiRI/rP//zPDqq2bXbt2iWXy6U///nP2rFjh37/+99r2bJl+tWvfnXOdf31OL744ovKzc3VggULtGXLFg0bNkw5OTkqKSlpsv2GDRt04403avr06dq6dasmTpyoiRMnavv27R1ceet88MEHmjVrlj755BOtWbNGdXV1GjdunKqqqlpcLzY21uN4HThwoIMqPj+XXHKJR70fffRRs20D7RhK0ubNmz32b82aNZKkG264odl1/P0YVlVVadiwYVq6dGmTyx999FE9+eSTWrZsmTZu3KhOnTopJydHp06danabbf08t4rBeVm5cqWJi4s7a/7bb79trFarsdvt7nl/+tOfTGxsrKmpqWlyW19++aWRZDZv3uye97//+7/GYrGYw4cPt3vtF6K2ttYkJSWZBx98sMV2V155pfnlL3/ZMUW1g169epnf//73rW5fVlZmIiIizMsvv+yet3PnTiPJ5Ofne6HC9vfoo4+aPn36tNjGn4/j6NGjzaxZs9yvnU6nSU9PN4sXL26y/U9/+lMzYcIEj3mZmZnmf/7nf7xaZ3spKSkxkswHH3zQbJvmvpf81YIFC8ywYcNa3T7Qj6Exxvzyl780/fr1My6Xq8nlgXYMJZnXXnvN/drlcpnU1FTz2GOPueeVlZUZm81m/v73vze7nbZ+nluDMzjtLD8/X0OGDFFKSop7Xk5OjhwOh3bs2NHsOvHx8R5nRLKzs2W1WrVx40av19wW//znP3XixAlNmzbtnG2fe+45JSYm6tJLL1VeXp6qq6s7oMLz9/DDD6tbt24aMWKEHnvssRYvKxYUFKiurk7Z2dnueYMGDVLPnj2Vn5/fEeVesPLyciUkJJyznT8ex9raWhUUFHj8/VutVmVnZzf795+fn+/RXmr4bAbS8ZJ0zmNWWVmpXr16KSMjQ9ddd12z3zv+Ys+ePUpPT1ffvn01ZcoUFRUVNds20I9hbW2tnn32Wf33f/93iw96DrRjeKbCwkLZ7XaP4xQXF6fMzMxmj9P5fJ5bIyQftulNdrvdI9xIcr+22+3NrpOcnOwxLzw8XAkJCc2u4yvLly9XTk7OOR9W+rOf/Uy9evVSenq6Pv/8c913333avXu3Xn311Q6qtG3uvPNOfec731FCQoI2bNigvLw8HT16VI8//niT7e12uyIjI8/qh5WSkuJ3x6wpe/fu1VNPPaUlS5a02M5fj+Px48fldDqb/Kzt2rWryXWa+2wGwvFyuVyaM2eOvve97+nSSy9ttt3AgQO1YsUKDR06VOXl5VqyZInGjBmjHTt2eP0Bw+cjMzNTq1at0sCBA3X06FEtXLhQ3//+97V9+3Z16dLlrPaBfAwl6fXXX1dZWZluvvnmZtsE2jH8tsZj0ZbjdD6f59Yg4EiaN2+eHnnkkRbb7Ny585yd3wLJ+ezzoUOH9M477+ill1465/bP7D80ZMgQpaWlaezYsdq3b5/69et3/oW3QVv2MTc31z1v6NChioyM1P/8z/9o8eLFfj18+vkcx8OHD2v8+PG64YYbNGPGjBbX9YfjCGnWrFnavn17i/1TJCkrK0tZWVnu12PGjNHFF1+sP//5z1q0aJG3y2yza665xv3z0KFDlZmZqV69eumll17S9OnTfViZdyxfvlzXXHON0tPTm20TaMfQnxFwJM2dO7fFRC1Jffv2bdW2UlNTz+r53XhnTWpqarPrfLsjVX19vUpLS5td50Kdzz6vXLlS3bp1049//OM2v19mZqakhjMHHfWL8UKOa2Zmpurr67V//34NHDjwrOWpqamqra1VWVmZx1mc4uJirx2zprR1H48cOaIf/OAHGjNmjP7yl7+0+f18cRybkpiYqLCwsLPuWmvp7z81NbVN7f3F7Nmz3TcetPV/8BERERoxYoT27t3rperaV3x8vC666KJm6w3UYyhJBw4c0Hvvvdfms5+Bdgwbj0VxcbHS0tLc84uLizV8+PAm1zmfz3OrnHfvnRB3rk7GxcXF7nl//vOfTWxsrDl16lST22rsZPzpp5+6573zzjt+1cnY5XKZPn36mLlz557X+h999JGRZD777LN2rsw7nn32WWO1Wk1paWmTyxs7Gb/yyivuebt27fLrTsaHDh0yAwYMMJMnTzb19fXntQ1/Oo6jR482s2fPdr92Op2me/fuLXYy/uEPf+gxLysry287qLpcLjNr1iyTnp5uvvrqq/PaRn19vRk4cKC566672rk676ioqDBdu3Y1f/jDH5pcHmjH8EwLFiwwqamppq6urk3r+fsxVDOdjJcsWeKeV15e3qpOxm35PLeqtvNeM0QdOHDAbN261SxcuNB07tzZbN261WzdutVUVFQYYxr+MV566aVm3LhxZtu2bWb16tUmKSnJ5OXlubexceNGM3DgQHPo0CH3vPHjx5sRI0aYjRs3mo8++sgMGDDA3HjjjR2+f8157733jCSzc+fOs5YdOnTIDBw40GzcuNEYY8zevXvNgw8+aD799FNTWFho3njjDdO3b19zxRVXdHTZrbJhwwbz+9//3mzbts3s27fPPPvssyYpKclMnTrV3ebb+2iMMbfeeqvp2bOnWbdunfn0009NVlaWycrK8sUunNOhQ4dM//79zdixY82hQ4fM0aNH3dOZbQLpOL7wwgvGZrOZVatWmS+//NLMnDnTxMfHu+9g/MUvfmHmzZvnbv/xxx+b8PBws2TJErNz506zYMECExERYb744gtf7UKLbrvtNhMXF2fWr1/vcbyqq6vdbb69jwsXLjTvvPOO2bdvnykoKDCTJ082UVFRZseOHb7YhXOaO3euWb9+vSksLDQff/yxyc7ONomJiaakpMQYE/jHsJHT6TQ9e/Y0991331nLAvEYVlRUuH/3STKPP/642bp1qzlw4IAxxpiHH37YxMfHmzfeeMN8/vnn5rrrrjN9+vQxJ0+edG/j6quvNk899ZT79bk+z+eDgNNGN910k5F01vT++++72+zfv99cc801Jjo62iQmJpq5c+d6pPb333/fSDKFhYXueSdOnDA33nij6dy5s4mNjTXTpk1zhyZ/cOONN5oxY8Y0uaywsNDj76CoqMhcccUVJiEhwdhsNtO/f39zzz33mPLy8g6suPUKCgpMZmamiYuLM1FRUebiiy82Dz30kMcZt2/vozHGnDx50tx+++2ma9euJiYmxvzkJz/xCAz+ZOXKlU3+uz3zJG4gHsennnrK9OzZ00RGRprRo0ebTz75xL3syiuvNDfddJNH+5deeslcdNFFJjIy0lxyySXmrbfe6uCKW6+547Vy5Up3m2/v45w5c9x/HykpKebaa681W7Zs6fjiW2nSpEkmLS3NREZGmu7du5tJkyaZvXv3upcH+jFs9M477xhJZvfu3WctC8Rj2Pg77NtT4364XC5z//33m5SUFGOz2czYsWPP2vdevXqZBQsWeMxr6fN8PizGGHP+F7gAAAD8D+PgAACAoEPAAQAAQYeAAwAAgg4BBwAABB0CDgAACDoEHAAAEHQIOAAAIOgQcAAAQNAh4AAAgKBDwAEAAEGHgAMAAIIOAQcAAASd/x/0CCyWpsn/bQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "\n",
    "def sigmoid(x):\n",
    "    a = []\n",
    "    for item in x:\n",
    "        a.append(1/(1+math.exp(-item)))\n",
    "    return a\n",
    "\n",
    "def f2(x):\n",
    "    a = []\n",
    "    for item in x:\n",
    "        a.append(math.tanh(item))\n",
    "    return a\n",
    "\n",
    "x = np.arange(-10., 10., 0.2)\n",
    "y1 = sigmoid(x)\n",
    "y2 = f2(x)\n",
    "\n",
    "print(\"Sigmoid\")\n",
    "plt.plot(x,y1)\n",
    "plt.show()\n",
    "\n",
    "print(\"Hyperbolic Tangent(tanh)\")\n",
    "plt.plot(x,y2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jyCPwHyfnD2b"
   },
   "source": [
    "Both of these two functions compress their output to a specific range.  For the sigmoid function, this range is 0 to 1.  For the hyperbolic tangent function, this range is -1 to 1.\n",
    "\n",
    "LSTM maintains an internal state and produces an output.  The following diagram shows an LSTM unit over three timeslices: the current time slice (t), as well as the previous (t-1) and next (t+1) slice, as demonstrated by Figure 10.LSTM.\n",
    "\n",
    "**Figure 10.LSTM: LSTM Layers**\n",
    "![LSTM Layers](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_10_lstm1.png \"LSTM Layers\")\n",
    "\n",
    "The values $\\hat{y}$ are the output from the unit; the values ($x$) are the input to the unit, and the values $c$ are the context values.  The output and context values always feed their output to the next time slice.  The context values allow the network to maintain the state between calls.  Figure 10.ILSTM shows the internals of a LSTM layer.\n",
    "\n",
    "**Figure 10.ILSTM: Inside a LSTM Layer**\n",
    "![LSTM Layers](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_10_lstm2.png \"Inside the LSTM\")\n",
    "\n",
    "A LSTM unit consists of three gates:\n",
    "\n",
    "* Forget Gate ($f_t$) - Controls if/when the context is forgotten. (MC)\n",
    "* Input Gate ($i_t$) - Controls if/when the context should remember a value. (M+/MS)\n",
    "* Output Gate ($o_t$) - Controls if/when the remembered value is allowed to pass from the unit. (RM)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J0mVhDjAnD2c"
   },
   "source": [
    "## Simple LSTM Example\n",
    "\n",
    "The following code creates the LSTM network, an example of an RNN for classification.  The following code trains on a data set (x) with a max sequence size of 6 (columns) and six training elements (rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-lBrvaWKnD2c",
    "outputId": "37ee36c1-a820-4a8d-b5a7-e0d65f6bd311"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 1., 0., 0.],\n",
      "        [0., 0., 1., 0.],\n",
      "        [0., 0., 0., 1.],\n",
      "        [0., 0., 1., 0.],\n",
      "        [0., 0., 0., 1.],\n",
      "        [0., 1., 0., 0.]])\n",
      "Train...\n",
      "Epoch [10/200], Loss: 0.6272\n",
      "Epoch [20/200], Loss: 0.5205\n",
      "Epoch [30/200], Loss: 0.4737\n",
      "Epoch [40/200], Loss: 0.4391\n",
      "Epoch [50/200], Loss: 0.3834\n",
      "Epoch [60/200], Loss: 0.3432\n",
      "Epoch [70/200], Loss: 0.2931\n",
      "Epoch [80/200], Loss: 0.2421\n",
      "Epoch [90/200], Loss: 0.1975\n",
      "Epoch [100/200], Loss: 0.1602\n",
      "Epoch [110/200], Loss: 0.1146\n",
      "Epoch [120/200], Loss: 0.0834\n",
      "Epoch [130/200], Loss: 0.0584\n",
      "Epoch [140/200], Loss: 0.0479\n",
      "Epoch [150/200], Loss: 0.0241\n",
      "Epoch [160/200], Loss: 0.0174\n",
      "Epoch [170/200], Loss: 0.0144\n",
      "Epoch [180/200], Loss: 0.0120\n",
      "Epoch [190/200], Loss: 0.0122\n",
      "Epoch [200/200], Loss: 0.0095\n",
      "Predicted classes: [1 2 3 2 3 1]\n",
      "Expected classes: [1 2 3 2 3 1]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# Data\n",
    "max_features = 4\n",
    "x_data = [\n",
    "    [[0], [1], [1], [0], [0], [0]],\n",
    "    [[0], [0], [0], [2], [2], [0]],\n",
    "    [[0], [0], [0], [0], [3], [3]],\n",
    "    [[0], [2], [2], [0], [0], [0]],\n",
    "    [[0], [0], [3], [3], [0], [0]],\n",
    "    [[0], [0], [0], [0], [1], [1]]\n",
    "]\n",
    "x = torch.tensor(x_data, dtype=torch.float32)\n",
    "y = torch.tensor([1, 2, 3, 2, 3, 1], dtype=torch.int64)\n",
    "\n",
    "# Convert labels to one-hot encoding\n",
    "y2 = torch.nn.functional.one_hot(y, max_features).to(torch.float32)\n",
    "print(y2)\n",
    "\n",
    "# Model using a sequence\n",
    "class LSTMLayer(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        super(LSTMLayer, self).__init__()\n",
    "        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, batch_first=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out, _ = self.lstm(x)\n",
    "        return out\n",
    "\n",
    "model = nn.Sequential(\n",
    "    LSTMLayer(input_size=1, hidden_size=128),\n",
    "    nn.Dropout(p=0.2),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(128*6, 4),\n",
    "    nn.Sigmoid()\n",
    ")\n",
    "\n",
    "# Check for GPU availability\n",
    "model.to(device)\n",
    "x, y2 = x.to(device), y2.to(device)\n",
    "\n",
    "# Loss and optimizer\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "# Train the model\n",
    "print('Train...')\n",
    "for epoch in range(200):\n",
    "    optimizer.zero_grad()\n",
    "    outputs = model(x)\n",
    "    loss = criterion(outputs, y2)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    if (epoch + 1) % 10 == 0:\n",
    "        print(f\"Epoch [{epoch + 1}/200], Loss: {loss.item():.4f}\")\n",
    "\n",
    "\n",
    "# Predictions\n",
    "with torch.no_grad():\n",
    "    outputs = model(x)\n",
    "    predicted_classes = torch.argmax(outputs, 1)\n",
    "    print(f\"Predicted classes: {predicted_classes.cpu().numpy()}\")\n",
    "    print(f\"Expected classes: {y.cpu().numpy()}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CMIZvStnnD2c"
   },
   "source": [
    "We can now present a sequence directly to the model for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F3otdNdwnD2d",
    "outputId": "86763263-1ad2-4cd3-8487-df183e86db29"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "def runit(model, inp):\n",
    "    inp = torch.tensor(inp, dtype=torch.float32).to(device)\n",
    "    with torch.no_grad():\n",
    "        outputs = model(inp)\n",
    "    return torch.argmax(outputs[0]).item()\n",
    "\n",
    "print(runit(model, [[[0], [0], [0], [0], [0], [1]]]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ShrkldWxnKFo"
   },
   "source": [
    "## Sun Spots Example\n",
    "\n",
    "This section shows an example of RNN regression to predict sunspots.  You can find the data files needed for this example at the following location.\n",
    "\n",
    "* [Sunspot Data Files](http://www.sidc.be/silso/datafiles#total)\n",
    "* [Download Daily Sunspots](http://www.sidc.be/silso/INFO/sndtotcsv.php) - 1/1/1818 to now.\n",
    "\n",
    "We begin by loading and preparing data for the LSTM model. Next, we define a list of the column headers for a dataset. Following that, we read a CSV file from the given URL using the **pd.read_csv* function. This dataset is sunspot activity. The CSV is provided by the USA government and has certain specifications:\n",
    "\n",
    "* It uses a semicolon (;) as a separator.\n",
    "* The dataset doesn't have a header, so the header=None argument ensures pandas doesn't mistakenly take the first row as column names. Instead, the predefined names list is used as the columns' header.\n",
    "* Any value of '-1' in the dataset is considered as a missing value (na_values=['-1']).\n",
    "* The dataset is read without setting an index column (index_col=False), meaning the default integer index will be used.\n",
    "* Once executed, this code will load the specified dataset into a pandas DataFrame named df.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OoH-IH_bELLs"
   },
   "outputs": [],
   "source": [
    "# Data Loading\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from torch.optim.lr_scheduler import ReduceLROnPlateau\n",
    "\n",
    "names = ['year', 'month', 'day', 'dec_year', 'sn_value',\n",
    "         'sn_error', 'obs_num', 'unused1']\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/SN_d_tot_V2.0.csv\",\n",
    "    sep=';', header=None, names=names,\n",
    "    na_values=['-1'], index_col=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Li2i8ygFn7oc"
   },
   "source": [
    "Next we perform data preprocessing tasks for the sunspot dataset. Initially, the code identifies the last occurrence where the 'obs_num' column has a value, this is to strip off incomplete data near the beginning of the file. The **sn_value** column of the dataframe is then converted into floating-point numbers, ensuring numerical computations are consistent. After this, the dataframe is split into two parts based on the 'year' column: any data prior to the year 2000 is assigned to **df_train**, and data from the year 2000 onwards is assigned to **df_test**. Subsequently, the **sn_value** column from both the training and testing dataframes is extracted, transformed into numpy arrays, and reshaped to form 2D arrays with a single column.\n",
    "\n",
    "This restructuring is done to meet the input requirements of PyTorch. The final segment of the code focuses on data normalization. A StandardScaler is initialized, which is a tool to standardize data to have a mean of 0 and a standard deviation of 1. This scaler is trained on the training data and then used to normalize both the training and testing data. After normalization, both datasets are flattened and converted to lists, resulting in one-dimensional lists of normalized **sn_value** data for both training and testing purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xwQ5JVNWmZFk"
   },
   "outputs": [],
   "source": [
    "# Data Preprocessing\n",
    "start_id = max(df[df['obs_num'] == 0].index.tolist()) + 1\n",
    "df = df[start_id:].copy()\n",
    "df['sn_value'] = df['sn_value'].astype(float)\n",
    "df_train = df[df['year'] < 2000]\n",
    "df_test = df[df['year'] >= 2000]\n",
    "\n",
    "spots_train = df_train['sn_value'].to_numpy().reshape(-1, 1)\n",
    "spots_test = df_test['sn_value'].to_numpy().reshape(-1, 1)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "spots_train = scaler.fit_transform(spots_train).flatten().tolist()\n",
    "spots_test = scaler.transform(spots_test).flatten().tolist()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "popojYcVpiDE"
   },
   "source": [
    "The following code prepars the sequence data. This must be done in tasks like time series prediction or sequential data processing. The primary goal is to transform a list of observations into overlapping sequences of a specified length.\n",
    "\n",
    "The constant **SEQUENCE_SIZE** is set to 10, meaning that each sequence (or window) will consist of 10 observations.\n",
    "\n",
    "The function **to_sequences** is defined to facilitate this transformation. This function takes in two arguments: the size of each sequence (seq_size) and the list of observations (obs). Within the function, two empty lists, **x** and **y**, are initialized. Iterating over the **obs** list, for every index **i**, a window of size **seq_size** is extracted from **obs** and appended to the **x** list. The observation immediately following this window, or the (i + seq_size)-th observation, is appended to the **y** list. Essentially, **x** contains the sequences, and **y** contains the observations immediately following each sequence that are to be predicted. Once the lists are filled, they're converted into PyTorch tensors with the appropriate shapes and data type (torch.float32).\n",
    "\n",
    "Using this **to_sequences** function, the previously prepared **spots_train** and **spots_test** lists are transformed into their corresponding sequence datasets: **x_train**, **y_train**, **x_test**, and **y_test**. In this setup, if we consider **x_train** and **x_test**, each entry will represent a sequence of 10 observations, and the corresponding entry in **y_train** or **y_test** will represent the observation immediately following that sequence. This structure is particularly useful for tasks like predicting the next value in a time series based on a sequence of previous values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eVoIjklkmdDv"
   },
   "outputs": [],
   "source": [
    "# Sequence Data Preparation\n",
    "SEQUENCE_SIZE = 10\n",
    "\n",
    "def to_sequences(seq_size, obs):\n",
    "    x = []\n",
    "    y = []\n",
    "    for i in range(len(obs) - seq_size):\n",
    "        window = obs[i:(i + seq_size)]\n",
    "        after_window = obs[i + seq_size]\n",
    "        x.append(window)\n",
    "        y.append(after_window)\n",
    "    return torch.tensor(x, dtype=torch.float32).view(-1, seq_size, 1), torch.tensor(y, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "x_train, y_train = to_sequences(SEQUENCE_SIZE, spots_train)\n",
    "x_test, y_test = to_sequences(SEQUENCE_SIZE, spots_test)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aoalnjZPqmlX"
   },
   "source": [
    "Next we set up data loaders for PyTorch, a crucial step when training neural networks in batches. First, the training data (**x_train** and **y_train**) is encapsulated into a TensorDataset. This structure pairs input data and its corresponding target, making it easier to manage. Similarly, the testing data (**x_test** and **y_test**) is also wrapped into a **TensorDataset**. Once the datasets are structured, they are passed to the **DataLoader** function.\n",
    "\n",
    "For the training data, a **DataLoader** is created with a batch size of 32, and the shuffle parameter is set to True, which means tha during each epoch of training, the training data will be divided into batches of 32 samples, and these batches will be randomly shuffled. This shuffling is to ensure the model isn't exposed to any inherent order in the data during training, promoting better generalization.\n",
    "\n",
    "Conversely, for the testing data, while the batch size remains 32, the shuffle parameter is set to False, indicating that the order of the test data remains unchanged. This is typical as shuffling the test data isn't necessary and can often make evaluation metrics easier to interpret. By the end of this code, two data loaders (train_loader and test_loader) are established, ready to feed data in batches to a neural network during both training and evaluation phases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "449lnEVlmgki"
   },
   "outputs": [],
   "source": [
    "# Setup data loaders for batch\n",
    "train_dataset = TensorDataset(x_train, y_train)\n",
    "train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "test_dataset = TensorDataset(x_test, y_test)\n",
    "test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fbhQn0SovB_U"
   },
   "source": [
    "\n",
    "We define an **LSTMModel** class, which is a neural network architecture using PyTorch that provides more precise control than a sequence. This model is structured with an LSTM layer followed by a dropout layer for regularization and two subsequent fully connected layers. The LSTM layer processes input sequences, producing a series of hidden states. The model then utilizes only the last hidden state of this series, passing it through the dropout layer and the two linear layers in sequence. This architecture is geared towards processing sequential data, where the LSTM can capture temporal dependencies and the fully connected layers further refine the representation for the final output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HRhEO-hwmj5D"
   },
   "outputs": [],
   "source": [
    "# Model definition\n",
    "class LSTMModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LSTMModel, self).__init__()\n",
    "        self.lstm = nn.LSTM(input_size=1, hidden_size=64, batch_first=True)\n",
    "        self.dropout = nn.Dropout(0.2)\n",
    "        self.fc1 = nn.Linear(64, 32)\n",
    "        self.fc2 = nn.Linear(32, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x, _ = self.lstm(x)\n",
    "        x = self.dropout(x[:, -1, :])\n",
    "        x = self.fc1(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "model = LSTMModel().to(device)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h4G3DvX6vq77"
   },
   "source": [
    "\n",
    "The code outlines the training process for a neural network model. It sets up a mean squared error (MSE) loss function and uses the Adam optimizer with a learning rate of 0.001. The learning rate scheduler, **ReduceLROnPlateau**, adjusts the learning rate when the validation loss plateaus, decreasing it by a factor of 0.5 if there's no improvement for three epochs. The model trains for a maximum of 1000 epochs but incorporates early stopping; if the validation loss doesn't improve for five consecutive epochs, the training halts prematurely. During each epoch, the model's weights are updated using the training data. Subsequently, the model's performance is evaluated on the validation data, and the average validation loss is computed. The progress of the training, including the current epoch and corresponding validation loss, is printed to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uVPwS7EdmnO5",
    "outputId": "97faca55-8f23-49a3-937e-2b176931566f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000, Validation Loss: 0.0339\n",
      "Epoch 2/1000, Validation Loss: 0.0376\n",
      "Epoch 3/1000, Validation Loss: 0.0346\n",
      "Epoch 4/1000, Validation Loss: 0.0346\n",
      "Epoch 00005: reducing learning rate of group 0 to 5.0000e-04.\n",
      "Epoch 5/1000, Validation Loss: 0.0340\n",
      "Epoch 6/1000, Validation Loss: 0.0334\n",
      "Epoch 7/1000, Validation Loss: 0.0343\n",
      "Epoch 8/1000, Validation Loss: 0.0333\n",
      "Epoch 9/1000, Validation Loss: 0.0334\n",
      "Epoch 10/1000, Validation Loss: 0.0339\n",
      "Epoch 11/1000, Validation Loss: 0.0342\n",
      "Epoch 00012: reducing learning rate of group 0 to 2.5000e-04.\n",
      "Epoch 12/1000, Validation Loss: 0.0349\n",
      "Early stopping!\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "scheduler = ReduceLROnPlateau(optimizer, 'min', factor=0.5, patience=3, verbose=True)\n",
    "\n",
    "epochs = 1000\n",
    "early_stop_count = 0\n",
    "min_val_loss = float('inf')\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    model.train()\n",
    "    for batch in train_loader:\n",
    "        x_batch, y_batch = batch\n",
    "        x_batch, y_batch = x_batch.to(device), y_batch.to(device)\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(x_batch)\n",
    "        loss = criterion(outputs, y_batch)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    # Validation\n",
    "    model.eval()\n",
    "    val_losses = []\n",
    "    with torch.no_grad():\n",
    "        for batch in test_loader:\n",
    "            x_batch, y_batch = batch\n",
    "            x_batch, y_batch = x_batch.to(device), y_batch.to(device)\n",
    "            outputs = model(x_batch)\n",
    "            loss = criterion(outputs, y_batch)\n",
    "            val_losses.append(loss.item())\n",
    "\n",
    "    val_loss = np.mean(val_losses)\n",
    "    scheduler.step(val_loss)\n",
    "\n",
    "    if val_loss < min_val_loss:\n",
    "        min_val_loss = val_loss\n",
    "        early_stop_count = 0\n",
    "    else:\n",
    "        early_stop_count += 1\n",
    "\n",
    "    if early_stop_count >= 5:\n",
    "        print(\"Early stopping!\")\n",
    "        break\n",
    "    print(f\"Epoch {epoch + 1}/{epochs}, Validation Loss: {val_loss:.4f}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y8h1ZJOQwUAX"
   },
   "source": [
    "Finally, we evaluate the PyTorch neural network model on test data. It initiates by setting the model to evaluation mode using **model.eval**, which ensures that specific layers like dropout are fixed during inference. The predictions list is initialized to store the model's predictions on the test data. The **torch.no_grad** context is used to disable gradient calculations, optimizing memory usage and speed during evaluation. Inside this context, the code iterates over the **test_loader** to fetch batches of test data.\n",
    "\n",
    "Each data batch is then transferred to the computing device, and the model is subsequently used to generate predictions on this batch. These predictions are added to the predictions list. After processing all test batches, the code calculates the Root Mean Square Error (RMSE) between the predicted values and the actual targets (**y_test**). Notably, the predictions and targets are inverse-transformed using the scaler to revert the normalization and compute the RMSE in the original data scale. The computed RMSE, a measure of prediction error, is then printed to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hqjH9mKZmqvk",
    "outputId": "a81aae30-716d-4f26-c712-38414bf42f0f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (RMSE): 14.5177\n"
     ]
    }
   ],
   "source": [
    "# Evaluation\n",
    "model.eval()\n",
    "predictions = []\n",
    "with torch.no_grad():\n",
    "    for batch in test_loader:\n",
    "        x_batch, y_batch = batch\n",
    "        x_batch = x_batch.to(device)\n",
    "        outputs = model(x_batch)\n",
    "        predictions.extend(outputs.squeeze().tolist())\n",
    "\n",
    "rmse = np.sqrt(np.mean((scaler.inverse_transform(np.array(predictions).reshape(-1, 1)) - scaler.inverse_transform(y_test.numpy().reshape(-1, 1)))**2))\n",
    "print(f\"Score (RMSE): {rmse:.4f}\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}