LSTM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LSTM.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "eYhLQOn3YqGA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#**LSTM**\n",
        "The problem we are trying to solve here is to classify variable sized sequences of drawings into 10 categories (apple, banana, fork...). \n",
        "\n",
        "We are tackling this problem with an LSTM (Long-Short Memory Network). The input of the LSTM are the keypoints of the drawings in the order in which they were drawn.\n",
        "\n",
        "The dataset we will use is extracted from the Kaggle competition: **Quick Draw! Doodle Recognition Challenge ** (https://www.kaggle.com/c/quickdraw-doodle-recognition). "
      ]
    },
    {
      "metadata": {
        "id": "dwTJ68cmcdlr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#**1. Notebook Setting**\n",
        "\n",
        "Import Pytorch and Python libraries (Numpy, Matplotlib...)"
      ]
    },
    {
      "metadata": {
        "id": "b_k3w0d1hxzB",
        "colab_type": "code",
        "outputId": "129a04bb-b31c-4683-c874-3c9428c8777b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "cell_type": "code",
      "source": [
        "# http://pytorch.org/\n",
        "from os.path import exists\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n",
        "accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n",
        "\n",
        "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision\n",
        "  \n",
        "import numpy as np\n",
        "import os\n",
        "import sys\n",
        "import torch\n",
        "import torchvision\n",
        "import random\n",
        "import codecs\n",
        "import torch.utils.data\n",
        "import torch.optim as optim\n",
        "import torchvision.datasets as datasets\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from matplotlib import pyplot as plt\n",
        "from tqdm import tqdm\n",
        "from torchvision import datasets, transforms\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tcmalloc: large alloc 1073750016 bytes == 0x579c8000 @  0x7f42104c92a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ADImzkBIqHd_",
        "colab_type": "code",
        "outputId": "054bf527-8c4e-4240-b987-9e7908069c9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "cell_type": "code",
      "source": [
        "#Training on the GPU\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(device)\n",
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cuda:0\n",
            "Sun Dec 16 18:24:47 2018       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   31C    P8    32W / 149W |     11MiB / 11441MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1euniB3GOHmk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **2. Dataset Preparation**\n",
        "\n",
        "Download, reduce, reshape and reorganize dataset\n"
      ]
    },
    {
      "metadata": {
        "id": "E1ww9hWadKaR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.1 Download the Dataset:**\n",
        "\n",
        "The dataset is downloaded from Google Drive and it comes in the form of csv files with the sequences. The link is public."
      ]
    },
    {
      "metadata": {
        "id": "cfmjdhj88dfN",
        "colab_type": "code",
        "outputId": "3f4ebfd3-7145-4ed5-f600-7865fce02232",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "cell_type": "code",
      "source": [
        "  !pip install googledrivedownloader\n",
        "  \n",
        "  from google_drive_downloader import GoogleDriveDownloader as gdd\n",
        "  urls = [\n",
        "        '1JmW1jzvcDCSRg2vr6nBfmnWyR2K665Ny', \n",
        "        '1_DRuC1dnG6Rsfb-a32eXD0fh_dpyiuJU',\n",
        "        '1cBXEmlIAwuOmwPEgd9kbnSvjofGEFIAU',\n",
        "        '1OZdyx5rXytzXnvTq8S3LZS_Ewl_iyIio',\n",
        "        '1585P-SU8G_vNGu78yEJhOpH9-ajxz3fm',\n",
        "        '1a9KvLtNi3crhi3iqBX93UwKlmPlmxnyi',\n",
        "        '1lnneEBuc2K4papzkui14ZiYN58dMM8VZ',\n",
        "        '1KdDex8cjZc-SNR8NVURsK-OaXtfxEskU',\n",
        "        '1fODQI_9LtXyXVk9RNGC7VWZxUlsahxKk',\n",
        "        '1hbRfukgoLJoQeGtjL820cR89hEqBkzk4'\n",
        "    ]\n",
        "  \n",
        "  class_name = ['apple', 'banana', 'book', 'fork', 'key', 'ladder', 'pizza', 'stop_sign', 'tennis_racquet', 'wheel']\n",
        "\n",
        "    \n",
        "  def createDir(path):\n",
        "    if not os.path.exists(path):\n",
        "        os.makedirs(path)\n",
        "  extension = '.csv'\n",
        "                  \n",
        "  for i in range(0, len(urls)):\n",
        "    createDir('data')\n",
        "    name = class_name[i] + extension\n",
        "    gdd.download_file_from_google_drive(file_id = urls[i],\n",
        "                                       dest_path = os.path.join('data', name))\n",
        "    \n",
        "    \n",
        "  print(\"Done!\")   "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.6/dist-packages (0.3)\n",
            "Downloading 1JmW1jzvcDCSRg2vr6nBfmnWyR2K665Ny into data/apple.csv... Done.\n",
            "Downloading 1_DRuC1dnG6Rsfb-a32eXD0fh_dpyiuJU into data/banana.csv... Done.\n",
            "Downloading 1cBXEmlIAwuOmwPEgd9kbnSvjofGEFIAU into data/book.csv... Done.\n",
            "Downloading 1OZdyx5rXytzXnvTq8S3LZS_Ewl_iyIio into data/fork.csv... Done.\n",
            "Downloading 1585P-SU8G_vNGu78yEJhOpH9-ajxz3fm into data/key.csv... Done.\n",
            "Downloading 1a9KvLtNi3crhi3iqBX93UwKlmPlmxnyi into data/ladder.csv... Done.\n",
            "Downloading 1lnneEBuc2K4papzkui14ZiYN58dMM8VZ into data/pizza.csv... Done.\n",
            "Downloading 1KdDex8cjZc-SNR8NVURsK-OaXtfxEskU into data/stop_sign.csv... Done.\n",
            "Downloading 1fODQI_9LtXyXVk9RNGC7VWZxUlsahxKk into data/tennis_racquet.csv... Done.\n",
            "Downloading 1hbRfukgoLJoQeGtjL820cR89hEqBkzk4 into data/wheel.csv... Done.\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jPNS_XT5nCIN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.2 Reduction, reshape and reorganization of the Dataset:**"
      ]
    },
    {
      "metadata": {
        "id": "JQuGOMzd-oBm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We selected 10 classes from the original datset, which was composed by more than 300 classes. \n",
        "For each drawing, we have a **variable sized** sequence that contains the ordered keypoints of the drawings. By keypoints we mean the important points in a drawing to define its structure (see section 2.3 Dataset Visualization for more information). We will get these keypoints from the csv file and save them to a folder (train, validation and test) in a \".npy\" format. \n",
        "\n",
        "We can specify how many samples of each class we want for each class, in this case 10.000. When we run this cell, we will have 10.000 samples/class proportionally distributed taking into account the percen array. By default: 6.000 samples/class for training, 3.000 samples/class for validation and 1.000 samples/class for testing."
      ]
    },
    {
      "metadata": {
        "id": "C7wbAlMHZ1kr",
        "colab_type": "code",
        "outputId": "43cd63be-0315-4e5a-f25d-f507b8827a51",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254
        }
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import json\n",
        "\n",
        "class_name = ['apple', 'banana', 'book', 'fork', 'key', 'ladder', 'pizza', 'stop_sign', 'tennis_racquet', 'wheel']\n",
        "step = ['train', 'validation', 'test']\n",
        "\n",
        "dire = r'data/'\n",
        "\n",
        "max_length = 10000 # Maximum number of files (drawings) per class\n",
        "percen=[0.6, 0.3, 0.1] # Percentage of training, validation and testing\n",
        "\n",
        "begin = [0, int(max_length * percen[0]), int(max_length * (percen[0] + percen[1]))]\n",
        "end = [int(max_length * (percen[0])), int(max_length * (percen[0] + percen[1])), max_length-10]\n",
        "print(begin)\n",
        "print(end)\n",
        "for c in range(0, len(class_name)):\n",
        "  print('Class ' + str(c+1) + ' out of ' + str(len(class_name)))\n",
        "  filename = dire + str(class_name[c]) + '.csv'\n",
        "  \n",
        "  csv = pd.read_csv(filename, sep = ',')\n",
        "  drawing = csv[csv['recognized']==True]\n",
        "  drawings = csv['drawing']\n",
        "  drawings = drawings.values\n",
        "  data = drawings\n",
        "  \n",
        "  for s in range(0, len(step)):\n",
        "    dire_step = str(dire) + str(step[s])\n",
        "    if not os.path.exists(dire_step):\n",
        "      os.makedirs(dire_step)\n",
        "    \n",
        "    for i in range(begin[s], end[s]):\n",
        "      dire_class = str(dire_step) + '/' + str(class_name[c])\n",
        "      if not os.path.exists(dire_class):\n",
        "        os.makedirs(dire_class)\n",
        "        \n",
        "       \n",
        "      x = np.array(json.loads(drawings[i]))\n",
        "      drawing_strokes = []\n",
        "      for elem in x:\n",
        "        mat = np.zeros((2, len(elem[0])))\n",
        "        mat[0, :] = elem[0][:]\n",
        "        mat[1, :] = elem[1][:]\n",
        "        drawing_strokes.append(mat)\n",
        "      aux = np.zeros((2,1)) \n",
        "      for stroke in drawing_strokes:\n",
        "        aux = np.hstack((aux, stroke))\n",
        "      sample_name = class_name[c] + '_' + str(step[s]) + '_' + str(i)\n",
        "      \n",
        "      np.save(os.path.join(dire_class, sample_name), aux[:, 1:])\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0, 6000, 9000]\n",
            "[6000, 9000, 9990]\n",
            "Class 1 out of 10\n",
            "Class 2 out of 10\n",
            "Class 3 out of 10\n",
            "Class 4 out of 10\n",
            "Class 5 out of 10\n",
            "Class 6 out of 10\n",
            "Class 7 out of 10\n",
            "Class 8 out of 10\n",
            "Class 9 out of 10\n",
            "Class 10 out of 10\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "HhoE0veRfHoT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.3 Dataset Visualization:**\n",
        "\n",
        "Let us draw the keypoints of a random drawing of an apple. The number next to each keypoint is the order in which this keypoint was drawn. \n",
        "\n",
        "Each time instant our LSTM will process a point in the drawing in the order it was drawn."
      ]
    },
    {
      "metadata": {
        "id": "jh0RVxYHnZbS",
        "colab_type": "code",
        "outputId": "bc325a66-4481-4981-dbc1-05a0f8641086",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 347
        }
      },
      "cell_type": "code",
      "source": [
        "csv = pd.read_csv('data/apple.csv', sep = ',',engine = 'python')\n",
        "\n",
        "# Header Format\n",
        "#['countrycode', 'drawing', 'key_id', 'recognized', 'timestamp', 'word'] \n",
        "\n",
        "drawing = csv[csv['recognized']==True]\n",
        "drawings = csv['drawing']\n",
        "drawings = drawings.values\n",
        "index = 100000\n",
        "\n",
        "x = np.array(json.loads(drawings[index]))\n",
        "j = 0\n",
        "for elem in x:\n",
        "  for i in range(0,len(elem[0])-1):\n",
        "    \n",
        "    plt.plot([elem[0][i], elem[0][i+1]], [-elem[1][i], -elem[1][i+1]], marker = 'o', color = 'b')\t\n",
        "    j += 1\n",
        "    plt.text(elem[0][i]+3, -elem[1][i] -15, str(j)) # The +3 and -15\n",
        "   \n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAFKCAYAAADScRzUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXl8TFf/x9+zTxYhJLETsdZeD10U\n4antp2ittVZr31ppVVvR1h6qaiktSuxFE0uppbYKYt9Kn+cR+xIUUYIss9/fH1cikZmIZCbreb9e\nXjF35t5z5uTmfs75nu+ikCRJQiAQCAQCQa5EmdMdEAgEAoFA4Bgh1AKBQCAQ5GKEUAsEAoFAkIsR\nQi0QCAQCQS5GCLVAIBAIBLkYIdQCgUAgEORi1DndAXvExDx22rW8vd158CDBadcTyIhxdR1ibF2H\nGFvXIMY16/j6FnL4Xr5fUavVqpzuQr5EjKvrEGPrOsTYugYxrq4l3wu1QCAQCAR5GSHUAoFAIBDk\nYoRQCwQCgUCQixFCLRAIBAJBLkYItUAgEAgEuRgh1AKBQCAQ5GKEUAsEAoFAkIsRQi0QCAQCQS4m\n24Q6JCSEd999l27dunHmzJnsalYgEAgEuYwNG9QEBrpTsqQngYHubNiQK5Nk5hqyZXSOHj3KtWvX\n+OWXX7h06RLBwcH88ssv2dG0QCAQCHIRGzaoGTTILfn12bOqJ68T6dDBknMdy8Vky4r60KFDNG/e\nHICKFSvy8OFD4uLisqNpgUAgEOQiZs3S2j0+e7b944JsWlHfu3ePGjVqJL8uWrQoMTExeHp62v28\nt7e7U3PHppfsXJB5xLi6DjG2rkOMrWvI6LieP+/ouEr8bhyQIxsDkiSl+74zq7D4+hZyajUugYwY\nV9chxtZ1iLF1DS8yrlWquHP2bNqFWJUqVmJiCm4FrhyvnuXn58e9e/eSX9+9exdfX9/saFogEAgE\nuYigIJPd4yNG2D8uyCahfuONN9i+fTsA//3vf/Hz83No9hYIBAKBcwgO1lG2rCd+fp6ULetJcLAu\nx9stU8YGgIeHhFotUb26lQULhCNZemSL6btevXrUqFGDbt26oVAoGDt2bHY0KxAIBAWW4GAdixY9\nddAyGkl+HRJizLF2Z82SRXvVqkRef93qsn7kJxTS8zaMcwBn7iGJPSnXIMbVdYixdR35fWwTEuC/\n/1Vy5oyKMWN02GwKO5+SKFvWuY99pVKJzSavlKOjFUDadnU6ia1bE3jzTQ9efdXCb78lOrUPeZ30\n9qhFlLlAIBDkQeLj4T//UXHmjCzMZ84oOXdO6UCccx6jEfr0keOnBw+W96ODg3WsWKHBaASdDnr3\nNrt0tZ9XEUItEAgEuZy4uKeifPq0/PPChdSi7O4u0aCBlTp1bNSubeWTT/SYTPZWtnDiRLxT+ydb\nKuRrli3ridGB1t64IbtFDRzoRvHiUvJryD7TfF5ECLVAIBDkIpJE+fTp1KIsSU9F18ND4pVXnopy\nnTo2Kla0oUoR9fTnn+ZUe8VJ9O5tdmn/e/e2327lyjYuXFDRrp2ZS5eU/O9/9nNlrFihEUL9DEKo\nBQKBIId4/DitKF+8mFqUPT0lXnvNSu3aNurUeSrKyufE7CSJXXablu21+847Ztau1VC1qpWFCw0o\nleDn54m9vWxHq/GCjBBqgUAgyAYeP4YzZ2RRln+quHxZkUqUCxWSaNjQSq1aSaJsJSBAeq4oOyIk\nxJgjq9Nn2/3sMx1Wq4KPPjIlfxedzr4o67IngixPIYRaIBAInMyjR/ZEObXaenlJvPFGypWyFX//\nzItybuXOHQWrV2soX96WKlbakYnc1ab5vIgQaoFAIMgCDx+mFeUrV1KrbeHCEo0bW5JFuXbt/CnK\n9vjxRy1Go4IPPzSiTqE4z5rIQUGNGlaxP20HIdQCgaDAk9EwodhYksU4yQP76lX7opy0n5wkyorc\nGTWVzMGDkXz2WRDh4ZsoUaIk8+fPZd++PSgUCpo0acbgwcNf+Jr378OyZRpKlrTx7rtpV8pt2+5m\n374gVq/eRPPmlblw4QYNGgSh03kxcuR8ka3sCUKoBQJBgcZRJi2jEdq1sySvlk+fVnH9empRLlJE\nokmT1KJcvnzuF+VnMRgMzJ8/By+vwgDs3r2DU6dOsGzZGhQKBcOHD2TPnl00a9b8ha77009aEhIU\njB5tTLP3nLLN3btVxMVdpVSpYSQmNsBiuS5qVKdACLVAICjQrFihcXBcy4oVTwXc21siMDC1KJcr\nl/dE2R6LFy+gVas2bNiwFoA9e3bRpk1btFr5+7dq1YY9e3a/kFA/fgyhoVqKFbPRq1fa1XTKNhcv\n1iJJKm7cWIaHRyQazXVArlEthDqbinIIBAJBbsRsTi8cSCIoyMjixYmcOBFHVFQc4eGJfPmliXbt\nLHly5WyPS5cucuzYEd59t2fysejo65QuXSb5denSZbh27eoLXXfJEi0PHyoYNMiMh0f6bV65osBi\nKY3V6pfqc+fPyxJ18GAkjRrV5++/b2GxWJg5cxo9enSiW7eOfPttCBZL/hZzIdQCgaDAceuWgqlT\ntdSr54G9WF6Q96qDg020bWuhbNn8IcrPIkkS06eHEBT0GeoUnl4GgwGt9qmtWqfTYTBkPDd3QgLM\nn6/By0uib9/U5SvttVmhgv3c41Wq2NKY5cPCVnP9+jWWLVvDihW/cPnyJbZu/S3DfcuLCKEWCAQF\nApsNduyAPn301KvnwYwZOhITZU9jexSEMKGNG9fj7x9AnTp1Ux13c3PDZHpqajAYDLi5uWf4uj//\nrOHePSX9+pnw8np+m8+KeRIjRpiSTeTu7nL7deu+TFDQKDQaDRqNhurVa3DlyuUM9y0vIvaoBQJB\nvub+fVizRsOyZVquXAHQULu2lQ8+MPPOO7JZtqAWh4iM3EtU1FkOHNgPQGzsAwYMeA+AGzdu0KAB\nT/5/HX//Chm6pskEP/ygxd1dYsCAtJMde22uXduTMWO+ZcOGN7h1S8LTU2LBgkRq145i06YjLFy4\nPHn/vHr1msnXslgsHDt2hN69P8j0GOQFhFALBIJ8hyTBiRNKli7VsnGjGqNRgV4v8f770K1bPC+/\nbEtlys6pDF45zfTp36d63blzO+bMWUBU1P9YvnwxrVu/hSRJbNq0gYEDh2XommFhGm7dUjJokAkf\nn7QmbUdtlixZihEjEti61cj27TbeecfM0KFpzfJJSJLEd999g69vcf797xYv8K3zHkKoBQJBviE+\nHtav17B0qYa//pKLPgQE2OjTx0i3bmaqVClETIwth3uZ+2nWrDnnzkXxwQc9AAUtWrSiUaMmzz3P\nYoHvv9ei1UoMHWrfnO2IX39dS1jYauLj44iPj6d9+1a4u3ukMcvL7ViYMmUCsbGxhIRMQ6WyX+Aj\nv6CQJMm5FcSdgDMLu+f3QvE5hRhX1yHG9sU5d07JsmUafvlFw+PHClQqiVatLLz/vpkmTazJGcDE\n2LqGpHFdv17N4MFu9O5t4rvvsmah+PTTj4iKOovyyS8vNvYBXl5eTJgwld9/30JCQgLjxk22u9rO\ni/j6FnL4Xv74hgKBoMBhMsG2bWqWLNFw8KD8KCte3MbAgSZ69zZTqlSuW4Pka2w2Oe5ZpZL48MMX\nW03bw5GJ/Pz5KK5cucy8eaH5RqSfR8H4lgKBIE9iz8lr6FATK1ZoWLlSQ0yMvNpq3FhePbdubUFj\nP39JlomI2M3SpaGYTEYKFy7CqFGjKVfOn3nz5nDoUCRGo5FOnbrSo8d7rulALmf7djVnz6ro3NmM\nv7/rJkkbN67n9u2/ee+9d5OP1axZm+DgsS5rM6cRpm9BphDj6jrE2Mo8m9rzKRKgoHBhiW7dzPTp\nY6JSpYw9xjI7trdv36Z//14sWrSCEiVKEha2mp07t9GmTXt27NjGrFk/YjabGTTofT77bAx16rz8\nwm3kZXx8CvGvf1k5dUrF/v3xVK0q/ABelPRM3yKOWiAQ5EocpfZUKGDWrEROn45j4kRjhkU6K6jV\nasaOnUSJEiUBqF+/AdevX+PYsSO0aNEanU6Hp6cnbdq0IyLiD5f3J7cQHKyjbFlPlEo4dUqFv79N\niLQLEEItEAhyJY5Se0oS9OhhwT3j+TeyjI+PDw0avAbIHsdbt26mUaNAFAqw2Z4mTHFzc+fmzejs\n61gOkmTxMBqfxrldvaokOFiXzlmCzCCEWiDIAZJWIn5+npQt6ykebs9gseCwVvOzVZiyk7Cw1bRv\n34rTp08xZMhHNGjwKps3b+Lx48c8fBjL9u1bMRqz7kiVF3BczMRFTgIFGOFMJhBkM47KKgIFMunG\nszx+DP37u2Gz2U+unZOpPbt27U6XLt3YtWs7Q4b0Zdmy1dy8eZOBA/tQrJgPDRq8ytWr+TudZRKO\nLB6Oi5wIMotYUQsEGSQzq2CDAWJiFFy+rOD0aSWRkSqWLRMrEUdERyto29adPXvUNG9uoU8fEzqd\nBEjodBL9+5tyZDJz9eoVjh07AoBCoaBFi9bEx8dz8+YNhg0bwerV65k79ydUKhUBAZWyvX/ZjcGQ\nOy0e+RWxohYIMoCjVfCBAypq1LARFwePHil4/DjpHzx+rMBkynjJJaMR/vtfJYGBrvgGuZ9Tp5T0\n6uVGTIxczGHiRCNqNXz7bc4v0WJjHzBp0lhCQ1fg4+PLmTN/YrFYuHjxAitXLmXs2Mncv/8PW7du\nZubMuTndXZdy7ZqCfv1yp8UjvyLCswSZoqCNa9mynqmcZhzh4SHh5SVRqJBEoUI8+Zn69axZWiwW\nx9cqXhwaNzYTGGghMNBKiRK57k/U6WzZomboUD0GA0yaZLRbzMEZZOW+XbcujA0bwrHZbGg0WgYP\nHkbduv9i4sSvuXDhHCqVigEDhvLmm/k37/TOnSqGDnXj4UMFPXua0Ghg9WoNRqMCnU4qMMVMXEF6\n4VlCqAWZoqCNq5+fJ/brFkucPBmPl5eEhwdkJOWwo/jgwEALvr4SkZEabt9+erxaNSuBgVYCAy28\n/roVD49Mf41chyTBjz9qmDBBh5sbLFiQSKtW9stOOoOCdt86C6sVvvlGy6xZOnQ6ialTjfTs+XQy\nJcY164gUogJBFtHp7DvJ6HRQpsyLzXWTVhyOyir6+GjYty+evXtV7N2r5tAhFQsWqFiwQItGI9Gg\nwVPhrlPHlqHJQW7EYoEvvtCxfLmWEiVsrFyZSO3aIgY3txETo2DwYD3796spX97G4sWJ1Kolfk/Z\niVhRCzJFQRtXR6tgVzg3PTu2RiMcO6Zi714VERFqzpxRIkny6r5IEYnGjS3Jwl2+fK77c7ZLkmf3\nnj1qatSw8vPPiZnOzW0vtWf58hWYO3cWhw8fQKlUUqNGLYKCRlG+fPECdd9mlaNHlQwY4Mbffytp\n1crCnDmJFCmS9nMF7XngCoTpW9xATqcgjmvKvNOgoHJlKwcOJDi9neeN7f37sH+/OnnFHR391P3W\n39+WvLfduLGFwoXt58vOyX3E6GgFvXq5cfasihYtLCxYkIinZ+au5Si1Z7t2Hdi+fSszZ/6AWq1m\nwoSvKFWqNGPGfF7g7tvMIEmwaJGGsWN12GwQHGxi+HCTQ0/vgvg8cDYihahA4ARCQoxER8dx+3Yc\nVatauXxZyZUrGffqdhZFi8Lbb1uYMcPI8ePxHD4cx9SpBv7v/8z884+CZcu09O3rRtWqntSo4ZEi\ne5QCo1HBokXaHEuwcuqUktat3Tl7VkX//iaWLcu8SIPj1J6XL1+kVq06aLValEolL7/8L65cueSk\nb5G/iYuDgQP1jBmjp0gRibVrE/noI8cinVEOHoykUaP6/P33LQD27t1Dt24d6NLlbcaMGUV8fJwT\nep8/EUItELwgSiWMHGnCalUwa1bOBo0qFBAQING3r5llywycOxfH5s3xjBplpH59KzEx9icSORGz\nvWWLmnfeceeffxRMnmwgJEQOv8oKjlJ7/utfDTh8+CCPHj3CaDRy8OB+6td/1QnfIn9z7pySVq3c\n2bhRwyuvWNi9O4FGjbLu3GcwGJg/fw5eXoUBuHXrJjNmTGX69O8JC/sVP78SHDgQmeV28itCqAWC\nTNCunYWqVa2EhaldsqqOiNjN++/3oEePTgwZ0o/Lly8C8ODBfYKChvLuu+/YPU+thldesTFqlInN\nmxMdXj87s0dJEvzwg4a+ffUoFLBsWaLTw6+eTe3ZuHFTKlWqzNtvt6Jt2+bExcXRvn0Hp7aZ31i/\nXk2rVu5cuKBi0CATGzYkUrKkc3ZGFy9eQKtWbXB/kqB9x45tBAb+mzJlyqJQKBgxYiQtW7Z2Slv5\nESHUAkEmUKngk0/kVfXs2bKTmSNxDQtbRc+enenevSNTp07EbE5fpG7dusX06VOYOvU7Vq1aR7Nm\nzZkyZQKPHj1k+PCBVKyY8cxXjrJEZVf2KLMZRo3SMX68nuLFJX77LcEl4Vddu3Zny5ZddO3anSFD\n+hIWtprY2Ads27aHbdv24O9fgdmzv3N6u/kBkwlGj9YxeLAbSiWEhiYycaLRaXW9L126yLFjR3j3\n3Z7Jxy5ePI9GoyEoaCjdunXk229DMBgMzmkwHyKEWiDIJO3bW6hSxcovv2g4fvy2XXH9z3/+Ijx8\nDfPnL2HVqnXExT0mPHxNutd1tO8KCqZMmc4bbzTJcB8dZYnKjuxRjx5Bz55uLF+upWZNK7//nuD0\nsB5HqT2PHz9CkybN0Ov1qNVqmjZ9kz//POnUtvMDN28qePttd0JDtVSrZmXHjnjatbM47fqSJDF9\neghBQZ+hTrHP8fhxHMeOHWHs2EksWfIzN2/eYPnyxU5rN78hhFogyCQq1dO96sWL3e2K6549u/j3\nv1tQqFAhFAoFb73Vnj17dqV7XT8/P7v7rl5eXpQr5/9CfQwJMdK/vwnI3nzZ0dEK2rVzJyJCTYsW\nFjZtSsh0+FV6JKX2vHcvBiA5tWeZMmU5fPggFossOocORRIQUNHp7edl9uxR8eab7pw4oaJzZzPb\ntiU4vbb3xo3r8fcPoE6duqmOe3p60LhxU7y9i+Lm5kaHDp05duywU9vOT4iEJwJBFmjf3sL06VbW\nrSvFpk2lMJlAp7PwxhvbaNQokOjo6zRq9HQFXLp0Ga5fv5qha4eFrWbp0kWULl2GKVMyb7YNCTGy\nfbv8p37iRHymr5NRUubs7t/fxIQJWXcac0TduvV4772+BAUNTU7tOX78ZGrWrMOMGd/Qs2dnFAol\n5cqVY9SoYNd0Io9hs8GMGVq+/VaLRgPffGPg/ffNKFwQwBAZuZeoqLMcOLAfkCdWAwa8h6+vH76+\nfsmfUyqVKJV5NHNPNiCEWiDIAioVlC4tceGCApMJihRZRrFiP3LhQjlKl56FxfI5Wu3TRClarT7D\ne3HPllRcuTIMnU7vqq/iFJJydhuNMHmywWU5u1PSqVNXOnXqmub42LGTXN52XuP+fRg2zI3du9WU\nKWMjNDSRl192XZax6dO/T/W6c+d2zJmzgH/++YfRo0fSo8d7FCvmw+bNG6lf/xWX9SOvI0zfAkEW\nOXTo6UogNrYPly4dJja2DwcO9EahUGAymZLfNxoNuLm5pXu9S5cu2d13lfepcyfPenYvX+58z25B\n1jh1Sknz5h7s3q2mWTMLu3bFu1Sk06NmzVr07TuQoUP707NnZwoXLkKvXu/nSF/yAmJFLRBkEaMR\ntNpLqNV3SEhoCCh4/Lgtfn4TUCgU3LgRnfzZ6Ojr+PsHpHu9+/fv2y2pWKpUaZd9B3tpOAMCKrF0\n6SJ27NiGzSZRpUpVPvtsDJ7PZCgxm2Wv4aSc3T//LHJB5yYkCZYt0/Dll7onXvhGPvnElCM54teu\n/S35/x06dKZDh87Z34k8iBBqgSCL6HSgVN6nRInPuHZtHVZrcfT6EygUFvr06ceECV/RrVtPvLwK\nEx6+hubNW6Z7vQYNGtjddz116iQ//jgbg8HA/fv/0KNHJ3x9/Zg9e16W+n/7tuyxnjIN55QpE+jR\n4z3++GMnixYtR693Y/z4Mfz88zIGDRqWfO6jR3LO7ogINTVrWlm5MvM5uwXOJz4eRo3Ss3atBm9v\niXnzEvn3v11XnUzgGoRQCwRZpHdvM4sWNeCff4ZQpswHKBQ2JElL7drTqFu3Ht2792bo0AGARP36\nr/LOO89fRTjad03pmOYs7IWDhYbOp3z5CgQHj8PdXa6rWbNmHU6ePJZ8njNzdgucz6VLCvr2lX8/\n9epZWbQo8YUrvQlyB0KoBYIskhTqFBrag4cPe6QpfNGlSze6dOmWk11MFx8fH3x8fIDU4WDPhjMd\nPnyQunVfBlJ7dg8YIHt259Vym/mRzZvVfPSRnrg4BR98IP9+sivJjcD5OE2o169fz+zZsylXrhwA\nDRs2ZMiQIURFRTFu3DgAqlatyvjx453VpECQawgJMXLzpoJt2zScOBGHn1/uWbkEB+uIjpZjb8qW\n9XRYPSu9cLBly0J58OAfOnfuliOe3YKMYTbDpEk65s3T4u4um7o7dXJeAhNBzuDUFXWbNm34/PPP\nUx2bPHkywcHB1K5dm5EjR7J3714CAwOd2axAkCuoVEl2oLp0SYmfX+7YB3y2jrbRSPLrZ8XaUThY\n9+7zuHjxENHRSwkI8MZqBXd32bO7Zcvc8T0FcPu2ggED9Bw5oqZSJSuLFxuoVk049eUHXBqeZTKZ\nuHnzJrVr1wagWbNmHDp0yJVNCgQ5Rkqhzi04qpKV8rijNJzXr1+jR49Qzp07zfXrK7DZimK1yuUy\nW7Y0C5HORRw4IGcZO3JETfv2ZnbsSBAinY9w6hPl6NGj9OvXjz59+vC///2PBw8e4OXllfx+sWLF\niImJcWaTAkGuISBANndfvJh7hNpRlayUxx2l4YyLi+PSpd+4dWs+kpTaS2zbtuwvkylIiyTB999r\n6dTJjQcPFEycaGDhQoNw6stnZMr0HR4eTnh4eKpjb731Fh9++CFNmzbl1KlTfP755yxatCjVZyQp\nY/t23t7uqNXO80zx9S3ktGsJniLGNTWvyem5iY7W4uurTfXe7t27+f777zGZTBQpUoTx48cTEBDA\n9OnTiYiIwGg00rNnT/r37w84b2x1Ovtirdcrktto0SKQu3eHMnLkcGw2G1qtllmzZrJ7926UyseU\nK9cl+TyzuTQ3b4ZiNCry7O8/r/b7WWJj4f33YeNGKFUKwsLgjTf0QM5kr8sv45obyZRQd+nShS5d\nujh8/+WXX+b+/ft4e3sTGxubfPzOnTv4+fk5PC+JBw8SMtMtu/j6FiIm5rHTrieQEeOaFkmCIkU8\nOXvWRkzM03s4JuYun332OfPmhVKhQgDr14cTHDyGli3bcPz4SRYtWonZbGbQoPfx969C8+ZNnDK2\nkgSlSrlz5UraSW+vXiZiYp4qeMuW7WnZsn2qz1SvXo9p08ZjNKZNAq3TScTExGW5j9lNfrlv//pL\nSb9+bly9qqRxYwvz5xvw9ZXIKYNlfhnXnCS9iY7TbHQLFy5k8+bNAJw/f56iRYui1WoJCAjg+PHj\nAOzYsYPGjRs7q0mBIFehUEDFijauXlWSsuS0Wq1m3LjJVKggZySrXbsuV65c5tixI7Ro0RqdToen\npydt2rQjIuIPp/Vn4UINV66oKF7chk6XuepZ3brlXJlMgX1Wr1bz1lvuXL2qJCjISFhYIr6+uSfK\nQOB8nOb13a5dO0aNGsWaNWuwWCxMnjwZgODgYL7++mtsNht16tShYcOGzmpSIMh1VKpk48QJFdHR\niuQ9a2/vorz22tP7/vDhA1SvXhOFAmy2pw5Zbm7u3LwZneaameHoUSXjxunw9bWxa1cCxYtn7kGe\nlApUpZKwWkkTIy7IPhITZS/+n3/W4uUlsXBhAq1aCYe+goDThLpEiRKsWLEizfFKlSqxatUqZzUj\nEORqKlaUhe3iRSUBAWkfosePHyUsbDWzZ8/j9OmT/Prrelq1egubzcr27VvR69Mv2JER7t1TMGCA\nGzYb/PSTIdMiLUkQGqpBrZY4eTKeEiXEqi07CQ7WsWKF5kkueShUSOKff5TUqmUlNDQRf3/x+ygo\niMxkAoETSSnUz4Yv7dsXwaxZ3zJt2kwqVAigbNly3Lx5k4ED+1CsmA8NGrzK1auXs9S+1QqDB+v5\n+28lX35p5I03Mr/iOnhQxdmzKt55xyxEOpt5Nv7dZIJ//lFQpYqVzZsTeE4BNkE+Qwi1QOBEkoT6\n2VjqY8eOMHv2dGbMmIu/fwVA3rseNmwEw4aNAGDJkoUEBFTKUvvTp2vZt09Nq1YWhg83Pf+EJ9ir\nnrVwYU2KFfuee/e20K2bjSpVqjJqVDCFCgnvXlfjKP792jWlEOkCiBBqgcCJVKhgQ6GQUgm1wWBg\nypQJhIRMTxZpgB07tnHgwD7Gjp3M/fv/sHXrZmbOnJvptv/4Q8WMGVrKlbMxZ04iygy6itqrnjV+\n/EQOHPiAkiUPsGrVSrRaLV9//QUrVixm6NARaa5hT+h//30LkZH7Uo1DkSLeLF68MtPfMb+TmAi/\n/67OUPy7oOAghFogcCJublC2bGqh3r8/gtjYB0yY8GWqz86YMZeIiD949913UKlUDB48nDJlymaq\n3Rs3FAwZ4oZWC4sXJ1KkSMbPtVc968cfF2AwVObdd0ej18txuS+//K/kDGYpcVQmc+HC5alEffr0\nqfj7+2fq++VnbDY4fFhFWJia337T8Phx2nC4JERhjYKJEGqBwMkEBNiIiFDz+DEUKgQtWrSmRYvW\ndj8bEvJtltszGuWa0A8eKJg+3UDt2i+WOvLZ6lmbNm0mLu5NChWqQv/+8QDExcWxZ89uWrduk+Z8\nR2UyU3L58kX+/PMkQUGfZuYr5ksuXVIQHq4hPFxDdLQ8sStd2ka/fiZu3FCwdq02zTkiLK5gIoQ6\nn5LSY1SE1GQvlSrZiIiQ96nr1nV9vuWxY3WcPKmia1dzlh7kSdWz9Pqy3Lw5j+HDzej1MG7cGPbv\nj6B581a0bt02zXmOymSmZPHihfTs+R5qdcF+5Ny/D7/+KovziRNyIhoPD4lu3cx07WqmYUNr8pZF\nkSKIv2EBAAopo3k9sxFnZrgpiBlznvUYTaJfPxNTpjjnDz29cS3ok4TQUA2jR+szXWLwRe7Z9evV\nDB7sxksvWdm2LQF39xduLhUxF9JWAAAgAElEQVQ2m0SzZhEYDLNYsyaMChVkW6vRaGTevO+5f/8+\nEyZMsXvus2Uyk8T7xo1oPvpoMGFhG3NcqHPieWAywc6dasLD1ezcqcZsVqBUSgQGWuna1Uzr1hY8\nPLK1S06nID5nnU16mckK9vQ2HxIbC0uX2vcYDQ3VsGqVBr0e9HrJ7k+dTsLNjXQ/o9dL+PmByaRG\nr5c/r9PJ782dqyE8PGNlFfMrKUO0XMm5c0o++USPp6fE4sWJmRbpq1evEBNzlwYNXuXoUTVnz7an\nWrVJHDkSjiS9TkBARXQ6He3adWDYsP4Or+OoTObu3Tto0qRpjot0diJJcPKkkrAwDb/+quHBA3nf\n+aWXZHHu1MkiQt4EGabg/OXkY4xG2LVLzdq18ozdYnHsjFKlig2jERITFSQkwP37SoxGMBgcn+OY\njMeJrFihKTBCnVTu8vJl1wl1XBz066cnIUFBaGgiFStm/qGfVD0rNHQFixaVRa8/gUZjITExkblz\nZzJ16gy0Wi0HDuyjYsXKac5PKfRJZTJnzvyW69evUblyVQ4ejOSDDwZk5evmGa5fV7B2rWzaTnIo\n9PW1MXiwbNquWVOUnhS8OEKo8yg2Gxw9qiI8XM2mTRoePpSFtlo1KxcuKJ/UDU6NTgc7d9oveGKz\n8USwZdFO/yeo1W7cu2dIPpaYqMBohCVLNEDatgtSWEmpUhJubpLLVtSSBJ9+quf8eRWDBplo1+7F\nzespqVu3Hu+915dhw4Zy5QqUK6dl4sTJ/OtfDZgzZwZ9+nRDkqB48eJ8/vmXac5PKfQ+Pr7JZTJL\nlSoNwKVLF1KFpeU3Hj+G335TExam4eBB+ZGq10t07GimSxczgYFWCpAxQeACxO2Tx7hwQcnatWrW\nrdNw/bosBMWL2+jRQ34o1KhhY8wY+3vU6TkaKZVyaJGcTCFpdeZ4lebrCzExaa+3apXGgSgrGDxY\nz+jRRsqXz98mP6VSjqe+dEmJJMnFOpzJkiUa1q/XUL++la++cs4MqFOnrpw/34s9e3TMmGGgYUP5\nd/vpp6Ofe26S0AcFDcVms6HRaBk/fjIeHp48evQQg8FA0aLFnNLP3ILFAnv3qggL07BtmzrZItWw\noYWuXc20bWvByyt7+nLwYCSffRZEePgmtm79jfXrwyhc+Gl83qBBwwkMbJY9nRG4BCHUeYA7dxT8\n+quatWs1nD791FP03XfNdO5splEjK6oUlQyTTMw54dDVu7fZ7iShWDEb69dr2LxZTd++Zj7+2Ii3\nt8u7k2NUrGjjf/9Tcfu2gpIlsz4xSemgB6DT2Vi0KBFt2qHOFAaDfL94e8srwRelU6eudOrUNc1x\nL6/CREYed0YXcwX/+Y+877xunZqYGHmiXLGija5dTXTqZKZcueydhBoMBubPn4OXV+HkYx07dqVf\nv0HZ2g+BaxFCnQuw5yU9ZoyRbdtkcY6IUGGzKVCpJJo3t9C5s+wpmp7zUEiIMUf2hB1NEiZNMrJh\ng5qQEB3z52tZvVpDUJCRfv3kEKD8RtI+9cWLSkqWzFqFI3te/EajgrlztVn+HaeeACioVSv9+6og\ncvu2gnXrZNP22bPyjNjbW+KDD0x07WqmXj2b060mGWXx4gW0atWGDRvW5kwHBNmCCM/KYRyFUqnV\nUrJTWL16Vjp3NvP225ZcU3c2s+NqMMDixRpmztTx8KGCsmVtjB5tpGNHS4ZTXuYFwsLUDB/uxrRp\nBt5//8VWqM+ObdmynhiN9nwOJKKj4zLdR0f33ovUq85rZPS+jY+HbdvUhIdr2LtXnihrNBItWljo\n2tVC8+YWp1kzMsulSxeZNOlrFi5cTrduHZgzZwFbt/7G8eNHkSSJhw9jadiwMYMGDUPr4s7m9uds\nXkCEZ+ViHCXft1jgk0+MdOlizpJHb25Dr4ehQ810725m1iwdoaEahg51Y/58K2PHGmncOH/U13VU\nnCMzuCrvs6N7ryB56KfEZpMrhoWFafjtNzXx8fLk6F//stKli5l33jFTtGgOd/IJkiQxfXoIQUGf\npQp7q1q1Gu7uHnTq1BWDIZEvvhjJzz8vKzBe9/kVIdQ5THoP2y++yHj1o8wQGbmXRYsWYDab8PIq\nzKhRo4mKOsvs2dMpVswn+XPy/uO7Tm3b2xvGjzfSr5+JkBAd69dr6NTJnTfftPD110Zeeilvh7E4\nM5Zap7N/n2Q177Mo/CBz4YKSsDB5m+nmTfn3VbasjUGDTLl2orxx43r8/QOoU6duquMpM8JptVre\nfbcHK1cuFUKdxxFCnYNcuaJAoZDDbZ7F1cn3Y2LuMmnSOObNC6VChQDWrw/n229DaNeuA02aNGPM\nmHEubf/ZScKyZWNYuLAGBw78yXvvjcPLK5FKlUowefJEfHx8XdoXV1CkCPj42JyyonbkoJfVvM+u\nmgDkBf75R3bQDAvTcOqUvO/s6SnRs6eJLl0svPaaNVdvxURG7iUq6iwHDuwH5BC5AQPeY+DAYbz5\nZgs8PDwBsFisBSrRTH4lF9+K+ZvDh1X83/+5I0n2vVBcnXxfrVYzbtxkKlQIAKB27bpcuXLZpW0m\nkTRJGDt2Ej//vJYWLVqzadMkli+PoXr1Eej1E/jrr90cOBDI6NG7eJxHt74qVrRx/boiyyvUiRON\naLUScrichE4nOWUf2dE9ll8LPxiNsG4dvPeenlq1PBg9Ws/p00refNPCggWJ/Oc/ccycaUyVbzu3\nMn3692zevJNNm7azadN2/PyKs3Dhck6ePM6CBT8gSRJGo5FNm9bz+uuNcrq7giySy2/H/El4uJrO\nnd14+FCudtS/vwmdzrkP4efh7V2U115rmPz68OEDVK9eE4ALF84zfPhAunXryJQpE4iLy7zDkj0c\nTRIiI/dSu3Y1IiIqM2tWIkplfzZuHMgrr3gQGqrBnMf0o1IlGzabgmvXsvZnduaMEpNJQc+eZu7e\njSM6Os4p90dIiDHFvQdKZfbce9mJJMGxY0pGjdJRq5YnnTvD779rqFbNxvjxBk6fjmf16kQ6dMgf\n3u4jRowkJuYu3bt35IMPelCpUmW6deuV090SZBHh9Z2N2GwwbZqWGTN0eHlJhIYmEhiY885Tx48f\nZcKEr5g9ex5xcY85evQw3bv3QqlUMWnSWNzd3QkOHpvqHGeO68qVSzl58jgBAZV4+DCWuLg4rly5\nTMWKVSlU6EsWLChOfLyCgAAbY8YYadvWkmPhMC/CnDlaJk7UsXRpIm3aZDx72LNjO2uWlpAQHQsX\nJvL221nLQuaISpU8KVXKxr599jPX5TWuXn2ayvPKlaeJgXr3VvLWW/HUqJG3fSByG7npOZtXSc/r\nW6yos4nERBg0SM+MGTrKl7exdWtCrhDpffsiCAkZz7RpM6lQIYBaterQr98g3N090Ov19O79AQcP\nRrqs/ePHjxIWtpoPP/wkeZIwbNgIVq4Mw81Ni8k0laNH4+nb18T16wr69XPjrbfcOXo099+6KWOp\ns8LevSoUConGjV0j0iA7T12/rrTrL7Fhg5rAQHdKlvQkMNCdDRtyx55ncLCOsmU98fPzpGxZT0aO\nlGPC27d345VXPJk2Tcft2wo6dTLzyy8J/PlnPN9+ixBpQZ4j9z/t8gF37yro2NGdjRs1vPKKhW3b\nEqhSJecfFseOHWH27OnMmDGXatWqA3Dnzm0ePHiQ/Bmr1eIyZ5RnJwkeHp7Ur9+AMmXKolar6dKl\nO0ePHsbXV2LqVCP798fz1ltmjh9X0batBx98oOfSpdy7tE7y/L58OfN9jI+Xc7rXrm1zaWhQ2bI2\nEhIU3L+fuq8bNqgZNMiNs2dVWK0Kzp5VMWiQW46LdVIMuBxfrsBoVLBihZaRI/UcOaKiUSML33+f\nyH//G8e8eQaaNUudvU8gyEvkjqlxPubsWSW9erkRHa2kc2czM2cacoVXrcFgYMqUCYSETE9VMOHX\nX9dx9eplJk78BoVCwbp1v7jEGSXlJCGp/bt3S7Nnzy1+/NGTKlVs9OihQaV6OpesWFFiyRIDR4+a\nGDdOz5YtGn7/Xc1775n59FNTrkkGk4S/vw2VKmvFOQ4fVmE2KwgMfP5qOiJiN0uXhmIyGSlcuAij\nRo0mIKBS8vtz584iImI3a9f+lubcpNSX0dEKihV7Oo6zZtlPlDF8uJ5Jk3JuvKOj7U9+1GqJo0fj\n+fFHLaNG6fnoo4JZE12QvxArahfyxx8q3nrLnehoJZ9/buSHHzIv0pGRe3n//R707NmZIUP6cfny\nxVTvz507i86d22X4evv3RxAb+4AJE76kR49Oyf/efrsjnp6F6NWrC716dUGlUjNs2IjMddoBSZOE\nyZO/TRbpDRvUhIb+H1brMVSq85w9q2LOnI34+b2W5vxXXrGxZUsCixcnUr68xJIlWl55xYMZM7TE\nxzu1q1lCq5UFMCvlLiMi5Ln087ZJbt++zfTpU5g69TtWrVpHs2bNmTJlQvL7Fy6cZ//+CIfnlykj\nr/6jo1P39fx5JZ6e2ylX7m38/VtTtmx3tNrzmM2gVJ5Ap2uHTtcSrfZ94G6mvqMzsVjgxx+1aVbb\nixZpCQ7OBTNkgSATiBW1E0mZN1mtlh8aOh389FMi77yT+f1FRzHP8+YtBp7/ELZHixatadGitd33\nXB1DnXKSkIS8P7qSO3emUKrUcECByVSZqCj7fVEooG1bC61aWVi+XMN332mZOlXH0qUaPv/cRLdu\n5lxh6qxY0cauXWpiY+XY6hdl714Vbm4Sr7ySvlCr1WrGjp1EiRIlAahfvwGhofMBsNlsfPfdVAYM\nGML8+XPtnl+27NMVdUoqV76ByTSW69fXYbGUpkiRZZQoEYyHRyhFigQxefK31KxZi5Url6JSbaB7\n9+zxMJbTqqY9rlYnlVpNS2iohrt3wcNDj7e3lOZfkSISRYvK/3fLeKl1gcDlCKF2Es/mTbY80eWW\nLU1ZEmmwH870008/ABl7COc27E0SSpb0xGpVEBfXkri4linekejUyUrTplaaNrVQo4YtVYyrRgP9\n+pnp2tXM3Lla5s/X8vHHehYs0PD110befNOaox7islDLDmX167+YX8Lt2wqiolQ0a2Z5riXGx8cH\nHx85m5zFYmHr1s3JWao2blxPQEBFatSo5fD8cuXsr6gHDJD46qvvsFjk2tIJCa9TrNj3tGy5g4cP\nq1GzpnzNXr3ef6HvllkeP4Zz55Q0aGAlMjLt4yspP749JAk2bQKwL+Qp0evTindaQSfVMW9vKcfz\nfwvyJ0KonYSjvMk7dmiArO2NpRfznJGHcF6gShVbcmWilOj1sH+/mv371UycqMPX10ZgoCzagYFW\niheXV4KFCsHo0Sbef9/MtGlyda4ePdxp1MjC2LFG6tTJGee9lDm/X1So9+6Vx6Np04xP9MLCVrN0\n6SJKly7DlCnf8c8/9wgLW8WCBUuJj3ccD3/6tCzQixdrOHhQRVCQiQ4dLPTu7Y2npy+zZ1s5f95G\nhQrreOmlphQuHAUUYfToT7ly5TJVq1bl448/p0hmzAZ2iIuTze7nzimJilJx7pz8/6QUn6mRUCqh\nenUb/fubGDVKj9lsr4gJ/P03nD8fR2ysggcPZOe52NinPx88ePrv/n0FN28qOXs24zM9T8+0gp5S\n7O0dK1yYLFl/7FXfE/vx+Qsh1E4iu/ImJ4UzzZ49L8MP4bxAUJCJQYPS2htnzzbQqJGVvXtVRESo\niYhQsXathrVr5YlR9epWmjWThfvVV62ULCkxc6aRAQPMTJqkY9cuNS1aqOnY0UxwsDHb6wUnhWhl\nJpXo3r0Z259OSdeu3enSpRu7dm1nyJC+VKpUhQ8+GICXl5fDe2TDBjWffCKPvSQ99ewGORFIhw4W\nzOYVyROAkJDvWLRoHkePHuaHHxZSokRJpk6dyPfff8fXX098oe+YkCDn2o6KUj4RY1mUr19PO14l\nStgIDLRQrZqNqlVtVK1qpWpVG15eqT/3n/84Trnq7a0lICApy1vGsFhIIeKkEnNH/y5eVJKQkDGB\nVyhksU4p4t7eqQXd3rFChWDMmNSWPKOR5NdCrPMPIuGJE7BYoEwZT2w255ciTMm+fRHMmvUtISHf\nUq1adcaODeaNNxrTsuX/8ffft/jww0F2PXpdgSvGdcMGNbNnazl/XkmVKjZGjJBXdSmRJNmTPiJC\nxZ49ag4fViWXgNTrJV5/XRbtZs3kh3hkpIrx43WcOaNCq5Xo189MUJARb2+ndt0ht28rqF3bk3bt\nzISGGjJ0jq9vIe7efUzNmh4oFPDXX/HPNd9fvXqFmJi7NGjwavKxNm3exGw24/Zkw9Vms/Lo0SOK\nFPFm7drfkksfBga627VmVK9uJSLiaQIUSZLYtWs7P/30I02aNCM29j5ffSUL87lzUYwc+SGbN++0\n27/ERNn8n1KQo6KUXL+uSJNG19fXlizGKUX5RRbrjlaZ2ZmYw2gk1Wr9/v2UYk6alXzS+yZTxgRe\npZKwWgFc+9zJCCLhSdYRZS5diNUKH36otyvS4Ly8yfbCmQ4ejOTUqRPMnTsr+SHcvn2rVA/hvETS\n6i09FArZxFm9uo2hQ80kJsohTEmr7T171OzZo2bsWHkF1rSplcGDTcTHK/j+ey3z5mlZtUpDUJCR\nfv3M6PWu/U7Fi0t4eLx4iNb//qckJkYO6cvIHnts7AMmTRpLaOgKfHx8OXPmTywWC7/+ujW5QIOj\nydz58/b7dv68MtUEQKFQ0KJFa2bO/BZPT09u3Lie/FmlUolKpcRolAU5yVQtC7OKq1cVaf5GfHxs\nNGxofSLESaJsdUq8eEiIMcdXlDqd/PtP2p7JCJIkWxnsiXhaE70cY28PoxHWrlXTsqUljcVBkPcQ\nQp0FbDb45BM969ZpqF/fSo0aVtascf5ekaOY55079yX/P7tX1LkFNzdo1kw2f4O8go2IkIV7714V\na9ZoWLNGg0IhUbOmDX9/CydPqhg/Xs/ixVpGjzbSsaPFZUUYFAooVkwiKkpJyZJyfHjS/m96JO1P\nZyR+GqBu3Xq8915fgoKGYrPZ0Gi0jB8/OVmk08ORf0CVKjaHE4DmzduxatXPLFx4hX/+qcr+/ZuI\njW1I9erHKVlyMJcv78ZiKY6f32Q8PQ9TpYoNX99XaN36c6pXV1G1qg0fn+wz5h08GMlnnwWxe/du\ndLrCyce//PIzYmNjmTv3p2zrS3ooFODhAR4eEqVLP398HHm/g4KhQ93QaCSaNLHStq2F1q0tqWLk\nBXkHYfrOJJIEn32mY9kyLS+/bCU8PMFlM9edO39nypQJyaE3Scyd+xNFixYDsl+o84Kpy2aD//xH\nmbzaPnJElexkpFbLZkNJUlC1qpXJk400aeL8lK5Jmb2eZcGCRIdi7etbiH//28KePWrOnImjRAnX\n/omm18e2bS0sWhTO77+HYzTaMJu1GI0fc/lyM/T6nfj4fEtSKF1CQjAlSgxCqbzL22+v5vHjrTx6\ndJapU0OwWi189NFgWrVqQ8eOXVz6fZ7FYDAwcGAf7t27x4YN65OF+uDBSGbM+IYSJUrmGqF+UZ6N\nNkmiUycTlStLbNmi5q+/5EmYUinRsKGVt96y8NZbFqfeV3nheZDbSc/0LYQ6E0gSfPmljoULtdSs\naWX9+oRMxcjmZXLqD9NisTBv3hx++eVn1q/fgp9f8QyfGxcHhw49NZNfuJB6FVmqlI2yZW2cOqXC\nZHKOVSSj+78pKVSoEEWLSvj729i7N3uKZKxbp+a777RcuaKkaFGJ8uVtxMfLTlHPelAXKiQ9MVWn\nNluvWzeLIkWKsGHDWubMWcC9ezF4exelTJmygJyUx2KxEBT0abZ8pyR+/HE2hQvL/fr555XodIUx\nGAz079+bHj3eY82alVy+fInw8E2ULFkqW/vmDJ7n9X31qoItW9Rs3qzhxImn92L9+lbatjXz1lsW\nypfPmgwIoc46Yo/aiUgSTJggi/RLL1kJD08scCKdk3zxxSe89FKNTJ3r6QktWlhp0UJeOd+4oWDv\nXjUbNqg4eFDNrVtKbt16agNP8qC1WGDatMyJdXr7v444eBASExUuWeFbrXDtmiLZuzrJueviRWWy\nU15MjIKYGCUeHhK1aj115kpy7CpVSkqzb37p0kWOHz/CwoXL2bBhLQC1atVJfv/evXscPnyQESM+\ncfp3So9Lly5y7FjqfgEsXvwTrVq1oWjRYty6dRMvr8LpXCV387z9eH9/iWHDzAwbZubWLQXbtqnZ\nvFnNoUMqjh/XM24c1K4tr7TbtrVQuXLO1yEQpEYI9QvyzTdafvhBS6VKski/6J5PZOReFi1agNls\nwsurcHI+5tDQBezevQObTaJKlaqMGhVMoUKOZ1gFlfff70/NmrVZsmRhlq9VpoxEz55mevY0Y7FA\nuXKedhNmLF0q73P7+kpUrGijYkUbJUpIlCiR9FOiZEkbhQqRRsDS2/91xM4njtMvEj/9LDabnGUs\nyZkrSZAvXFBiMKTupLu7xEsvpRXk0qWlDO3dS5LE9OkhBAV9ZreAy7BhAzh79n9069aT+vVftXMF\n1+CoX5cuXeTo0cMsWrSciRO/omjRYthsBUOcSpWSIx/69TMTE6Pg99/VbNmiZv9+FWfO6JgyRUfV\nqk/N4zVr2vJESdn8jhDqF2DGDLmWtL+/jXXrEvHzezGRdpQKtGPHrhw7doQlS35Go9Hy9ddfsGLF\nYoYOdW6O7fxAzZq1XXLdpJSvjjAYFERHK4iOVhIR4ehT8v2gUEBAgI2WLa3UrGm1K9QjRpjSHNuw\nQc2sWVrOnpVja2Ninj4hHZk3JUm2DKT0sD53Tsn582njePV6icqVU3tYV6tmo2zZjAmyIzZuXI+/\nfwB16tS1+/4PPywkPj6OkJDxzJs3h6FDP8p8Y1nslyRJfPfdVD7+eBTXrl0lKuosvr5+3LlzO1v6\nlJvw9ZXo3dtM795mHj6E7dvllXZEhJoZM3TJJXnbtrXQtq2Zl1+2uczpUpA+Yo86g8ydq2HCBD3l\nytn49dcEypR58WF78OA+585FJWcZu3jxAsOHD2DOHNmRpXLlKgCsW/cLx44dYerUGVnut6vI6T2p\nRo3qv/Ae9fOQPWjtx6Tu3x/PH3+o2blTxaFD6mQRVCgk9HrZVJ1R3N3lPeDixZ+uxm/fVrB6dVqn\noAULEjl2TGXXYcjHx0ZiooL4+NRt63QSlSqljUMuX15ySf7zTz/9iKiosyifPMVjYx/g5eXF559/\nSeXK1ShRogQAkZH7WLRoPkuXrnJ+JzLYL5vNhpubO25ueh49eoRWq8VkkidNq1evz5N71M4mLg7+\n+EMW7Z071cn3V8mStmTz+Kuvpi4bmtPPg/yA2KPOIj/9JIt0qVI21q3LnEiD41SgSQINEBcXx549\nu2nduk2W+y14MXr3dpzRyt9fom9fM337ymbyEydUyWFgJ044WmZIFCsme5dbrYonP8FkgqiojKWm\nHD5cj9lBKP69e4oUJmv530svyYLsihLi9spoRkWd5a+/TlOsmE/y5ywWMwsXLmfJkoXs37+XL774\nCoVCwaFDkVSsWCmdFpzL9Onfp3rduXO7ZGeyX39dx7lzZ2nRojWLF//E7dt/Z1u/spu//75Ft24d\nKF26TPKxl16qwVdfTbD7eU9PaN/eQvv2FgwGOVRw82YN27erWbRIrkzm42Pj//5PNo83auR8XwpB\naoRQP4elSzV8+aWe4sVtrF+fkGXvyCRSpgJNYty4MezfH0Hz5q1o3bqtU9oRZJwkh5zn5U1Wq+HV\nV628+qqVzz834efnOFbZ3T3pf2nvG5tNwmaTTe6ymTutcDsS6SSyyys8qYzmokUrKFGiJGFhq5ky\nZQIdOnShSZNmqSquJZVbHT48iO+++4aePTsjSRIVKgQwalRwtvT3eURG7iUq6iwREX+QmJiAzWZj\nwID3mDBhKvXq1c/p7jkdX18/Vq1a98Ln6fXQqpWVVq2smM0QGaliyxY1W7eqWbFCy4oVWgoXlmjf\nHpo3V9O0qQU3N5F/3NkI03c6rFqlJijIDR8fG7/+mpiuA9CL8Gwq0JQYjUbmzfue+/fvM2HCFKe0\n5wpy2tTlCtN3ZknPZJ7RNI7pXQPI8vWzyr1797hy5SINGsj1wS9fvsiQIf0YMeJTTp064fLSqM7C\n0X3buXM75sxZkC9N367IsWC1ylnRNm+WndGSoiXc3SWKF7dx5UraPZb+/U1CrNMhPdO3cA1wQHi4\nmo8/1lO0qI21a50n0ilTgSaJ9IkTx7h8+RIAOp2Odu06cPToIae0l5+4f/8fevToRI8enQD48MNB\n9OjRiZiYuznaL0dpYl8kfayjlbPZ7JzrZxUfH59kkX62jOaFC+cZPnwg3bp1ZMqUCcTF5e0CMfmR\n+Ph4Ro8eSY8enfjkkw+5evVKlq6nUsHrr8uJgk6ejOfwYRg+3Iifn2RXpMFxhUHB8xErajts2qRm\n4EA9hQrB+vUJ1KrlHJE2GAz06NGJkJDpVKv2UvLxJUsW8tdfp5k6dQZarZblyxdz5Mghfvgh6yFI\nriKnV9S5jayY+iIj9/LFFz9hNJqx2Ypw5854TKaK+PpOxdt7H+XLg8FQh2PHvsZg8MhRU+KzZTT/\n/vsmR48epnv3XiiVKiZNGou7uzvBwWOzvW8ZoSDetw8fxhIauoDu3XtTvHgJfvllFRs3rmflyjC7\n4XSZIWlcJQmKF/fE3jYOSNy9KyZxjhCZyV7getu2qenXT49eD2vXJlCvnvPiKx2lAp0xYy4rVy7l\nxIljT2704owc+QXlypV3WtvOpiA+8JKwFwv/++9biIx8mnvdYDBQpIg3ixevTPdaMTF36d37XTp3\nXkpwcE0KF/4ZL6/fePSoI4UKbWT06B/p1AkmTPiKUqVKM3DgUFd/veeSsorWypVh6HRPK5s8r4pW\nTlOQ79skJEmideumzJ+/hAoVApxyzZTj6mgbR6mUOHkynlKlcp3k5AqE13cG2bVLRf/+erRaWLPG\nuSIN0KJFa1q0aG33vV2shCcAACAASURBVE8/He3UtgSuwVEs/Lx5qePep0+fir+//3Ovp1arGTdu\nMq+9Vp5ixRKZNetlTKaZlCgRRYMGtenSRd6devnlf3H48AHXfKkM4KiKVlTUWcqV88f7Sd1Qq9Xi\ntFWawDk8evSIuLjHlCpVOvmYzWZz2e/JUfSEzabgjTc8+OILuXKduE0yjtijfkJEhIoPPnBDrYZV\nqxJ55ZWCkalI8GIkCWvSSqR27bpcuXI51WcuX77In3+e5J13Oj/3eilD9jp0sDBgwG4CA+sQEvIy\n9+9H8ujRI4xGIwcP7s/WrF7PklRF6969GIDkKlr79u1h2rRJWCwWrFYr69b9wuuvN8qxfgrSEhX1\nP0aMGMKDBw8A2LRpA8WLl0gl3M4kJMRI//6mJ46QEjqdRN++JmbONKDVwldf6WnZ0j2dsEbBs2R6\nTnP06FFGjBhBSEgIzZo1AyAqKopx48YBULVqVcaPHw/AokWL+P3331EoFAwfPpzAwMCs9zyLpNxT\n1GjklItqNSxfnkjDhiIuUGAfR7HwKVm8eCE9e76XasXiKHVsSpJC9lasWE7hwsXZu3cPb7/dCrVa\nTZUq1WjfvkO6fbMX5/xsG5nFURnNl1+uz3ffTaVXry4oFApq1arDsGEio15u4pVXXqNDh84MGdIP\npVKBr68fkyZNQ+WK7DdPcJR/vFUrCxMm6FizRkObNu706WNmzBgjhfNuqvVsIVN71NevX2fKlCko\nlUo6d+6cLNS9e/dm1KhR1K5dm5EjR9K+fXsCAgIYMWIEa9asIS4ujh49erBly5Z0bxJX71E7Kg3X\nsqWZlSsNTms7PyP2+mRhnTDhK2bPnpe8wr5xI5qPPhpMWNjGZKFO2odOaS7fuXMb8+YtTr5WypC9\nxo1f5ccfF3LkyEEmTZqGWq1m5sxpKBRKPv30C7t9uX37Nv3790oV57xz5zYWLlzu+oHIQ4j71jW8\n6LgePKjis890nD+vwsfHxsSJcl34gpxX3OnhWb6+vsydOzdV0QiTycTNmzepXVvOxdysWTMOHTrE\nkSNHaNy4MVqtlqJFi1K6dGkuXryYmWadhqMwgb17xaaJIGPs2xdBSMh4pk2bmcohZ/fuHTRp0jTV\navp55nJ7IXvHjh2mSZNm6PV61Go1TZu+yZ9/nnTYH7Vazdixk5IdFevXb8D169ec+p0FAmfRsKGV\nP/5I4MsvjcTHKxgyxI3Ond24dKkAK3U6ZEqZ3NzSFpl/8EDO75tEsWLFiImJoUiRIhQtWjT5eNGi\nRYmJiaFq1aoOr+/t7Y5a7TyzzLMzFaODqBajUZHurEaQmoI6VgcPHmTu3BksXbqEihUrpnrv2LFD\nDBs2LNXY+PoWokqVpx78GzYcp27duvj6FiIxMZFvvpnIjz/+QM2aT03oVatW5tSpo/Tp0wO1Ws2f\nfx7lpZeqOhxzX99CvPRSBUCOc46I2EHz5s0L7O8oPcSYuIYXGdc7d+7w5ZdfcO3aNRo39sBs/po9\nexoQGOjJ6NHwxRdyVjSBzHOFOjw8nPDw8FTHPvzwQxo3bpzueY4s6hmxtD944Ly0iPZMMjqd4yxQ\nMTEizi8jFFQTosFg4PPPvyAkZDpeXn5pxiAqKgpv7xIOx+b48aMsWbKU2bPnERPzmJ07f+f+/ft8\n/PHTOs0qlZLZs+czZ85MWrZshUKhpFy5cowaFfzcMX82zrkg/o7So6Det67mRcf1k08+5bXXGjJt\n2vecPHmcDRuW0rt3TcaM0TF+vJLly218842Bpk0Ljr9QlsKzunTpQpcuXZ7bSNGiRYmNjU1+fefO\nHfz8/PDz8+PKlStpjuck6RVfEAjSY//+CGJjHzBhwpepjs+d+xNqtRqDwUDRosXsnpu0D53SXG4v\nZC/poTd27KQX7l/Xrt3p0qUbu3ZtZ8iQvmninAWCnObOnducOxeVXDSlXr36T/KrW2ja1MI33+hY\nuFBD167udOxoZvx4I8WLF+zYa6f5x2s0GgICAjh+/DgAO3bsoHHjxrz22mtERERgMpm4c+cOd+/e\npVKl7KugY4/Bg02AhELxNHxA5KEVZIQWLVrzxx8HWbVqXap/RYsWw8urMJGRx9Fq004C7e1DO5Or\nV69w7NgRgOQ45/j4eLFPLch1XLx4gZIlSzFv3hy6d+/I8OEDOX8+CpArd02caGTnzgTq1bOyfr2G\nhg09CA3VYC04i+s0ZEqoIyIi6N27N/v372fGjBn07dsXgP9v777jo6rSx49/7tRMSBCCCQjSQl9D\nNQgqseDC2hZRINISwhKChViwEIMrrC5gxQUivRrqBpBVQRSlI92fiLiWfANLRE0hoaRNMjP398eQ\nQGACIUzLzPN+vfIic2cy95nDJM+cc895TnJyMtOmTWPw4ME0a9aMO+64g8aNGxMdHc3w4cN55pln\nmDRpUsX+sJ6yaJEBUJgxo4Ts7AIyMwskSQuXKSkpYerU15k8+R1atGjpknNUtc7ZVWtlhaipgoJz\nZGSk06VLV1auXEffvg8wYcLLWCyWisd07Ghjw4Yi3n67BEWBV14J4IEHAvnuO/9ce+13JUQLC6FL\nlyAMBns5O6PRaafyK3Ktr/qqKh2bkjLP4TB5Tdt27dp/89FHaRXrnJ944mkpPnIJed+6xrW0665d\nO5g27S3WrdsAXL2kaXa2wsSJRtau1aPRqIwaVUZSkplgH5sTKCVEL5KWpufMGYUXXyyVJC3c4kql\nY51pwIBoBgyIdvl5hLgejRrdRFFRITabDY1Gg6IoKIoGrdZxbzksTGX27BKGDClj/PgA5s838Mkn\nOiZPNvPww/6x9tqvxhFUFRYs0KPXq4wYIRPHhBDC3Vq1as2NN4byySfrAdiy5UuCg+vSuPHNV/y5\nu+6ysm1bIePHm8nPVxg1ysSQISaOH/f9TO1XPept27T8/LOWgQPLqjWLsKqyj0uWLOCLLz7DZlNp\n27YdL788gaCgIDe8AiGEqN0UReGNN95iypRJLFu2lPr16/PGG29Wa5MQoxFeeKGURx+19663bNFx\n1111eP75Un7/XWHlypptNevt/Ooa9bBhJjZv1vH554V07XrlTTeqKvsYHT2UxYvnM2fOIgICTPzj\nHxNo3Phmxox52mkx1wZyrc91pG1dR9rWNTzRrqoK69fr+PvfjWRnOx4crk2reZxeQrQ2yshQ2LxZ\nR2Sk9apJGqou+9i8eUuSkycRGFgHjUZDRERnjh/PuMqzCSGEcCZFse84t3t3IVqt4/5mVeWiaxu/\nGfouL3CSkFBarcdXtUtSeHjlkpF7935Nly5dnReoEEKIarvhBqpcY11Vuejaxi961OfOwcqVem66\nycZDD1mu/gOXKN9+MDFxXKXjS5cuJD//FAMHDnZWqEIIIa5RVSt4fGVlj08n6uRkIyYTtGoVRGGh\nQuPGNvTXOBJS1S5Jc+aksH37VqZN+8DhJiVCCCHco6ryz717X3vHzBv5bKIu33O6pATAPn3/0CEd\nycnV/4hVVdnHhQvncuTIYVJS5lKvXj0nRy6EEOJaTJliJj6+FKPRXhZar1fRaFR279bx00+1P83V\n/ldQhaomEVR3ckFVZR9//PG/bNq0kbfeep/AwDpOiVUIIcT1mTLFTGZmAdnZBZw8WcAHH5Rw9qzC\nsGEmsrNr91prn51MVvWe09X7+ap2SYqI6ERBwTkSEkZUHGvU6CamTUupaahCCCGcbMAAC8eOmXn7\nbSMjRphYt66I2nqV0mcTtdHoOClXd3KBu8o+CiGEcI0XXijl2DENaWl6xo4NYP78Ejy8J1SN1MKQ\nq6eqyQWy57QQQvgHRYFp00q4/XYLn3yiZ/Lky7egrQ18tkddXo3Gvn5a9bmSckIIIa7OaIQlS4p5\n8MFdfPLJdPbtU6lb114g5cSJ//HFF9u9fr6RzyZqsCfrzZsN2Gwqhw4VejocIYQQHlC/PixfHsWD\nD/6FzEyFlSuLsVo3sWXLF16fpMGHh76FEEKIcuHhKkuWlKDVwqhRGmbNms1TTz3r6bCqRRK1EEII\nv9Czp5UZM0rQaNZy8uSt6PVNPR1StUiiFkII4Tf69y+lZcuF/PrrKGJjTRQVeTqiq/P7RL1r13bi\n4oYybNhAnnxyFBkZ6QDk5+fx3HNP8fjj/T0coRBCCGf5/vvvaNTIRP/+LfjmGy1jxwZgu/qGih7l\n14k6Jyebf/5zEhMn/pPly9fQp8/9vPPOFM6ePcPYsQm0atXa0yEKIYRwoq+/3sXtt9/JtGkl3Hmn\nhU8/1fPPf3r3si2/TtRV7TkNClOnvsudd97l2QCFEEI4VXr6z7Ro0RKDARYtKqZVKxspKUYefdRE\n06ZBhIUF0bRp0DXtC+Fqfp2oq9pzum7dujRr1sJzgQkhhHCJnJxsQkIaAPZlWytWFGE02ti9W4fZ\nrAAKZrPCggUGr0nWfp2oL1bVntNCCCF8x9Klq+jR4/aK2y1bqthsjjftqO4mTq7m0wVPqmvHjm38\n61/vXLbntBBCCN9XVkVl6epu4uRqfp+oL95z+uLtLIUQQviH693EydX8eui7qj2nhRBC+A9v38TJ\nr3vUVe05HRMzktTUxZSUlJCXd4qhQwcQGhrG9OmzPRSpEEIIVynfrOnDD/WUlipoNCp/+5v3bOKk\nqKqqejqIS+XknHPac3XvHozNZpNNOZwsNDTYqf9P4gJpW9eRtnUNX2rXhx4K5OBBDYcPF9KokfvS\nY2hocJX3+fXQtxBCCHGxQYPKUFWFtWu9Z8BZErUQQghxXr9+Zej1KmvWeMfSLJBELYQQQlQICYH7\n7rNw9KiWH37wjhTpHVEIIYQQXmLQIAtApeHvDRs+ZvjwQQwbNpDnnnuKEyf+57Z4JFELIYQQF+nT\nx0JwsMratXpsNvjf/44za9Z03n//A5YvX8M99/Rm6tTX3RaPJGohhBDiIgEB9mvVv/2m4euvtRw/\nnsHNNzcjNDQMgG7dunPs2P+5LR5J1EIIIcQlBg60D3+vWaPjlls6cvLkr2RkpKOqKtu3byEysofb\nYvGe+edCCCGEl7j9ditNmtj45BM9U6eGMmbM04wcOQyTKRCTyURKyjy3xSI9aiGEEOISGg0MGFDG\nuXMKy5al8+GHi1i9+j9s2rSVJ54Yy/jx43BXvTBJ1EIIIYQD5cPfn3xygIiITjRq1AiA++7ry/Hj\nGZw+fdotcUiiFkIIIRxo395GRISVH35oxeHD33HmjD0x79mziwYNGlCvXj23xCHXqIUQQogqDBxY\nxqRJ99G48XeMGfM3FAXq1Ani9dffQlEUt8QgiVoIIYSowmOPWXj9dZWMjGfYuDHeIzHI0LcQQghR\nhUaNVKKirBw8qOXYMff0oC8liVoIIYS4goEDywBYu9YzG3VIohZCCCGu4KGHLJhMKmlpety0IquS\nGifq/fv3c/vtt7N169aKYzExMQwYMICYmBhiYmL4/vvvAViwYAEDBw5k0KBBbN++/fqjFkIIIdwk\nKAgeeMDCsWMavvnG/f3bGk0mO3HiBIsXL6Zbt26X3Td16lTatm1bcTszM5ONGzeyatUqCgoKGDp0\nKL169UKr1dY8aiGEEMKNBg4sY906PWvW6Ln1VrNbz12jjwahoaGkpKQQHBx81cfu27ePqKgoDAYD\nISEhNGnShPT09JqcVgghhPCIu++2cuONNtav11FW5t5z1yhRm0ymKnvEM2bMYNiwYbz22muUlJSQ\nm5tLSEhIxf0hISHk5OTULFohhBDCA/R6ePRRC6dOadi2zb0jwlcd+k5LSyMtLa3SscTERKKioi57\nbGxsLO3ataNZs2ZMnDiR5cuXX/aY6tRGrV8/EJ3OeQ2h0WgIDb16719cG2lT15G2dR1pW9fwh3Yd\nPRrmz4ePPw5k6FD3nfeqiXrQoEEMGjSoWk/Wp0+fiu979+7Nxo0b6dGjB8eOHas4npWVRVhY2BWf\nJz+/qFrnq55gbDYbOTmFTnxOERoaTE7OOU+H4ZOkbV1H2tY1/KVdmzeHunXrsHq1wurVYDRCTEwZ\nU6Zc/zXrK33Qcdr0NVVViYuL4+zZs4D92nSbNm3o2bMn27Zto7S0lKysLLKzs2ndurWzTiuEEEK4\nxYQJRs6e1QAKoGA2KyxYYCA52ejS89Zo1ve2bdtYuHAhGRkZHD16lNTUVBYtWkR0dDRxcXGYTCYa\nNmxIYmIiJpOJ6Ohohg8fjqIoTJo0CY1Glm8LIYSoXVJTHRc8SU3VO6VXXRVFddeGmtfAmUMo3bvb\nh74PHZKhb2fyl6EuT5C2dR1pW9fwl3YNCwvC3pu+lEp2dsF1Pbdbhr6FEEIIX2asYoS7quPOIola\nCCGEqIaYGMcLqKs67iySqIUQQohqmDLFTK9elvO3VIxGlfj4UpdenwbZj1oIIYSotg4dbOzaBZs2\nFdGtm80t55QetRBCCFFNP/5oT5tt27onSYMkaiGEEKLafvxRQ7NmNoKC3HdOSdRCCCFENeTmKuTk\naGjf3n29afDxRJ2cbOR//4PMTIWmTYNcXj1GCCGE7/rpJ3vKbNfOesXHffbZpwwfHs1jjz3EG2/8\nndLS0us6r88m6uRkIwsWGM7fcl+pNyGEEL6p/Pr0lXrUGRnppKS8z3vvzWDt2k+xWm2sWPHhdZ3X\nZxP1lUq9CSGEENeqOon60KGDdOvWnYYNG6EoCtHRQ9i2bct1nddnE7W5imVtVR0XQgghruTHHzVo\nNCpt2lSdqBUFbLYLQ+MmUyAnT2Ze13l9NlF7qtSbEEII36Oq8NNPWlq2VAkIqPpxt956GwcO7Ccj\nIx2LxcK6dWlyjboqnir1JoQQwvdkZSmcPq3Qvv2VJ5K1bBnO88+/xMSJySQkxNGiRUuCrnMtl89W\nJisv6bZsmYGSEhVQaNXK6vJSb0IIIXzPf/9bPuP76kuzHnjgYR544GEAvv32G8LDW1/XuX22Rw32\nZF1cDFlZBbRubSUzU8OpU462KBNCCCGqVr40q0OHKyfqX3/NJC5uKOfOncNisfDhh4t58MG/Xte5\nfTpRl1MUGDGijNJShdWrfXYQQQghhItUZ8Y3wM03NyUq6m7i4oYwePCjtG3brqJ3XVOKqqrqdT2D\nCzhzA/LyDc3z86Fz5yAaN1b5+utCNH7xEcV1/GWjeE+QtnUdaVvX8Id2vf/+QI4c0XD8eAF6F6zy\nDQ0NrvI+v0lX9etDv34WMjI07NqlrTi+a9d24uKGMmzYQJ58chQZGekA/PvfKxg2bCBDhjzGm2++\nQVmZTEITQgh/9MorRr75RkNZGYSHu7/Kpd8kaoARI+xT5JcutX8cysnJ5p//nMTEif9k+fI19Olz\nP++8M4Xvvz9CWtoq5sxZzIoVaykoOEda2irPBS6EEMIjkpONLFxoABQ8VeXSrxJ1ZKSNP/3Jymef\n6cjKUtDpdEyaNJmWLcMB6NSpC8eOZbB165f07t2H4OBgFEXhoYf6sXXrlx6OXgghhLt5Q5VLv0rU\n5ZPKLBaFlSv11K8fQs+ed1Tcv3fvbv70pwgyM0/QpMnNFcebNLmZEyeOeyBiIYQQnuQNVS79KlED\nDBxYRmCgSmqqHutF69YPHtzPv/+9ksTEcZjNJRgMhor7DIYASkpKPBCtEEIIT/n446pXCbmzyqXf\nJergYBgwoIzMTA3bttknle3YsY0pU/7B22+/T8uW4QQEBFQq+WY2l2AymTwVshBCCDfbu1fL008H\noKsiV7uzyqXfJWqA2Fh7Ay9dqufAgX1Mn/4u06al0L79nwBo3rwFv/56oYh6ZuYJWrQI90isQggh\n3OvnnzXExpqwWmH58mLi40sxGlVAxWhUiY8vdWuVS7+s/tG5s40uXaxs3lzGmTOv89Zb79KiRcuK\n+3v37kNy8ksMHjyMunVvIC1tFX/+c18PRiyEEMIdsrIUhgwxcfq0wowZxdx7r5V77/Vs+Wm/TNRg\nn1T22mtbyM/P5/XXX610X0rKPIYMieGpp0YDKpGRPejff6BnAhVCCOEWBQUwZIiJzEwNSUlmBg+2\neDokwI8qk12qsBA6dQqiTh2Vb74prPI6hHDMHyoReYq0retI27qGL7RrWRkMG2Zi2zYdMTGlvPuu\nGcWNW0NIZTIH6tSB6Ogy/vhDwxdfSJYWQgh/paowblwA27bp6NPHwltvXXuStlgszJz5Pr16RZKd\nnVVx3BlVLv02UUPlSWVCCCH801tvGVi9Wk/XrlbmzSuu0QhrUtI4AgMDKx1zVpVLv07UHTrYuO02\nC1u36jh+XLa/FEIIf5OaqmfaNCPNm9tYtqyYOnVq9jxxcfGMGjWm0jFnVbn060QN9kllAMuWSa9a\nCCH8yebNWl5+2UhIiI3Vq4sIDa35lK2IiE6XHXNWlUu/T9R//auF+vVVVqzQc1GNEyGEED7s//0/\nDaNHmzAYYNmyYsLDnT+v2llVLv0+UQcEwOOPl5Gbq2HjRplUJoQQvu7YMYVhw0yUlMDcucVERtpc\nch5nVbn0+0QNEBtbeftLIYQQvunUKYUhQwLJzdUwdaqZ+++3Xv2HashZVS4lUQOtW6tERVnYvVvH\nL79IkwghhC8qKoLhw01kZGh45hkzI0e6tl537959+PLLz8nLO4XFYqlxlUsZ6z0vNraMnTt1fPih\nnjfe8FypOCGEEM5ntcKTTwZw6JCWgQPLmDDBeZOS8vJOMXZsQsXtxMQxaLVapk+f7ZQql35bmexS\npaXQpUsdLBaFw4cLkM2yrswXKhF5K2lb15G2dQ1vb1dVhaQkI4sXG4iKsrByZTEXzfHyClKZrBoM\nBmjSxMbp0wrNmwfRtGkQyclu3HBUCCGES6SkGFi82ECHDlYWL/a+JH01MvR9XnKykcOHy5tDwWyG\nBQvs/5ue3DVFCCFEza1dq+ONN4w0bmxj5cpi6tb1dETXTnrU56WmOp7xXdVxIYQQ3m3nTi3PPBNA\n3boqK1cW07ix113prRbpUZ9nrqLTXNVxIYQQ3ic52Uhqqr7ib7eiwNKlxXTo4Jq10u4gPerzjFVc\njtZLh1oIIWqF5GQjCxYYMJsVwP6lqgobNtTuPqkk6vNiYhyvp1NVOHhQmkkIIbydr17ClAx03pQp\nZuLjSzEaVUDFaFTp3duCqsLgwYEcPixNJYQQ3sxXL2FK9rnIlClmMjMLyM4uIDOzgFWrivnggxIK\nCmDQoEC+/16aSwghvI3NBv/6V9Vrrqq6tFlb1CjzWCwWxo8fz5AhQ4iOjubgwYMA/PjjjwwePJjB\ngwczceLEiscvWLCAgQMHMmjQILZv3+6cyN3ksccs/OtfJZw5A4MGmfjxR0nWQgjhLc6ehbi4AKZM\nMVKnjuNZ3VVd2qwtapR1/vOf/2AymVi5ciWTJ0/mzTffBGDy5MkkJyezatUqCgoK2L59O5mZmWzc\nuJEVK1Ywd+5cpk6ditXquiLorjB4sIV33jFz6pSGAQNMpKcrng5JCCH83g8/aOjTpw6bNumJirJw\n4EDRZZcw4+NLa30tjBpNhevXrx8PP/wwACEhIZw+fZrS0lJOnjxJp072zbPvvfde9uzZQ05ODlFR\nURgMBkJCQmjSpAnp6em0a9fOea/CDWJjyygrg1de0TJkyEyMxiWsW7eBsLCGWCwWZs+eyZ49uzCb\nzQwYEM3QobGeDlkIIXzWunU6xo0LoKhI4ZlnzCQllaLT2S9h1vbEfKkaJWr9RWuWli5dysMPP0x+\nfj51Lyr50qBBA3JycqhXrx4hISEVx0NCQsjJyblioq5fPxCdTluT0By6Ug3Va5GUBF9+OZpvv+2I\n0QjFxUGEhgazYsUKfvnlv3z66SeUlpYSHR3NnXf2IDIy0inn9VbOaldxOWlb15G2dQ13tWtpKbz0\nEsyYAcHBsG4dPPqoEajlF6Kv4KqJOi0tjbS0tErHEhMTiYqKYvny5Rw9epQ5c+aQl5dX6TFV7fVR\nnT1A8vOLrvqY6nJ2sfgJE+L48stI1qz5gIEDbfznPwVs3bqDe+7pw9mz9t1Y/vKXh1i//lOaN69d\nowbXwtuL8Ndm0rauI23rGu5q1z/+UIiPD2D/fh3t2llZsqSYVq1UcnJcfmqXu9IHnasm6kGDBjFo\n0KDLjqelpbFlyxZmzZqFXq+vGAIvl5WVRVhYGGFhYRw7duyy47VVREQnIiJKWbMGTp7UMGBAIHfc\noWCzXbjubjIFcvJk5hWeRQghxLXYs0dLfHwAOTka+vcvY9q0EoKCPB2Ve9RoMllmZiarVq0iJSUF\n4/l573q9nvDw8IoZ4F988QVRUVH07NmTbdu2UVpaSlZWFtnZ2bRu3dp5r8CDRo4083//p2Hfvl6s\nX/8x586d48yZ03z++UbMZuftdSqEEP5KVWHOHD2PPWYiL0/hjTdKmDv3+pK0xWJh5sz36dUrkuzs\nrIrj+fl5PPfcUzz+eH8nRO48NbpGnZaWxunTp0lIuLBR9sKFC0lOTua1117DZrPRuXNn7rjjDgCi\no6MZPnw4iqIwadIkNBrfWOL0zDNlaDSlzJ37OIqSSXz8CEJDb6R79x4cP57h6fCEEKJWKyiAceMC\nWL9eT1iYjQULSujZ8/pXDSUljaNDh1sqHTt79gxjxybQs+cd/P77b9d9DmdS1OpcNHYzZ17rcNW1\nk169Ilm3bgOhoQ0rNiTv1MnK2rVFrFs3H1VV+dvfEq7+RLWUXOtzHWlb15G2dQ1XtGt6usLIkSZ+\n+knLbbdZWLiwhIYNnZOuvv/+OyIiOlX8HQ8La8jZs2c5fTqP3Nxc3nrrn6xevd4p56quK12j9o2u\nrQcpCtx773p69nyW775TiI4uYMOGT+nb9wFPhyaEELXSp5/q6Nu3Dj/9pCUhoZSPPip2WpIG+1yj\nS9WtW5dmzVo47RzOVLu3FHGzvLxTjB17oZecmDgGrVbL9OmzueWWLRQU/Jn8fB0wjnr1mnouUCGE\nqIUsFpg61cDMmUYCA1XmzCnmsccsng7L4yRRX4OQkAasWLHW4X1Tp76DxQJPPWW/nhIba2HZsmIC\nA90cpBBC1EI5NtmWdAAAGpVJREFUOQpPPBHAzp06wsNtLF5cu/eQdiYZ+nYinQ4++KCEBx8sY9cu\nHXFxJkpKPB2VEEJ4t0OHNPTpE8jOnTruv7+ML74olCR9EUnUTqbXw7x5JfTta2HbNh2jRpkolZVa\nQghxGVWFJUv09OsXyB9/KEyYYGbJkhIuKnIpkKFvlzAYYMGCYmJjTWzerCMhIYD580vQ1+69y4UQ\nwmmKi+HllwNYvVpPSIiNuXNLuPtu12/YVNVco5iYkaSmLqakpIS8vFMMHTqA0NAwpk+f7fKYrkaW\nZ7lQcTEMG2Zi1y4d/fuXMWtWCTof+Wgky1xcR9rWdaRtXaM67ZqcbCQ1VY/ZbO/MBAWp5OVp6NLF\nyqJFxdx8s9elIreS5VkeYjJBamoxPXpYWL9ez7PPBmCTyy5CCD+TnGxkwQIDZrMCKJSWKuTlaWjX\nzsrHHxf5fZK+GknULlanDqxYUcytt1pJS9Pz4otGSdZCCL+Smur4ut/x4xoCAtwcTC0kidoNgoNh\n1aoiOne2smyZgVdeMeJ9FxyEEML5iorAXMX20FUdF5VJonaTG26A1auL+NOfrCxebOC11yRZCyF8\nV1GRfTON7t3rAIrDxxh9dwtpp5JE7UYhIZCWVky7dlbmzjUwebJBkrUQwqdcnKBfey2AoiKFzp0d\nVxeLiSlzc3S1kyRqNwsNVVmzppjwcBszZhh55x2Dp0MSQojrVlQEs2dfSNDFxQrPP2/m0KECNm8u\nJj6+FKNRBVSMRpX4+FKmTJGx7+qQ5Vke8ttvCo88Esj//qdhwgQzzz5bu6qieGu7+gJpW9eRtnW+\noiJYuzaYqVNt5OZqCApSGT26lDFjSgkJ8XR0tceVlmf5yKre2qdxY5V164p45JFAJk82YjCoPPmk\nDAMJIWqHoiJYulTPzJkGcnMhKEhh3DgzY8aUUr++p6PzLZKoPahpU5W1a4vo3z+QiRMDMBhg1ChJ\n1kII71VYaE/QKSmGih70q69CTEyBJGgXkUTtYS1b2pP1I48E8sorAej1EBsryVoI4V0cJejyHnTb\ntsHk5Hg6Qt8lidoLtG6tsnZtMY8+auKll+zD4IMHyx6sQgjPu1KCdnYP2mKxMHv2TFavXs66dRsI\nC2sIwJIlC/jii8+w2VTatm3Hyy9PICgoyLkn92Iy69tLtG9vIy2tmBtugOeeC2DdOvkMJYTwnMJC\nmDXLPot70qQASkrs16APHSogKck116GTksYRGBhY6djWrV+yZctmFiz4kBUr1qAosHz5Uuef3ItJ\novYiERE20tKKCAqCp58O4JNPJFkLIdyrsBA++MC9CbpcXFw8o0aNqXSsefOWJCdPIjCwDhqNhoiI\nzhw/nuG6ILyQZAIv07mzjVWrihg0KJAxYwIwGIr5y19cv/WbEMK/FRba94b+4AP7EHdwsOuGuKsS\nEdHpsmPh4a0q3d6792u6dOnqnoC8hPSovVBkpI2VK4vPzwI3sWWL1tMhCSF81MU96H/8IwCz2d6D\nPnjQ9T3oa7V06ULy808xcOBgT4fiVpKovVTPnlZSU4vRaCAuzsSOHZKshRDO4yhBv/CCe4a4a2LO\nnBS2b9/KtGkfYDKZPB2OW0mi9mJRUVaWLCnGZoPYWBN790qyFkJcn8JCSElxnKDHjy+lXj1PR3i5\nhQvncuTIYVJS5lLPGwN0MUnUXq53byuLFhVTVgZDhpg4cED+y4QQ1+7iBP3667UjQQP8+ON/2bRp\nI2+99T6BgXU8HY5HyGSyWqBvXyvz5pUQHx/A4MGBrF1bRJcuNk+HJYTwUsnJRlJT9ZjNYDBA585W\nMjI0nDplnyT2wgv2SWLelJzz8k4xdmxCxe3ExDFotVo6d+5KQcE5EhJGVNzXqNFNTJuW4okwPUI2\n5ahF1q/X8cQTAeh0oKpQVmbfzzUmpsztu9D4Urt6G2lb1/GHtk1ONrJgweW78un1Ks8+W0pCgvMT\ntD+0q6tdaVMOGUetRfr3txAVZaW0VKGsTAEUzGaFBQsMJCfLDuxC+DOrFQ4c0LB4sd7h/RoNvPyy\nd/WiRfVIoq5lqppQlprq+JdTCOG7srIUVq3SkZAQQIcOQTz0UB2sVsXhY82y9XOtJdeoa5mqftnM\nZvjsMx19+1rQyuRwIXxSWRkcOqTlq6+0bNmi48iRC7/sTZrY+Otfy1i5Uo/FcnmyNsqgW60libqW\nMRqrStYKI0aYaNHCRkJCKYMHl+FHNeuF8Fm//aawdauOr77SsmOHjrNn7UnYYFC56y4LvXtbuO8+\nK23b2lAU+98IR9eoY2JkV77aShJ1LRMTU+bwl/Cxx0oJDIS0ND3JyQG8+aaRYcPKiI8vpWlTr5sv\nKISoQmkp7N+v5auvdGzZouW//73Qa27WzMaAAWX07m3hzjutDj+Ml08sLZ/17akJp8J5ZNZ3LXTx\n0otLfwlzcxWWLtWzaJGenBwNWq3KQw9ZGDOmlO7dnbekyxfb1VtI27qOt7ZtZqbCli32XvPOnToK\nC+295oAAlTvusJ7vNVsID1dRHF+C9ihvbdfa5EqzviVR+yizGT76SMfcuQaOHrV/Ir/1VitjxpTy\n8MMWdNc5luKv7eoO0rau4y1tW1Jinxj61Vc6tm7V8vPPF3rN4eE27rvPPqR9++1WLtn10St5S7vW\nZldK1DL07aOMRhg82MLjj1v4+mstc+fq+fxzHQkJJpo0sfG3v5UREyNLNYRwl2PH7L3mLVt07N6t\npajI3jUODFTp08eemHv3ttCypdf1nYSHSY/aj2RkKMyfb2DlSj1FRQqBgSqDB5eRkFBKePi1vQ2k\nXV1H2tZ13Nm2RUWwZ0/5tWYdGRkXVsO2bWuld2/7kHbPnlYCAtwSUiUWi4XZs2eyevVy1q3bQFhY\nQ8BeV/urr77AZlNp27YdL72UTHBw1b09kPesM0jBEz9ksViYOfN9evWKJDs7C4DmzS00ajSFHj36\nEhl5P40avcLixWXcfnsdYmJM7NypRVXt18CbNg0iLCyIpk2DpJiKENWgqpCerjBvnp7HHzfRvn0Q\nQ4YEsmCBgawshfvvL+Odd0o4dKiAXbuKeP11M/fc45kkDZCUNI7AS8bVN2/exIED+1i8eDkrVqzB\nZrOSmrrIMwGKCjL07aOSksbRocMtlY5t2PAxP//8I6mpq9DpdPzjH3+nsHAWR4+O4/PPdXz+uY6Q\nEBt5eRc+v5nNF5Z6yKxRISorKIDduy/0mk+cuPC706GD9fy1Ziu33WbFcPliDY+Ki4snIqITixfP\nrzjWokU4L7yQhNFo//TQteutHDiwz1MhivMkUfsoR7+EGRnpdOzYGcP5vxjdut3K3r27+eyzIg4e\n1DBvnoH16x2/JVJT9ZKohd9TVfjpJw1bttiT8759WkpL7deag4NVHn64jPvusw9p33ST111VrCQi\notNlx9q0aVvxfUFBAVu3fsX99z/ozrCEA5KofZSjX8Jbb+3OwoXzGDo0FqPRyNdf7+S2224HIDLS\nRmRkCevXO66SYjbDxIlGevSw9w5CQ10avhBe49w52LHDvqZ5yxYdJ09e6DV37Hih13zrrVb0PlLJ\nd9KkCezcuY0///kv3H//w54Ox+9JovYjUVH3sH37Vh555C/odDratm1Pv36PVnpM1ZXPYPZsA7Nn\n279v1w4iIy8k7pYtvXN9pxDXSlXh6FHN+RnaWvbv11aU5KxXT6V/f3vBkXvvtdKwoXf3mmtq0qTJ\nmM1mZs+eweuv/53XX5/q6ZD8miRqP5KWtorTp/P57LOt6HQ63n//baZPf48XX0yqeExVlc9GjCjj\nkUcs7NunZd8+LQcP6vjpJwPLl9vvDw210aOHteIrIsJ23Wu1hXCXM2dg+3ZdRTWwrCx7r1lRVLp0\nsVUsnerWzebTtfQPHTpA/fohhIe3wmg08te/PsrTT8d7Oiy/J39K/ciBA3u56657CTg/zfSee+5j\n+vT3Kj3mauUHe/WyAlC/fjA7dhRWJO59+7R8+qmeTz+1j/0FBqrcequ9t92jh5XISMflDoXwBJsN\njhzRVFQDO3RIW7HrVIMG9jKd991n4Z57rNx4o2/2mh357rtvOXLkMG++OQ2DwcDu3Tto1aqNp8Py\ne5Ko/UjTps3Zu/drHnzwr+h0Ovbs2UV4eKvLHjdlivmqE8d0OujY0UbHjjbi48tQVThxQmH/fnvS\n3r/fXgpx5077W0yjUYmIqNzr9tVhQ+F5jsrsvviimS+/hI8+CmDrVi25ufZes0aj0q3bhWpgnTvb\n0Pj4wtW8vFOMHZtQcTsxcQxarZbp02dz6lQuI0YMRlWhYcOGjB//qgcjFVDDgicWi4UJEyZw4sQJ\nrFYrL7/8MpGRkcTExFBUVFSxNm/8+PFERESwYMECNm3ahKIojB07lrvvvvuKzy8FT67Pxb+EJ078\njyZNbq74JZw1awY//PA9iqKhWbNmvPRSMqGhYdd8juq0a34+HDhwocf97bcXZsgCNG9eOXG3aWOT\n69z453vWmZKTjQ4v34AK2N9goaE2eve2TwS7+24L9eu7NUSfI+/Z6+f0Wt9r167lyJEjTJo0iV9+\n+YVXXnmFNWvWEBMTw9///nfatr0wxT8zM5Nnn32WVatWUVBQwNChQ9mwYQPaK1zokUTt/WrSriUl\n8O239t52+dfp0xcyc0iIjdtus9K9u40ePew9G3/cQ1fes1dmNkNWlsIffyhkZWnIylLO39bwxx8K\nO3ZoUdXLP/EpisrkyQo9ehRyyy2+32t2J3nPXj+n1/ru168fDz9sn7IfEhLC6dOnq3zsvn37iIqK\nwmAwEBISQpMmTUhPT6ddu3Y1ObWoxQICoGdPKz172q9z22zw88+aih73/v1aNm3Ss2kTgBGjUaVr\n1ws97u7drdxwg0dfgnCh8gRcnnQvTcDZ2fbv8/OvNuziuO+hqvDKK5CT47xd5IRwhxolav1FiwWX\nLl1akbQBZsyYQX5+Pq1atSI5OZnc3FxCQkIq7g8JCSEnJ0cSdS1XVlbGzJnvV6oTPGvWdHbt2lHx\nmJKSEurVq8+iRcscPodGA+3b22jf3saIEfZN7X/77cJ17vLkvXev/W2qKCrt21ceLr/5ZrnO7e0u\nTcD2hHt5Mr5aAq5bV6VRIxsRESoNG9q/t/9rv92wof12u3ZBDpcY+uPojPANV03UaWlppKWlVTqW\nmJhIVFQUy5cv5+jRo8yZMweA2NhY2rVrR7NmzZg4cSLLy9fuXKQ6I+316wei0zlvDcSVhhREzYwe\nPZqOHTsC0KBBEKGhwUycWHnSyaRJk2jVqtU1tX9oKHTuDKNH22+fPQt798KuXbBrl8LevVr++18t\nS5bY72/aFHr1gjvvtP8bEUGl5TPPPAPz59uH3QMC7M87Y8b1vHL3qA3vWbMZ/vgDfvvN/vX775d/\n//vvcOrUlZ/nhhugcWPo2tX+b+PGcNNNl/8bGKgAV/+7kJAAM2c6Ol5+fdr727Y2knZ1nRrvnpWW\nlsamTZuYNWsWRgcfVbdv387GjRvp0aMHx44d44UXXgBweB37UnKN2vv9+ms6N9/cml69IivtvFMu\nIyOd115LZsmSFeicuKC6rMy+rObiHnf57F2wl3Hs3t3e2/7uOw0bNlxeKio+vtSry6E64z3raNZz\ndV+z2UxFr7f8GnD59/Z/7V8X14R3pLwHbO/tXugFX9oDdsV+y1W9fvl74BrSrtfP6deoMzMzWbVq\nFcuWLatI0qqqMnLkSGbMmEHdunXZt28fbdq0oWfPnixevJjExETy8/PJzs6mdevWNXslwmt07dr1\nir+YixbNZ9iwWKcmaQC9Hrp1s9Gtm40nn7QvC8vIUM4nbh3792sr9vytOjY9p04paLX259Pp1Irv\n7f+q6HRc8djFP6PTlX9deky96L6qjl3+mOt16azn8o1VLBZITCytchLWtSTghg1t3HKLpVICvjAM\n7boEXF3VWWIoRG1Roz8LaWlpnD59moSEC+vwFi5cSHR0NHFxcZhMJho2bEhiYiImk4no6GiGDx+O\noihMmjQJjUy39Gm//prJDz98z6RJk11+LkWBVq1UWrWyMHSoBbD3Bg8c0DJyZADly3EuZrMpfPSR\n9xZl1mpBpwuqSNx6vXr+GFc4diHh79jheHh4yRIDS5ZUvYXTxQk4LMyedC9OwGFh9u/r1HHVKxdC\nOFLjoW9XkqFv71fero6GvpcuXUh+fh7PPfeSByOEpk2DMJsvT9RGo8rBg4WUlYHFUv6lXPS9o9tX\nf0xZmYLVWvXPlJVx/v5LjymV7gcdJSXWq/zM5cccLUmqTGXAAEtFr9eeiP0rAcvfA9eQdr1+Th/6\nFuJKvv56FyNHjvZ0GFXWLY+JKXNQFc17Pq/a/+gVXfPP2Wz2pB0eHlSpsEw5oxFmzy5xRog+wWKx\nMHv2zEorFy6WkvIvtm37ijVrPvFQhELYyRi0cLr/+79faNGipafDYMoUM/HxpRiNKqBiNKpeP5Hs\nemg0YDBAbGyZw/tjYhwf91dJSeMqqihe6pdffmbnzm3uDUiIKkiiFtcsL+8U999/P0OHDgDsdYKH\nDh1ATk42Z8+eoaSkhJCQBh6O0m7KFDOZmQVkZxeQmVngs0n6Yv72AaWm4uLiGTVqzGXHbTYb7733\nJqNHP+mBqIS4nAx9i2sWEtKATZs2VXlNateug26OSFxKZj1fXUREJ4fH//OfdYSHt+KWWzq6OSIh\nHJMetRBCnHfqVC7//vcKnngi0dOhCFFBErUQQpw3Y8Y0Ro4cTd26dT0dihAVJFELv2exWJg58316\n9YokOzur4vjhw98SExNNdPQjPPPME+Tm5ngwSuEOX3+9i5SUf9Gv318YPTqW7Ows+vX7C6WlpZ4O\nTfgxuUYt/F5S0jg6dLil0rHCwgJeey2JyZPfISKiI8uWLWHz5s8ZMmS4h6J0PkfLkzZu/ITp09+l\nQYMbKx43YEA0AwY87sFI3Wfz5gubyvz++28kJo6R5VnC4yRRC78XFxdPREQnFi+eX3Fs587ttGvX\nnogI+4Si4cPjPBSd6zj6gAJw1133MmHCJPcH5EZ5eacYO/ZCZcXExDFotVqmT59NaGiYByMT4nKS\nqIXfczT7Nz39F264oR6vvPIix45l0K5dO55/fjz16tXzQISu4egDir8ICWnAihVrr/iYm25qLL1p\n4RXkGrUQDhQUnGP//r08/fSzLFv2b/R6AzNmvOfpsJyqquVJv/zyM2PHJjB48GNMnfo6BQUFbo5M\nCHExSdRCOFCnThCRkd25+eam6HQ6Bg0awv79ez0dlss1bdqMqKi7efvt91myZAWFhYU+9wFFiNpG\nhr6FcKBRo5v49dcTFbc1Gg1are9/ru3YsTMdO3auuB0TM5IXXpA1xUJ4klcm6ivtIuINzyfsfLFd\nGzQIIjQ0mP79H2Lhwjnk5f1Gu3btmDXrU+688063vWZ3t2356/79998xGo2EhIQA8PvvRgwGvU/9\nX/vSa/Em0q6u45WJWgh3yc3NZfjwC0uuYmJi0Gq1LF26lKlTpzJ27FgURaFNmza88cYbHozUPVau\nXEl6ejrTp09Ho9GQmprKPffc4+mwhPBrXrkftRDCtS7+gHLs2DGaNWtW8QHl/fff55tvvkFRFLp1\n60ZycjLBwdJbEsJTJFELIYQQXsz3Z8cIIYQQtZgkaiGEEMKLSaIWQgghvJjPzvqeMmUKhw8fRlEU\nkpOT6dTJcRUmcXX79u3j2WefpU2bNgC0bduW+Ph4Xn75ZaxWK6GhobzzzjsYDAYPR1p7/Pzzzzz1\n1FPExcUxfPhwfv/9d4ft+fHHH7N06VI0Gg3R0dEMGjTI06F7vUvbNikpiaNHj1aUfx01ahT33HOP\ntO01evvttzl06BAWi4UxY8bQsWNHec+6i+qD9u3bpyYkJKiqqqrp6elqdHS0hyOq3fbu3asmJiZW\nOpaUlKRu3LhRVVVVfe+999Tly5d7IrRaqbCwUB0+fLj66quvqqmpqaqqOm7PwsJCtW/fvurZs2fV\n4uJi9aGHHlLz8/M9GbrXc9S248ePV7ds2XLZ46Rtq2/Pnj1qfHy8qqqqmpeXp959993ynnUjnxz6\n3rNnD3/+858BaNWqFWfOnJF6xU62b98+7rvvPgDuvfde9uzZ4+GIag+DwcD8+fMJC7uwS5Oj9jx8\n+DAdO3YkODiYgIAAunXrxjfffOOpsGsFR23riLTttenevTvTp08HoG7duhQXF8t71o18MlHn5uZS\nv379itshISHk5OR4MKLaLz09nSeeeIIhQ4awe/duiouLK4a6GzRoIO17DXQ6HQEBAZWOOWrP3Nzc\nigphIO/j6nDUtgDLli0jNjaW559/nry8PGnba6TVagkMDARgzZo13HXXXfKedSOfvUZ9MVWWil+X\nFi1aMHbsWB544AEyMzOJjY3FarVW3C/t61xVtae0c8088sgj1KtXjw4dOjBv3jxSUlLo2rVrpcdI\n21bPl19+yZo1a1i0aBF9+/atOC7vWdfyyR51WFgYubm5Fbezs7MJDQ31YES1W8OGDXnwwQdRFIVm\nzZpx4403cubMGUpKSgDIysq66lCjuLLAwMDL2tPR+1ja+drdfvvtdOjQAYDevXvz888/S9vWwM6d\nO5kzZw7z588nODhY3rNu5JOJ+s477+Tzzz8H4OjRo4SFhREUFOThqGqvjz/+mIULFwKQk5PDqVOn\neOyxxyra+IsvviAqKsqTIdZ6d9xxx2Xt2blzZ44cOcLZs2cpLCzkm2++ITIy0sOR1j6JiYlkZmYC\n9rkAbdq0kba9RufOnePtt99m7ty5FbPn5T3rPj5bQvTdd9/l4MGDKIrCxIkTad++vadDqrUKCgp4\n8cUXOXv2LGVlZYwdO5YOHTowfvx4zGYzjRs3ZurUqej1ek+HWit8//33vPXWW5w8eRKdTkfDhg15\n9913SUpKuqw9N23axMKFC1EUheHDh9OvXz9Ph+/VHLXt8OHDmTdvHiaTicDAQKZOnUqDBg2kba/B\n6tWrmTlzJi1btqw49uabb/Lqq6/Ke9YNfDZRCyGEEL7AJ4e+hRBCCF8hiVoIIYTwYpKohRBCCC8m\niVoIIYTwYpKohRBCCC8miVoIIYTwYpKohRBCCC8miVoIIYTwYv8fxbMd0OGQfToAAAAASUVORK5C\nYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f80bf884cf8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "SM8HHsj3Xw9p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **3. Data Loading**\n",
        "\n",
        "In this section we define the data loaders and the auxiliary classes that make possible loading sequences of different lenghts."
      ]
    },
    {
      "metadata": {
        "id": "9Dbh7DIbgvgc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **3.1  Auxiliary classes to get variable sized sequences**\n",
        "\n",
        "We have the data organized in folders. We have a folder for the training data, a folder for the validation data, and a folder for the test data. For example, for the training data, we have 10 folders, one for each class, each one containing the training samples in the \".npy\" format. This can be seen running the following commands, the first one shows the structure of the training folder and the second one counts the number of files in the data/training/apple directory.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "9WeS_kaZMsRR",
        "colab_type": "code",
        "outputId": "aef02353-cc7b-4dfc-fbf5-4fe2a57345aa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "!ls data/train\n",
        "!ls -l data/train/apple | egrep -c '^-'"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "apple  banana  book  fork  key\tladder\tpizza  stop_sign  tennis_racquet  wheel\n",
            "6000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hnkSCwVmORxG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Having this structure, we can take advantage of the PyTorch class [DatasetFolder](https://pytorch.org/docs/0.4.0/torchvision/datasets.html#datasetfolder). This class creates a datasetFolder object that **understands every folder in the root directory as a different class**, and inside every folder of each class it expects to find all the samples of that class. These samples can be in the extension that we want, in our case: \".npy\".  \n",
        "\n",
        "However, we need more things than a DatasetFolder object in order to load batches automatically while training our network, and we need ***even more*** things if our data is variable-sized. \n",
        "\n",
        "Let us go step by step. Once the DatasetFolder object is created, we need an object that **gets the data from that Dataset automatically**, that object is called [DataLoader](https://pytorch.org/docs/0.4.0/data.html#torch.utils.data.DataLoader ). The DataLoader is created taking as argument the DatasetFolder object so it knows where to get the data and how to get it. In addition, and here is where the variable-sized sequences come into play, **we can define a method that preprocess our data before building a batch and pass it as argument!** To do that, we create a class called PadCollate that deals with this problem. \n",
        "\n",
        "This class will take a batch of different length sequences, it will compute the length of all of them, and finally it will zero-pad all the sequences in decreasing length. That is, if we have a batch of 4 sequences with lengths {3, 4, 6, 10}, we will end with a batch of 4 sequences of length 10. We want to preserve the length of the sequence, otherwise, the LSTM won't be able to freeze its hidden layers when the padded zeros are passed as input (it will just take the important part of the input sequences). \n"
      ]
    },
    {
      "metadata": {
        "id": "6xNq_vmpwdTz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def pad_tensor(my_vec, my_pad, my_dim):\n",
        "    \"\"\"\n",
        "    args:\n",
        "        my_vec - tensor to my_pad\n",
        "        my_pad - the size to my_pad to\n",
        "        my_dim - my_dimension to my_pad\n",
        "\n",
        "    return:\n",
        "        a new tensor my_padded to 'my_pad' in my_dimension 'my_dim'\n",
        "    \"\"\"\n",
        "    my_pad_size = list(my_vec.shape)\n",
        "    my_pad_size[my_dim] = my_pad - my_vec.size(my_dim)\n",
        "    return torch.cat([my_vec, torch.zeros(*my_pad_size).double()], dim=my_dim)\n",
        "\n",
        "class PadCollate:\n",
        "\t\"\"\"\n",
        "\ta variant of collate_fn that pads according to the longest sequence in\n",
        "\ta batch of sequences\n",
        "\t\"\"\"\n",
        "\tdef __init__(self, dim=0):\n",
        "\t\t\"\"\"\n",
        "\t\targs:\n",
        "\t\tdim - the dimension to be padded (dimension of time in sequences)\n",
        "\t\t\"\"\"\n",
        "\t\tself.dim = dim\n",
        "\n",
        "\n",
        "\tdef pad_collate(self, batch):\n",
        "\n",
        "\t\t# find longest sequence\n",
        "\t\tlengths = np.flip(np.sort([x[0].shape[self.dim] for x in batch]), axis = 0)\n",
        "\t\tmax_len = max(map(lambda x: x[0].shape[self.dim], batch))\n",
        "\t\tbatch = list(map(lambda x: (pad_tensor(x[0], my_pad=max_len, my_dim=self.dim), x[1]), batch))\n",
        "\t\t# stack all\n",
        "\t\txs = torch.stack(list(map(lambda x: x[0], batch)), dim=1)\n",
        "\t\tys = torch.tensor(list(map(lambda x: x[1], batch)))\n",
        "\t\treturn xs, ys, lengths\n",
        "\n",
        "\tdef __call__(self, batch):\n",
        "\t\treturn self.pad_collate(batch)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HuX4rAijADid",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **3.2 Data Loaders definition**"
      ]
    },
    {
      "metadata": {
        "id": "4oU8VvQjVRhC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the cell below we define our DatasetFolder objects and the corresponding DataLoaders associated to each DatasetFolder. Notice that to handle the variable-sized sequences we must pass as argument a method that pads the sequence with zeros.\n",
        "\n",
        "We also define our batch size (here called **bs**), since the DataLoader needs to know how many samples it has to get for each batch."
      ]
    },
    {
      "metadata": {
        "id": "JP9g7gv78BUh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def load_sample(x):\n",
        "  \"\"\"\n",
        "  We have to tell to the DatasetFolder object HOW TO load a single example so it can form batches with a few samples each one.\n",
        "  \"\"\"\n",
        "  # We load them in format N x 2 in order to stack them in proper order\n",
        "  return torch.from_numpy(np.load(x).T).double()\n",
        "\n",
        "# BATCH SIZE\n",
        "bs = 30\n",
        "\n",
        "# DatasetFolder creation and DataLoader creation\n",
        "train_dir = r\"data/train\"\n",
        "val_dir = r\"data/validation\"\n",
        "test_dir = r\"data/test\"\n",
        "\n",
        "train_dataset = datasets.DatasetFolder(train_dir, extensions = ['.npy'], loader = load_sample)\n",
        "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = bs, shuffle = True, num_workers = 0, collate_fn = PadCollate(dim=0))\n",
        "train_iter = iter(train_loader)\n",
        "\n",
        "valid_dataset = datasets.DatasetFolder(val_dir, extensions = ['.npy'], loader = load_sample)\n",
        "valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size = bs, shuffle = True, num_workers = 0, collate_fn = PadCollate(dim = 0))\n",
        "valid_iter = iter(valid_loader)\n",
        "\n",
        "test_dataset = datasets.DatasetFolder(test_dir, extensions = ['.npy'], loader = load_sample)\n",
        "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = bs, shuffle = True, num_workers = 0, collate_fn = PadCollate(dim = 0))\n",
        "test_iter = iter(test_loader)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Q8pQn318ZHPE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To check that the resulting batch has the expected size, we load one single batch."
      ]
    },
    {
      "metadata": {
        "id": "yuDTFxOvYs6U",
        "colab_type": "code",
        "outputId": "3fe58906-4b0b-4db5-ce2f-155c46399ec1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        }
      },
      "cell_type": "code",
      "source": [
        "samples, labels, lengths = valid_iter.next() \n",
        "print(\"The size of the batch is:\")\n",
        "print(samples.size())\n",
        "print(\"The size of the labels is:\")\n",
        "print(labels.size())\n",
        "print(\"The size of the length is:\")\n",
        "print(np.shape(lengths))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The size of the batch is:\n",
            "torch.Size([108, 30, 2])\n",
            "The size of the labels is:\n",
            "torch.Size([30])\n",
            "The size of the length is:\n",
            "(30,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TPxaZbBYZmRm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 4. Model definition and training"
      ]
    },
    {
      "metadata": {
        "id": "fLE_h_uaHnsM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4.1. Model Construction"
      ]
    },
    {
      "metadata": {
        "id": "8TnHl4EaZr0B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that we have our dataset structured in folders, our dataloaders that get batches automatically, it is time to build the model! \n",
        "\n",
        "The model that we are going to train is an LSTM (Long-Short Term Memory Network). \n",
        "\n",
        "First of all, we must consider the sizes of the tensors that the network is going to take as input. Recalling the previous information about the batch size and the batch padding explained in section 3, we have:\n",
        "\n",
        "1.   Sequences of  ** * x 2** , where ** * ** is the length of the sequence,  **which is variable**, and ** 2 ** is given by the keypoints in the drawing, which have a range between (0, 0) and (255, 255).\n",
        "\n",
        "\n",
        "2.   We pad this sequences with zeros according to the longest sequence length. Thus, we end with a batch of padded sequence that will have the size: ** LONGEST_LENGTH x BATCH_SIZE x 2** \n",
        "\n",
        "\n",
        "Having explained the input to our network, we have to build our LSTM network. Since we are solving a classification problem, we will need a fully connected layer on top of the LSTM in order to classify the extracted features coming from the LSTM hidden layer.\n",
        "\n",
        "A very important point in the cell below is **how we are passing the padded batch to our model**. If we look at the forward pass method, it needs to know the lengths of the padded sequence in order to not take these padded values as valid inputs. Then, thanks to the PyTorch method [pack_padded_sequence](https://pytorch.org/docs/0.3.1/nn.html#torch.nn.utils.rnn.pack_padded_sequence) we can forward pass the batch to the network safely.\n",
        "\n",
        "Another important point is that if we look at the forward method in the cell below, it has not a softmax layer at the end. The reason is because we are using [CrossEntropyLoss](https://pytorch.org/docs/0.3.1/nn.html#torch.nn.CrossEntropyLoss) class and it computes the softmax for us. "
      ]
    },
    {
      "metadata": {
        "id": "ZicgElYxC7fO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Quick_draw_LSTM(nn.Module):\n",
        "  def __init__(self, lstm_input_size, lstm_units, lstm_hidden_units, batch_size, output_dim):\n",
        "    super(Quick_draw_LSTM, self).__init__()\n",
        "    self.lstm_input_size = lstm_input_size\n",
        "    # Number of stacked lstm that we will have \n",
        "    self.lstm_units = lstm_units\n",
        "    # Number of hidden units of each LSTM\n",
        "    self.lstm_hidden_units = lstm_hidden_units\n",
        "    self.batch_size = batch_size\n",
        "    self.output_dim = output_dim\n",
        "    self.__build_model()\n",
        "  \n",
        "  def __build_model(self):\n",
        "    self.lstm = nn.LSTM(input_size = self.lstm_input_size, hidden_size = self.lstm_hidden_units,num_layers = self.lstm_units)\n",
        "    self.hidden_to_class = nn.Linear(self.lstm_hidden_units, self.output_dim)\n",
        "    \n",
        "  def init_hidden(self):\n",
        "    hidden_a = torch.zeros(self.lstm_units, self.batch_size, self.lstm_hidden_units)\n",
        "    hidden_b = torch.zeros(self.lstm_units, self.batch_size, self.lstm_hidden_units)\n",
        "    return (hidden_a.to(device), hidden_b.to(device))\n",
        "  \n",
        "  def forward(self, X, X_lengths):\n",
        "    # at the beginning of each sequence we must reset the hidden states\n",
        "    self.hidden = self.init_hidden()\n",
        "    seq_len, batch_size, features_size = X.size()\n",
        "    \"\"\"We pack the batch with pack_padded_sequence, this method is useful because the LSTM won't see\n",
        "    the padded values. This function expects as arguments: a tensor of (T x B x *)\n",
        "    \"\"\"\n",
        "    X = nn.utils.rnn.pack_padded_sequence(X, X_lengths)\n",
        "    # Forward pass through the LSTM\n",
        "    X, self.hidden = self.lstm(X, self.hidden)\n",
        "    # Unpack the padded sequence\n",
        "    X = nn.utils.rnn.pad_packed_sequence(X)\n",
        "    X = X[0].contiguous()\n",
        "    X = X.view(-1, X.shape[2])\n",
        "    X = self.hidden_to_class(X)\n",
        "    \n",
        "    #X = F.log_softmax(X, dim=1) #If we use crossEntropy loss it computes the softmax for us!!\n",
        "    X = X.view(seq_len, batch_size, self.output_dim)\n",
        "    return X"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "esOsTmQft5fZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4.2. Training the network"
      ]
    },
    {
      "metadata": {
        "id": "yzCYcMkdTYCq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the cell below we create the model with 3 stacked LSTM units, each one containing a hidden layer of 512. We must provide it the batch size and the number of classes we are trying to classify."
      ]
    },
    {
      "metadata": {
        "id": "Qck8TM6sARP5",
        "colab_type": "code",
        "outputId": "bf6c4bf4-5d7f-4ff8-d764-3d02de54ca37",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "cell_type": "code",
      "source": [
        "\n",
        "model = Quick_draw_LSTM(2, 3, 512, bs, 10)\n",
        "# Move the model to device hence we can train it on the GPU!\n",
        "model.to(device)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Quick_draw_LSTM(\n",
              "  (lstm): LSTM(2, 512, num_layers=3)\n",
              "  (hidden_to_class): Linear(in_features=512, out_features=10, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "metadata": {
        "id": "uMU8v5vRUGx5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that we have our model properly built. Let us train the model. The process that is performed each epoch is the following one:\n",
        "\n",
        "\n",
        "1.   Get a padded batch from the DataLoader\n",
        "2.   Forward Pass that batch through the network\n",
        "3.   **Get only the outputs at the end of each sequence** instead of the padded sequences\n",
        "4.   Compute the loss with respect these outputs\n",
        "5.   Propagate backwards to compute the parameters update\n",
        "6.   At the end of each epoch, we compute a forward pass and its corresponding loss in the **entire validation set**\n",
        "\n",
        "Warning: If you plan to save this model to do inference later or you simply want to do early stopping, you must save the weight's values. This is done with the  [torch.save()](https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-state-dict-recommended) command.\n"
      ]
    },
    {
      "metadata": {
        "id": "eKoLV6eSt85B",
        "colab_type": "code",
        "outputId": "7db494ea-11ac-4f70-8823-cf2e953850eb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 30017
        }
      },
      "cell_type": "code",
      "source": [
        "#In these lists we save the loss and accuracy to plot them afterwards\n",
        "training_loss_list = []\n",
        "validation_loss_list = []\n",
        "training_accuracy_list = []\n",
        "validation_accuracy_list = []\n",
        "\n",
        "# Optimization Hyper-parameters\n",
        "epochs = 150\n",
        "learning_rate = 0.01\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)\n",
        "\n",
        "for epoch in range(epochs):\n",
        "  \n",
        "  running_loss = 0.0\n",
        "  training_accuracy = 0.0\n",
        "  training_total = 0.0\n",
        "  training_correct = 0.0\n",
        "  \n",
        "  for i, data in enumerate(train_loader, 0):\n",
        "    model.train()\n",
        "    # Sample a batch. Recall that we need the lengths for the padding and packing!\n",
        "    inputs, labels, lengths = data\n",
        "   \n",
        "    inputs = inputs.float().to(device)\n",
        "    labels = labels.to(device)\n",
        "    lengths = torch.from_numpy(lengths.copy()).to(device)\n",
        "    \n",
        "    # Zero the gradients because pytorch accumulates gradients\n",
        "    optimizer.zero_grad()\n",
        "    # Forward pass\n",
        "    outputs = model(inputs, lengths)\n",
        "   \n",
        "    # We take only the last output of each sequence! Valid outputs are the outputs computed in the sequence's lengths\n",
        "    valid_outputs = outputs[np.array(lengths-1), np.arange(0, bs), :]\n",
        "    \n",
        "    # Get the indices of the maximum predicted values\n",
        "    _, predicted = torch.max(valid_outputs.data, 1)\n",
        "    # Get the number of correct predictions to compute the accuracy afterwards\n",
        "    training_total = training_total + labels.size(0)\n",
        "    training_correct = training_correct + (predicted == labels).sum().item()\n",
        "    \n",
        "    # Compute the cross entropy loss\n",
        "    loss = criterion(valid_outputs, labels)\n",
        "    \n",
        "    # Backward pass and parameters update\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    \n",
        "    running_loss += loss.item()\n",
        "    \n",
        "    # Print loss information after 200 batches/epoch\n",
        "    if i % 200 == 199:\n",
        "      training_accuracy = training_correct/training_total\n",
        "      print('[%d, %5d] loss: %.3f - Training_Accuracy: %.3f' % (epoch + 1, i + 1, running_loss / 200, training_accuracy))\n",
        "      training_loss_list.append(running_loss)\n",
        "      training_accuracy_list.append(training_accuracy)\n",
        "      running_loss = 0.0\n",
        "      training_total = 0.0\n",
        "      training_correct = 0.0 \n",
        "      \n",
        "  with torch.no_grad():\n",
        "\n",
        "    validation_correct = 0.0\n",
        "    validation_accuracy = 0.0\n",
        "    validation_total = 0.0\n",
        "    running_validation_loss = 0.0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    for j, valid_data in enumerate(valid_loader, 0):\n",
        "      validation_inputs, validation_labels, validation_lengths = valid_data\n",
        "      \n",
        "      validation_inputs = validation_inputs.float().to(device)\n",
        "      validation_labels = validation_labels.to(device)\n",
        "      validation_lengths = torch.from_numpy(validation_lengths.copy()).to(device)\n",
        "      validation_outputs = model(validation_inputs, validation_lengths)\n",
        "\n",
        "      validation_valid_outputs = validation_outputs[np.array(validation_lengths-1), np.arange(0, bs), :]\n",
        "\n",
        "      valid_loss = criterion(validation_valid_outputs, validation_labels)\n",
        "      _, val_predicted = torch.max(validation_valid_outputs.data, 1)\n",
        "      validation_total = validation_labels.size(0) + validation_total\n",
        "      validation_correct = (val_predicted == validation_labels).sum().item() + validation_correct\n",
        "      running_validation_loss += valid_loss.item()\n",
        "\n",
        "    validation_accuracy = validation_correct/validation_total\n",
        "    print('[%d, %5d] Validation loss: %.3f - Validation_Accuracy: %.3f' %(epoch + 1, i + 1, running_validation_loss/(len(valid_loader)), validation_accuracy))\n",
        "    validation_loss_list.append(running_validation_loss/len(valid_loader))\n",
        "    validation_accuracy_list.append(validation_accuracy)\n",
        "    val_correct = 0.0\n",
        "    running_validation_loss = 0.0\n",
        "  #Save the model for early-stopping\n",
        "  torch.save(model.state_dict(), str(epoch))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1,   200] loss: 2.229 - Training_Accuracy: 0.161\n",
            "[1,   400] loss: 2.126 - Training_Accuracy: 0.194\n",
            "[1,   600] loss: 2.040 - Training_Accuracy: 0.225\n",
            "[1,   800] loss: 1.959 - Training_Accuracy: 0.271\n",
            "[1,  1000] loss: 1.897 - Training_Accuracy: 0.296\n",
            "[1,  1200] loss: 1.793 - Training_Accuracy: 0.337\n",
            "[1,  1400] loss: 1.764 - Training_Accuracy: 0.350\n",
            "[1,  1600] loss: 1.781 - Training_Accuracy: 0.357\n",
            "[1,  1800] loss: 2.015 - Training_Accuracy: 0.268\n",
            "[1,  2000] loss: 1.847 - Training_Accuracy: 0.331\n",
            "[1,  2000] Validation loss: 1.754 - Validation_Accuracy: 0.357\n",
            "[2,   200] loss: 1.966 - Training_Accuracy: 0.284\n",
            "[2,   400] loss: 1.895 - Training_Accuracy: 0.299\n",
            "[2,   600] loss: 1.788 - Training_Accuracy: 0.345\n",
            "[2,   800] loss: 1.716 - Training_Accuracy: 0.380\n",
            "[2,  1000] loss: 1.656 - Training_Accuracy: 0.411\n",
            "[2,  1200] loss: 1.551 - Training_Accuracy: 0.453\n",
            "[2,  1400] loss: 1.467 - Training_Accuracy: 0.482\n",
            "[2,  1600] loss: 1.434 - Training_Accuracy: 0.492\n",
            "[2,  1800] loss: 1.390 - Training_Accuracy: 0.505\n",
            "[2,  2000] loss: 1.340 - Training_Accuracy: 0.525\n",
            "[2,  2000] Validation loss: 1.325 - Validation_Accuracy: 0.535\n",
            "[3,   200] loss: 1.310 - Training_Accuracy: 0.544\n",
            "[3,   400] loss: 1.299 - Training_Accuracy: 0.546\n",
            "[3,   600] loss: 1.208 - Training_Accuracy: 0.575\n",
            "[3,   800] loss: 1.185 - Training_Accuracy: 0.582\n",
            "[3,  1000] loss: 1.152 - Training_Accuracy: 0.598\n",
            "[3,  1200] loss: 1.144 - Training_Accuracy: 0.605\n",
            "[3,  1400] loss: 1.137 - Training_Accuracy: 0.612\n",
            "[3,  1600] loss: 1.088 - Training_Accuracy: 0.621\n",
            "[3,  1800] loss: 1.056 - Training_Accuracy: 0.638\n",
            "[3,  2000] loss: 1.052 - Training_Accuracy: 0.635\n",
            "[3,  2000] Validation loss: 1.081 - Validation_Accuracy: 0.628\n",
            "[4,   200] loss: 1.034 - Training_Accuracy: 0.640\n",
            "[4,   400] loss: 0.973 - Training_Accuracy: 0.666\n",
            "[4,   600] loss: 0.960 - Training_Accuracy: 0.671\n",
            "[4,   800] loss: 0.934 - Training_Accuracy: 0.675\n",
            "[4,  1000] loss: 0.926 - Training_Accuracy: 0.682\n",
            "[4,  1200] loss: 0.908 - Training_Accuracy: 0.683\n",
            "[4,  1400] loss: 0.897 - Training_Accuracy: 0.691\n",
            "[4,  1600] loss: 0.862 - Training_Accuracy: 0.708\n",
            "[4,  1800] loss: 0.875 - Training_Accuracy: 0.696\n",
            "[4,  2000] loss: 0.848 - Training_Accuracy: 0.704\n",
            "[4,  2000] Validation loss: 0.831 - Validation_Accuracy: 0.715\n",
            "[5,   200] loss: 0.833 - Training_Accuracy: 0.714\n",
            "[5,   400] loss: 0.829 - Training_Accuracy: 0.717\n",
            "[5,   600] loss: 0.800 - Training_Accuracy: 0.721\n",
            "[5,   800] loss: 0.810 - Training_Accuracy: 0.727\n",
            "[5,  1000] loss: 0.760 - Training_Accuracy: 0.744\n",
            "[5,  1200] loss: 0.772 - Training_Accuracy: 0.730\n",
            "[5,  1400] loss: 0.792 - Training_Accuracy: 0.727\n",
            "[5,  1600] loss: 0.792 - Training_Accuracy: 0.724\n",
            "[5,  1800] loss: 0.769 - Training_Accuracy: 0.732\n",
            "[5,  2000] loss: 0.751 - Training_Accuracy: 0.742\n",
            "[5,  2000] Validation loss: 0.760 - Validation_Accuracy: 0.740\n",
            "[6,   200] loss: 0.769 - Training_Accuracy: 0.733\n",
            "[6,   400] loss: 0.734 - Training_Accuracy: 0.744\n",
            "[6,   600] loss: 0.730 - Training_Accuracy: 0.751\n",
            "[6,   800] loss: 0.744 - Training_Accuracy: 0.740\n",
            "[6,  1000] loss: 0.729 - Training_Accuracy: 0.757\n",
            "[6,  1200] loss: 0.752 - Training_Accuracy: 0.741\n",
            "[6,  1400] loss: 0.684 - Training_Accuracy: 0.762\n",
            "[6,  1600] loss: 0.718 - Training_Accuracy: 0.750\n",
            "[6,  1800] loss: 0.683 - Training_Accuracy: 0.758\n",
            "[6,  2000] loss: 0.714 - Training_Accuracy: 0.754\n",
            "[6,  2000] Validation loss: 0.697 - Validation_Accuracy: 0.761\n",
            "[7,   200] loss: 0.703 - Training_Accuracy: 0.757\n",
            "[7,   400] loss: 0.659 - Training_Accuracy: 0.775\n",
            "[7,   600] loss: 0.692 - Training_Accuracy: 0.762\n",
            "[7,   800] loss: 0.679 - Training_Accuracy: 0.768\n",
            "[7,  1000] loss: 0.687 - Training_Accuracy: 0.759\n",
            "[7,  1200] loss: 0.676 - Training_Accuracy: 0.767\n",
            "[7,  1400] loss: 0.675 - Training_Accuracy: 0.768\n",
            "[7,  1600] loss: 0.685 - Training_Accuracy: 0.760\n",
            "[7,  1800] loss: 0.675 - Training_Accuracy: 0.763\n",
            "[7,  2000] loss: 0.656 - Training_Accuracy: 0.776\n",
            "[7,  2000] Validation loss: 0.734 - Validation_Accuracy: 0.744\n",
            "[8,   200] loss: 0.660 - Training_Accuracy: 0.776\n",
            "[8,   400] loss: 0.638 - Training_Accuracy: 0.778\n",
            "[8,   600] loss: 0.656 - Training_Accuracy: 0.771\n",
            "[8,   800] loss: 0.644 - Training_Accuracy: 0.778\n",
            "[8,  1000] loss: 0.626 - Training_Accuracy: 0.785\n",
            "[8,  1200] loss: 0.635 - Training_Accuracy: 0.780\n",
            "[8,  1400] loss: 0.625 - Training_Accuracy: 0.784\n",
            "[8,  1600] loss: 0.597 - Training_Accuracy: 0.791\n",
            "[8,  1800] loss: 0.658 - Training_Accuracy: 0.771\n",
            "[8,  2000] loss: 0.645 - Training_Accuracy: 0.778\n",
            "[8,  2000] Validation loss: 0.619 - Validation_Accuracy: 0.786\n",
            "[9,   200] loss: 0.642 - Training_Accuracy: 0.776\n",
            "[9,   400] loss: 0.626 - Training_Accuracy: 0.785\n",
            "[9,   600] loss: 0.622 - Training_Accuracy: 0.783\n",
            "[9,   800] loss: 0.626 - Training_Accuracy: 0.784\n",
            "[9,  1000] loss: 0.621 - Training_Accuracy: 0.786\n",
            "[9,  1200] loss: 0.622 - Training_Accuracy: 0.785\n",
            "[9,  1400] loss: 0.624 - Training_Accuracy: 0.784\n",
            "[9,  1600] loss: 0.633 - Training_Accuracy: 0.776\n",
            "[9,  1800] loss: 0.589 - Training_Accuracy: 0.794\n",
            "[9,  2000] loss: 0.571 - Training_Accuracy: 0.801\n",
            "[9,  2000] Validation loss: 0.699 - Validation_Accuracy: 0.758\n",
            "[10,   200] loss: 0.610 - Training_Accuracy: 0.786\n",
            "[10,   400] loss: 0.578 - Training_Accuracy: 0.802\n",
            "[10,   600] loss: 0.578 - Training_Accuracy: 0.791\n",
            "[10,   800] loss: 0.579 - Training_Accuracy: 0.793\n",
            "[10,  1000] loss: 0.607 - Training_Accuracy: 0.792\n",
            "[10,  1200] loss: 0.571 - Training_Accuracy: 0.803\n",
            "[10,  1400] loss: 0.620 - Training_Accuracy: 0.784\n",
            "[10,  1600] loss: 0.578 - Training_Accuracy: 0.801\n",
            "[10,  1800] loss: 0.578 - Training_Accuracy: 0.803\n",
            "[10,  2000] loss: 0.584 - Training_Accuracy: 0.796\n",
            "[10,  2000] Validation loss: 0.665 - Validation_Accuracy: 0.769\n",
            "[11,   200] loss: 0.561 - Training_Accuracy: 0.796\n",
            "[11,   400] loss: 0.552 - Training_Accuracy: 0.810\n",
            "[11,   600] loss: 0.539 - Training_Accuracy: 0.811\n",
            "[11,   800] loss: 0.544 - Training_Accuracy: 0.809\n",
            "[11,  1000] loss: 0.582 - Training_Accuracy: 0.800\n",
            "[11,  1200] loss: 0.540 - Training_Accuracy: 0.814\n",
            "[11,  1400] loss: 0.590 - Training_Accuracy: 0.790\n",
            "[11,  1600] loss: 0.601 - Training_Accuracy: 0.795\n",
            "[11,  1800] loss: 0.585 - Training_Accuracy: 0.797\n",
            "[11,  2000] loss: 0.566 - Training_Accuracy: 0.806\n",
            "[11,  2000] Validation loss: 0.625 - Validation_Accuracy: 0.785\n",
            "[12,   200] loss: 0.583 - Training_Accuracy: 0.796\n",
            "[12,   400] loss: 0.583 - Training_Accuracy: 0.803\n",
            "[12,   600] loss: 0.534 - Training_Accuracy: 0.816\n",
            "[12,   800] loss: 0.575 - Training_Accuracy: 0.800\n",
            "[12,  1000] loss: 0.542 - Training_Accuracy: 0.814\n",
            "[12,  1200] loss: 0.577 - Training_Accuracy: 0.804\n",
            "[12,  1400] loss: 0.561 - Training_Accuracy: 0.806\n",
            "[12,  1600] loss: 0.551 - Training_Accuracy: 0.810\n",
            "[12,  1800] loss: 0.554 - Training_Accuracy: 0.805\n",
            "[12,  2000] loss: 0.554 - Training_Accuracy: 0.806\n",
            "[12,  2000] Validation loss: 0.570 - Validation_Accuracy: 0.801\n",
            "[13,   200] loss: 0.550 - Training_Accuracy: 0.808\n",
            "[13,   400] loss: 0.531 - Training_Accuracy: 0.815\n",
            "[13,   600] loss: 0.543 - Training_Accuracy: 0.805\n",
            "[13,   800] loss: 0.533 - Training_Accuracy: 0.813\n",
            "[13,  1000] loss: 0.544 - Training_Accuracy: 0.808\n",
            "[13,  1200] loss: 0.547 - Training_Accuracy: 0.809\n",
            "[13,  1400] loss: 0.551 - Training_Accuracy: 0.812\n",
            "[13,  1600] loss: 0.516 - Training_Accuracy: 0.820\n",
            "[13,  1800] loss: 0.522 - Training_Accuracy: 0.818\n",
            "[13,  2000] loss: 0.544 - Training_Accuracy: 0.809\n",
            "[13,  2000] Validation loss: 0.545 - Validation_Accuracy: 0.813\n",
            "[14,   200] loss: 0.547 - Training_Accuracy: 0.809\n",
            "[14,   400] loss: 0.559 - Training_Accuracy: 0.803\n",
            "[14,   600] loss: 0.538 - Training_Accuracy: 0.812\n",
            "[14,   800] loss: 0.531 - Training_Accuracy: 0.814\n",
            "[14,  1000] loss: 0.535 - Training_Accuracy: 0.813\n",
            "[14,  1200] loss: 0.528 - Training_Accuracy: 0.814\n",
            "[14,  1400] loss: 0.527 - Training_Accuracy: 0.815\n",
            "[14,  1600] loss: 0.536 - Training_Accuracy: 0.813\n",
            "[14,  1800] loss: 0.531 - Training_Accuracy: 0.815\n",
            "[14,  2000] loss: 0.538 - Training_Accuracy: 0.813\n",
            "[14,  2000] Validation loss: 0.614 - Validation_Accuracy: 0.784\n",
            "[15,   200] loss: 0.532 - Training_Accuracy: 0.812\n",
            "[15,   400] loss: 0.534 - Training_Accuracy: 0.818\n",
            "[15,   600] loss: 0.522 - Training_Accuracy: 0.816\n",
            "[15,   800] loss: 0.525 - Training_Accuracy: 0.806\n",
            "[15,  1000] loss: 0.535 - Training_Accuracy: 0.820\n",
            "[15,  1200] loss: 0.513 - Training_Accuracy: 0.821\n",
            "[15,  1400] loss: 0.511 - Training_Accuracy: 0.817\n",
            "[15,  1600] loss: 0.526 - Training_Accuracy: 0.813\n",
            "[15,  1800] loss: 0.529 - Training_Accuracy: 0.818\n",
            "[15,  2000] loss: 0.515 - Training_Accuracy: 0.817\n",
            "[15,  2000] Validation loss: 0.557 - Validation_Accuracy: 0.806\n",
            "[16,   200] loss: 0.494 - Training_Accuracy: 0.824\n",
            "[16,   400] loss: 0.489 - Training_Accuracy: 0.825\n",
            "[16,   600] loss: 0.513 - Training_Accuracy: 0.819\n",
            "[16,   800] loss: 0.507 - Training_Accuracy: 0.826\n",
            "[16,  1000] loss: 0.474 - Training_Accuracy: 0.834\n",
            "[16,  1200] loss: 0.490 - Training_Accuracy: 0.835\n",
            "[16,  1400] loss: 0.512 - Training_Accuracy: 0.820\n",
            "[16,  1600] loss: 0.504 - Training_Accuracy: 0.824\n",
            "[16,  1800] loss: 0.493 - Training_Accuracy: 0.830\n",
            "[16,  2000] loss: 0.501 - Training_Accuracy: 0.825\n",
            "[16,  2000] Validation loss: 0.585 - Validation_Accuracy: 0.796\n",
            "[17,   200] loss: 0.512 - Training_Accuracy: 0.818\n",
            "[17,   400] loss: 0.506 - Training_Accuracy: 0.823\n",
            "[17,   600] loss: 0.504 - Training_Accuracy: 0.826\n",
            "[17,   800] loss: 0.483 - Training_Accuracy: 0.830\n",
            "[17,  1000] loss: 0.498 - Training_Accuracy: 0.831\n",
            "[17,  1200] loss: 0.498 - Training_Accuracy: 0.824\n",
            "[17,  1400] loss: 0.489 - Training_Accuracy: 0.827\n",
            "[17,  1600] loss: 0.516 - Training_Accuracy: 0.818\n",
            "[17,  1800] loss: 0.488 - Training_Accuracy: 0.829\n",
            "[17,  2000] loss: 0.499 - Training_Accuracy: 0.826\n",
            "[17,  2000] Validation loss: 0.506 - Validation_Accuracy: 0.823\n",
            "[18,   200] loss: 0.473 - Training_Accuracy: 0.840\n",
            "[18,   400] loss: 0.481 - Training_Accuracy: 0.831\n",
            "[18,   600] loss: 0.512 - Training_Accuracy: 0.821\n",
            "[18,   800] loss: 0.483 - Training_Accuracy: 0.828\n",
            "[18,  1000] loss: 0.478 - Training_Accuracy: 0.832\n",
            "[18,  1200] loss: 0.492 - Training_Accuracy: 0.827\n",
            "[18,  1400] loss: 0.484 - Training_Accuracy: 0.828\n",
            "[18,  1600] loss: 0.486 - Training_Accuracy: 0.829\n",
            "[18,  1800] loss: 0.495 - Training_Accuracy: 0.820\n",
            "[18,  2000] loss: 0.498 - Training_Accuracy: 0.825\n",
            "[18,  2000] Validation loss: 0.561 - Validation_Accuracy: 0.804\n",
            "[19,   200] loss: 0.473 - Training_Accuracy: 0.830\n",
            "[19,   400] loss: 0.478 - Training_Accuracy: 0.832\n",
            "[19,   600] loss: 0.486 - Training_Accuracy: 0.834\n",
            "[19,   800] loss: 0.469 - Training_Accuracy: 0.835\n",
            "[19,  1000] loss: 0.485 - Training_Accuracy: 0.836\n",
            "[19,  1200] loss: 0.470 - Training_Accuracy: 0.830\n",
            "[19,  1400] loss: 0.477 - Training_Accuracy: 0.834\n",
            "[19,  1600] loss: 0.492 - Training_Accuracy: 0.826\n",
            "[19,  1800] loss: 0.480 - Training_Accuracy: 0.835\n",
            "[19,  2000] loss: 0.496 - Training_Accuracy: 0.829\n",
            "[19,  2000] Validation loss: 0.524 - Validation_Accuracy: 0.821\n",
            "[20,   200] loss: 0.492 - Training_Accuracy: 0.833\n",
            "[20,   400] loss: 0.514 - Training_Accuracy: 0.813\n",
            "[20,   600] loss: 0.458 - Training_Accuracy: 0.836\n",
            "[20,   800] loss: 0.503 - Training_Accuracy: 0.823\n",
            "[20,  1000] loss: 0.455 - Training_Accuracy: 0.842\n",
            "[20,  1200] loss: 0.473 - Training_Accuracy: 0.833\n",
            "[20,  1400] loss: 0.503 - Training_Accuracy: 0.819\n",
            "[20,  1600] loss: 0.475 - Training_Accuracy: 0.831\n",
            "[20,  1800] loss: 0.472 - Training_Accuracy: 0.840\n",
            "[20,  2000] loss: 0.473 - Training_Accuracy: 0.826\n",
            "[20,  2000] Validation loss: 0.577 - Validation_Accuracy: 0.799\n",
            "[21,   200] loss: 0.477 - Training_Accuracy: 0.836\n",
            "[21,   400] loss: 0.476 - Training_Accuracy: 0.829\n",
            "[21,   600] loss: 0.489 - Training_Accuracy: 0.832\n",
            "[21,   800] loss: 0.463 - Training_Accuracy: 0.842\n",
            "[21,  1000] loss: 0.487 - Training_Accuracy: 0.828\n",
            "[21,  1200] loss: 0.457 - Training_Accuracy: 0.840\n",
            "[21,  1400] loss: 0.475 - Training_Accuracy: 0.836\n",
            "[21,  1600] loss: 0.458 - Training_Accuracy: 0.835\n",
            "[21,  1800] loss: 0.469 - Training_Accuracy: 0.835\n",
            "[21,  2000] loss: 0.499 - Training_Accuracy: 0.823\n",
            "[21,  2000] Validation loss: 0.532 - Validation_Accuracy: 0.815\n",
            "[22,   200] loss: 0.479 - Training_Accuracy: 0.832\n",
            "[22,   400] loss: 0.450 - Training_Accuracy: 0.834\n",
            "[22,   600] loss: 0.468 - Training_Accuracy: 0.838\n",
            "[22,   800] loss: 0.436 - Training_Accuracy: 0.843\n",
            "[22,  1000] loss: 0.454 - Training_Accuracy: 0.834\n",
            "[22,  1200] loss: 0.436 - Training_Accuracy: 0.842\n",
            "[22,  1400] loss: 0.458 - Training_Accuracy: 0.837\n",
            "[22,  1600] loss: 0.483 - Training_Accuracy: 0.827\n",
            "[22,  1800] loss: 0.478 - Training_Accuracy: 0.830\n",
            "[22,  2000] loss: 0.485 - Training_Accuracy: 0.829\n",
            "[22,  2000] Validation loss: 0.515 - Validation_Accuracy: 0.822\n",
            "[23,   200] loss: 0.470 - Training_Accuracy: 0.838\n",
            "[23,   400] loss: 0.460 - Training_Accuracy: 0.835\n",
            "[23,   600] loss: 0.465 - Training_Accuracy: 0.832\n",
            "[23,   800] loss: 0.459 - Training_Accuracy: 0.836\n",
            "[23,  1000] loss: 0.442 - Training_Accuracy: 0.844\n",
            "[23,  1200] loss: 0.478 - Training_Accuracy: 0.832\n",
            "[23,  1400] loss: 0.464 - Training_Accuracy: 0.834\n",
            "[23,  1600] loss: 0.444 - Training_Accuracy: 0.841\n",
            "[23,  1800] loss: 0.454 - Training_Accuracy: 0.841\n",
            "[23,  2000] loss: 0.486 - Training_Accuracy: 0.824\n",
            "[23,  2000] Validation loss: 0.515 - Validation_Accuracy: 0.819\n",
            "[24,   200] loss: 0.439 - Training_Accuracy: 0.838\n",
            "[24,   400] loss: 0.447 - Training_Accuracy: 0.841\n",
            "[24,   600] loss: 0.471 - Training_Accuracy: 0.839\n",
            "[24,   800] loss: 0.449 - Training_Accuracy: 0.840\n",
            "[24,  1000] loss: 0.426 - Training_Accuracy: 0.848\n",
            "[24,  1200] loss: 0.434 - Training_Accuracy: 0.843\n",
            "[24,  1400] loss: 0.430 - Training_Accuracy: 0.844\n",
            "[24,  1600] loss: 0.454 - Training_Accuracy: 0.840\n",
            "[24,  1800] loss: 0.457 - Training_Accuracy: 0.837\n",
            "[24,  2000] loss: 0.458 - Training_Accuracy: 0.836\n",
            "[24,  2000] Validation loss: 0.514 - Validation_Accuracy: 0.819\n",
            "[25,   200] loss: 0.432 - Training_Accuracy: 0.846\n",
            "[25,   400] loss: 0.419 - Training_Accuracy: 0.848\n",
            "[25,   600] loss: 0.435 - Training_Accuracy: 0.847\n",
            "[25,   800] loss: 0.423 - Training_Accuracy: 0.853\n",
            "[25,  1000] loss: 0.425 - Training_Accuracy: 0.844\n",
            "[25,  1200] loss: 0.424 - Training_Accuracy: 0.851\n",
            "[25,  1400] loss: 0.438 - Training_Accuracy: 0.851\n",
            "[25,  1600] loss: 0.422 - Training_Accuracy: 0.846\n",
            "[25,  1800] loss: 0.422 - Training_Accuracy: 0.847\n",
            "[25,  2000] loss: 0.443 - Training_Accuracy: 0.840\n",
            "[25,  2000] Validation loss: 0.468 - Validation_Accuracy: 0.835\n",
            "[26,   200] loss: 0.416 - Training_Accuracy: 0.851\n",
            "[26,   400] loss: 0.420 - Training_Accuracy: 0.847\n",
            "[26,   600] loss: 0.401 - Training_Accuracy: 0.855\n",
            "[26,   800] loss: 0.428 - Training_Accuracy: 0.847\n",
            "[26,  1000] loss: 0.435 - Training_Accuracy: 0.844\n",
            "[26,  1200] loss: 0.420 - Training_Accuracy: 0.846\n",
            "[26,  1400] loss: 0.404 - Training_Accuracy: 0.855\n",
            "[26,  1600] loss: 0.441 - Training_Accuracy: 0.842\n",
            "[26,  1800] loss: 0.442 - Training_Accuracy: 0.844\n",
            "[26,  2000] loss: 0.438 - Training_Accuracy: 0.845\n",
            "[26,  2000] Validation loss: 0.484 - Validation_Accuracy: 0.832\n",
            "[27,   200] loss: 0.383 - Training_Accuracy: 0.859\n",
            "[27,   400] loss: 0.419 - Training_Accuracy: 0.854\n",
            "[27,   600] loss: 0.428 - Training_Accuracy: 0.850\n",
            "[27,   800] loss: 0.451 - Training_Accuracy: 0.834\n",
            "[27,  1000] loss: 0.429 - Training_Accuracy: 0.847\n",
            "[27,  1200] loss: 0.471 - Training_Accuracy: 0.834\n",
            "[27,  1400] loss: 0.428 - Training_Accuracy: 0.846\n",
            "[27,  1600] loss: 0.444 - Training_Accuracy: 0.845\n",
            "[27,  1800] loss: 0.408 - Training_Accuracy: 0.851\n",
            "[27,  2000] loss: 0.400 - Training_Accuracy: 0.857\n",
            "[27,  2000] Validation loss: 0.484 - Validation_Accuracy: 0.830\n",
            "[28,   200] loss: 0.408 - Training_Accuracy: 0.858\n",
            "[28,   400] loss: 0.411 - Training_Accuracy: 0.852\n",
            "[28,   600] loss: 0.399 - Training_Accuracy: 0.854\n",
            "[28,   800] loss: 0.413 - Training_Accuracy: 0.851\n",
            "[28,  1000] loss: 0.406 - Training_Accuracy: 0.853\n",
            "[28,  1200] loss: 0.394 - Training_Accuracy: 0.856\n",
            "[28,  1400] loss: 0.430 - Training_Accuracy: 0.846\n",
            "[28,  1600] loss: 0.417 - Training_Accuracy: 0.846\n",
            "[28,  1800] loss: 0.417 - Training_Accuracy: 0.853\n",
            "[28,  2000] loss: 0.412 - Training_Accuracy: 0.852\n",
            "[28,  2000] Validation loss: 0.466 - Validation_Accuracy: 0.840\n",
            "[29,   200] loss: 0.415 - Training_Accuracy: 0.848\n",
            "[29,   400] loss: 0.398 - Training_Accuracy: 0.860\n",
            "[29,   600] loss: 0.417 - Training_Accuracy: 0.849\n",
            "[29,   800] loss: 0.399 - Training_Accuracy: 0.857\n",
            "[29,  1000] loss: 0.409 - Training_Accuracy: 0.855\n",
            "[29,  1200] loss: 0.383 - Training_Accuracy: 0.863\n",
            "[29,  1400] loss: 0.408 - Training_Accuracy: 0.855\n",
            "[29,  1600] loss: 0.417 - Training_Accuracy: 0.848\n",
            "[29,  1800] loss: 0.431 - Training_Accuracy: 0.846\n",
            "[29,  2000] loss: 0.414 - Training_Accuracy: 0.849\n",
            "[29,  2000] Validation loss: 0.466 - Validation_Accuracy: 0.835\n",
            "[30,   200] loss: 0.389 - Training_Accuracy: 0.862\n",
            "[30,   400] loss: 0.402 - Training_Accuracy: 0.857\n",
            "[30,   600] loss: 0.400 - Training_Accuracy: 0.857\n",
            "[30,   800] loss: 0.411 - Training_Accuracy: 0.852\n",
            "[30,  1000] loss: 0.446 - Training_Accuracy: 0.843\n",
            "[30,  1200] loss: 0.399 - Training_Accuracy: 0.852\n",
            "[30,  1400] loss: 0.413 - Training_Accuracy: 0.848\n",
            "[30,  1600] loss: 0.431 - Training_Accuracy: 0.846\n",
            "[30,  1800] loss: 0.425 - Training_Accuracy: 0.851\n",
            "[30,  2000] loss: 0.441 - Training_Accuracy: 0.843\n",
            "[30,  2000] Validation loss: 0.475 - Validation_Accuracy: 0.831\n",
            "[31,   200] loss: 0.411 - Training_Accuracy: 0.851\n",
            "[31,   400] loss: 0.406 - Training_Accuracy: 0.854\n",
            "[31,   600] loss: 0.398 - Training_Accuracy: 0.857\n",
            "[31,   800] loss: 0.397 - Training_Accuracy: 0.857\n",
            "[31,  1000] loss: 0.400 - Training_Accuracy: 0.858\n",
            "[31,  1200] loss: 0.417 - Training_Accuracy: 0.852\n",
            "[31,  1400] loss: 0.406 - Training_Accuracy: 0.855\n",
            "[31,  1600] loss: 0.393 - Training_Accuracy: 0.857\n",
            "[31,  1800] loss: 0.419 - Training_Accuracy: 0.853\n",
            "[31,  2000] loss: 0.388 - Training_Accuracy: 0.861\n",
            "[31,  2000] Validation loss: 0.499 - Validation_Accuracy: 0.826\n",
            "[32,   200] loss: 0.400 - Training_Accuracy: 0.853\n",
            "[32,   400] loss: 0.417 - Training_Accuracy: 0.848\n",
            "[32,   600] loss: 0.393 - Training_Accuracy: 0.859\n",
            "[32,   800] loss: 0.404 - Training_Accuracy: 0.859\n",
            "[32,  1000] loss: 0.411 - Training_Accuracy: 0.854\n",
            "[32,  1200] loss: 0.409 - Training_Accuracy: 0.854\n",
            "[32,  1400] loss: 0.377 - Training_Accuracy: 0.868\n",
            "[32,  1600] loss: 0.407 - Training_Accuracy: 0.849\n",
            "[32,  1800] loss: 0.374 - Training_Accuracy: 0.867\n",
            "[32,  2000] loss: 0.395 - Training_Accuracy: 0.859\n",
            "[32,  2000] Validation loss: 0.443 - Validation_Accuracy: 0.844\n",
            "[33,   200] loss: 0.382 - Training_Accuracy: 0.862\n",
            "[33,   400] loss: 0.399 - Training_Accuracy: 0.852\n",
            "[33,   600] loss: 0.397 - Training_Accuracy: 0.859\n",
            "[33,   800] loss: 0.397 - Training_Accuracy: 0.853\n",
            "[33,  1000] loss: 0.393 - Training_Accuracy: 0.860\n",
            "[33,  1200] loss: 0.379 - Training_Accuracy: 0.860\n",
            "[33,  1400] loss: 0.392 - Training_Accuracy: 0.856\n",
            "[33,  1600] loss: 0.376 - Training_Accuracy: 0.860\n",
            "[33,  1800] loss: 0.383 - Training_Accuracy: 0.863\n",
            "[33,  2000] loss: 0.411 - Training_Accuracy: 0.850\n",
            "[33,  2000] Validation loss: 0.448 - Validation_Accuracy: 0.842\n",
            "[34,   200] loss: 0.369 - Training_Accuracy: 0.865\n",
            "[34,   400] loss: 0.390 - Training_Accuracy: 0.859\n",
            "[34,   600] loss: 0.421 - Training_Accuracy: 0.851\n",
            "[34,   800] loss: 0.417 - Training_Accuracy: 0.849\n",
            "[34,  1000] loss: 0.410 - Training_Accuracy: 0.849\n",
            "[34,  1200] loss: 0.417 - Training_Accuracy: 0.855\n",
            "[34,  1400] loss: 0.401 - Training_Accuracy: 0.858\n",
            "[34,  1600] loss: 0.392 - Training_Accuracy: 0.861\n",
            "[34,  1800] loss: 0.411 - Training_Accuracy: 0.854\n",
            "[34,  2000] loss: 0.408 - Training_Accuracy: 0.850\n",
            "[34,  2000] Validation loss: 0.525 - Validation_Accuracy: 0.817\n",
            "[35,   200] loss: 0.421 - Training_Accuracy: 0.848\n",
            "[35,   400] loss: 0.395 - Training_Accuracy: 0.852\n",
            "[35,   600] loss: 0.380 - Training_Accuracy: 0.860\n",
            "[35,   800] loss: 0.403 - Training_Accuracy: 0.853\n",
            "[35,  1000] loss: 0.395 - Training_Accuracy: 0.853\n",
            "[35,  1200] loss: 0.387 - Training_Accuracy: 0.865\n",
            "[35,  1400] loss: 0.389 - Training_Accuracy: 0.861\n",
            "[35,  1600] loss: 0.375 - Training_Accuracy: 0.864\n",
            "[35,  1800] loss: 0.409 - Training_Accuracy: 0.852\n",
            "[35,  2000] loss: 0.421 - Training_Accuracy: 0.840\n",
            "[35,  2000] Validation loss: 0.480 - Validation_Accuracy: 0.835\n",
            "[36,   200] loss: 0.409 - Training_Accuracy: 0.850\n",
            "[36,   400] loss: 0.414 - Training_Accuracy: 0.850\n",
            "[36,   600] loss: 0.397 - Training_Accuracy: 0.853\n",
            "[36,   800] loss: 0.410 - Training_Accuracy: 0.850\n",
            "[36,  1000] loss: 0.376 - Training_Accuracy: 0.864\n",
            "[36,  1200] loss: 0.401 - Training_Accuracy: 0.856\n",
            "[36,  1400] loss: 0.401 - Training_Accuracy: 0.858\n",
            "[36,  1600] loss: 0.361 - Training_Accuracy: 0.867\n",
            "[36,  1800] loss: 0.377 - Training_Accuracy: 0.859\n",
            "[36,  2000] loss: 0.389 - Training_Accuracy: 0.858\n",
            "[36,  2000] Validation loss: 0.437 - Validation_Accuracy: 0.844\n",
            "[37,   200] loss: 0.365 - Training_Accuracy: 0.868\n",
            "[37,   400] loss: 0.393 - Training_Accuracy: 0.858\n",
            "[37,   600] loss: 0.372 - Training_Accuracy: 0.868\n",
            "[37,   800] loss: 0.361 - Training_Accuracy: 0.869\n",
            "[37,  1000] loss: 0.394 - Training_Accuracy: 0.859\n",
            "[37,  1200] loss: 0.376 - Training_Accuracy: 0.866\n",
            "[37,  1400] loss: 0.394 - Training_Accuracy: 0.854\n",
            "[37,  1600] loss: 0.361 - Training_Accuracy: 0.865\n",
            "[37,  1800] loss: 0.379 - Training_Accuracy: 0.860\n",
            "[37,  2000] loss: 0.369 - Training_Accuracy: 0.862\n",
            "[37,  2000] Validation loss: 0.429 - Validation_Accuracy: 0.850\n",
            "[38,   200] loss: 0.365 - Training_Accuracy: 0.864\n",
            "[38,   400] loss: 0.357 - Training_Accuracy: 0.873\n",
            "[38,   600] loss: 0.352 - Training_Accuracy: 0.873\n",
            "[38,   800] loss: 0.378 - Training_Accuracy: 0.862\n",
            "[38,  1000] loss: 0.350 - Training_Accuracy: 0.869\n",
            "[38,  1200] loss: 0.342 - Training_Accuracy: 0.875\n",
            "[38,  1400] loss: 0.358 - Training_Accuracy: 0.868\n",
            "[38,  1600] loss: 0.369 - Training_Accuracy: 0.868\n",
            "[38,  1800] loss: 0.395 - Training_Accuracy: 0.859\n",
            "[38,  2000] loss: 0.355 - Training_Accuracy: 0.874\n",
            "[38,  2000] Validation loss: 0.466 - Validation_Accuracy: 0.836\n",
            "[39,   200] loss: 0.372 - Training_Accuracy: 0.866\n",
            "[39,   400] loss: 0.353 - Training_Accuracy: 0.875\n",
            "[39,   600] loss: 0.343 - Training_Accuracy: 0.875\n",
            "[39,   800] loss: 0.383 - Training_Accuracy: 0.866\n",
            "[39,  1000] loss: 0.400 - Training_Accuracy: 0.851\n",
            "[39,  1200] loss: 0.376 - Training_Accuracy: 0.863\n",
            "[39,  1400] loss: 0.381 - Training_Accuracy: 0.861\n",
            "[39,  1600] loss: 0.368 - Training_Accuracy: 0.863\n",
            "[39,  1800] loss: 0.382 - Training_Accuracy: 0.865\n",
            "[39,  2000] loss: 0.376 - Training_Accuracy: 0.858\n",
            "[39,  2000] Validation loss: 0.479 - Validation_Accuracy: 0.828\n",
            "[40,   200] loss: 0.386 - Training_Accuracy: 0.854\n",
            "[40,   400] loss: 0.376 - Training_Accuracy: 0.863\n",
            "[40,   600] loss: 0.401 - Training_Accuracy: 0.855\n",
            "[40,   800] loss: 0.416 - Training_Accuracy: 0.847\n",
            "[40,  1000] loss: 0.393 - Training_Accuracy: 0.860\n",
            "[40,  1200] loss: 0.384 - Training_Accuracy: 0.857\n",
            "[40,  1400] loss: 0.363 - Training_Accuracy: 0.866\n",
            "[40,  1600] loss: 0.397 - Training_Accuracy: 0.859\n",
            "[40,  1800] loss: 0.373 - Training_Accuracy: 0.868\n",
            "[40,  2000] loss: 0.367 - Training_Accuracy: 0.870\n",
            "[40,  2000] Validation loss: 0.462 - Validation_Accuracy: 0.839\n",
            "[41,   200] loss: 0.366 - Training_Accuracy: 0.866\n",
            "[41,   400] loss: 0.360 - Training_Accuracy: 0.867\n",
            "[41,   600] loss: 0.400 - Training_Accuracy: 0.852\n",
            "[41,   800] loss: 0.369 - Training_Accuracy: 0.866\n",
            "[41,  1000] loss: 0.399 - Training_Accuracy: 0.856\n",
            "[41,  1200] loss: 0.397 - Training_Accuracy: 0.858\n",
            "[41,  1400] loss: 0.406 - Training_Accuracy: 0.851\n",
            "[41,  1600] loss: 0.406 - Training_Accuracy: 0.851\n",
            "[41,  1800] loss: 0.418 - Training_Accuracy: 0.844\n",
            "[41,  2000] loss: 0.394 - Training_Accuracy: 0.856\n",
            "[41,  2000] Validation loss: 0.495 - Validation_Accuracy: 0.826\n",
            "[42,   200] loss: 0.368 - Training_Accuracy: 0.863\n",
            "[42,   400] loss: 0.358 - Training_Accuracy: 0.870\n",
            "[42,   600] loss: 0.361 - Training_Accuracy: 0.868\n",
            "[42,   800] loss: 0.398 - Training_Accuracy: 0.850\n",
            "[42,  1000] loss: 0.381 - Training_Accuracy: 0.859\n",
            "[42,  1200] loss: 0.401 - Training_Accuracy: 0.855\n",
            "[42,  1400] loss: 0.425 - Training_Accuracy: 0.851\n",
            "[42,  1600] loss: 0.404 - Training_Accuracy: 0.854\n",
            "[42,  1800] loss: 0.394 - Training_Accuracy: 0.855\n",
            "[42,  2000] loss: 0.419 - Training_Accuracy: 0.851\n",
            "[42,  2000] Validation loss: 0.454 - Validation_Accuracy: 0.843\n",
            "[43,   200] loss: 0.356 - Training_Accuracy: 0.869\n",
            "[43,   400] loss: 0.356 - Training_Accuracy: 0.872\n",
            "[43,   600] loss: 0.362 - Training_Accuracy: 0.867\n",
            "[43,   800] loss: 0.372 - Training_Accuracy: 0.865\n",
            "[43,  1000] loss: 0.396 - Training_Accuracy: 0.856\n",
            "[43,  1200] loss: 0.390 - Training_Accuracy: 0.861\n",
            "[43,  1400] loss: 0.363 - Training_Accuracy: 0.870\n",
            "[43,  1600] loss: 0.363 - Training_Accuracy: 0.869\n",
            "[43,  1800] loss: 0.328 - Training_Accuracy: 0.878\n",
            "[43,  2000] loss: 0.363 - Training_Accuracy: 0.861\n",
            "[43,  2000] Validation loss: 0.463 - Validation_Accuracy: 0.838\n",
            "[44,   200] loss: 0.370 - Training_Accuracy: 0.866\n",
            "[44,   400] loss: 0.357 - Training_Accuracy: 0.872\n",
            "[44,   600] loss: 0.356 - Training_Accuracy: 0.867\n",
            "[44,   800] loss: 0.340 - Training_Accuracy: 0.877\n",
            "[44,  1000] loss: 0.365 - Training_Accuracy: 0.862\n",
            "[44,  1200] loss: 0.352 - Training_Accuracy: 0.871\n",
            "[44,  1400] loss: 0.363 - Training_Accuracy: 0.865\n",
            "[44,  1600] loss: 0.376 - Training_Accuracy: 0.860\n",
            "[44,  1800] loss: 0.359 - Training_Accuracy: 0.867\n",
            "[44,  2000] loss: 0.364 - Training_Accuracy: 0.869\n",
            "[44,  2000] Validation loss: 0.447 - Validation_Accuracy: 0.843\n",
            "[45,   200] loss: 0.359 - Training_Accuracy: 0.862\n",
            "[45,   400] loss: 0.344 - Training_Accuracy: 0.873\n",
            "[45,   600] loss: 0.349 - Training_Accuracy: 0.874\n",
            "[45,   800] loss: 0.371 - Training_Accuracy: 0.862\n",
            "[45,  1000] loss: 0.368 - Training_Accuracy: 0.867\n",
            "[45,  1200] loss: 0.400 - Training_Accuracy: 0.856\n",
            "[45,  1400] loss: 0.360 - Training_Accuracy: 0.872\n",
            "[45,  1600] loss: 0.335 - Training_Accuracy: 0.875\n",
            "[45,  1800] loss: 0.348 - Training_Accuracy: 0.871\n",
            "[45,  2000] loss: 0.393 - Training_Accuracy: 0.858\n",
            "[45,  2000] Validation loss: 0.472 - Validation_Accuracy: 0.835\n",
            "[46,   200] loss: 0.335 - Training_Accuracy: 0.874\n",
            "[46,   400] loss: 0.366 - Training_Accuracy: 0.866\n",
            "[46,   600] loss: 0.376 - Training_Accuracy: 0.863\n",
            "[46,   800] loss: 0.363 - Training_Accuracy: 0.870\n",
            "[46,  1000] loss: 0.401 - Training_Accuracy: 0.849\n",
            "[46,  1200] loss: 0.370 - Training_Accuracy: 0.868\n",
            "[46,  1400] loss: 0.376 - Training_Accuracy: 0.860\n",
            "[46,  1600] loss: 0.376 - Training_Accuracy: 0.865\n",
            "[46,  1800] loss: 0.373 - Training_Accuracy: 0.862\n",
            "[46,  2000] loss: 0.343 - Training_Accuracy: 0.873\n",
            "[46,  2000] Validation loss: 0.464 - Validation_Accuracy: 0.840\n",
            "[47,   200] loss: 0.351 - Training_Accuracy: 0.870\n",
            "[47,   400] loss: 0.338 - Training_Accuracy: 0.875\n",
            "[47,   600] loss: 0.366 - Training_Accuracy: 0.866\n",
            "[47,   800] loss: 0.361 - Training_Accuracy: 0.867\n",
            "[47,  1000] loss: 0.336 - Training_Accuracy: 0.877\n",
            "[47,  1200] loss: 0.350 - Training_Accuracy: 0.875\n",
            "[47,  1400] loss: 0.375 - Training_Accuracy: 0.857\n",
            "[47,  1600] loss: 0.340 - Training_Accuracy: 0.869\n",
            "[47,  1800] loss: 0.352 - Training_Accuracy: 0.872\n",
            "[47,  2000] loss: 0.362 - Training_Accuracy: 0.863\n",
            "[47,  2000] Validation loss: 0.456 - Validation_Accuracy: 0.842\n",
            "[48,   200] loss: 0.367 - Training_Accuracy: 0.861\n",
            "[48,   400] loss: 0.346 - Training_Accuracy: 0.877\n",
            "[48,   600] loss: 0.348 - Training_Accuracy: 0.870\n",
            "[48,   800] loss: 0.345 - Training_Accuracy: 0.876\n",
            "[48,  1000] loss: 0.343 - Training_Accuracy: 0.875\n",
            "[48,  1200] loss: 0.385 - Training_Accuracy: 0.857\n",
            "[48,  1400] loss: 0.370 - Training_Accuracy: 0.863\n",
            "[48,  1600] loss: 0.355 - Training_Accuracy: 0.865\n",
            "[48,  1800] loss: 0.333 - Training_Accuracy: 0.879\n",
            "[48,  2000] loss: 0.358 - Training_Accuracy: 0.870\n",
            "[48,  2000] Validation loss: 0.508 - Validation_Accuracy: 0.825\n",
            "[49,   200] loss: 0.332 - Training_Accuracy: 0.880\n",
            "[49,   400] loss: 0.376 - Training_Accuracy: 0.868\n",
            "[49,   600] loss: 0.366 - Training_Accuracy: 0.860\n",
            "[49,   800] loss: 0.367 - Training_Accuracy: 0.862\n",
            "[49,  1000] loss: 0.384 - Training_Accuracy: 0.860\n",
            "[49,  1200] loss: 0.380 - Training_Accuracy: 0.856\n",
            "[49,  1400] loss: 0.370 - Training_Accuracy: 0.866\n",
            "[49,  1600] loss: 0.351 - Training_Accuracy: 0.869\n",
            "[49,  1800] loss: 0.365 - Training_Accuracy: 0.869\n",
            "[49,  2000] loss: 0.378 - Training_Accuracy: 0.864\n",
            "[49,  2000] Validation loss: 0.487 - Validation_Accuracy: 0.833\n",
            "[50,   200] loss: 0.333 - Training_Accuracy: 0.880\n",
            "[50,   400] loss: 0.367 - Training_Accuracy: 0.863\n",
            "[50,   600] loss: 0.332 - Training_Accuracy: 0.877\n",
            "[50,   800] loss: 0.358 - Training_Accuracy: 0.870\n",
            "[50,  1000] loss: 0.348 - Training_Accuracy: 0.870\n",
            "[50,  1200] loss: 0.353 - Training_Accuracy: 0.870\n",
            "[50,  1400] loss: 0.341 - Training_Accuracy: 0.874\n",
            "[50,  1600] loss: 0.341 - Training_Accuracy: 0.874\n",
            "[50,  1800] loss: 0.316 - Training_Accuracy: 0.883\n",
            "[50,  2000] loss: 0.301 - Training_Accuracy: 0.883\n",
            "[50,  2000] Validation loss: 0.448 - Validation_Accuracy: 0.846\n",
            "[51,   200] loss: 0.332 - Training_Accuracy: 0.878\n",
            "[51,   400] loss: 0.334 - Training_Accuracy: 0.875\n",
            "[51,   600] loss: 0.344 - Training_Accuracy: 0.870\n",
            "[51,   800] loss: 0.330 - Training_Accuracy: 0.877\n",
            "[51,  1000] loss: 0.353 - Training_Accuracy: 0.871\n",
            "[51,  1200] loss: 0.343 - Training_Accuracy: 0.876\n",
            "[51,  1400] loss: 0.329 - Training_Accuracy: 0.873\n",
            "[51,  1600] loss: 0.353 - Training_Accuracy: 0.864\n",
            "[51,  1800] loss: 0.326 - Training_Accuracy: 0.882\n",
            "[51,  2000] loss: 0.310 - Training_Accuracy: 0.884\n",
            "[51,  2000] Validation loss: 0.438 - Validation_Accuracy: 0.849\n",
            "[52,   200] loss: 0.302 - Training_Accuracy: 0.888\n",
            "[52,   400] loss: 0.317 - Training_Accuracy: 0.880\n",
            "[52,   600] loss: 0.327 - Training_Accuracy: 0.879\n",
            "[52,   800] loss: 0.318 - Training_Accuracy: 0.889\n",
            "[52,  1000] loss: 0.314 - Training_Accuracy: 0.880\n",
            "[52,  1200] loss: 0.332 - Training_Accuracy: 0.879\n",
            "[52,  1400] loss: 0.324 - Training_Accuracy: 0.883\n",
            "[52,  1600] loss: 0.327 - Training_Accuracy: 0.878\n",
            "[52,  1800] loss: 0.315 - Training_Accuracy: 0.879\n",
            "[52,  2000] loss: 0.334 - Training_Accuracy: 0.872\n",
            "[52,  2000] Validation loss: 0.443 - Validation_Accuracy: 0.850\n",
            "[53,   200] loss: 0.355 - Training_Accuracy: 0.872\n",
            "[53,   400] loss: 0.331 - Training_Accuracy: 0.877\n",
            "[53,   600] loss: 0.326 - Training_Accuracy: 0.882\n",
            "[53,   800] loss: 0.308 - Training_Accuracy: 0.884\n",
            "[53,  1000] loss: 0.327 - Training_Accuracy: 0.878\n",
            "[53,  1200] loss: 0.318 - Training_Accuracy: 0.884\n",
            "[53,  1400] loss: 0.338 - Training_Accuracy: 0.875\n",
            "[53,  1600] loss: 0.336 - Training_Accuracy: 0.873\n",
            "[53,  1800] loss: 0.328 - Training_Accuracy: 0.879\n",
            "[53,  2000] loss: 0.332 - Training_Accuracy: 0.879\n",
            "[53,  2000] Validation loss: 0.452 - Validation_Accuracy: 0.845\n",
            "[54,   200] loss: 0.314 - Training_Accuracy: 0.884\n",
            "[54,   400] loss: 0.308 - Training_Accuracy: 0.884\n",
            "[54,   600] loss: 0.327 - Training_Accuracy: 0.877\n",
            "[54,   800] loss: 0.307 - Training_Accuracy: 0.883\n",
            "[54,  1000] loss: 0.339 - Training_Accuracy: 0.879\n",
            "[54,  1200] loss: 0.299 - Training_Accuracy: 0.891\n",
            "[54,  1400] loss: 0.330 - Training_Accuracy: 0.879\n",
            "[54,  1600] loss: 0.342 - Training_Accuracy: 0.873\n",
            "[54,  1800] loss: 0.334 - Training_Accuracy: 0.876\n",
            "[54,  2000] loss: 0.348 - Training_Accuracy: 0.866\n",
            "[54,  2000] Validation loss: 0.470 - Validation_Accuracy: 0.839\n",
            "[55,   200] loss: 0.317 - Training_Accuracy: 0.878\n",
            "[55,   400] loss: 0.307 - Training_Accuracy: 0.880\n",
            "[55,   600] loss: 0.309 - Training_Accuracy: 0.884\n",
            "[55,   800] loss: 0.311 - Training_Accuracy: 0.880\n",
            "[55,  1000] loss: 0.324 - Training_Accuracy: 0.880\n",
            "[55,  1200] loss: 0.341 - Training_Accuracy: 0.870\n",
            "[55,  1400] loss: 0.339 - Training_Accuracy: 0.871\n",
            "[55,  1600] loss: 0.332 - Training_Accuracy: 0.873\n",
            "[55,  1800] loss: 0.332 - Training_Accuracy: 0.876\n",
            "[55,  2000] loss: 0.356 - Training_Accuracy: 0.869\n",
            "[55,  2000] Validation loss: 0.474 - Validation_Accuracy: 0.839\n",
            "[56,   200] loss: 0.323 - Training_Accuracy: 0.879\n",
            "[56,   400] loss: 0.322 - Training_Accuracy: 0.876\n",
            "[56,   600] loss: 0.320 - Training_Accuracy: 0.880\n",
            "[56,   800] loss: 0.327 - Training_Accuracy: 0.878\n",
            "[56,  1000] loss: 0.320 - Training_Accuracy: 0.877\n",
            "[56,  1200] loss: 0.314 - Training_Accuracy: 0.885\n",
            "[56,  1400] loss: 0.322 - Training_Accuracy: 0.875\n",
            "[56,  1600] loss: 0.325 - Training_Accuracy: 0.878\n",
            "[56,  1800] loss: 0.315 - Training_Accuracy: 0.885\n",
            "[56,  2000] loss: 0.339 - Training_Accuracy: 0.874\n",
            "[56,  2000] Validation loss: 0.474 - Validation_Accuracy: 0.839\n",
            "[57,   200] loss: 0.322 - Training_Accuracy: 0.878\n",
            "[57,   400] loss: 0.334 - Training_Accuracy: 0.871\n",
            "[57,   600] loss: 0.308 - Training_Accuracy: 0.889\n",
            "[57,   800] loss: 0.330 - Training_Accuracy: 0.876\n",
            "[57,  1000] loss: 0.309 - Training_Accuracy: 0.882\n",
            "[57,  1200] loss: 0.322 - Training_Accuracy: 0.876\n",
            "[57,  1400] loss: 0.324 - Training_Accuracy: 0.878\n",
            "[57,  1600] loss: 0.342 - Training_Accuracy: 0.871\n",
            "[57,  1800] loss: 0.302 - Training_Accuracy: 0.882\n",
            "[57,  2000] loss: 0.317 - Training_Accuracy: 0.879\n",
            "[57,  2000] Validation loss: 0.477 - Validation_Accuracy: 0.839\n",
            "[58,   200] loss: 0.328 - Training_Accuracy: 0.874\n",
            "[58,   400] loss: 0.322 - Training_Accuracy: 0.879\n",
            "[58,   600] loss: 0.329 - Training_Accuracy: 0.871\n",
            "[58,   800] loss: 0.339 - Training_Accuracy: 0.871\n",
            "[58,  1000] loss: 0.305 - Training_Accuracy: 0.885\n",
            "[58,  1200] loss: 0.320 - Training_Accuracy: 0.883\n",
            "[58,  1400] loss: 0.309 - Training_Accuracy: 0.883\n",
            "[58,  1600] loss: 0.310 - Training_Accuracy: 0.886\n",
            "[58,  1800] loss: 0.291 - Training_Accuracy: 0.888\n",
            "[58,  2000] loss: 0.302 - Training_Accuracy: 0.889\n",
            "[58,  2000] Validation loss: 0.451 - Validation_Accuracy: 0.850\n",
            "[59,   200] loss: 0.315 - Training_Accuracy: 0.883\n",
            "[59,   400] loss: 0.300 - Training_Accuracy: 0.890\n",
            "[59,   600] loss: 0.312 - Training_Accuracy: 0.883\n",
            "[59,   800] loss: 0.322 - Training_Accuracy: 0.882\n",
            "[59,  1000] loss: 0.354 - Training_Accuracy: 0.871\n",
            "[59,  1200] loss: 0.377 - Training_Accuracy: 0.859\n",
            "[59,  1400] loss: 0.355 - Training_Accuracy: 0.869\n",
            "[59,  1600] loss: 0.325 - Training_Accuracy: 0.878\n",
            "[59,  1800] loss: 0.324 - Training_Accuracy: 0.883\n",
            "[59,  2000] loss: 0.329 - Training_Accuracy: 0.877\n",
            "[59,  2000] Validation loss: 0.449 - Validation_Accuracy: 0.848\n",
            "[60,   200] loss: 0.313 - Training_Accuracy: 0.882\n",
            "[60,   400] loss: 0.340 - Training_Accuracy: 0.875\n",
            "[60,   600] loss: 0.308 - Training_Accuracy: 0.880\n",
            "[60,   800] loss: 0.314 - Training_Accuracy: 0.881\n",
            "[60,  1000] loss: 0.315 - Training_Accuracy: 0.880\n",
            "[60,  1200] loss: 0.355 - Training_Accuracy: 0.874\n",
            "[60,  1400] loss: 0.326 - Training_Accuracy: 0.879\n",
            "[60,  1600] loss: 0.304 - Training_Accuracy: 0.886\n",
            "[60,  1800] loss: 0.294 - Training_Accuracy: 0.889\n",
            "[60,  2000] loss: 0.314 - Training_Accuracy: 0.886\n",
            "[60,  2000] Validation loss: 0.459 - Validation_Accuracy: 0.846\n",
            "[61,   200] loss: 0.293 - Training_Accuracy: 0.890\n",
            "[61,   400] loss: 0.290 - Training_Accuracy: 0.889\n",
            "[61,   600] loss: 0.309 - Training_Accuracy: 0.885\n",
            "[61,   800] loss: 0.300 - Training_Accuracy: 0.883\n",
            "[61,  1000] loss: 0.312 - Training_Accuracy: 0.881\n",
            "[61,  1200] loss: 0.316 - Training_Accuracy: 0.883\n",
            "[61,  1400] loss: 0.312 - Training_Accuracy: 0.885\n",
            "[61,  1600] loss: 0.307 - Training_Accuracy: 0.886\n",
            "[61,  1800] loss: 0.319 - Training_Accuracy: 0.879\n",
            "[61,  2000] loss: 0.320 - Training_Accuracy: 0.879\n",
            "[61,  2000] Validation loss: 0.454 - Validation_Accuracy: 0.846\n",
            "[62,   200] loss: 0.296 - Training_Accuracy: 0.892\n",
            "[62,   400] loss: 0.297 - Training_Accuracy: 0.890\n",
            "[62,   600] loss: 0.301 - Training_Accuracy: 0.886\n",
            "[62,   800] loss: 0.298 - Training_Accuracy: 0.888\n",
            "[62,  1000] loss: 0.289 - Training_Accuracy: 0.893\n",
            "[62,  1200] loss: 0.333 - Training_Accuracy: 0.876\n",
            "[62,  1400] loss: 0.337 - Training_Accuracy: 0.876\n",
            "[62,  1600] loss: 0.312 - Training_Accuracy: 0.877\n",
            "[62,  1800] loss: 0.304 - Training_Accuracy: 0.887\n",
            "[62,  2000] loss: 0.300 - Training_Accuracy: 0.890\n",
            "[62,  2000] Validation loss: 0.457 - Validation_Accuracy: 0.846\n",
            "[63,   200] loss: 0.302 - Training_Accuracy: 0.889\n",
            "[63,   400] loss: 0.306 - Training_Accuracy: 0.885\n",
            "[63,   600] loss: 0.287 - Training_Accuracy: 0.892\n",
            "[63,   800] loss: 0.307 - Training_Accuracy: 0.885\n",
            "[63,  1000] loss: 0.284 - Training_Accuracy: 0.889\n",
            "[63,  1200] loss: 0.325 - Training_Accuracy: 0.880\n",
            "[63,  1400] loss: 0.312 - Training_Accuracy: 0.883\n",
            "[63,  1600] loss: 0.314 - Training_Accuracy: 0.880\n",
            "[63,  1800] loss: 0.288 - Training_Accuracy: 0.891\n",
            "[63,  2000] loss: 0.291 - Training_Accuracy: 0.887\n",
            "[63,  2000] Validation loss: 0.479 - Validation_Accuracy: 0.838\n",
            "[64,   200] loss: 0.340 - Training_Accuracy: 0.875\n",
            "[64,   400] loss: 0.295 - Training_Accuracy: 0.887\n",
            "[64,   600] loss: 0.285 - Training_Accuracy: 0.891\n",
            "[64,   800] loss: 0.297 - Training_Accuracy: 0.887\n",
            "[64,  1000] loss: 0.288 - Training_Accuracy: 0.892\n",
            "[64,  1200] loss: 0.282 - Training_Accuracy: 0.893\n",
            "[64,  1400] loss: 0.274 - Training_Accuracy: 0.897\n",
            "[64,  1600] loss: 0.285 - Training_Accuracy: 0.888\n",
            "[64,  1800] loss: 0.313 - Training_Accuracy: 0.881\n",
            "[64,  2000] loss: 0.309 - Training_Accuracy: 0.887\n",
            "[64,  2000] Validation loss: 0.457 - Validation_Accuracy: 0.847\n",
            "[65,   200] loss: 0.291 - Training_Accuracy: 0.891\n",
            "[65,   400] loss: 0.272 - Training_Accuracy: 0.890\n",
            "[65,   600] loss: 0.277 - Training_Accuracy: 0.895\n",
            "[65,   800] loss: 0.285 - Training_Accuracy: 0.892\n",
            "[65,  1000] loss: 0.305 - Training_Accuracy: 0.886\n",
            "[65,  1200] loss: 0.300 - Training_Accuracy: 0.888\n",
            "[65,  1400] loss: 0.267 - Training_Accuracy: 0.899\n",
            "[65,  1600] loss: 0.289 - Training_Accuracy: 0.890\n",
            "[65,  1800] loss: 0.277 - Training_Accuracy: 0.897\n",
            "[65,  2000] loss: 0.309 - Training_Accuracy: 0.885\n",
            "[65,  2000] Validation loss: 0.460 - Validation_Accuracy: 0.847\n",
            "[66,   200] loss: 0.274 - Training_Accuracy: 0.899\n",
            "[66,   400] loss: 0.301 - Training_Accuracy: 0.882\n",
            "[66,   600] loss: 0.280 - Training_Accuracy: 0.894\n",
            "[66,   800] loss: 0.276 - Training_Accuracy: 0.892\n",
            "[66,  1000] loss: 0.279 - Training_Accuracy: 0.893\n",
            "[66,  1200] loss: 0.271 - Training_Accuracy: 0.896\n",
            "[66,  1400] loss: 0.263 - Training_Accuracy: 0.900\n",
            "[66,  1600] loss: 0.262 - Training_Accuracy: 0.902\n",
            "[66,  1800] loss: 0.284 - Training_Accuracy: 0.887\n",
            "[66,  2000] loss: 0.282 - Training_Accuracy: 0.896\n",
            "[66,  2000] Validation loss: 0.415 - Validation_Accuracy: 0.860\n",
            "[67,   200] loss: 0.253 - Training_Accuracy: 0.901\n",
            "[67,   400] loss: 0.262 - Training_Accuracy: 0.896\n",
            "[67,   600] loss: 0.265 - Training_Accuracy: 0.898\n",
            "[67,   800] loss: 0.276 - Training_Accuracy: 0.896\n",
            "[67,  1000] loss: 0.281 - Training_Accuracy: 0.890\n",
            "[67,  1200] loss: 0.303 - Training_Accuracy: 0.882\n",
            "[67,  1400] loss: 0.287 - Training_Accuracy: 0.891\n",
            "[67,  1600] loss: 0.298 - Training_Accuracy: 0.886\n",
            "[67,  1800] loss: 0.307 - Training_Accuracy: 0.887\n",
            "[67,  2000] loss: 0.280 - Training_Accuracy: 0.895\n",
            "[67,  2000] Validation loss: 0.465 - Validation_Accuracy: 0.847\n",
            "[68,   200] loss: 0.288 - Training_Accuracy: 0.892\n",
            "[68,   400] loss: 0.267 - Training_Accuracy: 0.893\n",
            "[68,   600] loss: 0.277 - Training_Accuracy: 0.898\n",
            "[68,   800] loss: 0.285 - Training_Accuracy: 0.889\n",
            "[68,  1000] loss: 0.289 - Training_Accuracy: 0.889\n",
            "[68,  1200] loss: 0.266 - Training_Accuracy: 0.898\n",
            "[68,  1400] loss: 0.267 - Training_Accuracy: 0.896\n",
            "[68,  1600] loss: 0.304 - Training_Accuracy: 0.888\n",
            "[68,  1800] loss: 0.309 - Training_Accuracy: 0.880\n",
            "[68,  2000] loss: 0.289 - Training_Accuracy: 0.890\n",
            "[68,  2000] Validation loss: 0.473 - Validation_Accuracy: 0.844\n",
            "[69,   200] loss: 0.288 - Training_Accuracy: 0.888\n",
            "[69,   400] loss: 0.266 - Training_Accuracy: 0.898\n",
            "[69,   600] loss: 0.279 - Training_Accuracy: 0.891\n",
            "[69,   800] loss: 0.285 - Training_Accuracy: 0.893\n",
            "[69,  1000] loss: 0.285 - Training_Accuracy: 0.892\n",
            "[69,  1200] loss: 0.284 - Training_Accuracy: 0.890\n",
            "[69,  1400] loss: 0.270 - Training_Accuracy: 0.895\n",
            "[69,  1600] loss: 0.264 - Training_Accuracy: 0.900\n",
            "[69,  1800] loss: 0.284 - Training_Accuracy: 0.888\n",
            "[69,  2000] loss: 0.277 - Training_Accuracy: 0.897\n",
            "[69,  2000] Validation loss: 0.464 - Validation_Accuracy: 0.849\n",
            "[70,   200] loss: 0.263 - Training_Accuracy: 0.898\n",
            "[70,   400] loss: 0.256 - Training_Accuracy: 0.900\n",
            "[70,   600] loss: 0.260 - Training_Accuracy: 0.901\n",
            "[70,   800] loss: 0.244 - Training_Accuracy: 0.907\n",
            "[70,  1000] loss: 0.253 - Training_Accuracy: 0.899\n",
            "[70,  1200] loss: 0.279 - Training_Accuracy: 0.893\n",
            "[70,  1400] loss: 0.277 - Training_Accuracy: 0.895\n",
            "[70,  1600] loss: 0.271 - Training_Accuracy: 0.900\n",
            "[70,  1800] loss: 0.307 - Training_Accuracy: 0.880\n",
            "[70,  2000] loss: 0.323 - Training_Accuracy: 0.880\n",
            "[70,  2000] Validation loss: 0.549 - Validation_Accuracy: 0.816\n",
            "[71,   200] loss: 0.290 - Training_Accuracy: 0.890\n",
            "[71,   400] loss: 0.267 - Training_Accuracy: 0.898\n",
            "[71,   600] loss: 0.274 - Training_Accuracy: 0.895\n",
            "[71,   800] loss: 0.291 - Training_Accuracy: 0.888\n",
            "[71,  1000] loss: 0.280 - Training_Accuracy: 0.893\n",
            "[71,  1200] loss: 0.296 - Training_Accuracy: 0.888\n",
            "[71,  1400] loss: 0.260 - Training_Accuracy: 0.902\n",
            "[71,  1600] loss: 0.275 - Training_Accuracy: 0.892\n",
            "[71,  1800] loss: 0.291 - Training_Accuracy: 0.885\n",
            "[71,  2000] loss: 0.306 - Training_Accuracy: 0.882\n",
            "[71,  2000] Validation loss: 0.461 - Validation_Accuracy: 0.846\n",
            "[72,   200] loss: 0.278 - Training_Accuracy: 0.894\n",
            "[72,   400] loss: 0.317 - Training_Accuracy: 0.879\n",
            "[72,   600] loss: 0.306 - Training_Accuracy: 0.882\n",
            "[72,   800] loss: 0.269 - Training_Accuracy: 0.898\n",
            "[72,  1000] loss: 0.269 - Training_Accuracy: 0.900\n",
            "[72,  1200] loss: 0.294 - Training_Accuracy: 0.886\n",
            "[72,  1400] loss: 0.285 - Training_Accuracy: 0.889\n",
            "[72,  1600] loss: 0.308 - Training_Accuracy: 0.881\n",
            "[72,  1800] loss: 0.279 - Training_Accuracy: 0.892\n",
            "[72,  2000] loss: 0.279 - Training_Accuracy: 0.897\n",
            "[72,  2000] Validation loss: 0.488 - Validation_Accuracy: 0.837\n",
            "[73,   200] loss: 0.275 - Training_Accuracy: 0.892\n",
            "[73,   400] loss: 0.279 - Training_Accuracy: 0.894\n",
            "[73,   600] loss: 0.276 - Training_Accuracy: 0.897\n",
            "[73,   800] loss: 0.270 - Training_Accuracy: 0.896\n",
            "[73,  1000] loss: 0.262 - Training_Accuracy: 0.898\n",
            "[73,  1200] loss: 0.280 - Training_Accuracy: 0.890\n",
            "[73,  1400] loss: 0.308 - Training_Accuracy: 0.884\n",
            "[73,  1600] loss: 0.312 - Training_Accuracy: 0.885\n",
            "[73,  1800] loss: 0.320 - Training_Accuracy: 0.878\n",
            "[73,  2000] loss: 0.292 - Training_Accuracy: 0.888\n",
            "[73,  2000] Validation loss: 0.464 - Validation_Accuracy: 0.848\n",
            "[74,   200] loss: 0.297 - Training_Accuracy: 0.884\n",
            "[74,   400] loss: 0.267 - Training_Accuracy: 0.896\n",
            "[74,   600] loss: 0.276 - Training_Accuracy: 0.893\n",
            "[74,   800] loss: 0.288 - Training_Accuracy: 0.888\n",
            "[74,  1000] loss: 0.274 - Training_Accuracy: 0.892\n",
            "[74,  1200] loss: 0.297 - Training_Accuracy: 0.894\n",
            "[74,  1400] loss: 0.291 - Training_Accuracy: 0.888\n",
            "[74,  1600] loss: 0.316 - Training_Accuracy: 0.886\n",
            "[74,  1800] loss: 0.315 - Training_Accuracy: 0.875\n",
            "[74,  2000] loss: 0.276 - Training_Accuracy: 0.891\n",
            "[74,  2000] Validation loss: 0.470 - Validation_Accuracy: 0.845\n",
            "[75,   200] loss: 0.270 - Training_Accuracy: 0.897\n",
            "[75,   400] loss: 0.264 - Training_Accuracy: 0.899\n",
            "[75,   600] loss: 0.287 - Training_Accuracy: 0.895\n",
            "[75,   800] loss: 0.292 - Training_Accuracy: 0.890\n",
            "[75,  1000] loss: 0.283 - Training_Accuracy: 0.893\n",
            "[75,  1200] loss: 0.314 - Training_Accuracy: 0.885\n",
            "[75,  1400] loss: 0.307 - Training_Accuracy: 0.885\n",
            "[75,  1600] loss: 0.287 - Training_Accuracy: 0.889\n",
            "[75,  1800] loss: 0.268 - Training_Accuracy: 0.898\n",
            "[75,  2000] loss: 0.257 - Training_Accuracy: 0.897\n",
            "[75,  2000] Validation loss: 0.464 - Validation_Accuracy: 0.847\n",
            "[76,   200] loss: 0.281 - Training_Accuracy: 0.893\n",
            "[76,   400] loss: 0.296 - Training_Accuracy: 0.890\n",
            "[76,   600] loss: 0.283 - Training_Accuracy: 0.893\n",
            "[76,   800] loss: 0.284 - Training_Accuracy: 0.890\n",
            "[76,  1000] loss: 0.275 - Training_Accuracy: 0.894\n",
            "[76,  1200] loss: 0.259 - Training_Accuracy: 0.901\n",
            "[76,  1400] loss: 0.271 - Training_Accuracy: 0.895\n",
            "[76,  1600] loss: 0.280 - Training_Accuracy: 0.893\n",
            "[76,  1800] loss: 0.267 - Training_Accuracy: 0.900\n",
            "[76,  2000] loss: 0.263 - Training_Accuracy: 0.900\n",
            "[76,  2000] Validation loss: 0.444 - Validation_Accuracy: 0.853\n",
            "[77,   200] loss: 0.283 - Training_Accuracy: 0.893\n",
            "[77,   400] loss: 0.262 - Training_Accuracy: 0.900\n",
            "[77,   600] loss: 0.270 - Training_Accuracy: 0.897\n",
            "[77,   800] loss: 0.279 - Training_Accuracy: 0.898\n",
            "[77,  1000] loss: 0.280 - Training_Accuracy: 0.890\n",
            "[77,  1200] loss: 0.329 - Training_Accuracy: 0.875\n",
            "[77,  1400] loss: 0.272 - Training_Accuracy: 0.897\n",
            "[77,  1600] loss: 0.298 - Training_Accuracy: 0.885\n",
            "[77,  1800] loss: 0.290 - Training_Accuracy: 0.891\n",
            "[77,  2000] loss: 0.254 - Training_Accuracy: 0.901\n",
            "[77,  2000] Validation loss: 0.481 - Validation_Accuracy: 0.842\n",
            "[78,   200] loss: 0.252 - Training_Accuracy: 0.901\n",
            "[78,   400] loss: 0.268 - Training_Accuracy: 0.896\n",
            "[78,   600] loss: 0.273 - Training_Accuracy: 0.899\n",
            "[78,   800] loss: 0.254 - Training_Accuracy: 0.897\n",
            "[78,  1000] loss: 0.249 - Training_Accuracy: 0.904\n",
            "[78,  1200] loss: 0.266 - Training_Accuracy: 0.900\n",
            "[78,  1400] loss: 0.240 - Training_Accuracy: 0.906\n",
            "[78,  1600] loss: 0.253 - Training_Accuracy: 0.901\n",
            "[78,  1800] loss: 0.259 - Training_Accuracy: 0.897\n",
            "[78,  2000] loss: 0.267 - Training_Accuracy: 0.899\n",
            "[78,  2000] Validation loss: 0.446 - Validation_Accuracy: 0.854\n",
            "[79,   200] loss: 0.234 - Training_Accuracy: 0.907\n",
            "[79,   400] loss: 0.271 - Training_Accuracy: 0.897\n",
            "[79,   600] loss: 0.272 - Training_Accuracy: 0.895\n",
            "[79,   800] loss: 0.242 - Training_Accuracy: 0.909\n",
            "[79,  1000] loss: 0.277 - Training_Accuracy: 0.896\n",
            "[79,  1200] loss: 0.290 - Training_Accuracy: 0.887\n",
            "[79,  1400] loss: 0.272 - Training_Accuracy: 0.897\n",
            "[79,  1600] loss: 0.276 - Training_Accuracy: 0.894\n",
            "[79,  1800] loss: 0.301 - Training_Accuracy: 0.885\n",
            "[79,  2000] loss: 0.291 - Training_Accuracy: 0.890\n",
            "[79,  2000] Validation loss: 0.473 - Validation_Accuracy: 0.846\n",
            "[80,   200] loss: 0.274 - Training_Accuracy: 0.892\n",
            "[80,   400] loss: 0.263 - Training_Accuracy: 0.901\n",
            "[80,   600] loss: 0.246 - Training_Accuracy: 0.904\n",
            "[80,   800] loss: 0.272 - Training_Accuracy: 0.899\n",
            "[80,  1000] loss: 0.272 - Training_Accuracy: 0.891\n",
            "[80,  1200] loss: 0.271 - Training_Accuracy: 0.899\n",
            "[80,  1400] loss: 0.251 - Training_Accuracy: 0.901\n",
            "[80,  1600] loss: 0.285 - Training_Accuracy: 0.888\n",
            "[80,  1800] loss: 0.282 - Training_Accuracy: 0.890\n",
            "[80,  2000] loss: 0.298 - Training_Accuracy: 0.888\n",
            "[80,  2000] Validation loss: 0.476 - Validation_Accuracy: 0.845\n",
            "[81,   200] loss: 0.258 - Training_Accuracy: 0.903\n",
            "[81,   400] loss: 0.294 - Training_Accuracy: 0.889\n",
            "[81,   600] loss: 0.257 - Training_Accuracy: 0.900\n",
            "[81,   800] loss: 0.246 - Training_Accuracy: 0.901\n",
            "[81,  1000] loss: 0.251 - Training_Accuracy: 0.901\n",
            "[81,  1200] loss: 0.281 - Training_Accuracy: 0.889\n",
            "[81,  1400] loss: 0.288 - Training_Accuracy: 0.891\n",
            "[81,  1600] loss: 0.267 - Training_Accuracy: 0.899\n",
            "[81,  1800] loss: 0.271 - Training_Accuracy: 0.896\n",
            "[81,  2000] loss: 0.273 - Training_Accuracy: 0.897\n",
            "[81,  2000] Validation loss: 0.472 - Validation_Accuracy: 0.848\n",
            "[82,   200] loss: 0.250 - Training_Accuracy: 0.902\n",
            "[82,   400] loss: 0.265 - Training_Accuracy: 0.897\n",
            "[82,   600] loss: 0.263 - Training_Accuracy: 0.903\n",
            "[82,   800] loss: 0.273 - Training_Accuracy: 0.898\n",
            "[82,  1000] loss: 0.275 - Training_Accuracy: 0.896\n",
            "[82,  1200] loss: 0.289 - Training_Accuracy: 0.889\n",
            "[82,  1400] loss: 0.256 - Training_Accuracy: 0.899\n",
            "[82,  1600] loss: 0.269 - Training_Accuracy: 0.898\n",
            "[82,  1800] loss: 0.300 - Training_Accuracy: 0.891\n",
            "[82,  2000] loss: 0.281 - Training_Accuracy: 0.893\n",
            "[82,  2000] Validation loss: 0.472 - Validation_Accuracy: 0.849\n",
            "[83,   200] loss: 0.251 - Training_Accuracy: 0.900\n",
            "[83,   400] loss: 0.275 - Training_Accuracy: 0.901\n",
            "[83,   600] loss: 0.245 - Training_Accuracy: 0.911\n",
            "[83,   800] loss: 0.251 - Training_Accuracy: 0.905\n",
            "[83,  1000] loss: 0.256 - Training_Accuracy: 0.900\n",
            "[83,  1200] loss: 0.256 - Training_Accuracy: 0.902\n",
            "[83,  1400] loss: 0.249 - Training_Accuracy: 0.902\n",
            "[83,  1600] loss: 0.240 - Training_Accuracy: 0.905\n",
            "[83,  1800] loss: 0.232 - Training_Accuracy: 0.908\n",
            "[83,  2000] loss: 0.234 - Training_Accuracy: 0.909\n",
            "[83,  2000] Validation loss: 0.496 - Validation_Accuracy: 0.843\n",
            "[84,   200] loss: 0.265 - Training_Accuracy: 0.895\n",
            "[84,   400] loss: 0.256 - Training_Accuracy: 0.904\n",
            "[84,   600] loss: 0.255 - Training_Accuracy: 0.905\n",
            "[84,   800] loss: 0.253 - Training_Accuracy: 0.900\n",
            "[84,  1000] loss: 0.281 - Training_Accuracy: 0.896\n",
            "[84,  1200] loss: 0.279 - Training_Accuracy: 0.899\n",
            "[84,  1400] loss: 0.283 - Training_Accuracy: 0.891\n",
            "[84,  1600] loss: 0.282 - Training_Accuracy: 0.892\n",
            "[84,  1800] loss: 0.273 - Training_Accuracy: 0.892\n",
            "[84,  2000] loss: 0.251 - Training_Accuracy: 0.904\n",
            "[84,  2000] Validation loss: 0.486 - Validation_Accuracy: 0.846\n",
            "[85,   200] loss: 0.255 - Training_Accuracy: 0.900\n",
            "[85,   400] loss: 0.245 - Training_Accuracy: 0.910\n",
            "[85,   600] loss: 0.247 - Training_Accuracy: 0.904\n",
            "[85,   800] loss: 0.272 - Training_Accuracy: 0.893\n",
            "[85,  1000] loss: 0.258 - Training_Accuracy: 0.900\n",
            "[85,  1200] loss: 0.245 - Training_Accuracy: 0.901\n",
            "[85,  1400] loss: 0.257 - Training_Accuracy: 0.898\n",
            "[85,  1600] loss: 0.249 - Training_Accuracy: 0.902\n",
            "[85,  1800] loss: 0.286 - Training_Accuracy: 0.887\n",
            "[85,  2000] loss: 0.272 - Training_Accuracy: 0.897\n",
            "[85,  2000] Validation loss: 0.484 - Validation_Accuracy: 0.843\n",
            "[86,   200] loss: 0.251 - Training_Accuracy: 0.906\n",
            "[86,   400] loss: 0.264 - Training_Accuracy: 0.896\n",
            "[86,   600] loss: 0.270 - Training_Accuracy: 0.897\n",
            "[86,   800] loss: 0.251 - Training_Accuracy: 0.906\n",
            "[86,  1000] loss: 0.240 - Training_Accuracy: 0.903\n",
            "[86,  1200] loss: 0.245 - Training_Accuracy: 0.909\n",
            "[86,  1400] loss: 0.256 - Training_Accuracy: 0.905\n",
            "[86,  1600] loss: 0.257 - Training_Accuracy: 0.903\n",
            "[86,  1800] loss: 0.240 - Training_Accuracy: 0.906\n",
            "[86,  2000] loss: 0.248 - Training_Accuracy: 0.901\n",
            "[86,  2000] Validation loss: 0.461 - Validation_Accuracy: 0.854\n",
            "[87,   200] loss: 0.248 - Training_Accuracy: 0.904\n",
            "[87,   400] loss: 0.225 - Training_Accuracy: 0.912\n",
            "[87,   600] loss: 0.221 - Training_Accuracy: 0.912\n",
            "[87,   800] loss: 0.227 - Training_Accuracy: 0.913\n",
            "[87,  1000] loss: 0.225 - Training_Accuracy: 0.913\n",
            "[87,  1200] loss: 0.240 - Training_Accuracy: 0.904\n",
            "[87,  1400] loss: 0.266 - Training_Accuracy: 0.893\n",
            "[87,  1600] loss: 0.264 - Training_Accuracy: 0.898\n",
            "[87,  1800] loss: 0.278 - Training_Accuracy: 0.891\n",
            "[87,  2000] loss: 0.269 - Training_Accuracy: 0.894\n",
            "[87,  2000] Validation loss: 0.480 - Validation_Accuracy: 0.840\n",
            "[88,   200] loss: 0.240 - Training_Accuracy: 0.910\n",
            "[88,   400] loss: 0.257 - Training_Accuracy: 0.901\n",
            "[88,   600] loss: 0.234 - Training_Accuracy: 0.909\n",
            "[88,   800] loss: 0.255 - Training_Accuracy: 0.906\n",
            "[88,  1000] loss: 0.273 - Training_Accuracy: 0.892\n",
            "[88,  1200] loss: 0.260 - Training_Accuracy: 0.900\n",
            "[88,  1400] loss: 0.254 - Training_Accuracy: 0.901\n",
            "[88,  1600] loss: 0.273 - Training_Accuracy: 0.894\n",
            "[88,  1800] loss: 0.262 - Training_Accuracy: 0.899\n",
            "[88,  2000] loss: 0.250 - Training_Accuracy: 0.904\n",
            "[88,  2000] Validation loss: 0.453 - Validation_Accuracy: 0.854\n",
            "[89,   200] loss: 0.239 - Training_Accuracy: 0.911\n",
            "[89,   400] loss: 0.245 - Training_Accuracy: 0.904\n",
            "[89,   600] loss: 0.245 - Training_Accuracy: 0.906\n",
            "[89,   800] loss: 0.295 - Training_Accuracy: 0.891\n",
            "[89,  1000] loss: 0.272 - Training_Accuracy: 0.897\n",
            "[89,  1200] loss: 0.245 - Training_Accuracy: 0.906\n",
            "[89,  1400] loss: 0.259 - Training_Accuracy: 0.902\n",
            "[89,  1600] loss: 0.278 - Training_Accuracy: 0.894\n",
            "[89,  1800] loss: 0.265 - Training_Accuracy: 0.899\n",
            "[89,  2000] loss: 0.276 - Training_Accuracy: 0.894\n",
            "[89,  2000] Validation loss: 0.514 - Validation_Accuracy: 0.835\n",
            "[90,   200] loss: 0.245 - Training_Accuracy: 0.903\n",
            "[90,   400] loss: 0.296 - Training_Accuracy: 0.888\n",
            "[90,   600] loss: 0.273 - Training_Accuracy: 0.892\n",
            "[90,   800] loss: 0.263 - Training_Accuracy: 0.901\n",
            "[90,  1000] loss: 0.278 - Training_Accuracy: 0.897\n",
            "[90,  1200] loss: 0.267 - Training_Accuracy: 0.900\n",
            "[90,  1400] loss: 0.274 - Training_Accuracy: 0.893\n",
            "[90,  1600] loss: 0.271 - Training_Accuracy: 0.896\n",
            "[90,  1800] loss: 0.242 - Training_Accuracy: 0.908\n",
            "[90,  2000] loss: 0.247 - Training_Accuracy: 0.903\n",
            "[90,  2000] Validation loss: 0.463 - Validation_Accuracy: 0.855\n",
            "[91,   200] loss: 0.243 - Training_Accuracy: 0.908\n",
            "[91,   400] loss: 0.245 - Training_Accuracy: 0.905\n",
            "[91,   600] loss: 0.246 - Training_Accuracy: 0.903\n",
            "[91,   800] loss: 0.276 - Training_Accuracy: 0.896\n",
            "[91,  1000] loss: 0.250 - Training_Accuracy: 0.904\n",
            "[91,  1200] loss: 0.272 - Training_Accuracy: 0.899\n",
            "[91,  1400] loss: 0.247 - Training_Accuracy: 0.908\n",
            "[91,  1600] loss: 0.241 - Training_Accuracy: 0.903\n",
            "[91,  1800] loss: 0.263 - Training_Accuracy: 0.900\n",
            "[91,  2000] loss: 0.230 - Training_Accuracy: 0.911\n",
            "[91,  2000] Validation loss: 0.509 - Validation_Accuracy: 0.840\n",
            "[92,   200] loss: 0.250 - Training_Accuracy: 0.905\n",
            "[92,   400] loss: 0.281 - Training_Accuracy: 0.895\n",
            "[92,   600] loss: 0.273 - Training_Accuracy: 0.890\n",
            "[92,   800] loss: 0.242 - Training_Accuracy: 0.904\n",
            "[92,  1000] loss: 0.247 - Training_Accuracy: 0.903\n",
            "[92,  1200] loss: 0.284 - Training_Accuracy: 0.894\n",
            "[92,  1400] loss: 0.271 - Training_Accuracy: 0.894\n",
            "[92,  1600] loss: 0.255 - Training_Accuracy: 0.903\n",
            "[92,  1800] loss: 0.260 - Training_Accuracy: 0.899\n",
            "[92,  2000] loss: 0.242 - Training_Accuracy: 0.901\n",
            "[92,  2000] Validation loss: 0.468 - Validation_Accuracy: 0.853\n",
            "[93,   200] loss: 0.230 - Training_Accuracy: 0.911\n",
            "[93,   400] loss: 0.236 - Training_Accuracy: 0.912\n",
            "[93,   600] loss: 0.233 - Training_Accuracy: 0.908\n",
            "[93,   800] loss: 0.230 - Training_Accuracy: 0.909\n",
            "[93,  1000] loss: 0.231 - Training_Accuracy: 0.909\n",
            "[93,  1200] loss: 0.226 - Training_Accuracy: 0.911\n",
            "[93,  1400] loss: 0.223 - Training_Accuracy: 0.910\n",
            "[93,  1600] loss: 0.222 - Training_Accuracy: 0.910\n",
            "[93,  1800] loss: 0.227 - Training_Accuracy: 0.914\n",
            "[93,  2000] loss: 0.250 - Training_Accuracy: 0.903\n",
            "[93,  2000] Validation loss: 0.544 - Validation_Accuracy: 0.835\n",
            "[94,   200] loss: 0.264 - Training_Accuracy: 0.899\n",
            "[94,   400] loss: 0.253 - Training_Accuracy: 0.901\n",
            "[94,   600] loss: 0.256 - Training_Accuracy: 0.901\n",
            "[94,   800] loss: 0.254 - Training_Accuracy: 0.902\n",
            "[94,  1000] loss: 0.240 - Training_Accuracy: 0.909\n",
            "[94,  1200] loss: 0.242 - Training_Accuracy: 0.902\n",
            "[94,  1400] loss: 0.231 - Training_Accuracy: 0.909\n",
            "[94,  1600] loss: 0.243 - Training_Accuracy: 0.908\n",
            "[94,  1800] loss: 0.250 - Training_Accuracy: 0.902\n",
            "[94,  2000] loss: 0.254 - Training_Accuracy: 0.905\n",
            "[94,  2000] Validation loss: 0.520 - Validation_Accuracy: 0.839\n",
            "[95,   200] loss: 0.237 - Training_Accuracy: 0.910\n",
            "[95,   400] loss: 0.240 - Training_Accuracy: 0.905\n",
            "[95,   600] loss: 0.229 - Training_Accuracy: 0.909\n",
            "[95,   800] loss: 0.232 - Training_Accuracy: 0.909\n",
            "[95,  1000] loss: 0.220 - Training_Accuracy: 0.916\n",
            "[95,  1200] loss: 0.235 - Training_Accuracy: 0.910\n",
            "[95,  1400] loss: 0.261 - Training_Accuracy: 0.901\n",
            "[95,  1600] loss: 0.239 - Training_Accuracy: 0.910\n",
            "[95,  1800] loss: 0.234 - Training_Accuracy: 0.905\n",
            "[95,  2000] loss: 0.245 - Training_Accuracy: 0.904\n",
            "[95,  2000] Validation loss: 0.488 - Validation_Accuracy: 0.846\n",
            "[96,   200] loss: 0.212 - Training_Accuracy: 0.917\n",
            "[96,   400] loss: 0.273 - Training_Accuracy: 0.898\n",
            "[96,   600] loss: 0.248 - Training_Accuracy: 0.906\n",
            "[96,   800] loss: 0.247 - Training_Accuracy: 0.907\n",
            "[96,  1000] loss: 0.232 - Training_Accuracy: 0.911\n",
            "[96,  1200] loss: 0.242 - Training_Accuracy: 0.905\n",
            "[96,  1400] loss: 0.285 - Training_Accuracy: 0.892\n",
            "[96,  1600] loss: 0.287 - Training_Accuracy: 0.889\n",
            "[96,  1800] loss: 0.252 - Training_Accuracy: 0.899\n",
            "[96,  2000] loss: 0.267 - Training_Accuracy: 0.900\n",
            "[96,  2000] Validation loss: 0.507 - Validation_Accuracy: 0.839\n",
            "[97,   200] loss: 0.263 - Training_Accuracy: 0.896\n",
            "[97,   400] loss: 0.221 - Training_Accuracy: 0.913\n",
            "[97,   600] loss: 0.240 - Training_Accuracy: 0.904\n",
            "[97,   800] loss: 0.285 - Training_Accuracy: 0.893\n",
            "[97,  1000] loss: 0.250 - Training_Accuracy: 0.905\n",
            "[97,  1200] loss: 0.269 - Training_Accuracy: 0.896\n",
            "[97,  1400] loss: 0.255 - Training_Accuracy: 0.903\n",
            "[97,  1600] loss: 0.252 - Training_Accuracy: 0.901\n",
            "[97,  1800] loss: 0.276 - Training_Accuracy: 0.894\n",
            "[97,  2000] loss: 0.253 - Training_Accuracy: 0.899\n",
            "[97,  2000] Validation loss: 0.481 - Validation_Accuracy: 0.849\n",
            "[98,   200] loss: 0.258 - Training_Accuracy: 0.902\n",
            "[98,   400] loss: 0.271 - Training_Accuracy: 0.897\n",
            "[98,   600] loss: 0.245 - Training_Accuracy: 0.905\n",
            "[98,   800] loss: 0.231 - Training_Accuracy: 0.910\n",
            "[98,  1000] loss: 0.264 - Training_Accuracy: 0.898\n",
            "[98,  1200] loss: 0.250 - Training_Accuracy: 0.901\n",
            "[98,  1400] loss: 0.241 - Training_Accuracy: 0.909\n",
            "[98,  1600] loss: 0.238 - Training_Accuracy: 0.909\n",
            "[98,  1800] loss: 0.254 - Training_Accuracy: 0.897\n",
            "[98,  2000] loss: 0.245 - Training_Accuracy: 0.905\n",
            "[98,  2000] Validation loss: 0.462 - Validation_Accuracy: 0.856\n",
            "[99,   200] loss: 0.223 - Training_Accuracy: 0.916\n",
            "[99,   400] loss: 0.214 - Training_Accuracy: 0.915\n",
            "[99,   600] loss: 0.206 - Training_Accuracy: 0.918\n",
            "[99,   800] loss: 0.220 - Training_Accuracy: 0.910\n",
            "[99,  1000] loss: 0.223 - Training_Accuracy: 0.912\n",
            "[99,  1200] loss: 0.245 - Training_Accuracy: 0.908\n",
            "[99,  1400] loss: 0.209 - Training_Accuracy: 0.918\n",
            "[99,  1600] loss: 0.215 - Training_Accuracy: 0.915\n",
            "[99,  1800] loss: 0.235 - Training_Accuracy: 0.905\n",
            "[99,  2000] loss: 0.221 - Training_Accuracy: 0.911\n",
            "[99,  2000] Validation loss: 0.504 - Validation_Accuracy: 0.848\n",
            "[100,   200] loss: 0.217 - Training_Accuracy: 0.913\n",
            "[100,   400] loss: 0.213 - Training_Accuracy: 0.913\n",
            "[100,   600] loss: 0.225 - Training_Accuracy: 0.909\n",
            "[100,   800] loss: 0.215 - Training_Accuracy: 0.915\n",
            "[100,  1000] loss: 0.217 - Training_Accuracy: 0.917\n",
            "[100,  1200] loss: 0.237 - Training_Accuracy: 0.904\n",
            "[100,  1400] loss: 0.217 - Training_Accuracy: 0.914\n",
            "[100,  1600] loss: 0.222 - Training_Accuracy: 0.915\n",
            "[100,  1800] loss: 0.229 - Training_Accuracy: 0.909\n",
            "[100,  2000] loss: 0.217 - Training_Accuracy: 0.915\n",
            "[100,  2000] Validation loss: 0.461 - Validation_Accuracy: 0.861\n",
            "[101,   200] loss: 0.205 - Training_Accuracy: 0.920\n",
            "[101,   400] loss: 0.210 - Training_Accuracy: 0.918\n",
            "[101,   600] loss: 0.230 - Training_Accuracy: 0.908\n",
            "[101,   800] loss: 0.209 - Training_Accuracy: 0.918\n",
            "[101,  1000] loss: 0.207 - Training_Accuracy: 0.918\n",
            "[101,  1200] loss: 0.189 - Training_Accuracy: 0.924\n",
            "[101,  1400] loss: 0.238 - Training_Accuracy: 0.906\n",
            "[101,  1600] loss: 0.243 - Training_Accuracy: 0.906\n",
            "[101,  1800] loss: 0.240 - Training_Accuracy: 0.905\n",
            "[101,  2000] loss: 0.229 - Training_Accuracy: 0.909\n",
            "[101,  2000] Validation loss: 0.514 - Validation_Accuracy: 0.841\n",
            "[102,   200] loss: 0.247 - Training_Accuracy: 0.901\n",
            "[102,   400] loss: 0.217 - Training_Accuracy: 0.916\n",
            "[102,   600] loss: 0.228 - Training_Accuracy: 0.910\n",
            "[102,   800] loss: 0.229 - Training_Accuracy: 0.910\n",
            "[102,  1000] loss: 0.233 - Training_Accuracy: 0.910\n",
            "[102,  1200] loss: 0.220 - Training_Accuracy: 0.916\n",
            "[102,  1400] loss: 0.234 - Training_Accuracy: 0.913\n",
            "[102,  1600] loss: 0.227 - Training_Accuracy: 0.913\n",
            "[102,  1800] loss: 0.217 - Training_Accuracy: 0.916\n",
            "[102,  2000] loss: 0.225 - Training_Accuracy: 0.916\n",
            "[102,  2000] Validation loss: 0.541 - Validation_Accuracy: 0.835\n",
            "[103,   200] loss: 0.241 - Training_Accuracy: 0.905\n",
            "[103,   400] loss: 0.245 - Training_Accuracy: 0.899\n",
            "[103,   600] loss: 0.236 - Training_Accuracy: 0.905\n",
            "[103,   800] loss: 0.233 - Training_Accuracy: 0.910\n",
            "[103,  1000] loss: 0.222 - Training_Accuracy: 0.913\n",
            "[103,  1200] loss: 0.222 - Training_Accuracy: 0.913\n",
            "[103,  1400] loss: 0.214 - Training_Accuracy: 0.912\n",
            "[103,  1600] loss: 0.236 - Training_Accuracy: 0.909\n",
            "[103,  1800] loss: 0.209 - Training_Accuracy: 0.920\n",
            "[103,  2000] loss: 0.237 - Training_Accuracy: 0.909\n",
            "[103,  2000] Validation loss: 0.527 - Validation_Accuracy: 0.843\n",
            "[104,   200] loss: 0.219 - Training_Accuracy: 0.915\n",
            "[104,   400] loss: 0.221 - Training_Accuracy: 0.913\n",
            "[104,   600] loss: 0.237 - Training_Accuracy: 0.906\n",
            "[104,   800] loss: 0.211 - Training_Accuracy: 0.922\n",
            "[104,  1000] loss: 0.203 - Training_Accuracy: 0.920\n",
            "[104,  1200] loss: 0.221 - Training_Accuracy: 0.910\n",
            "[104,  1400] loss: 0.222 - Training_Accuracy: 0.912\n",
            "[104,  1600] loss: 0.216 - Training_Accuracy: 0.917\n",
            "[104,  1800] loss: 0.212 - Training_Accuracy: 0.917\n",
            "[104,  2000] loss: 0.222 - Training_Accuracy: 0.908\n",
            "[104,  2000] Validation loss: 0.502 - Validation_Accuracy: 0.850\n",
            "[105,   200] loss: 0.199 - Training_Accuracy: 0.923\n",
            "[105,   400] loss: 0.234 - Training_Accuracy: 0.909\n",
            "[105,   600] loss: 0.238 - Training_Accuracy: 0.911\n",
            "[105,   800] loss: 0.239 - Training_Accuracy: 0.905\n",
            "[105,  1000] loss: 0.225 - Training_Accuracy: 0.909\n",
            "[105,  1200] loss: 0.228 - Training_Accuracy: 0.912\n",
            "[105,  1400] loss: 0.207 - Training_Accuracy: 0.918\n",
            "[105,  1600] loss: 0.222 - Training_Accuracy: 0.912\n",
            "[105,  1800] loss: 0.208 - Training_Accuracy: 0.920\n",
            "[105,  2000] loss: 0.228 - Training_Accuracy: 0.912\n",
            "[105,  2000] Validation loss: 0.502 - Validation_Accuracy: 0.849\n",
            "[106,   200] loss: 0.204 - Training_Accuracy: 0.922\n",
            "[106,   400] loss: 0.206 - Training_Accuracy: 0.916\n",
            "[106,   600] loss: 0.228 - Training_Accuracy: 0.908\n",
            "[106,   800] loss: 0.206 - Training_Accuracy: 0.919\n",
            "[106,  1000] loss: 0.219 - Training_Accuracy: 0.911\n",
            "[106,  1200] loss: 0.223 - Training_Accuracy: 0.911\n",
            "[106,  1400] loss: 0.214 - Training_Accuracy: 0.915\n",
            "[106,  1600] loss: 0.218 - Training_Accuracy: 0.918\n",
            "[106,  1800] loss: 0.217 - Training_Accuracy: 0.911\n",
            "[106,  2000] loss: 0.213 - Training_Accuracy: 0.918\n",
            "[106,  2000] Validation loss: 0.506 - Validation_Accuracy: 0.849\n",
            "[107,   200] loss: 0.220 - Training_Accuracy: 0.914\n",
            "[107,   400] loss: 0.214 - Training_Accuracy: 0.913\n",
            "[107,   600] loss: 0.208 - Training_Accuracy: 0.920\n",
            "[107,   800] loss: 0.207 - Training_Accuracy: 0.922\n",
            "[107,  1000] loss: 0.225 - Training_Accuracy: 0.911\n",
            "[107,  1200] loss: 0.230 - Training_Accuracy: 0.910\n",
            "[107,  1400] loss: 0.206 - Training_Accuracy: 0.918\n",
            "[107,  1600] loss: 0.210 - Training_Accuracy: 0.918\n",
            "[107,  1800] loss: 0.216 - Training_Accuracy: 0.915\n",
            "[107,  2000] loss: 0.198 - Training_Accuracy: 0.920\n",
            "[107,  2000] Validation loss: 0.495 - Validation_Accuracy: 0.847\n",
            "[108,   200] loss: 0.191 - Training_Accuracy: 0.921\n",
            "[108,   400] loss: 0.205 - Training_Accuracy: 0.919\n",
            "[108,   600] loss: 0.206 - Training_Accuracy: 0.918\n",
            "[108,   800] loss: 0.198 - Training_Accuracy: 0.919\n",
            "[108,  1000] loss: 0.207 - Training_Accuracy: 0.917\n",
            "[108,  1200] loss: 0.211 - Training_Accuracy: 0.916\n",
            "[108,  1400] loss: 0.219 - Training_Accuracy: 0.915\n",
            "[108,  1600] loss: 0.226 - Training_Accuracy: 0.909\n",
            "[108,  1800] loss: 0.219 - Training_Accuracy: 0.913\n",
            "[108,  2000] loss: 0.207 - Training_Accuracy: 0.917\n",
            "[108,  2000] Validation loss: 0.515 - Validation_Accuracy: 0.845\n",
            "[109,   200] loss: 0.188 - Training_Accuracy: 0.927\n",
            "[109,   400] loss: 0.213 - Training_Accuracy: 0.919\n",
            "[109,   600] loss: 0.196 - Training_Accuracy: 0.919\n",
            "[109,   800] loss: 0.227 - Training_Accuracy: 0.910\n",
            "[109,  1000] loss: 0.228 - Training_Accuracy: 0.913\n",
            "[109,  1200] loss: 0.217 - Training_Accuracy: 0.912\n",
            "[109,  1400] loss: 0.208 - Training_Accuracy: 0.916\n",
            "[109,  1600] loss: 0.215 - Training_Accuracy: 0.918\n",
            "[109,  1800] loss: 0.204 - Training_Accuracy: 0.919\n",
            "[109,  2000] loss: 0.206 - Training_Accuracy: 0.920\n",
            "[109,  2000] Validation loss: 0.505 - Validation_Accuracy: 0.846\n",
            "[110,   200] loss: 0.214 - Training_Accuracy: 0.917\n",
            "[110,   400] loss: 0.198 - Training_Accuracy: 0.914\n",
            "[110,   600] loss: 0.205 - Training_Accuracy: 0.919\n",
            "[110,   800] loss: 0.201 - Training_Accuracy: 0.922\n",
            "[110,  1000] loss: 0.200 - Training_Accuracy: 0.921\n",
            "[110,  1200] loss: 0.214 - Training_Accuracy: 0.913\n",
            "[110,  1400] loss: 0.219 - Training_Accuracy: 0.913\n",
            "[110,  1600] loss: 0.194 - Training_Accuracy: 0.923\n",
            "[110,  1800] loss: 0.209 - Training_Accuracy: 0.915\n",
            "[110,  2000] loss: 0.212 - Training_Accuracy: 0.916\n",
            "[110,  2000] Validation loss: 0.483 - Validation_Accuracy: 0.856\n",
            "[111,   200] loss: 0.188 - Training_Accuracy: 0.928\n",
            "[111,   400] loss: 0.190 - Training_Accuracy: 0.925\n",
            "[111,   600] loss: 0.202 - Training_Accuracy: 0.919\n",
            "[111,   800] loss: 0.193 - Training_Accuracy: 0.925\n",
            "[111,  1000] loss: 0.228 - Training_Accuracy: 0.909\n",
            "[111,  1200] loss: 0.224 - Training_Accuracy: 0.912\n",
            "[111,  1400] loss: 0.226 - Training_Accuracy: 0.912\n",
            "[111,  1600] loss: 0.219 - Training_Accuracy: 0.912\n",
            "[111,  1800] loss: 0.228 - Training_Accuracy: 0.911\n",
            "[111,  2000] loss: 0.221 - Training_Accuracy: 0.911\n",
            "[111,  2000] Validation loss: 0.494 - Validation_Accuracy: 0.853\n",
            "[112,   200] loss: 0.214 - Training_Accuracy: 0.915\n",
            "[112,   400] loss: 0.214 - Training_Accuracy: 0.918\n",
            "[112,   600] loss: 0.225 - Training_Accuracy: 0.912\n",
            "[112,   800] loss: 0.206 - Training_Accuracy: 0.921\n",
            "[112,  1000] loss: 0.217 - Training_Accuracy: 0.917\n",
            "[112,  1200] loss: 0.217 - Training_Accuracy: 0.913\n",
            "[112,  1400] loss: 0.212 - Training_Accuracy: 0.918\n",
            "[112,  1600] loss: 0.233 - Training_Accuracy: 0.906\n",
            "[112,  1800] loss: 0.262 - Training_Accuracy: 0.904\n",
            "[112,  2000] loss: 0.234 - Training_Accuracy: 0.912\n",
            "[112,  2000] Validation loss: 0.549 - Validation_Accuracy: 0.839\n",
            "[113,   200] loss: 0.234 - Training_Accuracy: 0.908\n",
            "[113,   400] loss: 0.199 - Training_Accuracy: 0.921\n",
            "[113,   600] loss: 0.206 - Training_Accuracy: 0.921\n",
            "[113,   800] loss: 0.202 - Training_Accuracy: 0.917\n",
            "[113,  1000] loss: 0.220 - Training_Accuracy: 0.913\n",
            "[113,  1200] loss: 0.193 - Training_Accuracy: 0.926\n",
            "[113,  1400] loss: 0.213 - Training_Accuracy: 0.919\n",
            "[113,  1600] loss: 0.213 - Training_Accuracy: 0.914\n",
            "[113,  1800] loss: 0.198 - Training_Accuracy: 0.920\n",
            "[113,  2000] loss: 0.199 - Training_Accuracy: 0.922\n",
            "[113,  2000] Validation loss: 0.513 - Validation_Accuracy: 0.849\n",
            "[114,   200] loss: 0.203 - Training_Accuracy: 0.920\n",
            "[114,   400] loss: 0.190 - Training_Accuracy: 0.927\n",
            "[114,   600] loss: 0.201 - Training_Accuracy: 0.919\n",
            "[114,   800] loss: 0.212 - Training_Accuracy: 0.921\n",
            "[114,  1000] loss: 0.236 - Training_Accuracy: 0.910\n",
            "[114,  1200] loss: 0.208 - Training_Accuracy: 0.916\n",
            "[114,  1400] loss: 0.209 - Training_Accuracy: 0.917\n",
            "[114,  1600] loss: 0.195 - Training_Accuracy: 0.923\n",
            "[114,  1800] loss: 0.194 - Training_Accuracy: 0.925\n",
            "[114,  2000] loss: 0.198 - Training_Accuracy: 0.919\n",
            "[114,  2000] Validation loss: 0.481 - Validation_Accuracy: 0.856\n",
            "[115,   200] loss: 0.192 - Training_Accuracy: 0.925\n",
            "[115,   400] loss: 0.216 - Training_Accuracy: 0.914\n",
            "[115,   600] loss: 0.225 - Training_Accuracy: 0.913\n",
            "[115,   800] loss: 0.215 - Training_Accuracy: 0.916\n",
            "[115,  1000] loss: 0.196 - Training_Accuracy: 0.923\n",
            "[115,  1200] loss: 0.200 - Training_Accuracy: 0.922\n",
            "[115,  1400] loss: 0.211 - Training_Accuracy: 0.916\n",
            "[115,  1600] loss: 0.214 - Training_Accuracy: 0.917\n",
            "[115,  1800] loss: 0.248 - Training_Accuracy: 0.904\n",
            "[115,  2000] loss: 0.232 - Training_Accuracy: 0.910\n",
            "[115,  2000] Validation loss: 0.494 - Validation_Accuracy: 0.853\n",
            "[116,   200] loss: 0.209 - Training_Accuracy: 0.917\n",
            "[116,   400] loss: 0.207 - Training_Accuracy: 0.919\n",
            "[116,   600] loss: 0.198 - Training_Accuracy: 0.922\n",
            "[116,   800] loss: 0.203 - Training_Accuracy: 0.923\n",
            "[116,  1000] loss: 0.207 - Training_Accuracy: 0.918\n",
            "[116,  1200] loss: 0.213 - Training_Accuracy: 0.916\n",
            "[116,  1400] loss: 0.221 - Training_Accuracy: 0.912\n",
            "[116,  1600] loss: 0.201 - Training_Accuracy: 0.922\n",
            "[116,  1800] loss: 0.207 - Training_Accuracy: 0.917\n",
            "[116,  2000] loss: 0.234 - Training_Accuracy: 0.907\n",
            "[116,  2000] Validation loss: 0.545 - Validation_Accuracy: 0.841\n",
            "[117,   200] loss: 0.221 - Training_Accuracy: 0.915\n",
            "[117,   400] loss: 0.223 - Training_Accuracy: 0.915\n",
            "[117,   600] loss: 0.228 - Training_Accuracy: 0.907\n",
            "[117,   800] loss: 0.195 - Training_Accuracy: 0.923\n",
            "[117,  1000] loss: 0.196 - Training_Accuracy: 0.923\n",
            "[117,  1200] loss: 0.189 - Training_Accuracy: 0.924\n",
            "[117,  1400] loss: 0.208 - Training_Accuracy: 0.918\n",
            "[117,  1600] loss: 0.220 - Training_Accuracy: 0.917\n",
            "[117,  1800] loss: 0.212 - Training_Accuracy: 0.916\n",
            "[117,  2000] loss: 0.235 - Training_Accuracy: 0.909\n",
            "[117,  2000] Validation loss: 0.566 - Validation_Accuracy: 0.835\n",
            "[118,   200] loss: 0.201 - Training_Accuracy: 0.922\n",
            "[118,   400] loss: 0.193 - Training_Accuracy: 0.924\n",
            "[118,   600] loss: 0.199 - Training_Accuracy: 0.925\n",
            "[118,   800] loss: 0.194 - Training_Accuracy: 0.922\n",
            "[118,  1000] loss: 0.194 - Training_Accuracy: 0.923\n",
            "[118,  1200] loss: 0.226 - Training_Accuracy: 0.913\n",
            "[118,  1400] loss: 0.216 - Training_Accuracy: 0.913\n",
            "[118,  1600] loss: 0.200 - Training_Accuracy: 0.922\n",
            "[118,  1800] loss: 0.197 - Training_Accuracy: 0.925\n",
            "[118,  2000] loss: 0.237 - Training_Accuracy: 0.908\n",
            "[118,  2000] Validation loss: 0.537 - Validation_Accuracy: 0.845\n",
            "[119,   200] loss: 0.211 - Training_Accuracy: 0.917\n",
            "[119,   400] loss: 0.224 - Training_Accuracy: 0.916\n",
            "[119,   600] loss: 0.217 - Training_Accuracy: 0.912\n",
            "[119,   800] loss: 0.213 - Training_Accuracy: 0.914\n",
            "[119,  1000] loss: 0.200 - Training_Accuracy: 0.919\n",
            "[119,  1200] loss: 0.186 - Training_Accuracy: 0.927\n",
            "[119,  1400] loss: 0.191 - Training_Accuracy: 0.924\n",
            "[119,  1600] loss: 0.200 - Training_Accuracy: 0.918\n",
            "[119,  1800] loss: 0.202 - Training_Accuracy: 0.922\n",
            "[119,  2000] loss: 0.200 - Training_Accuracy: 0.919\n",
            "[119,  2000] Validation loss: 0.528 - Validation_Accuracy: 0.847\n",
            "[120,   200] loss: 0.189 - Training_Accuracy: 0.925\n",
            "[120,   400] loss: 0.214 - Training_Accuracy: 0.914\n",
            "[120,   600] loss: 0.196 - Training_Accuracy: 0.927\n",
            "[120,   800] loss: 0.190 - Training_Accuracy: 0.923\n",
            "[120,  1000] loss: 0.185 - Training_Accuracy: 0.926\n",
            "[120,  1200] loss: 0.201 - Training_Accuracy: 0.919\n",
            "[120,  1400] loss: 0.195 - Training_Accuracy: 0.921\n",
            "[120,  1600] loss: 0.203 - Training_Accuracy: 0.918\n",
            "[120,  1800] loss: 0.194 - Training_Accuracy: 0.926\n",
            "[120,  2000] loss: 0.206 - Training_Accuracy: 0.917\n",
            "[120,  2000] Validation loss: 0.498 - Validation_Accuracy: 0.856\n",
            "[121,   200] loss: 0.187 - Training_Accuracy: 0.929\n",
            "[121,   400] loss: 0.193 - Training_Accuracy: 0.920\n",
            "[121,   600] loss: 0.185 - Training_Accuracy: 0.924\n",
            "[121,   800] loss: 0.187 - Training_Accuracy: 0.925\n",
            "[121,  1000] loss: 0.183 - Training_Accuracy: 0.924\n",
            "[121,  1200] loss: 0.199 - Training_Accuracy: 0.919\n",
            "[121,  1400] loss: 0.193 - Training_Accuracy: 0.925\n",
            "[121,  1600] loss: 0.194 - Training_Accuracy: 0.921\n",
            "[121,  1800] loss: 0.200 - Training_Accuracy: 0.919\n",
            "[121,  2000] loss: 0.196 - Training_Accuracy: 0.920\n",
            "[121,  2000] Validation loss: 0.530 - Validation_Accuracy: 0.847\n",
            "[122,   200] loss: 0.205 - Training_Accuracy: 0.920\n",
            "[122,   400] loss: 0.194 - Training_Accuracy: 0.923\n",
            "[122,   600] loss: 0.192 - Training_Accuracy: 0.922\n",
            "[122,   800] loss: 0.195 - Training_Accuracy: 0.922\n",
            "[122,  1000] loss: 0.188 - Training_Accuracy: 0.926\n",
            "[122,  1200] loss: 0.182 - Training_Accuracy: 0.926\n",
            "[122,  1400] loss: 0.185 - Training_Accuracy: 0.924\n",
            "[122,  1600] loss: 0.180 - Training_Accuracy: 0.931\n",
            "[122,  1800] loss: 0.195 - Training_Accuracy: 0.921\n",
            "[122,  2000] loss: 0.184 - Training_Accuracy: 0.929\n",
            "[122,  2000] Validation loss: 0.501 - Validation_Accuracy: 0.856\n",
            "[123,   200] loss: 0.168 - Training_Accuracy: 0.935\n",
            "[123,   400] loss: 0.178 - Training_Accuracy: 0.927\n",
            "[123,   600] loss: 0.194 - Training_Accuracy: 0.921\n",
            "[123,   800] loss: 0.182 - Training_Accuracy: 0.929\n",
            "[123,  1000] loss: 0.197 - Training_Accuracy: 0.921\n",
            "[123,  1200] loss: 0.191 - Training_Accuracy: 0.924\n",
            "[123,  1400] loss: 0.203 - Training_Accuracy: 0.919\n",
            "[123,  1600] loss: 0.198 - Training_Accuracy: 0.923\n",
            "[123,  1800] loss: 0.178 - Training_Accuracy: 0.928\n",
            "[123,  2000] loss: 0.189 - Training_Accuracy: 0.921\n",
            "[123,  2000] Validation loss: 0.495 - Validation_Accuracy: 0.857\n",
            "[124,   200] loss: 0.172 - Training_Accuracy: 0.933\n",
            "[124,   400] loss: 0.189 - Training_Accuracy: 0.925\n",
            "[124,   600] loss: 0.190 - Training_Accuracy: 0.923\n",
            "[124,   800] loss: 0.210 - Training_Accuracy: 0.915\n",
            "[124,  1000] loss: 0.203 - Training_Accuracy: 0.924\n",
            "[124,  1200] loss: 0.202 - Training_Accuracy: 0.922\n",
            "[124,  1400] loss: 0.258 - Training_Accuracy: 0.900\n",
            "[124,  1600] loss: 0.195 - Training_Accuracy: 0.922\n",
            "[124,  1800] loss: 0.208 - Training_Accuracy: 0.918\n",
            "[124,  2000] loss: 0.215 - Training_Accuracy: 0.916\n",
            "[124,  2000] Validation loss: 0.532 - Validation_Accuracy: 0.845\n",
            "[125,   200] loss: 0.204 - Training_Accuracy: 0.922\n",
            "[125,   400] loss: 0.226 - Training_Accuracy: 0.914\n",
            "[125,   600] loss: 0.207 - Training_Accuracy: 0.920\n",
            "[125,   800] loss: 0.232 - Training_Accuracy: 0.915\n",
            "[125,  1000] loss: 0.208 - Training_Accuracy: 0.922\n",
            "[125,  1200] loss: 0.203 - Training_Accuracy: 0.921\n",
            "[125,  1400] loss: 0.210 - Training_Accuracy: 0.918\n",
            "[125,  1600] loss: 0.195 - Training_Accuracy: 0.925\n",
            "[125,  1800] loss: 0.204 - Training_Accuracy: 0.915\n",
            "[125,  2000] loss: 0.190 - Training_Accuracy: 0.925\n",
            "[125,  2000] Validation loss: 0.517 - Validation_Accuracy: 0.851\n",
            "[126,   200] loss: 0.175 - Training_Accuracy: 0.930\n",
            "[126,   400] loss: 0.209 - Training_Accuracy: 0.915\n",
            "[126,   600] loss: 0.182 - Training_Accuracy: 0.924\n",
            "[126,   800] loss: 0.189 - Training_Accuracy: 0.925\n",
            "[126,  1000] loss: 0.194 - Training_Accuracy: 0.924\n",
            "[126,  1200] loss: 0.169 - Training_Accuracy: 0.929\n",
            "[126,  1400] loss: 0.181 - Training_Accuracy: 0.930\n",
            "[126,  1600] loss: 0.179 - Training_Accuracy: 0.924\n",
            "[126,  1800] loss: 0.196 - Training_Accuracy: 0.921\n",
            "[126,  2000] loss: 0.205 - Training_Accuracy: 0.919\n",
            "[126,  2000] Validation loss: 0.499 - Validation_Accuracy: 0.856\n",
            "[127,   200] loss: 0.174 - Training_Accuracy: 0.930\n",
            "[127,   400] loss: 0.191 - Training_Accuracy: 0.925\n",
            "[127,   600] loss: 0.202 - Training_Accuracy: 0.921\n",
            "[127,   800] loss: 0.197 - Training_Accuracy: 0.921\n",
            "[127,  1000] loss: 0.202 - Training_Accuracy: 0.917\n",
            "[127,  1200] loss: 0.232 - Training_Accuracy: 0.909\n",
            "[127,  1400] loss: 0.182 - Training_Accuracy: 0.927\n",
            "[127,  1600] loss: 0.200 - Training_Accuracy: 0.921\n",
            "[127,  1800] loss: 0.174 - Training_Accuracy: 0.928\n",
            "[127,  2000] loss: 0.188 - Training_Accuracy: 0.925\n",
            "[127,  2000] Validation loss: 0.517 - Validation_Accuracy: 0.852\n",
            "[128,   200] loss: 0.181 - Training_Accuracy: 0.927\n",
            "[128,   400] loss: 0.178 - Training_Accuracy: 0.928\n",
            "[128,   600] loss: 0.191 - Training_Accuracy: 0.926\n",
            "[128,   800] loss: 0.177 - Training_Accuracy: 0.930\n",
            "[128,  1000] loss: 0.197 - Training_Accuracy: 0.926\n",
            "[128,  1200] loss: 0.201 - Training_Accuracy: 0.919\n",
            "[128,  1400] loss: 0.183 - Training_Accuracy: 0.923\n",
            "[128,  1600] loss: 0.175 - Training_Accuracy: 0.926\n",
            "[128,  1800] loss: 0.187 - Training_Accuracy: 0.925\n",
            "[128,  2000] loss: 0.196 - Training_Accuracy: 0.926\n",
            "[128,  2000] Validation loss: 0.531 - Validation_Accuracy: 0.853\n",
            "[129,   200] loss: 0.200 - Training_Accuracy: 0.919\n",
            "[129,   400] loss: 0.197 - Training_Accuracy: 0.926\n",
            "[129,   600] loss: 0.201 - Training_Accuracy: 0.920\n",
            "[129,   800] loss: 0.225 - Training_Accuracy: 0.912\n",
            "[129,  1000] loss: 0.196 - Training_Accuracy: 0.922\n",
            "[129,  1200] loss: 0.200 - Training_Accuracy: 0.922\n",
            "[129,  1400] loss: 0.204 - Training_Accuracy: 0.920\n",
            "[129,  1600] loss: 0.223 - Training_Accuracy: 0.915\n",
            "[129,  1800] loss: 0.207 - Training_Accuracy: 0.916\n",
            "[129,  2000] loss: 0.216 - Training_Accuracy: 0.918\n",
            "[129,  2000] Validation loss: 0.572 - Validation_Accuracy: 0.832\n",
            "[130,   200] loss: 0.207 - Training_Accuracy: 0.920\n",
            "[130,   400] loss: 0.198 - Training_Accuracy: 0.917\n",
            "[130,   600] loss: 0.195 - Training_Accuracy: 0.921\n",
            "[130,   800] loss: 0.210 - Training_Accuracy: 0.916\n",
            "[130,  1000] loss: 0.201 - Training_Accuracy: 0.921\n",
            "[130,  1200] loss: 0.236 - Training_Accuracy: 0.910\n",
            "[130,  1400] loss: 0.232 - Training_Accuracy: 0.910\n",
            "[130,  1600] loss: 0.223 - Training_Accuracy: 0.912\n",
            "[130,  1800] loss: 0.227 - Training_Accuracy: 0.910\n",
            "[130,  2000] loss: 0.231 - Training_Accuracy: 0.912\n",
            "[130,  2000] Validation loss: 0.580 - Validation_Accuracy: 0.834\n",
            "[131,   200] loss: 0.223 - Training_Accuracy: 0.913\n",
            "[131,   400] loss: 0.229 - Training_Accuracy: 0.911\n",
            "[131,   600] loss: 0.215 - Training_Accuracy: 0.918\n",
            "[131,   800] loss: 0.205 - Training_Accuracy: 0.917\n",
            "[131,  1000] loss: 0.199 - Training_Accuracy: 0.922\n",
            "[131,  1200] loss: 0.243 - Training_Accuracy: 0.906\n",
            "[131,  1400] loss: 0.231 - Training_Accuracy: 0.908\n",
            "[131,  1600] loss: 0.218 - Training_Accuracy: 0.916\n",
            "[131,  1800] loss: 0.232 - Training_Accuracy: 0.907\n",
            "[131,  2000] loss: 0.247 - Training_Accuracy: 0.905\n",
            "[131,  2000] Validation loss: 0.578 - Validation_Accuracy: 0.832\n",
            "[132,   200] loss: 0.219 - Training_Accuracy: 0.916\n",
            "[132,   400] loss: 0.206 - Training_Accuracy: 0.919\n",
            "[132,   600] loss: 0.242 - Training_Accuracy: 0.905\n",
            "[132,   800] loss: 0.240 - Training_Accuracy: 0.909\n",
            "[132,  1000] loss: 0.223 - Training_Accuracy: 0.908\n",
            "[132,  1200] loss: 0.212 - Training_Accuracy: 0.917\n",
            "[132,  1400] loss: 0.220 - Training_Accuracy: 0.914\n",
            "[132,  1600] loss: 0.205 - Training_Accuracy: 0.915\n",
            "[132,  1800] loss: 0.216 - Training_Accuracy: 0.914\n",
            "[132,  2000] loss: 0.206 - Training_Accuracy: 0.918\n",
            "[132,  2000] Validation loss: 0.543 - Validation_Accuracy: 0.842\n",
            "[133,   200] loss: 0.194 - Training_Accuracy: 0.922\n",
            "[133,   400] loss: 0.184 - Training_Accuracy: 0.929\n",
            "[133,   600] loss: 0.174 - Training_Accuracy: 0.930\n",
            "[133,   800] loss: 0.181 - Training_Accuracy: 0.924\n",
            "[133,  1000] loss: 0.194 - Training_Accuracy: 0.922\n",
            "[133,  1200] loss: 0.201 - Training_Accuracy: 0.919\n",
            "[133,  1400] loss: 0.207 - Training_Accuracy: 0.918\n",
            "[133,  1600] loss: 0.218 - Training_Accuracy: 0.914\n",
            "[133,  1800] loss: 0.226 - Training_Accuracy: 0.911\n",
            "[133,  2000] loss: 0.242 - Training_Accuracy: 0.904\n",
            "[133,  2000] Validation loss: 0.580 - Validation_Accuracy: 0.832\n",
            "[134,   200] loss: 0.208 - Training_Accuracy: 0.919\n",
            "[134,   400] loss: 0.196 - Training_Accuracy: 0.923\n",
            "[134,   600] loss: 0.198 - Training_Accuracy: 0.923\n",
            "[134,   800] loss: 0.208 - Training_Accuracy: 0.918\n",
            "[134,  1000] loss: 0.222 - Training_Accuracy: 0.913\n",
            "[134,  1200] loss: 0.221 - Training_Accuracy: 0.914\n",
            "[134,  1400] loss: 0.217 - Training_Accuracy: 0.918\n",
            "[134,  1600] loss: 0.216 - Training_Accuracy: 0.915\n",
            "[134,  1800] loss: 0.196 - Training_Accuracy: 0.920\n",
            "[134,  2000] loss: 0.223 - Training_Accuracy: 0.911\n",
            "[134,  2000] Validation loss: 0.583 - Validation_Accuracy: 0.834\n",
            "[135,   200] loss: 0.203 - Training_Accuracy: 0.919\n",
            "[135,   400] loss: 0.212 - Training_Accuracy: 0.917\n",
            "[135,   600] loss: 0.186 - Training_Accuracy: 0.926\n",
            "[135,   800] loss: 0.202 - Training_Accuracy: 0.917\n",
            "[135,  1000] loss: 0.192 - Training_Accuracy: 0.928\n",
            "[135,  1200] loss: 0.200 - Training_Accuracy: 0.921\n",
            "[135,  1400] loss: 0.203 - Training_Accuracy: 0.918\n",
            "[135,  1600] loss: 0.187 - Training_Accuracy: 0.927\n",
            "[135,  1800] loss: 0.197 - Training_Accuracy: 0.924\n",
            "[135,  2000] loss: 0.182 - Training_Accuracy: 0.922\n",
            "[135,  2000] Validation loss: 0.521 - Validation_Accuracy: 0.853\n",
            "[136,   200] loss: 0.183 - Training_Accuracy: 0.927\n",
            "[136,   400] loss: 0.176 - Training_Accuracy: 0.930\n",
            "[136,   600] loss: 0.173 - Training_Accuracy: 0.927\n",
            "[136,   800] loss: 0.177 - Training_Accuracy: 0.927\n",
            "[136,  1000] loss: 0.182 - Training_Accuracy: 0.929\n",
            "[136,  1200] loss: 0.196 - Training_Accuracy: 0.922\n",
            "[136,  1400] loss: 0.190 - Training_Accuracy: 0.922\n",
            "[136,  1600] loss: 0.185 - Training_Accuracy: 0.923\n",
            "[136,  1800] loss: 0.236 - Training_Accuracy: 0.910\n",
            "[136,  2000] loss: 0.227 - Training_Accuracy: 0.912\n",
            "[136,  2000] Validation loss: 0.552 - Validation_Accuracy: 0.844\n",
            "[137,   200] loss: 0.208 - Training_Accuracy: 0.917\n",
            "[137,   400] loss: 0.179 - Training_Accuracy: 0.930\n",
            "[137,   600] loss: 0.188 - Training_Accuracy: 0.927\n",
            "[137,   800] loss: 0.195 - Training_Accuracy: 0.922\n",
            "[137,  1000] loss: 0.179 - Training_Accuracy: 0.927\n",
            "[137,  1200] loss: 0.179 - Training_Accuracy: 0.929\n",
            "[137,  1400] loss: 0.179 - Training_Accuracy: 0.927\n",
            "[137,  1600] loss: 0.200 - Training_Accuracy: 0.924\n",
            "[137,  1800] loss: 0.204 - Training_Accuracy: 0.919\n",
            "[137,  2000] loss: 0.169 - Training_Accuracy: 0.933\n",
            "[137,  2000] Validation loss: 0.522 - Validation_Accuracy: 0.853\n",
            "[138,   200] loss: 0.184 - Training_Accuracy: 0.929\n",
            "[138,   400] loss: 0.176 - Training_Accuracy: 0.933\n",
            "[138,   600] loss: 0.174 - Training_Accuracy: 0.931\n",
            "[138,   800] loss: 0.189 - Training_Accuracy: 0.926\n",
            "[138,  1000] loss: 0.171 - Training_Accuracy: 0.932\n",
            "[138,  1200] loss: 0.167 - Training_Accuracy: 0.934\n",
            "[138,  1400] loss: 0.175 - Training_Accuracy: 0.931\n",
            "[138,  1600] loss: 0.170 - Training_Accuracy: 0.933\n",
            "[138,  1800] loss: 0.172 - Training_Accuracy: 0.932\n",
            "[138,  2000] loss: 0.171 - Training_Accuracy: 0.933\n",
            "[138,  2000] Validation loss: 0.520 - Validation_Accuracy: 0.853\n",
            "[139,   200] loss: 0.182 - Training_Accuracy: 0.928\n",
            "[139,   400] loss: 0.186 - Training_Accuracy: 0.928\n",
            "[139,   600] loss: 0.166 - Training_Accuracy: 0.933\n",
            "[139,   800] loss: 0.171 - Training_Accuracy: 0.936\n",
            "[139,  1000] loss: 0.179 - Training_Accuracy: 0.931\n",
            "[139,  1200] loss: 0.187 - Training_Accuracy: 0.924\n",
            "[139,  1400] loss: 0.186 - Training_Accuracy: 0.929\n",
            "[139,  1600] loss: 0.179 - Training_Accuracy: 0.930\n",
            "[139,  1800] loss: 0.176 - Training_Accuracy: 0.928\n",
            "[139,  2000] loss: 0.186 - Training_Accuracy: 0.928\n",
            "[139,  2000] Validation loss: 0.562 - Validation_Accuracy: 0.846\n",
            "[140,   200] loss: 0.170 - Training_Accuracy: 0.932\n",
            "[140,   400] loss: 0.180 - Training_Accuracy: 0.930\n",
            "[140,   600] loss: 0.192 - Training_Accuracy: 0.919\n",
            "[140,   800] loss: 0.183 - Training_Accuracy: 0.927\n",
            "[140,  1000] loss: 0.177 - Training_Accuracy: 0.933\n",
            "[140,  1200] loss: 0.174 - Training_Accuracy: 0.932\n",
            "[140,  1400] loss: 0.179 - Training_Accuracy: 0.931\n",
            "[140,  1600] loss: 0.176 - Training_Accuracy: 0.928\n",
            "[140,  1800] loss: 0.181 - Training_Accuracy: 0.928\n",
            "[140,  2000] loss: 0.171 - Training_Accuracy: 0.932\n",
            "[140,  2000] Validation loss: 0.546 - Validation_Accuracy: 0.852\n",
            "[141,   200] loss: 0.174 - Training_Accuracy: 0.932\n",
            "[141,   400] loss: 0.178 - Training_Accuracy: 0.933\n",
            "[141,   600] loss: 0.189 - Training_Accuracy: 0.924\n",
            "[141,   800] loss: 0.216 - Training_Accuracy: 0.917\n",
            "[141,  1000] loss: 0.193 - Training_Accuracy: 0.922\n",
            "[141,  1200] loss: 0.205 - Training_Accuracy: 0.918\n",
            "[141,  1400] loss: 0.215 - Training_Accuracy: 0.918\n",
            "[141,  1600] loss: 0.205 - Training_Accuracy: 0.919\n",
            "[141,  1800] loss: 0.195 - Training_Accuracy: 0.927\n",
            "[141,  2000] loss: 0.195 - Training_Accuracy: 0.921\n",
            "[141,  2000] Validation loss: 0.529 - Validation_Accuracy: 0.853\n",
            "[142,   200] loss: 0.193 - Training_Accuracy: 0.923\n",
            "[142,   400] loss: 0.177 - Training_Accuracy: 0.925\n",
            "[142,   600] loss: 0.178 - Training_Accuracy: 0.928\n",
            "[142,   800] loss: 0.183 - Training_Accuracy: 0.928\n",
            "[142,  1000] loss: 0.182 - Training_Accuracy: 0.924\n",
            "[142,  1200] loss: 0.174 - Training_Accuracy: 0.931\n",
            "[142,  1400] loss: 0.176 - Training_Accuracy: 0.931\n",
            "[142,  1600] loss: 0.170 - Training_Accuracy: 0.931\n",
            "[142,  1800] loss: 0.176 - Training_Accuracy: 0.933\n",
            "[142,  2000] loss: 0.179 - Training_Accuracy: 0.931\n",
            "[142,  2000] Validation loss: 0.518 - Validation_Accuracy: 0.858\n",
            "[143,   200] loss: 0.165 - Training_Accuracy: 0.935\n",
            "[143,   400] loss: 0.175 - Training_Accuracy: 0.932\n",
            "[143,   600] loss: 0.212 - Training_Accuracy: 0.922\n",
            "[143,   800] loss: 0.224 - Training_Accuracy: 0.910\n",
            "[143,  1000] loss: 0.216 - Training_Accuracy: 0.920\n",
            "[143,  1200] loss: 0.203 - Training_Accuracy: 0.924\n",
            "[143,  1400] loss: 0.200 - Training_Accuracy: 0.922\n",
            "[143,  1600] loss: 0.186 - Training_Accuracy: 0.925\n",
            "[143,  1800] loss: 0.212 - Training_Accuracy: 0.918\n",
            "[143,  2000] loss: 0.207 - Training_Accuracy: 0.920\n",
            "[143,  2000] Validation loss: 0.551 - Validation_Accuracy: 0.847\n",
            "[144,   200] loss: 0.213 - Training_Accuracy: 0.916\n",
            "[144,   400] loss: 0.235 - Training_Accuracy: 0.914\n",
            "[144,   600] loss: 0.217 - Training_Accuracy: 0.918\n",
            "[144,   800] loss: 0.199 - Training_Accuracy: 0.922\n",
            "[144,  1000] loss: 0.191 - Training_Accuracy: 0.925\n",
            "[144,  1200] loss: 0.203 - Training_Accuracy: 0.920\n",
            "[144,  1400] loss: 0.199 - Training_Accuracy: 0.922\n",
            "[144,  1600] loss: 0.194 - Training_Accuracy: 0.923\n",
            "[144,  1800] loss: 0.203 - Training_Accuracy: 0.919\n",
            "[144,  2000] loss: 0.192 - Training_Accuracy: 0.925\n",
            "[144,  2000] Validation loss: 0.550 - Validation_Accuracy: 0.847\n",
            "[145,   200] loss: 0.191 - Training_Accuracy: 0.922\n",
            "[145,   400] loss: 0.196 - Training_Accuracy: 0.926\n",
            "[145,   600] loss: 0.206 - Training_Accuracy: 0.920\n",
            "[145,   800] loss: 0.196 - Training_Accuracy: 0.923\n",
            "[145,  1000] loss: 0.201 - Training_Accuracy: 0.923\n",
            "[145,  1200] loss: 0.191 - Training_Accuracy: 0.927\n",
            "[145,  1400] loss: 0.200 - Training_Accuracy: 0.920\n",
            "[145,  1600] loss: 0.234 - Training_Accuracy: 0.909\n",
            "[145,  1800] loss: 0.215 - Training_Accuracy: 0.915\n",
            "[145,  2000] loss: 0.205 - Training_Accuracy: 0.920\n",
            "[145,  2000] Validation loss: 0.581 - Validation_Accuracy: 0.839\n",
            "[146,   200] loss: 0.189 - Training_Accuracy: 0.920\n",
            "[146,   400] loss: 0.188 - Training_Accuracy: 0.924\n",
            "[146,   600] loss: 0.219 - Training_Accuracy: 0.917\n",
            "[146,   800] loss: 0.194 - Training_Accuracy: 0.925\n",
            "[146,  1000] loss: 0.178 - Training_Accuracy: 0.930\n",
            "[146,  1200] loss: 0.166 - Training_Accuracy: 0.936\n",
            "[146,  1400] loss: 0.169 - Training_Accuracy: 0.933\n",
            "[146,  1600] loss: 0.178 - Training_Accuracy: 0.931\n",
            "[146,  1800] loss: 0.175 - Training_Accuracy: 0.929\n",
            "[146,  2000] loss: 0.178 - Training_Accuracy: 0.929\n",
            "[146,  2000] Validation loss: 0.546 - Validation_Accuracy: 0.850\n",
            "[147,   200] loss: 0.189 - Training_Accuracy: 0.928\n",
            "[147,   400] loss: 0.210 - Training_Accuracy: 0.914\n",
            "[147,   600] loss: 0.219 - Training_Accuracy: 0.910\n",
            "[147,   800] loss: 0.231 - Training_Accuracy: 0.910\n",
            "[147,  1000] loss: 0.221 - Training_Accuracy: 0.915\n",
            "[147,  1200] loss: 0.224 - Training_Accuracy: 0.913\n",
            "[147,  1400] loss: 0.197 - Training_Accuracy: 0.919\n",
            "[147,  1600] loss: 0.190 - Training_Accuracy: 0.926\n",
            "[147,  1800] loss: 0.185 - Training_Accuracy: 0.922\n",
            "[147,  2000] loss: 0.193 - Training_Accuracy: 0.918\n",
            "[147,  2000] Validation loss: 0.592 - Validation_Accuracy: 0.840\n",
            "[148,   200] loss: 0.205 - Training_Accuracy: 0.921\n",
            "[148,   400] loss: 0.190 - Training_Accuracy: 0.928\n",
            "[148,   600] loss: 0.191 - Training_Accuracy: 0.924\n",
            "[148,   800] loss: 0.193 - Training_Accuracy: 0.924\n",
            "[148,  1000] loss: 0.185 - Training_Accuracy: 0.931\n",
            "[148,  1200] loss: 0.180 - Training_Accuracy: 0.929\n",
            "[148,  1400] loss: 0.183 - Training_Accuracy: 0.930\n",
            "[148,  1600] loss: 0.184 - Training_Accuracy: 0.924\n",
            "[148,  1800] loss: 0.184 - Training_Accuracy: 0.928\n",
            "[148,  2000] loss: 0.187 - Training_Accuracy: 0.925\n",
            "[148,  2000] Validation loss: 0.517 - Validation_Accuracy: 0.854\n",
            "[149,   200] loss: 0.182 - Training_Accuracy: 0.926\n",
            "[149,   400] loss: 0.179 - Training_Accuracy: 0.929\n",
            "[149,   600] loss: 0.195 - Training_Accuracy: 0.922\n",
            "[149,   800] loss: 0.184 - Training_Accuracy: 0.926\n",
            "[149,  1000] loss: 0.191 - Training_Accuracy: 0.922\n",
            "[149,  1200] loss: 0.188 - Training_Accuracy: 0.921\n",
            "[149,  1400] loss: 0.196 - Training_Accuracy: 0.925\n",
            "[149,  1600] loss: 0.183 - Training_Accuracy: 0.929\n",
            "[149,  1800] loss: 0.174 - Training_Accuracy: 0.930\n",
            "[149,  2000] loss: 0.159 - Training_Accuracy: 0.934\n",
            "[149,  2000] Validation loss: 0.535 - Validation_Accuracy: 0.857\n",
            "[150,   200] loss: 0.160 - Training_Accuracy: 0.934\n",
            "[150,   400] loss: 0.157 - Training_Accuracy: 0.937\n",
            "[150,   600] loss: 0.164 - Training_Accuracy: 0.934\n",
            "[150,   800] loss: 0.164 - Training_Accuracy: 0.931\n",
            "[150,  1000] loss: 0.176 - Training_Accuracy: 0.931\n",
            "[150,  1200] loss: 0.177 - Training_Accuracy: 0.930\n",
            "[150,  1400] loss: 0.188 - Training_Accuracy: 0.926\n",
            "[150,  1600] loss: 0.197 - Training_Accuracy: 0.922\n",
            "[150,  1800] loss: 0.172 - Training_Accuracy: 0.928\n",
            "[150,  2000] loss: 0.171 - Training_Accuracy: 0.932\n",
            "[150,  2000] Validation loss: 0.529 - Validation_Accuracy: 0.854\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-nLcJY2Rvumf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4.3 Results Visualization"
      ]
    },
    {
      "metadata": {
        "id": "WTbBCBvsIw3y",
        "colab_type": "code",
        "outputId": "aee3de17-7f5a-43b6-b8e0-89771b008ff4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 244
        }
      },
      "cell_type": "code",
      "source": [
        "training_examples = 6e4\n",
        "plot_every = 200 # batches\n",
        "training_loss_np = np.asarray(training_loss_list)\n",
        "validation_loss_np = np.asarray(validation_loss_list)\n",
        "training_accuracy_np = np.asarray(training_accuracy_list)\n",
        "validation_accuracy_np = np.asarray(validation_accuracy_list)\n",
        "\n",
        "x_axis_train = np.arange(1, len(training_loss_list)+1)\n",
        "x_axis_validation = np.arange(1, len(validation_loss_list)+1)\n",
        "\n",
        "p1 = plt.plot(x_axis_train * bs * plot_every / training_examples, training_loss_np)\n",
        "p2 = plt.plot(x_axis_validation, validation_loss_np,color=\"r\")\n",
        "\n",
        "plt.xlabel('epochs')\n",
        "plt.title('Loss')\n",
        "plt.legend((p1[0], p2[0]), ('Training Loss', 'Validation Loss'))\n",
        "plt.show(),\n",
        "\n",
        "p3 = plt.plot(x_axis_train * bs * plot_every / training_examples, training_accuracy_np)\n",
        "p4 = plt.plot(x_axis_validation, validation_accuracy_np,color=\"r\")\n",
        "plt.xlabel('epochs')\n",
        "plt.title('Accuracy')\n",
        "plt.legend((p3[0], p4[0]), ('Training Accuracy', 'Validation Accuracy'))\n",
        "plt.show()"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-4e0178f869f3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mtraining_examples\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m6e4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mplot_every\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m200\u001b[0m \u001b[0;31m#batches\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mtraining_loss_np\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtraining_loss_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mvalidation_loss_np\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalidation_loss_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mtraining_accuracy_np\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtraining_accuracy_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'np' is not defined"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "VYGMvzBa1mky",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **5. Network Testing**"
      ]
    },
    {
      "metadata": {
        "id": "RvqA67EGv4tK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let us evaluate the model with data that it has never seen before, the test data. We set the model to evaluation, doing this we force PyTorch to not track all the grad_fn that it uses to do backpropagation, saving resources."
      ]
    },
    {
      "metadata": {
        "id": "YS9n4-vr4F-w",
        "colab_type": "code",
        "outputId": "dbad6660-a7cb-4ff7-e314-fc8fc9d6ee94",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "cell_type": "code",
      "source": [
        "model.eval()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Quick_draw_LSTM(\n",
              "  (lstm): LSTM(2, 512, num_layers=3)\n",
              "  (hidden_to_class): Linear(in_features=512, out_features=10, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "metadata": {
        "id": "GNp8rDLJw8c6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For testing our network we have to iterate over the test set simmilarly than when we did with the validation set. Hopefully, our test accuracy will be high if our model has been able to generalize well."
      ]
    },
    {
      "metadata": {
        "id": "GuiC_DNl1q_n",
        "colab_type": "code",
        "outputId": "34adbd17-e5ee-4f39-d4a9-0eca01f2bc9b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "running_test_loss=0.0\n",
        "test_total=0.0\n",
        "test_correct=0.0\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "for i, test_data in enumerate(test_loader,0): \n",
        "  test_inputs, test_labels, test_lengths = test_data\n",
        "  \n",
        "  test_inputs = test_inputs.float().to(device)\n",
        "  test_labels = test_labels.to(device)\n",
        "  test_lengths = torch.from_numpy(test_lengths.copy()).to(device)\n",
        "  test_outputs = model(test_inputs, test_lengths)\n",
        "  test_valid_outputs = test_outputs[np.array(test_lengths-1), np.arange(0, bs), :]\n",
        "  \n",
        "  test_loss = criterion(test_valid_outputs, test_labels)\n",
        "  running_test_loss += test_loss.item()\n",
        "  \n",
        "  _,predicted = torch.max(test_valid_outputs.data,1)\n",
        "  test_total = test_total+test_labels.size(0)\n",
        "  test_correct = test_correct + (predicted == test_labels).sum().item()        \n",
        "  \n",
        "test_accuracy = test_correct/test_total\n",
        "  \n",
        "print('Test Loss: %.3f - Test Accuracy: %.3f' %\n",
        "         (running_test_loss/len(test_loader), test_accuracy))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Loss: 0.426 - Test Accuracy: 0.854\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}