Notebooks/Fashion_MNIST_using_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R6gHiH-I7uFa"
   },
   "source": [
    "[![Google Colab](https://badgen.net/badge/Launch/on%20Google%20Colab/blue?icon=terminal)](https://colab.research.google.com/github/tirthajyoti/Computer_vision/blob/master/Fashion_MNIST_using_CNN.ipynb)\n",
    "\n",
    "# Improving Computer Vision Accuracy using Convolutions\n",
    "### Dr. Tirthajyoti Sarkar, Fremont CA 94536\n",
    "---\n",
    "\n",
    "This is a notebook made using **[Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb)** where deep learning networks are trained on [Google's GPU cluster](https://cloud.google.com/gpu/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sBrgJnw3A_39"
   },
   "source": [
    "## What are we doing?\n",
    "\n",
    "In this notebook, we will first see how to do fashion image recognition using a Deep Neural Network (DNN) containing three layers -- the input layer (in the shape of the data), the output layer (in the shape of the desired output) and a hidden layer. \n",
    "\n",
    "We will use a dataset called '[Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist)' as it consists of small images (28 pixels x 28 pixles) of fashion items, much in the same spirit as that of the [original famous MNIST dataset](http://yann.lecun.com/exdb/mnist/).\n",
    "\n",
    "We can experiment with the impact of different sized of hidden layer, number of training epochs etc on the final accuracy.\n",
    "\n",
    "After that, we will introduce [convolutional and pooling layers](http://cs231n.github.io/convolutional-networks/) and see their impact on the image classification task.\n",
    "\n",
    "![alt text](http://cs231n.github.io/assets/nn1/neural_net2.jpeg)\n",
    "\n",
    "![alt text](http://cs231n.github.io/assets/cnn/cnn.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3XniaZ3f2Nm5"
   },
   "source": [
    "We will keep track of wall clock time using the **`time.time()`** function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AafIMtq3AVe9"
   },
   "source": [
    "## First, we use simple densely connected neural net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pib5cCPf2c9t"
   },
   "outputs": [],
   "source": [
    "from time import time\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xcsRtq9OLorS"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nO9YE5BS-5kQ"
   },
   "source": [
    "### Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ny6m36-VFbGW"
   },
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LxwQaGOW-8TW"
   },
   "source": [
    "### Show few images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 318
    },
    "colab_type": "code",
    "id": "-gM5F_By9bCa",
    "outputId": "e43796e6-d112-4d27-8ff4-bb1a524a1655"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAEuCAYAAAD/feFYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XeYXHd1P/7zmb59VVarXi3Zlpvc\nKxg3XIAYMMU0m+BgAoRfSAjBgW8CKZTQQwnEAWMgYBIwYIrAxsYFF2xZstwkq3dppV1J23fqvb8/\nvFj7PudqZ3ZmtHtHer+ex4917t575+7cM7fs3PM5zvd9ISIiIiIiovCITPQGEBEREREREeKNGhER\nERERUcjwRo2IiIiIiChkeKNGREREREQUMrxRIyIiIiIiChneqBEREREREYUMb9SIiIiIiIhChjdq\nREREREREIVPRjZpz7irn3Drn3Ebn3C3V2ig6NjB/qFzMHaoE84fKxdyhSjB/aKyc7/vlLehcVETW\ni8gVIrJTRFaIyFt8319zuGUSLumnpKGs16NwS8uAZP2MK3X+seZPTeVOQx2EsTlZiIe6U2aR2CB+\nDp2nPpcBH9N8Pf6dxbXk8efZGMSp3RmzDj+fN9MmQp8c7PJ9v62UeY+2Y49LxCHOtCUgTnbmIPaz\nmE9V04h5m6/D/Ip1Ddplyjx/VNOxcuxxdfa4kW2OQhxrxlzJFdTPD9i/zUb70xB79Ul8jVa7LZMb\nB/B1PHydgW7MpXgHzh8mx/KxJyxcEo95fuYIHeOq7Fg59gTR561cC+7D1BQ8rmTVsUhEJJfGdZjr\nnKg9v7TW43moe7AeX3cHvq7veWYdYVHqsSdWbIZRnCMiG33f3ywi4pz7kYhcKyKHPWClpEHOdZdV\n8JIUVo/79411kTHlzxHJHRdwfK3GhefJp0A46Uu7IH7ulyeYRaatwhNTNFOA2GXtwabrNDxARV+9\nH+L9WydBfMK/bjHrKOzdZ6ZNhHv9n2wbw+xH1bEnNnMOxBveMxvixf+9G+L8lrG8VaXzzjod4v1L\n8cZg2m2rzDJ+xt78j7eaPPaUIXKcPW7svmIyxJOuxlzZc7AZ4mk/whsoEZGmP2yEOH3GAoi3vN7e\n3L3tvMcg3pvB13nsp6dBPOvfHzXrCItj+dgTFtG5mHOFjfZcFUY1cezR1zlV+uOaPm/tuQbPW0ve\nvg7iHX32Lz57NuA9SkTd8xZa8DpIROTaM56C+K7VyyA+4YP4ul5fn1lHUUfq2lAp9dhTyaOPs0Rk\nx4h45/A04Jy72Tn3pHPuyZxM/EmdQqNo/jB36DB47KFK8NhD5eKxhyrBYw+N2REfTMT3/Vt93z/L\n9/2z4pIsvgDRMOYOVYL5Q+Vi7lAlmD9ULuYOaZU8+rhLREZ+9zl7eBpRKY58/hT7yr+Er7ILrzgD\n4k1vth+Zf77kpxCnfXyccH68E+Jp7/mNWceyZOUH5G/3TIc4txCfCX/363aI9kgG/1bz3qfeBvGs\nL6pnyEXEPbK63E2slpo99kQnTTLTtr8JHyF537XLIT74KqxReLZnplnHQC6pYqwXmN7Qa5ZpieOz\n/FdM+jnE//CH6yB2BfwsiIhMvfUxM60GhDJ/et96HsSz3ouPJB7M2BrBefFuXEcGH1c9ffZOiD/w\nhXvNOi5M4THgzn58jHHAw1wSEflDz/EQb+9Xj1m/ej3EF99w0KzjSysuh3jxO1eaeUIolLlTiimP\n2GPP8Y17IX6+bwbE/e+ZCnHheXysrBTR4xaYadf9Eo8b0+MvQPzrg/g429Yr7Pmx0N0z5m0JgYm/\n7gkQm41fCq/9e3yM8c8utJ/NSbFNEO/N4nVOUwzPL5+e/QuzjgWnNo66Xf1e2kxbPtgOcf5UvM5p\nexgfdVzbj9dFIiJP/nEJxMd/Dh+1zXfg52KiVfKN2goRWeycW+CcS4jI9SJi9wRRMOYPlYu5Q5Vg\n/lC5mDtUCeYPjVnZ36j5vp93zv2ViNwtIlERuc33/eertmV0VGP+ULmYO1QJ5g+Vi7lDlWD+UDkq\nefRRfN9fLiLLi85IFID5Q+Vi7lAlmD9ULuYOVYL5Q2N1xAcTISIiIiIiorGp6Bs1olArUkQbnTrF\nTBu6A4tb3zvvTogTzvb12JrFgut9WSzGf24AC3Xzvm38WBfBPmqL67CYdWcW+yWJiOTUejx/9L6b\nt6SnmWlT4/0Qf/ik30HcersdwODjz78G4umvXTvq69IhhYN2UIVED+bpHZ+5GuLzP7gC4nfOeMSs\n42WpLognRbHH3vPZIbPM1jwOLvChVW+EeObdmF/Z0eu+aQwip51opg28CQdHWLkWB2GI1Nvm9C6C\nueN7eAzYnsdj3McGXl902/Ie/v22EHBcOdCLA9wUCriMl8f4qZXHmXXEZ+CxZf2tZ0O85GbMe6pM\nMmrz59wGHBDi6uanIZ7+GxwefnMOz20iIu96+J0Q//rir0Gccg+bZTo9HBxkTQbPkfNS2BN0U3c4\nmz7XoqBjzzV34D6a0oMDcmzux2scEZGhvGp4rRpaD2RxEKKfPI99OkVE6hswv/RxJJu1tyjxOF6D\nzZ2M59TtMTyvNcZsi4PLXoZ53nk2ntz2fvd8s8yUb0/cwFn8Ro2IiIiIiChkeKNGREREREQUMrxR\nIyIiIiIiChnWqJVLNxUUKV4TNcXWGR28EhvvNf/wj2N+XRfDZ4X9XNbMM2ZBv59WQiPFMGu+y27/\n9VOw/ufxvkUQ67owEZG6aA7ioQLuj4jD10k4Wyug53lmAJsgxwJq47R4CfNo+7JNEHfl8FntoLq3\nfz3pLoi/fg42RpYnnh3zdhzLvAS+x7FuD+IHv3MOxPF32f18oID7bXIUaw/XphebZW5/ARsst3+/\nDuKeBZjrdZ24XVS+9R9OmWlelz22jKTr0UREkkk89uTzuI6cqhXbtt3WmkR68TLAS+F+dp49BviJ\nIrmgl4nZbS/swDrKthOxLqnn7ZifIiIt/1Pk/EiHtaG7zUzLTsF8WTU0H+Jlqe0Qvyxlz12Lb1wF\n8RcfvwLiD0+/xyzzbBrPbw0RrCN6tg9r1kS6hUpU5Lrs4KdzZtpj3Xids6UXr1VTMbvf9bVBRtWo\nOXVNo+vRREQyGTz25FVNWixuz3VN9dgEW9fKZQq4jt6MPdZGI3jd0xDHa+bj3mUbu/f+FGvfgurN\njxR+o0ZERERERBQyvFEjIiIiIiIKGd6oERERERERhQxr1MrkoraewM/jc7yRZUshXvse24gootob\nxQewHiU2hLUA8XuetK9brCYtqK5Nb7/De/ZS6txcbET62EeYQyd/6ZkQXzPlLjPPqoH5ENer/mbJ\ngF90WqIX4isasK/YzCg+qx139u8jfR6utz6C+yfj25oQvZamCPYtGfTwWfTNeftx/03fqbhMAdch\nAaWKaR+fCV//F/gM+JIn7DJ0ePF+zI/Bqbhnm7dhbqz4x7PMOu6bg/U86am445q32vyZ3oXP/w+2\nqb58Ol1KKFul0sz7nj1/9HwAjyMH92Mdhb/P1loMNqqdlB/9b68uG1BvNhWPcWaO3rieIi49tr/x\nRgJet9CM+de5qxXiJaxHq6pd22zf0IbFWDekj+37PexfFnVYHxTkj7vnQbxkju2BdrfqozY9jjVo\n7Un8LHQWfVU6nNjC+RCfMmWPmWfHAH726uN47ZAJuHaYnMI+iG11eJ6KOTzn5H17zMiqerKsh8fF\n1oTt/zkjhf0mMx7mrB4jIGNOZCJ7h/DYquvY2lPYR05EZN1bT4N42tcfNfMcKfxGjYiIiIiIKGR4\no0ZERERERBQyvFEjIiIiIiIKGdaolQnqs4bpGrUdV+Jzv287/w9mmUc6F0K8LTkd14mtjSR2+flm\nHUv+cxfE+a3Y+ySor4beVi06CXtGSMH2syj0jniOvAZaqu28FOuvpsT6zTyTYvjcte6blorYHiRd\nOXze+fr//BDEDbvxWe2mbbafSP8cfGa/cRfO40dsjUcki+stJFUPpWaM951uc/Zf3vIDiFcOLIBY\n1+iJiOR8XM+XLrkD4m/IcWYZOrxIXn94cF8PTh29v5aISH0X5kJjB64zVx9QFzkb96Nuw+f0ZtXA\nZ7xWBNUaD553AcTnXPkCxE88ZXvhOdWfLFKPn1fvAB5XgmrF/C48LkYzOE+hLuD8oV431of5lZuC\n5xcv4G/CkXqc5/gP4nlr7F0haTRN622tYeoKPJ95qo5oRxbr2npSG806vIuWqSmYg/sKA2aZiKpf\nanC4zLZB3XO2y6yDSpOf1gzxhS22tur33gkQN8fw+mNm0vaxG/TwuDE5hvtZXzvpfS5ie7/q/EsG\nXG9FBdejr0f06+gaNhERwbdEVvfNxh/HbG1c+hWqbu3rdrVHCr9RIyIiIiIiChneqBEREREREYUM\nb9SIiIiIiIhChjdqREREREREIcPBRMrkpYs3fsyejoNVvKHFFpDrwSkejGAh5K7fz4G4cKodAGPb\nF3EwC+8pLEqf8pwty25+Cpsedr18FsSdZ2KxeHtA79FJ92566d/uQPhT6dVXPw7xgGq6KWL3h270\nODVmGyFuGGqHeOZnsVi3783YjHjvOWqEGBGZ8QVcZtctuA+nPmuLanNTsUjWj+IgAPUdWKA97+O2\nE3X6zbgOPXjI1Lj9fXfncJCc97Y+D/E3z7wWt2sl/pyQHijGqcF/Iurj6wWMLZJurcLf3PQ4E2oM\nCS/GjtdH0tx/wWPAa9+2DeKn2/EYLSKS3o/HksIgJkdsEPMi1l98H5qBQgZsbqn6ffHiKmf7VfP0\nZjt4Vds92GS20LW/6LZR+Rp32sEc9DlQD+7QFMXrnPuH2sw6fvW//w3x5hyeq347gA2wRURSDufR\nA0Ds6m+BuJmDiZSt83RsOK7fexGRC1o2QazzIO7s57crjyNyPHxgEcRPb8cBOqLb8fMuIhIbwONR\nVI2zFh+wAxnp8UUKSVxH90m4rX998T1mHfuyuO1LGvZBPDdh8+0P9YvMtPHCb9SIiIiIiIhChjdq\nREREREREIcMbNSIiIiIiopAJf2FRWDj1bH9AE+n+N2Et0g1LH4B4U84+3z07cQDiN85ciTO8HeOv\nrbvYrGNgMz7PHWnAbes4z96P77oWt8XP4XO9k1apJoI37jXr6M0eatZduM/We4XNP0zDhuO/Us2d\nRWyDxUlx+1y/trCuE+LnBJuE/uGL/wnxrgI21RYRuXjJ30C85TW4zMuffZ1Z5ncn/S/E9RFsQPnx\nzpMg/uNptvHjoKpR0PmY9u0yOQ9z464BrJ3Z8zLMx+kqpQllG/HYoksno2n8PPsBNWq6l6iexy+h\nvEz1GjVxwZYYUJlcPGGm+TmsD/3+1epY/+/F1xtVNWm6iXlQ8+roECaHzp2gZSKqKbbOFbuAndT6\nvceKLETV1LjT1tV3e/UQ61ox3bR4n6pLEhH5ykGs0W6K4OvoeicRkfXp6RBPiWHtfcTZnKPytH0D\nP2ffu/cSM8/GP8d9mDyxB+JZn7InHX/Fs2oKXgcdp+Jos80d19SI62zAmluv2dbzF+rwmiTWh4Vt\n076+BuLfCNbUi4ic+RTm+UUN6yHelZ9klrl85jqIV47j91z8Ro2IiIiIiChkeKNGREREREQUMrxR\nIyIiIiIiChnWqP2JrkErw3kfwT5VlzSuOcych8xSzYoGfKxd6C5gD4yPL/21WUfnEuyjllNNbr61\nAXtyiYj0q7q2aB5///Pe9RTE101eYdbx2TtPeenfEX/A/Hyi+Rcug/jxzAsQB/VR08/T654j0+P4\n7LaIyFODtk/MSNdc906II0O2j8ncOfj+X/NPr4S4ydm6tjdkrsQJqh9X9+VLcB1im+E9dBDnecVk\nfA5b1ygETevMY/6lz1e9/r5sVkEj6J5Upp5Ml8cG/XlNz1PGMhHVKkcvE9S/jcqj69GC5DdvxXjL\n+WaexDw87ubTWHMU1X3TAkpude8ifRyJBRza01NG7/Wn/wSc3GlrXWl8xXcfNNOua8Bp3+zBOiJ9\nbI/q5opie29qfZ4tbo2qREx7mB/pHB4UsZKJxmL9N8/BCQHlfzMexIluNeZBdpLto3b9Wuw9pvfp\npvQ0iNf02nqzXX24ZzN5PMn4vj1gOYc1kO1NeL1x02zsP/mTfWeadaz6C6zJW92DPdL83XZMBm/Q\nXoONF36jRkREREREFDK8USMiIiIiIgoZ3qgRERERERGFDG/UiIiIiIiIQoaDifxJQAPrsdrQj8WT\n+5uxULIjbxvvTYliIWRTZAji+fEuiDsLWNwrIhJVTZmzarCHfz7pl2aZ9IlYvKsH0bggtRviN665\nwayjQTabaWGy98NYJT892gvxVrENyDOqqLldDR4S1PBzsIADwOQvOwPioTZc59Bk+/cR9bIyMB2L\nWyN2/BGJqUbIhQQOApBpxTj9l3YwggsaH4R4Xw5/vyWpPWYZXVDeEsXRBm488XGIHxRbREyH6EE7\nYoOjN7gOGhhEDx4S0GM2YKHRf2wGmaAJ5UfsDmtpxPPFftXAuJDEZeJ9dtAsfeyJqP1eZKwIESme\nb3X7Kh+siyqT37Kt6Dz6OqCU5tVaQf39v97ZBEqqkYvqVdJ19+AgalOLviodzqx78bO32/a7lq5r\n8Tjy2bPuhPhDv367WeZ7/+81EGdacL/34iWM5BsCTjhqkh9T5764XcZl8fcZ8HBgvM/93/UQJ/rs\nOg5+RA3ClMNrc68br+VFRG65FK+j77r0VFzHng6zTLXwGzUiIiIiIqKQ4Y0aERERERFRyBS9UXPO\n3eac2+ece27EtMnOud855zYM/3/Skd1MqlXMHyoXc4cqwfyhcjF3qBLMH6qmUmrUbheRr4nI90ZM\nu0VE7vN9/zPOuVuG449Uf/NqS1sS6810s+SEs00Dd+fws7ph6HiI1/fis7JXtT9v1qEbEOsaoqDn\nymfGsdFl2sdCBV0SdWG7rUdbbaYEul0mKH/yT+B7++9Tr4b4zdNsE+/FCWziOCeK9X/f6TnZLJPx\n8GO0/HvfhDjnF1Rsmzim1bSUU8/5R2zD2Ij6O0vGx70Wd5gXm3O20O22AxdCPCuJeaFz+MX1Yh4/\n2H0CxI/cjc9uz5NHzTpKdLscA8eewGbUI+hG0y6gaXEpdWzFqDSWaAaPI0NtNVdndLvUUv5E1E70\n8LhRv8fu1OhJKhnULNGM7nxuX9ZL4MRoGpcp2H7FElPz6Lq27GTcrsZdxWubXBxrfUtpCn4E3S61\nlDtlOugNjfpzXW8WF7sf9Tz6ekTHIvacGVUHNa+v5huk3y4hyZ+Xf+wxiPsLSTPPyq45EN+2+yKI\nb7jkIbPMx9+0ZtTX7fewvvGAZ69/06q4uqDiQd/eoqTU9WxLBOPZMRwb4vmszfGPbXstxBu6sAoy\n9Yw96H1tMy4zY0/Z1zVjVvR07vv+QyJyQE2+VkS+O/zv74rIa4UoAPOHysXcoUowf6hczB2qBPOH\nqqncUR/bfd//03BwHSLSfrgZnXM3i8jNIiIpqT/cbHRsKSl/mDsUgMceqgSPPVQuHnuoEjz2UFkq\nHkzE931fRhno2ff9W33fP8v3/bPiYr9ypWPbaPnD3KHR8NhDleCxh8rFYw9VgsceGotyv1Hb65yb\n4fv+HufcDBHZV3SJsHP4bKyL4nPVfh6fr41OsnWgF7c+C3FnAXtSdRfsX0dao4MQ9+Xx2dgDQ7jM\nCUnb12rV4HyI2xJYZ6RfQ0RkaxafyV2cxB4Qn917GcRzUvpbfJH8ZS9/6d/+44+Zn49iXPJn9qfw\nGeKeT+HPb5tu+4oNnYrPanfcjM9Zf+JU25Pu+f6ZEH9hP9axbRjEOsOGaFBfmYBGaWMUcaPXJu7P\nYW8aEZHj6vGt/+7G8yCedu0LJbwy1mZWUJNWipo+9sSm2z+imhIOXQqme81UaaxeXevmxfCF46pP\nX1AfnEgD5pQ3MGDmCZmazZ/mrQF1Xuoz7yVUT03VurNhh02eSF71X5yM60x029pEXW6tD2m651sV\nDm9hULO5czi5MfaP1fVoIiJRUfVl6gCW8QPqq1XeFtRBLTpwVA5IPiH58+N7sA79zIvWmXk+vOge\niP/uiTdCvOm3C80y32t7OcQNO3Gf6fOaroEWESnUjd4zNIhTx6uYKkHTx5oclqyJiEh6Dh6wNl59\nK8R/PvMVZpnvzcM6vctXvgvi6AOrAra2Osr9NPxCRG4c/veNInJXdTaHjhHMHyoXc4cqwfyhcjF3\nqBLMHypLKcPz3yEij4nI8c65nc65m0TkMyJyhXNug4hcPhwTGcwfKhdzhyrB/KFyMXeoEswfqqai\njz76vv+Ww/zossNMJ3oJ84fKxdyhSjB/qFzMHaoE84eqqdwataOPelbbxfCt0TVqO2460azi0nqs\nX3o0PQvitlifWUb3GJmR7IG4qR1rpILq3CbHsEaor1AHcb1uchOwLWckuiD+m3vPwO04eb9ZR3N8\nxBeyNddiSSTfsddMi6tps4ZOhzh1my220M/kt8SwJlDv02TE9hMJ6jUzku4zIyISUcVLeh1T47iP\ne/OYFyI2DzJPTB51O6gy/qDt6RLVH8+xlY2UtkzA53Os/dsSvXYlNVCTdtSIDwT1Xyxy4FWLBO1z\n3VZJH2qSB21ypafi6waUv6rXqMETxDEg7sa2X3Q9mohIShcFqVmC+rh6Km91H1evbUJ76B1V6o7v\nhvhg2l5D/qF3CcQNK/BaYehce5x/1WLso+apg0spdff6mkWvIxJ03aPqG/X1VN7Ddaw6gOMOiIj0\n/gTHFfi3s3FcgSd2zDPLnNLxVojnrNoIcfFOkeU7Kis2iYiIiIiIahlv1IiIiIiIiEKGN2pERERE\nREQhwxs1IiIiIiKikOFgIsNcPAGxl04fZs4XTX3WFrt2FbAgtjWCg0okAopqs6qY8oLJWyDuVAOD\nrBpaYNbRFMUBCtoiOEDEnLgdCOTZNBZYLh84DuKbXn0vxHfceoVZR+K3hxobO9821Q4dVTgdSSbN\nLGa/q0FmNmexebWISEIVs+oC2aAmoZoeLEQ3AK2GUop71bgngcxAOwWV12Nsonos8QPem1KafI4H\np7ZNDzJBR5g3ejl6JGcL6/ftb8Z5snjcSHQXP44kcawByeXwOBkwBpHU7cNcGWpTTWj7dVLbbaeJ\nFy0yCpgePCRoUKu46n4+IHjgCBoQol51SB/0cJnFs2u+l3hovHzWZojrdHd6Ebmq5RmIH+s4B+Le\nIdu0fKiA18y7BlsgjkVwv2fy9nYjHsVjnh4IxA8YLMmpwUSmpnCgk8E8butJrR1mHSsGcTCRBUnM\nt6XT7TKLGnHAvefmH48zPNNrlqkWfqNGREREREQUMrxRIyIiIiIiChneqBEREREREYVM+GrUVB2R\ni+Hzpi6q7i0j9l7TS6sOskWe/RcR8XNja7D4H//1NTNtR74V4o4cxq1RW8dVUM+I/3EIn/PVzSTb\nYvY52F4voIhghD4vZabpOir9Oh+ZsgHin/ZcPupr1ARVg+NlbCNwLf4c1gxuHGw389RF8b07mB+9\n+6tukC1im1eX0jxR1wvofaq3ozFW/PdN9JZQXxZV9Sd528Cbgun6vsB5SmhSPB7r8CPqWByUlBGV\nCyUca+kwiryXmVabO60tByE+MIjzZCbjeS3oCOC6VH12vapLarbnRi9bpLAygseRvrn2HKSPkmM9\nB1PlIkUaXutm1ZESag11XVtObK7oeum0h9d5V7ZjM+W7BWsxqXSxCO7DA1l7faIbjid6cZl4na1v\nz6uTSkK9TiKK1wX6Gido2/IOcyWovjGvrnPiah2NcVwmqDa/vnP0a5YTmvbaZXRd5VzMyRSW+VUV\nv1EjIiIiIiIKGd6oERERERERhQxv1IiIiIiIiEJmQmvUguo1fFXvop9b94u3girL0LXYN2LHa/G5\n17ed/gTEHfkms46nBudD3KL6mzVEbIWAfjZ4d3YSxLp2bHKs36xjmqpb0z24duVwnUF0/dzOPL5O\n359hbzYRkdbvFV1tqDldayU2/wq9+D70BtSftcZxPw+q/iL62eagZ7V13ZquPwtaRtcPFBzu94P5\neohnJGyTNF1z4ArsgXYkuYZ6O1G95apNjOhWMkG1YroGrZzebL6uD9Y93wJ62kTqsPbIGxgw81CJ\nitT31XfY88fetVMgbt6leqDV4/klFtAedGga7ueIqj9LbLc5G1WbklOnw7oOXOfgTB5XJpo78yQz\nrSWyGmJd56x7hAbR/WH1uSvq2zqjqDro6T5qZ9Vj76+7ZVnR7aBgps5Qn2BEJOfjtXiyCw8UqTqb\nBzkPc0XXm3kB5wtNz6Ovg4K+SRpSfdJycdwOPWaA7ucmIpLaidezXXmsN8t49t4kqT4L2WbcOluF\nWz38Ro2IiIiIiChkeKNGREREREQUMrxRIyIiIiIiChneqBEREREREYXMhA4mogduKEVsxnSIcwts\nA+IDJ2Lx8+B0LFBcds1as8w7278DcWcBiwvjDrd1Rw6LuEVETq/fCvHve5ZC3BVrNMvoAUcuaMBG\n090e/i4zY9jgVETkIxvfAHF7PRZKfmvecrNMThX4rsthMW+PKhT9/5beb9bxM2kz02qJ75VQ4K4K\n/LMBRaaeGslBF8jqYt4gOdXwUw8iE0Q3g9Svo7dDF4qL2ELwgP6SVinvGwULajCrJpka7FLe7uJ1\n22OmBxcJfNmAAXnoyNh1sR3Uo3Erxi1bVSH9EH6+Y912QJJ8Kx7705PxWBQfCBgMIoPr7Z+VMPOM\ndHCaXUds3hzcjm07cAbdAFyEDdUrcOAU2zT6t4O47/sLOCRCUwSvT4KkHOZcKU2x9bnqgBqk68Ik\nriNzzdlmHcnlK4q+Dll6UAwRkay6Noht3wdxU8oOolaMHrREN8gWEUmpAUhiouKAgUD0YDVZda0a\n9PtpLo3HQX0tFfS6eoARL3oETrqHwW/UiIiIiIiIQoY3akRERERERCHDGzUiIiIiIqKQmdAatczV\n9rnjaR/DRofLmndCvLTuYYjTqrZHxNb3rBmaBfGgZ5+n35DF2rce1SxYPxe7L2sbXn9hy+UQ33fO\nNyH+f7uvMstE6vA53v0FrGO7rhGbWYvY3/c9cx+CeGECny/+1cAMs8xu1QS7PY7NkOfHOyF+fdN6\ns45ar1ErxysmrTPT1gzOhFgt/XtbAAAgAElEQVQ/I60bkAfVrOn8qgb9On0F25JRP5tdTqNkGoPY\nEXqDdR1bCY/PF2tw7atn8ANzI2GPR1SCEuqvoscfB/HQCbZbdWEr1hhlW3F/ZCbj6zRttscAVR4k\nA/NwO+I99jIh16T/xjt6IWW03/5NePOfY43a3E+oGjXWo1VV1yuyZlpBRq+njqo6o0JQ03tVk+aV\n8Pf/pLpG042Of9A3DeIDN/ebdcywpfcUoJTG07oBeb5jL8Sp2Nyi682rWjF9bZEp2ONIzOncUQ2w\nC8VzKV3AY55eZzSgZtJvwOPg+kG8/m+NDRZ93YDLqSOG36gRERERERGFDG/UiIiIiIiIQoY3akRE\nRERERCEzvjVqTsTFDr3kuZ+yfTAua3oe4kFf9XhRNWm61ipIi3reNJOzv/a+nO0xMtKSZAfEr2te\nbeZ56GvnQnxR+gMQb7oUe7WJiNw3hM/1duZxO67fcinEq7bjc/0iIufN3wLxKU27INb1diIiTVGs\nd9B94gY8fN//mLY94GqeP/a6sLRfvCanJYa9Z3TOBtWjRVR9UEQ9M66f3Raxz5UPqiKixhj2CjmY\ns3mge8AV4qUUN1W/nu6YEdCbTJcsqrIQ01ctoB2NVULvNVOTFimy74N+PEUdf7v2F39hKqn+asef\nYZ1O3Qt2nkIK92FClTQPzsXPatMu+9k9cII6H6pZ6nfZHd99Mr5uah+uIzMZf79Et03aoZl4znGn\nnwSx/xReC1Bl3njaSjOtr1AHsa4d0/U9BbG1laX0/NQS6npjagxr0A6oWv2PnHiPWcf3xF4L0ZHR\nkrD99HRftGK9yEqpwzf1dAHnHF3z76nrkf48XrvGI/ZYW2jAcSoe2Ib1wG9d8qRZpiePn5USSv+q\nht+oERERERERhQxv1IiIiIiIiEKGN2pEREREREQhM641arlpDbL7Hee8FH+i5atmnh8eOA/iOakD\nEM9LdEF8Wt22oq/bFMF6rOOb82aeXw3MhviB7hMgnhHvhvgPg4vMOn70ic9B/M6/+RDE5y//S7NM\n73y8V8434LP/zadhzcf/O/3XZh0JVeTSXcBapMnJAbNMa3T0PhG6NrApYp9RHtnnx2192Pz8aNSV\ns/3zdN803acvqZ7HzwU0pNI1aPq5/x5VSyBie+DUR7EmTdefdXij12GKiGRbx/HB62OQn7Q1jrrm\nrOiz70E/L6EmbaxcoUixnIh49Ukzjapj4CT8PDc8b99rXVdY0LMkdF2I/dtssd6JzrPJ5Tx83Qhu\nqtTNwpqjfJ899sR68YX7jsO6pManRt8uGpvrWu2YAM+msc5L91ErlPC3/JTDc5WuISqFro2bEsX8\nubhuj1nmf+qPh9gbLN776li0YwjriKendG9eO0aBNiVp39s+VQumrzfyJZSy6+ueiBu9Vl/E1pzp\nurahPJ5jg9ahj5uZnXjsqT/B9hw86ON19Xj2nOU3akRERERERCHDGzUiIiIiIqKQKXqj5pyb45y7\n3zm3xjn3vHPur4enT3bO/c45t2H4/8XHyadjCnOHKsH8oXIxd6gSzB8qF3OHqq2Ub9TyIvIh3/eX\nish5IvJ+59xSEblFRO7zfX+xiNw3HBONxNyhSjB/qFzMHaoE84fKxdyhqio6mIjv+3tEZM/wv/uc\nc2tFZJaIXCsirxie7bsi8oCIfGS0dUVyIvV7D1UY/qp3mZlnYV0nxHrwhrv7T4F4dt1Bs46WKA58\ncZxqVr063WqW+W0nNtucWYcFl3tzLRDvzzWYdQyqJtHf/tIXIf7C3svNMq+bvAri0xI4eEi3h/fS\na7LTzTr6vBTEuilzTyGo4TW+RzkfUyGqmgi2Rmwxae8pU176d2GvTaVq5k5YBA0EUoxu9OiVsA5d\n1B1UEKvpYl7dgFL/XMQ2Ns+nzCyGHzC4wJFwNOaPHw/Y97qhtZ5lfN5uieRHf6HAvrYhfXi+FnMn\ncjIOYBXtwEGJzEAhIhJX40R5+jCcx+TK1xXfYU4tE9Sn1jeDlGDSpodw2702O1hBsgM3drAN14Hl\n/eOrFvNHi01vh/jMhD32PDqIB/zJahCPghqoIersMUKfV/T1hz6XiYhEVCPt1igm8i1Pvh7in1/w\nDbOOoVfgNVtyuR0sZSJMdO5EUrhP9WAbQftjY8ZeV47UEMuYaQP5RMCcI18X86I+ZgfoyKoDlh5M\nJEgqiicivY6CumbWA5aIiPhxnKdhO8aNURyAUEQk42Fee/HxG3htTKdZ59x8ETldRB4XkfbhhBQR\n6RCR9sMsRsTcoYowf6hczB2qBPOHysXcoWoo+UbNOdcoIneKyAd934evm3zf9+Uwf/d1zt3snHvS\nOfdkPmOHiaejXzVyJyf2Lzp0bGD+ULmYO1QJ5g+Vi7lD1VLSjZpzLi4vJtwPfN//6fDkvc65GcM/\nnyEi+4KW9X3/Vt/3z/J9/6xY0j4uSEe3auVOXNiv6VjE/KFyMXeoEswfKhdzh6qpaI2ac86JyLdF\nZK3v+yOLrn4hIjeKyGeG/39XsXVFs5407Tj0FwL97KyIyO+78Dn99lQfxMuadkC8btA+W/vs0EyI\nV8XmQlwXtcUWLQl8JlU/kzs1jtuxIGk/Y7rx9Io0vu572x4wy2zP48A/vxxYAvGaQfxdJsVsrdiz\nvTjPoHp2OFOwuzmdx1q/liT+/mdPxkbi62SGWUfnaYfu8/OPmB9XNXfCIuj57sAGxCOU0wBUN6DU\ndW6lvI7e1qDPm66rzNePU0FUCY7G/AlqeG1nwlDv+jLSqSy6XCCoRi3fhPkzjj1AR1WLuTOwCJtC\n6/ffDzhbF1SZiKljU42pTQ1bAK8Vjz2RfEDOxnDjdF1lbBvWyfgL7XnL78SNyWIZuMRm2HN7fk+H\nmXYk1GL+aD0Xzoc46uyBY1AlTFsMr3N0jVpQY+Q2Ve/eGsV9HVTX7anvCPR56KKFmyCuDzjv7l+K\neTlzuZllQkx07rz4Zd0h+rwfdP370P7FaspeiJIRu9917Ve+yIkpEnChVKzBddA68+p6NhbRtfj4\nOmnPHr+yLbiOyevwPWmI2G8y9e8bcDl1xJRw2JYLReQdIvKsc2718LSPyovJ9n/OuZtEZJuIvOnI\nbCLVMOYOVYL5Q+Vi7lAlmD9ULuYOVVUpoz4+LIf/zuCy6m4OHU2YO1QJ5g+Vi7lDlWD+ULmYO1Rt\nIR1cmYiIiIiI6NhVyqOP1dM/JJEHn3op/PE9F5pZ/vHaH0P8YDfWrP2qA2urerO22LKtHkeXbFb1\nZZN18xkRaVG1Xyn1LPbBPA6EkonY514L6o8oHRl86P4RTz8HLJLz8PntjIr188QHslPNOmbW9UDc\np5phbe2bbJbp6sEuNel6TIWHC4sgvmr682YddfsO/b4BjzCHn1+deqxUYIOpwwuqWSvWJy1Zwmvo\nZ6h1H7VYxD7nn1aFL2W0iaMxKCQD3mBdC6Y/S7rPWlW3aMR6VVrqspBIzr5y92I8/k55oMobdQzx\nYqPXQKhSIBERKdSpdcRxH7ls8Z5oOqESDdjvKLBGLYvJMjQTk3bKKszzKedhf1ARkY17cePVqU+8\naVi/LSIi41SjdjTYdTV+gFdmbB+rflWjpuvJsur8MD/WZdahz2ZNEUzUadE+0dZncWT6Pg9z4fwW\nrFEbDDgx9S+1vw9Zup9ZUJ39C3unQTxP1agFLaNrwXSftJg62CSj9iJRX/9q+hrmxdfF3yer1lFK\nL7Z0izo+re6GOKgW01yjjWONGr9RIyIiIiIiChneqBEREREREYUMb9SIiIiIiIhChjdqRERERERE\nITO+g4koCz/ymJn2n8+8Aed53zqIr57+HMSrerGptIjIdjV4xtOqAXY8YgsU6+NYCJlSg3gkolhM\nGTT4gx7MoSGK69RNtEVEJidxYJOmKDaeDiqm1KJqW57omQ9xe70t5j2uGYuCdWNBXcx725YLzDra\nv/roS//e6tsBWkLPBVSDFhlgpFcN1CIiUp8YW1FzUANQPSBJ2scC/qBi3qD1jKSLbqMBRbYZ1Qyy\npGbKfvGcpGD9c2z+aGZQjyINsEUCBoEpYcQRP6IHmlBNjNXHI2jAoPqugAbwVJahKbjjvQTuj7pO\nu8zBpaq5bQrjWB+uUzfIFrH7taURB4MoJHAgLRGRSBrXO2cpDvLhL8fBCfb0NZl1eAlMZL8Vc8mP\nc2SjSiycvw/jmP0Av7wJr6/0IApPD83D+QMOX+d+5MMQt34fr+t+sOMRs8zM2FaIN+eazTwjzQ64\nUj17yRaIe+wsJPa6NPBaYqf9jI/Unas30zYewIHt+vrV4ECF4qNt+AV1souowZCCBgZRq9WXcfEE\n5nBrAgcKFBHJNaqFNm6HUF9Ti4jk1InZG8e7J36jRkREREREFDK8USMiIiIiIgoZ3qgRERERERGF\nzPjXqEVGPHfu2WdlW37wR4j3/wB//pPrroT43I+uMOt49fynIT4hoZr3iS3ySKnCjwZVv5FWtUtB\nd7gPD82BuKDm+v3BE80y3Tl8rnfvID6rHY8WrwHRjQeHVIPSniH7YHlUPQucfgCfN96yBhuNtyy3\n7zO9KK6KPHTdl65nDHpGXE/Tz0jrZupB82h6mWJNtUXY8PpIi6UDaltVP2H9WL7pCRrw6L9OqVL2\nY1Q1sNavo2vhzHP9IhLbyhq1aklP1UWBqkZtv32vu5pVssRUjVoH7tRCwuZf8iBO6xvE80V9GX/O\nTfRhzW1/t61xcZ5q8D2I2zowx9bN1D859m05Vu27ZzbEBxbb656IuhYqqDqc9njxyq9E/+g1y4MB\ndd/dRQp8dI12V8Hm/ooXFkC8RGxT9WORc2M/78f7R68na43bOq/6BH7Gsyncp7NbsYl0pmD3ebaA\nn/lSekjrhtZRNeZEVz8eN2akes06Hp+uzn0DOMZCa9SOuVCnxq3Q5+0jid+oERERERERhQxv1IiI\niIiIiEKGN2pEREREREQhM/41agF1aWPRcOfjED93p53nOcFnl93Zfwbx0HSsCxMRSe7HHmd983Ce\n5k34zGokY3uSeE+vtRsD+ov8XEQEn6fNHWau0ehWOW0lLbW+jFeqcUV6pgVZ2TXHTJsz+wDEg6pZ\nke53FtT/rDGaGXWeoGV0PUFGPfdfHy1eqKTX4UdLacA19veNXtR0nz1GHFxyMsSZVnxSP4ZtrQLZ\nnmeqH00Zu2xwuu6zZudJrd4KMSvWypdvULUXQ6pOelLQ5xnPQ9EUxpEcHou8mK0CSU9V8X489yUa\nAipHpmK/z6WTsI/aE4tnQOx7AWcy3TNJ1axlm+zfkW2lGx3OzM8+CvGiDzaaeSJyEOIVmVkQF+vV\nKWL7L2or0jPNND1uQK+HdZGL4vtVbLf9xC/itRKPPcPiWDw1kMdjwKBnmynq84f2v7+9yEzLN+M7\nnuzCXNkSxfEWAkrz7XaodAvcLt1HTZ2XXB5n+HHvGWYVs1eOvjEDXtJMy6rrq5J6zlYJv1EjIiIi\nIiIKGd6oERERERERhQxv1IiIiIiIiEJm/GvUJoC/4lmIbVcxq/nR0X8+eucQOlrNaeq20+JYo1Yf\nyUJ8dt1miBMB2RNXD1q3RMb+xP2geqA7pQqTftlv+/jNimONQv0C23PEiKgHySusOz2WFHrt+zvn\na9j3sfvaUyAemop/T8vZ9lLmeflIoXhHGr2MriFo3oo5OfkXa8w6gn4fKo+/EHsV+duwIitfwolL\n9xgqqHLsKJaWiYjIzEewPnbzWzB3glpeTXoAN+aeiOq7qXKrvsUWWg4NYt1RwzY8rkz5pa3n5JGm\nfK984zvNtHt+fLuasguiA6aeydY3DU7D/aZHAHhZ3R6zzLQoHsTq3T6IF6iatAv+5i/NOprW/NFM\nI5FII763UXVtEdTHNdcy+hXtwlseq3zDaogX8B2W7keXaxm/Wn1+o0ZERERERBQyvFEjIiIiIiIK\nGd6oERERERERhQxv1IiIiIiIiELmmBhMhCiQCxhwoUgz58efW2SmPZHEBuvSgw0n/XgJQ8+oP5lE\n+9WEoM6PauAA3ehRNzmOBPSczaqC2LYniw9CwcFDKhCQc97AAMTNP8QieWwbKhKbMd2sIz9vGsSZ\nSdiwM6jhdd0OHAjE37pz1O0K3Ov692Ez9LItvAEHz/BzOCiRGcRHRNrUZzFyGg4Y5K/BdbrjF5p1\neM+9APGS+4puqjHlW0VmuHXs6+RRprrcI6vNtCtnLoM4/ZpzIN6/FC8R617WZdbRfh8OFpJXPz93\n+QfNMg1tOHBO451NELf8AI+BTcKBQ0qV34PN59dvOhvijXvwXCEi0raiyHc2QddK2lF07P/bu99m\npk2ahwOvTV3NwUSIiIiIiIiOWbxRIyIiIiIiChneqBEREREREYWM88fxuVLnXKeIbBORqSJiH3YO\np1rZ1oneznm+77cdqZUzd464id5W5o9VK9s60dvJ3LG4raVj/li1sq0TvZ3MHYvbWrqS8mdcb9Re\nelHnnvR9/6xxf+Ey1Mq21sp2VqqWfk9ua/jU0u9ZK9taK9tZqVr6Pbmt4VNLv2etbGutbGelaun3\n5LZWHx99JCIiIiIiChneqBEREREREYXMRN2oldFVZcLUyrbWynZWqpZ+T25r+NTS71kr21or21mp\nWvo9ua3hU0u/Z61sa61sZ6Vq6ffktlbZhNSoERERERER0eHx0UciIiIiIqKQGdcbNefcVc65dc65\njc65W8bztYtxzt3mnNvnnHtuxLTJzrnfOec2DP9/0kRu45845+Y45+53zq1xzj3vnPvr4emh3N5q\nYf5UjrnD3KkE84f5Uy7mDnOnEswf5k+5aj13xu1GzTkXFZGvi8jVIrJURN7inFs6Xq9fgttF5Co1\n7RYRuc/3/cUict9wHAZ5EfmQ7/tLReQ8EXn/8HsZ1u2tGPOnapg7zJ1KMH+YP+Vi7jB3KsH8Yf6U\nq7Zzx/f9cflPRM4XkbtHxP8gIv8wXq9f4jbOF5HnRsTrRGTG8L9niMi6id7Gw2z3XSJyRa1sL/Mn\nPP8xd8LxXy3mDvNn4retlvOHuROO/2oxd5g/E79ttZw/tZY74/no4ywR2TEi3jk8Lczafd/fM/zv\nDhFpn8iNCeKcmy8ip4vI41ID21sB5k+VMXdCLfT7g/kTaqHeH8ydUAv9/mD+hFqo90ct5g4HEymR\n/+Itd6iGyHTONYrInSLyQd/3e0f+LIzbeywL2/5g7tSOMO4P5k/tCNv+YO7UjjDuD+ZP7Qjb/qjV\n3BnPG7VdIjJnRDx7eFqY7XXOzRARGf7/vgnenpc45+LyYsL9wPf9nw5PDu32VgHzp0qYO8ydSjB/\nmD/lYu4wdyrB/GH+lKuWc2c8b9RWiMhi59wC51xCRK4XkV+M4+uX4xcicuPwv2+UF59rnXDOOSci\n3xaRtb7vf3HEj0K5vVXC/KkC5g5zpxLMH+ZPuZg7zJ1KMH+YP+Wq+dwZ5wK+a0RkvYhsEpGPTXSB\nntq2O0Rkj4jk5MXngG8SkSny4kgwG0TkXhGZPNHbObytF8mLX9E+IyKrh/+7Jqzby/wJz/5g7jB3\nmD/MH+YOc6eWcof5w/w5lnPHDf8SREREREREFBIcTISIiIiIiChkeKNGREREREQUMrxRIyIiIiIi\nChneqBEREREREYUMb9SIiIiIiIhChjdqREREREREIcMbNSIiIiIiopDhjRoREREREVHI8EaNiIiI\niIgoZHijRkREREREFDK8USMiIiIiIgqZim7UnHNXOefWOec2OuduqdZG0bGB+UPlYu5QJZg/VC7m\nDlWC+UNj5XzfL29B56Iisl5ErhCRnSKyQkTe4vv+msMtk3BJPyUNZb0ehVtaBiTrZ1yp8481f2o5\nd7zWeoijQwUzj5/JjHm9ri4Fca4hCnGsa2DM65wofXKwy/f9tlLm5bGHRuKxhypxtB57XBTPB+nZ\nSTNPa/0gxL0HcTvjHRNzDslPxe1ITLHnx3Qf/j6J3eO/rTz2VFk9XtNkW6Jmloi+fPIwjPflIPYz\n2Wps2RFR6rEnVsFrnCMiG33f3ywi4pz7kYhcKyKHPWClpEHOdZdV8JIUVo/79411kTHlzxHJHVfC\n8bXMP2SMNHTJORA3rtlv5ims3zTm9UaOOwHifRdMgnjqrY+NeZ0T5V7/J9vGMDuPPfSSmjz2UGgc\nrceeaHMLxGv/abGZ5/Wnr4L4t3eeB/HsTz1a/Q0rQdd150M87+0bzTxrf4+/z9xPjP+28thTXe6k\nkyHe9qpmM0/yIMbRNF6jtd+/F+LChs3V2ThNXz+Wca1Y6rGnkkcfZ4nIjhHxzuFpwDl3s3PuSefc\nkzkZ+7cGdNQqmj/MHToMHnuoEjz2ULl47KFK8NhDY3bEBxPxff9W3/fP8n3/rLjYr96JDoe5Q5Vg\n/lC5mDtUCeYPlYu5Q1oljz7uEpE5I+LZw9OISjHx+eMC/k7h2fqxkaJLFplp69+Djxjf/YbPQ7wo\nvnrs21YSXG/Gx2ezB/8R4wu+9XdmDXP/uYzHRSLqufEi79kRMPG5Q7WM+UPlCm3ubPrhMoj/Zhk+\nlnedW2eW+WMvns9+8O4vQfzEOxZAfO/+E806Vm6ZC7HXF4c41mprhN576kMQt0SxVm5x8hsQ39d3\nklnHq978LMS/u2IpxD3vnWaW8Z55wUwbZ6HNn6or49HAqV/ZCfG3Zv/KzNOgrtsmRXEMAPkEhkse\nusGs43XHPwPxjEQ3xF996hKzzJL34uO3Xl8fxC6Gt1N+Pm/WUa5KvlFbISKLnXMLnHMJEbleRH5R\nnc2iYwDzh8rF3KFKMH+oXMwdqgTzh8as7G/UfN/PO+f+SkTuFpGoiNzm+/7zVdsyOqoxf6hczB2q\nBPOHysXcoUowf6gclTz6KL7vLxeR5VXaFjrGMH+oXMwdqgTzh8rF3KFKMH9orI74YCJEREREREQ0\nNhV9o0ZUU8oYBOOCp7EQ+qZJ3zXzTI4kIN6jVvvAEP49pC1qG3M+m5kJ8do0xpc0rjXLzIxhMevu\nfBPE7VEcTGTlu79s1vHMjfievPfZt0E87dqA4mv9vk384CLHloD+f7q5rV9Q+6CUHi/F+gpWoaeg\niEjmmrMhTi5fgZtxFvbS8VcGPBlUpW0hGbf9PlYdP7cDV7T9Rx3E0fuxD1ikHgcW8AZxkIqjxcB1\n55ppp87GflHf3ngBxNMa+80yEYf79vN7roT4jObtEL++Dd9vEZFpSVzv8ufx83v1Evv57StgY+PV\nfbMh/ub+l0N8/OR9Zh337jke4jlNOCDE0OfTZpnkK80kOlL0YG1+8euCc1u2QPxcdoqZpzWCn+nu\nDI6MuTSBfWofvhAHphERmRbFJuL9HubKBy/dapZZ+NWbIF78zpUQuzo8NvlqsJFK8Bs1IiIiIiKi\nkOGNGhERERERUcjwRo2IiIiIiChkWKNGRy9de1FC7dSJK/Ej8eEpT0D8cHqSWaZVNev0fHxWuTUy\nBHHaVzVdInJx3Q6IL6/Hxo+7C3aZbg9r49qjWCuwt9CoYrMKaYrgs9lPnf0jiC/53bVmmcQV23CC\nfl/LaHRJVVbOe16F/TT4Oqyd2X+yzdv0ogzEF/8T1lZGZCvEuy/Fz5PI0Vt7NKqgWrJi+6yUz6Ke\nppcp43VdEutG/EzGzONfiA2a3/yt30J8U8tqs8wlH8XjUfR+NYPnjbpdR4tdl9n3f+/OWRAnklij\nnM5jI2oRkVQM59nYPRWXKeD5UNe0iYgkInj8P2cx1hkdyGI9kIhIR7oZ4wGMz5iG58PONJ7LRESi\nalue2zsD4qmNthY88ypVH/vrFWYeqg5TNx1w/RVJYa3i65qwnnFn3h77Uw4bSZ+satK25bFOdXV6\nnlnHW5uw+Xu3Om5ExDZpP/7zmE/mSKPrwquI36gRERERERGFDG/UiIiIiIiIQoY3akRERERERCHD\nGjU6ehWpozjwrvPNtC9M/zrEvx3CZ+fjYp9DbnL4nH9O9Q/xfKzxKIit+disnquOCm573NnX1fNk\nVO2brlnLBfxdZtDDuoVfDOB2/O8JPzTLXPvWD0Hc/MM/4gysSRubsdb0Bfzcz+cDZhxdx19jn6UZ\nD/dAvOuSFojffuPvzDoeObAI4r+f/S2I/6cTX0NE5IHnsP/Rzo8cB3HkwacOs8XHuKC8GGMPNBcr\n4ZSvaktcImFm8XSPINVLUdekDV17jlnHV778VYh7faxr+2Y31lyJiNS9D19HHxW9gFq4o1HDdFt/\nNdiH75+oMJ23+z4exXewIYG1Of05XMn+QVtvlozhsUfXseU8e96Z0dAL8eQU1pzqmrS9g1jHKmLP\nq9GIN+rPRUQ6XobvwYJfm1moXOpY5KK43328TBIRke7XYZ3q7BheS2wOOK21uNE/400RzOH5iU4z\nz6QoXufUR3DjPrf/FLNMoRk/C/EFWPuW36Jq93V/WZGye8zyGzUiIiIiIqKQ4Y0aERERERFRyPBG\njYiIiIiIKGRYo0ZHDV1/UaxuZ8W/fcNMW5nBZRbGDkC8JjvdLNPn4/P1DarPh6dq0lIB9WYJ1ZUj\nqI6tGL2MrlkLWmfE4es2q75qL+RsTcJjn/8mxK96DHsb6We1XdzWuPg526eEqugcfMbej9vn5bMX\nYZ3R+tOxp01T60GIv/Ozy806Zj2A+/Gz958Kce6yJWaZ+rMxHyIZrKWMLFsKsbd6jVkHDRtjPWhJ\ntYxqnqAeaIaqvYgej3WHP/zqF80im/NYh5RStb63f/I1ZpmWDaoe9ljp2ajqXYJ6hG3vxc/voIrr\nkwFFQkoyivs+FVXLYGnP8Dy4zEAeP991YvdJTNWTpaKYY3F1XqqP2W0/kAnYmBEKATVq0UX9AXNS\nVajPnpdOH2bGQzrPxPghtciatK1TvallO76OunbarS6vzk5inzURkYLqdbs5h/n1nfteYZaJX4X5\nNG0V1qzV6eueiM0/v8w2j/xGjYiIiIiIKGR4o0ZERERERBQyvFEjIiIiIiIKGd6oERERERERhQwH\nE6GjRrFC+fy9cyFem/hWw6IAACAASURBVH3UzLM1h4OFvLahG+I1AWNg5NSgHbbMGyXKrSgdIz14\nSNBgImkfG17rgv7t+clmmX2F3RDvuWomxG3fwKJaP1+8iP2YNsYBEKLNzWZaz5UnQtywC6uyYwds\nVrbf3gpx7gNYdL2nYxLEi//pMbOO2Lw5EOfV75J6aotZxp11AsTbr8RBJdS4AjJrtVkF/YluqlpG\nQ9XYfDwu5qdho/NMGw5KISKy9yw8bmSm4ev6UcyDp7NTzToe6sM8WJLqgHjKw7vMMmNv6350iJyC\ng/JEI/bzHEvhcTbXi4MdHOyxA0MlVLPqRS3Y9D5dwP3cGLcDy+gG17FIYdSfi4gMqgFH9KAlepm8\nb79T0A2t+4ZsnmontmOOFTtX0xiUMbDPOeevg9hT+3nNIF5biIi8uQsHm3rttKdwHer7p3OS+8w6\n/rkT1/HxNhywyo/bbV/4IxxYrrBmvZkH1lHKwE0l4jdqREREREREIcMbNSIiIiIiopDhjRoRERER\nEVHIsEaNjhmfXnRn0Xlao/jUetTh3zJ0TVcQ/Zy1qQ0L6GUdDWgKWin9unq7XnxdrJfTv19rBJt5\ni4hMiWCzyIOn47PYbXqBo7UJbZWYRu0FVWek3j83CWuIRERiaZyn6zRsBtv7Mpt0G1/x3xBf8Ld/\nCfHiH6nmwgHy23aM+nN/druZljyA2zo4E7ft6jdhLdzTD51m1uEefbroth0LXFzlTkbVB52GtYve\nF7HJuYjI7KZOiHcNYr3Q+2c9aJa5t+ckiP+67X6Ib97wVoh/13OyWUdLbAjiAwWsofLjR+byRH/e\npAZKaIdmYx1nOms32vfU8V195CM7bA1Xp2o83T2Ax3ZddtRSj/tMRCSbx/ez4LlRfy4iEo9inh5M\n4usW1O8ylLXn3d69+J5E6vE8VN9o6+m2dmPN9Yw5WMeX37HTLEOlcTHcR34uoKBfedO0FRDPifVC\n/JWZ+HMRkdUZ3K+6nnFDdhrEN2x7uVnH5ZOwJu2MJ98M8eK/etwsY6p/y6jJKxe/USMiIiIiIgoZ\n3qgRERERERGFDG/UiIiIiIiIQoY1an+inzfVdJ2Ifs5dSqgtiWPvEJHSnuMFum+OSFm9czSXxGe1\n/azarqOgzqgjj7U9rYlOM4+tQcP3Vtd0iYj0efh8fVMEn+Mf8PC9TUVsfYGuH8uq3mxRZ1837nDb\niq0jSEMEn/feX8Dn/nXNnojIngLWrX33cqx1+qQsK/q6dEix44aZv9/uE11+2P9y3Eczf4Q5KCJy\n5VtxPzVJ8Zq0scq0295N2WY81k5bibUly7PnQzw9ZWtNUrMO9ddxe4vXjR6t/Ix9b0bynl4LcewG\n25do6y5dd4Tx12WJWHgMe59cBPFnt/wE4raoPc99ft8lEN+xHGtJFmy0ffuK1XOa87Kzf4se8zk3\nBAbb8Pfq3GvrVOubsXfiB5fdB/GXf/Vqs4zXgecuvx3XkUjifu5P2+NINqf2iTp8eQW7D7IOz03J\nOB4DMmqdvZ14XhIReeXpz0Gc93CdD24+ziwTb8Tc7l+Gn4cUa9TKVqxfavo155hp5yYfhvjL+/E4\ncmq9rYE+JYn9FZ/NzIL4ghT2cf1Dwh6/bmjugvj7DbYWP0z4jRoREREREVHI8EaNiIiIiIgoZHij\nRkREREREFDK8USMiIiIiIgoZDibyJ2MdLCOgSFn8vJ028sdlFDHv/OgFEH/lpv8y83x20SljXq9W\nrCi9FnkvOx3is1Xh6oa8LVBui2JD2B5PNXOO2QE6OvPNEMcdLqMHKIn6dmCQnK+ahpbQrLqgpnnq\n7y4RNfBJ0IAkeqATPc8pCWxAKSLSrRqaDnq2wJzGYIzHnsL+A2Za3V1PQLzgruLriTQ1Qez19499\nu4o0/eyfaQf6SB7EeRLdWIQ+9xeYc0NzcTtFRDJLph96yZ5jdzCRscrv2m0nqgGqbBPtsZ8b3r8W\nG14/eNodZp6NfW0QLzl/K8RBQxP4+SLn2CI/FxGJzVIDqtTA+BFDbfg5SzbYa4lPn/oziM9O7oP4\nx8vONMt0PIbvxbSlPRB39uI5MqubaotIRDXNzuUwn+IJu09iUVymKYk5Nr8Fj3GP78JzrIhIZxq3\n7TPzfg7x5IQddOnRfQtwHadhrs/5pVmESlXkfLH7LTZn4+r8oQdae2ZwjlmmM4/ng/4CNnLXg7l1\nZex1nlnnXfg6Pf893cyz5N2q+bb+fY9gA2x+o0ZERERERBQyvFEjIiIiIiIKmaI3as6525xz+5xz\nz42YNtk59zvn3Ibh/086sptJtYr5Q+Vi7lAlmD9ULuYOVYL5Q9VUSo3a7SLyNRH53ohpt4jIfb7v\nf8Y5d8tw/JHqb94EKvK8aTn1Zvvef4GZ1n0KPr/9+Ut/BHFHfj/ETw4uNOvo+iU29Jv6mvVj3rZI\nCp/z3fCvWN+16MO2+WiJbpcJyh8vjn+HSKm6wqC6rzkxfFY+42MeRMU+d9wUHRp1noTTTbMDnl1W\ndW0RtW1B9WV6Pdkij0TrBtkiAc24XU793K40rerprqrH9+xLo2/GWNwux+KxpwxFGwFHizc/1/OU\nUu9TjK6tERGJ96uc0h/DKC6TbbLbHskfWodvX+JPbhfmD9LnNRERVTNbSk2aiydwGXU+9O/A+rPk\nMltHGFO1TW9ofxLiO5pONMt4fX1mGjjvVAjbv7TVzPK0bhZ9beCabpcQ5c7Mzz0KcXSpbeL7+S9d\nCXHjB/CDtfM9uE9ERNxibPTbn8H9aurN4vYc4qm6NT1PUMplsni86h7Cxtsz6rFW7tzTNpp19L0B\nt/Wqj/4txKkZtkZt3g2bIW4c3GzmqZLbJUT5cySYc06R88Unz/qZmbYq0wrx1U3PQDwzhtdWIiJN\n6jqu27PXRiOtbdo16s9FRNq/gp8vF3Ctnr93LsSxy7fjDFWsSdOKfqPm+/5DIqKr168Vke8O//u7\nIvLaKm8XHSWYP1Qu5g5VgvlD5WLuUCWYP1RN5Y762O77/p7hf3eISPvhZnTO3SwiN4uIpKS+zJej\no0xJ+cPcoQA89lAleOyhcvHYQ5XgsYfKUvFgIr7v+yJBz3K99PNbfd8/y/f9s+LCYbwJjZY/zB0a\nDY89VAkee6hcPPZQJXjsobEo9xu1vc65Gb7v73HOzRCRfUWXCJPA5/T90WO9itNPMtM2XY+9Phae\ntQPiB47/glnmf3rxWfN7unG9Owaw3vTqac+bdfzfqbdB/D65KGCLR7f7PWdAvOiM7YeZsyrGJX/2\nnYUHucYIxrpXmYjt69GjarQ68qq+QUTmx7sg7vVSZp5ir1usJ1rQIT2qas70MrreTMdBdA+49mjC\nzLM5g/UE2/NYT5C98iyIE3dj7UmFavvYc4SU01/KV/U+uubAKOW4qeQD/iB8wRtXQ3z/fcsgPu52\n3K5En62LiQ0c+n1cQB3lKI7t/KlSHYWugdRav481zc/8a9rMM78B66/Xp2dAfPBae45t2ob1czd9\n6+dqjm0QnZK0feP+/m1vgfg5M8dhhSZ3CmtsHXrdlWoe9fPWNdPMMgvPxWuU5zpwH+hPfFD66MNC\nJIIzRZxdKJrAuqKePjynpFuxpjERsfmW39MB8eIPdJh5tNGrmY640ORPNfhFjrveRXhcf1PjajPP\nj/rw+nZKFOsKOwv2+uOuwcUQv715LcTLVe+1tFe8z+b7N+Dn6ZuvWmTmWf6xX0B8pSwz8xwp5X6j\n9gsRuXH43zeKSAntVYlewvyhcjF3qBLMHyoXc4cqwfyhspQyPP8dIvKYiBzvnNvpnLtJRD4jIlc4\n5zaIyOXDMZHB/KFyMXeoEswfKhdzhyrB/KFqKvroo+/7bznMjy6r8rbQUYj5Q+Vi7lAlmD9ULuYO\nVYL5Q9VUbo3a+ImoHjoePqus+3+JiHhp+zw8KOE5/Wg7Ps+97vOzIL7zom+aZXYVsH7pgV7sA/P3\nuy81yzRG8Zn7tkQ/xPdvxudxB6faZ3av+f6HIZ4vWB8Qm4fP7IqIbLkBpz35ni9DfN2rboQ4e+mZ\nZh2x368008JE91eKO8ylnG/Tv29s9S4iIhJRPc76PHzefkoU92nWt32hUhGsH8upeYJ6oBlq03Ut\n3JSI7Y/0QgGLiObGDkKcdPb5bt17bXIE38fev+qFeOrdwZt7TCjSjzFMdB1b0Zq1gGW01H477Xdr\nlkLcvkyVanRjjVr3cba2ZsaDhz5TrhDe93TClZN/ehkX8OCNP7Zqn9/0nWKmLazrhPiUFNZL/dtn\nnzXLFNTr/lEd0vSx973r3mrWUbd5y6jbGkpqnwT2RdR9EFU/vKmr8LgsIrLvzU24jDppOtXrLqiP\nWj6Pr+t5umjNbqruoadfd3+6AeKL2jaZdXTK6LVH1Th+0Si80a9Jtl+Fn8V+z16Xd6vrj84C5mN9\nwDXLwuReiCdFcR0P9eC4D2c2Yd1q0Lb8Gaab3PIp3PYgO+/EGtrZ19nxI6ql4lEfiYiIiIiIqLp4\no0ZERERERBQyvFEjIiIiIiIKGd6oERERERERhUz4BhPRRbMRjHUNc9GBQwIMvOFcM23Pa7MQ/+Zl\nX4N4VXo2xF/fZwcGGSpgcev8eqykP7Vxp1lmXw6bZHdkML5h6RMQP35wvlnHW1/zIMRXvhWLsDsK\nG80y39j+CohfN/d8iKONuyBOddtGz2Evw433F/m5s79Bj2qO2OvjwBl64BARkUSRNpp6mWhAw+tq\niMjog5rUO2xMLWKbZE+O4qAm63O2YDjhcL3dHr6PTUn8LB3TQjx4SDHlFNp7F58OcdtTg2ae9m9i\ni+EDbz8b4o7X4eAhAT1PRdaNGBAibQvOaVg18q/IoAGl+P0pDWba5c/hoDGX1eHrnPEv7zXL5Jrx\n2PnV9+CgXnNi3RDvfwAbOIuIzJYaHExE7cfAz2aRJuTRnoFRfy4iksvhwCDJJJ4P9MAhIiLRqB4Y\nBH8e1PDaU4OHJFP4OgcH1UAUeTwPD68lYNqI7Qh6P2r4eDyRggZm0TkYqcdBPb54/Xcg/tfO88w6\nbp78MMQt6nq/J2Bwt01FGliv626H+E1TnjDz6IHlNuXwYnHNBf9jlvn5QCPEnzwV2+B9+h3vgLj1\n+zioXyX4jRoREREREVHI8EaNiIiIiIgoZHijRkREREREFDLhq1Er5VnsIrb/0wUQ/9X1v4T4ZfX/\nYZbRDTm/vA8byOv6s3ObNxfdDt1Q2fPtfbFuZJz38NnZ1T1YGze3ARsSB7ll43UQJ1+5NWAurJfb\n9DmsUfvW6/8L4l92LzNrWHPToca17oVHim7XeLvxfcsh1k0OB7zJZpkpEaypOS0xBLFuRC0S/Az+\nREioXDqgcharAF40WTXjblLNbTcX8LlsEZHpUWycuruA9QMPnPxziK90WLckIqwVCCFdh1DKsXfL\np/G4kZuEOXjC12yT3d0fOAfi1H7Mhek/Xgdx/oS5Zh0ja5N95lJlijTFdnFbJOjn1dFE7wO1zjt3\n2HqNjTmc58qZWDveJsVrPLrfjXUxKYfbNf/7ttlt2Gury+VieLz3c1gr7CdtbU+mgHVeXg6P/7F6\n/PlQzp7/Ugn8zOcKOE/Q+THv4es0prDOdCiL23rP9hPMOmbKGjMNBDZqr7ze8lhUyrlg3WfwGvpV\n9Y9CvCaNNakiIr/sPxnim1vWQ9wUs/X8W/Oj1yZq+jpcRCQmmKNZdW2+J28HOGiN4Dz71bVR1yvx\n+jLbjPchIiLTvv6omVYKfqNGREREREQUMrxRIyIiIiIiChneqBEREREREYXMhNaoFS45w0zb/kqs\nd4keh8+K1qkeTadN223WcXbqDxCvG5wO8YMHlphlFjRgz7PWGNYqHVeH21EIuMfdk22FuCmKz6wG\n9eBKq54QdaqPVU49O9uVsTVDB7L4nP4/LsKavOgm+4z4vBjWjiwfwN//+534fG170taavPCXh7Yl\n/cnw3fO/uUn1a1Jv/5SofQ5Z9xH7Wf9CiGfGbI1gVPD9LRyhPmljpfOt27Mf9/nxAxDXRzAfg36X\npKqFq1f96O7sn4ILsIaoJug6hOhJx0O87V9s/tRFsW9Vrgv7ZW2+fpJZpmWj+ryo0hlvwUyIIxlb\nH1HzGaXrwkTERVX9T1CNjWJ6RVWh51nR1xAp+pk++yncZzdsfo2ZZ+DlnWPelkgqBbGuSftZz5kQ\n53diP9Bj2eD8VjMtk8Pzeiw5ei1SY73tWZjNj34ZqXumiYgkYvg6mRyuQ9e1Ba0jumQRxIX1myDW\nPXhFbB9eOoyIOhaVcFz5z2tuh/j5LNb3X9OI12MiIptyeK3w7/uxnv2drY+bZU5J4D3Anf3YK3Fa\nPdbCdRfw+vhFuG2eus6ZEbPX2dtUXe5Hn3otxMe946mA16mO8F1dExERERERHeN4o0ZERERERBQy\nvFEjIiIiIiIKGd6oERERERERhcy4DiaSndEgO/7i0CAVZ1xjGxaenMQBHqKC1Z+9+TqIG2K2uHVv\nphliPajCzLoes4xuwLgjjUXwG/02iFNR2z5YN6uenMABSYIGE5kUx3mSEVxvWwLjKfEBsw49aMmG\nDA6ekvZto8tnVTn+oIdNTaeq/TA/1WXWETbRxTjwx4zYaohXZrAIdWYU33sRO+BGVjVL1E2lReyA\nL3oe3SQ7qAFjg8NtC3odTe/XrHod2wDbFtUeH8di8j5VNNyZn2aWWRzHz8+A+uy8Wg3Mc6vgfqlV\n5TSEHi962/RAFJE6HIRBRKTQqwYIOgcblnqfwYFmBjdj0baIyPRZOLjO9He/gDOUMJCMdzEWkPcs\nwULuSfdtNstMaNvagIFAijV8Nj8PeF/ClE+ghIEEBn6Ln/GfbsRjzZw32IEEjBIGMHAJPE/NjOGx\n6Kdrl0G8UPAccFQrMlJGx/kBjX/VQCAJ1bw6GsF1prP2WqIhhecu3ay64NnvA3SD694hPD7F1Ovq\n+UVEsrNacFvXqxn04DwiImH9jI2nUo5fJXzmu284H+Kr6vGztjaL65wesDuOU9cS8+MrIf7M3ivM\nMs0xvN59bSsuk4jgtq9N4+BUIiL5xn0Qp/2AjVN+dPBc3NY3P1N0mWrhN2pEREREREQhwxs1IiIi\nIiKikOGNGhERERERUciMa41asjMj8/9rw0vxrhXHmXmevFA9A30C1kotm4UNLOfVYT2MiMjSemyC\n3RDB55t1k2kRkbhq2nt2Iz7nem5qB8S5gEbAKdWksUU9c1/v8Pn6F1939Gdjt+fx99+Rt3VG3R5O\nG/Cwabjn2/vxzjzW8bWoeq1dGWyOeTCPjWxFROb85tC/99t+2OOu4/L2UX+eVrVhrRH7vHpPHvdr\nV64J4mWpbWaZXh/f74J6v3VNWikNsY9E02ydJyIiuwv4vLeup1uYwGe5RUTq1TPuner3Szr7+Toa\nlFRDFPT8P6zkyLRq1tuma9ZMPZrYms6Nf4vbHnl4LsRtZ9o61earN5lpY6ZKa7woboffE4KDy0hB\n+7BYTVoZ3NlYM7jupjozz9JP4rkuv2Pn6CvVdWAiph4l0oDHem/A1kVv+CrWa1wxGes1tl6FDWVL\nUkJdjG6+nVJ1uG6rfY8Ms6+KL1ILAhuTj5BbkLYT83iuaqjDa6VUHI8rQTVqunl1No85FlSjpjUk\nsc6tbwjPqamEHRNg/4lY1zbtfjWDd5Ts2Gqr0jnoNX+Hb/id/XhNecfeyyG+fMpas462GDanvq4R\nj/X/Nfsxs8y3e0Yfg+G7834Pcca35+3OAuZbk2mObq/Vp8b71RRb9w1KqQUsEb9RIyIiIiIiChne\nqBEREREREYUMb9SIiIiIiIhCZlxr1EREZMSzoHWPbzA/nne37XE2Uk891tk8fNLZZp6DJ2Afnr55\n+KxoeoZ9lttPqmn68dKI7jNhnz+N7cdnZWMDOE8S2xK9OK0b15vqxu1IHsBnaaP9tp9IpG/0egA/\nZZ+3LVpLsxtrk9Z12+d86/wnDm2Db3uSjbeIfYwd7C9gXsQTWTNPRNUZLq3DmsiELqgRkb7/v707\nj46rvu4Afn+zaOTRYlmyLMu2ZMnGChhoWAwYCEsCCYsTmgRCkxwSmoXkJEBYm2KcnCyQtqEHSDjh\n0JoToBRKSgIUNyFpgwsUcAO2D2YxxniVsWzLq3ZLGs38+ocV0Pf+nmdGM6PRe/L3c44P3Jn33vzm\nvTtv5mnm/q6q69L1jpn6mx1aB2/rTU1Ke78Xvd2U4A7p8OijtidZkXaZD8fw+YuIlKq6yl7rkV9H\nqjGqQQNZ/PY9m3q6jbfh6yG5C4999FisF5iyyD1fF4JRtST9U/H5pfo9amuKyRgxsdiI0N3/Nonn\nBZvAc4uu6frXRfc523ih5xh1y6sQ3Tt5tWjPn4N13o8fM91ZBnjVganno2vSwh9ya8mXnP80xL/5\n/EfVEliPEqqoEC3VjfUp2fRRC9VhP1Pdw3LGS9nUkU6Av09nUWtoonhenjbVrfXsG8BlrMVcyKZS\nujyavo/aUNLd32H1Ptuvlgmpz1sDCfejatc8fM3pjp+ZavZohAyvvcT5JzurLK65H+LVg7jObQ3L\nIH5z0O3D+dTekyD+9904joZJ2KdTRORHdSsg3qxe8t9uw3PR3TNecLZRFcJ82q+2kbBu7mzqq1W3\ndDvLgAJ+FpgAZywiIiIiIqKJhRdqREREREREPsMLNSIiIiIiIp8pao2aHRqSZPsHtU/hqsnugOY0\n4TpOfwMU2t3h3FazEXvJTC3D2gs74NYmaSaifrOrf28adn8jbuOqr4Laho25PUhSJarnSByXGaxU\nv/eejv1FREQGK7DnmW4T51W7lVJHfiiO+znaXQ1xOOH+3rZyy4jakTUrnPuLre4P2/CGH2OYEt3f\nzK030z05dL1Zr3WPoa59KzW4w/XjxkNuPZ/uB6THEfZo9pOp9k2Pw4t+fnHVc7Ai5O6jPvVacPv0\nuetMCFn0ygrXYaVEqgHj3ga3H2H8qVdGN44cfvu+8WcLndtMEo91w/xdEMc+sXXUj6PrYrzo+i0b\nUTXENT7rf2St2IEP9lUuozv6OOzDeWap+zfSpKyHuETw9fzywSZnnYWTtkC89MufgbjqYbcPkSND\nPjU94vZmu/1PiyBued2tnxvJqUfL0UAjvi+1qX6gsWdWFuRx/M54fC7Sb2fhqbiv9hxw6wSnV2Pd\n2oFerI2uLcN6xd0Jdxthj/eIkSJh935dCx5Vy1hVI1QScWuGypvTz2eQTT1mUWqKi03Vmxn1WdWz\ndi9DD8MZP3b7ZR5I4dwIKYuff/dY/Nx9cbzd2calqueZ3ubSA1jDJiJyz/4PQ/yRcjxvXlL9GsR3\n7f8LZxvfmILnq4T6vNWTcueC2NJVA3EsU41aAfEbNSIiIiIiIp/hhRoREREREZHPZLxQM8Y0GGOe\nM8a8bYxZa4y5bvj2amPMH40xG4b/O2Xsh0tBwtyhfDB/KFfMHcoH84dyxdyhQsvmG7UhEbnJWjtf\nRBaKyNXGmPkicouILLfWzhOR5cMx0UjMHcoH84dyxdyhfDB/KFfMHSqojJOJWGt3isjO4f/vNsas\nE5GZIvKXInLu8GL/IiLPi8jfjubBkx0exaBet6Xh1UjTxFRBu27+WuWuYyfhOqmS9LvGRtxrXD3x\niRnKPKmCDeN2jCpuLenAwvv4Vo/G0qpA1kZV8ajXc9Fj089H3R/qdh83uXFEIbt1m26PZe542X7Z\n7LT362bOHSn3+JyqJnx5uR+fd0fKbRqtJ/EoUxNyJNVkG/16thcR6VATcETVRAJJj9ajpWqWGD35\ngF6nIuweoz2qGF9vs9SjuW+/ylE9qUmhJhMpdv5kMaCMi/SdjDnY1YivvViXxwQklXgMkl1uY9rR\nCs+bA/HJC9xm1VNKMLe3nurmx6ipGQ2yajqr0iUxK/NkTxmHUcDcSVXF5eC5p74f62MqIlL/wOu4\njmoafUbN5oxjfmdgBsRv9c6EeO8ATlokIrK9Agvcb1jyK4gffDj9OdHL0LONEF9b+ytnmdbv4UQV\nWbSZLoiBKXju3DE0+i8lnIk4PFLUd+ceLYum3YNHYYPhijKP92gVl5bg+b8siu9luiG2iEi5WiZe\ngpNK9A64Ewyl1HYmx7Cp/Z4hnHRpcMidvG1QNcEe2ZReRGACoPeX0RNr6M+GBTCmuePxfhxSzzvV\nj/vSZpgoxMvWn5wO8UOz/tFZ5uubL4X47qYnIO5TnwvWJ9xjWGrwGE1Wjc4/Vv62s87CUo9m7yMc\nf9dXIR5Y0OMs872z34F4cwL3Udxjsp4d+3Dyw+a0oyisUdWoGWOaROREEXlFROqGE1JEZJeI1BV0\nZDShMHcoH8wfyhVzh/LB/KFcMXeoELK+UDPGlIvIEyJyvbUW/uRrrbVymFmLjTHfMMasMsasSoj7\nFw6a+Jg7lA/mD+WqELkzNNDrtQgdAXjuoVwxd6hQsrpQM8ZE5VDCPWqtfXL45nZjTP3w/fUisttr\nXWvtUmvtAmvtgqi4PcBoYmPuUD6YP5SrQuVOJOb2vqOJj+ceyhVzhwopY42aMcaIyC9FZJ219q4R\ndy0TkStF5B+G//v0mIwwA89Gmpn60O3KcH+O0rfmzm6dTNsY/a+NCyOXxy127kTO25v2/u4kNvPc\nn3J/O69/d3z97VdDvOyH7m+1J4dwO1uGcG8lVI1aRwrHIeI2uNZ1b141arrR9KBapCaENQm1HjVq\nLXH8EPqVbWdB/OnGF5111g161EmmEWlqdG4b2rrNY0lU8PwZ+fv+XBqdZtEsVTfcrc1is2Pyml6K\ndQpX1LmNj3/x15dDbGRN3g/r1Hx41FRoJoX78aS5rRDn0la0kLmTjBnpnPPBW+X/3nSns8yz1+Kv\nmFoHp0J8vqq12OZRG9OTxNqeT1bh8fhE3G1gP2DxtpjB88jif/ocxB+63/12sP/v8bYH5z0C8ZfW\nfdlZp6wtc83dWOipx/qUjf2j//WYTWV+7fv9c0829h2L+VRX4V4XtHVi3c2MSqyP7U3ghULYo/F0\naRhzsKoU32e86GiMigAADoxJREFUatQOJjBPGysOqMfFdby2MSmGtazhWnzNDW1vc9bJprYvX2Oa\nOx7vObomzRlPRNXyHX2Us8w71+G8DVsW3QfxNW3nOeucOmUrxDdsxZq1Wxt/C3FTxK097lD1ybq6\nfVDS16OJiJx2y7cgnvHwCoh7/oD12l7043hlSaLLzUEwhs3UM16oiciZIvIlEXnTGPPnd45b5VCy\nPW6M+ZqItIrI5YdZn45czB3KB/OHcsXcoXwwfyhXzB0qqGxmfXxJDv9Fj3uZTTSMuUP5YP5Qrpg7\nlA/mD+WKuUOFNvbfAxMREREREdGoZPPTR6JAmBTFuo8tCeyf0VCyD+KE0//LVf0A1vacccqNzjK/\n+PjDEM+J7If4BNXnZPlB949tNaH0dV+DHn9T0TVqXSmsSWguwd+ED3j8ZvqmnSdB/NbS43CB290a\ntYQai66vE8GahW2Xz3K2MeOOzDVqBZfvb8azWV/9Tj32PNbQnFXj9jP7t/sugHjavSucZTLZdOdC\niNe13Atxy++/6azT8vKqUT/OWLAR3Gct5VhLs3qc/54Ybe+V6Xd/cEyWfPFcZ5nvTPsfiI+P7YS4\nX51rnu9rcrYxS52f5pdg3c5qjzqd2jC+xkNqlrgtlyzFFS5xNiGvDuDrtV3V8sZvwz5/nkLqXJpD\n76ZsDGJJlWzs1VWg+yWjMRqb3wxMwddVZYlby7Q1gf3wGssx5zZ04v6NRNwemfp9KGJwmVjUrcfs\n7MUcm1u2B+KdfZhzA0PuR9VIWNWCN2KNmvGqUZuA9l2FPc+e/D7W0eteqNPC7nk/qfpfvpvAOsMb\npi131vlB2ychvnbWsxA/su8MiG+e9pyzjbj6KNSZwvPI2fiRRkRETlmCNWnVD7v11yOVhEf/ek95\n9IKN7XL73440lj36+I0aERERERGRz/BCjYiIiIiIyGd4oUZEREREROQzvFAjIiIiIiLyGU4mQhOG\nnuqhOVoO8duJ/Is7W771qnPbPXL0qLYRKitzb6ueom5QVbZeTVrV5BZWNb68cy9OTuANi2arRRXm\n3u6uEVZ7uiyEExjsTmID3ekXvOdu5I4shlZAqaoy6fvYae/H4QG3WLikEydViOzG5q/ShZPT2D63\ngXiqB5fpGsBq6CsqX3fW6bwqDvFrv22CeKgV91/3X+HEISIiv/nszyH+SitOUHL0NW+5Y3VuGQNZ\nTMBiVa53DMXVEukbuhbbyzuandvunoHH+Xd9OOtFhWo+f9akrc42ouol36r2Q3XI3Q9JvXvVNt4Y\nxHX2J/W+FRHByY5e6m3BTb6cRSN0m382hcpwbKlut9V5YjI+4fV7p0E8TU0m4nWuTfW6Tb8DR78/\neOibje93Pap5tYjbo3dGaQfEK7Y3QVxa4jZd1xrL8Bi81zXZWSaRwIkXmmM4mcjaWD3EvYPuRDoh\ng7kwOBmXcZ+tZLXf/Cwyp8m5bfHfPApxbwq/f9mcxM9B7xiv1yquU2pwX9ao93gRkZ81/A7iG7df\nBPFlU3HSks0Jd1Ki00txu7MiOGHHR974rLNO9YPpJw/RdHN1L/0pXCZp3ebcJZ0ZNjKGzdT5jRoR\nEREREZHP8EKNiIiIiIjIZ3ihRkRERERE5DOsUaMJY/KVWB8kr2E4M4w/Mo56/FZ7wI79S8KrRsKv\ndROP97j1BWeU4n5eO4i/ga8JYR1D66tuw+tm8ahbG0PJmEhX0wd1ET2NHrUKU7Gmo6wCqxwSCax3\n6T/g0Y0zhds1O7DZ5jn7rnZWiazF7cawj6h0noZNZz/a4tab3bTpcxCX3IzHJNX/trNOKK5qgvrS\nN10fK+FezJf/fuEEiOfKn4o5nIxq73CPe/TXWHNzURwbB4fU30S3eZTLrk/ga60jiXmRCKvzm4hU\nqLq1CvXai6pKxFLj1hjNjmD93Pd/eA7EcXnFHewYNLg2umDKQzKGdUkdezHPsWLNbUJ7RFFvbz2D\nbtVWXNUIdQ5hI2pdS+bVvLq+FN9Xj4/juf3F1FxnnWg0fb5EQjj4RNL9TqE0gmMxmcth3abEmVfx\nldbLZzi3nRDbAfHzfUdB3BDFWnW32k+kNoyfP0oNHp9+676OqtXe+7uZz0D8jKp1/WmrW1t94xxs\nkn1pOdaFl1242WO0KNP7WFefx/u0UhXO/N4Xby9KVbcnfqNGRERERETkM7xQIyIiIiIi8hleqBER\nEREREfkMa9Rowki274b44vOwbuf6//wPiOdFsY5EROSUlV+FuF7W5T8wVc/hVTdhwvg3E5tF/ynP\n3mojt5FUdQBedSS6LkQ97q0r3T4mb5zzzxDPjWIPnEXrPwNx8+LR9T0ZC9H2Xpl+94q8thGZifUB\ng3PqnGX6a7ECoHsW1oVY49aJ9DaoOqJTMS9byrE26cXnjne2cdRDeAyS692aNG28atK0cC/2rHn6\nsqUQ33jz6cUcTkZefcUumIF1dV1fwHqMs7+LdXY/rXO3Mdep2+lylnHpihOvCpT0rnoPe+7Fn/So\nSSsC53zl4cQFGyFet9t9DcI2szmPTlChQXxPSaQ86rxUzdmbB/AcZ9U6/YNuT6ryMNa59VvMwc5O\nt3dfSamqYx6YCnFE1Y+nPMauRQ5m7pOaTY75WcO9bzq3LfnUJRBfNf0FiOdEsIbQq9JK93Dst3hD\nwrr7f0dS17/i/VdUbIV44dFuvdkxUcyns67+NsSe9bGKHXR7no00lMhcp9qdwjq2eMjNk5Ke9DVq\nJoqXUzaRflyjwW/UiIiIiIiIfIYXakRERERERD7DCzUiIiIiIiKfYY0aTVjJdRsg1r0ymqPYg0dE\n5IS6Nojb1f3hKrevWLKj07kNqNow61ErZt32RkVhIvgbcf276tI3sa+OiEjP2TjY2eos0nl/A8SV\ngvs0qIbasF9NSMUiIroaw63OGD1dZdPssT+DXHmRXLse4k8tux7ieVnUKfhN5WNYk7bmMbz/AsGa\nNhERc/KxELefpvqqHefW4JTXY/3izMl4LrKq1mRTO9YCiYjM/aJbL4cD8+hvVoC+ac4ms6iZ3HkP\n9uSa/Qb2iNKjsgcPypGqau5+iBsqOpxl+oawnmxO+V6MK3D/Vkbc/bmgDGuP5qm+Xc/MdmtqT6zC\nXms/qMWa2msGKyCeWu72GQ3pM+NAkM+C2Ul1dzu3HTgT4ztbPg3xhq9jd8FF5610tvGjuhchbgy5\n7/v5qg25fSDPvhnP9ZVPFb5nptns8S6MrSJlfhTPm492z3ZWqVi1HWJ9NraJzDWSueI3akRERERE\nRD7DCzUiIiIiIiKf4YUaERERERGRz/BCjYiIiIiIyGc4mQhNXKoI/qqfXwdx6X63GWp5G06mEZHV\nEKd6J1hxuk3fxLF0j7uPdiWxgWSHahZp0m+SKK153wne5CGFYFevhXgannoEpwQ4zDYy3D9XtmdY\nwmujRWoancXjlP8acyPT9BF2aOwK/MdVFo2be9bUQLyypspZJrYHPwJuGWiGuHQvHhPj8bC/r8fm\n7v3TcaHqNe73Aa0xnBTmkQac3UFPXxPu85jQ5nicWGNO626IPY98wBteZyP57iaI53wX43Ue61wu\np0OsJzbqbMHJXURE+qfgcZ20H9/4K9fj8bGv4flNRKRSCj95iNa05P+c285a9U2I4zvxc12k3Z0g\nbqitNe3jFLLBtcZv1IiIiIiIiHyGF2pEREREREQ+wws1IiIiIiIinzG2WL8/FxFjzB4RaRWRqSKy\nN8PifhGUsY73OGdba2vHauPMnTE33mNl/riCMtbxHidzx8WxZo/54wrKWMd7nMwdF8eavazyp6gX\nau8/qDGrrLULiv7AOQjKWIMyznwF6XlyrP4TpOcZlLEGZZz5CtLz5Fj9J0jPMyhjDco48xWk58mx\nFh5/+khEREREROQzvFAjIiIiIiLymfG6UFs6To+bi6CMNSjjzFeQnifH6j9Bep5BGWtQxpmvID1P\njtV/gvQ8gzLWoIwzX0F6nhxrgY1LjRoREREREREdHn/6SERERERE5DO8UCMiIiIiIvKZol6oGWMu\nNMasN8ZsNMbcUszHzsQY84AxZrcx5q0Rt1UbY/5ojNkw/N8p4znGPzPGNBhjnjPGvG2MWWuMuW74\ndl+Ot1CYP/lj7jB38sH8Yf7kirnD3MkH84f5k6ug507RLtSMMWERuVdELhKR+SLyBWPM/GI9fhYe\nEpEL1W23iMhya+08EVk+HPvBkIjcZK2dLyILReTq4X3p1/HmjflTMMwd5k4+mD/Mn1wxd5g7+WD+\nMH9yFezcsdYW5Z+InC4i/zUiXiwii4v1+FmOsUlE3hoRrxeR+uH/rxeR9eM9xsOM+2kR+XhQxsv8\n8c8/5o4//gUxd5g/4z+2IOcPc8cf/4KYO8yf8R9bkPMnaLlTzJ8+zhSR90bE24dv87M6a+3O4f/f\nJSJ14zkYL8aYJhE5UURekQCMNw/MnwJj7via748H88fXfH08mDu+5vvjwfzxNV8fjyDmDicTyZI9\ndMntq14GxphyEXlCRK631naNvM+P4z2S+e14MHeCw4/Hg/kTHH47Hsyd4PDj8WD+BIffjkdQc6eY\nF2ptItIwIp41fJuftRtj6kVEhv+7e5zH8z5jTFQOJdyj1tonh2/27XgLgPlTIMwd5k4+mD/Mn1wx\nd5g7+WD+MH9yFeTcKeaF2koRmWeMaTbGlIjI50VkWREfPxfLROTK4f+/Ug79rnXcGWOMiPxSRNZZ\na+8acZcvx1sgzJ8CYO4wd/LB/GH+5Iq5w9zJB/OH+ZOrwOdOkQv4LhaRd0Vkk4gsGe8CPTW2x0Rk\np4gk5NDvgL8mIjVyaCaYDSLyrIhUj/c4h8f6ETn0Fe0bIrJm+N/Ffh0v88c/x4O5w9xh/jB/mDvM\nnSDlDvOH+XMk544ZfhJERERERETkE5xMhIiIiIiIyGd4oUZEREREROQzvFAjIiIiIiLyGV6oERER\nERER+Qwv1IiIiIiIiHyGF2pEREREREQ+wws1IiIiIiIin/l/MCmhDTs4mT4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x360 with 12 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=6,figsize=(15,5))\n",
    "ax = axes.ravel()\n",
    "for i in range(12):\n",
    "  ax[i].imshow(training_images[i].reshape(28,28))\n",
    "  \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "11faCfet_5vO"
   },
   "source": [
    "### Reshaping the images for feeding into the neural net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1Zdi__3XFbWd"
   },
   "outputs": [],
   "source": [
    "training_images=training_images / 255.0\n",
    "test_images=test_images / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XiHJI37-_-1-"
   },
   "source": [
    "### Keras model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 104
    },
    "colab_type": "code",
    "id": "uC7bTBFiFhd2",
    "outputId": "5c110771-492c-46d9-c504-9b238bf7309a"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0703 03:03:41.810330 139679994218368 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(256, activation=tf.nn.relu),\n",
    "  tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VxjlhCk0ADOf"
   },
   "source": [
    "### Compile the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Zt31P1knFhmt"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r6CiYgYvAGXD"
   },
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 351
    },
    "colab_type": "code",
    "id": "FN6WIRO7FldB",
    "outputId": "431bf8bf-4b7f-414e-a7a3-f38f4bd61225"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 6s 108us/sample - loss: 0.4856 - acc: 0.8274\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.3642 - acc: 0.8678\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.3274 - acc: 0.8804\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.3064 - acc: 0.8865\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.2853 - acc: 0.8939\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.2716 - acc: 0.8993\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 6s 97us/sample - loss: 0.2577 - acc: 0.9037\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.2474 - acc: 0.9072\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 6s 97us/sample - loss: 0.2382 - acc: 0.9103\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 6s 98us/sample - loss: 0.2292 - acc: 0.9139\n"
     ]
    }
   ],
   "source": [
    "t1=time()\n",
    "model.fit(training_images, training_labels, epochs=10)\n",
    "t2=time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cPnbHMWuAJ1U"
   },
   "source": [
    "### Print the time taken for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "fsd_1F5jFlha",
    "outputId": "a723ebf7-cf74-4e55-b712-741024b53f05"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training wall clock time: 60.275 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\nTraining wall clock time: {} seconds\\n\".format(round(t2-t1,3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fA3OIbQ0AMrF"
   },
   "source": [
    "### Compute accuracy of the model on the test/validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "5Zy6ESSvFpZP",
    "outputId": "4cbaeef3-79ca-4c0b-8159-aa9739b37798"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 67us/sample - loss: 0.3636 - acc: 0.8763\n",
      "\n",
      "Test accuracy:  0.8763\n"
     ]
    }
   ],
   "source": [
    "test_loss = model.evaluate(test_images, test_labels)\n",
    "print(\"\\nTest accuracy: \",test_loss[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zldEXSsF8Noz"
   },
   "source": [
    "Your accuracy is probably about 89% on training and 87% on validation...not bad...But how do you make that even better? One way is to use something called Convolutions. I'm not going to details on Convolutions here, but the ultimate concept is that they narrow down the content of the image to focus on specific, distinct, details. \n",
    "\n",
    "If you've ever done image processing using a filter (like this: https://en.wikipedia.org/wiki/Kernel_(image_processing)) then convolutions will look very familiar.\n",
    "\n",
    "In short, **you take an array (usually 3x3 or 5x5) and pass it over the image**. By changing the underlying pixels based on the formula within that matrix, you can do things like edge detection. So, for example, if you look at the above link, you'll see a 3x3 that is defined for edge detection where the middle cell is 8, and all of its neighbors are -1. In this case, for each pixel, you would multiply its value by 8, then subtract the value of each neighbor. Do this for every pixel, and you'll end up with a new image that has the edges enhanced.\n",
    "\n",
    "This is perfect for computer vision, because often** it's features that can get highlighted like this that distinguish one item for another, and the amount of information needed is then much less**...because you'll just train on the highlighted features.\n",
    "\n",
    "That's the concept of Convolutional Neural Networks. Add some layers to do convolution before you have the dense layers, and then the information going to the dense layers is more focussed, and possibly more accurate.\n",
    "\n",
    "## Adding convolution and pooling layers\n",
    "Run the below code -- this is the same neural network as earlier, but this time with Convolutional layers added first. It will take longer, but look at the impact on the accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C0tFgT1MMKi6"
   },
   "outputs": [],
   "source": [
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zrS0jiQ7Gn_U"
   },
   "outputs": [],
   "source": [
    "model2 = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(128, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "colab_type": "code",
    "id": "Rfen5MMdGoCZ",
    "outputId": "65452bb9-7c7a-456f-f553-e05aeab80c0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_8 (Conv2D)            (None, 26, 26, 128)       1280      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_8 (MaxPooling2 (None, 13, 13, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_9 (Conv2D)            (None, 11, 11, 64)        73792     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_9 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_5 (Flatten)          (None, 1600)              0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 128)               204928    \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 281,290\n",
      "Trainable params: 281,290\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 184
    },
    "colab_type": "code",
    "id": "vGuETdmCGtok",
    "outputId": "ba518720-08ae-4f6a-e3c0-412362b92876"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 19s 322us/sample - loss: 0.1176 - acc: 0.9634\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 19s 323us/sample - loss: 0.0381 - acc: 0.9881\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 18s 304us/sample - loss: 0.0264 - acc: 0.9914\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 18s 303us/sample - loss: 0.0187 - acc: 0.9939\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 18s 296us/sample - loss: 0.0154 - acc: 0.9951\n"
     ]
    }
   ],
   "source": [
    "t1=time()\n",
    "history=model2.fit(training_images, training_labels, epochs=5)\n",
    "t2=time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 288
    },
    "colab_type": "code",
    "id": "Co9uXb3pIFal",
    "outputId": "92b8ffe0-7ca8-4fc9-9a57-cb864e0314fe"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEPCAYAAABsj5JaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcFdWZ//HPQwMKNtIKBFCRmAn5\nJaggS1DUmMYtmBlRkVeisogGGYOY+FNiVEbMYNBE/U1CRFExRBEUHaIGDaMwistklAABWaIgisqO\nLM0ia3c/vz+q2lwut7vr3r5b9/2+X696UVXnVNVD6a2HqlN1jrk7IiIiyWqU6wBERKR+UgIREZGU\nKIGIiEhKlEBERCQlSiAiIpISJRAREUlJVhOImU02s81mtqyacjOz35nZKjNbYmbdY8quNrMPw+nq\n7EUtIiKJZPsO5Amgbw3lFwGdwmk4MBHAzI4F7gJOB3oBd5nZMRmNVEREapTVBOLubwHbaqhyCTDF\nA+8CJWbWHvgeMMfdt7n7dmAONSciERHJsMa5DiDO8cCamOW14brq1h/GzIYT3L3QrFmzHh06dEg5\nmMrKSho1yr9mIsWVHMWVHMWVnIYY18qVK7e4e5va6uVbAqkzd38MeAygZ8+evmDBgpT39cYbb1Ba\nWpqmyNJHcSVHcSVHcSWnIcZlZp9GqZdvaXMdEHvLcEK4rrr1IiKSI/mWQGYCQ8K3sc4Adrj7BuBV\n4EIzOyZsPL8wXCciIjmS1UdYZvYMUAq0NrO1BG9WNQFw90eAWcD3gVXAHuCasGybmd0NzA93Ndbd\na2qMFxGRDMtqAnH3K2spd+CGasomA5MzEZeIiCQv3x5hiYhIPaEEIiIiKVECERGRlCiBiIg0MJWV\nlezduzfjx2lwHxKKiOSbqgt61bRnz55DlqOUJbPNgQMHADh48CCNG2fuMq8EIiIFJ/6CXpcL+Jo1\na2jevHmN21Rd0LPpiCOOYN++fRQXF2fsGEogIpJ3Kisr2blzJ9u2bWPbtm1s3bqVd999lzVr1qTl\nor9///6s/52aNWt22NS8efOE6+tadsQRR/Dmm29mNHmAEoiIZFBFRQU7duz4MhFUJYPY5URl27dv\np7KyMqOxpevC/dFHH/Htb3+7xu2OPPJIzCyjf59cUAIRkVqVl5dTVlZW7QW/umRQVlZG8H1w8o4+\n+miOPfbYL6f9+/dz4oknpuVf6um8oOdrZ4rZoAQiUkDKy8urveBXXfQ/+OAD7r333kPWl5WVpXzM\nkpKSQxJB1dSqVatq15eUlNCkSZND9lPIF+p8pQQiUg8dOHCA7du3R3ocFDvt3LkzpeOZ2SGJINHF\nP9G6kpKSjL4FJLml/7IiObR///5Ij4Ti1+/evTul4zVq1Ihjjjmmxov/xo0bOfvssw9Z37JlS4qK\nitL8t5f6TglEJM327NnDxo0bD5s2bdp0yPLmzZvZt29fSsdo1KhRrY+BEq1v2bJlraPU6VGRRKUE\nIhLBwYMH2bx5c6TEsGvXrsj7bdy4ceR2gdjlFi1a5OUwqlJYlECkYFVWVrJt27ZISWHLli2R99u0\naVPatWt32NS2bdtD5lesWMFFF13UIF/vlMKgBCINiruze/fuQy7+b7/9NnPmzDksMWzatIny8vJI\n+zUzvvKVr9SaGNq1a0dJSUmkpLBmzRolD6nXlECkXti/f/9hdwXV3THs2bMn8n5LSkoSJoX4xNC6\ndWu9TSQSR78IyZmKigq2bNlSbVKITQzbt2+PvN9mzZodkggqKiro0aPHYYmhbdu2HHnkkRn8G4o0\nbEogklbuzo4dO2pMClWJYfPmzZG7qygqKjrsUVF1dwwtWrQ45NGQ3ioSyQwlEKmzZcuW8bOf/YxF\nixZRVlaWVEd1rVq1qjEpVCWGVq1a6a0jkTyjBCIpq6ys5MEHH+TnP//5IUmjuLi41qTQrl072rRp\nQ9OmTXP4NxCRulACkZRs2LCBoUOHMnv2bACGDRtGaWkpl156KUcddVSOoxORbNAzAUnaCy+8wKmn\nnsrs2bNp1aoVzz//PJMmTeL4449X8hApIEogEtnu3bsZNmwY/fv3Z+vWrVx44YUsWbKEyy67LNeh\niUgO6BGWRDJv3jwGDRrEqlWrOOKII7jvvvsYOXKkGrZFCph+/VKj8vJy7r77bs466yxWrVpFly5d\nWLBgAT/5yU+UPEQKnO5ApFqrV69m0KBB/O///i8AN998M+PGjdPHdyICKIFIAu7OlClTuPHGG9m1\naxfHHXccTz75JOeff36uQxORPKJnEHKIbdu28cMf/pChQ4eya9cuLr/8cpYsWaLkISKH0R2IfOn1\n119nyJAhrFu3juLiYh588EGuvvpq9RgrIgnpDkTYv38/o0aN4rzzzmPdunX07t2bxYsXM3ToUCUP\nEamW7kAK3PLlyxk4cCDvvfceRUVF3HnnnYwePVpdl4tIrXSVKFDuzoQJE7j11lvZt28f//RP/8TU\nqVM544wzch2aiNQTSiAFaMOGDVx77bW88sorAFx77bX89re/pUWLFjmOTETqEyWQAvOnP/2JYcOG\nsWXLFo499lgmTZpE//79cx2WiNRDWW9EN7O+ZrbCzFaZ2W0Jyjua2WtmtsTM3jCzE2LK7jOz5Wb2\nvpn9ztTCG9kXX3zB8OHDufTSS9myZQsXXHABS5cuVfIQkZRlNYGYWRHwEHAR0Bm40sw6x1V7AJji\n7l2AscC94bZnAmcBXYBTgG8D381S6PXa/Pnz6datG5MmTaJp06b85je/4ZVXXuG4447LdWgiUo9F\nSiBm1ilNx+sFrHL3j939ADAduCSuTmfg9XB+bky5A0cCTYEjgCbApjTF1SBVVFQwbtw4zjzzTD78\n8ENOOeUUFixYwE033aR+rESkzszda69kVgn8BZgMPOfuX6R0MLMBQF93HxYuDwZOd/eRMXWeBua5\n+3gz6w/8EWjt7lvN7AFgGGDABHcfneAYw4HhAG3btu0xffr0VEIFgu7Li4uLU94+U6LEtWHDBu65\n5x6WLVsGwIABA7juuusyOgJgfT5fuaC4kqO4klOXuPr06bPQ3XvWWtHda52AvgR3C3uBXQSJ5DtR\nto3bzwDg8ZjlwQSJILbOccDzwCJgPLAWKAG+DvwZKA6nd2qLoUePHl4Xc+fOrdP2mVJTXJWVlT5l\nyhRv0aKFA96+fXufPXt2zuPKJcWVHMWVnIYYF7DAI1zTIz3HcPdX3P0KoD3wc4I2iDfDhvDRsQ3d\ntVgHdIhZPiFcF3us9e7e3927AaPDdWXAZcC77r7b3XcD/wX0jnjcgrB9+3auvPJKhgwZwq5du+jf\nvz9Lly7lggsuyHVoItIAJfUg3N3L3P1hd+8FnAqsJ2joXm1mr5rZRbXsYj7QycxOMrOmwBXAzNgK\nZtbazKriup3gbgfgM+C7ZtbYzJoQNKC/n0z8DdncuXPp0qULzz77LEcddRS///3vmTFjBq1atcp1\naCLSQCXdkmpmx4Wv384geCvqdeAmYD8w08x+Xd227l4OjAReJbj4P+fuy81srJn1C6uVAivMbCXQ\nFhgXrp8BfAQsBd4D3nP3l5KNv6HZv38/t956K+eddx5r167l9NNPZ/HixVx77bXqx0pEMirSh4Th\n3cKlwDXA+cAG4A/AH9z9k7DaQ2ED9v0Ej7kScvdZwKy4dWNi5mcQJIv47SqAf40Sb6F4//33ueqq\nq1i8eDGNGjVizJgxjB49miZNmuQ6NBEpAFG/RN8INAdeAi4GXg0bWuK9C+xJU2xSDXfnoYceYtSo\nUezbt4+vfe1rTJ06ld691SQkItkTNYHcDTzl7ltqquTuSwga2iVDNm7cyO233868efMAuOaaaxg/\nfrz6sRKRrIv6FtZvakseknkvvfQSXbp0Yd68eRxzzDH853/+J5MnT1byEJGciPol+iNmNq2asqfM\n7KH0hiWxvvjiC66//nr69evH559/Tvfu3VmyZAkDBgzIdWgiUsCiPsL6Z2BUNWUvETSc35CWiOQQ\nCxYsYODAgaxcuZKmTZty7733ctppp3HCCVE/vRERyYyor/G2AT6vpmwLweu2kkYVFRXcc8899O7d\nm5UrV3LyySfz17/+lZtvvln9WIlIXoh6JVpD8M1HImcR9zW51M0nn3xCnz59GD16NOXl5fz0pz9l\n/vz5dO3aNdehiYh8KeojrCnAHWa2nqCr9YPh1+CDCb4WvzdTARaaadOmMWLECHbu3Em7du144okn\n+N73vpfrsEREDhM1gdwD/B9gEvCImW0HjgGKCJLLuBq2lQjKysoYMWIEzzzzDACXXnopkyZNonXr\n1jmOTEQksUgJJPwKfJCZ3QecC7QCtgKvufvSDMZXEN58800GDx7MmjVraN68OePHj+dHP/qRuiIR\nkbyW1Jjo4YeCSzIUS8E5cOAAY8aM4b777sPd6dWrF1OnTqVTp3SN3yUikjlJJRAzawN0IhgZ8BDu\n/vrhW0h1PvjgAwYOHMjf/vY3GjVqxL/9279x5513qh8rEak3onameBQwlaAfrOqeqxSlK6iGzN2Z\nOHEio0aNYu/evZx00kk89dRTnHVWdS+5iYjkp6iv8Y4DvgVcSJBArgQuAqYBnwDfyURwDc2mTZu4\n+OKLueGGG9i7dy9Dhgxh8eLFSh4iUi9FTSAXEySRN8Plj919trsPIRhmdmS1WwoAL7/8Mqeeeip/\n/vOfKSkp4dlnn+XJJ5/k6KOPznVoIiIpiZpA2gOfhG9j7SF4C6vKTOD76Q6sodizZw8jRozg4osv\n5vPPP+fcc89l6dKl/OAHP8h1aCIidZLMl+hVSWMV0DemrBuwL51BNRQLFy6ke/fuTJw4kSZNmnD/\n/fczZ84c9WMlIg1C1LewXiP4/uNF4HfA42bWlWAY2/OBCZkJr36qqKjg/vvv584776S8vJzOnTsz\nbdo0TjvttFyHJiKSNlETyM+BFgDuPtnM9gIDgGbArcCDmQmv/vnss88YPHgwb731FgA33ngjv/71\nr2nWrFmOIxMRSa9aE0g4HvqFwKKqde7+DPBMBuOql5555hl+/OMfs2PHDtq2bcsf/vAHLrroolyH\nJSKSEbW2gbj7AYLXdU/MfDj1U1lZGYMGDeKqq65ix44d9OvXj6VLlyp5iEiDFrURfTnwtUwGUl+9\n9dZbdO3alWnTptG8eXMee+wxXnzxRdq0aZPr0EREMipqArkFuN3Mzs9kMPXJgQMHuOOOOygtLeWz\nzz6jZ8+eLFq0iOuuu06dIIpIQYjaiP4U0BJ41cwOEoxO6LEV3L1gHnGtWLGCgQMHsnDhQho1asQd\nd9zBXXfdpX6sRKSgRE0g04hLGIXI3Xn00Ue5+eab2bt3Lx07dmTq1KmcffbZuQ5NRCTroo4Hclum\nA8l3mzdvZtiwYbz00ksADBo0iAkTJtCyZcscRyYikhtJdedeqGbNmsU111zD5s2bKSkpYeLEiVxx\nxRW5DktEJKeiduf+Vm113P2cuoeTX/bt28cNN9zAww8/DEBpaSlTpkyhQ4cOOY5MRCT3ot6BrOfw\nNpBjgF7ALuAv6QwqHyxatIjrr7+eTz/9lCZNmvDLX/6SW265haIiDXsiIgLR20ASPq8xs5bAS8Ar\n6Qwq15544gmGDx/OwYMH+da3vsW0adPo1q1brsMSEckrUb8DScjddwAPAHelJ5z80LVrV8yMSy+9\nlAULFih5iIgkkI5G9OYcOj5IvdetWzdWrlzJ6tWrad68ea7DERHJS1Eb0c9NsLopwTC3twJvpzOo\nfNCxY0dWr16d6zBERPJW1DuQ/yZoRI/vo8MJhrT913QGJSIi+S9qAvlWgnX7gPXufjCZA5pZX2A8\nUAQ87u6/iivvCEwG2gDbgEHuvjYsOxF4HOhAkLy+7+6fJHN8ERFJj6hvYa1Ix8HMrAh4CLgAWAvM\nN7OZ7v73mGoPAFPc/cnw0dm9wOCwbAowzt3nmFkxUJmOuEREJHmR3sIys+vN7O5qyu42s+ERj9cL\nWOXuH4fjjEwHLomr0xl4PZyfW1VuZp2Bxu4+B8Ddd7v7nojHFRGRNDP32vtINLO/A79198cSlP0I\nuNndT46wnwFAX3cfFi4PBk5395ExdZ4G5rn7eDPrD/wRaA18BxgGHABOImiXuc3dK+KOMRwYDtC2\nbdse06dPr/XvV53du3dTXFyc8vaZoriSo7iSo7iS0xDj6tOnz0J371lrRXevdQL2AqXVlJUCX0Tc\nzwCCdo+q5cHAhLg6xwHPEwyhO57gUVdJuO0OgoGtGhMklh/VdLwePXp4XcydO7dO22eK4kqO4kqO\n4kpOQ4wLWOARrulRPyQsA75eTVknYHfE/awjaACvckK47kvuvt7d+7t7N2B0uK6MIJEs9uDxVznw\nItA94nFFRCTNoiaQWcAvzOwbsSvD5THAyxH3Mx/oZGYnmVlT4ApgZtw+W5tZVVy3E7yRVbVtiZlV\njRV7LhDb+C4iIlkUNYHcRnCXsczM3jaz58zsbWAZsBP4eZSdhHcOI4FXgfeB59x9uZmNNbN+YbVS\nYIWZrQTaAuPCbSuAUcBrZraU4JuUSRHjFxGRNIv6Gu/nZtYduA7oA7QHthKMlf57T+JtKHefRXBH\nE7tuTMz8DGBGNdvOAbpEPZaIiGRO5L6wwiQxPpxERKTARf0O5BwzG1hN2VVmpkHBRUQKTNQ2kF8B\nJ1ZT1iEsFxGRAhI1gZxC8BZUIgvCchERKSBRE0glwRC2iRybxH5ERKSBiHrhfwf4v2Z2SKN7uHwT\nDXBMdBERqVnUt7DuIBg0akXYV9UGgld5ryT4VuM7mQlPRETyVdTvQBaZ2ZnAvwM3AkcTfED4GjDG\n3ZdnLkQREclHyXwHsgS4DIJxPTyuF1wRESksKTV+K3mIiEjkO5CwK5NrgW8AR8aXu/s5aYxLRETy\nXNQv0UuBdwm+9ziPYFCnxkBvgi7Z12YoPhERyVNRH2GNAyYSJA8DbnX3MwmGn60grkt2ERFp+KIm\nkJMJkkQl4EBzAHf/ELgrnEREpIBETSAHCMZPd2AjwZjkVbZRfT9ZIiLSQEVtRF9K0Hj+38AbwO1m\n9glBYvkFGhlQRKTgRL0D+R3QJJy/neBR1psEXZx0JBhlUERECkjUL9H/FDP/mZl1JWhAbwYsdfe9\nGYpPRETyVOTvQGKFHxIuTXMsIiJSj6gbdhERSYkSiIiIpEQJREREUqIEIiIiKVECERGRlER6C8vM\nZtVQXEkwuNRiYIq7b0xHYCIikt+i3oFUAKcCFxIMZds4/PNCoCtwPMGwt++bWbcMxCkiInkmagJ5\nmqDPq6+7ezd3v9DduwGdwvWTga8BHwG/zkikIiKSV6ImkLHAL9z9k9iV7r6aYJz0u9x9G3A/cEZa\nIxQRkbwUNYEcDxRVU1YEtAvnN9ZQT0REGpCoCeQvwD1mdnLsSjM7BbgH+J9w1deBz9IXnoiI5Kuo\nCWQ4UA4sMbOPzOxdM/sIeA84GJZDMFb6f6Q/TBERyTdRe+NdHd599Ad6EjyymgvMB14IB5rC3R/K\nVKAiIpJfIvfGGyaJP4aTiIgUuKS6czezRgR3H0fGl7n7x+kKSkRE8l/UL9G/AjwM9OPwt6wM8ATr\nRUSkAYt6B/I40Bu4k2D88wOpHtDM+gLjCRLO4+7+q7jyjgQfJrYh+EhxkLuvjSk/OozhRXfXULoi\nIjkSNYF8F/ixuz9dl4OZWRHwEHABsBaYb2Yz3f3vMdUeIOhT60kzOxe4FxgcU3438FZd4hARkbqL\n+hrvFoIOE+uqF7DK3T929wPAdOCSuDqdgdfD+bmx5WbWA2gLzE5DLCIiUgcWvoFbcyWzIcC1wPfd\nfU/KBzMbAPR192Hh8mDg9NhHUWb2NDDP3cebWX+Ct75aA9sJEssg4HygZ6JHWGY2nPC7lLZt2/aY\nPn16quGye/duiouLU94+UxRXchRXchRXchpiXH369Fno7j1rqxf1EdYFwEnAp2b2DlAWV+7ufnWS\nMVZnFDDBzIYSPKpaR9Ab8AhglruvNbNqN3b3x4DHAHr27OmlpaUpB/LGG29Ql+0zRXElR3ElR3El\np5DjippAvg6sD+fbhFOs2m9jAuuADjHLJ4Tr/rEj9/UEHyxiZsXA5e5eZma9ge+Y2QigGGhqZrvd\n/baIxxYRkTSK+iV67zQdbz7QycxOIkgcVwBXxVYws9bANnevBG4neCMLdx8YU2cowSMsJQ8RkRzJ\n6pC27l4OjAReBd4HnnP35WY21sz6hdVKgRVmtpKgwXxcNmMUEZFoqr0DMbNrgT+5+9ZwvkbuPjnK\nAd19FjArbt2YmPkZwIxa9vEE8ESU44mISGbU9AjrcWAZsDWcr4kTPmoSEZHCUFMCaebu+6vmsxGM\niIjUH9UmkJjkcci8iIgIJN8b74kEr94m6o339cO3EBGRhipqb7zfAKYB3Ql6342n3nhFRApM1DuQ\nx4BjCb7ZqFNvvCIi0jBETSA9gIHuPjOTwYiISP0R9UPCT0myvURERBq2qAnkZ8AdZnZCJoMREZH6\nI+pdxR3A8cCqsIuR+N54cfdz0hmYiIjkt6gJZG04iYiIANF7470y04GIiEj9ktXeeEVEpOGoqTfe\nscAj7r4+nK9RbI+6IiLS8NX0COs64HmCkQivq2U/DiiBiIgUkJo6U2yfaF5ERATUBiIiIilKtjfe\nnsA3SNwbrwaUEhEpIFF7420NzAZOI2jvqOqR12OqKYGIiBSQqI+wHgAOAp0Iksd3gG8C44BVwMkZ\niU5ERPJW1EdY5wI3A6vD5YPuvhIYY2aVwP8D/jkD8YmISJ6KegdyLLDJ3SuBnUCbmLK3APWDJSJS\nYKImkNVAu3D+feCHMWV9gR3pDEpERPJf1AQymyBRANwLXGVmH5nZ+8AtwMOZCE5ERPJX1M4UbzGz\nRuH8TDP7LtAfaAbc4e4vZDBGERHJQ7UmEDM7AhhJcBeyFMDd/wL8JbOhiYhIPqv1EZa77wfGEjSk\ni4iIANHbQBYCXTMZiIiI1C9RvwP5KTDdzPYAs4BNHPoVOuErviIiUiCiJpCF4Z+P1lCnqI6xiIhI\nPRI1gYwg7o5DREQKW00jEp4D/M3dd7v7I1mMSURE6oGaGtHnAp2zFYiIiNQvNSUQq6FMREQKXNZH\nJDSzvma2wsxWmdltCco7mtlrZrbEzN4wsxPC9aeZ2Ttmtjws++HhexcRkWyprRH9+2b2zSg7cvcp\ntdUxsyLgIeACYC0w38xmuvvfY6o9AExx9yfN7FyCvrcGA3uAIe7+oZkdByw0s1fdvSxKfCIikl61\nJZAxEffjQK0JBOgFrHL3jwHMbDpwCRCbQDoTjD0CQTvMiwDh+COE8+vNbDNBt/JKICIiOWDuid/O\nDQeK6gMsiLIjd/+i1oOZDQD6uvuwcHkwcLq7j4yp8zQwz93Hm1l/4I9Aa3ffGlOnF/AkcHL8B4xm\nNhwYDtC2bdse06dPjxJ+Qrt376a4uDjl7TNFcSVHcSVHcSWnIcbVp0+fhe7es9aK7p5wAiqBXtWV\npzIBA4DHY5YHAxPi6hwHPA8sAsYTPOoqiSlvD6wAzqjteD169PC6mDt3bp22zxTFlRzFlRzFlZyG\nGBewwCNc06N+SJgu64AOMcsnhOu+5O7rCbqKx8yKgcs9bOcws6OBPwOj3f3drEQsIiIJZfstrPlA\nJzM7ycyaAlcAM2MrmFnrqrFHgNuByeH6psALBA3sM7IYs4iIJFBtAnH3Ru7+13QezN3LCcYWeZVg\naNzn3H25mY01s35htVJghZmtBNoC48L1PyAYe32omS0Op9PSGZ+IiESX7UdYuPssgh59Y9eNiZmf\nARx2h+HuU4GpGQ9QREQiyfqHhCIi0jAogYiISEqUQEREJCVKICIikhIlEBERSYkSiIiIpEQJRERE\nUqIEIiIiKVECERGRlCiBiIhISpRAREQkJUogIiKSEiUQERFJiRKIiIikRAlERERSogQiIiIpUQIR\nEZGUKIGIiEhKlEBERCQlSiAiIpISJRAREUmJEoiIiKRECURERFKiBCIiIilRAhERkZQogYiISEqU\nQEREJCVKICIikhIlEBERSYkSiIiIpEQJREREUqIEIiIiKVECERGRlCiBiIhISpRAREQkJVlPIGbW\n18xWmNkqM7stQXlHM3vNzJaY2RtmdkJM2dVm9mE4XZ3dyEVEJFZWE4iZFQEPARcBnYErzaxzXLUH\ngCnu3gUYC9wbbnsscBdwOtALuMvMjslW7CIicqhs34H0Ala5+8fufgCYDlwSV6cz8Ho4Pzem/HvA\nHHff5u7bgTlA3yzELCIiCTTO8vGOB9bELK8luKOI9R7QHxgPXAa0MLNW1Wx7fPwBzGw4MDxc3G1m\nK+oQb2tgSx22zxTFlRzFlRzFlZyGGFfHKJWynUCiGAVMMLOhwFvAOqAi6sbu/hjwWDoCMbMF7t4z\nHftKJ8WVHMWVHMWVnEKOK9sJZB3QIWb5hHDdl9x9PcEdCGZWDFzu7mVmtg4ojdv2jUwGKyIi1ct2\nG8h8oJOZnWRmTYErgJmxFcystZlVxXU7MDmcfxW40MyOCRvPLwzXiYhIDmQ1gbh7OTCS4ML/PvCc\nuy83s7Fm1i+sVgqsMLOVQFtgXLjtNuBugiQ0HxgbrsuktDwKywDFlRzFlRzFlZyCjcvcPdPHEBGR\nBkhfoouISEqUQEREJCUFn0AidK1yhJk9G5bPM7Ov5klcQ83sczNbHE7DshTXZDPbbGbLqik3M/td\nGPcSM+ueJ3GVmtmOmPM1JktxdTCzuWb2dzNbbmY/TVAn6+csYlxZP2dmdqSZ/dXM3gvj+vcEdbL+\nm4wYV05+k+Gxi8xskZm9nKAsc+fL3Qt2AoqAj4CvAU0JPmLsHFdnBPBIOH8F8GyexDUUmJCDc3YO\n0B1YVk3594H/Agw4A5iXJ3GVAi/n4Hy1B7qH8y2AlQn+W2b9nEWMK+vnLDwHxeF8E2AecEZcnVz8\nJqPElZPfZHjsm4GnE/33yuT5KvQ7kChdq1wCPBnOzwDOMzPLg7hywt3fAmp6++0Sgr7M3N3fBUrM\nrH0exJUT7r7B3f8Wzu8iePswvgeFrJ+ziHFlXXgOdoeLTcIp/k2frP8mI8aVE2GHs/8MPF5NlYyd\nr0JPIFG6R/myjgevIe8AWuWPtvgvAAAFXUlEQVRBXACXh488ZphZhwTluRA19lzoHT6C+C8zOznb\nBw8fHXQj+NdrrJyesxrighycs/BxzGJgM0H/d9Weryz+JqPEBbn5Tf4WuBWorKY8Y+er0BNIffYS\n8FUPei2ewz/+hSGJ/Q3o6O5dgQeBF7N58LBXhT8CN7n7zmweuya1xJWTc+buFe5+GkFvE73M7JRs\nHLc2EeLK+m/SzP4F2OzuCzN9rEQKPYHU2rVKbB0zawy0BLbmOi533+ru+8PFx4EeGY4pqijnNOvc\nfWfVIwh3nwU0MbPW2Ti2mTUhuEhPc/fnE1TJyTmrLa5cnrPwmGUEPXLH97qdi99krXHl6Dd5FtDP\nzD4heNR9rplNjauTsfNV6Amk1q5VwuWqwasGAK972BqVy7jinpH3I3iGnQ9mAkPCN4vOAHa4+4Zc\nB2Vm7aqe+5pZL4L/9zN+0QmP+XvgfXf/j2qqZf2cRYkrF+fMzNqYWUk43wy4APggrlrWf5NR4srF\nb9Ldb3f3E9z9qwTXidfdfVBctYydr3zsjTdr3L3czKq6VikCJnvYtQqwwN1nEvzInjKzVQSNtFfk\nSVw/saD7l/IwrqGZjgvAzJ4heDuntZmtJRjkq0kY9yPALIK3ilYBe4Br8iSuAcCPzawc2AtckYV/\nCEDwL8TBwNLw+TnAHcCJMbHl4pxFiSsX56w98KQFg881Iuju6OVc/yYjxpWT32Qi2Tpf6spERERS\nUuiPsEREJEVKICIikhIlEBERSYkSiIiIpEQJREREUqIEIhLHzH5hZl7NFP+OfTbi8fC1bpG8UtDf\ngYjUYAeHfwENwbcaIoISiEh1ysOecUWkGnqEJZIkM/tq+FjpKjN7ysx2WTCY1V0J6p4bDuKzz8w2\nmdnDYQeGsXVamdmjZrYhrLfCzG6K21WRmd1jwYBFm83sITM7ImYfJWb2uJmtD/fxmZlNytApEAF0\nByJSrbDjuUOE3WFXuR94maDLj3OAu8xsi7s/FG5/MvAKQc+slxN0aPcrgoHC+oZ1mgFvAF8B/p2g\nf6Wvh1OsW4DXgUFAF+Be4FPgvrD8P4Azgf8LbAyPdU6qf3eRKNSViUgcM/sFQV9aiZwU/rmaYEyI\nC2O2m0TQp1UHd680s+kEPbJ+090rwjo/AJ4FznT3d8zsX4GJBKMDLiYBM3PgbXc/J2bdi0A7dz8j\nXF4GPOruD6b69xZJlu5ARBLbAZyfYP164Lhw/oW4sueBYQTdsX9GMLLkjKrkEfojQWd7ZwPvAOcC\ni6pLHjFmxy3/HegZs7wY+JmZVQD/7e4ra9mfSJ2pDUQksXJ3X5BgOhBTZ3PcNlXL7WP+3BRbIUwm\nW4Fjw1WtgChdt5fFLR8AjoxZHkkw4NMYYIWZfWhm2eilVgqYEohI6r5SzfKGmD8PqRN2B96Kf4zf\nvpV/JJyUuXuZu//E3dsBXQmGp51mZp3rum+R6iiBiKTusrjl/gRJY224PA+4LEwasXUaA/8TLr8G\ndDOzLukKyt2XAD8j+H1/M137FYmnNhCRxBqHowPGWxMzf7KZPUrQrnEO8CPgp+5eGZb/ElgEvGhm\nEwnaRn4NvOru74R1pgA3ALPDxvsVBA3133D326IGa2b/Q9Amswxw4DrgC+CvUfchkiwlEJHEWhI0\ncse7E6gac/pW4F8IEsg+4G5gQlXFcBTJi4B7CBrYdwLPhNtV1dlnZucSvN47Fjga+AR4OMl43yEY\nAe+rQAVB4rrI3dfWsI1Ineg1XpEkmdlXCV7jvdjdX85tNCK5ozYQERFJiRKIiIikRI+wREQkJboD\nERGRlCiBiIhISpRAREQkJUogIiKSEiUQERFJyf8HztvTbj1ebhQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'],c='k',lw=2)\n",
    "plt.grid(True)\n",
    "plt.xlabel(\"Epochs\",fontsize=15)\n",
    "plt.ylabel(\"Training accuracy\",fontsize=15)\n",
    "plt.ylim(0.9,1.0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "sa-0m42NGtr5",
    "outputId": "9863dfae-5bb5-4c2c-c8fb-6a75dc96c2ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training wall clock time: 93.696 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\nTraining wall clock time: {} seconds\\n\".format(round(t2-t1,3)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 67
    },
    "colab_type": "code",
    "id": "SQde_rHqGycu",
    "outputId": "77d87c33-7164-49e7-b6db-8967d6fcafec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 109us/sample - loss: 0.0503 - acc: 0.9846\n",
      "\n",
      "Test accuracy:  0.9846\n"
     ]
    }
   ],
   "source": [
    "test_loss = model.evaluate(test_images, test_labels)\n",
    "print(\"\\nTest accuracy: \",test_loss[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uRLfZ0jt-fQI"
   },
   "source": [
    "## How was the code organized anyway?\n",
    "\n",
    "Then, look at the code again, and see, step by step how the Convolutions were built:\n",
    "\n",
    "Step 1 is to gather the data. You'll notice that there's a bit of a change here in that the training data needed to be reshaped. That's because the first convolution expects a single tensor containing everything, so instead of 60,000 28x28x1 items in a list, we have a single 4D list that is 60,000x28x28x1, and the same for the test images. If you don't do this, you'll get an error when training as the Convolutions do not recognize the shape. \n",
    "\n",
    "```\n",
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SS_W_INc_kJQ"
   },
   "source": [
    "Next is to define your model. Now instead of the input layer at the top, you're going to add a Convolution. The parameters are:\n",
    "\n",
    "1. The number of convolutions you want to generate. Purely arbitrary, but good to start with something in the order of 32\n",
    "2. The size of the Convolution, in this case a 3x3 grid\n",
    "3. The activation function to use -- in this case we'll use relu, which you might recall is the equivalent of returning x when x>0, else returning 0\n",
    "4. In the first layer, the shape of the input data.\n",
    "\n",
    "You'll follow the Convolution with a MaxPooling layer which is then designed to compress the image, while maintaining the content of the features that were highlighted by the convlution. By specifying (2,2) for the MaxPooling, the effect is to quarter the size of the image. Without going into too much detail here, the idea is that it creates a 2x2 array of pixels, and picks the biggest one, thus turning 4 pixels into 1. It repeats this across the image, and in so doing halves the number of horizontal, and halves the number of vertical pixels, effectively reducing the image by 25%.\n",
    "\n",
    "You can call model.summary() to see the size and shape of the network, and you'll notice that after every MaxPooling layer, the image size is reduced in this way. \n",
    "\n",
    "\n",
    "```\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RMorM6daADjA"
   },
   "source": [
    "Add another convolution\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b1-x-kZF4_tC"
   },
   "source": [
    "Now flatten the output. After this you'll just have the same DNN structure as the non convolutional version\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Flatten(),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qPtqR23uASjX"
   },
   "source": [
    "The same 128 dense layers, and 10 output layers as in the pre-convolution example:\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C0GSsjUhAaSj"
   },
   "source": [
    "Now compile the model, call the fit method to do the training, and evaluate the loss and accuracy from the test set.\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
    "print(test_acc)\n",
    "```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8KVPZqgHo5Ux"
   },
   "source": [
    "## EXERCISES\n",
    "\n",
    "1. Try editing the convolutions. Change the 32s to either 16 or 64. What impact will this have on accuracy and/or training time.\n",
    "\n",
    "2. Remove the final Convolution. What impact will this have on accuracy or training time?\n",
    "\n",
    "3. How about adding more Convolutions? What impact do you think this will have? Experiment with it.\n",
    "\n",
    "4. Remove all Convolutions but the first. What impact do you think this will have? Experiment with it. \n",
    "\n",
    "5. In the previous lesson you implemented a callback to check on the loss function and to cancel training once it hit a certain amount. See if you can implement that here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TcRY3qW48geP"
   },
   "source": [
    "### Only a single convolutional layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 636
    },
    "colab_type": "code",
    "id": "ZpYRidBXpBPM",
    "outputId": "57cae1a0-f8d7-4290-d313-4c78029a4cf7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.14.0\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 0s 0us/step\n",
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_2 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 128)               692352    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 693,962\n",
      "Trainable params: 693,962\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 10s 161us/sample - loss: 0.1553 - acc: 0.9537\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 10s 162us/sample - loss: 0.0510 - acc: 0.9843\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 10s 160us/sample - loss: 0.0322 - acc: 0.9900\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 10s 162us/sample - loss: 0.0216 - acc: 0.9930\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 10s 161us/sample - loss: 0.0151 - acc: 0.9951\n",
      "\n",
      "Training wall clock time: 48.846 seconds\n",
      "\n",
      "10000/10000 [==============================] - 1s 87us/sample - loss: 0.0447 - acc: 0.9860\n",
      "\n",
      "Test accuracy:  0.986\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)\n",
    "\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.summary()\n",
    "\n",
    "t1=time()\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "t2=time()\n",
    "print(\"\\nTraining wall clock time: {} seconds\\n\".format(round(t2-t1,3)))\n",
    "\n",
    "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
    "\n",
    "print(\"\\nTest accuracy: \",test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rb0QY9d58n-w"
   },
   "source": [
    "### Three (3) convolutional layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 736
    },
    "colab_type": "code",
    "id": "DxIwYSq48RHN",
    "outputId": "5ab52576-b806-4d96-be6d-2da69faed0f7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.14.0\n",
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_3 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 11, 11, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 3, 3, 32)          18464     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 1, 1, 32)          0         \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 128)               4224      \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 42,794\n",
      "Trainable params: 42,794\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 13s 219us/sample - loss: 0.2384 - acc: 0.9265\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 13s 219us/sample - loss: 0.0869 - acc: 0.9732\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 13s 216us/sample - loss: 0.0606 - acc: 0.9809\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 13s 217us/sample - loss: 0.0478 - acc: 0.9849\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 13s 217us/sample - loss: 0.0388 - acc: 0.9878\n",
      "\n",
      "Training wall clock time: 65.851 seconds\n",
      "\n",
      "10000/10000 [==============================] - 1s 102us/sample - loss: 0.0503 - acc: 0.9846\n",
      "\n",
      "Test accuracy:  0.9846\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)\n",
    "\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.summary()\n",
    "\n",
    "t1=time()\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "t2=time()\n",
    "print(\"\\nTraining wall clock time: {} seconds\\n\".format(round(t2-t1,3)))\n",
    "\n",
    "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
    "\n",
    "print(\"\\nTest accuracy: \",test_acc)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Fashion MNIST using CNN.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}