LeNet-Lab-Traffic-Sign-Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LeNet Traffic Sign Classifier\n",
    "![LeNet Architecture](lenet.png)\n",
    "Source: Yan LeCun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "Load the MNIST data, which comes pre-loaded with TensorFlow.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Image Shape: (34799, 32, 32, 3)\n",
      "\n",
      "Training Set:   34799 samples\n",
      "Validation Set: 4410 samples\n",
      "Test Set:       12630 samples\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "# Load pickled data\n",
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "def rgb2gray(rgb):\n",
    "    return rgb\n",
    "\n",
    "    r, g, b = rgb[:,:,:,0], rgb[:,:,:,1], rgb[:,:,:,2]\n",
    "    gray = rgb;\n",
    "    #gray = 0.2989 * r + 0.5870 * g + 0.1140 * b\n",
    "    gray = np.delete(gray,1, 3)\n",
    "    gray = np.delete(gray,1, 3)\n",
    "    gray[:,:,:,0] = 0.2989 * r + 0.5870 * g + 0.1140 * b\n",
    "    return gray\n",
    "\n",
    "# TODO: Fill this in based on where you saved the training and testing data\n",
    "\n",
    "training_file = \"/home/mark/code/udacity/CarND-Traffic-Sign-Classifier-Project/train.p\"\n",
    "validation_file= \"/home/mark/code/udacity/CarND-Traffic-Sign-Classifier-Project/valid.p\"\n",
    "testing_file = \"/home/mark/code/udacity/CarND-Traffic-Sign-Classifier-Project/test.p\"\n",
    "\n",
    "with open(training_file, mode='rb') as f:\n",
    "    train = pickle.load(f)\n",
    "with open(validation_file, mode='rb') as f:\n",
    "    valid = pickle.load(f)\n",
    "with open(testing_file, mode='rb') as f:\n",
    "    test = pickle.load(f)\n",
    "    \n",
    "X_train, y_train = (rgb2gray(train['features']) - 0) / 1, train['labels']\n",
    "X_validation, y_validation = (rgb2gray(valid['features']) - 0) / 1, valid['labels']\n",
    "X_test, y_test = (rgb2gray(test['features']) - 0) / 1, test['labels']\n",
    "\n",
    "assert(len(X_train) == len(y_train))\n",
    "assert(len(X_validation) == len(y_validation))\n",
    "assert(len(X_test) == len(y_test))\n",
    "\n",
    "print()\n",
    "print(\"Image Shape: {}\".format(X_train.shape))\n",
    "print()\n",
    "print(\"Training Set:   {} samples\".format(len(X_train)))\n",
    "print(\"Validation Set: {} samples\".format(len(X_validation)))\n",
    "print(\"Test Set:       {} samples\".format(len(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data that TensorFlow pre-loads comes as 28x28x1 images.\n",
    "\n",
    "However, the LeNet architecture only accepts 32x32xC images, where C is the number of color channels.\n",
    "\n",
    "In order to reformat the MNIST data into a shape that LeNet will accept, we pad the data with two rows of zeros on the top and bottom, and two columns of zeros on the left and right (28+2+2 = 32).\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Data\n",
    "\n",
    "View a sample from the dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFwAAABZCAYAAACzIkPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADL1JREFUeJztnE2IJVcVx3/nVtV7r3s6kkmUOInBD3AhRokgunAjUUHc\nRDdiFqIgxI2i4MLgymUW6laIGFAQRDRgFgEJogs3khiCmgQ1iOaDGDWRJNPT3e9V3ePinnNvvdc9\nM+91T2qaTB2Yqer6uPe+c//3fNcVVWWk4Shc7QFcazQyfGAaGT4wjQwfmEaGD0wjwwemkeED04kY\nLiKfFJG/iMjTInLPlRrUG5nkuI6PiFTAX4FPAM8BjwB3qeqTV254bzyqT/Duh4CnVfXvACLyU+BO\n4KIMP3v2er3llnOgoKSJFiQdJS22EAISfOH5PXBYiJ0oMR2jomrn+aj5BeXigBL8md41sT5Dlcfl\nYxNZftP/fuaZZ3nppZeENegkDL8FeLb393PAh1cfEpG7gbsBbr75rTzw8x/Tdi0xpp8Z7IdNmhkA\nW1tnmM7OpHtV7W2UCbIV2bVzABbzfRbzvXwO0M4PICbmt3aMdkSEEIxhNkb1Z1RpplMA6tmbAJjO\nrqNp7FpV2bh8MgRQ7rjj42uwK9FJGL4Wqep9wH0At932Ho0xIlqUhxjj/UdrhKOlnCwdQj0BoJEK\npKARIARo52lCQmsT1MXemCpryxtL71VS0Uy3AagnCQB1PaW2vkJwpNvkixD1UmvoMJ2E4c8Dt/b+\nfptduySFIEQFMgN87RdxkPWKMTCq5qWe3/KlXwcmdh58TkQQ+2kxngegCmXFS37Qp91W2WTGZLLM\n8KapCdWKbZHHFdNK3YDjJ7FSHgHeLSLvFJEJ8DngwRO0d03QsRGuqq2IfAX4FQki96vqE2u8iMY2\ny1QXAy5HVOMhkSJZtS5fzWcmUqpmyy5UuV1lkY6xBaBTzavDdURdJxndNDMaR7jJ7RAEKSp7qees\n3NdSl4lOJMNV9SHgoZO0ca3R6640+yRAJUIb9ZC51v9LM9pdvmvPfDz8RpbnhtimqvJ91Q6AxdwQ\n3u73FG8DwGSaVkbTzGiapCAlK8iQ9cfRP2oDeDO69oPToAgHUCTZ1dkSMeuEPqqXLZdkftl5Ng97\nyJLSdjoRKpPBi/mBPW5yvqqKGegy3HyAZjJFqmUTc2ms3l3PY5LS61o0MMPFlmhFkJivJdLen4ft\nrFXvM4sMINq5TwoxErukLOvGFaOJik4INhnN9Iw9M7U+6iI+3AxVLilSNhMoo0gZnAYXKZhIuZiy\nUYpCdcQGCS5dsqu+iEkZHrRz9hfJpd9fLKyRmM3O4OaniYVJtcWsLl4kQHRxY2GGNEqOPPdRgiN/\nVJqnmgZHuKAEgWAIPzoS4RG7Ire7Lpl1+21C8YEhdoHQWfxjcib9nFCF3GqwpdHN03vzvQMWi9TW\nXvsaAI11szWZMrPgVROSyRhCfUgt5sgAG3n1wOAMT6HUqLGEUvM9t0J6IVKzFA4WLQcLiw6SRElr\ny1+rhqpKCrFw4jAbqpnb1xPiQYouamtRxi613V2YE21iZxOzzWtJlg0l9FIsKkY7/LTTVTALBdWe\nN2l3NBbbWwztLkbm7YKFIbo1Bdd5gqBuwKJ5/Ta93dXoYjVp/HH0wFbZ3COVLXv7u6ndbKUKQWx1\nZDG4eQzFaUT4wDS8WSgBFTFJXMjRGWNHa5E9TQe6EGjV0BU862JDF8nxEs3xluIdFq+wOFZSm/mn\nSUGqmZhd14GNbL7YK+8b3CfBEhHe6DHSwSPCB6ZhES6ChAaRSKg9I+O5RrM6EBYmu6PhYSGCuono\nAji4PC3I1p7FsOqO579Vc7uYdWPApWvbrDcWbUJ6CC2ham2IFo30lF7Khl8kJXg0Da40o1RcmCtq\ng14sEsObYJ7hBKQ1WeKTUNeF4f3IUW61MNPvZfHik6Elp1m0nSlgLSamBDMRzQxtu5bKJqGOJQ1Y\n2tlMrowiZWAaXGlGEXbn++zuJ6XUWfxj1lR2f8FsZycNziJ8HZqVbEm29UTGCsii6uGVkJMaXb7m\nlQIEE2FBc9mG6WhibImulJdWST+Lv95vhxHhg9PgCK8Fzu5ssz01R2aREgRexjCb1mRxbais+im2\nbjn6l5LShkCXsW3LwlaOJyDmduy6Ba0VEXXm0tfWdx0qppZ2m/kgAqgr8c6R3jMxN/39Gz5/IhKU\nisj2tGHWpKF2nVkrvuRF6ExpzmPy+vZffYU9u3awZ6LICn20bVGzKLQtjNGViqtcGqeREl61CTXL\nZ2fnTUy3r0vPWRBrOQFlRUV2rI7hao4iZWAaPqepEaEDdTUY/YYdJIdl53sXANjdPc8r518BoLX6\nQbXlLbFXtZKRWBIQ2V7vhYMd7e4xhkkSI4sgRBMp9ZZn8quc+3Ql2U/2bZqEGBE+MA2O8CBWx+R6\nR11hpaGo1LSGg+kkKdbd87tsmVd4IBZn8ax6JThuakNnqGoqO8dj2Z5Mbpossyvzdre2UwIjItSG\nVkdzBIIlI6pQ4jfp4I7P+s7PZREuIreKyG9E5EkReUJEvmbXbxCRh0Xkb3Y8u3av1zCtg/AW+Iaq\nPiYi1wF/EJGHgS8Cv1bVe+1zk3uAb16ypST0ljCh6ojyupFpjnXUIT21c/ZGJvH69Jyhs851gQXN\npay4znLXV4Iu+Uvu7tux81LpBaGzOha71khFY+17m55sLtGc9emyDFfVF4AX7Pw1EXmKVIx/J/BR\ne+xHwG+5HMMVK8QPuWtPV3WdD79DapsEW/Lb2zcysx8t9fKy7k/eoqc8V9XY0bUlXuZsoeF5sfed\nybWEDIZyDIfaWJc2Upoi8g7gA8DvgZtsMgD+Bdx0kXfuFpFHReTRl1/+30aDeyPS2kpTRHaAXwBf\nV9VX+4hRVRU5InOb7uUvIN7/vveqqrJoO1pD9P6+hUHNVJttBzwJ4EX1ogFRw0Z0D7Afx1gpJxZ6\nJcbLqbwlMg/Vj0G1ODP+wUAIh0zL46TWnNZCuIg0JGb/RFUfsMsvisg5u38O+Pfxh3Ht0GURLgnK\nPwSeUtXv9W49CHwBuNeOv7xsbyJUVcP8wgG7F5KLvreXlFQ2w8IOW7XV/Pk17fDYdVhBmfb/cHON\nPgiX4bhUmOnnna8ypcIrtMzJ6aXYVpHdN2/XpXVEykeAzwN/EpHH7dq3SIz+mYh8Cfgn8NnNur42\naR0r5XdcXGp9bJPOhITw6WTG1IrguWFZ1iJVNu8qKVaEp91cCqpZKxJCNvl0Ge/LffeQXQr9zcU3\n/VGj1G52uhsvgbKoPB5eJPGmrv2gnqaixNjSNCHHOjqLqbgSrpuKSbZ3Qz56meWBlSF3/uPraTbX\nlqpRXLzocqwmZfSXI4mVvTapapo8yc5wKZ8jriQ1NH8yeAU9zZGuLA3+jY9UgsSOaB5ddNPM5j52\nkTC1Kin/9iY0EOfWiD3fJVS1nRAr8wRrMxUDFHW6IrLaNkcavZQ5UIrv3TMlfwEReqk0WW5TddPS\nwhHhQ9PAdSmBupnSdR21C85o31EaylKMxBGXnpkGqMVi0rEfqYP9ruNg3wrxJx4RDHReiJ8L8m0I\nqtReZ2hlGK0hviPmZLUrbJWeW9WrNwTLHm0I8WFFighNM6Gu6t639ctly6qlLtyDUiEIWO3IxANH\nnu8Mkamt09bETtzrsqfoGxLkYFeoV4JPKV4C0MWWiYmlYO0HIckaoPNa8179Sy2bCYlRpAxMV+Eb\nH0Akb4GhvS06IEXkVgvyoawEFxEuFmpqginNvLqliCPfxqNfz5KNuLxBgo9hku/mT9Pzf/QUaUnX\nRY0bOZsjwgemq/QVW68AP4dBijKUnkkG7mCUSCAUpKuGXhHn4TLi3NYRMCwF/KX8IWTTrzg5qzsB\n9T8VTVuFrE8jwgemwa2UhDgte8MsJWT9mcMVsgVcy/V8/T2vekGP3qcs7tz0x+Ay2MxP35XIdviB\nXlanF4M5FA7PcZRTGkuBXrBnRZQ4pWW+/Fl42jRsxWP0J0JgaY3bsVxxk849T8mTVsSUx2Iihzf/\nCT12lhhKf3xjLOUU07H3LTxWZyL/AXaB/w7W6fHpzaw/zrer6lvWeXBQhgOIyKOq+sFBOz0GvV7j\nHEXKwDQyfGC6Ggy/7yr0eRx6XcY5uAy/1mkUKQPTYAw/zXuNX6JC+Nsi8ryIPG7/PnXivoYQKad9\nr3GrHDvXrxAGPk2qtTmvqt+5Un0NhfC817iqzgHfa/xUkKq+oKqP2flrgFcIX3EaiuFH7TX+uvyg\nk9JKhTDAV0XkjyJy/5X46GBUmj1arRAGvg+8C7idVCP/3ZP2MRTDj7XX+JB0VIWwqr6oqp2m+O8P\nSKLxRDQUw0/1XuMXqxD2cmyjzwB/Pmlfg8TDj73X+HB0sQrhu0TkdlLA+x/Al0/a0ehpDkyj0hyY\nRoYPTCPDB6aR4QPTyPCBaWT4wDQyfGAaGT4w/R8vpCA64VEdoAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1fdd8eb9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "index = random.randint(0, len(X_train))\n",
    "image = X_train[index].squeeze()\n",
    "\n",
    "plt.figure(figsize=(1,1))\n",
    "plt.imshow(image, cmap=\"gray\")\n",
    "print(y_train[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess Data\n",
    "\n",
    "Shuffle the training data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "X_train, y_train = shuffle(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup TensorFlow\n",
    "The `EPOCH` and `BATCH_SIZE` values affect the training speed and model accuracy.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "EPOCHS = 200\n",
    "BATCH_SIZE = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SOLUTION: Implement LeNet-5\n",
    "Implement the [LeNet-5](http://yann.lecun.com/exdb/lenet/) neural network architecture.\n",
    "\n",
    "This is the only cell you need to edit.\n",
    "### Input\n",
    "The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.\n",
    "\n",
    "### Architecture\n",
    "**Layer 1: Convolutional.** The output shape should be 28x28x6.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 14x14x6.\n",
    "\n",
    "**Layer 2: Convolutional.** The output shape should be 10x10x16.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 5x5x16.\n",
    "\n",
    "**Flatten.** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using `tf.contrib.layers.flatten`, which is already imported for you.\n",
    "\n",
    "**Layer 3: Fully Connected.** This should have 120 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 4: Fully Connected.** This should have 84 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 5: Fully Connected (Logits).** This should have 10 outputs.\n",
    "\n",
    "### Output\n",
    "Return the result of the 2nd fully connected layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.layers import flatten\n",
    "input_size = 1\n",
    "\n",
    "def LeNet(x):    \n",
    "    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer\n",
    "    mu = 0\n",
    "    sigma = 0.1\n",
    "    \n",
    "    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
    "    c1 = 6*2\n",
    "    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, input_size, c1), mean = mu, stddev = sigma))\n",
    "    conv1_b = tf.Variable(tf.zeros(c1))\n",
    "    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "\n",
    "    # SOLUTION: Activation.\n",
    "    conv1 = tf.nn.relu(conv1)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
    "    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.\n",
    "    c2 = 16*4\n",
    "    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, c1, c2), mean = mu, stddev = sigma))\n",
    "    conv2_b = tf.Variable(tf.zeros(c2))\n",
    "    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    conv2 = tf.nn.relu(conv2)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
    "    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.\n",
    "    fc0   = flatten(conv2)\n",
    "    fc1_out = 5*5*c2\n",
    "    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
    "    fc1_W = tf.Variable(tf.truncated_normal(shape=(fc1_out, 120), mean = mu, stddev = sigma))\n",
    "    fc1_b = tf.Variable(tf.zeros(120))\n",
    "    fc1   = tf.matmul(fc0, fc1_W) + fc1_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc1    = tf.nn.relu(fc1)\n",
    "\n",
    "    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
    "    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))\n",
    "    fc2_b  = tf.Variable(tf.zeros(84))\n",
    "    fc2    = tf.matmul(fc1, fc2_W) + fc2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc2    = tf.nn.relu(fc2)\n",
    "    fc2    = tf.layers.dropout(fc2, 0.4)\n",
    "\n",
    "    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 43.\n",
    "    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))\n",
    "    fc3_b  = tf.Variable(tf.zeros(43))\n",
    "    logits = tf.matmul(fc2, fc3_W) + fc3_b\n",
    "    \n",
    "    return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features and Labels\n",
    "Train LeNet to classify [MNIST](http://yann.lecun.com/exdb/mnist/) data.\n",
    "\n",
    "`x` is a placeholder for a batch of input images.\n",
    "`y` is a placeholder for a batch of output labels.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (None, 32, 32, input_size))\n",
    "y = tf.placeholder(tf.int32, (None))\n",
    "one_hot_y = tf.one_hot(y, 43)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Pipeline\n",
    "Create a training pipeline that uses the model to classify MNIST data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rate = 0.001 # 0.001\n",
    "\n",
    "logits = LeNet(x)\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)\n",
    "loss_operation = tf.reduce_mean(cross_entropy)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = rate)\n",
    "training_operation = optimizer.minimize(loss_operation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "Evaluate how well the loss and accuracy of the model for a given dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))\n",
    "accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "def evaluate(X_data, y_data):\n",
    "    num_examples = len(X_data)\n",
    "    total_accuracy = 0\n",
    "    sess = tf.get_default_session()\n",
    "    for offset in range(0, num_examples, BATCH_SIZE):\n",
    "        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]\n",
    "        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "        total_accuracy += (accuracy * len(batch_x))\n",
    "    return total_accuracy / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "Run the training data through the training pipeline to train the model.\n",
    "\n",
    "Before each epoch, shuffle the training set.\n",
    "\n",
    "After each epoch, measure the loss and accuracy of the validation set.\n",
    "\n",
    "Save the model after training.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "\n",
      "EPOCH 1 ...\n",
      "Train Accuracy = 0.586\n",
      "Validation Accuracy = 0.457\n",
      "\n",
      "EPOCH 2 ...\n",
      "Train Accuracy = 0.820\n",
      "Validation Accuracy = 0.653\n",
      "\n",
      "EPOCH 3 ...\n",
      "Train Accuracy = 0.909\n",
      "Validation Accuracy = 0.725\n",
      "\n",
      "EPOCH 4 ...\n",
      "Train Accuracy = 0.929\n",
      "Validation Accuracy = 0.724\n",
      "\n",
      "EPOCH 5 ...\n",
      "Train Accuracy = 0.950\n",
      "Validation Accuracy = 0.771\n",
      "\n",
      "EPOCH 6 ...\n",
      "Train Accuracy = 0.972\n",
      "Validation Accuracy = 0.785\n",
      "\n",
      "EPOCH 7 ...\n",
      "Train Accuracy = 0.974\n",
      "Validation Accuracy = 0.787\n",
      "\n",
      "EPOCH 8 ...\n",
      "Train Accuracy = 0.969\n",
      "Validation Accuracy = 0.784\n",
      "\n",
      "EPOCH 9 ...\n",
      "Train Accuracy = 0.991\n",
      "Validation Accuracy = 0.807\n",
      "\n",
      "EPOCH 10 ...\n",
      "Train Accuracy = 0.986\n",
      "Validation Accuracy = 0.812\n",
      "\n",
      "EPOCH 11 ...\n",
      "Train Accuracy = 0.991\n",
      "Validation Accuracy = 0.810\n",
      "\n",
      "EPOCH 12 ...\n",
      "Train Accuracy = 0.989\n",
      "Validation Accuracy = 0.781\n",
      "\n",
      "EPOCH 13 ...\n",
      "Train Accuracy = 0.990\n",
      "Validation Accuracy = 0.811\n",
      "\n",
      "EPOCH 14 ...\n",
      "Train Accuracy = 0.991\n",
      "Validation Accuracy = 0.807\n",
      "\n",
      "EPOCH 15 ...\n",
      "Train Accuracy = 0.992\n",
      "Validation Accuracy = 0.806\n",
      "\n",
      "EPOCH 16 ...\n",
      "Train Accuracy = 0.991\n",
      "Validation Accuracy = 0.814\n",
      "\n",
      "EPOCH 17 ...\n",
      "Train Accuracy = 0.996\n",
      "Validation Accuracy = 0.822\n",
      "\n",
      "EPOCH 18 ...\n",
      "Train Accuracy = 0.996\n",
      "Validation Accuracy = 0.828\n",
      "\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-130-2ad68d8d5273>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     18\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Validation Accuracy = {:.3f}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalidation_accuracy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m         \u001b[0msaver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'./lenet_traffic_sign_classifier'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m     \u001b[0msaver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'./lenet_traffic_sign_classifier'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36msave\u001b[0;34m(self, sess, save_path, global_step, latest_filename, meta_graph_suffix, write_meta_graph, write_state)\u001b[0m\n\u001b[1;32m   1384\u001b[0m           checkpoint_file, meta_graph_suffix=meta_graph_suffix)\n\u001b[1;32m   1385\u001b[0m       \u001b[0;32mwith\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgraph\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_default\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1386\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexport_meta_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmeta_graph_filename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1387\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1388\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_is_empty\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36mexport_meta_graph\u001b[0;34m(self, filename, collection_list, as_text, export_scope, clear_devices)\u001b[0m\n\u001b[1;32m   1417\u001b[0m         \u001b[0mas_text\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1418\u001b[0m         \u001b[0mexport_scope\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexport_scope\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1419\u001b[0;31m         clear_devices=clear_devices)\n\u001b[0m\u001b[1;32m   1420\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1421\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mrestore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msave_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/tensorflow/python/training/saver.py\u001b[0m in \u001b[0;36mexport_meta_graph\u001b[0;34m(filename, meta_info_def, graph_def, saver_def, collection_list, as_text, graph, export_scope, clear_devices, **kwargs)\u001b[0m\n\u001b[1;32m   1639\u001b[0m       \u001b[0mexport_scope\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexport_scope\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1640\u001b[0m       \u001b[0mclear_devices\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mclear_devices\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1641\u001b[0;31m       **kwargs)\n\u001b[0m\u001b[1;32m   1642\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mmeta_graph_def\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1643\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py\u001b[0m in \u001b[0;36mexport_scoped_meta_graph\u001b[0;34m(filename, graph_def, graph, export_scope, as_text, unbound_inputs_col_name, clear_devices, **kwargs)\u001b[0m\n\u001b[1;32m    645\u001b[0m         \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdirname\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    646\u001b[0m         \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbasename\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 647\u001b[0;31m         as_text=as_text)\n\u001b[0m\u001b[1;32m    648\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    649\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mscoped_meta_graph_def\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar_list\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/tensorflow/python/framework/graph_io.py\u001b[0m in \u001b[0;36mwrite_graph\u001b[0;34m(graph_or_graph_def, logdir, name, as_text)\u001b[0m\n\u001b[1;32m     67\u001b[0m     \u001b[0mfile_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0matomic_write_string_to_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgraph_def\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m   \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m     \u001b[0mfile_io\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0matomic_write_string_to_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgraph_def\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializeToString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m   \u001b[0;32mreturn\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mSerializeToString\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1034\u001b[0m           'Message %s is missing required fields: %s' % (\n\u001b[1;32m   1035\u001b[0m           self.DESCRIPTOR.full_name, ','.join(self.FindInitializationErrors())))\n\u001b[0;32m-> 1036\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializePartialToString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1037\u001b[0m   \u001b[0mcls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializeToString\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSerializeToString\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mSerializePartialToString\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1043\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mSerializePartialToString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1044\u001b[0m     \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBytesIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1045\u001b[0;31m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_InternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1046\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetvalue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1047\u001b[0m   \u001b[0mcls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializePartialToString\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSerializePartialToString\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mInternalSerialize\u001b[0;34m(self, write_bytes)\u001b[0m\n\u001b[1;32m   1049\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mInternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1050\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mListFields\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1051\u001b[0;31m       \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_encoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtag_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue_bytes\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_unknown_fields\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m       \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtag_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/encoder.py\u001b[0m in \u001b[0;36mEncodeField\u001b[0;34m(write, value)\u001b[0m\n\u001b[1;32m    764\u001b[0m       \u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    765\u001b[0m       \u001b[0mlocal_EncodeVarint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mByteSize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 766\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_InternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    767\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mEncodeField\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    768\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mInternalSerialize\u001b[0;34m(self, write_bytes)\u001b[0m\n\u001b[1;32m   1049\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mInternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1050\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mListFields\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1051\u001b[0;31m       \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_encoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtag_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue_bytes\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_unknown_fields\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m       \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtag_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/encoder.py\u001b[0m in \u001b[0;36mEncodeRepeatedField\u001b[0;34m(write, value)\u001b[0m\n\u001b[1;32m    758\u001b[0m         \u001b[0mwrite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    759\u001b[0m         \u001b[0mlocal_EncodeVarint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melement\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mByteSize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 760\u001b[0;31m         \u001b[0melement\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_InternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    761\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mEncodeRepeatedField\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    762\u001b[0m   \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mInternalSerialize\u001b[0;34m(self, write_bytes)\u001b[0m\n\u001b[1;32m   1049\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mInternalSerialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1050\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mListFields\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1051\u001b[0;31m       \u001b[0mfield_descriptor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_encoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtag_bytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue_bytes\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_unknown_fields\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m       \u001b[0mwrite_bytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtag_bytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/encoder.py\u001b[0m in \u001b[0;36mEncodeField\u001b[0;34m(write, value)\u001b[0m\n\u001b[1;32m    821\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mEncodeField\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    822\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 823\u001b[0;31m       \u001b[0mentry_msg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmessage_type\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_concrete_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    824\u001b[0m       \u001b[0mencode_message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mentry_msg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    825\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36minit\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m    510\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fields\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mfield\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    511\u001b[0m       \u001b[0;32melif\u001b[0m \u001b[0mfield\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcpp_type\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_FieldDescriptor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mCPPTYPE_MESSAGE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 512\u001b[0;31m         \u001b[0mcopy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfield\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_default_constructor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    513\u001b[0m         \u001b[0mnew_val\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfield_value\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    514\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfield_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36mMakeSubMessageDefault\u001b[0;34m(message)\u001b[0m\n\u001b[1;32m    426\u001b[0m           \u001b[0m_OneofListener\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfield\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    427\u001b[0m           \u001b[0;32mif\u001b[0m \u001b[0mfield\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontaining_oneof\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 428\u001b[0;31m           else message._listener_for_children)\n\u001b[0m\u001b[1;32m    429\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    430\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mMakeSubMessageDefault\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/mark/anaconda3/envs/IntroToTensorFlow/lib/python3.6/site-packages/google/protobuf/internal/python_message.py\u001b[0m in \u001b[0;36m_SetListener\u001b[0;34m(self, listener)\u001b[0m\n\u001b[1;32m   1290\u001b[0m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_listener\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmessage_listener_mod\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNullMessageListener\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1291\u001b[0m   \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1292\u001b[0;31m     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_listener\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlistener\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1293\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1294\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    num_examples = len(X_train)\n",
    "    \n",
    "    print(\"Training...\")\n",
    "    print()\n",
    "    for i in range(EPOCHS):\n",
    "        X_train, y_train = shuffle(X_train, y_train)\n",
    "        for offset in range(0, num_examples, BATCH_SIZE):\n",
    "            end = offset + BATCH_SIZE\n",
    "            batch_x, batch_y = X_train[offset:end], y_train[offset:end]\n",
    "            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "            \n",
    "        validation_accuracy = evaluate(X_validation, y_validation)\n",
    "        train_accuracy = evaluate(X_train, y_train)\n",
    "        print(\"EPOCH {} ...\".format(i+1))\n",
    "        print(\"Train Accuracy = {:.3f}\".format(train_accuracy))\n",
    "        print(\"Validation Accuracy = {:.3f}\".format(validation_accuracy))\n",
    "        print()\n",
    "        saver.save(sess, './lenet_traffic_sign_classifier')\n",
    "        \n",
    "    saver.save(sess, './lenet_traffic_sign_classifier')\n",
    "    print(\"Model saved\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model\n",
    "Once you are completely satisfied with your model, evaluate the performance of the model on the test set.\n",
    "\n",
    "Be sure to only do this once!\n",
    "\n",
    "If you were to measure the performance of your trained model on the test set, then improve your model, and then measure the performance of your model on the test set again, that would invalidate your test results. You wouldn't get a true measure of how well your model would perform against real data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "\n",
    "    test_accuracy = evaluate(X_test, y_test)\n",
    "    print(\"Test Accuracy = {:.3f}\".format(test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}