MLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Multilayer Perceptron.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/telecombcn-dl/2018-dlai-team10/blob/master/Multilayer_Perceptron.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "XHZhbuFaJTSq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Multilayer Perceptron\n",
        "The problem we are trying to solve here is to classify grayscale images of handwritten objects (28 pixels by 28 pixels), into 10 categories (apple, banana, fork...). The dataset we will use is extracted from the Kaggle competition: **Quick Draw! Doodle Recognition Challenge**.\n",
        "\n",
        "In this notebook, we will approach this task by implementing a Multilayer Perceptron. For this project we have also implemented other two approaches (Convolutional Neural Network and Long-Short Term Memory Network), that also have a corresponding self-contained notebooks.\n",
        "\n",
        "*For more details about out project please visit: https://telecombcn-dl.github.io/2018-dlai-team10/*"
      ]
    },
    {
      "metadata": {
        "id": "3OeVrUACIZ4x",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 1. Notebook Setting\n",
        "\n",
        "In this section the Pytorch and relevant Python libraries (Numpy, Matplotlib...) have been imported. Additionally, the notebook environment is set to train on the GPU for a faster results."
      ]
    },
    {
      "metadata": {
        "id": "i0hJACKxoOa3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from os.path import exists\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n",
        "accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n",
        "\n",
        "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision\n",
        "\n",
        "\n",
        "import os\n",
        "import sys\n",
        "import time\n",
        "import torch\n",
        "import requests\n",
        "import numpy as np\n",
        "from tqdm import tqdm\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from google.colab import files\n",
        "import matplotlib.pyplot as plt\n",
        "import torch.nn.functional as F\n",
        "import torchvision.datasets as datasets\n",
        "\n",
        "\n",
        "#Check if GPU is available\n",
        "CUDA = torch.cuda.is_available()\n",
        "\n",
        "# Constants\n",
        "IMAGE_WIDTH = 28\n",
        "IMAGE_HEIGHT = 28\n",
        "N_CLASSES = 10\n",
        "BATCH_SIZE = 30\n",
        "LEARNING_RATE = 0.00001\n",
        "WEIGHT_DECAY = 0.001\n",
        "EPOCHS = 50\n",
        "TRAINING_EX = 6e4\n",
        "PLOT_EVERY = 200\n",
        "\n",
        "TRAIN_FOLDER = r\"data/train\"\n",
        "VALID_FOLDER = r\"data/validation\"\n",
        "TEST_FOLDER = r\"data/test\"\n",
        "RESULT_FOLDER = r\"results\"\n",
        "MODEL_FOLDER = r\"saved_model\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4_QfsPQzWR5b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 2. Dataset Preparation\n",
        "In this section we will download a part of the original dataset, we will reduce the number of samples, distribute them in training, validation and test, reshape them into images and organize them in a structured way.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "TAjk3JIbLgcc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 2.1 Dataset Download\n",
        "The dataset is downloaded from the Google APIs and it comes in the form of a set of Numpy arrays. The Quick! Draw challenge dataset actually contains more than 300 classes, however we will only use 10 of them for our project, for a simplification purpose. We have manually selected the classes we will work with in order to have some interesting inter-class variability (wheeel and pizza are very similar while apple is very different...).\n",
        "\n",
        "You can access to the dataset clicking in this url: https://console.cloud.google.com/storage/browser/quickdraw_dataset/full/numpy_bitmap"
      ]
    },
    {
      "metadata": {
        "id": "q_Of8TNeMF7L",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "urls = [\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/key.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/banana.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/ladder.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/tennis%20racquet.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/pizza.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/stop%20sign.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/wheel.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/fork.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/book.npy',\n",
        "      'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/apple.npy',\n",
        "  ]\n",
        "\n",
        "class_name = ['apple', 'banana', 'book', 'fork', 'key', 'ladder', 'pizza', 'stop_sign', 'tennis_racquet', 'wheel']\n",
        "\n",
        "def createDir(path):\n",
        "  if not os.path.exists(path):\n",
        "      os.makedirs(path)\n",
        "\n",
        "def gen_bar_updater(pbar):\n",
        "  def bar_update(count, block_size, total_size):\n",
        "      if pbar.total is None and total_size:\n",
        "          pbar.total = total_size\n",
        "      progress_bytes = count * block_size\n",
        "      pbar.update(progress_bytes - pbar.n)\n",
        "  return bar_update   \n",
        "\n",
        "\n",
        "def download_url(url, root, filename):\n",
        "    from six.moves import urllib\n",
        "    root = os.path.expanduser(root)\n",
        "    fpath = os.path.join(root, filename + \".npy\")\n",
        "\n",
        "    createDir(root)\n",
        "    \n",
        "    #Create model folder \n",
        "    createDir(MODEL_FOLDER)\n",
        "\n",
        "    # Download files\n",
        "    if !os.path.isfile(fpath):\n",
        "        try:\n",
        "            print('Downloading ' + url + ' to ' + fpath)\n",
        "            urllib.request.urlretrieve(\n",
        "                url, fpath,\n",
        "                reporthook = gen_bar_updater(tqdm(unit='B', unit_scale=True))\n",
        "            )\n",
        "        except OSError:\n",
        "            if url[:5] == 'https':\n",
        "                url = url.replace('https:', 'http:')\n",
        "                print('Failed download. Trying https -> http instead.'\n",
        "                      ' Downloading ' + url + ' to ' + fpath)\n",
        "                urllib.request.urlretrieve(\n",
        "                    url, fpath,\n",
        "                    reporthook = gen_bar_updater(tqdm(unit='B', unit_scale=True))\n",
        "                )\n",
        "\n",
        "                \n",
        "for i in range(0, len(urls)):\n",
        "  download_url(urls[i], \"data\", class_name[i])\n",
        "\n",
        "print(\"The dataset is successfully  download\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MhDG2Kr2MV7j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 2.2 Dataset Reduction, Reshaping and Reorganization\n",
        "As we are implementing a MLP (we are willing to exploit the local connectivity of the data), we want to have the data as images. Furthermore, we have decided to work with a reduced dataset, so the number of samples per class will be max_length. We also split the data into training, validation and test by the percentages defined by percen and place each sample in its corresponding folder."
      ]
    },
    {
      "metadata": {
        "id": "Lyu_uctCMqY9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_name = ['apple', 'banana', 'book', 'fork', 'key', 'ladder', 'pizza', 'stop_sign', 'tennis_racquet', 'wheel']\n",
        "step = ['train', 'validation', 'test']\n",
        "dire = r'data/'\n",
        "\n",
        "createDir(RESULT_FOLDER)\n",
        "\n",
        "max_length = 10000         # Maximum number of files (drawings) per class\n",
        "percen = [0.6, 0.3, 0.1]   # Percentage of training, validation and testing\n",
        "\n",
        "begin = [0, int(max_length * percen[0]), int(max_length * (percen[0] + percen[1]))]\n",
        "end = [int(max_length * (percen[0])), int(max_length * (percen[0] + percen[1])), max_length-10]\n",
        "\n",
        "for c in range(0, len(class_name)):\n",
        "  print('Class ' + str(c+1) + ' out of ' + str(len(class_name)))\n",
        "  filename = dire + str(class_name[c]) + '.npy'\n",
        "  data = np.load(filename)\n",
        "\n",
        "  for s in range(0, len(step)):\n",
        "    dire_step = str(dire) + str(step[s])\n",
        "    if not os.path.exists(dire_step):\n",
        "      os.makedirs(dire_step)\n",
        "\n",
        "    for i in range(begin[s], end[s]):\n",
        "      dire_class = str(dire_step) + '/' + str(class_name[c])\n",
        "      if not os.path.exists(dire_class):\n",
        "        os.makedirs(dire_class)\n",
        "\n",
        "      # Reshape the raw data into 28x28 images\n",
        "      data_sample = data[i,:].reshape((28, 28))\n",
        "      sample_name = class_name[c] + '_' + str(step[s]) + '_' + str(i)\n",
        "      np.save(os.path.join(dire_class, sample_name), data_sample)\n",
        "        \n",
        "        \n",
        "print(\"The reduction & reshape is complete\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "l7PyIVmYNKRy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 3. Network Definition\n",
        "In this section we will define mini-batchs, will set the architecture of the network and the forward pass, and will also define the loss function and the optimizer."
      ]
    },
    {
      "metadata": {
        "id": "C6XSn_ysNbQB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 3.1 Mini-Batch Definition\n",
        "We define a mini-batch of size bs. This sample subsets of data is what is going to be forward propagated through the network. We use a mini-batch instead of the whole batch because it would be very expensive to use the complete training set."
      ]
    },
    {
      "metadata": {
        "id": "fv6aSrcYNobM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def load_sample(x):\n",
        "\treturn np.load(x)\n",
        "\n",
        "\n",
        "train_dataset = datasets.DatasetFolder(TRAIN_FOLDER, extensions = ['.npy'], loader = load_sample)\n",
        "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = BATCH_SIZE, shuffle = True, num_workers = 0)\n",
        "\n",
        "test_dataset = datasets.DatasetFolder(TEST_FOLDER, extensions = ['.npy'], loader = load_sample)\n",
        "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = BATCH_SIZE, shuffle = True, num_workers = 0)\n",
        "\n",
        "val_dataset = datasets.DatasetFolder(VALID_FOLDER, extensions = ['.npy'], loader = load_sample)\n",
        "val_loader = torch.utils.data.DataLoader(val_dataset, batch_size = BATCH_SIZE, shuffle = True, num_workers = 0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VnkSplSyO14i",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 3.2 MLP Definition and Forward Pass\n",
        "Followingly the multilayer perceptron network architecture is defined and the forward pass is implemented. Other architectures had been tried but this one has resulted to be the best one in terms of performance."
      ]
    },
    {
      "metadata": {
        "id": "BEytNHW0Uk9v",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class MLP(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(MLP, self).__init__()\n",
        "        self.fc1 = nn.Linear(IMAGE_WIDTH * IMAGE_HEIGHT, 500)\n",
        "        self.fc2 = nn.Linear(500, 500)\n",
        "        self.fc3 = nn.Linear(500, 500)\n",
        "        self.fc4 = nn.Linear(500, 500)\n",
        "        self.fc5 = nn.Linear(500, 256)\n",
        "        self.fc6 = nn.Linear(256, N_CLASSES)\n",
        "    def forward(self, x):\n",
        "        x = x.view(-1, IMAGE_WIDTH * IMAGE_HEIGHT)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.relu(self.fc2(x))\n",
        "        x = F.relu(self.fc3(x))\n",
        "        x = F.relu(self.fc4(x))\n",
        "        x = F.relu(self.fc5(x))\n",
        "        x =self.fc6(x)\n",
        "        return x\n",
        "\n",
        "net = MLP()\n",
        "print(\"The multilayer perceptron network model:\")\n",
        "print(net)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "IqMvUXtRQV_K",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 3.3 Loss Function and Optimizer Definition\n",
        "As we are working on a classification task, we have chosen to use the Cross Entropy Loss. For the optimizer we will use ADAM, because it is been observed that it gives better results than the Gradient Descent)."
      ]
    },
    {
      "metadata": {
        "id": "pj094-UzQQ9D",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "#The weight_decay is use for regulation\n",
        "optimizer = optim.Adam(net.parameters(), lr = LEARNING_RATE, WEIGHT_DECAY = 0.001)\n",
        "\n",
        "#If cuda exist, the network will use the GPU\n",
        "if CUDA:\n",
        "    net.cuda()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RxI8CBxERBMw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 4. Network Training\n",
        "In this section we will train our model and validate it with the validation data. At the end of the training, we will plot the lossses and the accuracies obtained for each epoch both for the training and the validation data."
      ]
    },
    {
      "metadata": {
        "id": "TjJ9oN07RmsT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 4.1 Training and Validation"
      ]
    },
    {
      "metadata": {
        "id": "wS0IVlCiQPRD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "training_loss = []\n",
        "training_accuracy = []\n",
        "val_loss = []\n",
        "val_accuracy = []\n",
        "\n",
        "def train(epoch, net, train_loader, opt):\n",
        "    # set model to train mode\n",
        "    net.train()\n",
        "    correct = 0.0\n",
        "    running_training_loss = 0.0\n",
        "    total = 0.0\n",
        "    for j, item in enumerate(train_loader, 0):\n",
        "      inputs, labels = item\n",
        "\n",
        "      inputs = inputs.view(BATCH_SIZE, 1, IMAGE_WIDTH, IMAGE_HEIGHT).float()\n",
        "      if CUDA:\n",
        "          inputs = inputs.cuda()\n",
        "          labels = labels.cuda()\n",
        "\n",
        "      # Reset gradients\n",
        "      opt.zero_grad()\n",
        "\n",
        "      # Forward pass\n",
        "      outputs = net(inputs)\n",
        "\n",
        "      pred = outputs.data.max(1)[1]   # get the index of the max log-probability\n",
        "      correct += (pred == labels).sum().item()\n",
        "      total += labels.size()[0]\n",
        "      loss = criterion(outputs, labels)\n",
        "      loss.backward()                 # calculate the gradients (backpropagation)\n",
        "      optimizer.step()                # update the weights\n",
        "      running_training_loss += loss.item()\n",
        "      if j % 200 == 199:\n",
        "        accuracy = correct / (BATCH_SIZE*200)\n",
        "        txt = '[%d, %5d] loss: %.3f - training accuracy: %.3f' % (epoch, j + 1, running_training_loss/200, accuracy)\n",
        "        training_loss.append(running_training_loss/200)\n",
        "        training_accuracy.append(accuracy)\n",
        "        running_training_loss = 0.0\n",
        "        correct = 0.0\n",
        "        total = 0.0\n",
        "        print(txt)\n",
        "       \n",
        "      \n",
        "def validate(net, val_loader, epoch):\n",
        "    # set model to validation mode\n",
        "    net.eval()\n",
        "    val_correct = 0.0\n",
        "    running_val_loss = 0.0\n",
        "    val_total = 0.0\n",
        "    for inputs, labels in val_loader:\n",
        "\n",
        "        inputs = inputs.view(BATCH_SIZE, 1, IMAGE_WIDTH, IMAGE_HEIGHT).float()\n",
        "        if CUDA:\n",
        "            inputs = inputs.cuda()\n",
        "            labels = labels.cuda()\n",
        "\n",
        "        outputs = net(inputs)\n",
        "        val_loss = criterion(outputs, labels)\n",
        "        val_pred = outputs.data.max(1)[1]\n",
        "        val_correct += (val_pred == labels).sum().item()\n",
        "        val_total += labels.size()[0]\n",
        "        running_val_loss += val_loss.item()\n",
        "\n",
        "    running_val_loss /= len(val_loader)\n",
        "    val_accuracy = val_correct / val_total\n",
        "    txt = '[ %d ] loss: %.3f - validation accuracy: %.3f' % (epoch, running_val_loss, val_accuracy)\n",
        "    val_loss.append(running_val_loss)\n",
        "    val_accuracy.append(val_accuracy)\n",
        "    print(\"[]\")\n",
        "    print(txt)\n",
        "              \n",
        "    return running_val_loss, val_accuracy\n",
        "\n",
        "\n",
        "for ep in range(EPOCHS):  # epochs loop\n",
        "\n",
        "    # train\n",
        "    loss_info = train(ep, net, train_loader, optimizer)\n",
        "    \n",
        "    # validate\n",
        "    val_loss, accuracy = validate(net, val_loader, ep)\n",
        "    \n",
        "    # save model weights\n",
        "    torch.save(net.state_dict(), MODEL_FOLDER + \"/model\" + str(ep))  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DijAWRwqT5Ry",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 4.2 Results Visualization"
      ]
    },
    {
      "metadata": {
        "id": "Z_La4U50R60R",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "np_training_loss = np.asarray(training_loss)\n",
        "np_validation_loss = np.asarray(val_loss)\n",
        "\n",
        "x_axis = np.asarray(range(0, len(np_training_loss)))\n",
        "x_axis_val = np.arange(0, len(np_validation_loss))\n",
        "plt.title(\"Loss\")\n",
        "plt.plot(x_axis * PLOT_EVERY * BATCH_SIZE / TRAINING_EX, np_training_loss)\n",
        "plt.plot(x_axis_val, np_validation_loss)\n",
        "plt.show()\n",
        "plt.title(\"Accuracy\")\n",
        "plt.plot(x_axis * PLOT_EVERY * BATCH_SIZE / TRAINING_EX, training_accuracy)\n",
        "plt.plot(x_axis_val, val_accuracy)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ucdi3CooQcld",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 5. Network Testing\n",
        "In this section, we will comput the test accuracy and the test loss, we will plot the confusion matrix to see which classess performed better and we will do a little performance demo."
      ]
    },
    {
      "metadata": {
        "id": "8Wn7eN31Y4N_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 5.1 Test Accuracy and Loss Computation\n",
        "Let's evaluate the model on the test data. To do so, we will pass to the network mini-batches of test data and compare their results with the ground truth to compute its loss and accuracy.\n",
        "\n",
        "Additionally, to see how well the network performs on different categories, we have created a plot that shows the accuracy for each class. It can be noted that classes that were very similar (wheel and pizza for example) have lower accuracy than the others, while very different and clear objects such as apple, have a very high accuracy."
      ]
    },
    {
      "metadata": {
        "id": "KRsZnM26QeTA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "running_test_loss = 0.0\n",
        "test_total = 0.0\n",
        "test_correct = 0.0\n",
        "  \n",
        "for i, test_data in enumerate(test_loader,0): \n",
        "  test_inputs, test_labels = test_data\n",
        "  test_inputs = test_inputs.view(BATCH_SIZE, 1, IMAGE_WIDTH, IMAGE_HEIGHT).float()\n",
        "  test_inputs = test_inputs.cuda()\n",
        "  test_labels = test_labels.cuda()\n",
        "  test_outputs = net(test_inputs)\n",
        "  test_loss = criterion(test_outputs, test_labels)\n",
        "  running_test_loss += test_loss.item()\n",
        "  \n",
        "  _,predicted = torch.max(test_outputs.data,1)\n",
        "  test_total = test_total + test_labels.size(0)\n",
        "  test_correct = test_correct + (predicted == test_labels).sum().item()        \n",
        "  \n",
        "test_accuracy = test_correct / test_total\n",
        "  \n",
        "print('Test Loss: %.3f - Test Accuracy: %.3f' % (running_test_loss/len(test_loader), test_accuracy))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pQgPuAU7ZHEO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 5.2 Performance Demo\n",
        "Finally, in this little demo we can see how the network performs for a random image of the test set. An interesting experiment to do is to first try to classify the image by ourselfs and then looking to the predicted class and the ground true value to see if the network performed better than a human..."
      ]
    },
    {
      "metadata": {
        "id": "LWePHgiXRg-S",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_correct = list(0. for i in range(10))\n",
        "class_total = list(0. for i in range(10))\n",
        "with torch.no_grad():\n",
        "    for data in test_loader:\n",
        "        test_inputs, test_labels = data\n",
        "        test_inputs = test_inputs.view(BATCH_SIZE, 1, 28, 28).float()\n",
        "        test_inputs = test_inputs.cuda()\n",
        "        test_labels = test_labels.cuda()\n",
        "        test_outputs = net(test_inputs)\n",
        "        _, predicted = torch.max(test_outputs.data,1)\n",
        "        c = (predicted == test_labels).squeeze()\n",
        "        for i in range(BATCH_SIZE):\n",
        "            label = test_labels[i]\n",
        "            class_correct[label] += c[i].item()\n",
        "            class_total[label] += 1\n",
        "\n",
        "x=np.arange(len(class_name))\n",
        "plt.barh(x, class_correct, align='center', alpha=0.5)\n",
        "plt.yticks(x, class_name)\n",
        "plt.xlabel('Accuracy')\n",
        "plt.title('Accuracy by Class')\n",
        " \n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}