lab-session-4-CNN-MNIST-student.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mE2F5Cj5X-cq"
   },
   "source": [
    "# Lab session 4 | Convolutional Neural Networks (CNN) with pytorch\n",
    "\n",
    "pierre-henri.conze@imt-atlantique.fr \\\\\n",
    "francois.rousseau@imt-atlantique.fr \\\\\n",
    "simon.benaichouche@imt-atlantique.fr \\\\\n",
    "aurelien.colin@imt-atlantique.fr\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aZNAqdzCX-cs"
   },
   "source": [
    "## Objective of this lab session: perform classification on MNIST using convolutional neural networks\n",
    "\n",
    "In lab session 3, MNIST classification has been performed relying on Multi-Layer Perceptron (MLP) models. The obtained accuracy was 92% with a softmax regressor and 97% (or 98%) with a deeper MLP. This can be further improved! Let us jump from such simple models to something moderately more sophisticated, namely **Convolutional Neural Networks** (CNN).\n",
    "\n",
    "For recall, **MNIST** is a computer vision dataset which consists of handwritten digit images with associated label. Each image in MNIST has a corresponding label - a number between 0 and 9 - representing the digit drawn in the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "85qr0BEYX-cu"
   },
   "source": [
    "### 1- Data management\n",
    "\n",
    "Start with these lines of code to automatically download the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "W6fsgjPOX-cv"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\anaconda3\\lib\\site-packages\\torchvision\\datasets\\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ..\\torch\\csrc\\utils\\tensor_numpy.cpp:180.)\n",
      "  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)\n"
     ]
    }
   ],
   "source": [
    "from torchvision import datasets\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "transform = transforms.ToTensor() # convert data to torch.FloatTensor\n",
    "\n",
    "train_data = datasets.MNIST(root = 'data', train = True, download = True, transform = transform)\n",
    "test_data = datasets.MNIST(root = 'data', train = False, download = True, transform = transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FQKjqImJ4mKx"
   },
   "source": [
    "#### **Question 1.1** - Complete the following cell to create **data loaders** ([documentation](https://pytorch.org/docs/stable/data.html)) for training, validation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ho-I57ync1Ya"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "import numpy as np\n",
    "\n",
    "batch_size = 20 # how many samples per batch to load\n",
    "valid_size = 0.2 # percentage of training set to use as validation\n",
    "\n",
    "def create_data_loaders(batch_size, valid_size, train_data, test_data):\n",
    "# ! By ourselves\n",
    "    num_train, num_test = len(train_data), len(test_data)\n",
    "    # obtain training indices that will be used for validation\n",
    "    indices = list(range(num_train))\n",
    "    np.random.shuffle(indices)\n",
    "    split = int(np.floor(valid_size * num_train))\n",
    "    train_index, valid_index = indices[split:], indices[:split]\n",
    "\n",
    "    # define samplers for obtaining training and validation subsets\n",
    "    train_sampler = SubsetRandomSampler(train_index)\n",
    "    valid_sampler = SubsetRandomSampler(valid_index)\n",
    "\n",
    "    # prepare data loaders\n",
    "    train_loader = torch.utils.data.DataLoader(train_data, batch_size = batch_size, sampler = train_sampler)\n",
    "    valid_loader = torch.utils.data.DataLoader(train_data, batch_size = batch_size, sampler = valid_sampler)\n",
    "    test_loader = torch.utils.data.DataLoader(test_data, batch_size = batch_size)\n",
    "\n",
    "    \n",
    "    return train_loader, valid_loader, test_loader\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gXWfzusFrlmQ"
   },
   "outputs": [],
   "source": [
    "train_loader, valid_loader, test_loader = create_data_loaders(batch_size, valid_size, train_data, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18G47IQQX-c3"
   },
   "source": [
    "Let us visualize some images from the training set with corresponding ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 249
    },
    "executionInfo": {
     "elapsed": 2019,
     "status": "ok",
     "timestamp": 1601150749171,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "arAFBf_q0K3p",
    "outputId": "bda651dd-5b73-4afe-ea7a-919340592e39"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "data_iter = iter(train_loader) \n",
    "images, labels = data_iter.next() # obtain one batch from the train set\n",
    "images = images.numpy()\n",
    "# plot images in the batch, along with the corresponding labels\n",
    "fig = plt.figure(figsize=(25, 4))\n",
    "for idx in np.arange(20):\n",
    "    ax = fig.add_subplot(2, 10, idx+1, xticks=[], yticks=[])\n",
    "    ax.imshow(np.squeeze(images[idx]), cmap='gray') # .npsqueeze removes single-dimensional entries from the shape of an array\n",
    "    ax.set_title(str(labels[idx].item())) # .item() gets the value contained in a Tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P9vqCd8jpJnE"
   },
   "source": [
    "### 2- Simple CNN architecture\n",
    "\n",
    "Let us define a simple **CNN architecture**. The network will take as inputs 28x28 images instead of 784-dimensional tensors of pixel values as for MLP models. As in lab session 3, it will produce as output a tensor of length 10 (i.e. the number of classes) that indicates the class scores for each input image. A CNN architecture is a stack of layers including:\n",
    "  - convolutional layer using *nn.Conv2D* ([documentation](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html))\n",
    "  - max-pooling layer using *nn.MaxPool2D* ([documentation](https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html))\n",
    "  - regular densely-connected layer using *nn.Linear* ([documentation](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "897aqt3UyfHz"
   },
   "source": [
    "#### **Question 2.1** - In this first network architecture (*Net1*), employ:\n",
    " - 2 consecutive convolutional layers using 32 3x3 filters with stride 1 followed by *ReLU* activation,\n",
    " - a max pooling with vertical and horizontal downscale of 2,\n",
    " - a flatten operator to flatten the input array,\n",
    " - a dense layer with 10 units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4lLiWMnC5jBy"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net1(nn.Module):  \n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net1,self).__init__()\n",
    "        # ! By ourselves\n",
    "        super(Net1,self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1,32,3, stride=1)\n",
    "        self.conv2 = nn.Conv2d(32,32,3, stride=1)\n",
    "        self.rel1  = nn.ReLU()\n",
    "        self.rel2  = nn.ReLU()\n",
    "        self.pool  = nn.MaxPool2d(2,2)\n",
    "        self.fc    = nn.Linear(32*32,10)\n",
    "\n",
    "        \n",
    "        \n",
    "        \n",
    "    def forward(self,x):\n",
    "        # ! By ourselves\n",
    "        x = self.conv1(x)\n",
    "        x = self.rel1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.rel2(x)\n",
    "        x = self.pool(x)\n",
    "        x = x.view(-1,32*32) \n",
    "        x = self.fc(x)\n",
    "        return x\n",
    "        \n",
    "\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "executionInfo": {
     "elapsed": 2210,
     "status": "ok",
     "timestamp": 1601150749537,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "OFwVrc_XT3cs",
    "outputId": "da238c36-0e52-400b-a0d5-8ce00297fa57"
   },
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print('Using device '+str(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K48q4mnBKBx_"
   },
   "source": [
    "**torchsummary** provides information complementary to what is provided by *print(your_model)* in PyTorch, similar to the Tensorflow *model.summary()* routine ([documentation](https://pypi.org/project/torch-summary/)). This can be helpful while debugging your network!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OG8wQhudOvOE"
   },
   "source": [
    "#### **Question 2.2** - Describe input/output sizes of each layer. Confirm your analysis by using *summary()* from *torchsummary*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "executionInfo": {
     "elapsed": 6470,
     "status": "ok",
     "timestamp": 1601150753817,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "JbdKfY3h-wAy",
    "outputId": "f08d9a8e-4d36-45ad-9293-3d61375341cb"
   },
   "outputs": [],
   "source": [
    "from torchsummary import summary\n",
    "\n",
    "cnn_1 = Net1() # initialize the neural network\n",
    "cnn_1.to(device=device)\n",
    "\n",
    "summary(cnn_1, (1, 28, 28))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OWkV71ZLI2ZO"
   },
   "source": [
    "#### **Question 2.3** - Before training a model, configure the learning process by indicating the criterion (i.e the objective loss function the model will try to minimize) as well as the optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BeZr3uLL58Qw"
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "z_IkVZB-01tJ"
   },
   "outputs": [],
   "source": [
    "optimizer = torch.optim.SGD(cnn_1.parameters(),lr = 0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dxYFf5sG0_XC"
   },
   "source": [
    "### 3- Training\n",
    "\n",
    "#### **Question 3.1** - Complete the following cell to perform the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TbGZO4wt6jTR"
   },
   "outputs": [],
   "source": [
    "n_epochs = 20 # number of epochs to train the model\n",
    "\n",
    "def training(n_epochs, train_loader, valid_loader, model, criterion, optimizer):\n",
    "\n",
    "  train_losses, valid_losses = [], []\n",
    "  # initialize tracker for minimum validation loss\n",
    "  valid_loss_min = np.Inf  # set initial \"min\" to infinity\n",
    "\n",
    "  for epoch in range(n_epochs):\n",
    "      train_loss, valid_loss = 0, 0 # monitor losses\n",
    "      \n",
    "      # train the model\n",
    "      model.train() # prep model for training\n",
    "      for data, label in train_loader:\n",
    "          data = data.to(device=device, dtype=torch.float32)\n",
    "          label = label.to(device=device, dtype=torch.long)\n",
    "          # ! By ourselves\n",
    "          optimizer.zero_grad()\n",
    "          output = model(data) # forward pass: compute predicted outputs by passing inputs to the model\n",
    "          loss = criterion(output, label) # calculate the loss\n",
    "          loss.backward() # backward pass: compute gradient of the loss with respect to model parameters\n",
    "          optimizer.step()      \n",
    "          train_loss += loss.item() * data.size(0) # update running training loss\n",
    "      \n",
    "      # validate the model\n",
    "      model.eval()\n",
    "      for data, label in valid_loader:\n",
    "          data = data.to(device=device, dtype=torch.float32)\n",
    "          label = label.to(device=device, dtype=torch.long)\n",
    "          with torch.no_grad():\n",
    "          # ! By ourselves\n",
    "            output = model(data)\n",
    "          loss = criterion(output,label)\n",
    "          valid_loss += loss.item() * data.size(0)\n",
    "      \n",
    "      # calculate average loss over an epoch\n",
    "      train_loss /= len(train_loader.sampler)\n",
    "      valid_loss /= len(valid_loader.sampler)\n",
    "      train_losses.append(train_loss)\n",
    "      valid_losses.append(valid_loss)\n",
    "      \n",
    "      print('epoch: {} \\ttraining Loss: {:.6f} \\tvalidation Loss: {:.6f}'.format(epoch+1, train_loss, valid_loss))\n",
    "\n",
    "      # save model if validation loss has decreased\n",
    "      if valid_loss <= valid_loss_min:\n",
    "          print('validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(\n",
    "          valid_loss_min,\n",
    "          valid_loss))\n",
    "          torch.save(model.state_dict(), 'model.pt')\n",
    "          valid_loss_min = valid_loss\n",
    "      \n",
    "  return train_losses, valid_losses      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 527
    },
    "executionInfo": {
     "elapsed": 180302,
     "status": "ok",
     "timestamp": 1601150927690,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "fulupe2f71cJ",
    "outputId": "f3c9fc23-7732-4a43-bc29-e4223b1e83e1"
   },
   "outputs": [],
   "source": [
    "train_losses_1, valid_losses_1 = training(n_epochs, train_loader, valid_loader, cnn_1, criterion, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "twmMCMSz1NoN"
   },
   "source": [
    "To study the **convergence** of the training process, we plot the evolution of the loss function for training and validation sets with respect to epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 310
    },
    "executionInfo": {
     "elapsed": 180281,
     "status": "ok",
     "timestamp": 1601150927691,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "HpT1Y7cNBA5F",
    "outputId": "2c57e90a-9ff2-4770-fd31-a634544a2de9"
   },
   "outputs": [],
   "source": [
    "plt.plot(range(n_epochs), train_losses_1)\n",
    "plt.plot(range(n_epochs), valid_losses_1)\n",
    "plt.legend(['train', 'validation'], prop={'size': 10})\n",
    "plt.title('loss function', size=10)\n",
    "plt.xlabel('epoch', size=10)\n",
    "plt.ylabel('loss value', size=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1C17bsUC1W_Y"
   },
   "source": [
    "Let us load the model corresponding to the lowest validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "executionInfo": {
     "elapsed": 180263,
     "status": "ok",
     "timestamp": 1601150927692,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "qzToZJN3FBbS",
    "outputId": "9957d43f-b7ae-43db-96d1-d21b0d52f459"
   },
   "outputs": [],
   "source": [
    "cnn_1.load_state_dict(torch.load('model.pt', map_location=device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NM5BWPs_3Pbz"
   },
   "source": [
    "### 4- Testing\n",
    "\n",
    "#### **Question 4.1** - Complete the following cell to test the (best) model on previously unseen test data and evaluate its performance through per-class and global accuracy scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KQIsKly4D6bw"
   },
   "outputs": [],
   "source": [
    "def evaluation(model, test_loader, criterion):\n",
    "\n",
    "  # initialize lists to monitor test loss and accuracy\n",
    "  test_loss = 0.0\n",
    "  class_correct = list(0. for i in range(10))\n",
    "  class_total = list(0. for i in range(10))\n",
    "\n",
    "  model.eval() # prep model for evaluation\n",
    "  for data, label in test_loader:\n",
    "      data = data.to(device=device, dtype=torch.float32)\n",
    "      label = label.to(device=device, dtype=torch.long)\n",
    "      with torch.no_grad():\n",
    "          output = ... # forward pass: compute predicted outputs by passing inputs to the model\n",
    "      loss = criterion( ...  )\n",
    "    \n",
    "      test_loss += loss.item()*data.size(0)\n",
    "      _, pred = torch.max(output, 1) # convert output probabilities to predicted class\n",
    "      correct = np.squeeze(pred.eq(label.data.view_as(pred))) # compare predictions to true label\n",
    "      # calculate test accuracy for each object class\n",
    "      for i in range(len(label)):\n",
    "          digit = label.data[i]\n",
    "          class_correct[digit] += correct[i].item()\n",
    "          class_total[digit] += 1\n",
    "\n",
    "  # calculate and print avg test loss\n",
    "  test_loss = test_loss/len(test_loader.sampler)\n",
    "  print('test Loss: {:.6f}\\n'.format(test_loss))\n",
    "  for i in range(10):\n",
    "      print('test accuracy of %1s: %2d%% (%2d/%2d)' % (str(i), 100 * class_correct[i] / class_total[i], np.sum(class_correct[i]), np.sum(class_total[i])))\n",
    "  print('\\ntest accuracy (overall): %2.2f%% (%2d/%2d)' % (100. * np.sum(class_correct) / np.sum(class_total), np.sum(class_correct), np.sum(class_total)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "executionInfo": {
     "elapsed": 182130,
     "status": "ok",
     "timestamp": 1601150929600,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "Dc3tr9mEZ2rP",
    "outputId": "9f7a8a20-fbd8-4a46-bb1a-4e7387d8834f"
   },
   "outputs": [],
   "source": [
    "evaluation( ... ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H6Qr8y1JrplH"
   },
   "source": [
    "#### **Question 4.2** - What is the overall accuracy achieved for test data? **Answer** : ... %."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Brj2Tz_6QMU"
   },
   "source": [
    "### 5- Assessment\n",
    "\n",
    "The following cell displays test images and their labels in this format: predicted (ground-truth). The text will be green for accurately classified examples and red for incorrect predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7rqBNImBD-E0"
   },
   "outputs": [],
   "source": [
    "def visualization(model, test_loader):\n",
    "\n",
    "  data_iter = iter(test_loader)\n",
    "  images, labels = data_iter.next() # obtain one batch of test images\n",
    "  images = images.to(device=device, dtype=torch.float32)\n",
    "  labels = labels.to(device=device, dtype=torch.long)\n",
    "  with torch.no_grad():\n",
    "      output = model(images) # get model output\n",
    "  _, preds = torch.max(output, 1) # convert output probabilities to predicted class\n",
    "  images = images.cpu().numpy() # prep images for display\n",
    "  # plot the images in the batch, along with predicted and true labels\n",
    "  fig = plt.figure(figsize=(25, 4))\n",
    "  for idx in np.arange(20):\n",
    "      ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])\n",
    "      ax.imshow(np.squeeze(images[idx]), cmap='gray')\n",
    "      ax.set_title(\"{} ({})\".format(str(preds[idx].item()), str(labels[idx].item())), color=(\"green\" if preds[idx]==labels[idx] else \"red\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 249
    },
    "executionInfo": {
     "elapsed": 182823,
     "status": "ok",
     "timestamp": 1601150930323,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "osfUv94NaEi4",
    "outputId": "b1ec8c03-0497-4eab-e420-a608ecd0967f"
   },
   "outputs": [],
   "source": [
    "visualization(cnn_1, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2pbH6LzoQ9i5"
   },
   "source": [
    "Let us extract predicted (*preds*) and ground truth (*targets*) labels for images arising from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jxxIS6e6KXrY"
   },
   "outputs": [],
   "source": [
    "def get_all_prediction(model, loader):\n",
    "    preds = torch.tensor([], dtype=torch.long)\n",
    "    targets = torch.tensor([], dtype=torch.long)\n",
    "    for data, label in loader:\n",
    "        data = data.to(device=device, dtype=torch.float32)\n",
    "        label = label.to(device=device, dtype=torch.long)\n",
    "        with torch.no_grad():\n",
    "            output = model(data)\n",
    "        targets = torch.cat((targets, label.cpu()), dim = 0)\n",
    "        preds = torch.cat((preds, torch.max(output.cpu(), 1)[1]), dim = 0)\n",
    "    return targets.numpy(), preds.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g8hqY6sX64Nm"
   },
   "outputs": [],
   "source": [
    "targets, preds_1 = get_all_prediction(cnn_1, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tq8IGvWn67Rs"
   },
   "source": [
    "#### **Question 5.1** - Visualize some wrongly predicted data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 245
    },
    "executionInfo": {
     "elapsed": 184595,
     "status": "ok",
     "timestamp": 1601150932129,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "-dcFjx1QGpkm",
    "outputId": "1908a110-1ce1-4089-c2eb-fef35fccf890"
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YwTFkR2RX-dx"
   },
   "source": [
    "#### **Question 5.2** - Display the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zw0oafsMX-dx"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import itertools\n",
    "\n",
    "class_names= ['0','1','2','3','4','5','6','7','8','9']\n",
    "\n",
    "def plot_confusion_matrix(cm, classes, normalize=False, title='confusion matrix', cmap=plt.cm.Blues):\n",
    "    # This function prints and plots the confusion matrix. Normalization can be applied by setting `normalize=True`\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"normalized confusion matrix\")\n",
    "    else:\n",
    "        print('confusion matrix, without normalization')\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('true label')\n",
    "    plt.xlabel('predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 458
    },
    "executionInfo": {
     "elapsed": 184992,
     "status": "ok",
     "timestamp": 1601150932556,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "nkuaLoa-8jOz",
    "outputId": "7785ad57-861e-47bf-c1ac-8a5b61d29743"
   },
   "outputs": [],
   "source": [
    "# compute confusion matrix\n",
    "cnf_matrix = \n",
    "\n",
    "\n",
    "# plot normalized confusion matrix\n",
    "\n",
    "\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z9AcBqA5qQ7h"
   },
   "source": [
    "### 6 - Robustness to noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3nEFKytdqvPz"
   },
   "outputs": [],
   "source": [
    "class AddGaussianNoise(object):\n",
    "    def __init__(self, mean, std):\n",
    "        self.std = std\n",
    "        self.mean = mean\n",
    "        \n",
    "    def __call__(self, tensor):\n",
    "        return tensor + torch.randn(tensor.size()) * self.std + self.mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ztLqmxHORxj6"
   },
   "source": [
    "Image transformations can be chained together using *Compose* ([documentation](https://pytorch.org/docs/stable/torchvision/transforms.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3Kpks3nVq0lQ"
   },
   "outputs": [],
   "source": [
    "transform=transforms.Compose([\n",
    "    transforms.ToTensor(),\n",
    "    AddGaussianNoise(0., 0.8)\n",
    "])\n",
    "\n",
    "train_data = datasets.MNIST(root = 'data', train = True, download = True, transform = transform)\n",
    "test_data = datasets.MNIST(root = 'data', train = False, download = True, transform = transform)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JqaBxTnLq5nQ"
   },
   "outputs": [],
   "source": [
    "train_loader_n, valid_loader_n, test_loader_n = create_data_loaders(batch_size, valid_size, train_data, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "atrFtOHzSEAM"
   },
   "source": [
    "Let us visualize some noisy images with corresponding ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 249
    },
    "executionInfo": {
     "elapsed": 186034,
     "status": "ok",
     "timestamp": 1601150933635,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "IaIAC26CsH9R",
    "outputId": "775ac7d9-5a72-4bc5-8d22-86071b2dc012"
   },
   "outputs": [],
   "source": [
    "data_iter = iter(train_loader_n) \n",
    "images, labels = data_iter.next() # obtain one batch from the train set\n",
    "images = images.numpy()\n",
    "# plot images in the batch, along with the corresponding labels\n",
    "fig = plt.figure(figsize=(25, 4))\n",
    "for idx in np.arange(20):\n",
    "    ax = fig.add_subplot(2, 10, idx+1, xticks=[], yticks=[])\n",
    "    ax.imshow(np.squeeze(images[idx]), cmap='gray') # .npsqueeze removes single-dimensional entries from the shape of an array\n",
    "    ax.set_title(str(labels[idx].item())) # .item() gets the value contained in a Tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_GIdzfq2RccF"
   },
   "source": [
    "#### **Question 6.1** - Train the network on noisy data and evaluate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "executionInfo": {
     "elapsed": 425740,
     "status": "ok",
     "timestamp": 1601151173359,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "4uNLhPjRsZom",
    "outputId": "8810d8b2-f279-4299-fb96-483b8dbc8611"
   },
   "outputs": [],
   "source": [
    "train_losses_n, valid_losses_n = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "executionInfo": {
     "elapsed": 425723,
     "status": "ok",
     "timestamp": 1601151173360,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "gGsSOrTJtAL8",
    "outputId": "71cce560-5fd9-400b-d880-23b45cea5bf2"
   },
   "outputs": [],
   "source": [
    "cnn_1.load_state_dict(torch.load('model.pt', map_location=device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "executionInfo": {
     "elapsed": 427793,
     "status": "ok",
     "timestamp": 1601151175449,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "qfz9omj3tGhD",
    "outputId": "a9533d8d-cdf9-4170-f4fe-4e861ad1eb89"
   },
   "outputs": [],
   "source": [
    "evaluation(...) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mpq2aqtw7MdR"
   },
   "source": [
    "### 7- Towards deeper CNN models\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FY_2gW_3SPGY"
   },
   "source": [
    "In this last part, we come back to the original dataset, i.e. without additional noise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W9vHjuf_ST3q"
   },
   "source": [
    "#### **Question 7.1** - Implement a deeper convolutional neural network by adding two convolutional layers, one max pooling layer as well as dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "i27yTyfnQj1t"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net2(nn.Module): \n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net2,self).__init__()\n",
    "        \n",
    "\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "    def forward(self,x):\n",
    "\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 493
    },
    "executionInfo": {
     "elapsed": 427769,
     "status": "ok",
     "timestamp": 1601151175451,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "TiddhzR3SkW-",
    "outputId": "c163ac47-f9d7-4a1d-e864-f450ccfd0609"
   },
   "outputs": [],
   "source": [
    "cnn_2 = Net2()\n",
    "cnn_2.to(device=device)\n",
    "\n",
    "summary(...) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cUP7EYwn-9lC"
   },
   "outputs": [],
   "source": [
    "optimizer = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 595
    },
    "executionInfo": {
     "elapsed": 650304,
     "status": "ok",
     "timestamp": 1601151398013,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "xkfZoZzxStIn",
    "outputId": "5cbd9922-20b2-4857-f5bb-5359c29cb77a"
   },
   "outputs": [],
   "source": [
    "train_losses_2, valid_losses_2 = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "executionInfo": {
     "elapsed": 650288,
     "status": "ok",
     "timestamp": 1601151398014,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "xp_M2Lw-1L6L",
    "outputId": "65df802a-17c0-412c-9f28-ce5604ca2f91"
   },
   "outputs": [],
   "source": [
    "cnn_2.load_state_dict(torch.load('model.pt', map_location=device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X3KyDic8UbmE"
   },
   "source": [
    "#### **Question 7.2** - Quantify the gain from *Net1* to *Net2* in term of overall accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "executionInfo": {
     "elapsed": 652435,
     "status": "ok",
     "timestamp": 1601151400179,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "Y6-N59GizdA9",
    "outputId": "f9b1b7ca-6b75-498c-8778-842f9bdde4e3"
   },
   "outputs": [],
   "source": [
    "evaluation(...) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9HGAp_PrunNf"
   },
   "source": [
    "#### **Question 7.3** - What are the results for incorrect predictions arising from *Net1*?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "10BGrED1z6qH"
   },
   "outputs": [],
   "source": [
    "_, preds_2 = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OYHp_7VYwuC7"
   },
   "source": [
    "You may display test images and labels in the following format: predicted from *Net1* (ground-truth) | predicted from *Net2*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 245
    },
    "executionInfo": {
     "elapsed": 654479,
     "status": "ok",
     "timestamp": 1601151402249,
     "user": {
      "displayName": "Pierre-Henri Conze",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhZ4Qbz7lM98QP5Y2jxN8hIcfomTY44lTm-4fcHOg=s64",
      "userId": "10921126790911374273"
     },
     "user_tz": -120
    },
    "id": "s_Yeqh9p95ti",
    "outputId": "35febc56-c642-4bf0-f0e9-b0351a4a8271"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(25, 4))\n",
    "for i in range(20):\n",
    "  plt.subplot(2, 10, i + 1)\n",
    "  plt.axis('off')\n",
    "  plt.imshow(test_set_array[index[i],:,:], cmap='gray')\n",
    "  plt.title(\"{} ({}) | {}\".format(str(np.int(preds_1[index[i]])), str(np.int(targets[index[i]])), str(np.int(preds_2[index[i]]))), color=(\"green\" if preds_2[index[i]]==targets[index[i]] else \"red\"))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DSu3G7LWWcde"
   },
   "source": [
    "### 8- Interactive demonstration\n",
    "\n",
    "To finish this lab session, let us play with the following interactive demo: [link](http://mnist-demo.herokuapp.com). This web application demonstrates the ability of both CNN and MLP models to classify handwritten digits.\n",
    "\n",
    "#### **Question 8.1** - Try to draw a digit which is more accurately classified by CNN than MLP. \n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "lab-session-4-CNN-MNIST-correction.ipynb",
   "provenance": []
  },
  "interpreter": {
   "hash": "7b4b4feff2f24a0f0a34464dbe537a36fda679851528fb8735cb41fa49dffb2d"
  },
  "kernelspec": {
   "display_name": "Python 3.8.8 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}