HW2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iHDIw_DL0SXh"
   },
   "source": [
    "# Problem Set 2 \n",
    "\n",
    "<h4>Designed by Xide Xia, with help from Ashok Cutkosky and Brian Kulis. <br> </h4>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "This assignment will introduce you to:\n",
    "1. Understanding the power of ReLU activation.\n",
    "2. Implementing your own autograd.\n",
    "3. Implementing a simple MLP.\n",
    "4. Basic functionality in PyTorch\n",
    "\n",
    "This code has been tested on Colab. \n",
    "\n",
    "---\n",
    "\n",
    "## Preamble\n",
    "\n",
    "To run and solve this assignment, you need an interface to edit and run ipython notebooks (`.ipynb` files). The easiest way to complete this assignment is to use Google Colab. You can just copy the assignment notebook to your google drive and open it, edit it and run it on Google Colab. All libraries you need are pre-installed on Colab.\n",
    "\n",
    "---\n",
    "\n",
    "### Local installation\n",
    "The alternative is to have a local installation, although we do not recommend it. If you are working on Google Colab, feel free to skip to the next section \"More instructions\". We recommend using virtual environments for all your installations. Following is one way to set up a working environment on your local machine for this assignment, using [Anaconda](https://www.anaconda.com/distribution/): \n",
    "\n",
    "- Download and install Anaconda following the instructions [here](https://docs.anaconda.com/anaconda/install/)\n",
    "- Create a conda environment using `conda create --name dl_env python=3` (You can change the name of the environment instead of calling it `dl_env`)\n",
    "- Now activate the environment using : `conda activate dl_env`\n",
    "- Install jupyter lab, which is the [jupyter project's](https://jupyter.org/index.html) latest notebook interface : `pip install jupyterlab`. You can also use the classic jupyter notebooks and there isn't any difference except the interface.\n",
    "- Install other necessary libraries. For this assignment you need `numpy`, `scipy` , [`pytorch`](https://pytorch.org/get-started/locally/) and `matplotlib`, all of which can be installed using : `pip install <lib_name>`. Doing this in the environment, would install these libraries for `dl_env`. You can also use `conda install`.\n",
    "- Now download the assignment notebook in a local directory and launching `jupyter lab` in the same directory should open a jupyter lab session in your default browser, where you can open and edit the ipython notebook.\n",
    "- For deactivating the environment when you are done with it, use : `conda deactivate`.\n",
    "\n",
    "For users running a Jupyter server on a remote machine :\n",
    "- Launch Jupyter lab on the remote server (in the directory with the homework ipynb file) using : `jupyter lab --no-browser --ip=0.0.0.0`\n",
    "- To access the jupyter lab interface on your local browser, you need to set up ssh port forwarding. This can be done by running : `ssh -N -f -L localhost:8888:localhost:8888 <remoteuser>@<remotehost>`. You can now open `localhost:8888` on your local browser to access jupyter lab. This assumes you are running jupyter lab on its default port 8888 on the server.\n",
    "- Check \"Making life easy\" section at the end of [this post](https://ljvmiranda921.github.io/notebook/2018/01/31/running-a-jupyter-notebook/) to find how to add functions to your bash run config to do this more easily each time. The post mentions functions for jupyter notebook, but just replace those with jupyter lab if you are using that interface.\n",
    "\n",
    "The above instructions specify one way of working on the assignment. You can use other virtual environments/ipython notebook interfaces etc. (**not recommended**).\n",
    "\n",
    "---\n",
    "\n",
    "### More instructions\n",
    "\n",
    "If you are new to Python or its scientific library, Numpy, there are some nice tutorials [here](https://www.learnpython.org/) and [here](http://www.scipy-lectures.org/).\n",
    "\n",
    "In an ipython notebook, to run code in a cell or to render [Markdown](https://en.wikipedia.org/wiki/Markdown)+[LaTeX](https://en.wikipedia.org/wiki/LaTeX) press `Ctrl+Enter` or `[>|]`(like \"play\") button above. To edit any code or text cell (double) click on its content. To change cell type, choose \"Markdown\" or \"Code\" in the drop-down menu above.\n",
    "\n",
    "To enter your solutions for the written questions, put down your derivations into the corresponding cells below using LaTeX. Show all steps when proving statements. If you are not familiar with LaTeX, you should look at some tutorials and at the examples listed below between \\$..\\$. We will not accept handwritten solutions.\n",
    "\n",
    "Put your solutions into boxes marked with **`[double click here to add a solution]`** and press Ctrl+Enter to render text. (Double) click on a cell to edit or to see its source code. You can add cells via **`+`** sign at the top left corner.\n",
    "\n",
    "**Submission instructions:** please upload your completed solution file (having run all code cells and rendered all markdown/Latex) to the Google Form posted on Piazza by the due date (see Schedule for due dates and late policy).  \n",
    "\n",
    "Note: `Vector` stands for `column vector` below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2JkxmpsAj7VK"
   },
   "source": [
    "# Problem 1: Universal approximation power of ReLU networks (20 points)\n",
    "\n",
    "As we dicussed in class, a two layer NN with sigmoid activation function is a universal approximator, i.e: with sufficient hidden units, it can approximate any real function with desired accuracy. In this problem we want to demonstrate universal approximation power of NNs using ReLU activation units.\n",
    "\n",
    "## **Q1.1** \n",
    "Show that by composing only 2 hidden units in a ReLU network $-\\sum_{i=1}^2a_i\\ max(0,b_ix+c_i) -$ we can build an approximation to the step function $1[x>0]$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R35DbojskG6p"
   },
   "source": [
    "**Solution:**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ysSRJ3sbkLMt"
   },
   "source": [
    "## **Q1.2** \n",
    "\n",
    "Show that by composing 4 hidden units in a ReLU network; we can build an approximation to the unit impulse function of duration $\\delta$\n",
    "\n",
    "\\begin{equation}\n",
    "u_\\delta(x) = 1[0\\leq x\\leq \\delta]\n",
    "\\end{equation}\n",
    "\n",
    "The approximator should have value $1$ between $\\frac{\\delta}{4}$ and $\\frac{3\\delta}{4}$ and should be increasing/decreasing on either side of this for a duration of $\\frac{\\delta}{2}$, i.e., it should be 0 for all values less than $\\frac{-\\delta}{4}$ and more than $\\frac{5\\delta}{4}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j5wGbZBjkLC1"
   },
   "source": [
    "**Solution:**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WbyanYybkK6C"
   },
   "source": [
    "## **Q1.3** \n",
    "Using your approximator for the unit impulse function in Q3.2, complete the code given bellow to draw the approximator for different duration values $\\delta$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KoCRLBNcktn8"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def relu(x):\n",
    "    return np.maximum(0,x)\n",
    "def hat_u_delta(x,delta):\n",
    "    y = 0 ## -- ! code required   \n",
    "    return y\n",
    "\n",
    "def draw_impulse(deltas):\n",
    "    x = np.arange(-2, 2, 0.01).reshape((-1,1))\n",
    "    for delta in deltas:\n",
    "        plt.plot(x,hat_u_delta(x,delta))\n",
    "    plt.legend(['$\\delta$ = 0.5', '$\\delta$ = 0.25', '$\\delta$ = 0.05']);\n",
    "    plt.xlabel('x')\n",
    "    plt.ylabel('y');\n",
    "\n",
    "draw_impulse([0.5, 0.25, 0.05])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "82j6X2HNkKwy"
   },
   "source": [
    "## **Q1.4**\n",
    "Imagine the idea of riemann integral, where we approximate the integrand function with unit impulse functions (fig(1)). \n",
    "We will approximate the function $f(x)$ defined over $[a,b]$, using N impulse functions as follows:\n",
    "\n",
    "$$\\hat{f(x)} = \\sum_{i=0}^{N-1} f(a + i \\delta)\\, u_\\delta(x-i\\delta), $$\n",
    "where: $$\\delta = \\lfloor \\frac{b-a}{N} \\rfloor$$\n",
    "\n",
    "![riemann-gif](https://drive.google.com/uc?id=1nY1BHbbEpdm7OE3USfc0BhbbEWlTmsvv)\n",
    "\n",
    "Using your implemented approximator for unit impulse function in Q3.2; complete the code given bellow to approximate the $sin(x)$ function over $[0, 2\\pi]$. The code will plot the approximation for different number of impule functions $N$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_IhOmYTMkVAt"
   },
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    return np.sin(x)\n",
    "\n",
    "def hat_f(x,N,a,b):\n",
    "    y = np.zeros(x.shape)\n",
    "    delta = (b-a)/N\n",
    "    for i in range(N):\n",
    "        y += 0 ## -- ! code required   \n",
    "    return y\n",
    "\n",
    "def draw_hat_f(N,a,b):\n",
    "    x = np.arange(a, 1.5*b, 0.01).reshape((-1,1))\n",
    "    plt.plot(x,f(x))\n",
    "    for n in N:\n",
    "        y = hat_f(x,n,a,b)\n",
    "        plt.plot(x,y);\n",
    "    plt.legend([\"f(x)\"]+[\"N = \" + str(n) for n in N],loc = 'upper right')\n",
    "\n",
    "draw_hat_f([3,10,20],0,2*3.15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OQtPkwn1kKnf"
   },
   "source": [
    "# Problem 2: Autograd implementation. (30 points)\n",
    "\n",
    "In class, we discussed the forward and back-propagation in network layers. We pass the input through a network layer and calculate the output of the layer straightforwardly. This step is called forward-propagation. Each layer also implements a function called 'backward'. Backward is responsible for the backward pass of back-propagation. The process of back-propagation follows the schemas: Input -> Forward calls -> Loss function -> derivative -> back-propagation of errors. In neural network, any layer can forward its results to many other layers, in this case, in order to do back-propagation, we sum the deltas coming from all the target layers. \n",
    "\n",
    "In this problem, we first show the example of bias layer. And you will implement both forward and backward for the most commonly used layers including: linear, ReLU, sigmoid, softmax, and cross-entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ko0kWtlUjutN"
   },
   "outputs": [],
   "source": [
    "'''backprop implementation with layer abstraction.\n",
    "This could be made more complicated by keeping track of an actual DAG of\n",
    "operations, but this way is not too hard to implement.\n",
    "'''\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "class Layer:\n",
    "    '''A layer in a network.\n",
    "\n",
    "    A layer is simply a function from R^n to R^d for some specified n and d.\n",
    "    A neural network can usually be written as a sequence of layers:\n",
    "    if the original input x is in R^d, a 3 layer neural network might be:\n",
    "\n",
    "    L3(L2(L1(x)))\n",
    "\n",
    "    We can also view the loss function as itself a layer, so that the loss\n",
    "    of the network is:\n",
    "\n",
    "    Loss(L3(L2(L1(x))))\n",
    "\n",
    "    This class is a base class used to represent different kinds of layer\n",
    "    functions. We will eventually specify a neural network and its loss function\n",
    "    with a list:\n",
    "\n",
    "    [L1, L2, L3, Loss]\n",
    "\n",
    "    where L1, L2, L3, Loss are all Layer objects.\n",
    "\n",
    "    Each Layer object implements a function called 'forward'. forward simply\n",
    "    computes the output of a layer given its input. So instead of\n",
    "    Loss(L3(L2(L1(x))), we write\n",
    "    Loss.forward(L3.forward(L2.forward(L1.forward(x)))).\n",
    "    Doing this computation finishes the forward pass of backprop.\n",
    "\n",
    "    Each layer also implements a function called 'backward'. Backward is\n",
    "    responsible for the backward pass of backprop. After we have computed the\n",
    "    forward pass, we compute\n",
    "    L1.backward(L2.backward(L3.backward(Loss.backward(1))))\n",
    "    We give 1 as the input to Loss.backward because backward is implementing\n",
    "    the chain rule - it multiplies gradients together and so giving 1 as an\n",
    "    input makes the multiplication an identity operation.\n",
    "\n",
    "    The outputs of backward are a little subtle. Some layers may have a\n",
    "    parameter that specifies the function being computed by the layer. For\n",
    "    example, a Linear layer maintains a weight matrix, so that\n",
    "    Linear(x) = xW\n",
    "    for some matrix W.\n",
    "    The input to backward should be the gradient of the final loss with respect\n",
    "    to the output of the current layer. The output of backprop should be the\n",
    "    gradient of the final loss with respect to the input of the current layer,\n",
    "    which is just the output of the previous layer. This is why it is correct\n",
    "    to chain the outputs of backprop together. However, backward should ALSO\n",
    "    compute the gradient of the loss with respect to the current layer's\n",
    "    parameter and store this internally to be used in training.\n",
    "    '''\n",
    "    def __init__(self, parameter=None, name=None):\n",
    "        self.name = name\n",
    "        self.forward_called = False\n",
    "        self.parameter = parameter\n",
    "        self.grad = None\n",
    "\n",
    "    def zero_grad(self):\n",
    "        self.grad = None\n",
    "\n",
    "    def forward(self, input):\n",
    "        '''forward pass. Should compute layer and save relevant state\n",
    "        needed for backward pass.\n",
    "        Args:\n",
    "            input: input to this layer.\n",
    "        returns output of operation.\n",
    "        '''\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        '''Performs backward pass.\n",
    "\n",
    "        This function should also set self.grad to be the gradient of the final\n",
    "        output of the computation with respect to the parameter.\n",
    "\n",
    "        Args:\n",
    "            downstream_grad: gradient from downstream operation in the\n",
    "                computation graph. This package will only consider\n",
    "                computation graphs that result in scalar outputs at the final\n",
    "                node (e.g. loss function computations). As a result,\n",
    "                the dimension of downstream_grad should match the dimension of\n",
    "                the output of this layer.\n",
    "\n",
    "                Formally, if this operation computes F(x), and the final\n",
    "                computation computes a scalar, G(F(x)), then input_grad is\n",
    "                dG/dF.\n",
    "        returns:\n",
    "            gradient to pass to upstream layers. If the layer computes F(x, w),\n",
    "            where x is the input and w is the parameter of the layer, then\n",
    "            the return value should be dF(x,w)/dx * downstream_grad. Here,\n",
    "            x is in R^n, F(x, w) is in R^m, dF(x, w)/dx is a matrix in R^(n x m)\n",
    "            downstream_grad is in R^m and * indicates matrix multiplication.\n",
    "\n",
    "        We should also compute the gradient with respect to the parameter w.\n",
    "        Again by chain rule, this is dF(x, w)/dw * downstream_grad\n",
    "        '''\n",
    "        raise NotImplementedError\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EKZ3SP2lTpSJ"
   },
   "source": [
    "Below shows an example of the full implementation of the Bias layer, including the forward and backward function. Notice self.grad stores the gradient of the loss with respect to the current layer's parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JF30rs0ITosh"
   },
   "outputs": [],
   "source": [
    "class Bias(Layer):\n",
    "    '''adds a constant bias.\n",
    "    The bias is assumed to be a K-dimensional ndarray for arbitary K that is\n",
    "    added to the last K dimensions of a D dimensional input and broadcast\n",
    "    over the first D-K dimensions.'''\n",
    "\n",
    "    def __init__(self, bias, name=\"bias\"):\n",
    "        super(Bias, self).__init__(np.squeeze(bias), name)\n",
    "        self.weights = np.squeeze(bias)\n",
    "\n",
    "    def forward(self, input):\n",
    "        self.input = input\n",
    "        return self.parameter + self.input\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        self.grad = np.sum(downstream_grad, tuple(range(downstream_grad.ndim - self.parameter.ndim)))\n",
    "        return downstream_grad\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ntLOrh72mHOW"
   },
   "source": [
    "## **Q2.1** Multiplication layer.\n",
    "\n",
    "Let's start with the basic linear and bias layer. Show the derivatives of linear layer with respect to $X$.\n",
    "\n",
    "$Z_{linear} = XW$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UhpqN_9O2fmz"
   },
   "source": [
    "**Solution:**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5gyrJM3O2k2U"
   },
   "source": [
    "Complete the forward and backward function of the linear layer. In backward, you should ALSO set the self.grad to be the gradient of the loss with respect to the current layer's parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4L8Dcfq-lTRd"
   },
   "outputs": [],
   "source": [
    "class Linear(Layer):\n",
    "    '''Linear layer. Parameter is NxM matrix L, input is matrix v of size B x N\n",
    "    where B is batch size, output is vL.'''\n",
    "\n",
    "    def __init__(self, weights, name=\"Linear\"):\n",
    "        super(Linear, self).__init__(weights, name)\n",
    "        self.weights = weights\n",
    "\n",
    "    def forward(self, input):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        '''downstream_grad should be BxM.'''\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "icklJkILnhNB"
   },
   "source": [
    "## **Q2.2** Activation layers.\n",
    "\n",
    "Now let's look at the activation layers. Show the derivatives of ReLU and sigmoid. \n",
    "<p>\n",
    "$ReLU(x) = max(0,x)$\n",
    "</p> \n",
    "<p>\n",
    "$\\sigma(x)=\\frac{1}{1+e^{-x}}$\n",
    "</p> \n",
    "\n",
    "Hint: Let's assume the gradient of ReLU is 0 when x is 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nq5dtn6k49SR"
   },
   "source": [
    "**Solution:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NZPP7OTt49NQ"
   },
   "source": [
    "Complete the forward and backward functions. There is no need to update self.grad since there is no parameter in activation layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FgbalOrBlYkS"
   },
   "outputs": [],
   "source": [
    "class ReLU(Layer):\n",
    "    '''ReLU layer. No parameters.'''\n",
    "\n",
    "    def __init__(self, name=\"ReLU\"):\n",
    "        super(ReLU, self).__init__(name=name)\n",
    "\n",
    "    def forward(self, input):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "\n",
    "\n",
    "class Sigmoid(Layer):\n",
    "    '''Sigmoid layer. No parameters.'''\n",
    "\n",
    "    def __init__(self, name=\"Sigmoid\"):\n",
    "        super(Sigmoid, self).__init__(name=name)\n",
    "\n",
    "    def forward(self, input):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zU1LRCr_oI5o"
   },
   "source": [
    "## **Q2.3**  Loss *layers*.\n",
    "\n",
    "Show the derivatives of softMax and Cross-entropy. \n",
    "\n",
    "<p>\n",
    "$\\sigma(j)=\\frac{\\exp(w_j^Tx)}{\\sum_{k}\\exp(w_k^Tx)}$. \n",
    "</p> \n",
    "\n",
    "<p>\n",
    "$L(\\hat{y},y)=-yln(\\hat{y}) - (1-y)ln(1-\\hat{y})$\n",
    "</p> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VghiDCb45L4M"
   },
   "source": [
    "**Solution:** \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EyxKVXh76VIi"
   },
   "source": [
    "Complete the forward and backward functions. There is no need to update self.grad since there is no parameter in the layers. \n",
    "\n",
    "Hint: You may find [np.einsum](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html) helpful when computing the gradient of SoftMax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CxdqItKPlY-x"
   },
   "outputs": [],
   "source": [
    "class SoftMax(Layer):\n",
    "    '''SoftMax Layer, no parameters.'''\n",
    "\n",
    "    def __init__(self, name=\"SoftMax\"):\n",
    "        super(SoftMax, self).__init__(name=\"softmax\")\n",
    "\n",
    "    def forward(self, input):\n",
    "        '''input is BxN array, B is batch dimension.'''\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        '''downstream grad should be BxN array.'''\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "\n",
    "class CrossEntropy(Layer):\n",
    "    '''cross entropy loss.'''\n",
    "\n",
    "    def __init__(self, labels, name=\"Cross Entropy\"):\n",
    "        '''labels is BxC 1-hot vector for correct label.'''\n",
    "        super(CrossEntropy, self).__init__(name=\"Cross Entropy\")\n",
    "        self.labels = labels\n",
    "\n",
    "    def forward(self, input):\n",
    "        '''input is BxN, output is a scalar'''\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n",
    "    def backward(self, downstream_grad):\n",
    "        ## -- ! code required  \n",
    "        raise NotImplementedError\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sa8Lz_8Ro5F-"
   },
   "source": [
    "## **Q2.4** Test your implementations.\n",
    "\n",
    "Now let's build a simple model using your layers, and compare the autograd results with the numeric derivatives. If everything is implemented in the correct way, the autograd results should be very closed to numeric grad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3bcJ46cGlh3c"
   },
   "outputs": [],
   "source": [
    "def numerical_derivative(layers, input):\n",
    "    base_output = forward_layers(layers, input)\n",
    "    delta = 1e-7\n",
    "    \n",
    "    for layer in layers:\n",
    "        if layer.parameter is None:\n",
    "            continue\n",
    "        size = layer.parameter.size # total number of params\n",
    "        shape = layer.parameter.shape # shape of params\n",
    "        base_param = np.copy(layer.parameter)\n",
    "        perturb = np.zeros(size)\n",
    "        grad = np.zeros(size)\n",
    "         \n",
    "        for i in range(size):\n",
    "            perturb[i] = delta # only current i-th perturb is non-zero\n",
    "            layer.parameter = base_param + np.reshape(perturb, shape) # make a small change (delta) on the i-th parameter\n",
    "            perturb_output = forward_layers(layers, input) # new output after adding a small change (delta) on the i-th parameter\n",
    "            grad[i] = (perturb_output - base_output) / delta # update the grad of i-th parameter\n",
    "            perturb[i] = 0.0 # set it back to zero\n",
    "            \n",
    "        layer.parameter = base_param\n",
    "        layer.grad = np.reshape(np.copy(grad), shape)\n",
    "\n",
    "def forward_layers(layers, input):\n",
    "    '''Forward pass on all the layers. Must be called before backwards pass.'''\n",
    "    output = input\n",
    "    for layer in layers:\n",
    "        output = layer.forward(output)\n",
    "    #assert output.size == 1, \"only supports computations that output a scalar!\"\n",
    "    return output\n",
    "\n",
    "\n",
    "def backward_layers(layers):\n",
    "    '''runs a backward pass on all the layers.\n",
    "    after this function is finished, look at layer.grad to find the\n",
    "    gradient with respect to that layer's parameter.'''\n",
    "    downstream_grad = np.array([1])\n",
    "    for layer in reversed(layers):\n",
    "        downstream_grad = layer.backward(downstream_grad)\n",
    "\n",
    "\n",
    "def zero_grad(layers):\n",
    "    for layer in layers:\n",
    "        layer.zero_grad()\n",
    "\n",
    "        \n",
    "def test_autograd():\n",
    "    h = 2\n",
    "    b = 3\n",
    "    input = np.random.normal(np.zeros((b, h)))\n",
    "    labels = np.zeros((b, h))\n",
    "    for i in range(b):\n",
    "        labels[i, np.random.choice(range(h))] = 1.0\n",
    "\n",
    "    layers = [\n",
    "        Linear(np.random.normal(size=(h, 2 * h))),\n",
    "        Sigmoid(),\n",
    "        Bias(np.array([np.random.normal()])),\n",
    "        Linear(np.random.normal(size=(2 * h, 3 * h))),\n",
    "        ReLU(),\n",
    "        Linear(np.random.normal(size=(3 * h, h))),\n",
    "        SoftMax(),\n",
    "        CrossEntropy(labels)\n",
    "    ]\n",
    "    output = forward_layers(layers, input)\n",
    "    backward_layers(layers)\n",
    "    analytics = [np.copy(layer.grad)\n",
    "                 for layer in layers if layer.grad is not None]\n",
    "    zero_grad(layers)\n",
    "\n",
    "    numerical_derivative(layers, input)\n",
    "    numerics = [np.copy(layer.grad)\n",
    "                for layer in layers if layer.grad is not None]   \n",
    "    diff = np.sum([np.linalg.norm(analytic - numeric)/np.linalg.norm(numeric)\n",
    "                   for analytic, numeric in zip(analytics, numerics)])\n",
    "    \n",
    "    assert diff < 1e-5, \"autograd differs by {} from numeric grad!\".format(diff)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    test_autograd()\n",
    "    print(\"looking good!\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y613iZJ0pjO2"
   },
   "source": [
    "## Problem 3: Implementing a simple MLP. (25 points)\n",
    "\n",
    "In this problem we will develop a neural network with fully-connected layers, aka Multi-Layer Perceptron (MLP) using the layers from Problem 2. Below, we initialize toy data  that we will use to develop your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "gD5cHE09pVAr",
    "outputId": "e8f7f7d9-6752-4f17-b7ab-550b8374fea3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X =  [[ 16.24345364  -6.11756414  -5.28171752 -10.72968622]\n",
      " [  8.65407629 -23.01538697  17.44811764  -7.61206901]\n",
      " [  3.19039096  -2.49370375  14.62107937 -20.60140709]\n",
      " [ -3.22417204  -3.84054355  11.33769442 -10.99891267]\n",
      " [ -1.72428208  -8.77858418   0.42213747   5.82815214]]\n",
      "\n",
      "\n",
      "y =  [0 1 2 2 1]\n"
     ]
    }
   ],
   "source": [
    "# setup\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))\n",
    "\n",
    "\n",
    "# Create a small net and some toy data to check your implementations.\n",
    "# Note that we set the random seed for repeatable experiments.\n",
    "input_size = 4\n",
    "hidden_size = 10\n",
    "num_classes = 3\n",
    "num_inputs = 5\n",
    "\n",
    "def init_toy_data():\n",
    "    np.random.seed(1)\n",
    "    X = 10 * np.random.randn(num_inputs, input_size)\n",
    "    y = np.array([0, 1, 2, 2, 1])\n",
    "    return X, y\n",
    "\n",
    "X, y = init_toy_data()\n",
    "print ('X = ', X)\n",
    "print('\\n')\n",
    "print('y = ', y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HtBLq2v7EiUK"
   },
   "source": [
    "We will use the following class `TwoLayerMLP` to implement our network. The network parameters are stored in the instance variable `self.params` where keys are string parameter names and values are numpy arrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HM1ia9y0RlUF"
   },
   "outputs": [],
   "source": [
    "class TwoLayerMLP(object):\n",
    "    def __init__(self, input_size, hidden_size, num_classes, std=1e-1, activation='sigmoid'):\n",
    "        np.random.seed(0)\n",
    "        self.input_size = input_size\n",
    "        self.num_classes = num_classes\n",
    "        \n",
    "        self.params = {}\n",
    "        self.params['W1'] = std * np.random.normal(size=(input_size, hidden_size))\n",
    "        self.params['W2'] = std * np.random.normal(size=(hidden_size, num_classes))\n",
    "        self.params['b1'] = np.zeros(hidden_size)\n",
    "        self.params['b2']  = np.zeros(num_classes)\n",
    "\n",
    "        self.activation = 'sigmoid' \n",
    "\n",
    "        self.models = [\n",
    "                  Linear(self.params['W1']),\n",
    "                  Bias(self.params['b1']),\n",
    "                  Sigmoid(),\n",
    "                  Linear(self.params['W2']),\n",
    "                  Bias(self.params['b2']),\n",
    "                  SoftMax()\n",
    "                ]    \n",
    "           \n",
    "    def loss(self, X, y=None, reg=0.0):\n",
    "        # Unpack variables from the params dictionary\n",
    "        W1, b1 = self.params['W1'], self.params['b1']\n",
    "        W2, b2 = self.params['W2'], self.params['b2']\n",
    "        _, C = W2.shape\n",
    "        N, D = X.shape\n",
    "\n",
    "        ###########################################################################\n",
    "        # TODO: Finish the forward pass, and compute the loss. This should be the \n",
    "        # data loss. Store the result in the variable loss, which should be a scalar. \n",
    "        # Use the CrossEntropy loss. So that your results match ours.\n",
    "        ###########################################################################  \n",
    "        ## -- ! code required  \n",
    "              \n",
    "        ###########################################################################\n",
    "        #                            END OF YOUR CODE\n",
    "        ###########################################################################\n",
    "\n",
    "\n",
    "        grads = {}\n",
    "        ###########################################################################\n",
    "        # TODO: Compute the backward pass, computing the derivatives of the weights\n",
    "        # and biases. Store the results in the grads dictionary. For example,\n",
    "        # grads['W1'] should store the gradient on W1, and be a matrix of same size\n",
    "        ###########################################################################\n",
    "        ## -- ! code required  \n",
    "        \n",
    "        ###########################################################################\n",
    "        #                            END OF YOUR CODE\n",
    "        ###########################################################################\n",
    "        return loss, grads\n",
    "\n",
    "    def backward_layers(self, downstream_grad):\n",
    "        '''runs a backward pass on all the layers.\n",
    "        after this function is finished, look at layer.grad to find the\n",
    "        gradient with respect to that layer's parameter.'''\n",
    "        for layer in reversed(self.models):\n",
    "            downstream_grad = layer.backward(downstream_grad)\n",
    "\n",
    "    def train(self, X, y, X_val, y_val,\n",
    "            learning_rate=1e-3, learning_rate_decay=0.95,\n",
    "            reg=1e-5, num_epochs=10,\n",
    "            batch_size=200, verbose=False):\n",
    "\n",
    "        num_train = X.shape[0]\n",
    "        iterations_per_epoch = 1 #int(max(num_train / batch_size, 1))\n",
    "        epoch_num = 0\n",
    "\n",
    "        # Use SGD to optimize the parameters in self.model\n",
    "        loss_history = []\n",
    "        grad_magnitude_history = []\n",
    "        train_acc_history = []\n",
    "        val_acc_history = []\n",
    "\n",
    "        np.random.seed(1)\n",
    "        for epoch in range(num_epochs):\n",
    "            # fixed permutation (within this epoch) of training data\n",
    "            perm = np.random.permutation(num_train)\n",
    "\n",
    "            # go through minibatches\n",
    "            for it in range(iterations_per_epoch):\n",
    "                X_batch = None\n",
    "                y_batch = None\n",
    "\n",
    "                # Create a random minibatch\n",
    "                idx = perm[it*batch_size:(it+1)*batch_size]\n",
    "                X_batch = X[idx, :]\n",
    "                y_batch = y[idx]\n",
    "\n",
    "                # Compute loss and gradients using the current minibatch\n",
    "                loss, grads = self.loss(X_batch, y=y_batch, reg=reg)\n",
    "                loss_history.append(loss)\n",
    "\n",
    "                # do gradient descent\n",
    "                for param in self.params:\n",
    "                    self.params[param] -= grads[param] * learning_rate\n",
    "\n",
    "                # record gradient magnitude (Frobenius) for W1\n",
    "                grad_magnitude_history.append(np.linalg.norm(grads['W1']))\n",
    "\n",
    "            # Every epoch, check train and val accuracy and decay learning rate.\n",
    "            # Check accuracy\n",
    "            train_acc = (self.predict(X_batch) == y_batch).mean()\n",
    "            val_acc = (self.predict(X_val) == y_val).mean()\n",
    "            train_acc_history.append(train_acc)\n",
    "            val_acc_history.append(val_acc)\n",
    "            if verbose:\n",
    "                print ('Epoch %d: loss %f, train_acc %f, val_acc %f'%(\n",
    "                    epoch+1, loss, train_acc, val_acc))\n",
    "\n",
    "            # Decay learning rate\n",
    "            learning_rate *= learning_rate_decay\n",
    "\n",
    "        return {\n",
    "          'loss_history': loss_history,\n",
    "          'grad_magnitude_history': grad_magnitude_history, \n",
    "          'train_acc_history': train_acc_history,\n",
    "          'val_acc_history': val_acc_history,\n",
    "        }\n",
    "\n",
    "    def predict(self, X):        \n",
    "        ###########################################################################\n",
    "        # TODO: Implement this function; it should be VERY simple!\n",
    "        ###########################################################################\n",
    "        ## -- ! code required  \n",
    "        \n",
    "        ###########################################################################\n",
    "        #                              END OF YOUR CODE\n",
    "        ###########################################################################\n",
    "        return y_pred\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HSkRivC0soqb"
   },
   "source": [
    "### Q3.1 Forward pass\n",
    "\n",
    "Our 2-layer MLP uses a softmax output layer and the multiclass cross-entropy loss to perform classification. Both are defined in Problem 2.\n",
    "\n",
    "Please take a look at method `TwoLayerMLP.loss`. This function takes in the data and weight parameters, and computes the class scores (or the logits $z_k(x,\\theta)$), the loss ($L$), and the gradients on the parameters. \n",
    "\n",
    "- Use the layers designed in **Problem 2** and implement the first part of the function to compute `scores` and `loss`. Afterwards, run the following two test cases.\n",
    "\n",
    "Note 1: It should be VERY simple to compute the forward propagation by calling the **forward_layers**().\n",
    "\n",
    "Note 2: **If you're not careful, you could run into numerical underflow/overflow problems with softmax and cross-entropy.** In particular, it involves the [log-sum-exp operation](https://en.wikipedia.org/wiki/LogSumExp) where exponentiated numbers are summed. This can result in underflow/overflow, e.g. getting \"nan\" (stands for Not A Number) for seemingly ordinary numerical operations. Read about the solution in the link.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vGCgTsMXDuB-"
   },
   "outputs": [],
   "source": [
    "input_size = 4\n",
    "hidden_size = 10\n",
    "num_classes = 3\n",
    "\n",
    "net = TwoLayerMLP(input_size, hidden_size, num_classes)\n",
    "scores = forward_layers(net.models, X)\n",
    "print ('(1) Your scores:\\n')\n",
    "print (scores)\n",
    "print ('\\n')\n",
    "correct_scores = np.asarray([[0.374083, 0.312271,  0.313644],\n",
    "                            [0.378308, 0.311404, 0.310286],\n",
    "                            [0.392588, 0.306650, 0.300761],\n",
    "                            [0.391516, 0.310674, 0.297809],\n",
    "                            [0.389314, 0.313829, 0.296856]])\n",
    "\n",
    "# The difference should be very small (< 1e-7)\n",
    "print ('Difference between your scores and correct scores:')\n",
    "print (np.sum(np.abs(scores - correct_scores)))\n",
    "print ('\\n')\n",
    "\n",
    "loss, _ = net.loss(X, y, reg=0.1)\n",
    "correct_loss = 1.1443\n",
    "\n",
    "# The difference should be very small (< 1e-7)\n",
    "print ('(2) Your loss: %f'%(loss))\n",
    "print ('Difference between your loss and correct loss:')\n",
    "print (np.sum(np.abs(loss - correct_loss)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n9cchDnkEX6I"
   },
   "source": [
    "## **Q3.2** Backward pass\n",
    "- Implement the second part to compute gradient of the loss with respect to the variables `W1`, `b1`, `W2`, and `b2`, stored in `grads`. \n",
    "\n",
    "Hint: you can quickly get the gradients with respect to parameters by calling **self.backward_layers**(downstream_grad).\n",
    "\n",
    "Now debug your backward pass using a numeric gradient check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "b4QOW0VpGohH"
   },
   "outputs": [],
   "source": [
    "# Use numeric gradient checking to check your implementation of the backward pass.\n",
    "# If your implementation is correct, the difference between the numeric and\n",
    "# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.\n",
    "\n",
    "def eval_numerical_gradient(f, x, verbose=True, h=0.00001):\n",
    "  \"\"\" \n",
    "  a naive implementation of numerical gradient of f at x \n",
    "  - f should be a function that takes a single argument\n",
    "  - x is the point (numpy array) to evaluate the gradient at\n",
    "  \"\"\" \n",
    "\n",
    "  fx = f(x) # evaluate function value at original point\n",
    "  grad = np.zeros_like(x)\n",
    "  # iterate over all indexes in x\n",
    "  it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])\n",
    "  while not it.finished:\n",
    "\n",
    "    # evaluate function at x+h\n",
    "    ix = it.multi_index\n",
    "    oldval = x[ix]\n",
    "    x[ix] = oldval + h # increment by h\n",
    "    fxph = f(x) # evalute f(x + h)\n",
    "    x[ix] = oldval - h\n",
    "    fxmh = f(x) # evaluate f(x - h)\n",
    "    x[ix] = oldval # restore\n",
    "\n",
    "    # compute the partial derivative with centered formula\n",
    "    grad[ix] = (fxph - fxmh) / (2 * h) # the slope\n",
    "    if verbose:\n",
    "      print (ix, grad[ix])\n",
    "    it.iternext() # step to next dimension\n",
    "\n",
    "  return grad\n",
    "\n",
    "loss, grads = net.loss(X, y, reg=0.1)\n",
    "\n",
    "# these should all be very small\n",
    "for param_name in grads:\n",
    "    f = lambda W: net.loss(X, y, reg=0.1)[0]\n",
    "    param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False)\n",
    "    print ('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4So3UarqGLbl"
   },
   "source": [
    "## **Q3.3** Train the Sigmoid network\n",
    "To train the network we will use stochastic gradient descent (SGD), implemented in `TwoLayerNet.train`. Then we train a two-layer network on toy data.\n",
    "\n",
    "- Implement the prediction function `TwoLayerNet.predict`, which is called during training to keep track of training and validation accuracy.\n",
    "\n",
    "You should get the final training loss around 0.7, which is good, but not too great for such a toy problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rQW7gA0q2vru"
   },
   "outputs": [],
   "source": [
    "net = TwoLayerMLP(input_size, hidden_size, num_classes)\n",
    "scores = forward_layers(net.models, X)\n",
    "\n",
    "stats = net.train(X, y, X, y,learning_rate=0.5, reg=1e-5, num_epochs=200, verbose=False)\n",
    "print ('Final training loss: ', stats['loss_history'][-1])\n",
    "\n",
    "plt.plot(stats['loss_history'])\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('training loss')\n",
    "plt.title('Training Loss history')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NyQVjefVIwcQ"
   },
   "source": [
    "# Problem 4: Pytorch Intro (25 points)\n",
    "## **Q4.0**: Pytorch tutorials\n",
    "This homework will introduce you to [PyTorch](https://pytorch.org), currently the fastest growing deep learning library, and the one we will use in this course.\n",
    "\n",
    "Before starting the homework, please go over these introductory tutorials on the PyTorch webpage:\n",
    "\n",
    "*   [60-minute Blitz](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p9GI4YuDL-8D"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cxw9G9oyI67G"
   },
   "source": [
    "The `torch.Tensor` class is the basic building block in PyTorch and is used to hold data and parameters. The `autograd` package provides automatic differentiation for all operations on Tensors. After reading about Autograd in the tutorials above,  we will implement a few simple examples of what Autograd can do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0OHhRhYMJeYq"
   },
   "source": [
    "## **Q4.1**. Simple function\n",
    " Use `autograd` to do backpropagation on the simple function we saw in lecture, $f=(x+y)*z$. \n",
    "\n",
    "**Q4.1.1** Create the three inputs with values $x=-2$, $y=5$ and $z=-4$ as tensors and set `requires_grad=True` to track computation on them.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z3IehHMZ1-gY"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4zcdXFCzJ23X"
   },
   "source": [
    "**Q4.1.2** Compute the $q=x+y$ and $f=q \\times z$ functions, creating tensors for them in the process. Print out $q,f$, then run `f.backward(retain_graph=True)`, to compute the gradients w.r.t. $x,y,z$. The `retain_graph` attribute tells autograd to keep the computation graph around after backward pass as opposed deleting it (freeing some memory). Print the gradients. Note that the gradient for $q$ will be `None` since it is an intermediate node, even though `requires_grad` for it is automatically set to `True`. To access gradients for intermediate nodes in PyTorch you can use hooks as mentioned in [this answer](https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/2). Compute the values by hand (or check the slides) to verify your solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HFGohjfLJxFV"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7Cx02VhLJ9E6"
   },
   "source": [
    "**Q4.1.3** If we now run `backward()` again, it will add the gradients to their previous values. Try it by running the above cell multiple times. This is useful in some cases, but if we just wanted to re-compute the gradients again, we need to zero them first, then run `backward()`. Add this step, then try running the  backward function multiple times to make sure the answer is the same each time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_gqSSmRNJxQ3"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iZtPvY8IKGkO"
   },
   "source": [
    "## **Q4.2** Neuron\n",
    " Implement the function corresponding to one neuron (logistic regression unit) that we saw in the lecture and compute the gradient w.r.t. $x$ and $w$. The function is $f=\\sigma(w^Tx)$ where $\\sigma()$ is the sigmoid function. Initialize $x=[-1, -2, 1]$ and the weights to $w=[2, -3, -3]$ where $w_3$ is the bias. Print out the gradients and double check their values by hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ssa0Mqt7JxpQ"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QzykKC_TKPZr"
   },
   "source": [
    "## **Q4.3**. torch.nn\n",
    " We will now implement the same neuron function $f$ with the same variable values as in Q4.2, but using the `Linear` class from `torch.nn`, followed by the [Sigmoid](https://pytorch.org/docs/stable/nn.html#torch.nn.Sigmoid) class. In general, many useful functions are already implemented for us in this package. Compute the gradients $\\partial f/\\partial w$ by running `backward()` and print them out (they will be stored in the Linear variable, e.g. in `.weight.grad`.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c7ppyJIGJxl-"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xocHX3eHKUid"
   },
   "source": [
    "## **Q4.4** Module\n",
    " Now lets put these two functions (Linear and Sigmoid) together into a \"module\". Read the [Neural Networks tutorial](https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html) if you have not already.\n",
    "\n",
    "**Q4.4.1** Make a subclass of the `Module` class, called `Neuron`. Set variables to the same values as above. You will need to define the `__init__` and `forward`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XAEdoDGUJxiw"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "erHyJz6GKYi3"
   },
   "source": [
    "**Q4.4.2** Now create a  variable of your `Neuron` class called `my_neuron` and run backpropagation on it. Print out the gradients again. Make sure you zero out the gradients first, by calling `.zero_grad()` function of the parent class. Even if you will not re-compute the backprop, it is good practice to do this every time to avoid accumulating gradient!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PnVGQjtbJxbA"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "But_2wecKkJL"
   },
   "source": [
    "## **Q4.5**. Loss and SGD\n",
    " Now, lets train our neuron on some data. The code below creates a toy dataset containing a few inputs $x$ and outputs $y$ (a binary 0/1 label), as well as a function that plots the data and current solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JwRpJrqEKdrR"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# create some toy 2-D datapoints with binary (0/1) labels\n",
    "x = torch.tensor([[1.2, 1], [0.2, 1.4], [0.5, 0.5], \n",
    "                  [-1.5, -1.3], [0.2, -1.4], [-0.7, -0.5]])\n",
    "y = torch.tensor([0, 0, 0, 1, 1, 1 ])\n",
    "\n",
    "def plot_soln(x, y, params):\n",
    "  plt.plot(x[y==1,0], x[y==1,1], 'r+')\n",
    "  plt.plot(x[y==0,0], x[y==0,1], 'b.')\n",
    "  plt.grid(True)\n",
    "  plt.axis([-2, 2, -2, 2])\n",
    "  \n",
    "  # NOTE : This may depend on how you implement Neuron.\n",
    "  #   Change accordingly\n",
    "  w0 = params[0][0][0].item()\n",
    "  w1 = params[0][0][1].item()\n",
    "  bias = params[1][0].item()\n",
    "  \n",
    "  print(\"w0 =\", w0, \"w1 =\", w1, \"bias =\", bias)\n",
    "  dbx = torch.tensor([-2, 2])\n",
    "  dby = -(1/w1)*(w0*dbx + bias)  # plot the line corresponding to the weights and bias\n",
    "  plt.plot(dbx, dby)\n",
    "\n",
    "params = list(my_neuron.parameters())\n",
    "plot_soln(x, y, params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fAbcItAKKvTi"
   },
   "source": [
    "**Q4.5.1** Declare an object `criterion` of type `nn.CrossEntropyLoss`. Note that this can be called as a function on two tensors, one representing the network outputs and the other, the targets that the network is being trained to predict, to return the loss. Print the value of the loss on the dataset using the initial weights and bias defined above in Q4.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DVUAaX_9Kd3O"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6BD6wtblKwef"
   },
   "source": [
    "**Q4.5.2** Print out the chain of `grad_fn` functions backwards starting from `loss.grad_fn`  to demonstrate what backpropagation will be run on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J35nnDwSKeEw"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u6Yd5zbIKxMp"
   },
   "source": [
    "**Q4.5.3** Run the Stochastic Gradient Descent (SGD) optimizer from the `torch.optim` package to train your classifier on the toy dataset. Use the entire dataset in each batch. Use a learning rate of $0.01$ (no other hyperparameters). You will need to write a training loop that uses the `.step()` function of the optimizer. Plot the solution and print the loss after 10000 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ijxmOyUUKvrU"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ed2aFhiYK-vb"
   },
   "source": [
    "**Q4.5.4** How many thousands of iterations does it take (approximately) until the neuron learns to classify the data correctly?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0tuMoY6JLG_C"
   },
   "source": [
    "***Solution***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7MdGRt1jOmBX"
   },
   "source": [
    "## **Q4.6**. Hidden space ablation\n",
    "\n",
    "Now let's look at the size of network's hidden space. We will create and train a **2-layer MLP** network on the [SVHN Dataset](http://ufldl.stanford.edu/housenumbers/).\n",
    "\n",
    "The SVHN dataset consists of photos of house numbers, collected automatically using Google's Street View. Recognizing multi-digit numbers in photographs captured at street level is an important component of modern-day map making. Google’s Street View imagery contains hundreds of millions of geo-located 360 degree panoramic images. The ability to automatically transcribe an address number from a geo-located patch of pixels and associate the transcribed number with a known street address helps pinpoint, with a high degree of accuracy, the location of the building it represents. Below are example images from the dataset. Note that for this dataset, each image (32x32 pixels) has been cropped around a single number in its center, which is the number we want to classify.\n",
    "\n",
    "![SVHN images](http://ufldl.stanford.edu/housenumbers/32x32eg.png)\n",
    "\n",
    "In this problem, we turn the input images into grayscale and then flat them into 1-D vector. First, download the SVHN dataset using `torchvision` and display the images in the first batch. Take a look at the [Training a Classifier](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) tutorial for an example. Follow the settings used there, such as the normalization, batch size of 4 for the `torch.utils.data.DataLoader`, etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 278,
     "referenced_widgets": [
      "0d7b69ad6f064b0ca3d8c53c50667f1f",
      "d5460ae85b3d4a5bbd931985f6ea08ab",
      "f2ee0013a8c048919e6eb5888e52290e",
      "5d9248be301d411b8c47ab2b44e0e0d8",
      "3a7c43d85ba44d8cad9f6cd95e890cea",
      "49a42be81f124e2fb3af4b4d9817a00d",
      "9c4b9044203c4f528afd5c2ad2facac8",
      "fde7de6f43f34105b166e3bbe391d860"
     ]
    },
    "colab_type": "code",
    "id": "dtdl8h1eY2dZ",
    "outputId": "dc0626e4-9dd1-4029-cf46-c47f42cbfb41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://ufldl.stanford.edu/housenumbers/train_32x32.mat to ./data/train_32x32.mat\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d7b69ad6f064b0ca3d8c53c50667f1f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    4     8     4     1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAACyCAYAAACN8fHlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dfaxlV33e8eeHX8A2zlzPeDwMHoOnKsSioY0jC4hSVREElbQo5o8oBdLWSamsSq1K2lSNSf6okFqJqlXSVmmpUKA4EoIgQosVhbYWxUojFYLBKQFcgksMY8+LZ8Yev/BiMKz+cY+be5575j7nd/edO+c6348U4XXPy1577XX22Tn7md+qMYYAAACwvOdd7A4AAADsNVxAAQAANHEBBQAA0MQFFAAAQBMXUAAAAE1cQAEAADRNuoCqqjdU1Zer6oGqumOnOgUAALDKart1oKrqEkl/LOn1kh6S9BlJbxljfOl8r7nyyivH2tratrYHAACwm06cOHFmjHFw0WOXTnjfV0l6YIzxVUmqqg9JulXSeS+g1tbWdPvtt0/YJAAAwO545zvf+bXzPTblFt71ko5taD80+9ucqrq9qu6tqnu/+c1vTtgcAADAarjgIfIxxnvGGLeMMW658sorL/TmAAAALrgpF1APS7phQ/vI7G8AAADPaVMuoD4j6WVVdbSqLpf0Zkl37Uy3AAAAVte2Q+RjjGeq6h9I+m+SLpH0vjHGF3esZwAAACtqyr/C0xjjdyX97g71BQAAYE+YdAG10+6777659ve+97259ne/+9259ne+85259re//e259tNPP71l29/Pt/f9739/Ux+9btai52x0ySWXzLWf97z5u6ZVtWX70ku3PkTeHx8T75+/f+qfS693vn3vr7ff+ta3bvl+kvTqV796y/dIfB98n30e+D4+88wzW77en+/zzH3jG9+Ya/sxvOyyy+baPqa+ff/HGv7485///Ln2Qw89tKlPDzzwwFw7jbHvo/+LW9+nNG/9s+puuummLR8/dOjQltvzth8z/9xdfvnlc+0XvOAFc+0rrrhiy8f99Ys+Zz4vd1r3c9J9fjoXJmn/0/v76++55564zWPHjs21/Tj5Z8U/i6kPPo98nvn7+ePe9jHwtp+7nnjiibn2k08+OddO54bUTud/78+i1/g++Pk1Pe7b8DG96qqrtnzczz0dLOUCAADQxAUUAABAExdQAAAATSuVgUp5oNTu5nd2Qvc9U4YoZTH89SkX4PeLUxakmzNIY55s55ikeZCyG+n5vk/p8ZRLeOqpp+bannk6e/bsXNuzfD4n9u3bN9f2/I3vX8oxLMoAHD9+fK6dsnSeQ/B55zxr4vvoj0/JKSxj6rmmm21cNO/TGLup2b+p0rlk6ucwba/7+kW6ecbuvPDPhT8/5XmSlBM+c+bMXPvcuXNzbc8D+bnE80O+lm3K0C6Ti/O8pOe0UnbZvfCFL2z1cQp+gQIAAGjiAgoAAKCJCygAAICmlcpAdbMn3dzBTuhuw/uY6st4O9Ud8XvMfn841c5KNTfSMUg1ky7EMZp6XLuvT9kLHwPPOD3yyCNzbc88+eMpA+X3+D2XsH///rn2gQMH5topVyflz6K/JtXP8dpU3meXchFJOsbdOZCyMd3szCLdDNFOZ5C62UJvp9xbt7/d/m9Hyu75vJ6ahUs5N2/7+dvHxB/3c8ejjz461z59+vRc2/fPzy3eX/9cpzpXi46Rj7nnsk6dOjXX9lpW6TvHzy0pd+Y5rw5+gQIAAGjiAgoAAKCJCygAAICmlcpAdW2n7kfn/Ra9f7cOSFr7yLMh3k734FMdkG9961tbvj7lGNI9fr8H7m3fX29fiAxU2sdu/RgfY99HHzO/h+81lfye/+OPP77l+/mYeR7IMwJ+zD238AM/8ANz7UV5pKNHj861PQuR1vDyeeJrw3nWwvfZc2Hd+jh+jFK9s27eJn2uU62ZRdtLNXTSPvgYprU+0z74GKZ6Zy6dW9L6h97/bv5oJ3RzWt3zWZrXKaPln3XPX/q5wR/3MfZj6t9Haa3VZcbDj7Of/x577LG5dso/+rz1c4332fOVZKAAAAB2ERdQAAAATVxAAQAANK1UBmpqnZCpmahl7qF3M09pjS+//3r11Vdv+fxUV8Tv9/r94W4WxPfP369b/yfVEVlGqlGU9sH5GExdC8/Xn0pr3XnOIdWS8ed7hsDH1Nfi8wyCZ6Ik6cYbb9z0t408Z5DGLH0OulmOC22nazIts85dyualvKMfV2+nc4MfU2+77j779n1eprxO+pz4HFtGd50036eUj+yeq1w3s+pjnOr+Oe+vz6FuXcBF89zPf55x8j53c2bpGPr2p+AXKAAAgCYuoAAAAJq4gAIAAGja0xmo9HqXahgtUxumm4FK27ziiivm2p6JSlmTtBaeby+tlef77K/3/nh/ve0ZKN/ftNbfMrrr66V76qmdsh1+j93baczTelL+uGcIPBPl62H52njXXHONnD/H++DzoLtuWspxeSbKx8yzdK5bo6ibn+zWkdpOvaA0zzyf4pkhH0N/vefQUk2iVMMtHeOUffH6P14vzffXt+/nqkXZPufzKH3WU+bJH0/Zt6n5TO9/Nyvoc8L7lx5P4+M5N0k6efLkXDvVwUsZ0JQ7dj6PpuAXKAAAgCYuoAAAAJq4gAIAAGhaqQxUyve4lFXxe6Epp5C2t8w2U+bJcwep9opnhtIYpXXafPuea/D38+d7psnXNPM6Vv54WutvJ3TXykuP+5h6dsPv6fsY+jz0e/g+Jp5JSpmntCba6dOn59ovfelLt3x/KX9WUj0Z52PqeZ1jx47NtX0tPM8t+Lxy3UxTd06kWl2pP4vONSlf4mPumSLPm3g7ZaDSucSl9f9SzSI/pimz5Tm47dR9cqnP3fX/unWWUt5xap2o9H2U+ptybL6unI/PogxUyupNXfcyfe93338r/AIFAADQxAUUAABAExdQAAAATSuVgfJ7l6mGxtTMU8oTLVMHKr2n5ww80+SPp7XlumsPpcyTP+777M/3/vj+eJ7H214nKq3VdyF0187zxz2b4ff0ve6SP+71aXzdueuvv36u7fPaM1fO197zrInnGBblOrqfJefzMtWH8RpAnovweZRMzUB1M1HdDNQy/fPMT6oD5WPm+RQ/7p639M+iby9lTZz337ef6vGkdTW9v96/Zca4m3FKdY+655Lu2nLeTnPAM07pcz215l0an0XPSccp1Vb077C0HuFO4hcoAACAJi6gAAAAmriAAgAAaIoXUFX1vqp6pKq+sOFv+6vq7qr6yux/Ny+mBQAA8By1TIj8/ZJ+XdJvbvjbHZI+McZ4V1XdMWv/0k53LgU1u8HM7qKxi4Kh3RC5BzVTYUkPZacQuYcE08K03k4h8hRCT8HSdMzSPxTYCWkxyu7iwh7c9EB0WvTUQ+RHjhyZa99www1z7RRs9cU4FxWv2ygFU6XN8y4VRXTeZy/+efz48bm2F/v0Ph08eHDL7bmp/xhhp0PiyxTS9H1OofG0aLUHilNo2889foxT4Nrbqeii98fPbf458SK9qSjvgw8+qCQVWUznDu9zCixv5ztnozSPPLifPqfp+6FbHDWFzKX+GHb7lD4X3scp4i9QY4zfk/So/flWSXfO/vtOSW/asR4BAACsuO1moA6NMU7M/vukpEPne2JV3V5V91bVvf7/gQAAAOxFk0PkY/33s/P+Xj7GeM8Y45Yxxi3dWi4AAACraLuFNE9V1eExxomqOizpkfiKFbAT+Zru/Vq/aPT7+N3Cmumeud8DT8VGU/G5bpGylCnw3IPfQ/fx2And4napmFxa5DUtyOzFRNMCzP7+ng3xdsqypAKHi/rs0pg6z2WdOHFiru3FR3376Zi5brG+7vu7NO+XyZCljJNnmlLhzLQ4r++zz4u0D2mM/PUpL5Pyo2tra3Ntn/fbyUB1C2mmTGfK60xt+/Z9DD2j6lL+KH0/+PikObvMYsKuu6BzynGl3O8U2/0F6i5Jt83++zZJH9uZ7gAAAKy+ZcoYfFDS/5L0g1X1UFW9TdK7JL2+qr4i6SdmbQAAgD8T4i28McZbzvPQ63a4LwAAAHvCSi0mnO51Tq3t0r2Hv8xiwn5/NdV98ryL19tJi/1261Cle9y+PX//lHnq1glJGYMLkYFyaUHNZWr2bDS1Vkzavh8DP2bd3IRbNM+7GSPvU8q/eHaiu2Bz+gcpO11PrFuvx20nA5UyTz4mKQPl2/RzSzrfpqxg2kffv5SB8sc9o5XynstI+9ytU5eygCkz5Z91H7Pu4sXdzGua56nulM9Jr1EnbT6uU2us+fv5GKVc8RQs5QIAANDEBRQAAEATF1AAAABNK52B6t6DT9mSlGVZprZNyjx5hscf97bfj001LFIGy3XHzO+Rp+en/qUszHZybem+fLfuU3r/VEck1fZKc6a7nmDKhqT1vFJtMmnzmHbX/HK+j17DJ61X1c1AdXNmKfsxdb2u9DmQ8tp3KROVxnBqzaOUVfH38332eer71828+pzYTrYl1TlyO52X7K4vmN7fpfN7t6aSv1+qibdonqdcWXcfu2ud+vl3Cn6BAgAAaOICCgAAoIkLKAAAgKaVykAl6Z68369N91q7+R5pc92mlIHy56caPam2StJ9fhqD9Px0z3yn7+lL+T683wN3aYy8ppHnBjwzdOjQobn28ePH59qeTfH+eTvNW89+pHo/PgdT7TFpc34m1aZyfow883TkyJG5tq8P6GtoeV4mSRmkbjaw2+7W0ZI2j3nKuqU8YTqfpfo4KZvn++DzPGWy/PGUkfLnp3poPqcW6WbX0pqH3sc0z/z1Puap9lX33OHS90/6PvJj6ueidC5e9J5pm90x9TFL6wV28AsUAABAExdQAAAATVxAAQAANK1UBqp7/zmty+P3Z1NmwO8/e1ZE2ry2ndejSXWe/H6t3+dPOQe/fzt13bR0T9/3x8fEt5dyFGn/ltG9b++m1gvznNs111yzZfvUqVNz7SeeeGKufeLEiS2352P62GOPbfl+nsPw1/v6VGfOnNm0zbW1tS37lOadt/39fF56jswzUN7nRx99dMv+uTTPXbcmUZpDy2T/Up6lm9tKWb6U10znDuef5bQ/Lq2l5/kaX/vPX79MBsqljGo6zt2s3dT+pVybS/M+HbO0vmGq7SXlDFLKPKX1BFMmaifxCxQAAEATF1AAAABNXEABAAA0rVQGqluDwu/HpgxUWiPMMwCL1tvyDJTfZ09ZEN8Hz6v4PWPfB88k+f3kVKcprTvkY+Tv77kJv+eeMmGprsmFkMbE51GqpeL75PmeF73oRXNtz+94Hadjx47NtT3T5NvzfJDPqZQxOHfu3FzbM1WSdO211275Hj5mKdvh8yTlc3ze+5ikDFSa1936Pikbkh5fJvu302s0pnxid13OdD5O+Us/v/q5JK3D6f3ZiXNJt77Y1FqErrumYsrxdutaOT9m6Ts0ZZ4W5eZSlq47Zuk7MOWM/XzdwS9QAAAATVxAAQAANHEBBQAA0LRSGajuelNpHTa/N5qe7/fkPSMg5ZyBt9N6UX5/NuUguuvAdddOSnzMPAPm68SlDFSq1bVIqn+T9inlFlJWzp+f1srzOeP1bKau8eXz1ted8+15hurJJ5+US2Ocshjd2iv+es/L7Nu3r/V+3ZpEKRuY9s8f93nux8BzcFJeOy5l3dI+TF2/z5/v894fT2v7Oe+/f258TqS19ZaRxiid31Odv5QhSueqReuxbvX+6f26c8LH1PcvZSEX9T/1IdXaStk8l9aUnIJfoAAAAJq4gAIAAGjiAgoAAKDpOZWBSvejnWcIvIaRZ0ukzffhvZ1yCKkejN//9X3210/NQLk05mm9wFRbxqXaXouk49pdCy/1yaUx9zpLnnPzPM9111031/YMk9ce279//1zbPwfHjx+faz/44INz7YcffnjL/kqbMztpnvu8SPPa1zHz1y+qwdaRjmF6fsphpOyHZ0d8f318pf5n1cfM50Hah1TXqZuJ8jniWT0fs1QbzM8l3vYx9LavQbnIovVOt5JqXblUqzCtLee69cxct7Zien3av0WZrJTrStvszsv0flPwCxQAAEATF1AAAABNXEABAAA07akMVKq/49K6QGmdt0V1oPw56R6639NO9WmcP57WbUv5HL9/nOr9pDFPY+xS/5YxtS7I1Fpbfgy8zsjp06e33N7hw4fn2kePHp1re/7Hx9znYKpj5dmQs2fPzrUXrSvnzzl06NBc28fYMz7+2UnHqDtPk5SzSHMg1X1KGS+fE2k9rkXbcN15nuqZdV+f2v5+fn71bJ9npDwT1a255+OzTAYq1ftK89Tb3Xylvz7Nm7RWqtcX69aJSrm5NM/T99MyffAx7OaKU74yHaMOfoECAABo4gIKAACgKV5AVdUNVfXJqvpSVX2xqt4++/v+qrq7qr4y+99rLnx3AQAALr5lMlDPSPrFMcbnqupqSZ+tqrsl/ZykT4wx3lVVd0i6Q9IvTelMyuOkfE66p5/Wykt1R6RcuyTVyUj74NL94G6NDJfeL90vTvfw/XG/P53WgVsk1crq1sPp1n3q5l38HnxaP9DnmG/f3z9l+dbW1ubantvzvJO0OT/itaicZy8OHjw41061rfxzkjJGSXdedz+XKcuX1mlbVAcqrRGWznfdeZyyfD7P0ufK560/38+naW2/tJZpmjPL8OOQxtT7lMY8rX2a6uylTFE6hqnuVDq/p2Oe1n5d9J3cXbMxfa/7mPpn0Y9Zqk3YEX+BGmOcGGN8bvbfT0q6X9L1km6VdOfsaXdKetOO9QoAAGCFtTJQVXWjpJslfVrSoTHGidlDJyUdOs9rbq+qe6vqXv/1AQAAYC9a+gKqql4o6bcl/cIY44mNj4313/UW/pY5xnjPGOOWMcYtU5dnAAAAWAVL1YGqqsu0fvH0gTHGR2d/PlVVh8cYJ6rqsKRHLlQnN/Rjrp1qEKXnp3vui+6Vplol3bWCEu+T33NO9+xTjmJqxsr3x+/5++uffPLJubb/KnnttdcqSXVE0j3zVDsr1Wrp1jDq1jhKGQGfU6lWV5pDi8bTj9Pjjz8+1/bj/tRTT216j408m5GyhP789P6uW+sr1YpJx8T76/PaM2KLfo1PxzWdr9I88WPm++xj1s2BdfM13Rp0F2KNs+7acem4+xinTGh3n7rtlBNLmdP0fZNyxMtI6+el82X67HbzjR3L/Cu8kvReSfePMX51w0N3Sbpt9t+3SfrYjvUKAABghS3zC9SPSfpbkv6oqv5w9rdflvQuSR+uqrdJ+pqkn7kwXQQAAFgt8QJqjPH7ks732+jrdrY7AAAAq2+l1sLrZlu6tV783me3LW2+35oyUGkNre56fakuR7oH7u+X1t/qrivn97P9/T3L4tmQZTJQ3T539yG10/b8GKZMlEvzNuXoUnbGH1/0ufPMk2eivO3r/505c2aufd111821Dxw4sGUf0/p9XWlOLPPZ3yhl/3xe+3g98cTcv8NZ+B7eh5S/TGuEeTvlzrrrF3brUvnzPT+T1h1dZn3BJGWAut8x3TUWu5nT7jqdU2tt+fedH4Pt1BH05/hnxT8HqU5Uynx6n3e1DhQAAADmcQEFAADQxAUUAABA00pnoHY6E5WyMMvUEUn3eFONilSTJ+Ue/PFujSK/v+z3zFM+J60r5zxXkdrLSPMkjalLOYbuOmwpp9CtDZOyLmkOdmsaLepj2ifP9Hgmyh9/9NFHN21zI8+3eIbI1/ebKs1rP0bpmHr/vb2oDpTvYzoX+Dz3PqYMlGeM/Bj7mo2eUfLn+z6mee5j7oWWff1EfzytP7gMz8d0a7al83s6l3RryKX1/lItr25eyNup5lI3Syhtnjfpe9i36etq+jzxeU4GCgAA4CLiAgoAAKCJCygAAICmlcpApZxBytsk3fvRi7bn96C7WZFUW8XvMafcQTeDlGq/uFSXxMcj1aGaegwX9SHlV9LaSOn9k5QzcGn9rFQbLO1fGp9l6vX4NlKNHu+z13Hyz9a5c+fm2j4GKQfRzUB1P/vdOZNq06QaTtLmz2bqQ9oHH9Nu3jHlW1KWzvOWPie8f54L82zM1VdfPddO56Jl+HFKY5TmgR/D7rqWqY5Tmlf+OfUx9ffz13t+yHNw3bpPabwWvUc6BqnOnZ+b0nfoFPwCBQAA0MQFFAAAQBMXUAAAAE0rlYFKmafuPe5UEynVD/J7+NLm+6cpu+HbSPe0u2srdaVsSfcYpHvgKQO2nZocKWOU8ihJN+fl++C5gZMnT861fT1Ab+/fv3+u7ff00xzzOZkyV4vmWMq7dDNRnsVIuYeUv0n8s57q7aTPZWqn3JrX0/HxkzZnfFKexPfJM0be9nmQ8i7+uPc5ZbZSDbqUkfI10ryWWDpXpnXupJzRTGuXpjp83Rpsad6n7wM/Rj5PU82llCfyOZHylIv6mzKf6Tsq1VLcjRzus/gFCgAAoIkLKAAAgCYuoAAAAJpWKgOV7qemfE66P+z3l1O2ZZGU1XD+nqlmUKo74s/3fU61X7rtVEcq7Z9L63kto5uHSbmEVE8n8X3y3IDzNc/OnDkz1/Y1wFKWw4+Rv7+vS5dqMEmbMzspC3H48OG5to+pr33nfU51knx7STpXpGyez5FuNsQf92N6zTXXbOqz51PS+cw/q56l8wyRZ448L7Nv374t277mmI+JZ6jSZztluHz/fH+6x2wRH/OUO0vnjm6mKa116m0fE/8sp++wlI/0Y+Lz+sCBA3Nt73/6/pA2Z9nSd0iqVZjWCn3sscfm2n5+vPHGG7d8/63wCxQAAEATF1AAAABNXEABAAA0rVQGqrvGWarvkNYAS/UnFmUQ0v3YlJHy+/LeTjV6Ut2kVGsltVONDt++Z2VSLmE7dZ9ct/aIz4u0nlOSarN4VsSzJp4HevDBB+faKTvi2/dj6JmqU6dOzbU99+H9lXL+xfMuPgaeAbruuuvm2otyVxv558KzGGfPnt3y9X7Mu3Wg0npbqd6O99f3d9H+p/NXOv95BsqzHj5P/LPox9hzW35MUx4y1cry1z/++ONzbc9EpWPUzUZK/QxSaqf3S5moVEvLj3mqnZXWak017/xc4dJ4LDqXpX3ybfo+pPpgqZ6j16Sbgl+gAAAAmriAAgAAaOICCgAAoIkLKAAAgKaVCpG7FKL0MFoKYKfwcHo/aXNILgU7U4jcQ9gp8JzCtx7A88CdB+j8+WnxzFRQMC0w6o9vJ/iZgpYpINwNnafFJ30fPIx77bXXzrWPHz8+1/aw79e//vW5th8jD6r64x4i97Y7ePDgpr95YUwPRfu89cKQ3sdUFDLNew+GdkPkSTo3pHCsz+u0/UUFeLtjlBZHd97HFCL3fyjgwfl0LvQx8jnj7+//cMHPXX7uS4HsNEcW9akb8k7zoluYMy3G7vxxD1ynxYK7RYXTd2ran0V9SOff9LlIxaPTuWUKfoECAABo4gIKAACgiQsoAACAppXKQKV7nc7vz6ZMQLqf7AW3Ft2/9b91izL6PfO0cK2/n9+z98e9sJoX1/PHfZ/TMUj96WYI0uLIy/QhSff1Uzvlwvz919bW5tqegTp58uRcOxUQ9NyaZ1f8cS8s5/P+6quvnmsfOnRIznNR3WxGKrDaXfi7m5VLi3y7lF1xPke6BWQXfe7TmHYXrvXMUspneh/TYrypv/45TUUm0+LGPmYpf7NMBiotROtj3l1cOLWTlBfyz1n3mHeLS6fM7zLFqLuv8c+mHzN/fSrM2T0GW+EXKAAAgCYuoAAAAJriBVRVvaCq/qCq/ndVfbGq3jn7+9Gq+nRVPVBVv1VV0xc5AwAA2AOWyUA9Lem1Y4ynquoySb9fVR+X9I8l/doY40NV9R8lvU3Su6d0JmWWUjalWw9iO/eru/ViUh2k9H6pDpQ/7vd7U54m1ZXyMXR+D951a9sss9hwGqNunadU/8vHIC326/V0XvziF8+1PZtx+vTpubZnmHzxYZ9Tfoy9vwcOHNiyP0eOHJHzLIXrjplLn5OU1UhSPmY79Ws26taqSdkSafOYdWvupExRN/sxNWvYHeP02d/pRcEXvSbtQ+pTmjepz+mYTq3Tl86FKQvo20+f80VZv5Q1TvXGPMPp++jnorQg8hTxF6ix7tkk8mWz/xuSXivpI7O/3ynpTRekhwAAACtmqQxUVV1SVX8o6RFJd0v6v5LOjTGevfx8SNL153nt7VV1b1Xd679+AAAA7EVLXUCNMb43xvhhSUckvUrSTctuYIzxnjHGLWOMW3w5CAAAgL2oVQdqjHGuqj4p6UclrVXVpbNfoY5IenhqZ7qZp24mqlsnajv1Irp1m9L9We9TylD5Wkh+vzmtfef7nOrvpHv0fgy87bmHReuyuZQb6NaW8n3wdqo7kmql7N+/f6598803z7V9bTyvE+W1vPyY+v76/6Nyww03zLVf8pKXzLV9HTtp8z54H1LOLM3TlCtL9XUutG79nqnrLy4yNffluvme9DlL75/er5tp2sk1zJ7V7bPr1sKamr1LGaa0vmva37TGY7LMHEt5xLRmo2ek0nqs586dm2t318ncyjL/Cu9gVa3N/vsKSa+XdL+kT0r66dnTbpP0sR3rFQAAwApb5heow5LurKpLtH7B9eExxu9U1Zckfaiq/rmk+yS99wL2EwAAYGXEC6gxxucl3bzg71/Veh4KAADgz5SVWgsvZVf88ZRdmbpW3k7cK/U+pVyWP57yLilX4Jkob6e1+FLdD297f71Gkbe3k4GamlXr1hjqrsPm/J68Z478cc9MpRpLfkx9//z9vC7UoppP/i9muzmyi51hSmv3dfNCFyLjlLbppmaEupmj7nqCKf+SauYly6yz1tXNtnXfr9v2Y5S+4/zc0f0O9LykZ3bTWqfJopp5fs73djo/e598H/x86M/3x6dgKRcAAIAmLqAAAACauIACAABoWqkMVKqJ0W1fiLoi3ZyCPz/dk04Zo25+J71/9x5/qtWVHvftLbP2XbcP3fWlptaX8dd7fijlGvwevtc1Sesp+vb8GPsY++sXZf3SPPPjmJ7fzRd2M0uu+1mfWudpFXTrOnUfd90x2426Tl1T85TddTZTnSaXMvtS/2kAAAYASURBVE9+7vB1OD3v4/276qqr5tq+zpznI1O2cDv1zTyjlHJffn7ct2/fln3wMZiaad2IX6AAAACauIACAABo4gIKAACgaaUyUK985SsvdhewB9xzzz0XuwtYcWfPnr3YXcAesGgdyFWSMlKeWfL2KvIc10033XRBt/fyl7/8gr03v0ABAAA0cQEFAADQxAUUAABAExdQAAAATVxAAQAANHEBBQAA0MQFFAAAQFPt5ppOVXVa0tckXSvpzK5t+LmJMZyOMZyG8ZuOMZyOMZyOMTy/l44xDi56YFcvoP7/RqvuHWPcsusbfg5hDKdjDKdh/KZjDKdjDKdjDLeHW3gAAABNXEABAAA0XawLqPdcpO0+lzCG0zGG0zB+0zGG0zGG0zGG23BRMlAAAAB7GbfwAAAAmriAAgAAaNrVC6iqekNVfbmqHqiqO3Zz23tVVd1QVZ+sqi9V1Rer6u2zv++vqrur6iuz/73mYvd11VXVJVV1X1X9zqx9tKo+PZuPv1VVl1/sPq6yqlqrqo9U1f+pqvur6keZhz1V9Y9mn+MvVNUHq+oFzMOtVdX7quqRqvrChr8tnHe17t/NxvLzVfUjF6/nq+M8Y/ivZp/lz1fVf66qtQ2PvWM2hl+uqr96cXq9+nbtAqqqLpH07yX9pKRXSHpLVb1it7a/hz0j6RfHGK+Q9BpJf382bndI+sQY42WSPjFrY2tvl3T/hva/lPRrY4w/L+kxSW+7KL3aO/6tpP86xrhJ0l/S+lgyD5dUVddL+oeSbhlj/JCkSyS9WczD5P2S3mB/O9+8+0lJL5v93+2S3r1LfVx179fmMbxb0g+NMf6ipD+W9A5Jmn2/vFnSX5i95j/Mvr9hdvMXqFdJemCM8dUxxnckfUjSrbu4/T1pjHFijPG52X8/qfUvreu1PnZ3zp52p6Q3XZwe7g1VdUTSX5f0G7N2SXqtpI/MnsIYbqGq9kn6K5LeK0ljjO+MMc6Jedh1qaQrqupSSVdKOiHm4ZbGGL8n6VH78/nm3a2SfnOs+5Sktao6vDs9XV2LxnCM8d/HGM/Mmp+SdGT237dK+tAY4+kxxp9IekDr398wu3kBdb2kYxvaD83+hiVV1Y2Sbpb0aUmHxhgnZg+dlHToInVrr/g3kv6ppO/P2gckndtwAmE+bu2opNOS/tPsNuhvVNVVYh4ubYzxsKR/LenrWr9welzSZ8U83I7zzTu+Z7bn70j6+Oy/GcMlESLfI6rqhZJ+W9IvjDGe2PjYWK9FQT2K86iqN0p6ZIzx2Yvdlz3sUkk/IundY4ybJX1DdruOebi1WU7nVq1fjL5Y0lXafFsFTcy7aarqV7QeFfnAxe7LXrObF1APS7phQ/vI7G8IquoyrV88fWCM8dHZn089+9P07H8fuVj92wN+TNJPVdWDWr91/Fqt53nWZrdSJOZj8pCkh8YYn561P6L1Cyrm4fJ+QtKfjDFOjzG+K+mjWp+bzMO+8807vmcaqurnJL1R0s+OPy0KyRguaTcvoD4j6WWzf3FyudZDanft4vb3pFlW572S7h9j/OqGh+6SdNvsv2+T9LHd7tteMcZ4xxjjyBjjRq3Pu/8xxvhZSZ+U9NOzpzGGWxhjnJR0rKp+cPan10n6kpiHHV+X9JqqunL2uX52DJmHfeebd3dJ+tuzf433GkmPb7jVhw2q6g1ajzX81BjjmxseukvSm6vq+VV1VOuB/D+4GH1cdbtaibyq/prWsyiXSHrfGONf7NrG96iq+suS/qekP9Kf5nd+Wes5qA9Leomkr0n6mTGGBy1hqurHJf2TMcYbq+rPaf0Xqf2S7pP0N8cYT1/M/q2yqvphrYfwL5f0VUk/r/X/J4x5uKSqeqekv6H1Wyb3Sfq7Ws+XMA/Po6o+KOnHJV0r6ZSkfybpv2jBvJtdmP661m+NflPSz48x7r0Y/V4l5xnDd0h6vqSzs6d9aozx92bP/xWt56Ke0Xps5OP+nmApFwAAgDZC5AAAAE1cQAEAADRxAQUAANDEBRQAAEATF1AAAABNXEABAAA0cQEFAADQ9P8AzK+8MdF7Sl8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PBDzvc_nLfGJ"
   },
   "source": [
    "### Q4.6.1 2-layer MLP \n",
    "\n",
    "Next, we will train a 2-layer MLP on the data. We have defined a simple 2-layer MLP for you with two fc layers and ReLU activation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dWu68j0AaYD1"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Neuron(nn.Module):\n",
    "    def __init__(self, hidden_size):\n",
    "        super(Neuron, self).__init__()\n",
    "        self.l1 = nn.Linear(1024, hidden_size)\n",
    "        self.l2 = nn.Linear(hidden_size, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 1024)\n",
    "        x = F.relu(self.l1(x))\n",
    "        x = self.l2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n7OMjOH0MXpR"
   },
   "source": [
    "You can check the number of parameters in the model by printing out the model summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "tyODwJ0rJYq1",
    "outputId": "9b010936-b762-470b-81f9-869595a82f11"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_summary\n",
      "\n",
      "Layer_name\t\t\t\t\t\t\tNumber of Parameters\n",
      "====================================================================================================\n",
      "\t\t\t\t\t\t\t\t\t\t\n",
      "\n",
      "Linear(in_features=1024, out_features=10, bias=True)\t\t\t10240\n",
      "\n",
      "Linear(in_features=10, out_features=10, bias=True)\t\t\t10\n",
      "====================================================================================================\n",
      "Total Params:10250\n"
     ]
    }
   ],
   "source": [
    "def model_summary(model):\n",
    "  print(\"model_summary\")\n",
    "  print()\n",
    "  print(\"Layer_name\"+\"\\t\"*7+\"Number of Parameters\")\n",
    "  print(\"=\"*100)\n",
    "  model_parameters = [layer for layer in model.parameters() if layer.requires_grad]\n",
    "  layer_name = [child for child in model.children()]\n",
    "  j = 0\n",
    "  total_params = 0\n",
    "  print(\"\\t\"*10)\n",
    "  for i in layer_name:\n",
    "    print()\n",
    "    param = 0\n",
    "    try:\n",
    "      bias = (i.bias is not None)\n",
    "    except:\n",
    "      bias = False  \n",
    "    if not bias:\n",
    "      param =model_parameters[j].numel()+model_parameters[j+1].numel()\n",
    "      j = j+2\n",
    "    else:\n",
    "      param =model_parameters[j].numel()\n",
    "      j = j+1\n",
    "    print(str(i)+\"\\t\"*3+str(param))\n",
    "    total_params+=param\n",
    "  print(\"=\"*100)\n",
    "  print(f\"Total Params:{total_params}\")       \n",
    "\n",
    "my_neuron = Neuron(10)\n",
    "model_summary(my_neuron)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LOqfDwW7MmCM"
   },
   "source": [
    "Instantiate the cross-entropy loss `criterion`, and an SGD optimizer from the `torch.optim` package with learning rate $.001$ and momentum $.9$. You may also want to enable GPU training using `torch.device()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QSGrluSuM0Th"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rnYmjlVnM-a0"
   },
   "source": [
    "### Q4.6.2 Training\n",
    "Complete the training loop that makes five full passes through the dataset (five epochs) using SGD. Your batch size should be 4 and hidden size is 10. \n",
    "*italicized text*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5JcsoLpkcA0N"
   },
   "outputs": [],
   "source": [
    "my_neuron = Neuron(10)\n",
    "\n",
    "# create your optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(my_neuron.parameters(), lr=0.001, momentum=0.9)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# num of epoch\n",
    "stats = []\n",
    "for epoch in range(5):\n",
    "  running_loss = 0.0\n",
    "  for i, data in enumerate(trainloader, 0):\n",
    "    inputs, labels = data\n",
    "    inputs, labels = inputs.to(device), labels.to(device)\n",
    "\n",
    "    ## -- ! code required  \n",
    "    \n",
    "\n",
    "    # print statistics\n",
    "    running_loss += loss.item()\n",
    "    if i % 2000 == 1999:    # print every 2000 mini-batches\n",
    "        print('[%d, %5d] loss: %.3f' %\n",
    "              (epoch + 1, i + 1, running_loss / 2000))\n",
    "        stats.append(running_loss / 2000)\n",
    "        running_loss = 0.0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yCnLhyMWNdU9"
   },
   "source": [
    "Train the model again but this time set the hidden size as 100. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yZ2AEnXYdVgu"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4cGSgfUiNn63"
   },
   "source": [
    "### Q4.6.3\n",
    "\n",
    "Compare the performance when hidden_size=10 with hidden_size=100, and explain the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t8aVy24ddj9B"
   },
   "outputs": [],
   "source": [
    "# solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZXCDFdORONV2"
   },
   "source": [
    "***[ double click to enter solution ]***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1VLcR3G_f-xC"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ysPlY-XIgLOj"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "HW2_v2.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0d7b69ad6f064b0ca3d8c53c50667f1f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f2ee0013a8c048919e6eb5888e52290e",
       "IPY_MODEL_5d9248be301d411b8c47ab2b44e0e0d8"
      ],
      "layout": "IPY_MODEL_d5460ae85b3d4a5bbd931985f6ea08ab"
     }
    },
    "3a7c43d85ba44d8cad9f6cd95e890cea": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "49a42be81f124e2fb3af4b4d9817a00d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5d9248be301d411b8c47ab2b44e0e0d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fde7de6f43f34105b166e3bbe391d860",
      "placeholder": "​",
      "style": "IPY_MODEL_9c4b9044203c4f528afd5c2ad2facac8",
      "value": " 182042624/? [00:09&lt;00:00, 19919902.30it/s]"
     }
    },
    "9c4b9044203c4f528afd5c2ad2facac8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d5460ae85b3d4a5bbd931985f6ea08ab": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f2ee0013a8c048919e6eb5888e52290e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_49a42be81f124e2fb3af4b4d9817a00d",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3a7c43d85ba44d8cad9f6cd95e890cea",
      "value": 1
     }
    },
    "fde7de6f43f34105b166e3bbe391d860": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}