2. PyTorch Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor （张量）\n",
    "* PyTorch 中处理的最基本的操作对象就是Tensor，表示的是一个多维的矩阵。如下示意图，零维就是一个点，一维就是向量，二维就是矩阵，多维就相当于多维数组，这和 NumPy 是对应的，而且 **PyTorch 的 Tensor 可以和 NumPy 的 ndarray 相互转换，唯一不同的是前者可以在 GPU 上运行，而后者只能在 CPU 上运行。**\n",
    "\n",
    "<img src=\"./Images/tensor.jpg\" width=750, heigth=300>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 把 PyTorch 当做 NumPy 用\n",
    "* PyTorch 的很多操作和 NumPy 都是类似的，但是因为其能够在 GPU 上运行，所以有着比 NumPy 快很多倍的速度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.20686427 -0.81371042 -1.09025823  1.6196171  -0.26244709]\n",
      " [-1.23683592  0.45255505  0.38243564  0.90032699  1.34383286]\n",
      " [-0.33081421 -1.25847333  0.34835858  0.67918023  0.04971731]\n",
      " [-0.78090567  1.46037783 -0.60218052  2.0784988   0.75044257]]\n",
      "float64\n"
     ]
    }
   ],
   "source": [
    "# 创建一个 numpy ndarray\n",
    "numpy_tensor = np.random.randn(4, 5)\n",
    "print(numpy_tensor)\n",
    "print(numpy_tensor.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 我们可以使用下面两种方式将numpy的ndarray转换到tensor上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2069, -0.8137, -1.0903,  1.6196, -0.2624],\n",
      "        [-1.2368,  0.4526,  0.3824,  0.9003,  1.3438],\n",
      "        [-0.3308, -1.2585,  0.3484,  0.6792,  0.0497],\n",
      "        [-0.7809,  1.4604, -0.6022,  2.0785,  0.7504]], dtype=torch.float64)\n",
      "tensor([[-0.2069, -0.8137, -1.0903,  1.6196, -0.2624],\n",
      "        [-1.2368,  0.4526,  0.3824,  0.9003,  1.3438],\n",
      "        [-0.3308, -1.2585,  0.3484,  0.6792,  0.0497],\n",
      "        [-0.7809,  1.4604, -0.6022,  2.0785,  0.7504]], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "pytorch_tensor1 = torch.tensor(numpy_tensor,dtype=torch.float64)\n",
    "print(pytorch_tensor1)\n",
    "pytorch_tensor2 = torch.from_numpy(numpy_tensor)\n",
    "print(pytorch_tensor2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 使用以上两种方法进行转换的时候，会直接将 NumPy ndarray 的数据类型转换为对应的 PyTorch Tensor 数据类型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 同时我们也可以使用下面的方法将 pytorch tensor 转换为 numpy ndarray\n",
    "* 需要注意 GPU 上的 Tensor 不能直接转换为 NumPy ndarray，需要使用.cpu()先将 GPU 上的 Tensor 转到 CPU 上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.89924081  0.61037961 -1.23154489  0.08557894 -0.98492479]\n",
      " [ 1.78812816 -1.1839331   1.6943687   1.02783301 -0.46929173]\n",
      " [ 0.4528172  -0.60487508  1.89272887  1.2121682   2.18522087]\n",
      " [-1.82837114  0.66942428 -1.78702466 -0.25672374  0.8285337 ]]\n"
     ]
    }
   ],
   "source": [
    "# 如果 pytorch tensor 在 cpu 上\n",
    "numpy_array = pytorch_tensor1.numpy()\n",
    "print(numpy_array)\n",
    "\n",
    "# 如果 pytorch tensor 在 gpu 上\n",
    "numpy_array = pytorch_tensor1.cpu().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **PyTorch Tensor 使用 GPU 加速**，我们可以使用以下两种方式将 Tensor 放到 GPU 上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU available? ——> False\n"
     ]
    }
   ],
   "source": [
    "# 判定是否有 GPU 可用\n",
    "print('GPU available? ——>',torch.cuda.is_available())\n",
    "\n",
    "# 第一种方式是定义 cuda 数据类型\n",
    "dtype = torch.cuda.FloatTensor # 定义默认 GPU 的 数据类型\n",
    "gpu_tensor = torch.randn(10, 20).type(dtype)\n",
    "\n",
    "# 第二种方式更简单，推荐使用\n",
    "gpu_tensor = torch.randn(10, 20).cuda(0) # 将 tensor 放到第一个 GPU 上\n",
    "gpu_tensor = torch.randn(10, 20).cuda(1) # 将 tensor 放到第二个 GPU 上"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 使用第一种方式将 tensor 放到 GPU 上的时候会将数据类型转换成定义的类型，而用第二种方式能够直接将 tensor 放到 GPU 上，类型跟之前保持一致。\n",
    "\n",
    "* **推荐在定义 tensor 的时候就明确数据类型（比如上面执行torch.tensor()时指定dtype参数），然后直接使用第二种方法将 tensor 放到 GPU 上。**\n",
    "\n",
    "* 而将 tensor 放回 CPU 的操作非常简单"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cpu_tensor = gpu_tensor.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 访问 Tensor 的属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 5])\n",
      "torch.DoubleTensor\n",
      "2\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "## 得到 tensor 的大小\n",
    "print(pytorch_tensor1.shape)\n",
    "\n",
    "# 得到 tensor 的数据类型\n",
    "print(pytorch_tensor1.type())\n",
    "\n",
    "# 得到 tensor 的维度\n",
    "print(pytorch_tensor1.dim())\n",
    "\n",
    "# 得到 tensor 的所有元素个数\n",
    "print(pytorch_tensor1.numel())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor 的基本操作\n",
    "* Tensor 操作中的 api 和 NumPy 非常相似，如果你熟悉 NumPy 中的操作，那么 tensor 基本是一致的，以下列举其中的一些操作\n",
    "* <pr> .dtype属性是得到元素的数据类型，type()是得到数据结构类型，tensor 数据类型可参考 https://pytorch.org/docs/0.3.0/tensors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]]) torch.float32 torch.FloatTensor\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(2, 2)\n",
    "print(x, x.dtype, x.type()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1, 1],\n",
      "        [1, 1]]) torch.int64 torch.LongTensor\n",
      "tensor([[1., 1.],\n",
      "        [1., 1.]]) torch.float32 torch.FloatTensor\n"
     ]
    }
   ],
   "source": [
    "# 将其转化为整形\n",
    "x = x.long()\n",
    "# x = x.type(torch.LongTensor)  # 两种方式都可以\n",
    "print(x, x.dtype, x.type())\n",
    "\n",
    "# 再将其转回 float\n",
    "x = x.float()\n",
    "# x = x.type(torch.FloatTensor)\n",
    "print(x, x.dtype, x.type())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.5075,  0.2061,  0.2546],\n",
      "        [-1.6923,  0.3566, -0.3965],\n",
      "        [ 1.2329,  2.0489,  0.1152],\n",
      "        [ 1.2037,  0.1911,  0.7752]]) torch.float32\n"
     ]
    }
   ],
   "source": [
    "# 新建一个 float32的 tensor\n",
    "x = torch.randn(4, 3)\n",
    "print(x, x.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.2546, 0.3566, 2.0489, 1.2037]) tensor([2, 1, 1, 0])\n"
     ]
    }
   ],
   "source": [
    "# 沿着行取最大值\n",
    "max_value, max_idx = torch.max(x, dim=1)  ## 类似np.max(x, dim=1)，返回的第一个是每行的最大值，第二个是最大值的下标\n",
    "print(max_value, max_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.0468, -1.7322,  3.3971,  2.1699])\n"
     ]
    }
   ],
   "source": [
    "# 沿着行对 x 求和\n",
    "sum_x = torch.sum(x, dim=1)\n",
    "print(sum_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 改变维度（增加、减少、互换、改造）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 3])\n",
      "torch.Size([1, 4, 3])\n",
      "torch.Size([1, 1, 4, 3])\n"
     ]
    }
   ],
   "source": [
    "# 增加维度或者减少维度\n",
    "print(x.shape)\n",
    "\n",
    "x = x.unsqueeze(0) # 在第一维增加\n",
    "print(x.shape)\n",
    "\n",
    "x = x.unsqueeze(1) # 在第二维增加\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 4, 3])\n",
      "torch.Size([4, 3])\n"
     ]
    }
   ],
   "source": [
    "x = x.squeeze(0) # 减少第一维\n",
    "print(x.shape)\n",
    "\n",
    "x = x.squeeze() # 将 tensor 中所有的一维全部都去掉\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 4, 5])\n",
      "torch.Size([4, 3, 5])\n",
      "torch.Size([5, 3, 4])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, 4, 5)\n",
    "print(x.shape)\n",
    "\n",
    "# 使用permute和transpose进行维度交换\n",
    "x = x.permute(1, 0, 2) # permute 可以重新排列 tensor 的维度\n",
    "print(x.shape)\n",
    "\n",
    "x = x.transpose(0, 2)  # transpose 交换 tensor 中的两个维度\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 4, 5])\n",
      "torch.Size([12, 5])\n",
      "torch.Size([3, 20])\n"
     ]
    }
   ],
   "source": [
    "# 使用 view 对 tensor 进行 reshape\n",
    "x = torch.randn(3, 4, 5)\n",
    "print(x.shape)\n",
    "\n",
    "x = x.view(-1, 5) # -1 表示任意的大小，5 表示第二维变成 5\n",
    "print(x.shape)\n",
    "\n",
    "x = x.view(3, 20) # 重新 reshape 成 (3, 20) 的大小\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 基本的四则运算（元素级操作）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 2.0366, -1.5867, -0.7534, -0.4305],\n",
      "        [ 1.6674, -0.8818,  1.9291,  0.2475],\n",
      "        [ 1.3854,  1.7577,  0.8529, -0.0385]])\n",
      "tensor([[ 0.9495,  0.6282, -0.4515, -0.1176],\n",
      "        [ 0.5759, -0.2458,  0.6417, -0.9131],\n",
      "        [ 0.4784,  0.5167, -1.7392, -0.8053]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3, 4)\n",
    "y = torch.randn(3, 4)\n",
    "\n",
    "# 两个 tensor 求和\n",
    "z = x + y  # z = torch.add(x, y)\n",
    "m = x * y\n",
    "print(z)\n",
    "print(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 另外，pytorch中大多数的操作都支持 inplace 操作，也就是可以直接对 tensor 进行操作而不需要另外开辟内存空间，方式非常简单，一般都是在操作的符号后面加_，比如"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3])\n",
      "torch.Size([1, 3, 3])\n",
      "torch.Size([3, 1, 3])\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(3, 3)\n",
    "print(x.shape)\n",
    "\n",
    "# unsqueeze 进行 inplace\n",
    "x.unsqueeze_(0)\n",
    "print(x.shape)\n",
    "\n",
    "# transpose 进行 inplace\n",
    "x.transpose_(1, 0)\n",
    "print(x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "tensor([[2., 2., 2.],\n",
      "        [2., 2., 2.],\n",
      "        [2., 2., 2.]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.ones(3, 3)\n",
    "y = torch.ones(3, 3)\n",
    "print(x)\n",
    "\n",
    "# add 进行 inplace\n",
    "x.add_(y)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**小练习**\n",
    "\n",
    "访问[文档](http://pytorch.org/docs/0.3.0/tensors.html)了解 tensor 更多的 api，实现下面的要求\n",
    "\n",
    "创建一个 float32、4 x 4 的全为1的矩阵，将矩阵正中间 2 x 2 的矩阵，全部修改成2\n",
    "\n",
    "参考输出\n",
    "$$\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "1 & 1 & 1 & 1 \\\\\n",
    "1 & 2 & 2 & 1 \\\\\n",
    "1 & 2 & 2 & 1 \\\\\n",
    "1 & 1 & 1 & 1\n",
    "\\end{matrix}\n",
    "\\right] \\\\\n",
    "torch.FloatTensor\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1., 1.],\n",
      "        [1., 2., 2., 1.],\n",
      "        [1., 2., 2., 1.],\n",
      "        [1., 1., 1., 1.]]) torch.FloatTensor\n"
     ]
    }
   ],
   "source": [
    "# 答案\n",
    "x = torch.ones(4, 4).float()\n",
    "x[1:3, 1:3] = 2\n",
    "print(x, x.type())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable （变量）\n",
    "* <p>tensor 是 PyTorch 中的完美组件，但是构建神经网络还远远不够，我们需要能够构建计算图的 tensor，这就是 Variable， **关键在于 Variable 提供了自动求导的功能。** \n",
    "    \n",
    "* <p>Variable 和 tensor 本质上没有区别，不过 Variable 会被放入一个计算图中，然后进行前向传播，反向传播，自动求导。**Variable 有三个重要的组成特性：data, grad，grad_fn**，通过 data 可以取出 Variable 里面的 tensor 数值，grad_fn 表示的是得到这个 Variable 的操作，比如通过加减还是乘除来得到的，最后 grad 是这个 Variable 的反向传播梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 通过下面这种方式导入 Variable\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_tensor = torch.randn(10, 5)\n",
    "y_tensor = torch.randn(10, 5)\n",
    "\n",
    "# 将 tensor 变成 Variable\n",
    "x = Variable(x_tensor, requires_grad=True) # 默认 Variable 是不需要求梯度的，所以我们用这个方式申明需要对其进行求梯度\n",
    "y = Variable(y_tensor, requires_grad=True)\n",
    "\n",
    "# 可用 x.requires_grad查看该variable是否可导\n",
    "print(x.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(28.0715)\n",
      "<SumBackward0 object at 0x000001A28F7D1240>\n"
     ]
    }
   ],
   "source": [
    "z = torch.sum(2 * x + y)\n",
    "print(z.data)\n",
    "print(z.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 上面打出了 z 中的 tensor 数值，同时通过grad_fn知道了其是通过 Sum 这种方式得到的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.],\n",
      "        [2., 2., 2., 2., 2.]])\n",
      "tensor([[1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.],\n",
      "        [1., 1., 1., 1., 1.]])\n"
     ]
    }
   ],
   "source": [
    "# 求 x 和 y 的梯度\n",
    "z.backward()\n",
    "\n",
    "print(x.grad)\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 通过.grad我们得到了 x 和 y 的梯度，这里我们使用了 PyTorch 提供的自动求导机制，非常方便，下一小节会具体讲自动求导。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 自动微分\n",
    "* 自动求导是 PyTorch 中非常重要的特性，能够让我们避免手动去计算非常复杂的导数，这能够极大地减少我们构建模型的时间，这也是其前身 Torch 这个框架所不具备的特性。\n",
    "\n",
    "### 简单情况的自动微分\n",
    "* \"简单\"体现在计算的结果都是标量，也就是一个数，我们对这个标量进行求导。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比如：\n",
    "$$\n",
    "z = (x + 2)^2 + 3\n",
    "$$\n",
    "\n",
    "z 对 x 求导在 x = 2 的结果就是 \n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial x} = 2 (x + 2) = 2 (2 + 2) = 8\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1b6a9caedd8>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfR0lEQVR4nO3deXhV5b328e8vE5AQAiEhhDHMowwSUMQ6V6k41Kq1aJUqHrS1x6H1VKt1aB2qx/HVaitWD2gpirO1WgeKxREJEBFImEIYQwZCEiBz9vP+kY2lXiAh2TtrD/fnunJlZ2Un614J3Fn7Wc9ay5xziIhI+InxOoCIiLSOClxEJEypwEVEwpQKXEQkTKnARUTCVFx7riwtLc1lZWW15ypFRMLesmXLypxz6d9c3q4FnpWVRU5OTnuuUkQk7JnZ5oMt1xCKiEiYUoGLiIQpFbiISJhSgYuIhCkVuIhImFKBi4iEKRW4iEiYUoGLiARRbUMTd765mpI9tQH/3ipwEZEgevSD9cz5tJD1xXsD/r1V4CIiQfLVtkqe/qiAi7L7MmVwWsC/vwpcRCQIGpp8/OqVlXRPSuCWaSOCso52vRaKiEi0mL24gLyiKp66dAIpneKDsg7tgYuIBNiGkr38v4XrmXZUJmeM6hm09ajARUQCyOdz3PzKSjrFx3LnOaOCui4VuIhIAP1lyWZyNu/m9rNGkp7cIajrUoGLiATItt3V3P9OPicMTecHR/cO+vpU4CIiAeCc49evfoUD7j1vNGYW9HWqwEVEAmBBzlY+Wl/Gr88cQZ9uie2yThW4iEgbFVXWcPdbeRw7MJVLJvVrt/WqwEVE2mD/0Emjz/G/548lJib4Qyf7qcBFRNrg5WXb+HBtKb+aOox+3dtn6GQ/FbiISCsVV9Vy11trmJSVyozJWe2+fhW4iEgrOOe45dWvqGv0cf8FY9p16GQ/FbiISCu8nrudhfkl/M8ZwxiQluRJBhW4iMgR2llZyx1vrGZC/25cPmWAZzlU4CIiR8A5x82vrqS+yceDF44l1oOhk/1U4CIiR2BBzlY+XFvKTVOHezZ0sp8KXESkhbbtruYu/wk7Xsw6+SYVuIhIC/h8jpteWYlzjgcuaN8Tdg7lsAVuZn3NbJGZ5ZnZajO7zr/8TjPbbma5/rczgx9XRMQb877YwicbdnHLtBH0TW3fE3YOpSW3VGsEfumcW25mycAyM3vf/7lHnHMPBi+eiIj3tuyq5vdv5/GdIWlc3I7XOjmcwxa4c64IKPI/3mNmeUDwL3QrIhICmnyOXyzIJdaM+88f0y6XiW2pIxoDN7MsYDywxL/o52a20syeNbNuh/iaWWaWY2Y5paWlbQorItLenv6ogJzNu/ntuaPo1bWT13H+Q4sL3Mw6A68A1zvnqoA/AoOAcTTvoT90sK9zzs12zmU757LT09MDEFlEpH3kFVXx8HvrmDqqJ+eND72BhxYVuJnF01ze85xzrwI454qdc03OOR/wNDApeDFFRNpXXWMTN7yYS5dO8dz7g6NCauhkv5bMQjHgGSDPOffwAcszD3jaecCqwMcTEfHGI++vJ3/nHu4//yhSkxK8jnNQLZmFMgW4FPjKzHL9y24BppvZOMABhcBVQUkoItLOlhaW89TijfxoYl9OHZHhdZxDaskslI+Bg712eDvwcUREvLW3rpFfLMilT7dO/OaskV7H+VYt2QMXEYkad7+1hm27a1hw1WQ6dwjtitSp9CIifv9YtZMXlm7l6hMHMTEr1es4h6UCFxEBSqpq+fWrKxnduws3nDbU6zgtogIXkajnnOPGl1dS09DEoxeNJyEuPKoxPFKKiATR3E8LWbyulFunjWRwj85ex2kxFbiIRLV1xXv4/Tv5nDK8Bz8+JnQuVNUSKnARiVp1jU1c/0IunTvEhdyFqloitOfIiIgE0cPvrWNNURV/viyb9OQOXsc5YtoDF5Go9NH6Up5aXMAlx/TjtJGhe7blt1GBi0jU2bW3jl8s+JIhPTrzm2mhfbblt9EQiohEFecc//PySiprGnjuikl0Soj1OlKraQ9cRKLK3E8L+Wd+Cbd8bzgjMrt4HadNVOAiEjXyiqq41z9lcMZxWV7HaTMVuIhEhZr6Jq6dv4KUTvE8cEH4TRk8GI2Bi0hUuPvva1hfspfnrphE987hN2XwYLQHLiIR7+8ri5i3ZAuzThjICUMj5968KnARiWhby6u5+ZWVjO3blRtPH+Z1nIBSgYtIxGpo8vHz+SvA4A/Tw+cqgy2lMXARiVgPvruWL7dW8MTFR9M3NdHrOAEXWX+ORET8Fq0t4anFBVx8TD+mjcn0Ok5QqMBFJOIUV9XyywVfMrxnMreH+I2J20IFLiIRpcnnuO6FFdTUN/GHi8fTMT58T5U/HI2Bi0hEefSDdXxeUM4DF4xhcI9kr+MElfbARSRiLF5Xyh8WbeDCCX24MLuv13GCTgUuIhFhZ2Ut17+Yy9Aeyfzu3NFex2kXKnARCXuNTT6unb+C2oYmnrjk6LC+ROyR0Bi4iIS9h95fxxeF5Tx60biwuqt8W2kPXETC2qL8Ev744UamT+rH98f39jpOu1KBi0jY2lpezQ0LchmR2YU7zo7c+d6HogIXkbBU29DEz+Ytp6nJ8cdLjo7o+d6HctgCN7O+ZrbIzPLMbLWZXedfnmpm75vZev/7bsGPKyLS7HdvreGr7ZU89MOxZKUleR3HEy3ZA28EfumcGwEcC1xjZiOBm4GFzrkhwEL/xyIiQffysm38dckWrj5xEKeP6ul1HM8ctsCdc0XOueX+x3uAPKA3cC4w1/+0ucD3gxVSRGS/NTuquPW1rzh2YCo3nj7U6zieOqIxcDPLAsYDS4AM51wRNJc80OMQXzPLzHLMLKe0tLRtaUUkqlXWNPDTectI6RTP49OPJi42ug/jtXjrzawz8ApwvXOuqqVf55yb7ZzLds5lp6dHzq2MRKR9+XyOG1/6ku27a3jykqNJT46M+1q2RYsK3MziaS7vec65V/2Li80s0//5TKAkOBFFRODJDzfw/ppifn3mCLKzUr2OExJaMgvFgGeAPOfcwwd86k1ghv/xDOCNwMcTEYEP15bw0PvrOHdcL66YkuV1nJDRklPppwCXAl+ZWa5/2S3AfcACM5sJbAEuDE5EEYlmW3ZVc90LuQzLSOa+H4yheZ9SoAUF7pz7GDjUT+zUwMYREQHmzYNbb8Vt2UKHrj2YevIMrpl9R9RcpKqlovsQroiEnnnzYNYs2LwZc46M3cXc+/bj9PvHa14nCzkqcBEJLbfeCtXV/7Eotramebn8BxW4iISWLVuObHkUU4GLSEhp7N3n4J/o1699g4QBFbiIhIx9dY08eOIMauK/cZJOYiLcc483oUKYClxEQsL+My1n9z2Wwt8/Cv37g1nz+9mz4ZJLvI4YcnRLNREJCX9YtIF3Vu3kN9NGMOI70+CXV3sdKeRpD1xEPPfe6p08/P46fjC+NzOPH+B1nLChAhcRT+XvrOKGF3MZ2yeFe39wlM60PAIqcBHxTNneOq6cm0NShzieujQ7Km+L1hYaAxcRT9Q1NnH188so3VPHgqsm0zOlo9eRwo4KXETanXOOW19bRc7m3Tw+fTxj+3b1OlJY0hCKiLS7pz8q4OVl27ju1CGcPbaX13HClgpcRNrVwrxifv9OPtOOyuS6U4d4HSesqcBFpN3kFVVx7fwVjO6VwoMXjiUmRjNO2kIFLiLtoqSqlplzlpLcMZ6nL8vWtb0DQAcxRSToqusbmTk3h4qaBl66WjNOAkV74CISVE0+x/Uv5LJ6RyWPTx/PqF4pXkeKGCpwEQmq+97J4701xdx21khOHZHhdZyIogIXkaCZt2QzT3+0iRmT+3P5FF3jJNBU4CISFIvWlnD7G6s5eVg6t5010us4EUkFLiIBt2p7JdfMW86wjGQev/ho4mJVNcGgn6qIBNTW8moun7OUbokJzLl8Ip07aLJbsOgnKyIBU1ndwOVzllLX0MRfrzyGHl00XTCYVOAiEhB1jU381/M5bNlVzXMzJzEkI9nrSBFPBS4ibdZ8P8uVfLGpnMemj+fYgd29jhQVNAYuIm3inOOet/P425c7uPl7wzlHVxdsNypwEWmT2YsLeObjTfzkuCyuOmGg13GiigpcRFrt1eXb+P07+Zw1JpPbzxqp+1m2MxW4iLTKorUl/OrllUwZ3J2HfqhLw3pBBS4iR2zFlt387C/LGdYzmT/9eAId4nRpWC8ctsDN7FkzKzGzVQcsu9PMtptZrv/tzODGFJFQsbF0L1fMWUp6cgfmXD6J5I7xXkeKWi3ZA58DTD3I8kecc+P8b28HNpaIhKLtFTVc+uclxMYYz10xifTkDl5HimqHLXDn3GKgvB2yiEgIK9tbx6V/XsKeukaeu+IYstKSvI4U9doyBv5zM1vpH2LpdqgnmdksM8sxs5zS0tI2rE5EvFJV28CMZ79gR2UNz/5kIiN7dfE6ktD6Av8jMAgYBxQBDx3qic652c65bOdcdnp6eitXJyJeqW1o4sq5OazduYc//ngCE7NSvY4kfq0qcOdcsXOuyTnnA54GJgU2loiEgoYmH9fMW87SwnIevmgcJw/r4XUkOUCrCtzMMg/48Dxg1aGeKyLhqcnn+MWCL1mYX8Jd547WKfIh6LAXszKz+cBJQJqZbQPuAE4ys3GAAwqBq4KYUUTamc/nuPmVlV9f3+THx/b3OpIcxGEL3Dk3/SCLnwlCFhEJAc457vzbal5ato1rTx3C1ScO8jqSHILOxBSRrznnuO8f+Tz32WZmnTCQG04b4nUk+RYqcBH52mMLN/DUvwq49Nj+/Pp7w3VxqhCnAhcRAJ7610Ye+WAdF0zow2/PGaXyDgMqcBHhzx8V8Pt38jl7bC/uP3+MriwYJlTgIlHu2Y83cfff85h2VCaP/HAssSrvsKECF4licz8t5HdvrWHqqJ48+qNxxMWqEsKJflsiUer5zzdzx5ur+e7IDB6bPp54lXfY0W9MJArNW7KZ215fxWkjevDExUeTEKcqCEf6rYlEmec+K+TW11ZxyvAePHGJyjucHfZMTBGJHM98vIm73lrDd0dm8IeLx+tWaGFOBS4SJWYv3si9b+czdVRPHps+XnveEUAFLhIFnli0gQfeXcu0MZk8etE4HbCMECpwkQjmnOOxhRt45IN1nDuuFw9dOFZTBSOIClwkQjnnuO+dfJ5aXMD5R/fhfy8Yo5N0IowKXCQC+XyO295YxbwlW7hscn/uPHuUTo+PQCpwkQjT2OTjVy+v5NUV27n6xEHcNHWYLkwVoVTgIhGkvtHHdS+s4J1VO7nx9KFcc/JglXcEU4GLRIjq+kZ++pfl/GtdKbedNZKZxw/wOpIEmQpcJAJUVNdzxZyl5G6t4P7zj+Kiif28jiTtQAUuEuZ2VtZy2bNLKCyr5slLJjB1dE+vI0k7UYGLhLGC0r1c+swXVNY0MOeKiRw3KM3rSNKOVOAiYWrV9kpmPPsFAC/MOpbRvVM8TiTtTQUuEoYWryvlp39ZRtfEBJ6fOYmB6Z29jiQeUIGLhJlXlm3jpldWMrhHZ+ZcPomeKR29jiQeUYGLhAnnHE9+uJEH3l3LcYO686dLJ9ClY7zXscRDKnCRMNDkc9zuPzX+3HG9eOCCsbocrKjARUJddX0j187P5YO8Yq46cSA3nTFc1zURQAUuEtKKq2qZOXcpa3ZUcefZI/nJFJ1dKf+mAhcJUXlFVVwxZymVNQ08fVk2p47I8DqShBgVuEgIWrS2hJ/PW07njnEsuGqy5njLQR32KIiZPWtmJWa26oBlqWb2vpmt97/vFtyYItHj+c8KmTlnKf27J/H6NVNU3nJILTmMPQeY+o1lNwMLnXNDgIX+j0WkDRqafNz2+ipue2M1Jw3rwUtXTyYzpZPXsSSEHXYIxTm32MyyvrH4XOAk/+O5wIfATQHMJRJVKqrr+dm85Xy6cRezThjITVOH6/ZnclitHQPPcM4VATjnisysRwAziUSVDSV7mDk3h6KKWh68cCwXTOjjdSQJE0E/iGlms4BZAP366RrFIgdalF/CtfNX0CE+lvmzjmVCfx1OkpZr7alcxWaWCeB/X3KoJzrnZjvnsp1z2enp6a1cnUhkcc7xxKINXDF3Kf26J/Lmz6eovOWItbbA3wRm+B/PAN4ITByRyLe3rvnWZw+8u5azx/Tipasn06urDlbKkTvsEIqZzaf5gGWamW0D7gDuAxaY2UxgC3BhMEOKRIqNpXu56vllbCrbx2+mjWDm8QN002FptZbMQpl+iE+dGuAsIhHtgzXF3PBiLvFxMTw/c5LuniNtpjMxRYKsscnHw++v48kPN3JU7xT+dOkEemvIRAJABS4SRCV7arl2/go+LyjnRxP7cuc5o+gYH+t1LIkQKnCRIPm8YBf/PX8Fe2obeOjCsZyv+d0SYCpwkQDz+RxPLS7ggXfzyeqexPMzJzG8ZxevY0kEUoGLBFDpnjp++dKXLF5XyrQxmdx//hg6d9B/MwkO/csSCZCP15dx/Yu57Klt4J7zRnPxpH6aIihBpQIXaaMG/yyTP/1rI4PTOzPvymMY1jPZ61gSBVTgIm2wZVc11724ghVbKpg+qS+3nzWKTgmaZSLtQwUu0grOOV5ato3fvrmamBjj8enjOXtsL69jSZRRgYscofJ99fz61ZW8u7qYYwem8tAPx+nEHPGEClzkCCxaW8KvXl5JRXU9t5w5nCuPH0iMbrwgHlGBi7TA3rpG7n07j78u2cLQjM7MvXwSI3tpbrd4SwUuchifbCjjVy+vZEdlDf/1nQH88vRhOh1eQoIKXOQQ9tU1ct87+Tz/+WYGpCXx8tWTmdA/1etYIl9TgYscxCcbyrj51ZVs213DzOMHcOPpwzQ9UEKOClzkABXV9dz99zxeXraNrO6JvDhrMpMGaK9bQpMKXITmed1vrSzit39bze7qBn520iCuPXWIxrolpKnAJeptr6jh9tdXsTC/hDF9UnjuimM0w0TCggpcolZ9o48/f1zAYwvXYxi/mTaCy6cMIFbzuiVMqMAlKn26sYzbXl/FxtJ9nDEqg9vPHqWzKSXsqMAlqpRU1XLP23m8kbuDvqmdePYn2ZwyPMPrWCKtogKXqFDb0MQzH2/iiUUbaGxyXHvKYH528mAdpJSwpgKXiOac493Vxdzz9hq2ltfw3ZEZ3HrmCLLSkryOJtJmKnCJWGt2VHHXW2v4rGAXQzM685eZx3D8kDSvY4kEjApcIs72ihoeem8tr63YTkqneO46dxTTJ/UjLjbG62giAaUCl4hRWd3Akx9u4P8+LQRg1gkD+dmJg0lJjPc2mEiQqMAl7NXUN/HcZ4U8+eFGqmob+MH4Pvzi9KGaFigRTwUuYau2oYn5X2zhiUUbKdtbx4lD07lp6nCdRSlRQwUuYaehycdLOdt4/J/rKaqs5ZgBqfzxx0czMUsXnZLoogKXsFHb0MRLy7bxpw83sr2ihvH9uvLghWM5blB3zHT6u0QfFbiEvOr6Rv66ZAuzFxdQsqeOcX27cvf3R3PSsHQVt0S1NhW4mRUCe4AmoNE5lx2IUCIAu/fV85fPN/N/nxZSvq+eyQO788hF47THLeIXiD3wk51zZQH4PiIAbN61j2c+3sSCnK3UNvg4aVg6/33KYN3OTOQbNIQiIcE5x/Itu/nzR5v4x+qdxMUY3x/Xmyu/M5BhPZO9jicSktpa4A54z8wc8JRzbvY3n2Bms4BZAP369Wvj6iTS1DY08WbuDuZ+VsjqHVWkdIrnpycOYsZxWWR06eh1PJGQ1tYCn+Kc22FmPYD3zSzfObf4wCf4S302QHZ2tmvj+iRCbNlVzbwlm3kxZysV1Q0My0jm7u+P5rzxvUnqoBeGIi3Rpv8pzrkd/vclZvYaMAlY/O1fJdGqtqGJ99YUs2DpVj7eUEZsjHHGqAwum5zFMQNSdWBS5Ai1usDNLAmIcc7t8T8+HfhdwJJJxMjfWcWLS7fy2ortVFQ30LtrJ244bSg/nNiHzBSd7i7SWm3ZA88AXvPvNcUBf3XO/SMgqSTsFVXW8GbuDl7P3UFeURUJsTF8d1QGP5rYlymD0ojRfSdF2qzVBe6cKwDGBjCLhLmK6nreW13M67nb+axgF87BuL5d+e05ozh7bC9SkxK8jigSUXS0SNqkbG8d760u5p1VRXy2cReNPkdW90SuO3UI547rzQDd+UYkaMKiwOd8sonPC8o5ZUQPTh7Wg/TkDl5HimoFpXv5Z34JH+QV88WmcnwOsron8l8nDOR7o3tyVO8UHZAUaQdhUeCNPkfu1gr+sXonAGP7pHDK8AxOHp7OqF4pxGo8NajqGpvIKdzNP/NL+Gd+CZvK9gEwpEdnrjl5MN8bncmIzGSVtkg7M+fab2p2dna2y8nJadXXOudYU1TFovwSFuaXkLu1Auega2I8xw3qzpTBaRw/OI1+qYkqkjby+Zp/1p9sKOPjDWUsLSyntsFHQmwMkwd151T/K6G+qYleRxWJCma27GDXmgqbAv+msr11fLS+lE827OKTDWUUVdYC0LtrJyZmdSM7K5WJWakM6dFZMx4Oo77Rx6odleQUlrO0cDc5heXsrm4AYGhGZ44b1PzHcfKg7jrJRsQDEVfgB3LOsalsH59sKOOzgl0sLdxN6Z46ALp0jGNC/26M6dOVMX1SOKpPCj2So/cUbeccW8qrWbmtkq+2V5K7tYIvt1ZQ1+gDYEBaEtn9u3Hc4O5MGZRGD53OLuK5iC7wb3LOsbW8hqWF5eRsLmfZ5t1sKNmLz7+pPbt0ZHTvFIb3TGZoz2SGZSQzIC2JhLjIumt5dX0j64v3srZ4D+t27iFvZxVfbaukqrYRgIS4GEZkdmFi/25kZ3VjQv9UHSAWCUGHKvCIfD1sZvTrnki/7omcP6EPAPvqGllTVNW857mtgq+2V7JobQlN/laPizEGpCUxIC2JrLQk+ndPpH9q8/ueKR2Jjw3Ncq9taGJ7RQ2bd+2jsKyazbv2sbm8moLSfWwpr/76eR3iYhiS0ZlpY3o1vxLpncLQjOSI+6MlEk0issAPJqlDHBP94+L71TU2UVC6j3XFe1i7cw/rS/ZSuGsf/1pX+vWQAoAZpHfuQGZKR3qmdKRnl46kde5At6QEUpMS6JaYQLekeJI7xpOUEEtiQlyritE5R12jj311jVTXN1FZ08Du6nrK99Wze1895dUNlO2tY2dlLUWVteysrPl6rHq/5A5x9E9LZEyfFC6c0IchGckM65lMv9REzdYRiTBRU+AH0yEulhGZXRiR+Z93Mff5HMV7ar/eo20uy1qKqmrZVLaPTzfuYo9/GOJQ4mKMxIRYEuJiiYsxYmOM+Njm9w5o8jkamxyNPh9NPkddg4999Y1fD/McSmpSAj27dKRXSkcm9O9KZkonMlM60r97ElndE0lNStAsHJEoEdUFfigxMeYvxk5MHtT9oM+pb/RRUV1P+dd7yA3sq2tkX33z3vP+vej6Jh+NTT4afe7r0jbDX+oxze9jjY5xsSR1aN57T0yIJTEhluSO8aQmJZCaFE+3xARSOsUTF6JDOSLS/lTgrZQQF0OPLh01S0NEPKPdORGRMKUCFxEJUypwEZEwpQIXEQlTKnARkTClAhcRCVMqcBGRMKUCFxEJU+16NUIzKwU2t/LL04CyAMbxkrYl9ETKdoC2JVS1ZVv6O+fSv7mwXQu8Lcws52CXUwxH2pbQEynbAdqWUBWMbdEQiohImFKBi4iEqXAq8NleBwggbUvoiZTtAG1LqAr4toTNGLiIiPyncNoDFxGRA6jARUTCVFgVuJndZWYrzSzXzN4zs15eZ2otM3vAzPL92/OamXX1OlNrmNmFZrbazHxmFpbTvcxsqpmtNbMNZnaz13lay8yeNbMSM1vldZa2MLO+ZrbIzPL8/7au8zpTa5lZRzP7wsy+9G/LbwP6/cNpDNzMujjnqvyPrwVGOueu9jhWq5jZ6cA/nXONZnY/gHPuJo9jHTEzGwH4gKeAG51zOR5HOiJmFgusA74LbAOWAtOdc2s8DdYKZnYCsBd4zjk32us8rWVmmUCmc265mSUDy4Dvh+nvxIAk59xeM4sHPgauc859HojvH1Z74PvL2y8JCJ+/Pt/gnHvPObf/zsifA328zNNazrk859xar3O0wSRgg3OuwDlXD7wAnOtxplZxzi0Gyr3O0VbOuSLn3HL/4z1AHtDb21St45rt9X8Y738LWG+FVYEDmNk9ZrYVuAS43es8AXIF8I7XIaJUb2DrAR9vI0zLIhKZWRYwHljibZLWM7NYM8sFSoD3nXMB25aQK3Az+8DMVh3k7VwA59ytzrm+wDzg596m/XaH2xb/c24FGmnenpDUku0IY3aQZWH7yi6SmFln4BXg+m+8+g4rzrkm59w4ml9lTzKzgA1vhdxd6Z1zp7XwqX8F/g7cEcQ4bXK4bTGzGcBZwKkuhA9GHMHvJBxtA/oe8HEfYIdHWcTPP178CjDPOfeq13kCwTlXYWYfAlOBgBxoDrk98G9jZkMO+PAcIN+rLG1lZlOBm4BznHPVXueJYkuBIWY2wMwSgB8Bb3qcKar5D/w9A+Q55x72Ok9bmFn6/hlmZtYJOI0A9la4zUJ5BRhG86yHzcDVzrnt3qZqHTPbAHQAdvkXfR6OM2rM7DzgcSAdqABynXNneJvqyJjZmcCjQCzwrHPuHo8jtYqZzQdOovmypcXAHc65ZzwN1QpmdjzwEfAVzf/XAW5xzr3tXarWMbMxwFya/23FAAucc78L2PcPpwIXEZF/C6shFBER+TcVuIhImFKBi4iEKRW4iEiYUoGLiIQpFbiISJhSgYuIhKn/D5HsNUHu+7jGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "x = np.arange(-3, 3.01, 0.1)\n",
    "f = lambda x: (x + 2) ** 2 + 3\n",
    "plt.plot(x, f(x))\n",
    "plt.plot(2, f(2), 'ro')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([19.], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.Tensor([2]), requires_grad=True)\n",
    "z = f(x)  ## 这也是体现动态图优点的地方，直接把tensor当成numpy.ndarray用\n",
    "print(z)\n",
    "\n",
    "z.backward()\n",
    "print(x.grad)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 对于上面这样一个简单的例子，我们验证了自动微分，同时可以发现发现使用自动微分非常方便。如果是一个更加复杂的例子，那么手动微分就会显得非常的麻烦，所以自动微分的机制能够帮助我们省去麻烦的数学计算，下面我们可以看一个更加复杂的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = Variable(torch.randn(5, 10), requires_grad=True)\n",
    "y = Variable(torch.randn(5, 5), requires_grad=True)\n",
    "w = Variable(torch.randn(10, 5), requires_grad=True)\n",
    "\n",
    "out = torch.mean(y - torch.matmul(x, w)) # torch.matmul 是做矩阵乘法\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.1764,  0.0849, -0.0340,  0.0544,  0.0345, -0.1569,  0.0832, -0.0068,\n",
      "          0.0255, -0.1073],\n",
      "        [ 0.1764,  0.0849, -0.0340,  0.0544,  0.0345, -0.1569,  0.0832, -0.0068,\n",
      "          0.0255, -0.1073],\n",
      "        [ 0.1764,  0.0849, -0.0340,  0.0544,  0.0345, -0.1569,  0.0832, -0.0068,\n",
      "          0.0255, -0.1073],\n",
      "        [ 0.1764,  0.0849, -0.0340,  0.0544,  0.0345, -0.1569,  0.0832, -0.0068,\n",
      "          0.0255, -0.1073],\n",
      "        [ 0.1764,  0.0849, -0.0340,  0.0544,  0.0345, -0.1569,  0.0832, -0.0068,\n",
      "          0.0255, -0.1073]]) torch.Size([5, 10])\n"
     ]
    }
   ],
   "source": [
    "# 得到 x 的梯度\n",
    "print(x.grad, x.shape)  ## 计算的是out这个标量对 x 这个 tensor 中每一个元素的梯度\n",
    "# 关于标量对tensor中的每个元素进行求导，可参考一个blog (https://www.cnblogs.com/marsggbo/p/11549631.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0400, 0.0400, 0.0400, 0.0400, 0.0400],\n",
      "        [0.0400, 0.0400, 0.0400, 0.0400, 0.0400],\n",
      "        [0.0400, 0.0400, 0.0400, 0.0400, 0.0400],\n",
      "        [0.0400, 0.0400, 0.0400, 0.0400, 0.0400],\n",
      "        [0.0400, 0.0400, 0.0400, 0.0400, 0.0400]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 y 的的梯度\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.1318, -0.1318, -0.1318, -0.1318, -0.1318],\n",
      "        [-0.1305, -0.1305, -0.1305, -0.1305, -0.1305],\n",
      "        [ 0.1838,  0.1838,  0.1838,  0.1838,  0.1838],\n",
      "        [-0.0514, -0.0514, -0.0514, -0.0514, -0.0514],\n",
      "        [ 0.0860,  0.0860,  0.0860,  0.0860,  0.0860],\n",
      "        [ 0.1954,  0.1954,  0.1954,  0.1954,  0.1954],\n",
      "        [-0.0685, -0.0685, -0.0685, -0.0685, -0.0685],\n",
      "        [ 0.0226,  0.0226,  0.0226,  0.0226,  0.0226],\n",
      "        [-0.0282, -0.0282, -0.0282, -0.0282, -0.0282],\n",
      "        [ 0.0371,  0.0371,  0.0371,  0.0371,  0.0371]])\n"
     ]
    }
   ],
   "source": [
    "# 得到 w 的梯度\n",
    "print(w.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "梯度的size是跟被求梯度的量一样的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 复杂情况的自动微分\n",
    "* 上面展示了简单情况下的自动求导，都是对标量进行自动求导，那么如何对一个向量或者矩阵自动求导呢？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[2., 3.]], requires_grad=True)\n",
      "tensor([[0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "m = Variable(torch.FloatTensor([[2, 3]]), requires_grad=True) # 构建一个 1 x 2 的矩阵\n",
    "n = Variable(torch.zeros(1, 2)) # 构建一个相同大小的 0 矩阵\n",
    "print(m)\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 4., 27.]], grad_fn=<CopySlices>)\n"
     ]
    }
   ],
   "source": [
    "# 通过 m 中的值计算新的 n 中的值\n",
    "n[0, 0] = m[0, 0] ** 2\n",
    "n[0, 1] = m[0, 1] ** 3\n",
    "print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上面的式子写成数学公式，可以得到 \n",
    "$$\n",
    "n = (n_0,\\ n_1) = (m_0^2,\\ m_1^3) = (2^2,\\ 3^3) \n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们直接对 n 进行反向传播，也就是求 n 对 m 的导数。\n",
    "\n",
    "这时我们需要明确这个导数的定义，即如何定义\n",
    "\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m} = \\frac{\\partial (n_0,\\ n_1)}{\\partial (m_0,\\ m_1)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在 PyTorch 中，**如果要调用自动微分，需要往`backward()`中传入一个参数，这个参数的形状和 n 一样大**，比如是 $(w_0,\\ w_1)$，那么自动求导的结果就是：\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0}\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 4., 27.]], grad_fn=<CloneBackward>)\n"
     ]
    }
   ],
   "source": [
    "n.backward(torch.ones_like(n),create_graph=True) # 将 (w0, w1) 取成 (1, 1)\n",
    "print(m.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 除了backward()，另一种写法如下，并指定 grad_outputs参数，这个参数跟上面 backward()中传入的参数功能一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(tensor([[ 4., 27.]], grad_fn=<AddBackward0>),)\n"
     ]
    }
   ],
   "source": [
    "m_grad = torch.autograd.grad(n, m, grad_outputs=torch.ones_like(n),create_graph=True)\n",
    "print(m_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过自动微分我们得到了梯度是 4 和 27，我们可以验算一下\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_0} = w_0 \\frac{\\partial n_0}{\\partial m_0} + w_1 \\frac{\\partial n_1}{\\partial m_0} = 2 m_0 + 0 = 2 \\times 2 = 4\n",
    "$$\n",
    "$$\n",
    "\\frac{\\partial n}{\\partial m_1} = w_0 \\frac{\\partial n_0}{\\partial m_1} + w_1 \\frac{\\partial n_1}{\\partial m_1} = 0 + 3 m_1^2 = 3 \\times 3^2 = 27\n",
    "$$\n",
    "通过验算我们可以得到相同的结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多次自动微分\n",
    "* 通过调用 backward 我们可以进行一次自动求导，如果我们再调用一次 backward，会发现程序报错，没有办法再做一次。这是因为 PyTorch 默认做完一次自动求导之后，计算图就被丢弃了，所以两次自动求导需要手动设置一个东西，我们通过下面的小例子来说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([18.], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.FloatTensor([3]), requires_grad=True)\n",
    "y = x * 2 + x ** 2 + 3\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([8.])\n"
     ]
    }
   ],
   "source": [
    "y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图\n",
    "print(x.grad)  ## y_x = 2 + 2 * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([16.])\n"
     ]
    }
   ],
   "source": [
    "y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图\n",
    "print(x.grad)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 可以发现 x 的梯度变成了 16，因为这里做了两次自动求导，是将第一次的梯度 8 和第二次的梯度 8 加起来得到了 16 的结果。\n",
    "\n",
    "* <p> 通过设置 retain_graph=True 来保留计算图，以便多次求导。并且要注意，多次求导的话，它的导数值会不断累加。但是可以在求了一次导（设置retain\\_graph=True）之后，用 x.grad.data.zero_() 来清除之前的导数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([8.])\n"
     ]
    }
   ],
   "source": [
    "x.grad.data.zero_()\n",
    "y.backward(retain_graph=True) # 设置 retain_graph 为 True 来保留计算图\n",
    "print(x.grad) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**小练习**\n",
    "\n",
    "定义\n",
    "\n",
    "$$\n",
    "x = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "x_0 \\\\\n",
    "x_1\n",
    "\\end{matrix}\n",
    "\\right] = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "2 \\\\\n",
    "3\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "$$\n",
    "k = (k_0,\\ k_1) = (x_0^2 + 3 x_1,\\ 2 x_0 + x_1^2)\n",
    "$$\n",
    "\n",
    "我们希望求得\n",
    "\n",
    "$$\n",
    "j = \\left[\n",
    "\\begin{matrix}\n",
    "\\frac{\\partial k_0}{\\partial x_0} & \\frac{\\partial k_0}{\\partial x_1} \\\\\n",
    "\\frac{\\partial k_1}{\\partial x_0} & \\frac{\\partial k_1}{\\partial x_1}\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "参考答案：\n",
    "\n",
    "$$\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "4 & 3 \\\\\n",
    "2 & 6 \\\\\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4., 3.],\n",
      "        [2., 6.]])\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.FloatTensor([2,3]),requires_grad=True)\n",
    "k = Variable(torch.zeros_like(x))\n",
    "k[0] = x[0] ** 2 + 3 * x[1]\n",
    "k[1] = 2 * x[0] + x[1] ** 2\n",
    "\n",
    "\n",
    "j = torch.zeros(2,2)\n",
    "k.backward(torch.FloatTensor([1, 0]), retain_graph=True)\n",
    "j[0] = x.grad.data\n",
    "\n",
    "x.grad.data.zero_() # 这里需要对k做第二次求导，所以先归零之前求得的梯度\n",
    "\n",
    "k.backward(torch.FloatTensor([0, 1]))\n",
    "j[1] = x.grad.data\n",
    "print(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 也可以用torch.autograd.grad()来算梯度，因为没有使用 backward 来算 k 对 x 的梯度，所以梯度不会累加，那么就不用清零。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4., 3.],\n",
      "        [2., 6.]])\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.FloatTensor([2,3]),requires_grad=True)\n",
    "k = Variable(torch.zeros_like(x))\n",
    "k[0] = x[0] ** 2 + 3 * x[1]\n",
    "k[1] = 2 * x[0] + x[1] ** 2\n",
    "\n",
    "j = torch.zeros(2,2)\n",
    "k_1 = torch.autograd.grad(k, x, grad_outputs=torch.FloatTensor([1, 0]),retain_graph=True)\n",
    "k_2 = torch.autograd.grad(k, x, grad_outputs=torch.FloatTensor([0, 1]),retain_graph=True)\n",
    "j[0,:] = k_1[0]\n",
    "j[1,:] = k_2[0]\n",
    "print(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}