Module4_Lab3_FMML20210660.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Module4_Lab3_FMML20210660.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SreenadhSingamaneni/IIITH-FMML20210660/blob/main/Module4_Lab3_FMML20210660.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# FOUNDATIONS OF MODERN MACHINE LEARNING, IIIT Hyderabad\n",
        "# Module 4: Perceptron and Gradient Descent\n",
        "## Lab 3: Gradient Descent\n",
        "### Module Coordinator: Tanish Lad\n",
        "\n",
        "Gradient descent is a very important algorithm to understand, as it underpins many of the more advanced algorithms used in Machine Learning and Deep Learning.\n",
        "\n",
        "A brief overview of the algorithm is\n",
        "\n",
        "\n",
        "*   start with a random initialization of the solution.\n",
        "*   incrementally change the solution by moving in the direction of negative gradient of the objective function.\n",
        "*   repeat the previous step until some convergence criteria is met.\n",
        "\n",
        "The key equation for change in weight is:\n",
        "$$w^{k+1} \\leftarrow w^k - \\eta \\Delta J$$\n",
        "\n",
        "In this lab, we will discuss stochastic gradient descent, mini-batch gradient descent and batch gradient descent.\n"
      ],
      "metadata": {
        "id": "XYxxkQg6xCjD"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fr-MnaGs7JmZ"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ob_zZms7VOu"
      },
      "source": [
        "np.random.seed(42)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4Kix4bcChiy"
      },
      "source": [
        "# Creating the Data\n",
        "\n",
        "Let's generate some data with:\n",
        "\\begin{equation} y_0= 4 \\end{equation} \n",
        "\\begin{equation} y_1= 3 \\end{equation} \n",
        "\n",
        "and also add some noise to the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MtAS7eFZ9hX6"
      },
      "source": [
        "X = 2 * np.random.rand(100, 1)\n",
        "y = 4 + 3 * X + np.random.randn(100, 1)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zD95NaF-CxM-"
      },
      "source": [
        "Let's also plot the data we just created"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "id": "3IiEP4BQ7Wja",
        "outputId": "40aee3bd-ce15-4492-f553-e051472d6e3b"
      },
      "source": [
        "plt.plot(X, y, 'b.')\n",
        "plt.xlabel('x')\n",
        "plt.ylabel('y', rotation=0)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'y')"
            ]
          },
          "metadata": {},
          "execution_count": 4
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAZbElEQVR4nO3dfbAkVXnH8e/D5U2QUti9AkHWhSpLo1RUMrFcomYFE3V9IZpoQWKxKGZjogSSyotblGJBkjWWVWLKVIUrbAKlAQxKohgJRNhQhsuauwiCIAoIuETdK4hKjCyuT/7oHhmGebszfbrPOf37VG3duT0902f69j7nzHNe2twdERHJy15NF0BERKqn4C4ikiEFdxGRDCm4i4hkSMFdRCRDezddgK7Vq1f72rVrmy6GiEhSduzY8T13n+/fHk1wX7t2LUtLS00XQ0QkKWZ236DtSsuIiGRIwV1EJEMK7iIiGVJwFxHJkIK7iEiGFNxFRDI0U3A3s61mtsvMbuvZ9mYz+6qZ/czMOrMXUURkNouLsGVL8bMtZh3n/o/AR4GLe7bdBrwJOH/G9xYRmdniIpxwAuzeDfvuC1/4Aqxb13Spwpup5e7u1wMP9W27w93vnKlUIiIV2batCOx79hQ/t21rukT1aDTnbmabzGzJzJaWl5ebLIqIZGr9+qLFPjdX/Fy/vukS1aPR5QfcfQFYAOh0OrollIhUbt26IhWzbVsR2NuQkoGI1pYREQll3br2BPUuDYUUEcnQrEMhLwEWgeeY2U4zO83M3mhmO4F1wOfM7N+rKKiIiExuprSMu5885KkrZnlfERGZjdIyIiINCTm5Sh2qIiINCD25Si13EZEGhJ5cpeAuItKA0JOrlJYREWlA6MlVCu4iIg0JOblKaRkRkQwpuIuIZEjBXUQkQwruIiIZUnAXkdbL8TZ8Gi0jIq2W62341HIXkVYbNVM05Ra9Wu4iMrHFxfzuaNSdKdptuXdniqbeoldwF5GJpB7shhk2U3RQi36Wz1t3xajgLiITqTrYxWTQTNFhLfppNFExKriLyESqDHYpqHLtlyYqxpmCu5ltBV4H7HL3Y8pthwCXAWuBe4G3uPv3ZyumiDQt9EJXMapq7ZcmKkZz9+lfbPZy4BHg4p7g/kHgIXf/gJm9BzjY3f9i3Ht1Oh1fWlqauiwiIjELlXM3sx3u3unfPus9VK83s7V9m08E1pePLwK2AWODu4hIzkKuADlIiHHuh7r7t8vH3wEOHbajmW0ysyUzW1peXg5QFBGRdgo6icmLnM/QvI+7L7h7x9078/PzIYsiItIqIYL7d83scIDy564AxxARkRFCBPfPABvLxxuBfw1wDBFpuZSXBqjDrEMhL6HoPF1tZjuBs4EPAJ80s9OA+4C3zFpIEZFesc2WjXFZhllHy5w85KkTZnlfEZFRYpotG1tF06VVIUUkOd1JQXNzzc+WHbWqZJMU3EUkOd3Zsuee23xLub+iWbUqjr4ArS0jIkmqe1IQDM6t9y7LsGoVnHlmHCkaBXcRmUmMnYkhjMqtdyuaLVvi6QtQcBdJRIxBNNbOxBAm6cSNaeVMBXeRBMQaRGMatRLaJIE7ppUzFdxFEhBrEI2ppRrapIG7ib6AQRTcRRIQaxCNqaVah0kCdyzpMwV3kQTEHERjaanGIKb0mYK7SCIUROMXU/pMk5hERCoS08xZtdxFRCoSU/pMwV1EkhBLR+U4saTPFNxFJHoxdVSmQjl3EYlerCsvxkzBXUSiF1NHZSqCpWXM7Azg9wADPubu54U6lojkLaaOylQECe5mdgxFYH8xsBu4ysyudPe7QhxPRPLT34EaS0dlKkK13H8R2O7uPwYws/8E3gR8MNDxRCQj6kCdXaic+23Ay8xslZkdAGwAjuzfycw2mdmSmS0tLy8HKoqIpEYdqLMLEtzd/Q7gb4CrgauAm4E9A/ZbcPeOu3fm5+dDFEVEEtRUB+riYhy3yKtCsA5Vd78QuBDAzP4a2BnqWCKSl/5b13Vb7iFTM6FTQXVPwgo5WuYZ7r7LzNZQ5NtfEupYIpKfbgCsK/cectGvJvoQQo5z/5SZ3Q58FniXuz8c8FgikqE6c+8hU0FN9CGETMu8LNR7i0gcQqca6rxJScix9E3cbMXcPfxRJtDpdHxpaanpYojIhOpKNVRVgTS98Fio45vZDnfv9G/XwmEiMpW6bkxRxeSlGMbN1z0JS2vLiMhUUlrvpY3j5tVyFwmk6TRAaCmt9xLrDcZDUnAXCSCGNEAdUlnvJaWKqCoK7iIBxHSj5CqM+haSyjeUVCqiqii4iwSQUxpg1LeQGL6hpFK51E3BXSSAnNIAo76FNP0NJYbKJVYK7iKB5JIGGPUtpOlvKE1XLjFTcBeRkUZ9C5n0G0qo1EnTlUvMNENVRIKaJHUyS/Bve85dM1RFpBHjUiez5s1zSX9VTTNURRKWws0lxs1kbePs0Tqo5S6SqFRGiozLyytvHoaCu0iiUhopMip1ktOw0ZgouIskKvUWb39HqIJ6tULeZu+PgXcADtwKvM3dfxLqeCJtk3KLN5WUUsqCdKia2RHAHwEddz8GmANOCnEskTZbtw42b04vMKoTNbyQo2X2Bp5iZnsDBwD/E/BYIq2QwuiYSaS0FnyqgqRl3P0BM/sQcD/wf8DV7n51/35mtgnYBLBmzZoQRRHJRm6pjI0bi5+nnJL254hVqLTMwcCJwFHALwAHmtlb+/dz9wV377h7Z35+PkRRRLIROpVR17eCbiX1sY/BRReFPVYsmvjGFapD9ZXAN919GcDMPg0cB3w80PFEolTl1PiQo2Pq/FaQ0hDOKjT1jStUcL8feImZHUCRljkB0MIx0ipV/6cOOTqmzoCb+hDOlWqqMguVc99uZpcDNwE/Bb4MLIQ4lkisQvynDjUevM6Am/IQzmk0VZkFG+fu7mcDZ4d6f5HYpdRCHRdw277y4iyaqsy05K9IQDkExarTS7mN+mmalvwVaUAO0+qrTi+1rUO1KVryV0RGqnrCkSYw1UMtdxEZqeqccds6VJuinLuISMKG5dyVlpHK5LLuiUgOlJaRSmgEhEhc1HKXSmgJV5G4KLhLJTQColq5p7hy/3wxGJmWMbNzgIfc/bzy978Cdrn7R+oonKRDIyCqk3uKK/fPF4txLfetwCkAZrYXxd2UtLJjjVJq4aR6V6DY5J7iyv3zxWJky93d7zWzB83sRcChwJfd/cF6iiaxt3BymFoP8X2OlNakmUbuny8Wk4yWuQA4FTiMoiUvNYl5mnbsFc+kYvwcuae4cv98sZgkuF8BnAPsA/xO2OJIr5hbODFXPCsR6+fIYU2aUXL/fDEYG9zdfbeZXQc87O57aiiTlGJu4cRc8axELp9DpN/Y5QfKjtSbgDe7+zdCFUTLD6Qntlz1tHL5HNJOUy35a2bPA64ErlhJYDez5wCX9Ww6Gnhfd0hl6hQMCrl8tZ71c6R0PaRUVpnNuNEyt1ME5hVx9zuBFwKY2RzwAEXuPnkxdsBJc1K6HlIqq8yujhmqJwB3u/t9NRwrOI3RlV6xXQ+j5kXEVlYJq46Fw04CLqnhOLVQB1w66khBxHQ9jGuZx1RWCS9ocDezfYE3AJuHPL8J2ASwZs2akEWpTMwjWNquN5hDPSmImK6HccM6YyqrhBe65f4a4CZ3/+6gJ919AViAYrRM4LJUJpeOxJz0t1o3bqxv/Hos18MkLfNYyirhhQ7uJ5NRSkbi1d9qhfpSELGMQFHLXHoFu82emR0I3A8c7e4/GLe/xrnLLAblmyF8oKtjBEoslYfEaapx7rNw9/8FVoV6f4lPk0FoWKs1dDlCL1+g4YsyLd1mTyoRQxBqIp8cegRKrGvfSPwU3KUSbQ1CofPcGr4o01Jwl0q0OQit5BvDSlNX6iSVaSm4SyUUhMabNnWl4YsyDQV3qYyC0GhtTV1JM+pYW0ZEeDx1NTfXvtSV1E8td5GaKHUldVJwl0a0dWKOUldSFwV3qV0MY+JFcqece4uNWvs7JK0rLhKeWu4t1WTrediY+LamakRCUHBvqSaH5Q3qWEwtVaOKSGKn4N5STc8o7e9YTGkMeGoVkbSTcu4t1W09n3tuHMGpqjHgdfQjqM9AUqCWe4vFNCyvijHgIVrUg9IvTX/rEZmEgrtEY9bKpurUzrDKQpORJAUK7hFTp93KVN2iHlVZxPStR2SQYMHdzJ4OXAAcAzjwdneveUR1utRpt3JVt6hjSL+ogpdphWy5fwS4yt1/28z2BQ4IeKzspDR6JCZVtqibTr+ogpdZBAnuZvY04OXAqQDuvhvYHeJYuYqh1ViH2FumTaZfVMHLLEK13I8CloF/MLMXADuAM8qbZv+cmW0CNgGsWbMmUFHSFEOrMfSx1TIdrS0VvIQRKrjvDRwLnO7u283sI8B7gPf27uTuC8ACQKfT8UBlSVZTrcZxQbeqwK+W6WhNV/CStlDBfSew0923l79fThHcoxN7WqAJo4Jula1ttUzH06gcmVaQ4O7u3zGzb5nZc9z9TuAE4PYQx5pFDmmBEJXTqKBbZWs7hZapKn9JVcjRMqcDnyhHytwDvC3gsaaSelogVOXUH3ShmNK/fn31re2YW6Y5VP7SXsGCu7vfDHRCvX8VUk8LhKycukF3UICLvbVdlUnOr1r2EqtWz1BNIS0wSh2V06AAt3lzeudqGuPOr1r2ErNWB3eIOy3Qr7+VWEfllPq3m1mMO7+pp/Ukb60P7qkYtYhVyICS+rebWY06v3VUfEr7yLQU3BPR9J2TmhpvH3NgC13xKe0js1Bwr1DIYNS29EgqgS1kxae0j8xCwX1G3YC+ahWceWa4YNS29IgCW/sqdKmWgvsMeluXe+1VBKKf/SxcMEqp83dWCmztq9ClWgruM+htXboXAd6svcGoSgpshTZV6FKtVgT3ULnw/tbleefBgw+2OxhVSYFNZHrZB/eQHXNqXaYn9hE4IlXJPriH7pibpXUZW6CJrTxVS2UEjkgVsg/usXbMxRZoYitPCBqBI22SfXCPNXUyKNB0tzdRzjYEvlgrepEQsg/uEGfHXH+gWbXqyS1nqC/YtyHwxVrRi4TQiuAeo/5A099yvvhiuOii+tIkKwl8KefmY6zoRUJQcK/JoIDYH2h6W85Qf5pkksDXhty8SA6CBXczuxf4EbAH+Km713bjjmlaliFbo5MExEF3P+ptuceSJmlDbl4kB6Fb7q9w9+8FPsYTTNOyDN0anTQg9recY8wPtyE3L5KD7NIy07QsQ7dGpw2IMeaH1SkpkoaQwd2Bq83MgfPdfaF/BzPbBGwCWLNmTSUHnSaQhm6N5hYQY6x0ROSJzN3DvLHZEe7+gJk9A7gGON3drx+2f6fT8aWlpUqOHVvOPVfTnjOda5HqmNmOQX2awYJ738HfDzzi7h8atk+VwT1nsQTGafspNNpGpFrDgvtegQ52oJkd1H0M/AZwW4hjtUk3ML73vcXPxcXmyjJshm2o14nIygQJ7sChwBfN7BbgS8Dn3P2qQMdqjZgCY7efYm7uyf0Ui4uwZcvgymfU60SkOkE6VN39HuAFId47JytNscQ0DHFYJ/G4tEtuncsiscpuKOQkxgXVxcVi+j/AKaeE6SycJvccW2AcNGpmkmGlGm0jEl7rgvuooNoN6hdeCI89VmzbunXyce8rCdjTjq2PPTDG9O1CpM1C5dyjNSxv3Q3M55//eGCH4nGIzsJcc8/dbxfnnquRMCJNal3LfVjLshuY+0eG7rPP5IF3Ja3W2FIsVYr924VIGyQf3FfaKTksqPYG5rk52LABDjtsZTn3lQZsBUERCaWWSUyTmGYSU9UTYmKZICQiMqlhk5iSbrlXveCXWtIikoukO1SnnUgjIpK7pFvu006kERHJXdLBHaafSCMikrOk0zLD5DqGvClKcYmkJ/mW+yA5jyGvm1JcImnKMriDRr5URSkukTRlmZaR6ijFJZKmbFvuUg2luETSpOAuYynFJZKeoGkZM5szsy+b2ZUhjyMiIk8UOud+BnBH4GOIiEifYMHdzJ4JvBa4INQxYqKx4CISk5A59/OAPwcOCniMKGgsuIjEJkjL3cxeB+xy9x1j9ttkZktmtrS8vByiKLVYyR2YRETqECot86vAG8zsXuBS4Hgz+3j/Tu6+4O4dd+/Mz88HKkp4dY4FV/pHRCYRJC3j7puBzQBmth74U3d/a4hjxaCuseBK/4jIpDTOvSJ1jAXXUgAiMqngwd3dtwHbQh+nDVZyA24RaTe13BOipQBEZFIK7onRUgAiMgmtCikikiEFdxGRDCm4i4hkSMFdRCRDCu4T0sxQEUmJRstMQDNDRSQ1arlPQAuDiUhqWhfcp0mv6CbRIpKaVqVlpk2vaGaoiKSmVcF9loW3NDNURFLSqrRMk+kVjbYRkTq1quXeVHpFo21EpG6tCu7QTHpF67CLSN1alZZpikbbiEjdWtdyb4JG24hI3YIEdzPbH7ge2K88xuXufnaIY6VCo21EpE6hWu6PAse7+yNmtg/wRTP7vLvfGOh4IiLSI0hwd3cHHil/3af85yGOJSIiTxasQ9XM5szsZmAXcI27bx+wzyYzWzKzpeXl5VBFERFpnWDB3d33uPsLgWcCLzazYwbss+DuHXfvzM/PhyqKiEjrBB8K6e4PA9cBrw59LBERKQQJ7mY2b2ZPLx8/Bfh14GshjiUiIk9mRd9nxW9q9kvARcAcRQXySXc/Z8xrloH7Vnio1cD3pipkeCrbdGItW6zlApVtWrmU7Vnu/qS8dpDgXhczW3L3TtPlGERlm06sZYu1XKCyTSv3smn5ARGRDCm4i4hkKPXgvtB0AUZQ2aYTa9liLReobNPKumxJ59xFRGSw1FvuIiIygIK7iEiGog3uZvZqM7vTzO4ys/cMeH4/M7usfH67ma3teW5zuf1OM3tVA2X7EzO73cy+YmZfMLNn9Ty3x8xuLv99puZynWpmyz3Hf0fPcxvN7Bvlv41VlmvCsn24p1xfN7OHe54Lds7K999qZrvM7LYhz5uZ/W1Z9q+Y2bE9zwU7bxOU63fL8txqZjeY2Qt6nru33H6zmS1VWa4Jy7bezH7Q83d7X89zI6+FGsr2Zz3luq28vg4pnwt23szsSDO7rowNXzWzMwbsU9215u7R/aOY/HQ3cDSwL3AL8Ly+ff4Q+Pvy8UnAZeXj55X77wccVb7PXM1lewVwQPn4D7plK39/pMFzdirw0QGvPQS4p/x5cPn44DrL1rf/6cDW0Oes5/1fDhwL3Dbk+Q3A5wEDXgJsr+m8jSvXcd3jAa/plqv8/V5gdYPnbD1w5azXQoiy9e37euDaOs4bcDhwbPn4IODrA/6PVnatxdpyfzFwl7vf4+67gUuBE/v2OZFiFizA5cAJZmbl9kvd/VF3/yZwV/l+tZXN3a9z9x+Xv95IsXhaaJOcs2FeRbFy50Pu/n3gGqpdC2ilZTsZuKTC44/k7tcDD43Y5UTgYi/cCDzdzA4n8HkbVy53v6E8LtR3nXWPPe6cDTPLdRqibLVda+7+bXe/qXz8I+AO4Ii+3Sq71mIN7kcA3+r5fSdPPgk/38fdfwr8AFg14WtDl63XaRQ1cdf+VixzfKOZ/WYD5fqt8uve5WZ25ApfG7pslCmso4BrezaHOmeTGlb+0OdtJfqvMweuNrMdZrapoTKtM7NbzOzzZvb8cls058zMDqAIkJ/q2VzLebMijfwioH8p9MquNd1DNSAzeyvQAX6tZ/Oz3P0BMzsauNbMbnX3u2sq0meBS9z9UTP7fYpvPsfXdOxJnURxW8Y9PduaPGfRM7NXUAT3l/Zsfml5zp4BXGNmXytbtHW5ieLv9oiZbQD+BXh2jcefxOuB/3L33lZ+8PNmZk+lqFDOdPcfVvnevWJtuT8AHNnz+zPLbQP3MbO9gacBD0742tBlw8xeCZwFvMHdH+1ud/cHyp/3ANsoau9ayuXuD/aU5QLglyd9beiy9TiJvq/JAc/ZpIaVP/R5G8uKRfouAE509we723vO2S7gCqpNTY7l7j9090fKx/8G7GNmq4ngnPUYda0FOW9W3Hb0U8An3P3TA3ap7loL0XFQQcfD3hQdBkfxeKfL8/v2eRdP7FD9ZPn4+TyxQ/Uequ1QnaRsL6LoNHp23/aDgf3Kx6uBb1BRZ9KE5Tq85/EbgRv98c6ab5blO7h8fEid56zc77kUHVpWxznrO/ZahncOvpYndnJ9qY7zNkG51lD0KR3Xt/1A4KCexzcAr675nB3W/TtSBMj7y/M30bUQsmzl80+jyMsfWNd5Kz//xcB5I/ap7Fqr9IRW/MfZQNGbfDdwVrntHIqWMMD+wD+XF/eXgKN7XntW+bo7gdc0ULb/AL4L3Fz++0y5/Tjg1vKCvhU4reZybQG+Wh7/OuC5Pa99e3ku7wLeVvc5K39/P/CBvtcFPWflMS4Bvg08RpHLPA14J/DO8nkD/q4s+61Ap47zNkG5LgC+33OdLZXbjy7P1y3l3/usBs7Zu3uutRvpqYAGXQt1lq3c51SKgRe9rwt63ijSZg58pedvtiHUtablB0REMhRrzl1ERGag4C4ikiEFdxGRDCm4i4hkSMFdRCRDCu4iIhlScBcRyZCCu8gQZvYr5SJr+5vZgeUa3Mc0XS6RSWgSk8gIZvaXFLOhnwLsdPctDRdJZCIK7iIjmNm+wH8DP6GYQr9nzEtEoqC0jMhoq4CnUtw5Z/+GyyIyMbXcRUYo79l6KcUqhoe7+7sbLpLIRHSzDpEhzOwU4DF3/yczmwNuMLPj3f3aca8VaZpa7iIiGVLOXUQkQwruIiIZUnAXEcmQgruISIYU3EVEMqTgLiKSIQV3EZEM/T9VWltMwsdP/wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ScwxpouoDDyZ"
      },
      "source": [
        "## Cost Function\n",
        "\n",
        "The equation for calculating cost function is as shown below. The cost function is only for linear regression. For other algorithms, the cost function will be different and the gradients would have to be derived from the cost functions\n",
        "\n",
        "\\begin{equation}\n",
        "J(y_{pred}) = \\frac{1}{2m} \\sum_{i=1}^{m} (h(y_{pred})^{(i)} - y^{(i)})^2 \n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PUeTUAXH7ZaV"
      },
      "source": [
        "def cal_cost(y_pred, X, y):\n",
        "    '''\n",
        "    Calculates the cost for given X and Y.\n",
        "    y_pred = Vector of y_preds \n",
        "    X = Row of X's np.zeros((2, j))\n",
        "    y = Actual y's np.zeros((2, 1))\n",
        "    \n",
        "    where:\n",
        "        j is the no of features\n",
        "    '''\n",
        "    \n",
        "    m = len(y)\n",
        "    \n",
        "    predictions = X.dot(y_pred)\n",
        "    cost = (1 / 2 * m) * np.sum(np.square(predictions - y))\n",
        "\n",
        "    return cost"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FcXqsVNpDbKC"
      },
      "source": [
        "## Gradients\n",
        "\n",
        "\\begin{equation}\n",
        "y_{pred_0}: = y_{pred_0} -\\alpha . (1/m .\\sum_{i=1}^{m}(h(y_{pred}^{(i)} - y^{(i)}).X_0^{(i)})\n",
        "\\end{equation}\n",
        "\\begin{equation}\n",
        "y_{pred_1}: = y_{pred_1} -\\alpha . (1/m .\\sum_{i=1}^{m}(h(y_{pred}^{(i)} - y^{(i)}).X_0^{(i)})\n",
        "\\end{equation}\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        ".\n",
        "\n",
        "\\begin{equation}\n",
        "y_{pred_j}: = y_{pred_j} -\\alpha . (1/m .\\sum_{i=1}^{m}(h(y_{pred}^{(i)} - y^{(i)}).X_0^{(i)})\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fwxBFXP88NBW"
      },
      "source": [
        "def gradient_descent(X, y, y_pred, learning_rate=0.01, iterations=100):\n",
        "    '''\n",
        "    X = Matrix of X with added bias units\n",
        "    y = Vector of Y\n",
        "    y_pred = Vector of y_preds np.random.randn(j, 1)\n",
        "    learning_rate \n",
        "    iterations = no of iterations\n",
        "    \n",
        "    Returns the final y_pred vector and array of cost history over no of iterations\n",
        "    '''\n",
        "\n",
        "    m = len(y)\n",
        "    cost_history = np.zeros(iterations)\n",
        "    y_pred_history = np.zeros((iterations, 2))\n",
        "    \n",
        "    for it in range(iterations):    \n",
        "        prediction = np.dot(X, y_pred)\n",
        "        y_pred = y_pred - (1 / m) * learning_rate * (X.T.dot((prediction - y)))\n",
        "        y_pred_history[it,:] = y_pred.T\n",
        "        cost_history[it]  = cal_cost(y_pred, X, y)\n",
        "        \n",
        "    return y_pred, cost_history, y_pred_history    "
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1iSohSB2EtK1"
      },
      "source": [
        "Let's do 1000 iterations with a learning rate of 0.01. \n",
        "We will start with a random prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "18AX7hrU8bv5",
        "outputId": "8440d35c-8c71-4c14-a3db-e85a0831cc30"
      },
      "source": [
        "lr = 0.01\n",
        "n_iter = 1000\n",
        "\n",
        "y_pred = np.random.randn(2,1)\n",
        "X_b = np.c_[np.ones((len(X), 1)), X]\n",
        "y_pred, cost_history, y_pred_history = gradient_descent(X_b, y, y_pred, lr, n_iter)\n",
        "\n",
        "print('y_pred[0]: {:0.3f}\\ny_pred[1]: {:0.3f}'.format(y_pred[0][0], y_pred[1][0]))\n",
        "print('Final error: {:0.3f}'.format(cost_history[-1]))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "y_pred[0]: 3.903\n",
            "y_pred[1]: 3.046\n",
            "Final error: 4180.375\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7fao2MaE216"
      },
      "source": [
        "Plotting the error vs Number of iterations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "DrkrAAbk8hIs",
        "outputId": "d3820407-2083-432c-8623-6af5dae50026"
      },
      "source": [
        "fig, ax = plt.subplots(figsize=(12,8))\n",
        "\n",
        "ax.set_ylabel('Error')\n",
        "ax.set_xlabel('Number of iterations')\n",
        "\n",
        "ax.plot(range(n_iter), cost_history, 'b.')"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f5a38d29290>]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuYAAAHgCAYAAADpKKjTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7xddX3n+9eHE+KvVsOPDNoEm7Rm7KQ6Ip4iu/ahp8aB4PgweLUtXnuTVq6pVVvstBfJ3DvDXOk0amcGyowyjYAmrQ+RAZU8erU0PbKL0zn8OJEKAjKk+INwQTIkgredckz43D/W98jOycnvc/Zae+3X8/E4j7XWd62993ez2fA+3/P5fldkJpIkSZLqdULdHZAkSZJkMJckSZIawWAuSZIkNYDBXJIkSWoAg7kkSZLUAAZzSZIkqQEW1N2Bpjj11FNz2bJldXdDkiRJLbd9+/b/kZmLZ7YbzItly5YxOTlZdzckSZLUchHxndnaLWWRJEmSGsBgLkmSJDWAwVySJElqAIO5JEmS1AAGc0mSJKkBDOaSJElSAxjMJUmSpAYwmEuSJEkNYDCXJEmSGsBgLkmSJDWAwVySJElqAIO5JEmS1AAGc0mSJKkBDOaSJElSAxjMJUmSpAYwmEuSJEkNYDCv0cQEbNxYbSVJkjTcFtTdgWE1MQGrVsHUFCxcCOPj0OnU3StJkiTVxRHzmnS7VSjft6/adrt190iSJEl1MpjXZGysGikfGam2Y2N190iSJEl1spSlJp1OVb7S7Vah3DIWSZKk4WYwr1GnYyCXJElSxVIWSZIkqQEM5pIkSVIDGMwlSZKkBjCYS5IkSQ1gMJckSZIawGAuSZIkNYDBXJIkSWoAg7kkSZLUAAZzSZIkqQEM5jWbmICNG6utJEmShteCujswzCYmYNUqmJqChQthfBw6nbp7JUmSpDo4Yl6jbrcK5fv2Vdtut+4eSZIkqS4G8xqNjVUj5SMj1XZsrO4eSZIkqS6WstSo06nKV7rdKpRbxiJJkjS8DOY163QM5JIkSbKURZIkSWoEg7kkSZLUAPMWzCPi2oh4PCK+Mcu5342IjIhTy3FExJURsSMi7o6IM3uuXRcRD5afdT3tr4mIe8pjroyIKO0nR8S2cv22iDhpvt6jJEmSNFfmc8T808DqmY0RcTpwDvDdnubzgBXlZz1wVbn2ZOBS4LXAWcClPUH7KuA9PY+bfq1LgPHMXAGMl2NJkiSp0eYtmGfmrcDuWU5dDlwMZE/bGmBLVm4DFkXES4BzgW2ZuTsz9wDbgNXl3Asz87bMTGALcH7Pc20u+5t72iVJkqTG6muNeUSsAR7JzK/POLUEeLjneGdpO1T7zlnaAU7LzEfL/mPAaYfoz/qImIyIyV27dh3t25EkSZLmTN+CeUQ8H/iXwL/u12uW0fQ8xPlNmTmamaOLFy/uV7ckSZKkA/RzxPyngeXA1yPi28BS4GsR8WLgEeD0nmuXlrZDtS+dpR3ge6XUhbJ9fM7fiSRJkjTH+hbMM/OezPxHmbksM5dRlZ+cmZmPAVuBtWV1lrOBJ0s5ys3AORFxUpn0eQ5wczn3VEScXVZjWQvcVF5qKzC9esu6nnZJkiSpseZzucTPAhPAyyNiZ0RceIjLvwQ8BOwAPgm8DyAzdwOXAXeWnw+XNso1V5fH/C3w5dL+EeCfRcSDwJvKcaNNTMDGjdVWkiRJwymqMmyNjo7m5ORk3193YgJWrYKpKVi4EMbHodPpezckSZLUJxGxPTNHZ7Z758+adbtVKN+3r9p2u3X3SJIkSXUwmNdsbKwaKR8ZqbZjY3X3SJIkSXVYUHcHhl2nU5WvdLtVKLeMRZIkaTgZzBug0zGQS5IkDTtLWSRJkqQGMJhLkiRJDWAwlyRJkhrAYC5JkiQ1gMFckiRJagCDuSRJktQABnNJkiSpAQzmkiRJUgMYzCVJkqQGMJg3wMQEbNxYbSVJkjScFtTdgWE3MQGrVsHUFCxcCOPj0OnU3StJkiT1myPmNet2q1C+b1+17Xbr7pEkSZLqYDCv2dhYNVI+MlJtx8bq7pEkSZLqYClLzTqdqnyl261CuWUskiRJw8lg3gCdjoFckiRp2FnKIkmSJDWAwVySJElqAIO5JEmS1AAGc0mSJKkBDOaSJElSAxjMJUmSpAYwmEuSJEkNYDCXJEmSGsBg3hATE7BxY7WVJEnS8PHOnw0wMQGrVsHUFCxcCOPj3glUkiRp2Dhi3gDdbhXK9+2rtt1u3T2SJElSvxnMG2BsrBopHxmptmNjdfdIkiRJ/WYpSwN0OlX5SrdbhXLLWCRJkoaPwbwhOh0DuSRJ0jCzlEWSJElqAIO5JEmS1AAGc0mSJKkBDOaSJElSAxjMJUmSpAYwmEuSJEkNYDCXJEmSGsBg3hATE7BxY7WVJEnS8PEGQw0wMQGrVsHUFCxcWN0F1JsNSZIkDRdHzBug261C+b591bbbrbtHkiRJ6jeDeQOMjVUj5SMj1XZsrO4eSZIkqd8sZWmATqcqX+l2q1BuGYskSdLwmbcR84i4NiIej4hv9LT9YUR8MyLujogvRMSinnMbImJHRDwQEef2tK8ubTsi4pKe9uURcXtp/1xELCztzynHO8r5ZfP1HudSpwMbNhjKJUmShtV8lrJ8Glg9o20b8IrM/KfAfwc2AETESuAC4GfLYz4RESMRMQJ8HDgPWAm8s1wL8FHg8sx8GbAHuLC0XwjsKe2Xl+skSZKkRpu3YJ6ZtwK7Z7T9RWbuLYe3AUvL/hrgusx8OjO/BewAzio/OzLzocycAq4D1kREAG8EbiiP3wyc3/Ncm8v+DcCqcr0kSZLUWHVO/nw38OWyvwR4uOfcztJ2sPZTgO/3hPzp9v2eq5x/slwvSZIkNVYtwTwi/k9gL/CZOl6/px/rI2IyIiZ37dpVZ1ckSZI05PoezCPi14C3AO/KzCzNjwCn91y2tLQdrP0JYFFELJjRvt9zlfMvKtcfIDM3ZeZoZo4uXrz4ON+ZJEmSdOz6GswjYjVwMfDWzPz7nlNbgQvKiirLgRXAHcCdwIqyAstCqgmiW0ugvwV4R3n8OuCmnudaV/bfAXyl5xcASZIkqZHmbR3ziPgsMAacGhE7gUupVmF5DrCtzMe8LTPfm5n3RsT1wH1UJS7vz8x95Xk+ANwMjADXZua95SU+BFwXEb8P3AVcU9qvAf4kInZQTT69YL7eoyRJkjRXwsHkyujoaE5OTtbah4kJbzIkSZLUdhGxPTNHZ7Z758+GmJiAVatgagoWLqzuBGo4lyRJGh51LpeoHt1uFcr37au23W7dPZIkSVI/GcwbYmysGikfGam2Y2N190iSJEn9ZClLQ3Q6VfmKNeaSJEnDyWDeIJ2OgVySJGlYWcoiSZIkNYDBXJIkSWoAg7kkSZLUAAZzSZIkqQEM5pIkSVIDGMwlSZKkBjCYN8zEBGzcWG0lSZI0PFzHvEEmJmDVKpiaqu7+OT7uuuaSJEnDwhHzBul2q1C+b1+17Xbr7pEkSZL6xWDeIGNj1Uj5yEi1HRuru0eSJEnqF0tZGqTTqcpXut0qlFvGIkmSNDwM5g3T6RjIJUmShpGlLJIkSVIDGMwlSZKkBjCYS5IkSQ1gMJckSZIawGAuSZIkNYDBXJIkSWoAg3nDTEzAxo3VVpIkScPDdcwbZGICVq2Cqanqzp/j465pLkmSNCwcMW+QbrcK5fv2Vdtut+4eSZIkqV8M5g0yNlaNlI+MVNuxsbp7JEmSpH6xlKVBOp2qfKXbrUK5ZSySJEnDw2DeMJ2OgVySJGkYWcoiSZIkNYDBXJIkSWoAg7kkSZLUAAZzSZIkqQEM5pIkSVIDGMwbaGICNm6stpIkSRoOLpfYMBMTsGpVdefPhQurdc1dPlGSJKn9HDFvmG63CuX79lXbbrfuHkmSJKkfDOYNMzZWjZSPjFTbsbG6eyRJkqR+sJSlYTqdqnyl261CuWUskiRJw8Fg3kCdjoFckiRp2FjKIkmSJDWAwVySJElqAIO5JEmS1AAGc0mSJKkBDOaSJElSAxjMG2hiAjZurLaSJEkaDvMWzCPi2oh4PCK+0dN2ckRsi4gHy/ak0h4RcWVE7IiIuyPizJ7HrCvXPxgR63raXxMR95THXBkRcajXGBQTE7BqFfyrf1VtDeeSJEnDYT5HzD8NrJ7RdgkwnpkrgPFyDHAesKL8rAeugipkA5cCrwXOAi7tCdpXAe/pedzqw7zGQOh2YWoK9u2rtt1u3T2SJElSP8xbMM/MW4HdM5rXAJvL/mbg/J72LVm5DVgUES8BzgW2ZebuzNwDbANWl3MvzMzbMjOBLTOea7bXGAhjY7BwIYyMVNuxsbp7JEmSpH7o950/T8vMR8v+Y8BpZX8J8HDPdTtL26Had87SfqjXGAidDoyPVyPlY2PeAVSSJGlY9DuY/0hmZkRkna8REeupSmd46UtfOp9dOSqdjoFckiRp2PR7VZbvlTIUyvbx0v4IcHrPdUtL26Hal87SfqjXOEBmbsrM0cwcXbx48TG/KUmSJOl49TuYbwWmV1ZZB9zU0762rM5yNvBkKUe5GTgnIk4qkz7PAW4u556KiLPLaixrZzzXbK8hSZIkNda8lbJExGeBMeDUiNhJtbrKR4DrI+JC4DvAL5fLvwS8GdgB/D3w6wCZuTsiLgPuLNd9ODOnJ5S+j2rll+cBXy4/HOI1JEmSpMaKalETjY6O5uTkZN3dkCRJUstFxPbMHJ3Z7p0/G8q7f0qSJA2X2lZl0cFN3/1zaqpay3x83FVaJEmS2s4R8wby7p+SJEnDx2DeQN79U5IkafhYytJA3v1TkiRp+BjMG8q7f0qSJA0XS1kkSZKkBjCYS5IkSQ1gMJckSZIawGAuSZIkNYDBvMG8+6ckSdLwcFWWhvLun5IkScPFEfOG8u6fkiRJw8Vg3lDe/VOSJGm4WMrSUN79U5IkabgYzBvMu39KkiQND0tZJEmSpAYwmEuSJEkNYDCXJEmSGsBgLkmSJDWAwbzBvPOnJEnS8HBVlobyzp+SJEnDxRHzhvLOn5IkScPFYN5Q3vlTkiRpuFjK0lDe+VOSJGm4GMwbzDt/SpIkDQ9LWSRJkqQGMJhLkiRJDWAwlyRJkhrAYN5w3mRIkiRpODj5s8G8yZAkSdLwcMS8wbzJkCRJ0vAwmDeYNxmSJEkaHpayNJg3GZIkSRoeBvOG8yZDkiRJw8FSFkmSJKkBDOaSJElSAxjMJUmSpAYwmA8AbzIkSZLUfk7+bDhvMiRJkjQcHDFvOG8yJEmSNBwM5g3nTYYkSZKGg6UsDedNhiRJkoaDwXwAeJMhSZKk9rOURZIkSWqAWoJ5RPxORNwbEd+IiM9GxHMjYnlE3B4ROyLicxGxsFz7nHK8o5xf1vM8G0r7AxFxbk/76tK2IyIu6f87lCRJko5O34N5RCwBfhsYzcxXACPABcBHgcsz82XAHuDC8pALgT2l/fJyHRGxsjzuZ4HVwCciYiQiRoCPA+cBK4F3lmsHluuYS5IktV9dNeYLgOdFxA+B5wOPAm8E/tdyfjPwb4CrgDVlH+AG4D9FRJT26zLzaeBbEbEDOKtctyMzHwKIiOvKtffN83uaF65jLkmSNBwOO2IeESdExM/P1Qtm5iPAvwO+SxXInwS2A9/PzL3lsp3AkrK/BHi4PHZvuf6U3vYZjzlY+0ByHXNJkqThcNhgnpnPUJWGzImIOIlqBHs58BPAC6hKUfouItZHxGRETO7atauOLhyW65hLkiQNhyOtMR+PiLeXEpLj9SbgW5m5KzN/CHweeB2wKCKmS2uWAo+U/UeA0wHK+RcBT/S2z3jMwdoPkJmbMnM0M0cXL148B29t7k2vY37ZZZaxSJIktdmRBvPfAP4LMBURT0XEDyLiqWN8ze8CZ0fE80vQX0VV/30L8I5yzTrgprK/tRxTzn8lM7O0X1BWbVkOrADuAO4EVpRVXhZSTRDdeox9bYROBzZsMJRLkiS12RFN/szMH5+rF8zM2yPiBuBrwF7gLmAT8P8A10XE75e2a8pDrgH+pEzu3E0VtMnMeyPieqpQvxd4f2buA4iIDwA3U634cm1m3jtX/ZckSZLmQ1SDz0dwYcRbgdeXw25m/tm89aoGo6OjOTk5WXc3JEmS1HIRsT0zR2e2H1EpS0R8BLiIanT6PuCiiNg4t13UobiWuSRJUrsd6TrmbwbOKCu0EBGbqcpNNsxXx/Qs1zKXJElqv6O58+einv0XzXVHdHCuZS5JktR+Rzpi/gfAXRFxCxBUteaXzFuvtJ/ptcynR8xdy1ySJKl9DhvMI+IE4BngbODnSvOHMvOx+eyYnjW9lnm3W4Vyy1gkSZLa57DBPDOfiYiLM/N6Bnw98EHW6RjIJUmS2uxIa8z/MiJ+LyJOj4iTp3/mtWeSJEnSEDnSGvNfKdv397Ql8FNz2x1JkiRpOB1pjfklmfm5PvRHBzExYY25JElSmx1pjfn/ARjMa+I65pIkSe1njfkAcB1zSZKk9rPGfAC4jrkkSVL7HVEwz8zl890RHZzrmEuSJLXfIUtZIuLinv1fmnHuD+arUzpQpwMbNhjKJUmS2upwNeYX9OxvmHFu9Rz3RZIkSRpahwvmcZD92Y4lSZIkHaPDBfM8yP5sx5pnExOwcWO1lSRJUrscbvLnqyLiKarR8eeVfcrxc+e1Z9qPa5lLkiS12yFHzDNzJDNfmJk/npkLyv708Yn96qRcy1ySJKntjvQGQ6rZ9FrmIyOuZS5JktRGR3qDIdXMtcwlSZLazWA+QDodA7kkSVJbWcoiSZIkNYDBfMC4ZKIkSVI7WcoyQFwyUZIkqb0cMR8gLpkoSZLUXgbzAeKSiZIkSe1lKcsAcclESZKk9jKYDxiXTJQkSWonS1kkSZKkBjCYDxiXS5QkSWonS1kGiMslSpIktZcj5gPE5RIlSZLay2A+QFwuUZIkqb0sZRkgLpcoSZLUXgbzAeNyiZIkSe1kKcsAcmUWSZKk9nHEfMC4MoskSVI7OWI+YFyZRZIkqZ0M5gPGlVkkSZLayVKWAePKLJIkSe1kMB9ArswiSZLUPpayDCBXZZEkSWofR8wHjKuySJIktZMj5gPGVVkkSZLayWA+YFyVRZIkqZ1qCeYRsSgiboiIb0bE/RHRiYiTI2JbRDxYtieVayMiroyIHRFxd0Sc2fM868r1D0bEup7210TEPeUxV0ZE1PE+58P0qiyXXWYZiyRJUpvUNWL+R8CfZ+bPAK8C7gcuAcYzcwUwXo4BzgNWlJ/1wFUAEXEycCnwWuAs4NLpMF+ueU/P41b34T31TacDGzYYyiVJktqk78E8Il4EvB64BiAzpzLz+8AaYHO5bDNwftlfA2zJym3Aooh4CXAusC0zd2fmHmAbsLqce2Fm3paZCWzpea7WcGUWSZKkdqljVZblwC7gUxHxKmA7cBFwWmY+Wq55DDit7C8BHu55/M7Sdqj2nbO0t4Yrs0iSJLVPHaUsC4Azgasy89XA3/Fs2QoAZaQ757sjEbE+IiYjYnLXrl3z/XJzxpVZJEmS2qeOYL4T2JmZt5fjG6iC+vdKGQpl+3g5/whwes/jl5a2Q7UvnaX9AJm5KTNHM3N08eLFx/Wm+smVWSRJktqn78E8Mx8DHo6Il5emVcB9wFZgemWVdcBNZX8rsLasznI28GQpebkZOCciTiqTPs8Bbi7nnoqIs8tqLGt7nqsVXJlFkiSpfeq68+dvAZ+JiIXAQ8CvU/2ScH1EXAh8B/jlcu2XgDcDO4C/L9eSmbsj4jLgznLdhzNzd9l/H/Bp4HnAl8tPq3Q6BnJJkqQ2iaqcW6Ojozk5OVl3N47KxERVXz42ZkiXJEkaFBGxPTNHZ7bXNWKu4+TKLJIkSe1S1w2GdJxcmUWSJKldDOYDypVZJEmS2sVSlgE1vTKLNeaSJEntYDAfYK7MIkmS1B6WsgywiQnYuLHaSpIkabA5Yj6gXJVFkiSpXRwxH1CuyiJJktQuBvMB5aoskiRJ7WIpy4ByVRZJkqR2MZgPsOkwPl3GYjiXJEkaXAbzAeYEUEmSpPawxnyAOQFUkiSpPQzmA8wJoJIkSe1hKcsAcwKoJElSezhiLkmSJDWAI+YDzMmfkiRJ7eGI+QBz8qckSVJ7GMwHmJM/JUmS2sNSlgHm5E9JkqT2MJgPOO/+KUmS1A4G8wHnBFBJkqR2sMZ8wDkBVJIkqR0M5gPOCaCSJEntYCnLgHMCqCRJUjsYzFvACaCSJEmDz2DeAk4AlSRJGnzWmLeAE0AlSZIGn8G8BZwAKkmSNPgsZWmBTgeuuAJuvBHe/nbLWCRJkgaRwbwFJibggx+syli++lV45SsN55IkSYPGUpYWsMZckiRp8BnMW8Aac0mSpMFnKUsLeJMhSZKkwWcwbwlvMiRJkjTYDOYt4U2GJEmSBps15i3hBFBJkqTBZjBvCSeASpIkDTZLWVrCmwxJkiQNNoN5S3iTIUmSpMFmKUtLWGMuSZI02AzmLWGNuSRJ0mCzlKUlpm8ytGVL3T2RJEnSsXDEvGU2b4ZPfrJa03xiou7eSJIk6UgZzFvEOnNJkqTBZTBvEevMJUmSBldtwTwiRiLiroj4s3K8PCJuj4gdEfG5iFhY2p9TjneU88t6nmNDaX8gIs7taV9d2nZExCX9fm91ma4zf897YN26unsjSZKko1HniPlFwP09xx8FLs/MlwF7gAtL+4XAntJ+ebmOiFgJXAD8LLAa+EQJ+yPAx4HzgJXAO8u1Q8M6c0mSpMFTSzCPiKXAPweuLscBvBG4oVyyGTi/7K8px5Tzq8r1a4DrMvPpzPwWsAM4q/zsyMyHMnMKuK5cOxSsM5ckSRpMdY2YXwFcDDxTjk8Bvp+Ze8vxTmBJ2V8CPAxQzj9Zrv9R+4zHHKz9ABGxPiImI2Jy165dx/ueGmFsrKoxj6i21plLkiQNhr4H84h4C/B4Zm7v92vPlJmbMnM0M0cXL15cd3fmTMT+W0mSJDVfHSPmrwPeGhHfpiozeSPwR8CiiJi+4dFS4JGy/whwOkA5/yLgid72GY85WPtQ6HZh717IrLaWskiSJA2GvgfzzNyQmUszcxnV5M2vZOa7gFuAd5TL1gE3lf2t5Zhy/iuZmaX9grJqy3JgBXAHcCewoqzysrC8xtY+vLVGcMlESZKkwbTg8Jf0zYeA6yLi94G7gGtK+zXAn0TEDmA3VdAmM++NiOuB+4C9wPszcx9ARHwAuBkYAa7NzHv7+k5qNL1k4pYtdfdEkiRJRyOqwWeNjo7m5ORk3d2YExMT1VKJU1PVqPn4eBXYJUmSVL+I2J6ZozPbvfNnC7lkoiRJ0uAxmLfQdJ35CSdUK7OcckrdPZIkSdLhGMxbqNOBK66oJoA+8wx88IPeAVSSJKnpDOYt9cQTVSh/5hnLWSRJkgaBwbylXDZRkiRpsDRpuUTNoelylhtvhLe/3VVZJEmSms5g3lITE1Vt+dQUfPWr8MpXGs4lSZKazFKWlnLJREmSpMFiMG8pl0yUJEkaLAbzlnLJREmSpMFiMG8xl0yUJEkaHAbzFhsbq0bMI6qtSyZKkiQ1l8G85SL230qSJKmZDOYt1u3C3r2QWW0tZZEkSWoug3mLuTKLJEnS4DCYt5grs0iSJA0Og3nL9a7M8g//AFu21N0jSZIkzcZg3nLTK7NAVWv+qU85ai5JktREBvOW63Tg3e9+dlUWJ4FKkiQ1k8F8CKxdCyee6HrmkiRJTWYwHxKuZy5JktRsBvMh0Lue+dSUE0AlSZKayGA+BJwAKkmS1HwG8yHgBFBJkqTmM5gPCSeASpIkNZvBfIg4AVSSJKm5DOZDwgmgkiRJzWYwHxJOAJUkSWo2g/mQmDkB1FFzSZKkZjGYD5HpCaDgqLkkSVLTGMyHiMsmSpIkNZfBfMi4bKIkSVIzGcyHkMsmSpIkNY/BfMi4bKIkSVIzGcyHjMsmSpIkNZPBfMhMTwCd9sMfOgFUkiSpCQzmQ+jVr352/5ln4JRT6uuLJEmSKgbzIfTEE3BC+eQj4K676u2PJEmSDOZDaWwMFiyo9jPhk5+ETZtq7ZIkSdLQM5gPoZl15vv2wQc+4CRQSZKkOhnMh9Tatc+OmoN3AZUkSaqbwXxIdTrwL/7Fs8eZTgKVJEmqk8F8iC1a5CRQSZKkpjCYDzEngUqSJDWHwXyIOQlUkiSpOfoezCPi9Ii4JSLui4h7I+Ki0n5yRGyLiAfL9qTSHhFxZUTsiIi7I+LMnudaV65/MCLW9bS/JiLuKY+5MiKi3+9zUDgJVJIkqRnqGDHfC/xuZq4EzgbeHxErgUuA8cxcAYyXY4DzgBXlZz1wFVRBHrgUeC1wFnDpdJgv17yn53Gr+/C+BtJsk0C///36+iNJkjSs+h7MM/PRzPxa2f8BcD+wBFgDbC6XbQbOL/trgC1ZuQ1YFBEvAc4FtmXm7szcA2wDVpdzL8zM2zIzgS09z6VZLFpUTf6cdvnllrNIkiT1W6015hGxDHg1cDtwWmY+Wk49BpxW9pcAD/c8bGdpO1T7zlnadRBjYzAy8uyx5SySJEn9V1swj4gfA24EPpiZT/WeKyPd2Yc+rI+IyYiY3LVr13y/XGNZziJJklS/WoJ5RJxIFco/k5mfL83fK2UolO3jpf0R4PSehy8tbYdqXzpL+wEyc1Nmjmbm6OLFi4/vTQ04y1kkSZLqVceqLAFcA9yfmf+h59RWYHpllXXATT3ta8vqLGcDT5aSl5uBcyLipDLp8xzg5nLuqYg4u7zW2p7n0kHMVs6yZUtt3ZEkSRo6dYyYvw7434A3RsTflJ83Ax8B/llEPAi8qRwDfAl4CNgBfBJ4H0Bm7gYuA+4sPx8ubZRrri6P+Vvgy/14Y4Os04GPf/zZcJ4J11zjqLkkSVK/RFXOrdHR0ZycnKy7G7V729vgi1989vj88+ELX6ivP5IkSW0TEdszc3Rmu3f+1H5e/OL9j7duddRckiSpHwzm2s/atXBCz78VzzwDH/tYff2RJEkaFgZz7afTgbe+df+2m26CTZvq6Y8kSdKwMJjrABdfvP8KLZnwvvdZ0iJJkjSfDOY6QKcDn/jE/uua79tnSYskSdJ8MphrVuvXw5o1+/u2lYcAAA4sSURBVLc5EVSSJGn+GMx1UBdf7ERQSZKkfjGY66CcCCpJktQ/BnMdkhNBJUmS+sNgrkNyIqgkSVJ/GMx1WLNNBLWkRZIkaW4ZzHVEZk4EzYT3vtdwLkmSNFcM5jois00ENZxLkiTNHYO5jtjFF8OJJ+7fZjiXJEmaGwZzHbFOB/7qr2Dlyv3bDeeSJEnHz2Cuo9LpwNVXO3IuSZI01wzmOmqHGjn/jd+AD32onn5JkiQNMoO5jsn0yPkJs/wb9LGPwa/+av/7JEmSNMgM5jpmnQ5cddXs4fwzn4E3vME7hEqSJB0pg7mOy/r18F//K7z+9Qeeu/VWeN3rLG2RJEk6EgZzHbfpmvN3vevAc5lVacvy5U4MlSRJOhSDuebMn/5ptdb5bL797Wpi6BlnWN4iSZI0G4O55tRHPwp//McQMfv5r38dfv7nrT+XJEmayWCuObd+Pfz1X89edz7t1lurgP6Sl8Db3mZIlyRJMphrXkzXnf+3/1aVrxzMY4/BF79oSJckSTKYa151OnDXXVV5y0/+5KGvnRnS3/AG+M3fNKhLkqThYDBXX6xfX00APZKADlVIv/VW+M//+dmgvny5I+qSJKm9IjPr7kMjjI6O5uTkZN3dGBqbNsEVV8D3vge7dx/941/84upnz55qoulLXworV8LatdUovSRJUlNFxPbMHD2g3WBeMZjX53hD+kwrVsCCBfCc5zwb3BctgoUL4cILq9F7SZKkuhjMD8Ng3gzTIX3Pniqoz8e/nkuWVMF9OrD3hvennz4w0D/9NLz85dUa7Y7GS5Kk42UwPwyDefNMTMCWLXDfffCd78APfjA3I+rH48Uvhuc+9+Ah/kj2px9n4JckaTgZzA/DYD4YpkfU/+f/fDbwPv10NVl0kPUG/sMF+qMJ/4e69gUvgIsusrRHkqR+M5gfhsF8sE1MwMc+Bg888OxodG8w3bsXdu6su5fNdPLJ8MIXzs1fAebycU7olSS1lcH8MAzm7bdpE1xzDUxNHXlQPPFEePDBunuug03oreuXhmO91l82JElgMD8sg7kOZmat+1wEvr17DfzDrl+/bDTpryBHc+0ZZzj/QlJ7GcwPw2CufjtY4O9HGHvsscGvy9dwmHnPgib80lD345rct7l+T86FUVsZzA/DYK5hM12Xf9ddzfsfdxsm9EqaO71zYZryS0MTHmffjq9vdd7f5GDBfEH/uyKpCTod+MIX6u7FwR1uQm8T/yN/qGv9ZUM6drt3179crtrpjjuqbVP+KmMwl9RITf/F4VjU8cvGIP5C04R7FkgaHjfeaDCXpKHTxl825sts9yxowi8NTXhck/s2l+/JuTDql7e/ve4ePMtgLklqnPXrmzOCpfrMNhemCb80NOVx9m1wa8wPxmAuSZIayb8yadicUHcHJEmSJBnMJUmSpEYwmEuSJEkNYDCXJEmSGqC1wTwiVkfEAxGxIyIuqbs/kiRJ0qG0MphHxAjwceA8YCXwzohYWW+vJEmSpINrZTAHzgJ2ZOZDmTkFXAesqblPkiRJ0kG1NZgvAR7uOd5Z2vYTEesjYjIiJnft2tW3zkmSJEkztTWYH5HM3JSZo5k5unjx4rq7I0mSpCHW1mD+CHB6z/HS0iZJkiQ1UluD+Z3AiohYHhELgQuArTX3SZIkSTqoBXV3YD5k5t6I+ABwMzACXJuZ99bcLUmSJOmgWhnMATLzS8CX6u6HJEmSdCTaWsoiSZIkDRSDuSRJktQABnNJkiSpASIz6+5DI0TELuA7Nbz0qcD/qOF11V9+zsPBz3k4+DkPBz/n4VDX5/yTmXnATXQM5jWLiMnMHK27H5pffs7Dwc95OPg5Dwc/5+HQtM/ZUhZJkiSpAQzmkiRJUgMYzOu3qe4OqC/8nIeDn/Nw8HMeDn7Ow6FRn7M15pIkSVIDOGIuSZIkNYDBvEYRsToiHoiIHRFxSd390bGLiNMj4paIuC8i7o2Ii0r7yRGxLSIeLNuTSntExJXls787Is6s9x3oSEXESETcFRF/Vo6XR8Tt5bP8XEQsLO3PKcc7yvlldfZbRy4iFkXEDRHxzYi4PyI6fpfbJyJ+p/z3+hsR8dmIeK7f58EXEddGxOMR8Y2etqP+/kbEunL9gxGxrl/9N5jXJCJGgI8D5wErgXdGxMp6e6XjsBf43cxcCZwNvL98npcA45m5Ahgvx1B97ivKz3rgqv53WcfoIuD+nuOPApdn5suAPcCFpf1CYE9pv7xcp8HwR8CfZ+bPAK+i+rz9LrdIRCwBfhsYzcxXACPABfh9boNPA6tntB3V9zciTgYuBV4LnAVcOh3m55vBvD5nATsy86HMnAKuA9bU3Ccdo8x8NDO/VvZ/QPU/8iVUn+nmctlm4PyyvwbYkpXbgEUR8ZI+d1tHKSKWAv8cuLocB/BG4IZyyczPePqzvwFYVa5Xg0XEi4DXA9cAZOZUZn4fv8tttAB4XkQsAJ4PPIrf54GXmbcCu2c0H+3391xgW2buzsw9wDYODPvzwmBenyXAwz3HO0ubBlz5E+ergduB0zLz0XLqMeC0su/nP5iuAC4GninHpwDfz8y95bj3c/zRZ1zOP1muV7MtB3YBnyolS1dHxAvwu9wqmfkI8O+A71IF8ieB7fh9bquj/f7W9r02mEtzKCJ+DLgR+GBmPtV7LqslkFwGaUBFxFuAxzNze9190bxaAJwJXJWZrwb+jmf/7A34XW6DUpawhuoXsZ8AXkCfRkRVr6Z/fw3m9XkEOL3neGlp04CKiBOpQvlnMvPzpfl703/WLtvHS7uf/+B5HfDWiPg2VenZG6lqkReVP4XD/p/jjz7jcv5FwBP97LCOyU5gZ2beXo5voArqfpfb5U3AtzJzV2b+EPg81Xfc73M7He33t7bvtcG8PncCK8oM8IVUk0621twnHaNSa3gNcH9m/oeeU1uB6dnc64CbetrXlhnhZwNP9vyZTQ2UmRsyc2lmLqP6vn4lM98F3AK8o1w28zOe/uzfUa5v7CiNKpn5GPBwRLy8NK0C7sPvctt8Fzg7Ip5f/vs9/Tn7fW6no/3+3gycExEnlb+unFPa5p03GKpRRLyZqmZ1BLg2M/9tzV3SMYqIXwC+CtzDs/XH/5Kqzvx64KXAd4Bfzszd5X8E/4nqT6d/D/x6Zk72veM6JhExBvxeZr4lIn6KagT9ZOAu4Fcz8+mIeC7wJ1TzDXYDF2TmQ3X1WUcuIs6gmuC7EHgI+HWqgSy/yy0SEf838CtUq2rdBfzvVHXEfp8HWER8FhgDTgW+R7W6yhc5yu9vRLyb6v/jAP82Mz/Vl/4bzCVJkqT6WcoiSZIkNYDBXJIkSWoAg7kkSZLUAAZzSZIkqQEM5pIkSVIDGMwlqc8iIiPi3/cc/15E/Js5eu5PR8Q7Dn/lcb/OL0XE/RFxy4z2n4iIG8r+GWVZ2Ll6zUUR8b7ZXkuS2sBgLkn99zTwv0TEqXV3pFfPHQ+PxIXAezLzF3sbM/P/zczpXwzOAI4qmB+mD4uAHwXzGa8lSQPPYC5J/bcX2AT8zswTM0e8I+L/K9uxiPiriLgpIh6KiI9ExLsi4o6IuCcifrrnad4UEZMR8d8j4i3l8SMR8YcRcWdE3B0Rv9HzvF+NiK1Udz6c2Z93luf/RkR8tLT9a+AXgGsi4g9nXL+sXLsQ+DDwKxHxNxHxKxHxgoi4tvT5rohYUx7zaxGxNSK+AoxHxI9FxHhEfK289pry9B8Bfro83x9Ov1Z5judGxKfK9XdFxC/2PPfnI+LPI+LBiPhYzz+PT5e+3hMRB3wWktRvRzM6IkmaOx8H7p4OikfoVcA/obrz4EPA1Zl5VkRcBPwW8MFy3TLgLOCngVsi4mXAWqrbTf9cRDwH+OuI+Ity/ZnAKzLzW70vFhE/AXwUeA2wB/iLiDg/Mz8cEW+kuvvprHe5zMypEuBHM/MD5fn+gOpW5u+OiEXAHRHxlz19+KflbnwLgLdl5lPlrwq3lV8cLin9PKM837Kel3x/9bL5yoj4mdLXf1zOnUF1x8angQci4j8C/whYkpmvKM+16DD/7CVp3jliLkk1yMyngC3Abx/Fw+7MzEcz82ngb4HpYH0PVRifdn1mPpOZD1IF+J8BzgHWRsTfALcDpwAryvV3zAzlxc8B3czclZl7gc8Arz+K/s50DnBJ6UMXeC7VLbIBtmXm7rIfwB9ExN3AX1LdJv20wzz3LwB/CpCZ36S67fZ0MB/PzCcz8x+o/irwk1T/XH4qIv5jRKwGnjqO9yVJc8IRc0mqzxXA14BP9bTtpQyaRMQJwMKec0/37D/Tc/wM+//3PGe8TlKF3d/KzJt7T0TEGPB3x9b9oxbA2zPzgRl9eO2MPrwLWAy8JjN/GBHfpgrxx6r3n9s+YEFm7omIVwHnAu8Ffhl493G8hiQdN0fMJakmZYT4eqqJlNO+TVU6AvBW4MRjeOpfiogTSt35TwEPADcDvxkRJwJExD+OiBcc5nnuAN4QEadGxAjwTuCvjqIfPwB+vOf4ZuC3IiJKH159kMe9CHi8hPJfpBrhnu35en2VKtBTSlheSvW+Z1VKZE7IzBuB/4uqlEaSamUwl6R6/Xugd3WWT1KF4a8DHY5tNPu7VKH6y8B7SwnH1VRlHF8rEyb/mMP81TQzH6Wq674F+DqwPTNvOop+3AKsnJ78CVxG9YvG3RFxbzmezWeA0Yi4h6o2/pulP09Q1cZ/Y+akU+ATwAnlMZ8Dfq2U/BzMEqBbymr+FNhwFO9LkuZFZM78i6ckSZKkfnPEXJIkSWoAg7kkSZLUAAZzSZIkqQEM5pIkSVIDGMwlSZKkBjCYS5IkSQ1gMJckSZIawGAuSZIkNcD/DyD4ld7kXITOAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 864x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IG5tWAy-FCaW"
      },
      "source": [
        "Zooming in..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "WZ7BoFHy8kTk",
        "outputId": "d45bf65f-b63b-44b3-f498-2b1b85bd2d5d"
      },
      "source": [
        "fig,ax = plt.subplots(figsize=(10,8))\n",
        "ax.plot(range(200), cost_history[:200], 'b.')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f5a38811ad0>]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAHSCAYAAAC3lFz5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3db7Ck1X0f+O+PweA/iQRCE1YLTMAxcQonmxjfkrkVxzXlUdCg9RolcblQXMXEokRSlhIrzpYDcWmhTFJEya7lUJFJ4SAJXIqQotglKpGC0Viz9ouLrEGWhJAsM5YsMRQSBLCUWiWMgbMvnueKZnT/9r19++nuz6fqVneffrr7PPTM3C/nPL9zqrUWAACG5YxpdwAAgG8npAEADJCQBgAwQEIaAMAACWkAAAMkpAEADNCZ0+7AbnvlK1/ZLr744ml3AwBgUw8++OB/a63tX+u5uQtpF198cY4fPz7tbgAAbKqqvrzec6Y7AQAGSEgDABggIQ0AYICENACAARLSAAAGSEgDABggIQ0AYICENACAARLSAAAGSEgDABggIQ0AYICENACAARLSAAAGSEgDABggIQ0AYICENACAARLStmllJbn11u4WAGBSzpx2B2bJykpy6FBy6lRy1lnJ0aPJ8vK0ewUAzCMjadtw7FgX0J5/vrs9dmzaPQIA5pWQtg0HD3YjaPv2dbcHD067RwDAvDLduQ3Ly90U57FjXUAz1QkATIqQtk3Ly8IZADB5pjsBAAZISAMAGCAhDQBggIQ0AIABEtIAAAZISAMAGCAhDQBggIQ0AIABEtIAAAZISAMAGCAhbQdWVpJbb+1uAQB2k707x7Sykhw6lJw6lZx1Vrfxuj09AYDdYiRtTMeOdQHt+ee722PHpt0jAGCeCGljOniwG0Hbt6+7PXhw2j0CAOaJ6c4xLS93U5zHjnUBzVQnALCbhLQdWF4WzgCAyTDdCQAwQJuGtKp6V1U9UVWfXeO5f1JVrape2T+uqrqtqk5U1Weq6vKRY49U1SP9z5GR9h+qqof619xWVdW3v6Kq7u+Pv7+qzt2dUwYAGL6tjKS9J8nh0xur6qIkVyb5ykjzVUku7X+uT3J7f+wrktyU5IeTvDrJTSOh6/Ykbxp53epn3ZDkaGvt0iRH+8cAAAth05DWWvudJE+v8dQ7kvxCkjbSdnWSu1vngSTnVNWrkrw2yf2ttadba88kuT/J4f65l7XWHmittSR3J3n9yHvd1d+/a6QdAGDujXVNWlVdneSx1tqnT3vqgiSPjjw+2bdt1H5yjfYkOb+19nh//6tJzh+nrwAAs2jb1Z1V9d1J/lm6qc490VprVdXWe76qrk83vZoDBw7sVbcAACZmnJG0v5DkkiSfrqo/TnJhkk9W1f+S5LEkF40ce2HftlH7hWu0J8nX+unQ9LdPrNeh1todrbWl1trS/v37xzglAIBh2XZIa6091Fr7c621i1trF6ebory8tfbVJPcmubav8rwiydf7Kcv7klxZVef2BQNXJrmvf+4bVXVFX9V5bZIP9R91b5LVKtAjI+0AAHNvK0twvC/JSpLvr6qTVXXdBod/OMkXk5xI8mtJfjZJWmtPJ7klySf6n1/q29If8+/71/xRko/07f8yyd+sqkeSvKZ/DACwEKorqpwfS0tL7fjx43v6mSsrtocCALavqh5srS2t9ZxtoXZoZSU5dCg5darbaP3oUUENANg520Lt0LFjXUB7/vnu9tixafcIAJgHQtoOHTzYjaDt29fdHjw47R4BAPPAdOcOLS93U5yuSQMAdpOQtguWl4UzAGB3me4EABggIQ0AYICENACAARLSAAAGSEgDABggIQ0AYICENACAARLSAAAGSEgDABggIW2Xrawkt97a3QIAjMu2ULtoZSU5dCg5darbbP3oUdtFAQDjMZK2i44d6wLa8893t8eOTbtHAMCsEtJ20cGD3Qjavn3d7cGD0+4RADCrTHfuouXlborz2LEuoJnqBADGJaTtsuVl4QwA2DnTnQAAAySkAQAMkJAGADBAQhoAwAAJaQAAAySkAQAMkJAGADBAQhoAwAAJaRO0spLcemt3CwCwHXYcmJCVleTQoW6j9bPO6raLshMBALBVRtIm5NixLqA9/3x3e+zYtHsEAMwSIW1CDh7sRtD27etuDx6cdo8AgFliunNClpe7Kc5jx7qAZqoTANgOIW2ClpeFMwBgPKY7AQAGSEgDABggIQ0AYICENACAARLSAAAGSEgDABggIQ0AYICEtD1go3UAYLssZjthNloHAMZhJG3CbLQOAIxDSJswG60DAOMw3TlhNloHAMax6UhaVb2rqp6oqs+OtP3rqvqDqvpMVf1mVZ0z8tyNVXWiqr5QVa8daT/ct52oqhtG2i+pqo/37e+vqrP69rP7xyf65y/erZPea8vLyY03CmgAwNZtZbrzPUkOn9Z2f5K/3Fr735L8YZIbk6SqLktyTZIf6F/zq1W1r6r2JXlnkquSXJbkDf2xSfL2JO9orX1fkmeSXNe3X5fkmb79Hf1xAAALYdOQ1lr7nSRPn9b2W6215/qHDyS5sL9/dZJ7WmvPtta+lOREklf3Pydaa19srZ1Kck+Sq6uqkvxYkg/2r78ryetH3uuu/v4HkxzqjwcAmHu7UTjwxiQf6e9fkOTRkedO9m3rtZ+X5E9GAt9q+0veq3/+6/3xAABzb0chrap+MclzSd67O90Zux/XV9Xxqjr+5JNPTrMrAAC7YuyQVlV/L8mPJ/np1lrrmx9LctHIYRf2beu1P5XknKo687T2l7xX//zL++O/TWvtjtbaUmttaf/+/eOeEgDAYIwV0qrqcJJfSPITrbVvjjx1b5Jr+srMS5JcmuT3knwiyaV9JedZ6YoL7u3D3ceS/GT/+iNJPjTyXkf6+z+Z5LdHwiAAwFzbdJ20qnpfkoNJXllVJ5PclK6a8+wk9/fX8j/QWvsHrbWHq+oDST6Xbhr0za215/v3eUuS+5LsS/Ku1trD/Uf80yT3VNU/T/L7Se7s2+9M8utVdSJd4cI1u3C+AAAzoeZtcGppaakdP3582t1Y18qKhW0BgE5VPdhaW1rrOTsO7CGbrQMAW2Xvzj1ks3UAYKuEtD1ks3UAYKtMd+4hm60DAFslpO2x5WXhDADYnOlOAIABEtIAAAZISAMAGCAhDQBggIQ0AIABEtIAAAZISAMAGCAhbUpWVpJbb+1uAQBOZzHbKbDROgCwGSNpU2CjdQBgM0LaFNhoHQDYjOnOKbDROgCwGSFtSmy0DgBsxHQnAMAACWkAAAMkpAEADJCQBgAwQEIaAMAACWkAAAMkpA2AfTwBgNNZJ23K7OMJAKzFSNqU2ccTAFiLkDZl9vEEANZiunPK7OMJAKxFSBsA+3gCAKcz3QkAMEBCGgDAAAlpAAADJKQBAAyQkAYAMEBC2oDYHgoAWGUJjoGwPRQAMMpI2kDYHgoAGCWkDYTtoQCAUaY7B8L2UADAKCFtQGwPBQCsMt0JADBAQhoAwAAJaQAAAySkAQAMkJAGADBAQtpA2SIKABbbpiGtqt5VVU9U1WdH2l5RVfdX1SP97bl9e1XVbVV1oqo+U1WXj7zmSH/8I1V1ZKT9h6rqof41t1VVbfQZi2B1i6i3va27FdQAYPFsZSTtPUkOn9Z2Q5KjrbVLkxztHyfJVUku7X+uT3J70gWuJDcl+eEkr05y00jouj3Jm0Zed3iTz5h7togCADYNaa2130ny9GnNVye5q79/V5LXj7Tf3ToPJDmnql6V5LVJ7m+tPd1aeybJ/UkO98+9rLX2QGutJbn7tPda6zPmni2iAIBxdxw4v7X2eH//q0nO7+9fkOTRkeNO9m0btZ9co32jz5h7togCAHa8LVRrrVVV243OjPsZVXV9uunVHDhwYJJd2TO2iAKAxTZudefX+qnK9LdP9O2PJblo5LgL+7aN2i9co32jz/g2rbU7WmtLrbWl/fv3j3lKAADDMW5IuzfJaoXmkSQfGmm/tq/yvCLJ1/spy/uSXFlV5/YFA1cmua9/7htVdUVf1Xntae+11mcAAMy9Tac7q+p9SQ4meWVVnUxXpfkvk3ygqq5L8uUkP9Uf/uEkr0tyIsk3k/xMkrTWnq6qW5J8oj/ul1prq8UIP5uugvS7knyk/8kGnwEAMPeqK6qcH0tLS+348ePT7gYAwKaq6sHW2tJaz9lxAABggIS0gbM9FAAsph0vwcHkrG4PdepUt6jt0aOW5QCARWEkbcBsDwUAi0tIGzDbQwHA4jLdOWC2hwKAxSWkDZztoQBgMZnuBAAYICENAGCAhDQAgAES0gAABkhImyF2HwCAxaG6c0bYfQAAFouRtBlh9wEAWCxC2oyw+wAALBbTnTPC7gMAsFiEtBli9wEAWBymOwEABkhIAwAYICENAGCAhLQZZFFbAJh/CgdmjEVtAWAxGEmbMRa1BYDFIKTNGIvaAsBiMN05YyxqCwCLQUibQRa1BYD5Z7oTAGCAhDQAgAES0gAABkhIAwAYICFtxtl9AADmk+rOGWb3AQCYX0bSZpjdBwBgfglpM8zuAwAwv0x3zjC7DwDA/BLSZpzdBwBgPpnuBAAYICENAGCAhLQ5Ys00AJgfrkmbE9ZMA4D5YiRtTlgzDQDmi5A2J6yZBgDzxXTnnLBmGgDMFyFtjlgzDQDmh+lOAIABEtIAAAZoRyGtqv5xVT1cVZ+tqvdV1XdW1SVV9fGqOlFV76+qs/pjz+4fn+ifv3jkfW7s279QVa8daT/ct52oqht20lcAgFkydkirqguS/KMkS621v5xkX5Jrkrw9yTtaa9+X5Jkk1/UvuS7JM337O/rjUlWX9a/7gSSHk/xqVe2rqn1J3pnkqiSXJXlDfyybsKgtAMy+nRYOnJnku6rqT5N8d5LHk/xYkr/bP39XkpuT3J7k6v5+knwwyb+tqurb72mtPZvkS1V1Ismr++NOtNa+mCRVdU9/7Od22Oe5ZlFbAJgPY4+ktdYeS/J/J/lKunD29SQPJvmT1tpz/WEnk1zQ378gyaP9a5/rjz9vtP2016zXzgYsagsA82En053nphvZuiTJ/5rke9JNV+65qrq+qo5X1fEnn3xyGl0YDIvaAsB82Ml052uSfKm19mSSVNVvJPnrSc6pqjP70bILkzzWH/9YkouSnKyqM5O8PMlTI+2rRl+zXvtLtNbuSHJHkiwtLbUdnNPMs6gtAMyHnYS0ryS5oqq+O8n/SHIoyfEkH0vyk0nuSXIkyYf64+/tH6/0z/92a61V1b1J/kNV/XK6EblLk/xekkpyaVVdki6cXZMXr3VjAxa1BYDZN3ZIa619vKo+mOSTSZ5L8vvpRrP+S5J7quqf92139i+5M8mv94UBT6cLXWmtPVxVH0hXEPBckje31p5Pkqp6S5L70lWOvqu19vC4/QUAmCXV2nzNDi4tLbXjx49PuxsAAJuqqgdba0trPWfHgTlnzTQAmE02WJ9j1kwDgNllJG2OWTMNAGaXkDbHrJkGALPLdOccs2YaAMwuIW3OWTMNAGaT6U4AgAES0gAABkhIWxDWSwOA2eKatAVgvTQAmD1G0haA9dIAYPYIaQvAemkAMHtMdy4A66UBwOwR0haE9dIAYLaY7gQAGCAhDQBggIS0BWTNNAAYPtekLRhrpgHAbDCStmCsmQYAs0FIWzDWTAOA2WC6c8FYMw0AZoOQtoCsmQYAw2e6EwBggIS0BWYpDgAYLtOdC8pSHAAwbEbSFpSlOABg2IS0BWUpDgAYNtOdC8pSHAAwbELaArMUBwAMl+lOAIABEtJIYjkOABga051YjgMABshIGpbjAIABEtKwHAcADJDpTizHAQADJKSRxHIcADA0pjsBAAZISOMlLMUBAMNgupNvsRQHAAyHkTS+xVIcADAcQhrfYikOABgO0518i6U4AGA4hDRewlIcADAMpjtZl0pPAJgeI2msSaUnAEyXkTTWpNITAKZLSGNNKj0BYLp2FNKq6pyq+mBV/UFVfb6qlqvqFVV1f1U90t+e2x9bVXVbVZ2oqs9U1eUj73OkP/6Rqjoy0v5DVfVQ/5rbqqp20l+2brXS85ZbTHUCwDTsdCTt3yT5r621v5Tkryb5fJIbkhxtrV2a5Gj/OEmuSnJp/3N9ktuTpKpekeSmJD+c5NVJbloNdv0xbxp53eEd9pdtWF5ObrxRQAOAaRg7pFXVy5P8aJI7k6S1dqq19idJrk5yV3/YXUle39+/OsndrfNAknOq6lVJXpvk/tba0621Z5Lcn+Rw/9zLWmsPtNZakrtH3os9pMoTAPbeTqo7L0nyZJJ3V9VfTfJgkp9Lcn5r7fH+mK8mOb+/f0GSR0def7Jv26j95Brt36aqrk83OpcDBw6Mf0Z8G1WeADAdO5nuPDPJ5Ulub639YJL/Ly9ObSZJ+hGwtoPP2JLW2h2ttaXW2tL+/fsn/XELRZUnAEzHTkLaySQnW2sf7x9/MF1o+1o/VZn+9on++ceSXDTy+gv7to3aL1yjnT2kyhMApmPskNZa+2qSR6vq+/umQ0k+l+TeJKsVmkeSfKi/f2+Sa/sqzyuSfL2fFr0vyZVVdW5fMHBlkvv6575RVVf0VZ3XjrwXe0SVJwBMx053HPiHSd5bVWcl+WKSn0kX/D5QVdcl+XKSn+qP/XCS1yU5keSb/bFprT1dVbck+UR/3C+11p7u7/9skvck+a4kH+l/2GP28wSAvVfdZWPzY2lpqR0/fnza3ZhbKyvddWkHDwpuALBTVfVga21prefs3cmWqfQEgL1jWyi2TKUnAOwdIY0tU+kJAHvHdCdbtlrp6Zo0AJg8IY1tUekJAHvDdCdjs6cnAEyOkTTGotITACbLSBpjUekJAJMlpDEWlZ4AMFmmOxmLSk8AmCwjaYxteTm58cbuvgICANhdRtLYEQUEADAZRtLYEQUEADAZQho7ooAAACbDdCc7ooAAACZDSGPHRreKWlkR2ABgNwhp7BpFBACwe1yTxq5RRAAAu0dIY9coIgCA3WO6k12jiAAAdo+RNHaVXQgAYHcYSWPXKSAAgJ0zksauU0AAADsnpLHrFBAAwM6Z7mTXKSAAgJ0T0pgIuxAAwM4IaUyUIgIAGI9r0pgoRQQAMB4hjYlSRAAA4zHdyUQpIgCA8RhJY+LsQgAA22ckjT2hgAAAtsdIGntCAQEAbI+Qxp5QQAAA2yOksSdWCwhuuaW7TVyfBgAbcU0ae2Z1FwLXpwHA5oyksedcnwYAmxPS2HOuTwOAzZnuZM+NLnB73nkvjqSZ8gSAFwlpTMVqIHNtGgCszXQnU+PaNABYn5DG1Lg2DQDWJ6QxNdZOA4D1uSaNqbJ2GgCszUgag+D6NAB4KSGNQXB9GgC81I5DWlXtq6rfr6r/3D++pKo+XlUnqur9VXVW3352//hE//zFI+9xY9/+hap67Uj74b7tRFXdsNO+MlyuTwOAl9qNkbSfS/L5kcdvT/KO1tr3JXkmyXV9+3VJnunb39Efl6q6LMk1SX4gyeEkv9oHv31J3pnkqiSXJXlDfyxzank5ufHG7v6hQ8nb3tbdCmoALKIdhbSqujDJ/57k3/ePK8mPJflgf8hdSV7f37+6f5z++UP98Vcnuae19mxr7UtJTiR5df9zorX2xdbaqST39Mcy51yfBgA7H0n7lSS/kOSF/vF5Sf6ktfZc//hkkgv6+xckeTRJ+ue/3h//rfbTXrNeO3PO9WkAsIMlOKrqx5M80Vp7sKoO7l6XxurL9UmuT5IDBw5MsyvsAnt7AsDO1kn760l+oqpel+Q7k7wsyb9Jck5VndmPll2Y5LH++MeSXJTkZFWdmeTlSZ4aaV81+pr12l+itXZHkjuSZGlpqe3gnBgIe3sCsOjGnu5srd3YWruwtXZxugv/f7u19tNJPpbkJ/vDjiT5UH//3v5x+ud/u7XW+vZr+urPS5JcmuT3knwiyaV9tehZ/WfcO25/mT2uTQNgkU1inbR/muTnq+pEumvO7uzb70xyXt/+80luSJLW2sNJPpDkc0n+a5I3t9ae70fi3pLkvnTVox/oj2VBnH5t2nnnWZYDgMVR3WDW/FhaWmrHjx+fdjfYJSsrL16b9ta3mvoEYL5U1YOttaW1nrPjAIO2unbaU0+Z+gRgsQhpzATLcgCwaHZS3Ql7xrIcACwaIY2ZYVkOABaJ6U5mimU5AFgUQhozxbIcACwK053MlNOvTbMsBwDzykgaM8eyHAAsAiGNmTU69blvX/KVr5j2BGB+CGnMrNWpzze9KalKfu3XuspPQQ2AeSCkMdOWl5MDB5LnnjPtCcB8EdKYeSo+AZhHqjuZeSo+AZhHRtKYCyo+AZg3QhpzRcUnAPNCSGOuqPgEYF4IacwdFZ8AzAMhjbmk4hOAWae6k7mk4hOAWWckjbml4hOAWSakMfdUfAIwi4Q05p6KTwBmkZDGQlir4vPuuxUTADBcCgdYGKvTnqdOddOe7353F9oUEwAwREbSWBir05633JK88Y3WUQNg2IQ0Fspqxee111pHDYBhM93JQrKOGgBDZySNhbXWOmrPPpvcfLMRNQCmT0hj4a0WFJxxRvLCC8lHP2qJDgCmT0hj4a1Ofb7mNS8GNUt0ADBt1Vqbdh921dLSUjt+/Pi0u8EMWlnpRtBWl+ioskQHAJNVVQ+21pbWes5IGvQs0QHAkAhpMGKtJTrs9wnANAhpsAb7fQIwbUIarMN+nwBMk8VsYQP2+wRgWoykwQbWKyaw6C0AkyakwSZOLyaw6C0Ae0FIgy2y6C0Ae8litrBNFr0FYLdYzBZ2kevUANgLRtJgB1ZH1Z59tpv+POOM5Oyzk1/5leSpp7rqUCNrAKzHSBpMyFrXqT37bPKWtyRve5vCAgDGJ6TBDi0vd9OcZ5/dXaN2xhnd9KcFcAHYCYvZwi5YHVE7diw577zkrW+1AC4AOyOkwS5ZXn4xgP2Vv9IFtq98pdv3c7Sw4OabBTUANjf2dGdVXVRVH6uqz1XVw1X1c337K6rq/qp6pL89t2+vqrqtqk5U1Weq6vKR9zrSH/9IVR0Zaf+hqnqof81tVVU7OVnYK5stgHvHHaZAAdjYTq5Jey7JP2mtXZbkiiRvrqrLktyQ5Ghr7dIkR/vHSXJVkkv7n+uT3J50oS7JTUl+OMmrk9y0Guz6Y9408rrDO+gv7DmFBQCMa+yQ1lp7vLX2yf7+f0/y+SQXJLk6yV39YXcleX1//+okd7fOA0nOqapXJXltkvtba0+31p5Jcn+Sw/1zL2utPdC6dULuHnkvmBkbFRZYWw2A9ezKNWlVdXGSH0zy8STnt9Ye75/6apLz+/sXJHl05GUn+7aN2k+u0Q4zZ63CgtW11T760eR3f9faagC81I5DWlX9mST/KclbW2vfGL1srLXWqmriq+VW1fXpplBz4MCBSX8cjOX0woKbb+4C2ugU6AsvqAIFoLOjddKq6jvSBbT3ttZ+o2/+Wj9Vmf72ib79sSQXjbz8wr5to/YL12j/Nq21O1prS621pf379+/klGBPmAIFYDM7qe6sJHcm+Xxr7ZdHnro3yWqF5pEkHxppv7av8rwiydf7adH7klxZVef2BQNXJrmvf+4bVXVF/1nXjrwXzLzRPUDf+c4usKkCBWDV2Ht3VtWPJPndJA8leaFv/mfprkv7QJIDSb6c5Kdaa0/3QevfpqvQ/GaSn2mtHe/f6439a5PkX7TW3t23LyV5T5LvSvKRJP+wbdJhe3cyq1ZWXjoFesYZ3SjbCy90t298Y7ekh2lQgPmx0d6dNliHAVndsP3UqaSqC2gv9P8LVJV853cqMACYJxuFNDsOwICsVQX6P/9n0lr3o8AAYHHYYB0GZnW3guuv70LY3//7CgwAFpHpTpgBKytrr7F2xhldgDMFCjCbTHfCjNvOGmsCG8B8MJIGM2i9AgMVoQCzRXUnzKHTp0BVhALMHiEN5tzpgW21IjQxugYwZEIaLJCVleTuu5N3vzt57jmjawBDJqTBAhpndC3pXiO4AewNIQ0W3FZG177jO7rb554zLQqwV4Q0IMnGo2tV3e3oY9OiAJMlpAHf5vTRtX37umB26pSiA4C9IqQB61odXTt4sHu83aKDxHVsAOMS0oBt2U7RgevYAMYnpAFj26joYLPr2M47z/VsABsR0oAdW2uHg42uY3v++bU3gRfcAF4kpAG7ajvXsSWCG8B6hDRg4k4faXv22RcD2RlnbD+4KUoAFoGQBuyp0cC2OlK2neC2UVFCIrgB80NIA6Zuu8FtraKE9XZFSAQ3YDYJacAgrRfc1itKGCe4ud4NGDIhDZgZGxUlbDe4/emfbl6okBiFA6ZHSANm2k6DW7J2ocKZZ744CnfWWUIcsPeENGDubCW4jY6krVWoMBrmNgtxm02lrrYJc8B2CGnAQjg9uG1WqDAawtZa322rU6mbhbm1+iPMAYmQBpDk2wsVTg9P64W4zaZSt1LQsN1r5IzSwWIQ0gC2YL0Qt9lU6mZhbrvXyG1nlG69MLfZ88IeDIOQBrBDG02lbhTmxr1Gbr22rU657kbY204AFPpgPEIawB5ZK8xt9xq57YzSrdps5G67YW87AXCcqVuhEDpCGsAAbXaN3DhTruNOya4aJwBud+p2FkPhOO8jSLIVQhrAHNjKlOs4U7I7DU3jTt1upW0IoXDc99lqle+k2qb53kLq1glpAHzLVsPeVn85b3fqdtZC4bhtW6ny3Y3/TpMMnNO47nGvA+dmnzfpwCmkATBR25m6nbVQuN332U6V7yTbpv3e41z3uNeBc7PPW92N5OjRyQU1IQ2AubKXoXC7bdup8p3HkbSdXve4Vts0A+6+fckttyQ33piJENIAYA9ttcp3Hq9J28l1j0bSTntOSAMAdtNOrnt0TdrIc0IaAMB0bBTSztjrzgAAsDkhDQBggIQ0AIABEtIAAAZISAMAGCAhDQBggIQ0AIABEtIAAAZISAMAGCAhDQBggAYf0qrqcFV9oapOVNUN0+4PAMBeGHRIq6p9Sd6Z5KoklyV5Q1VdNt1eAQBM3qBDWpJXJznRWvtia+1UknuSXD3lPgEATNzQQ9oFSR4deXyyb3uJqrq+qo5X1fEnn3xyz7ArAvgAAAXVSURBVDoHADApQw9pW9Jau6O1ttRaW9q/f/+0uwMAsGNDD2mPJblo5PGFfRsAwFyr1tq0+7CuqjozyR8mOZQunH0iyd9trT28wWueTPLlCXftlUn+24Q/Y8ic/+Ke/yKfe+L8nf/inv8in3sy2fP/8621NacBz5zQB+6K1tpzVfWWJPcl2ZfkXRsFtP41E5/vrKrjrbWlSX/OUDn/xT3/RT73xPk7/8U9/0U+92R65z/okJYkrbUPJ/nwtPsBALCXhn5NGgDAQhLSxnPHtDswZc5/cS3yuSfO3/kvrkU+92RK5z/owgEAgEVlJA0AYICEtG1apA3fq+qiqvpYVX2uqh6uqp/r22+uqseq6lP9z+um3ddJqao/rqqH+vM83re9oqrur6pH+ttzp93PSaiq7x/5jj9VVd+oqrfO8/dfVe+qqieq6rMjbWt+39W5rf+34DNVdfn0er471jn/f11Vf9Cf429W1Tl9+8VV9T9G/hz8u+n1fOfWOfd1/6xX1Y39d/+FqnrtdHq9e9Y5//ePnPsfV9Wn+vZ5++7X+103/b/7rTU/W/xJtwzIHyX53iRnJfl0ksum3a8Jnu+rklze3/+z6dasuyzJzUn+z2n3b4/+G/xxklee1vavktzQ378hydun3c89+O+wL8lXk/z5ef7+k/xoksuTfHaz7zvJ65J8JEkluSLJx6fd/wmd/5VJzuzvv33k/C8ePW7Wf9Y59zX/rPf/Dn46ydlJLul/L+yb9jns9vmf9vz/k+T/mtPvfr3fdVP/u28kbXsWasP31trjrbVP9vf/e5LPZ429UxfQ1Unu6u/fleT1U+zLXjmU5I9aa5NeKHqqWmu/k+Tp05rX+76vTnJ36zyQ5JyqetXe9HQy1jr/1tpvtdae6x8+kG7nl7mzzne/nquT3NNae7a19qUkJ9L9fphZG51/VVWSn0ryvj3t1B7Z4Hfd1P/uC2nbs6UN3+dRVV2c5AeTfLxveks/zPuueZ3u67Ukv1VVD1bV9X3b+a21x/v7X01y/nS6tqeuyUv/gV6U7z9Z//texH8P3phuBGHVJVX1+1X1/1bV35hWpyZsrT/ri/bd/40kX2utPTLSNpff/Wm/66b+d19IY1NV9WeS/Kckb22tfSPJ7Un+QpK/luTxdMPg8+pHWmuXJ7kqyZur6kdHn2zd2Pdcl0hX1VlJfiLJf+ybFun7f4lF+L7XU1W/mOS5JO/tmx5PcqC19oNJfj7Jf6iql02rfxOysH/WT/OGvPR/0ubyu1/jd923TOvvvpC2PQu34XtVfUe6P7Tvba39RpK01r7WWnu+tfZCkl/LjA/zb6S19lh/+0SS30x3rl9bHdrub5+YXg/3xFVJPtla+1qyWN9/b73ve2H+Paiqv5fkx5P8dP/LKv1U31P9/QfTXZf1F6fWyQnY4M/6In33Zyb520nev9o2j9/9Wr/rMoC/+0La9nwiyaVVdUk/unBNknun3KeJ6a9DuDPJ51trvzzSPjr3/reSfPb0186Dqvqeqvqzq/fTXUD92XTf+ZH+sCNJPjSdHu6Zl/xf9KJ8/yPW+77vTXJtX+l1RZKvj0yNzI2qOpzkF5L8RGvtmyPt+6tqX3//e5NcmuSL0+nlZGzwZ/3eJNdU1dlVdUm6c/+9ve7fHnlNkj9orZ1cbZi3736933UZwt/9aVdVzNpPuqqOP0z3fw6/OO3+TPhcfyTd8O5nknyq/3ldkl9P8lDffm+SV027rxM6/+9NV8H16SQPr37fSc5LcjTJI0k+muQV0+7rBP8bfE+Sp5K8fKRtbr//dGH08SR/mu46k+vW+77TVXa9s/+34KEkS9Pu/4TO/0S6629W/w34d/2xf6f/e/GpJJ9M8n9Mu/8TOPd1/6wn+cX+u/9Ckqum3f9JnH/f/p4k/+C0Y+ftu1/vd93U/+7bcQAAYIBMdwIADJCQBgAwQEIaAMAACWkAAAMkpAEADJCQBgAwQEIaAMAACWkAAAP0/wM6wyBImvbdQwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYhOp3fjnh2G"
      },
      "source": [
        "# Stochastic Gradient Descent"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In Batch Gradient Descent we were considering all the examples for every step of Gradient Descent. But what if our dataset is very huge. Deep learning models crave for data. The more the data the more chances of a model to be good. Suppose our dataset has 5 million examples, then just to take one step the model will have to calculate the gradients of all the 5 million examples. This does not seem an efficient way. To tackle this problem we have Stochastic Gradient Descent. In Stochastic Gradient Descent (SGD), we consider just one example at a time to take a single step."
      ],
      "metadata": {
        "id": "10N2dcwWUctJ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aVwD7Cqw8m1d"
      },
      "source": [
        "def stocashtic_gradient_descent(X, y, y_pred, learning_rate=0.01, iterations=10):\n",
        "    '''\n",
        "    X = Matrix of X with added bias units\n",
        "    y = Vector of Y\n",
        "    y_pred = Vector of y_pred np.random.randn(j,1)\n",
        "    learning_rate \n",
        "    iterations = no of iterations\n",
        "    \n",
        "    Returns the final y_pred vector and array of cost history over no of iterations\n",
        "    '''\n",
        "\n",
        "    m = len(y)\n",
        "    cost_history = np.zeros(iterations)\n",
        "    \n",
        "    for it in range(iterations):\n",
        "        cost = 0.0\n",
        "        \n",
        "        for i in range(m):\n",
        "            rand_ind = np.random.randint(0,m)\n",
        "            X_i = X[rand_ind, :].reshape(1, X.shape[1])\n",
        "            y_i = y[rand_ind].reshape(1,1)\n",
        "            prediction = np.dot(X_i, y_pred)\n",
        "\n",
        "            y_pred = y_pred - (1 / m) * learning_rate *(X_i.T.dot((prediction - y_i)))\n",
        "            cost += cal_cost(y_pred, X_i, y_i)\n",
        "\n",
        "        cost_history[it]  = cost\n",
        "        \n",
        "    return y_pred, cost_history"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Yk6pfB5c8tPz",
        "outputId": "5db37ffc-954f-42df-ac35-9608c9ddda63"
      },
      "source": [
        "lr = 0.5\n",
        "n_iter = 50\n",
        "y_pred = np.random.randn(2, 1)\n",
        "X_b = np.c_[np.ones((len(X), 1)), X]\n",
        "y_pred, cost_history = stocashtic_gradient_descent(X_b, y, y_pred, lr, n_iter)\n",
        "\n",
        "print('y_pred[0]: {:0.3f}\\ny_pred[1]: {:0.3f}'.format(y_pred[0][0], y_pred[1][0]))\n",
        "print('Final error: {:0.3f}'.format(cost_history[-1]))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "y_pred[0]: 4.121\n",
            "y_pred[1]: 2.755\n",
            "Final error: 41.313\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 515
        },
        "id": "YiJUgS7o8u2e",
        "outputId": "a2392101-2c7d-4231-ff5a-714f63f15611"
      },
      "source": [
        "fig, ax = plt.subplots(figsize=(10,8))\n",
        "\n",
        "ax.set_ylabel('Error')\n",
        "ax.set_xlabel('Number of iterations')\n",
        "y_pred = np.random.randn(2,1)\n",
        "\n",
        "ax.plot(range(n_iter), cost_history, 'b.')"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f5a3879a210>]"
            ]
          },
          "metadata": {},
          "execution_count": 12
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAHgCAYAAAAL2HHvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeC0lEQVR4nO3df7Bmd10f8Pcnu0QcpQTImsEkuCBxKGMl4EqzlbELqTagQ2gFhKElaqbRFm2sWgxOp1ZtCdRR/DHWaQpIqKikKE2GoUJcsmo7y48NCQkQKJFCSRrICiFgHUmTfPrHc1Yvy/64d3PP/T733tdrZuc5v+55Pvc5d8993+/3e86p7g4AAOOcNroAAIDtTiADABhMIAMAGEwgAwAYTCADABhMIAMAGGzn6AIeijPPPLN37949ugwAgJO68cYb/6y7dx1r3aYOZLt3786hQ4dGlwEAcFJV9cnjrdNlCQAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEB2AgcPJldeuXgFAJjLztEFLKuDB5MLL0zuuy85/fRk//5k797RVQEAW5EWsuM4cGARxh54YPF64MDoigCArUogO459+xYtYzt2LF737RtdEQCwVemyPI69exfdlAcOLMKY7koAYC4C2Qns3SuIAQDzm7XLsqo+UVW3VtXNVXVoWvboqrq+qj42vT5qWl5V9atVdXtV3VJVT5uzNgCAZbERY8ie2d3nd/eeaf6KJPu7+7wk+6f5JHl2kvOmf5cl+Y0NqA0AYLgRg/ovTnL1NH11kuetWP7GXnh3kjOq6rED6gMA2FBzB7JO8s6qurGqLpuWndXdd03Tn05y1jR9dpJPrfjaO6ZlAABb2tyD+p/R3XdW1dclub6qPrJyZXd3VfVadjgFu8uS5HGPe9z6VQoAMMisLWTdfef0eneStyZ5epLPHOmKnF7vnja/M8m5K778nGnZ0fu8qrv3dPeeXbt2zVk+AMCGmC2QVdXXVNUjjkwn+a4kH0xyXZJLps0uSXLtNH1dkpdOV1tekOTeFV2bAABb1pxdlmcleWtVHXmf3+7uP6iq9yW5pqouTfLJJC+ctn97kuckuT3JXyT5gRlrAwBYGrMFsu7+eJKnHGP5Z5NceIzlneRlc9UDALCsPMsSAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGCw2QNZVe2oqpuq6m3T/OOr6j1VdXtVvbmqTp+Wf9U0f/u0fvfctQEALIONaCG7PMltK+ZfneQ13f3EJPckuXRafmmSe6blr5m2AwDY8mYNZFV1TpLvTvLaab6SPCvJW6ZNrk7yvGn64mk+0/oLp+0BALa0uVvIfjnJy5M8OM0/Jsnnu/v+af6OJGdP02cn+VSSTOvvnbb/MlV1WVUdqqpDhw8fnrN2AIANMVsgq6rvSXJ3d9+4nvvt7qu6e09379m1a9d67hoAYIidM+7725M8t6qek+ThSf5Gkl9JckZV7Zxawc5Jcue0/Z1Jzk1yR1XtTPLIJJ+dsT4AgKUwWwtZd7+iu8/p7t1JXpTkXd39kiQ3JHn+tNklSa6dpq+b5jOtf1d391z1AQAsixH3IfupJD9eVbdnMUbsddPy1yV5zLT8x5NcMaA2AIANN2eX5V/p7gNJDkzTH0/y9GNs85dJXrAR9QAALBN36gcAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhMIAMAGEwgAwAYTCADABhstkBWVQ+vqvdW1Qeq6kNV9bPT8sdX1Xuq6vaqenNVnT4t/6pp/vZp/e65agMAWCZztpB9KcmzuvspSc5PclFVXZDk1Ule091PTHJPkkun7S9Ncs+0/DXTdgAAW95sgawX/nyafdj0r5M8K8lbpuVXJ3neNH3xNJ9p/YVVVXPVBwCwLGYdQ1ZVO6rq5iR3J7k+yZ8m+Xx33z9tckeSs6fps5N8Kkmm9fcmecyc9QEALINZA1l3P9Dd5yc5J8nTkzzpoe6zqi6rqkNVdejw4cMPuUYAgNE25CrL7v58khuS7E1yRlXtnFadk+TOafrOJOcmybT+kUk+e4x9XdXde7p7z65du2avHQBgbnNeZbmrqs6Ypr86yXcmuS2LYPb8abNLklw7TV83zWda/67u7rnqAwBYFjtPvskpe2ySq6tqRxbB75rufltVfTjJ71bVv01yU5LXTdu/Lsl/rqrbk3wuyYtmrA0AYGnMFsi6+5YkTz3G8o9nMZ7s6OV/meQFc9UDALCs3KkfAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYLCTBrKqOq2q/s5GFAMAsB2dNJB194NJfn0DagEA2JZW22W5v6q+t6pq1moAALah1QayH0ryX5LcV1VfqKovVtUXZqwLAGDb2Lmajbr7EXMXAgCwXa0qkCVJVT03yXdMswe6+23zlAQAsL2sqsuyql6V5PIkH57+XV5VV85ZGADAdrHaFrLnJDl/uuIyVXV1kpuSvGKuwgAAtou13Bj2jBXTj1zvQgAAtqvVtpC9MslNVXVDkspiLNkVs1UFALCNnDSQVdVpSR5MckGSb5sW/1R3f3rOwgAAtouTBrLufrCqXt7d1yS5bgNqAgDYVlY7huwPq+onq+rcqnr0kX+zVgYAsE2sdgzZ902vL1uxrJM8YX3LAQDYflY7huyK7n7zBtQDALDtnLTLcrr32L/cgFoAALYlY8gAAAYzhgwAYLBVBbLufvzchQAAbFcn7LKsqpevmH7BUeteOVdRAADbycnGkL1oxfTRDxK/aJ1rAQDYlk4WyOo408eaBwDgFJwskPVxpo81DwDAKTjZoP6nVNUXsmgN++ppOtP8w2etDABgmzhhIOvuHRtVCADAdrXaG8MCADATgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgsNkCWVWdW1U3VNWHq+pDVXX5tPzRVXV9VX1sen3UtLyq6ler6vaquqWqnjZXbQAAy2TOFrL7k/xEdz85yQVJXlZVT05yRZL93X1ekv3TfJI8O8l507/LkvzGjLUBACyN2QJZd9/V3e+fpr+Y5LYkZye5OMnV02ZXJ3neNH1xkjf2wruTnFFVj52rPgCAZbEhY8iqaneSpyZ5T5KzuvuuadWnk5w1TZ+d5FMrvuyOaRkAwJY2eyCrqq9N8ntJfqy7v7ByXXd3kl7j/i6rqkNVdejw4cPrWCkAwBizBrKqelgWYexN3f370+LPHOmKnF7vnpbfmeTcFV9+zrTsy3T3Vd29p7v37Nq1a77iAQA2yJxXWVaS1yW5rbt/acWq65JcMk1fkuTaFctfOl1teUGSe1d0bQIAbFk7Z9z3tyf5x0luraqbp2U/neRVSa6pqkuTfDLJC6d1b0/ynCS3J/mLJD8wY20AAEtjtkDW3f89SR1n9YXH2L6TvGyuegAAlpU79QMADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkAEADCaQAQAMJpABAAwmkK2TgweTK69cvAIArMXO0QVsBQcPJhdemNx3X3L66cn+/cnevaOrAgA2Cy1k6+DAgUUYe+CBxeuBA6MrAgA2E4FsHezbt2gZ27Fj8bpv3+iKAIDNRJflOti7d9FNeeDAIozprgQA1kIgWyd79wpiAMCp0WUJADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAwmEAGADCYQAYAMJhABgAw2GyBrKpeX1V3V9UHVyx7dFVdX1Ufm14fNS2vqvrVqrq9qm6pqqfNVRcAwLKZs4XsDUkuOmrZFUn2d/d5SfZP80ny7CTnTf8uS/IbM9YFALBUZgtk3f3HST531OKLk1w9TV+d5Hkrlr+xF96d5IyqeuxctQEALJONHkN2VnffNU1/OslZ0/TZST61Yrs7pmUAAFvesEH93d1Jeq1fV1WXVdWhqjp0+PDhGSoDANhYGx3IPnOkK3J6vXtafmeSc1dsd8607Ct091Xdvae79+zatWvWYgEANsJGB7LrklwyTV+S5NoVy186XW15QZJ7V3RtAgBsaTvn2nFV/U6SfUnOrKo7kvxMklcluaaqLk3yySQvnDZ/e5LnJLk9yV8k+YG56gIAWDazBbLufvFxVl14jG07ycvmqgUAYJm5Uz8AwGACGQDAYAIZAMBgAtk2cvBgcuWVi1cAYHnMNqif5XLwYHLhhcl99yWnn57s35/s3Tu6KgAg0UK2bRw4sAhjDzyweD1wYHRFAMARAtmSWu/uxX37Fi1jO3YsXvftW5/9AgAPnS7LJTRH9+LevYv9HDiwCGO6KwFgeQhkS+hY3YvrEaD27l3/IHbwoJAHAA+VQLbBVhNgjnQvHmkhO1n34qhQ5EIBAFgfAtkGWm2AWUv34shQNFdLHgBsNwLZBlpLgFlt9+LIULTWljwA4NgEsg00R4AZGYpcKAAA66O6e3QNp2zPnj196NCh0WWsyRzjvQysB4DlV1U3dveeY64TyAAA5neiQObGsAAAgwlkAACDCWQAAIMJZAAAgwlkAACDCWRsiIMHkyuvXLwCAF/OjWGZnWdeAsCJaSFjdsd6vBMA8NcEMmZ35PFOO3Z45iUAHIsuS2bnmZcAcGICGRti797VBzHP5gRguxHIWCpruQBAcANgqxDIWCrHugDgWGHLlZsAbCUG9XNMo+4bttoLAFy5CcBWooWMrzCy9Wm1FwAcCW5HanTlJgCbmUDGV1htt+FcVnMBgCs3gSOMJ2UrEMj4Cpul9WktV24CW5PxpGwVxpDxFY60Pv38zzu5LSPPBYW/ZjwpW4UWMo5J69Ny0hoAX26ztOjDyWghg01Ea8Dy20otmJvhe9Giz1ahhQw2Ea0By20r3dh4M7XGatFnKxDIYBNxdely20o3Nh59tTU8VMv+R8/RBDJYEqs9eWgNOLnVfpbrfcJebQvmZgg7WmO3js0WTNbDZvij52gCGVveyJPRWoLByJPHVjphr/aznOMz30o3NtYauzWMPreMshn+6DmaQMaWttaT0Xq2rKzlvUeePEafsNc7DK72s5zrMx95Y+P1/iy1xm5+mzGYrIfN8EfP0QQytrS1nIzWu2VlLe89x8ljtb+cN0sYXO33s9rPcvQJe73DzuhgzXIa/XO+Fuv5B8VmbOEVyNjS1nIyWu+WlbW893qfPNbyy3nkCXuOQfCr/Sw34wn7RLZrS8hmMWpYwOif85HDNtbyR88yDNsQyNjS1nIyWu+WlbWeCNezxWQtv5xHnrDnGgS/2s9yK3XJrSVYL8Mvn5OZo8ZR3/fo1stRP+eGbayNQMaWt5ZfzuvdsjLqRLjWVq9RdW6lQfCjrfazXJZfPicyR40jv++1ho3NEJhXY/SwjTnqnJNABitslZaV0YPG1/ILZeQg+K1mNZ/lsvzyOZE5ahz5fa+19XLZA/NqjRy2sRbL8gefQAZb1KhB43P9Qln2ELxZLMsvnxOZo8a5vu/V/PGxlrCxGQLzao0ctrEWy/IHn0AGrMp6X/TA6sxxK4tlv+XGHDWuZZ9zDERfbdjYDIE52Xo3sl6GOgUyYFU2y+0ktpLN0to4+gq59dzn6IHoy9JacyJbqVt1mQhkwKps19tJjLRZWhvXUueyD1hfhoHoy9BacyKjfy6X/WfoVAlkwKptlYseNovN0tq42jo3Q8vKZhmInowLJiN/LjfDz9CpEsgAltToX/irtdo6R7esrMZmGYg+MpiM/LncDD9Dp0ogA1him6W1cTV1bpYWv83wmc8VTJZ9sP5m+Rk6FQIZABtis7T4bQZzPf922bsDt/LPkEAGwIbZDK1Pm8EcwWSzdAdu1Z8hgQwANqH1DiZbuTtwMxDIAE5iq15mDytt5e7AzUAgAziBzTCuBtbLVu0O3AxOG10AwDI71rgagPUmkAGcwJFxNTt2GFcDzEeXJcAJGFcDbASBDOAkjKsB5qbLEgBgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYDCBDABgMIEMAGAwgQwAYLClCmRVdVFVfbSqbq+qK0bXAwCwEZYmkFXVjiS/nuTZSZ6c5MVV9eSxVQEAzG9pAlmSpye5vbs/3t33JfndJBcPrgkAYHbLFMjOTvKpFfN3TMu+TFVdVlWHqurQ4cOHN6w4AIC57BxdwFp191VJrkqSqjpcVZ+c+S3PTPJnM78Hp87xWV6OzXJzfJab47O8Hsqx+YbjrVimQHZnknNXzJ8zLTuu7t41a0VJqupQd++Z+304NY7P8nJslpvjs9wcn+U117FZpi7L9yU5r6oeX1WnJ3lRkusG1wQAMLulaSHr7vur6keSvCPJjiSv7+4PDS4LAGB2SxPIkqS7357k7aPrOMpVowvghByf5eXYLDfHZ7k5PstrlmNT3T3HfgEAWKVlGkMGALAtCWQn4FFOy6WqXl9Vd1fVB1cse3RVXV9VH5teHzWyxu2qqs6tqhuq6sNV9aGqunxa7vgMVlUPr6r3VtUHpmPzs9Pyx1fVe6bz25uni6kYpKp2VNVNVfW2ad7xWRJV9YmqurWqbq6qQ9OydT+3CWTH4VFOS+kNSS46atkVSfZ393lJ9k/zbLz7k/xEdz85yQVJXjb9f3F8xvtSkmd191OSnJ/koqq6IMmrk7ymu5+Y5J4klw6skeTyJLetmHd8lsszu/v8Fbe7WPdzm0B2fB7ltGS6+4+TfO6oxRcnuXqavjrJ8za0KJIk3X1Xd79/mv5iFr9Yzo7jM1wv/Pk0+7DpXyd5VpK3TMsdm4Gq6pwk353ktdN8xfFZdut+bhPIjm9Vj3JiuLO6+65p+tNJzhpZDElV7U7y1CTvieOzFKbusJuT3J3k+iR/muTz3X3/tInz21i/nOTlSR6c5h8Tx2eZdJJ3VtWNVXXZtGzdz21LddsLeCi6u6vKZcMDVdXXJvm9JD/W3V9Y/KG/4PiM090PJDm/qs5I8tYkTxpcEpOq+p4kd3f3jVW1b3Q9HNMzuvvOqvq6JNdX1UdWrlyvc5sWsuNb86OcGOIzVfXYJJle7x5cz7ZVVQ/LIoy9qbt/f1rs+CyR7v58khuS7E1yRlUd+aPc+W2cb0/y3Kr6RBZDY56V5Ffi+CyN7r5zer07iz9onp4Zzm0C2fF5lNPmcF2SS6bpS5JcO7CWbWsa8/K6JLd19y+tWOX4DFZVu6aWsVTVVyf5zizG+N2Q5PnTZo7NIN39iu4+p7t3Z/F75l3d/ZI4Pkuhqr6mqh5xZDrJdyX5YGY4t7kx7AlU1XOy6Ns/8iinfze4pG2tqn4nyb4kZyb5TJKfSfJfk1yT5HFJPpnkhd199MB/ZlZVz0jyJ0luzV+Pg/npLMaROT4DVdW3ZDHoeEcWf4Rf090/V1VPyKJF5tFJbkryj7r7S+MqZeqy/Mnu/h7HZzlMx+Gt0+zOJL/d3f+uqh6TdT63CWQAAIPpsgQAGEwgAwAYTCADABhMIAMAGEwgAwAYTCAD1kVVdVX94or5n6yqf7NO+35DVT3/5Fs+5Pd5QVXdVlU3HLX866vqLdP0+dMtcdbrPc+oqn92rPcCtg+BDFgvX0ryD6vqzNGFrLTibuercWmSf9Ldz1y5sLv/T3cfCYTnJ1lTIDtJDWck+atAdtR7AduEQAasl/uTXJXkXxy94ugWrqr68+l1X1X9UVVdW1Ufr6pXVdVLquq9VXVrVX3jit38vao6VFX/c3r+35GHZv9CVb2vqm6pqh9asd8/qarrknz4GPW8eNr/B6vq1dOyf53kGUleV1W/cNT2u6dtT0/yc0m+r6purqrvm+7k/fqp5puq6uLpa76/qq6rqncl2V9VX1tV+6vq/dN7Xzzt/lVJvnHa3y8cea9pHw+vqt+ctr+pqp65Yt+/X1V/UFUfq6p/v+LzeMNU661V9RXHAlhOHi4OrKdfT3LLkYCwSk9J8jeTfC7Jx5O8trufXlWXJ/nRJD82bbc7i2fIfWOSG6rqiUlemuTe7v62qvqqJP+jqt45bf+0JN/c3f9r5ZtV1dcneXWSb01yT5J3VtXzprvXPyuLO6UfOlah3X3fFNz2dPePTPt7ZRaPu/nB6RFF762qP1xRw7d09+emVrJ/MD10/cwk754C4xVTnedP+9u94i1ftnjb/ltV9aSp1m+a1p2f5KlZtEx+tKp+LcnXJTm7u7952tcZJ/nsgSWhhQxYN939hSRvTPLP1/Bl7+vuu6bHwvxpkiOB6tYsQtgR13T3g939sSyC25OyeK7cS6vq5iwe0/SYJOdN27/36DA2+bYkB7r7cHffn+RNSb5jDfUe7buSXDHVcCDJw7N4nEqSXL/icSqV5JVVdUuSP0xydpKzTrLvZyT5rSTp7o9k8YiWI4Fsf3ff291/mUUr4Ddk8bk8oap+raouSvKFh/B9ARtICxmw3n45yfuT/OaKZfdn+gOwqk5LcvqKdSufz/fgivkH8+XnqKOf89ZZhJwf7e53rFwxPRPw/55a+WtWSb63uz96VA1/+6gaXpJkV5Jv7e7/V1WfyCK8naqVn9sDSXZ29z1V9ZQkfz/JDyd5YZIffAjvAWwQLWTAuppahK7JYoD8EZ/IooswSZ6b5GGnsOsXVNVp07iyJyT5aJJ3JPmnVfWwJKmqb6qqrznJft6b5O9W1ZlVtSPJi5P80Rrq+GKSR6yYf0eSH62qmmp46nG+7pFJ7p7C2DOzaNE61v5W+pMsglymrsrHZfF9H9PUFXpad/9ekn+VRZcpsAkIZMAcfjHJyqst/1MWIegDSfbm1Fqv/ncWYeq/JfnhqavutVl0171/Ggj/H3OSlv/uviuLcVs3JPlAkhu7+9o11HFDkicfGdSf5OezCJi3VNWHpvljeVOSPVV1axZj3z4y1fPZLMa+ffDoiwmS/Ickp01f8+Yk3z917R7P2UkOTN2nv5XkFWv4voCBqvvoXgAAADaSFjIAgMEEMgCAwQQyAIDBBDIAgMEEMgCAwQQyAIDBBDIAgMEEMgCAwf4/Nn/EzPM6zKYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ScckWktynk1o"
      },
      "source": [
        "# Mini Batch Gradient Descent"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have seen the Batch Gradient Descent. We have also seen the Stochastic Gradient Descent. Batch Gradient Descent can be used for smoother curves. SGD can be used when the dataset is large. Batch Gradient Descent converges directly to minima. SGD converges faster for larger datasets. But, since in SGD we use only one example at a time, we cannot implement the vectorized implementation on it. This can slow down the computations. To tackle this problem, a mixture of Batch Gradient Descent and SGD is used.\n",
        "Neither we use all the dataset all at once nor we use the single example at a time. We use a batch of a fixed number of training examples which is less than the actual dataset and call it a mini-batch. Doing this helps us achieve the advantages of both the former variants we saw."
      ],
      "metadata": {
        "id": "ZTVz-QssUkuE"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4JtxFVL78wEm"
      },
      "source": [
        "def minibatch_gradient_descent(X, y, y_pred, learning_rate=0.01, iterations=10, batch_size=20):\n",
        "    '''\n",
        "    X = Matrix of X without added bias units\n",
        "    y = Vector of Y\n",
        "    y_pred = Vector of y_preds np.random.randn(j, 1)\n",
        "    learning_rate \n",
        "    iterations = no of iterations\n",
        "    \n",
        "    Returns the final theta vector and array of cost history over no of iterations\n",
        "    '''\n",
        "\n",
        "    m = len(y)\n",
        "    cost_history = np.zeros(iterations)\n",
        "    n_batches = int(m / batch_size)\n",
        "    \n",
        "    for it in range(iterations):\n",
        "        cost = 0.0\n",
        "        indices = np.random.permutation(m)\n",
        "        X = X[indices]\n",
        "        y = y[indices]\n",
        "\n",
        "        for i in range(0, m, batch_size):\n",
        "            X_i = X[i: i + batch_size]\n",
        "            y_i = y[i: i + batch_size]\n",
        "            \n",
        "            X_i = np.c_[np.ones(len(X_i)), X_i]\n",
        "            prediction = np.dot(X_i, y_pred)\n",
        "\n",
        "            y_pred = y_pred - (1 / m) * learning_rate * (X_i.T.dot((prediction - y_i)))\n",
        "            cost += cal_cost(y_pred, X_i, y_i)\n",
        "\n",
        "        cost_history[it]  = cost\n",
        "        \n",
        "    return y_pred, cost_history"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SpbsVwA28znL",
        "outputId": "182c34d7-7d4d-43de-f7aa-a46e59a6bac9"
      },
      "source": [
        "lr = 0.1\n",
        "n_iter = 200\n",
        "y_pred = np.random.randn(2,1)\n",
        "y_pred, cost_history = minibatch_gradient_descent(X, y, y_pred, lr, n_iter)\n",
        "\n",
        "print('y_pred[0]: {:0.3f}\\ny_pred[1]: {:0.3f}'.format(y_pred[0][0], y_pred[1][0]))\n",
        "print('Final error: {:0.3f}'.format(cost_history[-1]))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "y_pred[0]: 4.148\n",
            "y_pred[1]: 2.827\n",
            "Final error: 807.488\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 515
        },
        "id": "Q_ivOYHT817C",
        "outputId": "47b614f3-16ef-46ba-d247-aaa8bb043404"
      },
      "source": [
        "fig, ax = plt.subplots(figsize=(10,8))\n",
        "\n",
        "ax.set_ylabel('Error')\n",
        "ax.set_xlabel('Number of iterations')\n",
        "y_pred = np.random.randn(2,1)\n",
        "\n",
        "ax.plot(range(n_iter), cost_history, 'b.')"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f5a3870b210>]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnAAAAHgCAYAAAAlnVB9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df5Ctd10f8PcnN1xw8EdCuM1gEk3A2Da1NeAasvVHF2NDkrEGf2EYp4nCGK1godVqop3CgGNAi1ocxAaJJJUa8Ndwx8GGcM3WTucC2UBMCD/MFUGShuRKkGCtuebm2z+eZ+Gw7N679949e/a7+3rNnHnO+Z7nPM/nu2d/vPf7fZ7zVGstAAD046RZFwAAwLER4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMyfPuoDN9tSnPrWdffbZsy4DAOCo7rjjjr9qre1Z2b7jAtzZZ5+dpaWlWZcBAHBUVfXx1dpNoQIAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8BtsP37k+uuG5YAANNw8qwL2E72708uuig5dCjZvTvZty+Zn591VQDAdmMEbgMtLg7h7fDhYbm4OOuKAIDtSIDbQAsLw8jbrl3DcmFh1hUBANuRKdQNND8/TJsuLg7hzfQpADANAtwGm58X3ACA6TKFCgDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM5MLcBV1VlVdVtVfbCq7qmql47tr6iq+6vqzvF22cRrrq2qA1X1kap67kT7JWPbgaq6ZqL9nKp6z9j+1qraPa3+AABsFdMcgXssyU+01s5LcmGSF1fVeeNzv9xaO3+8vSNJxueuSPJPklyS5NeqaldV7Ury+iSXJjkvyQsmtvOacVtfk+TTSV40xf4AAGwJUwtwrbUHWmvvG+9/NsmHkpxxhJdcnuTm1tqjrbW/SHIgyQXj7UBr7aOttUNJbk5yeVVVkm9L8rvj629M8rzp9AYAYOvYlGPgqursJM9M8p6x6SVVdVdV3VBVp45tZyT5xMTL7hvb1mo/Lclft9YeW9G+2v6vrqqlqlo6ePDgBvQIAGB2ph7gqupLk/xekpe11h5J8oYkz0hyfpIHkrx22jW01q5vrc211ub27Nkz7d0BAEzVydPceFU9IUN4e0tr7feTpLX24MTzb0zyh+PD+5OcNfHyM8e2rNH+qSSnVNXJ4yjc5PoAANvWNM9CrSRvSvKh1tovTbQ/bWK170rygfH+3iRXVNUTq+qcJOcmeW+S25OcO55xujvDiQ57W2styW1Jvnd8/VVJ3j6t/gAAbBXTHIH7piT/OsndVXXn2PYzGc4iPT9JS/KxJD+SJK21e6rqbUk+mOEM1he31g4nSVW9JMktSXYluaG1ds+4vZ9OcnNV/VyS92cIjAAA21oNA1k7x9zcXFtaWpp1GQAAR1VVd7TW5la2uxIDAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ2ZWoCrqrOq6raq+mBV3VNVLx3bn1JVt1bVvePy1LG9qup1VXWgqu6qqmdNbOuqcf17q+qqifZvqKq7x9e8rqpqWv0BANgqpjkC91iSn2itnZfkwiQvrqrzklyTZF9r7dwk+8bHSXJpknPH29VJ3pAMgS/Jy5M8O8kFSV6+HPrGdX544nWXTLE/AABbwtQCXGvtgdba+8b7n03yoSRnJLk8yY3jajcmed54//IkN7XBu5OcUlVPS/LcJLe21h5urX06ya1JLhmf+/LW2rtbay3JTRPbAgDYtjblGLiqOjvJM5O8J8nprbUHxqc+meT08f4ZST4x8bL7xrYjtd+3Svtq+7+6qpaqaungwYMn1BcAgFmbeoCrqi9N8ntJXtZae2TyuXHkrE27htba9a21udba3J49e6a9OwCAqZpqgKuqJ2QIb29prf3+2PzgOP2ZcfnQ2H5/krMmXn7m2Hak9jNXaQcA2NameRZqJXlTkg+11n5p4qm9SZbPJL0qydsn2q8cz0a9MMlnxqnWW5JcXFWnjicvXJzklvG5R6rqwnFfV05sCwBg2zp5itv+piT/OsndVXXn2PYzSV6d5G1V9aIkH0/y/PG5dyS5LMmBJH+b5IeSpLX2cFW9Ksnt43qvbK09PN7/sSRvTvIlSf5ovAEAbGs1HIa2c8zNzbWlpaVZlwEAcFRVdUdrbW5luysxAAB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAW6K9u9PrrtuWAIAbJSTZ13AdrV/f3LRRcmhQ8nu3cm+fcn8/KyrAgC2AyNwU7K4OIS3w4eH5eLirCsCALYLAW5KFhaGkbddu4blwsKsKwIAtgtTqFMyPz9Mmy4uDuHN9CkAsFEEuCmanxfcAICNZwoVAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnphbgquqGqnqoqj4w0faKqrq/qu4cb5dNPHdtVR2oqo9U1XMn2i8Z2w5U1TUT7edU1XvG9rdW1e5p9QUAYCuZ5gjcm5Ncskr7L7fWzh9v70iSqjovyRVJ/sn4ml+rql1VtSvJ65NcmuS8JC8Y102S14zb+pokn07yoin2BQBgy5hagGut/UmSh9e5+uVJbm6tPdpa+4skB5JcMN4OtNY+2lo7lOTmJJdXVSX5tiS/O77+xiTP29AOAABsUbM4Bu4lVXXXOMV66th2RpJPTKxz39i2VvtpSf66tfbYivZVVdXVVbVUVUsHDx7cqH4AAMzEZge4NyR5RpLzkzyQ5LWbsdPW2vWttbnW2tyePXs2Y5cAAFNz8mburLX24PL9qnpjkj8cH96f5KyJVc8c27JG+6eSnFJVJ4+jcJPrAwBsa5s6AldVT5t4+F1Jls9Q3Zvkiqp6YlWdk+TcJO9NcnuSc8czTndnONFhb2utJbktyfeOr78qyds3ow8AALM2tRG4qvrtJAtJnlpV9yV5eZKFqjo/SUvysSQ/kiSttXuq6m1JPpjksSQvbq0dHrfzkiS3JNmV5IbW2j3jLn46yc1V9XNJ3p/kTdPqCwDAVlLDYNbOMTc315aWlmZdBgDAUVXVHa21uZXtrsQAANAZAQ4AoDNHDXBVdVJV/fPNKAYAgKM7aoBrrT2e4XJWAABsAeudQt1XVd8zXsIKAIAZWm+A+5Ekv5PkUFU9UlWfrapHplgXAABrWNfnwLXWvmzahQAAsD7r/iDfqvrOJN86Plxsrf3hkdYHAGA61jWFWlWvTvLSDFdK+GCSl1bVddMsDACA1a13BO6yJOePZ6Smqm7McPmqa6dVGAAAqzuWD/I9ZeL+V2x0IQAArM96R+B+Psn7q+q2JJXhWLhrplYVAABrOmqAq6qTkjye5MIk3zg2/3Rr7ZPTLAwAgNUdNcC11h6vqp9qrb0tyd5NqAkAgCNY7zFw76qqn6yqs6rqKcu3qVYGAMCq1nsM3PePyxdPtLUkT9/YcgAAOJr1HgN3TWvtrZtQDwAAR3HUKdTxs9/+wybUAgDAOjgGDgCgM46BAwDozLoCXGvtnGkXAgDA+hxxCrWqfmri/veteO7np1UUAABrO9oxcFdM3F954fpLNrgWAADW4WgBrta4v9pjAAA2wdECXFvj/mqPAQDYBEc7ieHrq+qRDKNtXzLez/j4SVOtDACAVR0xwLXWdm1WIQAArM96P8gXAIAtQoADAOiMALcJ9u9PrrtuWAIAnKj1XkqL47R/f3LRRcmhQ8nu3cm+fcn8/KyrAgB6ZgRuyhYXh/B2+PCwXFycdUUAQO8EuClbWBhG3nbtGpYLC7OuCADonSnUKZufH6ZNFxeH8Gb6FAA4UQLcJpifF9wAgI1jChUAoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0JmpBbiquqGqHqqqD0y0PaWqbq2qe8flqWN7VdXrqupAVd1VVc+aeM1V4/r3VtVVE+3fUFV3j695XVXVtPoCALCVTHME7s1JLlnRdk2Sfa21c5PsGx8nyaVJzh1vVyd5QzIEviQvT/LsJBckefly6BvX+eGJ163cFwDAtjS1ANda+5MkD69ovjzJjeP9G5M8b6L9pjZ4d5JTquppSZ6b5NbW2sOttU8nuTXJJeNzX95ae3drrSW5aWJbAADb2mYfA3d6a+2B8f4nk5w+3j8jyScm1rtvbDtS+32rtK+qqq6uqqWqWjp48OCJ9QAAYMZmdhLDOHLWNmlf17fW5lprc3v27NmMXQIATM1mB7gHx+nPjMuHxvb7k5w1sd6ZY9uR2s9cpR0AYNvb7AC3N8nymaRXJXn7RPuV49moFyb5zDjVekuSi6vq1PHkhYuT3DI+90hVXTiefXrlxLYAALa1k6e14ar67SQLSZ5aVfdlOJv01UneVlUvSvLxJM8fV39HksuSHEjyt0l+KElaaw9X1auS3D6u98rW2vKJET+W4UzXL0nyR+MNAGDbq+FQtJ1jbm6uLS0tzboMAICjqqo7WmtzK9tdiQEAoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwLcJtu/P7nuumEJAHA8pnYtVL7Y/v3JRRclhw4lu3cn+/Yl8/OzrgoA6I0RuE20uDiEt8OHh+Xi4qwrAgB6JMBtooWFYeRt165hubAw64oAgB6ZQt1E8/PDtOni4hDeTJ8CAMdDgNtk8/OCGwBwYkyhAgB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwABwDQGQEOAKAzAhwAQGcEOACAzghwAACdEeAAADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8DNyP79yXXXDUsAgGNx8qwL2In2708uuig5dCjZvTvZty+Zn591VQBAL4zAzcDi4hDeDh8elouLs64IAOiJADcDCwvDyNuuXcNyYWHWFQEAPTGFOgPz88O06eLiEN5MnwIAx0KAm5H5ecENADg+plABADojwAEAdEaAAwDojAAHANAZAQ4AoDMCHABAZwQ4AIDOCHAAAJ0R4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnZhLgqupjVXV3Vd1ZVUtj21Oq6taqundcnjq2V1W9rqoOVNVdVfWsie1cNa5/b1VdNYu+AABstlmOwD2ntXZ+a21ufHxNkn2ttXOT7BsfJ8mlSc4db1cneUMyBL4kL0/y7CQXJHn5cugDANjOttIU6uVJbhzv35jkeRPtN7XBu5OcUlVPS/LcJLe21h5urX06ya1JLtnsogEANtusAlxL8s6quqOqrh7bTm+tPTDe/2SS08f7ZyT5xMRr7xvb1mr/IlV1dVUtVdXSwYMHN6oPAAAzcfKM9vvNrbX7q+ofJLm1qj48+WRrrVVV26idtdauT3J9kszNzW3YdjfK/v3J4mKysJDMz8+6GgBgq5tJgGut3T8uH6qqP8hwDNuDVfW01toD4xTpQ+Pq9yc5a+LlZ45t9ydZWNG+OOXSN9z+/clFFyWHDiW7dyf79glxAMCRbfoUalU9uaq+bPl+kouTfCDJ3iTLZ5JeleTt4/29Sa4cz0a9MMlnxqnWW5JcXFWnjicvXDy2dWVxcQhvhw8Py8XFWVcEAGx1sxiBOz3JH1TV8v7/e2vtf1TV7UneVlUvSvLxJM8f139HksuSHEjyt0l+KElaaw9X1auS3D6u98rW2sOb142NsbAwjLwtj8AtLMy6IgBgq6vWttwhYVM1NzfXlpaWZl3GF3AMHACwmqq6Y+Ij1z5nVicxMGF+XnADANZvK30OHAAA6yDAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwAt4Xs359cd92wBABYi4vZbxH79ycXXZQcOpTs3p3s2+cC9wDA6ozAbRGLi0N4O3x4WC4uzroiAGCrEuC2iIWFYeRt165hubAw64oAgK3KFOoWMT8/TJsuLg7hzfQpALAWAW4LmZ8X3ACAozOFCgDQGQEOAKAzAhwAQGcEOACAzghwAACdEeC2KJfVAgDW4mNEtiCX1QIAjsQI3BbksloAwJEIcFuQy2oBAEdiCnULclktAOBIBLgtymW1AIC1mEIFAOiMALfF+TgRAGAlU6hbmI8TAQBWYwRuC/NxIgDAagS4LczHiQAAqzGFuoX5OBEAYDUC3BY3+XEi+/cLcwCAANcNJzQAAMscA9cJJzQAAMsEuE6sPKHhtNN8PhwA7FSmUDsxeULDaaclL3uZ6VQA2KmMwHVkfj659trkU5/6/HTqo48mr3iFkTgA2EkEuA4tT6eedFLy+OPJu941nOAgxAHAziDAdWh5OvXbv/3zIe7QoeSmmxwXBwA7QbXWZl3Dppqbm2tLS0uzLmNDTH60yK5dSVXy2GPD/Re+MLnySsfGAUDPquqO1trcF7ULcH1b/nDfv/zL5I1vHI6LS4Yw96QnJb/yK8Mxcz78FwD6I8CNtluAW7Y8Gvd3f5csv6UnnTSMxj3++BeOyiWu6AAAPRDgRts1wCVDiLvppuQ3f3OYSq0awtvjjw/PVyVPeMLqU62JUAcAW40AN9rOAW7Z8rTq8ufFTY7KVQ3LycfrCXWT9wU8ANgcAtxoJwS4SStH5ZZPdjh06NhC3ZEC3mmnOc4OAKZBgBvttAC3bHlUbnk07VhC3ZEC3t///TBFe9JJyROf+PmTJiZDXSLoAcDxEOBGOzXArWY9oW69o3bJ50+aOHz486Hu5JOPHPTWCnirtQl9AOw0AtxIgDuylaHuaAFvMpgtf6jw8kkTyZGD3vLZsSu3c7TQd7Sgt1qb8AdAjwS4kQB3YlYLeMtBafmkiUcfPXIYmwx6qwW8Exnd28gRv9XaBEEANpMANxLgpmvyDNi1gtBy0Ju8gsTRQth6R/c2asTveEYBTzQcTm5HUAQgEeA+R4DbGo40kncio3sbNeJ3rKOAa00rr7fGlds50pm+y20bFRiPZduCJcDmEuBGAly/1jO6t1EjfsczCngi4XC9Z/pudGA8lm2f6HGIR2qbZhjdKuF38nlBGFgvAW4kwO1Mxzrid6yjgBsVqI52pu+0AuN6tn0ixyEeT2DciG1vlfB7PCOsPYRR+9MX+5v+P2QC3EiA40QcaRRwI36ZHO1M31mGkKONQJ5I2zTD6GbvbyNGWHsIo/anL/Y3zEzs2zfdECfAjQQ4trr1jhZu9n+kJ3Ic4nb6hX0i2z6WEdb1ts06jNqfvuzk/e3albzqVcm112ZqBLiRAAfH73iPQ9xuUybHu+31jrD2EEbtT1/szwjcphLggFnaiOMxt0IYtT99sT/HwG0qAQ4A6MVaAe6kWRQDAMDxE+AAADojwAEAdEaAAwDojAAHANCZ7gNcVV1SVR+pqgNVdc2s6wEAmLauA1xV7Ury+iSXJjkvyQuq6rzZVgUAMF1dB7gkFyQ50Fr7aGvtUJKbk1w+45oAAKaq9wB3RpJPTDy+b2z7AlV1dVUtVdXSwYMHN604AIBp6D3ArUtr7frW2lxrbW7Pnj2zLgcA4IT0HuDuT3LWxOMzxzYAgG2r9wB3e5Jzq+qcqtqd5Ioke2dcEwDAVJ086wJORGvtsap6SZJbkuxKckNr7Z4ZlwUAMFVdB7gkaa29I8k7Zl0HAMBm6X0KFQBgxxHgAAA6U621WdewqarqYJKPT3k3T03yV1Pex1am/zu3/zu574n+7+T+7+S+J/o/zf5/dWvtiz4DbccFuM1QVUuttblZ1zEr+r9z+7+T+57o/07u/07ue6L/s+i/KVQAgM4IcAAAnRHgpuP6WRcwY/q/c+3kvif6v5P7v5P7nuj/pvffMXAAAJ0xAgcA0BkBboNV1SVV9ZGqOlBV18y6nmmqqrOq6raq+mBV3VNVLx3bX1FV91fVnePtslnXOi1V9bGqunvs59LY9pSqurWq7h2Xp866zmmoqn848R7fWVWPVNXLtvP7X1U3VNVDVfWBibZV3+8avG78XXBXVT1rdpWfuDX6/otV9eGxf39QVaeM7WdX1f+b+B749dlVvjHW6P+a3+tVde343n+kqp47m6o3zhr9f+tE3z9WVXeO7dvq/T/C37qZ/uybQt1AVbUryZ8l+ZdJ7ktye5IXtNY+ONPCpqSqnpbkaa2191XVlyW5I8nzkjw/yd+01v7zTAvcBFX1sSRzrbW/mmj7hSQPt9ZePYb4U1trPz2rGjfD+L1/f5JnJ/mhbNP3v6q+NcnfJLmptfZ1Y9uq7/f4x/zHk1yW4evyX1prz55V7Sdqjb5fnOSPx+tSvyZJxr6fneQPl9fbDtbo/yuyyvd6VZ2X5LeTXJDkK5O8K8nXttYOb2rRG2i1/q94/rVJPtNae+V2e/+P8LfuBzPDn30jcBvrgiQHWmsfba0dSnJzkstnXNPUtNYeaK29b7z/2SQfSnLGbKvaEi5PcuN4/8YMP+jb3UVJ/ry1Nu0PyZ6p1tqfJHl4RfNa7/flGf7Ytdbau5OcMv4h6NJqfW+tvbO19tj48N1Jztz0wjbJGu/9Wi5PcnNr7dHW2l8kOZDh70O3jtT/qqoM/7j/9qYWtUmO8Ldupj/7AtzGOiPJJyYe35cdEmjG/7iemeQ9Y9NLxqHjG7brFOKoJXlnVd1RVVePbae31h4Y738yyemzKW1TXZEv/OW9U97/ZO33e6f9Pnhhkj+aeHxOVb2/qv5nVX3LrIraBKt9r++09/5bkjzYWrt3om1bvv8r/tbN9GdfgOOEVdWXJvm9JC9rrT2S5A1JnpHk/CQPJHntDMubtm9urT0ryaVJXjxOM3xOG45R2NbHKVTV7iTfmeR3xqad9P5/gZ3wfq+mqn42yWNJ3jI2PZDkq1prz0zy75P896r68lnVN0U79nt9hRfkC/+B25bv/yp/6z5nFj/7AtzGuj/JWROPzxzbtq2qekKGb+i3tNZ+P0laaw+21g631h5P8sZ0PnVwJK21+8flQ0n+IENfH1weLh+XD82uwk1xaZL3tdYeTHbW+z9a6/3eEb8PquoHk3xHkh8Y/4hlnDr81Hj/jiR/nuRrZ1bklBzhe31HvPdJUlUnJ/nuJG9dbtuO7/9qf+sy4599AW5j3Z7k3Ko6ZxyVuCLJ3hnXNDXjcQ9vSvKh1tovTbRPzvV/V5IPrHztdlBVTx4PaE1VPTnJxRn6ujfJVeNqVyV5+2wq3DRf8N/3Tnn/J6z1fu9NcuV4RtqFGQ7wfmC1DfSqqi5J8lNJvrO19rcT7XvGE1tSVU9Pcm6Sj86myuk5wvf63iRXVNUTq+qcDP1/72bXt0m+PcmHW2v3LTdst/d/rb91mfXPfmvNbQNvGc46+bMM/3H87KzrmXJfvznDkPFdSe4cb5cl+W9J7h7b92Y4e2fm9U6h/09P8qfj7Z7l9zvJaUn2Jbk3w9lnT5l1rVP8Gjw5yaeSfMVE27Z9/zME1QeS/H2G41petNb7naSSvH78XXB3hrOVZ96HDe77gQzH+iz//P/6uO73jD8TdyZ5X5J/Nev6p9T/Nb/Xk/zs+N5/JMmls65/Gv0f29+c5EdXrLut3v8j/K2b6c++jxEBAOiMKVQAgM4IcAAAnRHgAAA6I8ABAHRGgAMA6IwAB8xMVbXxItjLj39yvED4Rmz7zVX1vRuxraPs5/uq6kNVdduK9q+sqt8d758/XuB6o/Z5SlX92Gr7AnYGAQ6YpUeTfHdVPXXWhUwaP11+vV6U5Idba8+ZbGyt/Z/W2nKAPD/D50ZtVA2nJPlcgFuxL2AHEOCAWXosyfVJ/t3KJ1aOoFXV34zLhfEC2W+vqo9W1aur6geq6r1VdXdVPWNiM99eVUtV9WdV9R3j63dV1S9W1e3jRch/ZGK7/6uq9ib54Cr1vGDc/geq6jVj23/K8CGfb6qqX1yx/tnjuruTvDLJ91fVnVX1/eNVPG4Ya35/VV0+vuYHq2pvVf1xkn1V9aVVta+q3hBmsWkAAANvSURBVDfu+/Jx869O8oxxe7+4vK9xG0+qqt8c139/VT1nYtu/X1X/o6rurapfmPh6vHms9e6q+qL3Ath6juW/TIBpeH2Su5YDxTp9fZJ/nOThDJfo+Y3W2gVV9dIkP57kZeN6Z2e4PuUzktxWVV+T5MoMl7b5xqp6YpL/XVXvHNd/VpKva639xeTOquork7wmyTck+XSSd1bV81prr6yqb0vyk621pdUKba0dGoPeXGvtJeP2fj7JH7fWXlhVpyR5b1W9a6KGf9Zae3gchfuu1toj4yjlu8eAec1Y5/nj9s6e2OWLh922f1pV/2isdfk6lOcneWaGkc+PVNWvJvkHSc5orX3duK1TjvK1B7YAI3DATLXWHklyU5J/ewwvu7219kBr7dEMl6tZDmB3Zwhty97WWnu8tXZvhqD3jzJcs/bKqrozyXsyXA7n3HH9964Mb6NvTLLYWjvYWnssyVuSfOsx1LvSxUmuGWtYTPKkJF81Pndra+3h8X4l+fmquivDpXrOSHL6Ubb9zUl+K0laax9O8vF8/kLi+1prn2mt/V2GUcavzvB1eXpV/ep4bdNHTqBfwCYxAgdsBb+S4ZqJvznR9ljGfzKr6qQkuyeee3Ti/uMTjx/PF/5eW3mtwJYhFP14a+2WySeqaiHJ/z2+8o9ZJfme1tpHVtTw7BU1/ECSPUm+obX291X1sQxh73hNft0OJzm5tfbpqvr6JM9N8qNJnp/khSewD2ATGIEDZm4ccXpbhhMCln0sw5Rlknxnkiccx6a/r6pOGo+Le3qGC4vfkuTfVNUTkqSqvraqnnyU7bw3yb+oqqdW1a4kL0jyP4+hjs8m+bKJx7ck+fGqqrGGZ67xuq9I8tAY3p6TYcRste1N+l8Zgl/GqdOvytDvVY1Tsye11n4vyX/MMIULbHECHLBVvDbJ5Nmob8wQmv40yXyOb3TsLzOErz9K8qPj1OFvZJg+fN944P9/zVFmI1prD2Q47uy2JH+a5I7W2tuPoY7bkpy3fBJDkldlCKR3VdU94+PVvCXJXFXdneHYvQ+P9Xwqw7F7H1h58kSSX0ty0viatyb5wXGqeS1nJFkcp3N/K8m1x9AvYEaqtZUzDAAAbGVG4AAAOiPAAQB0RoADAOiMAAcA0BkBDgCgMwIcAEBnBDgAgM4IcAAAnfn/2QSOCgiV+zkAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x576 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Things to try out:\n",
        "\n",
        "1. Change batch size in mini-batch gradient descent.\n",
        "2. Test all the three out on real datasets.\n",
        "3. Compare the effects of changing learning rate by the same amount in Batch GD, SGD and Mini-batch GD."
      ],
      "metadata": {
        "id": "0neTARjKUoP4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#question 1 - Change batch size of mini batch gradient descent\n",
        "#intially it was 20, I am going to use 40 instead now.\n",
        "#keeping all the other factors same that is learning rate same = 0.01 and iterations 10\n",
        "\n",
        "def minibatch_gradient_descent(X, y, y_pred, learning_rate=0.01, iterations=10, batch_size=40):\n",
        "\n",
        "    m = len(y)\n",
        "    cost_history = np.zeros(iterations)\n",
        "    n_batches = int(m / batch_size)\n",
        "    \n",
        "    for it in range(iterations):\n",
        "        cost = 0.0\n",
        "        indices = np.random.permutation(m)\n",
        "        X = X[indices]\n",
        "        y = y[indices]\n",
        "\n",
        "        for i in range(0, m, batch_size):\n",
        "            X_i = X[i: i + batch_size]\n",
        "            y_i = y[i: i + batch_size]\n",
        "            \n",
        "            X_i = np.c_[np.ones(len(X_i)), X_i]\n",
        "            prediction = np.dot(X_i, y_pred)\n",
        "\n",
        "            y_pred = y_pred - (1 / m) * learning_rate * (X_i.T.dot((prediction - y_i)))\n",
        "            cost += cal_cost(y_pred, X_i, y_i)\n",
        "\n",
        "        cost_history[it]  = cost\n",
        "        \n",
        "    return y_pred, cost_history"
      ],
      "metadata": {
        "id": "Dy2ZJlRPv0Tt"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "lr = 0.1\n",
        "n_iter = 200\n",
        "y_pred = np.random.randn(2,1)\n",
        "y_pred, cost_history = minibatch_gradient_descent(X, y, y_pred, lr, n_iter)\n",
        "\n",
        "print('y_pred[0]: {:0.3f}\\ny_pred[1]: {:0.3f}'.format(y_pred[0][0], y_pred[1][0]))\n",
        "print('Final error: {:0.3f}'.format(cost_history[-1]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eqYzavUlwcQU",
        "outputId": "0b0ef3ee-a229-4a20-f1c6-212dac640e14"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "y_pred[0]: 4.248\n",
            "y_pred[1]: 2.741\n",
            "Final error: 1454.862\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# y is closer to 4 but the x has gone a little down from 3 \n",
        "#as compared to the previous batch the final error almost doubled"
      ],
      "metadata": {
        "id": "3-PQuK1fwkaT"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "YB22pSnpzcxp"
      },
      "execution_count": 18,
      "outputs": []
    }
  ]
}