NNFS/08_nnfs_gradients_partial_derivatives_and_the_chain_rule.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/theaveas/DeepLearning/blob/main/NNFS/08_nnfs_gradients_partial_derivatives_and_the_chain_rule.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a5904a7c-d42e-4a11-9f5a-b52256912d1e",
      "metadata": {
        "id": "a5904a7c-d42e-4a11-9f5a-b52256912d1e"
      },
      "source": [
        "# Gradients, Partial Derivatives, and the Chain Rule\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "af06bd34-db00-44ed-a540-21954e5021f9",
      "metadata": {
        "id": "af06bd34-db00-44ed-a540-21954e5021f9",
        "outputId": "b9815a50-99c3-400c-ae51-fa1cbcd914e4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "3.9.7\n",
            "1.21.2\n",
            "3.5.0\n"
          ]
        }
      ],
      "source": [
        "import platform\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "print(platform.python_version())\n",
        "print(np.__version__)\n",
        "print(matplotlib. __version__)\n",
        "\n",
        "#python version 3.9.7\n",
        "#numpy version 1.21.2\n",
        "#matplotlib version 3.5.0"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6180c1a8-23ed-4c33-900f-a9eac20c825c",
      "metadata": {
        "id": "6180c1a8-23ed-4c33-900f-a9eac20c825c"
      },
      "source": [
        "## The Partial Deravative\n",
        "Measures how much impact a single input has on a function's output.\\\n",
        "The method for calculating a partial derivative is the same as for derivatives, simple have to repeat this process for each of the independent inputs.\\\n",
        "`f(x, y, z)` -> `df(x, y, z)/dx`, `df(x, y, z)/dy`, `df(x, y, z)/dz`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a6020470-d088-44e5-9655-fb37501d44a1",
      "metadata": {
        "id": "a6020470-d088-44e5-9655-fb37501d44a1"
      },
      "source": [
        "## The Partial Derivative of a Sum\n",
        "`f(x, y)` = `x + y` -> `df(x, y)/dx`, `df(x, y)/dy` -> `1+0`, `0+1`\\\n",
        "Example:\\\n",
        "`f(x, y)` = `2x + 3y**2` -> `df(x, y)/dx = 2`, `df(x, y)/dy = 6y`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c06561a4-3494-42f4-99e1-456f86bc5948",
      "metadata": {
        "id": "c06561a4-3494-42f4-99e1-456f86bc5948"
      },
      "source": [
        "## The Partial Derivative of Multiplication\n",
        "`f(x, y)` = `x * y` -> `df(x,y)/dx = y`, `df(x, y)/dy = x`\\\n",
        "Example:\\\n",
        "`f(x, y, z)` = `3x**3z - y**2 + 5z + 2yz`\\\n",
        "`df(x, y, z)/dx` -> `9x**2z`\\\n",
        "`df(x, y, z)/dy` -> `-2 + 2z`\\\n",
        "`df(x, y, z)/dz` -> `3x**3 + 2y + 5`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "186e43a0-ea6f-4a06-8fa5-48373973d394",
      "metadata": {
        "id": "186e43a0-ea6f-4a06-8fa5-48373973d394"
      },
      "source": [
        "## The Partial Derivative of Max\n",
        "`f(x, y) = max(x, y)` -> `df(x, y)/dx` = `dmax(x, y)/dx` = `(x>y)`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b18d872f-522d-4bf1-a061-752877f6b9dd",
      "metadata": {
        "id": "b18d872f-522d-4bf1-a061-752877f6b9dd"
      },
      "source": [
        "## The Gradient\n",
        "The gradient is a vector composed of all of the partial derivatives of a function, calculated woth respect to each input variable\\\n",
        "We'll be using **derivatives** of a single-parameter functions and **gradients** of multivariate functions to perform **gradient desenct** using the **chain rule**."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "68b20b48-89ad-4f09-8656-d046da34aacf",
      "metadata": {
        "id": "68b20b48-89ad-4f09-8656-d046da34aacf"
      },
      "source": [
        "## The Chain Rule"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "adf977fb-d529-4bc9-b54b-6898ccebbd1e",
      "metadata": {
        "id": "adf977fb-d529-4bc9-b54b-6898ccebbd1e",
        "outputId": "72c0bbfb-8317-47c9-e0ee-09eb3645701c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'\\nloss = -np.log(\\n    np.sum(\\n        y* np.exp(\\n            np.dot(\\n                np.maximum(\\n                    0,\\n                    np.dot(\\n                        np.maximum(\\n                            0,\\n                            np.dot(\\n                                X, \\n                                w1.T\\n                            ) + b1\\n                        ),\\n                        w2.T\\n                    ) + b2\\n                ),\\n                w3.T\\n            ) + b3\\n        ) /\\n        np.sum(\\n            np.exp(\\n                np.dot(\\n                    np.maximum(\\n                        0,\\n                        np.dot(\\n                            np.maximum(\\n                                0, \\n                                np.dot(\\n                                    X, \\n                                w1.T\\n                            ) + b1\\n                        ),\\n                        w2.T\\n                    ) + b2\\n                ),\\n                w3.T\\n            ) + b3\\n        ),\\n            axis=1\\n            keepdims=True\\n        )\\n    )\\n)\\n'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# code for a forward pass of an example neural network model\n",
        "\"\"\"\n",
        "loss = -np.log(\n",
        "    np.sum(\n",
        "        y* np.exp(\n",
        "            np.dot(\n",
        "                np.maximum(\n",
        "                    0,\n",
        "                    np.dot(\n",
        "                        np.maximum(\n",
        "                            0,\n",
        "                            np.dot(\n",
        "                                X, \n",
        "                                w1.T\n",
        "                            ) + b1\n",
        "                        ),\n",
        "                        w2.T\n",
        "                    ) + b2\n",
        "                ),\n",
        "                w3.T\n",
        "            ) + b3\n",
        "        ) /\n",
        "        np.sum(\n",
        "            np.exp(\n",
        "                np.dot(\n",
        "                    np.maximum(\n",
        "                        0,\n",
        "                        np.dot(\n",
        "                            np.maximum(\n",
        "                                0, \n",
        "                                np.dot(\n",
        "                                    X, \n",
        "                                w1.T\n",
        "                            ) + b1\n",
        "                        ),\n",
        "                        w2.T\n",
        "                    ) + b2\n",
        "                ),\n",
        "                w3.T\n",
        "            ) + b3\n",
        "        ),\n",
        "            axis=1\n",
        "            keepdims=True\n",
        "        )\n",
        "    )\n",
        ")\n",
        "\"\"\"                                   "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8e0e933b-e0ed-41d0-a0f8-55d2146d0368",
      "metadata": {
        "id": "8e0e933b-e0ed-41d0-a0f8-55d2146d0368"
      },
      "source": [
        "This Chain rule can visualize as\\\n",
        "`df(g(y,h(x,z)))` / `dx` -> \\\n",
        "`df(g(y,h(x,z)))` / `d(y,h(x,z))` * `df(y,h(x,z))` / `d(x,z)` * `d(x,z)` / `dx`"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "torch110",
      "language": "python",
      "name": "torch110"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "name": "08_nnfs_gradients_partial_derivatives_and_the_chain_rule.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}