NNFS/04_nnfs_activation_functions.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "[![Open in Colab](https://img.shields.io/static/v1?label=&message=Open%20in%20Colab&labelColor=grey&color=blue&logo=google-colab)](https://colab.research.google.com/github/theaveas/DeepLearning/blob/main/NNFS/04_nnfs_activation_functions.ipynb#scrollTo=x96Asv5243q4)"
      ],
      "metadata": {
        "id": "x96Asv5243q4"
      },
      "id": "x96Asv5243q4"
    },
    {
      "cell_type": "markdown",
      "id": "9830cb44-2dd8-4fb6-a0cb-f238849efdfa",
      "metadata": {
        "id": "9830cb44-2dd8-4fb6-a0cb-f238849efdfa"
      },
      "source": [
        "# Activation Functions\n",
        "The activation function is applied to the output of a neuron (or layer of neurons), which modifies output. We use activation functions because if the activation function itself is nonlinear, it allows for neural netowrks with usually two or more hidden layers map nonlinear functions.\\\n",
        "\\\n",
        "There are two types of activation functions:\n",
        "   - activation functions used in hidden layers\n",
        "   - activation functions used in output layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "6c078c01-7559-4f67-b1b4-ed117e76a0b8",
      "metadata": {
        "id": "6c078c01-7559-4f67-b1b4-ed117e76a0b8",
        "outputId": "94ef9bff-ebdb-4073-d64a-4dbba73ae2bd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3.7.12\n",
            "1.19.5\n",
            "3.2.2\n"
          ]
        }
      ],
      "source": [
        "import platform\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "print(platform.python_version())\n",
        "print(np.__version__)\n",
        "print(matplotlib. __version__)\n",
        "\n",
        "#python version 3.9.7\n",
        "#numpy version 1.21.2\n",
        "#matplotlib version 3.5.0"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# install library\n",
        "!pip install nnfs"
      ],
      "metadata": {
        "id": "-z5RqXqTCWTT",
        "outputId": "019acd5b-fa72-4c3c-afce-314adb05bb32",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "id": "-z5RqXqTCWTT",
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting nnfs\n",
            "  Downloading nnfs-0.5.1-py3-none-any.whl (9.1 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from nnfs) (1.19.5)\n",
            "Installing collected packages: nnfs\n",
            "Successfully installed nnfs-0.5.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1f85f2f6-b79e-4d4b-82b2-75101a8dabe0",
      "metadata": {
        "id": "1f85f2f6-b79e-4d4b-82b2-75101a8dabe0"
      },
      "source": [
        "## The Step Activation Function\n",
        "Simple activation function that try to mimic a neuron `fire` or `not firing` based on input information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "99690444-5ea1-43fc-8667-4f4e525c75f7",
      "metadata": {
        "id": "99690444-5ea1-43fc-8667-4f4e525c75f7",
        "outputId": "70184d49-4e06-48df-8c19-9d67ab8fcf30",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1 0\n"
          ]
        }
      ],
      "source": [
        "# step activation function\n",
        "def activation_step(x):\n",
        "    if x > 0:\n",
        "        y = 1\n",
        "    else:\n",
        "        y = 0\n",
        "    \n",
        "    return y\n",
        "    \n",
        "y1 = activation_step(3)\n",
        "y2 = activation_step(-4)\n",
        "print(y1, y2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "43c74b49-7f35-4249-9a59-751e39ea962c",
      "metadata": {
        "id": "43c74b49-7f35-4249-9a59-751e39ea962c"
      },
      "source": [
        "## The Linear Activation Function\n",
        "Is simply the equation of a line `y = wx + b`\\\n",
        "This activation function is usually applied to the last layer's ouput in the case of a regression model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "348ea3ce-bd17-41cf-a176-219e75b5f2fb",
      "metadata": {
        "id": "348ea3ce-bd17-41cf-a176-219e75b5f2fb",
        "outputId": "a9f46cf8-2320-473e-8fcb-81209e9080f8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "7\n"
          ]
        }
      ],
      "source": [
        "# linear activation function\n",
        "def activation_linear(x, w, b):\n",
        "    return w * x + b\n",
        "\n",
        "y1 = activation_linear(3, 2, 1)\n",
        "print(y1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "80a60c72-b37c-40ed-96e9-930dd8355895",
      "metadata": {
        "id": "80a60c72-b37c-40ed-96e9-930dd8355895"
      },
      "source": [
        "## The Sigmoid Activation Function\n",
        "The original, more granular, than the step activation function `y = 1 / (1 + e^^-x)`\\\n",
        "This function return a value in the range of 0 for negative infinity, through 0.5 for the input of 0, and to 1 for positive infinity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "e3b7080d-0e9f-4f60-9c64-c19cc134b738",
      "metadata": {
        "id": "e3b7080d-0e9f-4f60-9c64-c19cc134b738",
        "outputId": "f2319c84-4707-46fd-b4c0-f1bf2eaa0789",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.9820137900379085\n"
          ]
        }
      ],
      "source": [
        "# sigmoid activation function\n",
        "def activation_sigmoid(x):\n",
        "    s = 1 / (1 + np.exp(-x))\n",
        "    \n",
        "    return s\n",
        "\n",
        "y1 = activation_sigmoid(4)\n",
        "print(y1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eae14421-e703-460a-a78d-776cb37f4e43",
      "metadata": {
        "id": "eae14421-e703-460a-a78d-776cb37f4e43"
      },
      "source": [
        "## The Rectified Linear Activation Function\n",
        "The `ReLU` activation function is simpler than the sigmoid, It's quite literally `y = x`, clipped at `0` from the negative side. `y = x if x > 0 else y = 0`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "2c565c93-056b-4923-934e-b6e76221c304",
      "metadata": {
        "id": "2c565c93-056b-4923-934e-b6e76221c304",
        "outputId": "bd2988be-4e32-49e4-baba-044a3bc4b578",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3.4 0\n"
          ]
        }
      ],
      "source": [
        "# relu activation function\n",
        "def activation_relu(x):\n",
        "    if x > 0:\n",
        "        y = x\n",
        "    elif x < 0:\n",
        "        y = 0\n",
        "    \n",
        "    return y\n",
        "\n",
        "y1 = activation_relu(3.4)\n",
        "y2 = activation_relu(-3)\n",
        "print(y1, y2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "e5b2dafa-1ced-4585-8689-4325465521ea",
      "metadata": {
        "id": "e5b2dafa-1ced-4585-8689-4325465521ea",
        "outputId": "dc5bb83c-8417-4a60-c6e1-722d5caaea37",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0, 2, 0, 3.3, 0, 1.1, 2.2, 0]\n"
          ]
        }
      ],
      "source": [
        "inputs = [0, 2, -1, 3.3, -2.7, 1.1, 2.2, -100]\n",
        "\n",
        "output = []\n",
        "for i in inputs:\n",
        "    output.append(max(0, i))\n",
        "\n",
        "print(output)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ebd2610-1f1a-4a27-9faf-dd9a804a54bb",
      "metadata": {
        "id": "5ebd2610-1f1a-4a27-9faf-dd9a804a54bb"
      },
      "source": [
        "## The Softmax Activation Function\n",
        "The softmax activation function is return confidence scores for each class and will add up to 1. `Sij = ezi,j / sum(ezi,j)`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "ebb36de3-1372-4bad-ad1f-7281d7460371",
      "metadata": {
        "id": "ebb36de3-1372-4bad-ad1f-7281d7460371",
        "outputId": "a03524ba-5b70-482e-e58a-e2a129ae3836",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[121.51041752   3.35348465  10.85906266]\n"
          ]
        }
      ],
      "source": [
        "# exponentiate the output, we do this with Euler's number \"e\"\n",
        "E = 2.71828182846     # E = math.e\n",
        "\n",
        "# values from the previous output\n",
        "layer_outputs = [4.8, 1.21, 2.385]\n",
        "\n",
        "# for each value in a vector, calc the exponential value\n",
        "exp_values = np.exp(layer_outputs)\n",
        "print(exp_values)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f2344b2c-66f9-4c37-8746-9d7b49d68947",
      "metadata": {
        "id": "f2344b2c-66f9-4c37-8746-9d7b49d68947"
      },
      "source": [
        "The purpose of this exponentiatioin is to Get the probabilities of this outputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "092dab19-db9f-491e-9581-3cdac1dc801d",
      "metadata": {
        "id": "092dab19-db9f-491e-9581-3cdac1dc801d",
        "outputId": "dc95dbcd-81b1-4cd1-8189-ee93f97fe5ed",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Normalized exponentiated values:  [0.89528266 0.02470831 0.08000903]\n"
          ]
        }
      ],
      "source": [
        "# calculate the norm probabilities \n",
        "# first normalize values\n",
        "norm_base = sum(exp_values)\n",
        "\n",
        "norm_values = exp_values / np.sum(exp_values)\n",
        "print('Normalized exponentiated values: ', norm_values)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "1c9f2099-451c-47b1-b8db-d9388ad68663",
      "metadata": {
        "id": "1c9f2099-451c-47b1-b8db-d9388ad68663",
        "outputId": "40b8e9ad-4768-48b6-e8fb-d1f14eaae661",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Note\n",
            "18.172\n",
            "We only need to add up the rows axis and keepdims\n",
            "[[8.395]\n",
            " [7.29 ]\n",
            " [2.487]]\n"
          ]
        }
      ],
      "source": [
        "layer_outputs = np.array([[4.8, 1.21, 2.385], \n",
        "                          [8.9, -1.81, 0.2], \n",
        "                          [1.41, 1.051, 0.026]])\n",
        "print('Note')\n",
        "print(np.sum(layer_outputs))\n",
        "\n",
        "print('We only need to add up the rows axis and keepdims')\n",
        "print(np.sum(layer_outputs, axis=1, keepdims=True))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "3f89e948-c691-482a-959a-ca7e9ea61973",
      "metadata": {
        "id": "3f89e948-c691-482a-959a-ca7e9ea61973",
        "outputId": "cc4e28da-3563-48e6-aba2-31389812a98c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[8.95282664e-01 2.47083068e-02 8.00090293e-02]\n",
            " [9.99811129e-01 2.23163963e-05 1.66554348e-04]\n",
            " [5.13097164e-01 3.58333899e-01 1.28568936e-01]]\n"
          ]
        }
      ],
      "source": [
        "# get normalize probabilities\n",
        "probabilities = np.exp(layer_outputs)/np.sum(np.exp(layer_outputs), axis=1, keepdims=True)\n",
        "\n",
        "print(probabilities)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6e3d6c24-e334-4026-bb94-4b15325824c4",
      "metadata": {
        "id": "6e3d6c24-e334-4026-bb94-4b15325824c4"
      },
      "source": [
        "## Why Use Activation Functions?\n",
        "In real world and real problems, there are a number of factors that come into play, that lead to make our model nonlinear.\\\n",
        "So use linear activation is just not going to work."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ae8c4c6-254c-4565-ad8b-ad5a290a8dc7",
      "metadata": {
        "tags": [],
        "id": "1ae8c4c6-254c-4565-ad8b-ad5a290a8dc7"
      },
      "source": [
        "---\n",
        "## Our code so far"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "8038def3-2641-418f-b1a3-1deee9846f8d",
      "metadata": {
        "id": "8038def3-2641-418f-b1a3-1deee9846f8d"
      },
      "outputs": [],
      "source": [
        "# import dataset \n",
        "import nnfs\n",
        "from nnfs.datasets import spiral_data\n",
        "\n",
        "# set random seed to 0, create float32 dtype, overrides the original dot product from Numpy\n",
        "nnfs.init()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "cd589ffb-e306-45df-a02b-8fbdf5d1b321",
      "metadata": {
        "id": "cd589ffb-e306-45df-a02b-8fbdf5d1b321"
      },
      "outputs": [],
      "source": [
        "class Dense:\n",
        "    def __init__(self, n_inputs, n_neurons):\n",
        "        \"\"\" Initialize the weights and biases of each neurons\n",
        "        n_inputs = number of input features\n",
        "        n_neurons = number of desired neurons\n",
        "        \"\"\"\n",
        "        # using np.random.randn and * 0.01 is to break the symetry of the neurons\n",
        "        self.weights = np.random.randn(n_inputs, n_neurons) * 0.01\n",
        "        # biases can be initialize as zeros\n",
        "        self.biases = np.zeros((1, n_neurons))\n",
        "    \n",
        "    def forward(self, inputs):\n",
        "        \"\"\" Calculate the output layer using The Dot product of input feature and weight plus bias\n",
        "        Input:\n",
        "        inputs = Training examples\n",
        "        \n",
        "        Output:\n",
        "        output = Output of the training example\n",
        "        \"\"\"\n",
        "        # calculate the output layer\n",
        "        output = np.dot(inputs, self.weights) + self.biases\n",
        "        \n",
        "        return output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "98b85654-e53f-4111-aa62-b0b8382cefc3",
      "metadata": {
        "id": "98b85654-e53f-4111-aa62-b0b8382cefc3"
      },
      "outputs": [],
      "source": [
        "# ReLU activation\n",
        "class Activation_ReLU:\n",
        "    def forward(self, inputs):\n",
        "        output = np.maximum(0, inputs)\n",
        "        \n",
        "        return output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "a9aeaec7-ec6b-45dd-95cf-0c437a9be431",
      "metadata": {
        "id": "a9aeaec7-ec6b-45dd-95cf-0c437a9be431"
      },
      "outputs": [],
      "source": [
        "# Sotfmax activation\n",
        "class Activation_Softmax:\n",
        "    def forward(self, inputs):\n",
        "        # input - np.max to prevent the exponential function from overflowing\n",
        "        exp_values = np.exp(inputs - np.max(inputs, axis=1, keepdims=True))\n",
        "        \n",
        "        softmax = exp_values / np.sum(exp_values, axis=1, keepdims=True)\n",
        "        return softmax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "88032b5e-a7e9-4143-9292-e82348476a83",
      "metadata": {
        "id": "88032b5e-a7e9-4143-9292-e82348476a83",
        "outputId": "d301649c-79ed-43d0-ad7a-cafb59629a27",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.33333334 0.33333334 0.33333334]\n",
            " [0.33332068 0.33335868 0.33332068]\n",
            " [0.3332981  0.33340386 0.3332981 ]\n",
            " [0.3332748  0.3334504  0.3332748 ]\n",
            " [0.33325398 0.33349204 0.33325398]\n",
            " [0.33329442 0.3334112  0.33329442]\n",
            " [0.33321366 0.33357266 0.33321366]\n",
            " [0.33321416 0.33357167 0.33321416]\n",
            " [0.33318758 0.33362478 0.33318758]\n",
            " [0.33315328 0.33369344 0.33315328]]\n"
          ]
        }
      ],
      "source": [
        "# create dataset\n",
        "X, y = spiral_data(samples=100, classes=3)\n",
        "\n",
        "# create dense layer with 2 input features and 3 output values\n",
        "l1 = Dense(2, 3)\n",
        "a1 = Activation_ReLU()\n",
        "\n",
        "# create dense layer with 3 input features and 3 output values\n",
        "l2 = Dense(3, 3)\n",
        "a2 = Activation_Softmax()\n",
        "\n",
        "# forward pass through activation func\n",
        "yhat1 = a1.forward(l1.forward(X))\n",
        "yhat2 = a2.forward(yhat1)\n",
        "\n",
        "print(yhat2[:10])"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "torch110",
      "language": "python",
      "name": "torch110"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "name": "04_nnfs_activation_functions.ipynb",
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}