Introduction_to_callbacks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Introduction_to_callbacks.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": false
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m0DK47idkkGN"
      },
      "source": [
        "# Introduction to callbacks in keras\n",
        "\n",
        "Hi everyone my name is yashwanth let's see why callbacks is needed and how to implement it.\n",
        "\n",
        "Building Deep Learning models without callbacks is like driving a car with no functioning brakes — you have little to no control over the whole process that is very likely to result in a disaster. In this article, you will learn how to monitor and improve your Deep Learning models using Keras callbacks like ModelCheckpoint and EarlyStopping.\n",
        "\n",
        "Often, when training a very deep neural network, we want to stop training once the training accuracy reaches a certain desired threshold. Thus, we can achieve what we want (optimal model weights) and avoid wastage of resources (time and computation power).\n",
        "In this brief tutorial, let’s learn how to achieve this in Tensorflow and Keras.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-tHPp8j1lLcB"
      },
      "source": [
        "## What is callback in keras\n",
        "\n",
        "A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training.\n",
        "\n",
        "A custom callback is a powerful tool to customize the behavior of a Keras model during training, evaluation, or inference, including reading/changing the Keras model. Examples include tf.keras.callbacks.TensorBoard where the training progress and results can be exported and visualized with TensorBoard, or tf.keras.callbacks.ModelCheckpoint where the model is automatically saved during training, and more. In this guide, you will learn what Keras callback is, when it will be called, what it can do, and how you can build your own. Towards the end of this guide, there will be demos of creating a couple of simple callback applications to get you started on your custom callback."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "aaYA2Onjkev-",
        "colab": {}
      },
      "source": [
        "# Import tensorflow\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "05sg3wYhmoEP"
      },
      "source": [
        "## First I will say how to stop training a neural-network using callback"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iqlJdzsVm1W-"
      },
      "source": [
        "1.First, set the accuracy threshold till which you want to train your model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LLNVMQaom5rp",
        "colab": {}
      },
      "source": [
        "acc_thresh = 0.96"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_4AxNwqJnGrR"
      },
      "source": [
        "2.For implementing the callback first you have to create class and function.\n",
        "\n",
        "Now, let us implement it to stop training when accuracy reaches acc_thresh."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tJ6I4t2GnDR_",
        "colab": {}
      },
      "source": [
        "class myCallback(tf.keras.callbacks.Callback): \n",
        "    def on_epoch_end(self, epoch, logs={}): \n",
        "        if(logs.get('acc') > ACCURACY_THRESHOLD):   \n",
        "          print(\"\\nWe have reached %2.2f%% accuracy, so we will stopping training.\" %(acc_thresh*100))   \n",
        "        self.model.stop_training = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ma3XyXPVoJ1O"
      },
      "source": [
        "Now you may wonder what is going on here?\n",
        "\n",
        "We are creating new class by extending tf.keras.callbacks.Callback and implementing the on_epoch_end() method which will invoke at the end of epoch.\n",
        "\n",
        "Next, we are fetching the value of accuracy at the end of that epoch, and if it is greater than our threshold, we are setting the stop_training of model to True using the “self” keyword to access the attributes and methods of the class in python."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rTH45IPTpxsE"
      },
      "source": [
        "3.Now, let us create the instance of an object of myCallback class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rbImHEbJn7gh",
        "colab": {}
      },
      "source": [
        "callbacks = myCallback()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VtwQmfVUqqvZ"
      },
      "source": [
        "4.Next, build a Neural Network or Conv-Net or any model following the normal steps of TensorFlow or Keras.You can pass a callbacks (as the keyword argument callbacks) to any of tf.keras.Model.fit(), tf.keras.Model.evaluate(), and tf.keras.Model.predict() methods. The methods of the callbacks will then be called at different stages of training/evaluating/inference.\n",
        "\n",
        "callbacks=[the newly instantiated object of myCallback class] "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0GkIxQBnqlqm",
        "colab": {}
      },
      "source": [
        "model.fit(x_train, y_train, epochs=20, callbacks=[callbacks])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8kBc4cZRraaG"
      },
      "source": [
        "And that’s all! While training, as soon as accuracy reaches the value set in acc_thresh, training will be stopped."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e-OsI3Ddr08y"
      },
      "source": [
        "In the above example you can also use on_epoch_begin() which is called at the beginning of an epoch during training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "smhlvT8Gzb4U"
      },
      "source": [
        "## Saving the model automatically in training.\n",
        "Let's say you want to save your model when validation accuracy reaches minimum in between training before overfitting this can be implemented using the ModelCheckpoint in callback which can be done as shown below. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HvI5LKa8GK-j",
        "colab_type": "text"
      },
      "source": [
        "ModelCheckpoint\n",
        " \n",
        "This callback saves the model after every epoch. Here are some relevant metrics:\n",
        "\n",
        "filepath: the file path you want to save your model in\n",
        "\n",
        "\n",
        "monitor: the value being monitored\n",
        "\n",
        "\n",
        "save_best_only: set this to True if you do not want to overwrite the latest best model\n",
        "\n",
        "\n",
        "mode: auto, min, or max. For example, you set mode=’min’ if the monitored value is val_loss and you want to minimize it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LtBt7gSWGK-k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import ModelCheckpoint\n",
        "best_model = ModelCheckpoint(filepath,\n",
        "                             monitor='val_loss',\n",
        "                             mode='min',\n",
        "                             save_best_only=True,\n",
        "                             verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eYmz1U150crs",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import ModelCheckpoint\n",
        "# autosave best Model\n",
        "best_model_file = \"vgg.h5\"\n",
        "best_model = ModelCheckpoint(best_model_file, monitor='val_acc', verbose=1, save_best_only=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aY-mBb7U0ztS"
      },
      "source": [
        "Here we are monitoring the validation accuracy say you want training accuracy then just change the monitor value as 'acc' as shown below you can also put 'loss' for training loss and 'val_loss' for validation accuracy.\n",
        "\n",
        "```\n",
        "best_model = ModelCheckpoint(best_model_file, monitor='acc', verbose=1, save_best_only=True)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "HZPrugZJ0m8W"
      },
      "source": [
        "This can be passed same as last example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VLLRKxQ60jew",
        "colab": {}
      },
      "source": [
        "model.fit(x_train, y_train, epochs=20, callbacks=[best_model])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2bIXXczGK-v",
        "colab_type": "text"
      },
      "source": [
        "## LearningRateScheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4QNy889gGK-w",
        "colab_type": "text"
      },
      "source": [
        "This one is pretty straightforward: it adjusts the learning rate over time using a schedule that you already write beforehand. This function returns the desired learning rate (output) based on the current epoch (epoch index as input)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Ep2vPStGK-x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import LearningRateScheduler\n",
        "lrs = LearningRateScheduler(schedule, verbose=0) # schedule is a function"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3F_0FjZbscBO"
      },
      "source": [
        "## Early stopping at minimum loss and Usage of logs dict"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GXazoj46GK-0",
        "colab_type": "text"
      },
      "source": [
        "Overfitting is a nightmare for Machine Learning practitioners. One way to avoid overfitting is to terminate the process early. The EarlyStoppingfunction has various metrics/arguments that you can modify to set up when the training process should stop. Here are some relevant metrics:\n",
        "\n",
        "monitor: value being monitored, i.e: val_loss\n",
        "\n",
        "\n",
        "min_delta: minimum change in the monitored value. For example, min_delta=1 means that the training process will be stopped if the absolute change of the monitored value is less than 1\n",
        "\n",
        "\n",
        "patience: number of epochs with no improvement after which training will be stopped\n",
        "\n",
        "\n",
        "restore_best_weights: set this metric to True if you want to keep the best weights once stopped\n",
        "\n",
        "\n",
        "The code example below will define an EarlyStopping function that tracks the val_loss value, stops the training if there are no changes towards val_loss after 3 epochs, and keeps the best weights once the training stops:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CAWQomNqGK-2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "earlystop = EarlyStopping(monitor = 'val_loss',\n",
        "                          min_delta = 0,\n",
        "                          patience = 3,\n",
        "                          verbose = 1,\n",
        "                          restore_best_weights = True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qzQyNCgLynU5",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-P3XWb14va4e"
      },
      "source": [
        "Let's build a small model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NS2Jl9hevfbl",
        "colab": {}
      },
      "source": [
        "# Define the Keras model to add callbacks to\n",
        "def get_model():\n",
        "  model = tf.keras.Sequential()\n",
        "  model.add(tf.keras.layers.Dense(1, activation = 'linear', input_dim = 784))\n",
        "  model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.1), loss='mean_squared_error', metrics=['mae'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xc1N7sf8vjph"
      },
      "source": [
        "Then, we will load the MNIST data for training and testing from Keras datasets API:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "aj9tJlNRvqpx",
        "colab": {}
      },
      "source": [
        "# Load example MNIST data and pre-process it\n",
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
        "x_train = x_train.reshape(60000, 784).astype('float32') / 255\n",
        "x_test = x_test.reshape(10000, 784).astype('float32') / 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Bl30_LYixMEq"
      },
      "source": [
        "The logs dict contains the loss value, and all the metrics at the end of a batch or epoch. Example includes the loss and mean absolute error."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Jqoj4MyExPs_",
        "outputId": "ec5cab85-7531-43a9-d875-b1d9a17213dc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 319
        }
      },
      "source": [
        "class LossAndErrorPrintingCallback(tf.keras.callbacks.Callback):\n",
        "\n",
        "  def on_train_batch_end(self, batch, logs=None):\n",
        "    print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))\n",
        "\n",
        "  def on_test_batch_end(self, batch, logs=None):\n",
        "    print('For batch {}, loss is {:7.2f}.'.format(batch, logs['loss']))\n",
        "\n",
        "  def on_epoch_end(self, epoch, logs=None):\n",
        "    print('The average loss for epoch {} is {:7.2f} and mean absolute error is {:7.2f}.'.format(epoch, logs['loss'], logs['mae']))\n",
        "    \n",
        "model = get_model()\n",
        "_ = model.fit(x_train, y_train,\n",
        "          batch_size=64,\n",
        "          steps_per_epoch=5,\n",
        "          epochs=3,\n",
        "          verbose=0,\n",
        "          callbacks=[LossAndErrorPrintingCallback()])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "For batch 0, loss is   33.11.\n",
            "For batch 1, loss is 1044.98.\n",
            "For batch 2, loss is   28.39.\n",
            "For batch 3, loss is   11.97.\n",
            "For batch 4, loss is    7.52.\n",
            "The average loss for epoch 0 is  225.19 and mean absolute error is    8.96.\n",
            "For batch 0, loss is    6.47.\n",
            "For batch 1, loss is    6.41.\n",
            "For batch 2, loss is    4.25.\n",
            "For batch 3, loss is    6.59.\n",
            "For batch 4, loss is    4.22.\n",
            "The average loss for epoch 1 is    5.59 and mean absolute error is    1.95.\n",
            "For batch 0, loss is    5.12.\n",
            "For batch 1, loss is    5.22.\n",
            "For batch 2, loss is    5.25.\n",
            "For batch 3, loss is    5.40.\n",
            "For batch 4, loss is    4.37.\n",
            "The average loss for epoch 2 is    5.07 and mean absolute error is    1.83.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yNGN6CWltBtP"
      },
      "source": [
        "Belowshowcases the creation of a Callback that stops the Keras training when the minimum of loss has been reached by mutating the attribute model.stop_training (boolean). Optionally, the user can provide an argument patience to specify how many epochs the training should wait before it eventually stops."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZBbpvbL0waNX"
      },
      "source": [
        "Now let us create class for it and pass it to callbacks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "4N-MCB_pu8xQ",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "class EarlyStoppingAtMinLoss(tf.keras.callbacks.Callback):\n",
        "  \"\"\"Stop training when the loss is at its min, i.e. the loss stops decreasing.\n",
        "\n",
        "  Arguments:\n",
        "      patience: Number of epochs to wait after min has been hit. After this\n",
        "      number of no improvement, training stops.\n",
        "  \"\"\"\n",
        "\n",
        "  def __init__(self, patience=0):\n",
        "    super(EarlyStoppingAtMinLoss, self).__init__()\n",
        "\n",
        "    self.patience = patience\n",
        "\n",
        "    # best_weights to store the weights at which the minimum loss occurs.\n",
        "    self.best_weights = None\n",
        "\n",
        "  def on_train_begin(self, logs=None):\n",
        "    # The number of epoch it has waited when loss is no longer minimum.\n",
        "    self.wait = 0\n",
        "    # The epoch the training stops at.\n",
        "    self.stopped_epoch = 0\n",
        "    # Initialize the best as infinity.\n",
        "    self.best = np.Inf\n",
        "\n",
        "  def on_epoch_end(self, epoch, logs=None):\n",
        "    current = logs.get('loss')\n",
        "    if np.less(current, self.best):\n",
        "      self.best = current\n",
        "      self.wait = 0\n",
        "      # Record the best weights if current results is better (less).\n",
        "      self.best_weights = self.model.get_weights()\n",
        "    else:\n",
        "      self.wait += 1\n",
        "      if self.wait >= self.patience:\n",
        "        self.stopped_epoch = epoch\n",
        "        self.model.stop_training = True\n",
        "        print('\\n Restoring model weights from the end of the best epoch.')\n",
        "        self.model.set_weights(self.best_weights)\n",
        "\n",
        "  def on_train_end(self, logs=None):\n",
        "    if self.stopped_epoch > 0:\n",
        "      print('Epoch %05d: early stopping' % (self.stopped_epoch + 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qFruNOmdryhX",
        "outputId": "b9db3fb9-d134-4257-b452-73b93fb3b00d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 470
        }
      },
      "source": [
        "model = get_model()\n",
        "_ = model.fit(x_train, y_train,\n",
        "          batch_size=64,\n",
        "          steps_per_epoch=5,\n",
        "          epochs=30,\n",
        "          verbose=0,\n",
        "          callbacks=[LossAndErrorPrintingCallback(),EarlyStoppingAtMinLoss()])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "For batch 0, loss is   34.13.\n",
            "For batch 1, loss is  811.08.\n",
            "For batch 2, loss is   22.56.\n",
            "For batch 3, loss is    7.79.\n",
            "For batch 4, loss is   11.28.\n",
            "The average loss for epoch 0 is  177.37 and mean absolute error is    8.24.\n",
            "For batch 0, loss is    6.41.\n",
            "For batch 1, loss is    5.56.\n",
            "For batch 2, loss is    5.50.\n",
            "For batch 3, loss is    4.42.\n",
            "For batch 4, loss is    4.99.\n",
            "The average loss for epoch 1 is    5.38 and mean absolute error is    1.89.\n",
            "For batch 0, loss is    4.15.\n",
            "For batch 1, loss is    4.79.\n",
            "For batch 2, loss is    4.93.\n",
            "For batch 3, loss is    5.29.\n",
            "For batch 4, loss is    4.05.\n",
            "The average loss for epoch 2 is    4.64 and mean absolute error is    1.74.\n",
            "For batch 0, loss is    5.63.\n",
            "For batch 1, loss is    6.46.\n",
            "For batch 2, loss is    8.21.\n",
            "For batch 3, loss is   13.19.\n",
            "For batch 4, loss is   35.99.\n",
            "The average loss for epoch 3 is   13.90 and mean absolute error is    2.88.\n",
            "\n",
            " Restoring model weights from the end of the best epoch.\n",
            "Epoch 00004: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Xf6NwuIiwoqB"
      },
      "source": [
        "Now, as you can see above as the loss is increased or did.t improve it stopped."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8uV0wmzat_9Z",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}