docs/r2/hyperparameter_tuning_with_hparams.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hyperparameter_tuning_with_hparams.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "TsHV-7cpVkyK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "metadata": {
        "id": "atWM-s8yVnfX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TB0wBWfcVqHz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Hyperparameter Tuning with the HParams Dashboard\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tensorboard/r2/hyperparameter_tuning_with_hparams\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/r2/hyperparameter_tuning_with_hparams.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorboard/blob/master/docs/r2/hyperparameter_tuning_with_hparams.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "metadata": {
        "id": "elH58gbhWAmn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "When building machine learning models, you need to choose various [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)), such as the dropout rate in a layer or the learning rate. These decisions impact model metrics, such as accuracy. Therefore, an important step in the machine learning workflow is to identify the best hyperparameters for your problem, which often involves experimentation. This process is known as \"Hyperparameter Optimization\" or \"Hyperparameter Tuning\".\n",
        "\n",
        "The HParams dashboard in TensorBoard provides several tools to help with this process of identifying the best experiment or most promising sets of hyperparameters. \n",
        "\n",
        "This tutorial will focus on the following steps:\n",
        "1. Experiment setup and HParams summary\n",
        "2. Adapt TensorFlow runs to log hyperparameters and metrics\n",
        "3. Start runs and log them all under one parent directory\n",
        "4. Visualize the results in TensorBoard's HParams dashboard\n",
        "\n",
        "Note: The HParams summary APIs and dashboard UI are in a preview stage and will change over time. \n",
        "\n",
        "Start by installing TF 2.0 and loading the TensorBoard notebook extension:"
      ]
    },
    {
      "metadata": {
        "id": "8p3Tbx8cWEFA",
        "colab_type": "code",
        "outputId": "b118c764-4a7f-45a6-ea3a-303bb1565991",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install -q tf-nightly-2.0-preview\n",
        "# Load the TensorBoard notebook extension\n",
        "%load_ext tensorboard.notebook "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K    100% |████████████████████████████████| 79.8MB 324kB/s \n",
            "\u001b[K    100% |████████████████████████████████| 348kB 19.2MB/s \n",
            "\u001b[K    100% |████████████████████████████████| 3.0MB 11.4MB/s \n",
            "\u001b[K    100% |████████████████████████████████| 61kB 18.2MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lEWCCQYkWIdA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Clear any logs from previous runs\n",
        "!rm -rf ./logs/ "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9GtR_cTTkf9G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Import TensorFlow and other packages needed for the HParams dashboard:"
      ]
    },
    {
      "metadata": {
        "id": "mVtYvbbIWRkV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import datetime\n",
        "import tensorflow as tf\n",
        "\n",
        "# Imports for the HParams plugin\n",
        "from tensorboard.plugins.hparams import api_pb2\n",
        "from tensorboard.plugins.hparams import summary as hparams_summary\n",
        "from google.protobuf import struct_pb2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XfCa27_8kov6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Download the [FashionMNIST](https://github.com/zalandoresearch/fashion-mnist) dataset and scale it:"
      ]
    },
    {
      "metadata": {
        "id": "z8b82G7YksOS",
        "colab_type": "code",
        "outputId": "bd771c37-6624-448d-92be-8e7700270f01",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "fashion_mnist = tf.keras.datasets.fashion_mnist\n",
        "\n",
        "(x_train, y_train),(x_test, y_test) = fashion_mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
            "32768/29515 [=================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
            "26427392/26421880 [==============================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
            "8192/5148 [===============================================] - 0s 0us/step\n",
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
            "4423680/4422102 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "0tsTxO85WaYq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 1. Experiment setup and the HParams experiment summary\n",
        "\n",
        "Experiment with three hyperparameters in the model:\n",
        "\n",
        "1. Number of units in the first dense layer\n",
        "2. Dropout rate in the dropout layer\n",
        "3. Optimizer\n",
        "\n",
        "List the values to try:"
      ]
    },
    {
      "metadata": {
        "id": "5Euw0agpWb4V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "num_units_list = [16, 32]\n",
        "dropout_rate_list = [0.1, 0.2] \n",
        "optimizer_list = ['adam', 'sgd'] "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DCbssXK_Wle3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Log an experiment summary. This is how the HParams dashboard knows what are the hyperparameters and metrics (API details are [here](https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/hparams/api.proto)):"
      ]
    },
    {
      "metadata": {
        "id": "t_usTF5rWl8t",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def create_experiment_summary(num_units_list, dropout_rate_list, optimizer_list):\n",
        "  num_units_list_val = struct_pb2.ListValue()\n",
        "  num_units_list_val.extend(num_units_list)\n",
        "  dropout_rate_list_val = struct_pb2.ListValue()\n",
        "  dropout_rate_list_val.extend(dropout_rate_list)\n",
        "  optimizer_list_val = struct_pb2.ListValue()\n",
        "  optimizer_list_val.extend(optimizer_list)\n",
        "  return hparams_summary.experiment_pb(\n",
        "      # The hyperparameters being changed\n",
        "      hparam_infos=[\n",
        "          api_pb2.HParamInfo(name='num_units',\n",
        "                             display_name='Number of units',\n",
        "                             type=api_pb2.DATA_TYPE_FLOAT64,\n",
        "                             domain_discrete=num_units_list_val),\n",
        "          api_pb2.HParamInfo(name='dropout_rate',\n",
        "                             display_name='Dropout rate',\n",
        "                             type=api_pb2.DATA_TYPE_FLOAT64,\n",
        "                             domain_discrete=dropout_rate_list_val),\n",
        "          api_pb2.HParamInfo(name='optimizer',\n",
        "                             display_name='Optimizer',\n",
        "                             type=api_pb2.DATA_TYPE_STRING,\n",
        "                             domain_discrete=optimizer_list_val)\n",
        "      ],\n",
        "      # The metrics being tracked\n",
        "      metric_infos=[\n",
        "          api_pb2.MetricInfo(\n",
        "              name=api_pb2.MetricName(\n",
        "                  tag='accuracy'),\n",
        "              display_name='Accuracy'),\n",
        "      ]\n",
        "  )\n",
        "\n",
        "exp_summary = create_experiment_summary(num_units_list, dropout_rate_list, optimizer_list)\n",
        "root_logdir_writer = tf.summary.create_file_writer(\"logs/hparam_tuning\")\n",
        "with root_logdir_writer.as_default():\n",
        "  tf.summary.import_event(tf.compat.v1.Event(summary=exp_summary).SerializeToString())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "va9XMh-dW4_f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 2. Adapt TensorFlow runs to log hyperparameters and metrics\n",
        "\n",
        "The model will be quite simple: two dense layers with a dropout layer between them. The training code will look familiar, although the hyperparameters are no longer hardcoded. Instead, the hyperparameters are provided in an `hparams` dictionary and used throughout the training function:"
      ]
    },
    {
      "metadata": {
        "id": "hG-zalNfW5Zl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_test_model(hparams):\n",
        "\n",
        "  model = tf.keras.models.Sequential([\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dense(hparams['num_units'], activation=tf.nn.relu),\n",
        "    tf.keras.layers.Dropout(hparams['dropout_rate']),\n",
        "    tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
        "  ])\n",
        "  model.compile(optimizer=hparams['optimizer'],\n",
        "                loss='sparse_categorical_crossentropy',\n",
        "                metrics=['accuracy'])\n",
        "\n",
        "  model.fit(x_train, y_train, epochs=1) # Run with 1 epoch to speed things up for demo purposes\n",
        "  _, accuracy = model.evaluate(x_test, y_test)\n",
        "  return accuracy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "46oJF8seXM7v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For each run, log an hparams summary with the hyperparameters and final accuracy:"
      ]
    },
    {
      "metadata": {
        "id": "8j-fO6nEXRfW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def run(run_dir, hparams):\n",
        "  writer = tf.summary.create_file_writer(run_dir)\n",
        "  summary_start = hparams_summary.session_start_pb(hparams=hparams)\n",
        "\n",
        "  with writer.as_default():\n",
        "    accuracy = train_test_model(hparams)\n",
        "    summary_end = hparams_summary.session_end_pb(api_pb2.STATUS_SUCCESS)\n",
        "      \n",
        "    tf.summary.scalar('accuracy', accuracy, step=1, description=\"The accuracy\")\n",
        "    tf.summary.import_event(tf.compat.v1.Event(summary=summary_start).SerializeToString())\n",
        "    tf.summary.import_event(tf.compat.v1.Event(summary=summary_end).SerializeToString())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "u2nOgIKAXdcb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 3. Start runs and log them all under one parent directory\n",
        "\n",
        "You can now try multiple experiments, training each one with a different set of hyperparameters. \n",
        "\n",
        "For simplicity, try all combinations (this is called a grid search). For more complex scenarios, it might be more effective to choose each hyperparameter value randomly (this is called a random search). There are more advanced methods that can be used.\n",
        "\n",
        "Run a few experiments, which will take a few minutes:"
      ]
    },
    {
      "metadata": {
        "id": "KbqT5n-AXd0h",
        "colab_type": "code",
        "outputId": "833c2156-edf2-488f-a2a0-4dc814294dd2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561
        }
      },
      "cell_type": "code",
      "source": [
        "session_num = 0\n",
        "\n",
        "for num_units in num_units_list:\n",
        "  for dropout_rate in dropout_rate_list:\n",
        "    for optimizer in optimizer_list:\n",
        "      hparams = {'num_units': num_units, 'dropout_rate': dropout_rate, 'optimizer': optimizer}\n",
        "      print('--- Running training session %d' % (session_num + 1))\n",
        "      print(hparams)\n",
        "      run_name = \"run-%d\" % session_num\n",
        "      run(\"logs/hparam_tuning/\" + run_name, hparams)\n",
        "      session_num += 1\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--- Running training session 1\n",
            "{'num_units': 16, 'dropout_rate': 0.1, 'optimizer': 'adam'}\n",
            "60000/60000 [==============================] - 4s 60us/sample - loss: 0.7210 - accuracy: 0.7441\n",
            "10000/10000 [==============================] - 0s 38us/sample - loss: 0.5077 - accuracy: 0.8219\n",
            "--- Running training session 2\n",
            "{'num_units': 16, 'dropout_rate': 0.1, 'optimizer': 'sgd'}\n",
            "60000/60000 [==============================] - 3s 52us/sample - loss: 1.8030 - accuracy: 0.4162\n",
            "10000/10000 [==============================] - 0s 39us/sample - loss: 1.3829 - accuracy: 0.6366\n",
            "--- Running training session 3\n",
            "{'num_units': 16, 'dropout_rate': 0.2, 'optimizer': 'adam'}\n",
            "60000/60000 [==============================] - 3s 58us/sample - loss: 0.7570 - accuracy: 0.7339\n",
            "10000/10000 [==============================] - 0s 39us/sample - loss: 0.5112 - accuracy: 0.8199\n",
            "--- Running training session 4\n",
            "{'num_units': 16, 'dropout_rate': 0.2, 'optimizer': 'sgd'}\n",
            "60000/60000 [==============================] - 3s 51us/sample - loss: 1.8379 - accuracy: 0.3270\n",
            "10000/10000 [==============================] - 0s 38us/sample - loss: 1.4468 - accuracy: 0.5909\n",
            "--- Running training session 5\n",
            "{'num_units': 32, 'dropout_rate': 0.1, 'optimizer': 'adam'}\n",
            "60000/60000 [==============================] - 4s 61us/sample - loss: 0.5985 - accuracy: 0.7936\n",
            "10000/10000 [==============================] - 0s 40us/sample - loss: 0.4662 - accuracy: 0.8335\n",
            "--- Running training session 6\n",
            "{'num_units': 32, 'dropout_rate': 0.1, 'optimizer': 'sgd'}\n",
            "60000/60000 [==============================] - 3s 56us/sample - loss: 1.7895 - accuracy: 0.4130\n",
            "10000/10000 [==============================] - 0s 41us/sample - loss: 1.3227 - accuracy: 0.6278\n",
            "--- Running training session 7\n",
            "{'num_units': 32, 'dropout_rate': 0.2, 'optimizer': 'adam'}\n",
            "60000/60000 [==============================] - 4s 64us/sample - loss: 0.6546 - accuracy: 0.7748\n",
            "10000/10000 [==============================] - 0s 39us/sample - loss: 0.4731 - accuracy: 0.8340\n",
            "--- Running training session 8\n",
            "{'num_units': 32, 'dropout_rate': 0.2, 'optimizer': 'sgd'}\n",
            "60000/60000 [==============================] - 3s 55us/sample - loss: 1.6611 - accuracy: 0.4529\n",
            "10000/10000 [==============================] - 0s 40us/sample - loss: 1.1841 - accuracy: 0.6557\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qSyjWQ3mPKT9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4. Visualize the results in TensorBoard's HParams plugin"
      ]
    },
    {
      "metadata": {
        "id": "5VBvplwyP8Vy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The HParams dashboard can now be opened. Start TensorBoard and click on \"HParams\" at the top."
      ]
    },
    {
      "metadata": {
        "id": "Xf4KM-U2bbP_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%tensorboard --logdir logs/hparam_tuning"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UTWg9nXnxWWI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img class=\"tfo-display-only-on-site\" src=\"images/hparams_table.png?raw=1\"/>"
      ]
    },
    {
      "metadata": {
        "id": "4RPGbR9EWd4o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The left pane of the dashboard provides filtering capabilities that are active across all the views in the HParams dashboard:\n",
        "- Filter which hyperparameters/metrics are shown in the dashboard\n",
        "- Filter which hyperparameter/metrics values are shown in the dashboard\n",
        "- Filter on run status (running, success, ...)\n",
        "- Sort by hyperparameter/metric in the table view\n",
        "- Number of session groups to show (useful for performance when there are many experiments)\n"
      ]
    },
    {
      "metadata": {
        "id": "Z3Q5v28XaCt8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The HParams dashboard has three different views, with various useful information:\n",
        "* The **Table View** lists the runs, their hyperparameters, and their metrics.\n",
        "* The **Parallel Coordinates View** shows each run as a line going through an axis for each hyperparemeter and metric. Click and drag the mouse on any axis to mark a region which will highlight only the runs that pass through it. This can be useful for identifying which groups of hyperparameters are most important. The axes themselves can be re-ordered by dragging them.\n",
        "* The **Scatter Plot View** shows plots comparing each hyperparameter/metric with each metric. This can help identify correlations. Click and drag to select a region in a specific plot and highlight those sessions across the other plots. \n",
        "\n",
        "A table row, a parallel coordinates line, and a scatter plot market can be clicked to see a plot of the metrics as a function of training steps for that session (although in this tutorial only one step is used for each run)."
      ]
    },
    {
      "metadata": {
        "id": "fh3p0DtcdOK1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To further explore the capabilities of the HParams dashboard, download a set of pregenerated logs with more experiments:"
      ]
    },
    {
      "metadata": {
        "id": "oxrSUAnCeFmQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "wget -q 'https://storage.googleapis.com/download.tensorflow.org/tensorboard/hparams_demo_logs.zip'\n",
        "unzip -q hparams_demo_logs.zip -d logs/hparam_demo"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "__8xQhjqgr3D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "View these logs in TensorBoard:"
      ]
    },
    {
      "metadata": {
        "id": "KBHp6M_zgjp4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%tensorboard --logdir logs/hparam_demo"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Po7rTfQswAMT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img class=\"tfo-display-only-on-site\" src=\"images/hparams_parallel_coordinates.png?raw=1\"/>"
      ]
    },
    {
      "metadata": {
        "id": "IlDz2oXBgnZ9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You can try out the different views in the HParams dashboard. \n",
        "\n",
        "For example, by going to the parallel coordinates view and clicking and dragging on the accuracy axis, you can select the runs with the highest accuracy. As these runs pass through 'adam' in the optimizer axis, you can conclude that 'adam' performed better than 'sgd' on these experiments."
      ]
    }
  ]
}