codes/ipython/1-basics/graph.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "graph.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MDevcewD85Im"
      },
      "source": [
        "## Intorduction to TensorFlow graphs.\n",
        "\n",
        "For long, the big complaint about TensorFlow was *It's not flexible for debugging!* With the advent of TensorFlow 2.0, that changed drastically.\n",
        "\n",
        "Now TensorFlow allows you to run the oprations **eagerly**. That means, you can run TensorFlow operations by Python and return the outputs to Python again. That creates a lot of flexibility, especially for debugging. \n",
        "\n",
        "But there are some merits in NOT using the eagerly option. You can run operations on TensorFlow graphs that in some scenarios leads to significant speed up. According to TensorFlow:\n",
        "\n",
        "> Graphs are data structures that contain a set of [tf.Operation](https://www.tensorflow.org/api_docs/python/tf/Operation) objects, which represent units of computation; and [tf.Tensor](https://www.tensorflow.org/api_docs/python/tf/Tensor) objects, which represent the units of data that flow between operations. They are defined in a [tf.Graph](https://www.tensorflow.org/api_docs/python/tf/Graph) context. Since these graphs are data structures, they can be saved, run, and restored all without the original Python code.\n",
        "\n",
        "Let's have some example for transforming functions to graphs!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJoLp_aUvBFR"
      },
      "source": [
        "# Loading necessary libraries\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import timeit"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8a9c8qUj-MSj"
      },
      "source": [
        "### Operation\n",
        "\n",
        "We can take a Python function on graph with [@tf.function](https://www.tensorflow.org/api_docs/python/tf/function) decorator."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V-ugyJGRHc7S",
        "outputId": "0bc0f944-4f6b-4fc2-95e2-8f517141cac6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "@tf.function\n",
        "def multiply_fn(a, b):\n",
        "  return tf.matmul(a, b)\n",
        "\n",
        "# Create some tensors\n",
        "a = tf.constant([[0.5, 0.5]])\n",
        "b = tf.constant([[10.0], [1.0]])\n",
        "\n",
        "# Check function\n",
        "print('Multiple a of shape {} with b of shape {}'.format(a.shape, b.shape))\n",
        "print(multiply_fn(a, b).numpy())\n"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Multiple a of shape (1, 2) with b of shape (2, 1)\n",
            "[[5.5]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zoh1CmJ59n-N",
        "outputId": "addb1205-94a9-4f7b-d5b1-882ebed24a8a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "# Function without neing take to graph, i.e., with eager execution.\n",
        "def add_fn(a, b):\n",
        "  return tf.add(a, b)\n",
        "\n",
        "# Create some tensors\n",
        "a = tf.constant([[0.5, 0.5]])\n",
        "b = tf.constant([[10.0], [1.0]])\n",
        "\n",
        "# Check function\n",
        "print('Add a of shape {} with b of shape {}'.format(a.shape, b.shape))\n",
        "print(add_fn(a, b).numpy())"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Add a of shape (1, 2) with b of shape (2, 1)\n",
            "[[10.5 10.5]\n",
            " [ 1.5  1.5]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-WywaJU_MXR"
      },
      "source": [
        "### Speedup\n",
        "\n",
        "Now let's define a custom model and run it:\n",
        "\n",
        "1. eagerly\n",
        "2. on graph\n",
        "\n",
        "To check how to define models refer to: https://www.tensorflow.org/api_docs/python/tf/keras/Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ynGAYcDC-y2W",
        "outputId": "98d5fb24-d34b-4c53-85a9-b7456cfca4da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "class ModelShallow(tf.keras.Model):\n",
        "\n",
        "  def __init__(self):\n",
        "    super(ModelShallow, self).__init__()\n",
        "    self.dense1 = tf.keras.layers.Dense(10, activation=tf.nn.relu)\n",
        "    self.dense2 = tf.keras.layers.Dense(20, activation=tf.nn.relu)\n",
        "    self.dense3 = tf.keras.layers.Dense(30, activation=tf.nn.softmax)\n",
        "    self.dropout = tf.keras.layers.Dropout(0.5)\n",
        "\n",
        "  def call(self, inputs, training=False):\n",
        "    x = self.dense1(inputs)\n",
        "    if training:\n",
        "      x = self.dropout(x, training=training)\n",
        "    x = self.dense2(x)\n",
        "    out = self.dense3(x)\n",
        "    return out\n",
        "\n",
        "class ModelDeep(tf.keras.Model):\n",
        "\n",
        "  def __init__(self):\n",
        "    super(ModelDeep, self).__init__()\n",
        "    self.dense1 = tf.keras.layers.Dense(1000, activation=tf.nn.relu)\n",
        "    self.dense2 = tf.keras.layers.Dense(2000, activation=tf.nn.relu)\n",
        "    self.dense3 = tf.keras.layers.Dense(3000, activation=tf.nn.softmax)\n",
        "    self.dropout = tf.keras.layers.Dropout(0.5)\n",
        "\n",
        "  def call(self, inputs, training=False):\n",
        "    x = self.dense1(inputs)\n",
        "    if training:\n",
        "      x = self.dropout(x, training=training)\n",
        "    x = self.dense2(x)\n",
        "    out = self.dense3(x)\n",
        "    return out\n",
        "\n",
        "# Create the model with eager esxecution by default\n",
        "model_shallow_with_eager = ModelShallow()\n",
        "\n",
        "# Take model to graph. \n",
        "# NOTE: Instead of using decorators, we can ditectly operate tf.function on the model.\n",
        "model_shallow_on_graph = tf.function(ModelShallow())\n",
        "\n",
        "# Model deep\n",
        "model_deep_with_eager = ModelDeep()\n",
        "model_deep_on_graph = tf.function(ModelDeep())\n",
        "\n",
        "# sample input\n",
        "sample_input = tf.random.uniform([60, 28, 28])\n",
        "\n",
        "# Check time for shallow model\n",
        "print(\"Shallow Model - Eager execution time:\", timeit.timeit(lambda: model_shallow_with_eager(sample_input), number=1000))\n",
        "print(\"Shallow Model - Graph-based execution time:\", timeit.timeit(lambda: model_shallow_on_graph(sample_input), number=1000))\n",
        "\n",
        "# Check time for deep model\n",
        "print(\"Deep Model - Eager execution time:\", timeit.timeit(lambda: model_deep_with_eager(sample_input), number=1000))\n",
        "print(\"Deep Model - Graph-based execution time:\", timeit.timeit(lambda: model_deep_on_graph(sample_input), number=1000))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shallow Model - Eager execution time: 2.758659444999921\n",
            "Shallow Model - Graph-based execution time: 1.1618621510001503\n",
            "Deep Model - Eager execution time: 477.634194022\n",
            "Deep Model - Graph-based execution time: 460.01053104599987\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBvht7tVAX4I"
      },
      "source": [
        ""
      ],
      "execution_count": 4,
      "outputs": []
    }
  ]
}