DLwithPython/chapter07_working_with_keras_a_deep_dive.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/theaveas/DeepLearning/blob/main/DLwithPython/chapter07_working_with_keras_a_deep_dive.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "55ea2cb6",
      "metadata": {
        "id": "55ea2cb6"
      },
      "source": [
        "# Working with Keras a Deep Dive \n",
        "- Creating Keras models with the `Sequential` class, the `Function API`, and `Model Subclassing`\n",
        "- Using built-in **Keras** `Training` and `Evaluation Loops`\n",
        "- Using **Keras** `callbacks` to customize training\n",
        "- Using **TensorBoard** to `Monitor training` and `Evaluation Metrics`\n",
        "- Writing `Training` and `Evaluation Loops` from sratch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9d238f0f-1404-4dfa-95bf-cd824bc7b125",
      "metadata": {
        "id": "9d238f0f-1404-4dfa-95bf-cd824bc7b125"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "from tensorflow import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d5d8ccc9-7c48-46f0-be06-75efd9941ead",
      "metadata": {
        "id": "d5d8ccc9-7c48-46f0-be06-75efd9941ead"
      },
      "source": [
        "## 3 different ways to build Keras model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7c38b5d7",
      "metadata": {
        "id": "7c38b5d7"
      },
      "source": [
        "#### The Sequential Model\n",
        "It's the most user-friendly API; it's just a Python list. Pros: Ideal for inexperienced users. Cons: constrained to simple layer stacks\n",
        "\n",
        "The Sequential class"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "78d16fd3",
      "metadata": {
        "id": "78d16fd3",
        "outputId": "1e6cf601-d126-41da-a8c1-a7e646584d11"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metal device set to: Apple M1 Pro\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:32:08.194205: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.\n",
            "2022-03-22 19:32:08.194567: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential([\n",
        "    keras.layers.Dense(64, activation='relu'),\n",
        "    keras.layers.Dense(64, activation='relu'),\n",
        "    keras.layers.Dense(32, activation='relu'),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "66cfd218",
      "metadata": {
        "id": "66cfd218"
      },
      "source": [
        "Incrementally building a Sequential model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4e5051d9",
      "metadata": {
        "id": "4e5051d9"
      },
      "outputs": [],
      "source": [
        "model.add(keras.layers.Dense(10, activation='softmax'))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c29ebeec",
      "metadata": {
        "id": "c29ebeec"
      },
      "source": [
        "Build a model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8dfe0738",
      "metadata": {
        "id": "8dfe0738"
      },
      "outputs": [],
      "source": [
        "model.build(input_shape=(None, 3))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d123354d",
      "metadata": {
        "id": "d123354d"
      },
      "source": [
        "Checking a model weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "50798562",
      "metadata": {
        "id": "50798562",
        "outputId": "e46b4f6d-c603-4b79-eba3-7f7628711d20"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<tf.Variable 'dense/kernel:0' shape=(3, 64) dtype=float32, numpy=\n",
              " array([[ 1.25467777e-05, -1.20943949e-01, -1.53224885e-01,\n",
              "         -8.68571550e-02, -2.70546883e-01,  2.32463598e-01,\n",
              "         -1.17774010e-02, -1.55422240e-01,  1.75913930e-01,\n",
              "         -1.48749545e-01, -1.92857236e-01,  4.96582389e-02,\n",
              "          2.62617826e-01, -2.12030932e-01, -4.90057766e-02,\n",
              "          2.77251422e-01, -2.07304090e-01,  7.02629387e-02,\n",
              "         -2.47921050e-01, -1.81252211e-01, -4.16259468e-02,\n",
              "          1.98020071e-01, -2.68627912e-01, -7.23579228e-02,\n",
              "         -2.56985307e-01,  1.61339074e-01,  1.27872527e-02,\n",
              "         -8.86335522e-02,  1.85892224e-01,  6.56902790e-02,\n",
              "         -1.59544766e-01, -1.51976869e-01,  2.46656239e-01,\n",
              "         -2.89082438e-01,  2.63829231e-01,  2.53752351e-01,\n",
              "         -1.26475513e-01,  2.46111333e-01, -2.90879339e-01,\n",
              "          1.89787298e-01,  1.33743256e-01,  1.66108102e-01,\n",
              "          2.44359970e-02, -1.22500747e-01, -2.04461336e-01,\n",
              "          4.82690930e-02,  1.86969787e-01, -1.63155094e-01,\n",
              "         -2.08721057e-01, -5.14401346e-02, -2.15568990e-01,\n",
              "         -2.17748135e-01,  1.22167051e-01,  2.93354452e-01,\n",
              "          1.04533672e-01, -2.82850862e-03,  1.83087051e-01,\n",
              "          7.65433908e-02, -1.83464423e-01,  8.05057883e-02,\n",
              "         -1.40189499e-01, -2.56147325e-01, -5.23834229e-02,\n",
              "         -1.08752251e-01],\n",
              "        [ 1.68894917e-01, -1.19604483e-01, -2.77842015e-01,\n",
              "          1.91288233e-01,  1.80112958e-01, -8.97714794e-02,\n",
              "          7.43171871e-02,  1.78879559e-01, -2.48219490e-01,\n",
              "         -1.49840727e-01,  1.17870748e-01, -1.90737069e-01,\n",
              "         -4.67436314e-02,  2.81482458e-01,  3.78446877e-02,\n",
              "         -1.37740999e-01,  1.75786376e-01,  1.72731489e-01,\n",
              "          9.64832008e-02,  1.90989971e-02,  1.27421588e-01,\n",
              "          1.30673021e-01,  1.15753859e-01, -4.97652590e-02,\n",
              "         -1.15272969e-01,  2.10058093e-01, -6.37007654e-02,\n",
              "         -1.78784966e-01, -2.10460722e-01, -2.42572054e-01,\n",
              "         -5.02876639e-02,  8.88800025e-02,  1.55306786e-01,\n",
              "          2.59564877e-01,  5.80516160e-02, -1.33494541e-01,\n",
              "          1.55130655e-01, -2.01331243e-01,  9.04893875e-02,\n",
              "         -2.62216866e-01,  1.83067441e-01, -1.72395080e-01,\n",
              "         -2.77689546e-01, -2.07145214e-02,  2.86661983e-02,\n",
              "          2.55653203e-01, -2.65052021e-02,  2.40538061e-01,\n",
              "          1.73357129e-02, -3.96214426e-02, -1.28612652e-01,\n",
              "         -1.03285253e-01, -3.93720865e-02,  2.35715091e-01,\n",
              "         -2.90021390e-01,  1.89053059e-01, -3.21231782e-02,\n",
              "         -2.05761492e-01, -1.44897997e-01, -2.88010240e-01,\n",
              "          1.21548265e-01,  2.63834178e-01, -1.37903377e-01,\n",
              "          4.20773029e-02],\n",
              "        [-1.12175852e-01, -2.45310664e-01, -5.38835824e-02,\n",
              "          1.55822933e-01,  4.33686674e-02,  6.78300858e-02,\n",
              "          2.03861713e-01, -7.74129629e-02,  2.63399065e-01,\n",
              "         -1.97912067e-01,  2.78954804e-02,  9.22730565e-02,\n",
              "          7.91594684e-02,  2.90642440e-01, -1.75511539e-01,\n",
              "         -2.79767245e-01,  4.07696962e-02,  1.87247545e-01,\n",
              "          1.35470331e-02, -2.95490295e-01,  2.48761594e-01,\n",
              "         -1.67214975e-01, -5.64879775e-02, -1.81534886e-02,\n",
              "          1.99342281e-01, -2.90745974e-01, -2.64091581e-01,\n",
              "         -3.60403061e-02, -2.72606194e-01, -7.85619318e-02,\n",
              "          2.65550554e-01,  5.49407899e-02,  6.45723343e-02,\n",
              "         -4.84595895e-02, -5.01751900e-03,  9.32589173e-03,\n",
              "          2.70602226e-01,  2.45234370e-01, -1.61137313e-01,\n",
              "         -3.71607244e-02, -1.80515558e-01,  9.34494436e-02,\n",
              "         -7.83915669e-02, -6.30508661e-02,  2.70294070e-01,\n",
              "         -3.95724177e-02,  1.80946559e-01,  1.88871354e-01,\n",
              "          2.94512749e-01,  1.29878789e-01, -1.98482841e-01,\n",
              "         -2.75816023e-01,  1.23565674e-01,  2.17490494e-01,\n",
              "          9.77080166e-02, -2.44492024e-01, -9.09340829e-02,\n",
              "         -1.34325877e-01, -1.80202991e-01, -2.08540320e-01,\n",
              "         -2.10384235e-01,  2.93843389e-01, -1.83829367e-02,\n",
              "         -1.86482206e-01]], dtype=float32)>,\n",
              " <tf.Variable 'dense/bias:0' shape=(64,) dtype=float32, numpy=\n",
              " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>,\n",
              " <tf.Variable 'dense_1/kernel:0' shape=(64, 64) dtype=float32, numpy=\n",
              " array([[ 9.08970833e-06, -8.75017196e-02, -1.10856637e-01, ...,\n",
              "         -1.85319975e-01, -3.78988683e-02, -7.86811411e-02],\n",
              "        [ 1.22193739e-01, -8.65326226e-02, -2.01015860e-01, ...,\n",
              "          1.90881327e-01, -9.97716859e-02,  3.04424912e-02],\n",
              "        [-8.11580867e-02, -1.77479744e-01, -3.89842093e-02, ...,\n",
              "          2.12592676e-01, -1.32998824e-02, -1.34917974e-01],\n",
              "        ...,\n",
              "        [-1.19983777e-01, -1.08133465e-01, -1.47370979e-01, ...,\n",
              "         -1.15739912e-01,  2.14914873e-01, -6.13653958e-02],\n",
              "        [-1.87747359e-01, -1.21977307e-01, -1.28846049e-01, ...,\n",
              "          1.34940743e-02, -3.64735723e-04, -6.61859065e-02],\n",
              "        [ 2.03194335e-01, -7.90140778e-02, -5.04009426e-03, ...,\n",
              "         -2.16090769e-01,  2.09318697e-02, -1.81587443e-01]], dtype=float32)>,\n",
              " <tf.Variable 'dense_1/bias:0' shape=(64,) dtype=float32, numpy=\n",
              " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>,\n",
              " <tf.Variable 'dense_2/kernel:0' shape=(64, 32) dtype=float32, numpy=\n",
              " array([[ 1.04904175e-05, -1.01038277e-01, -1.28006220e-01, ...,\n",
              "          5.48785925e-02, -1.33285940e-01, -1.26963615e-01],\n",
              "        [ 2.06060112e-01, -2.41503537e-01,  2.20406651e-01, ...,\n",
              "         -2.13989079e-01, -4.37618494e-02, -9.08531547e-02],\n",
              "        [ 1.41097188e-01, -9.99192595e-02, -2.32113123e-01, ...,\n",
              "         -2.02648103e-01, -4.20110226e-02,  7.42515922e-02],\n",
              "        ...,\n",
              "        [ 2.45189428e-01, -1.37803197e-01, -8.40236545e-02, ...,\n",
              "          1.81472421e-01, -6.10299706e-02, -1.67017341e-01],\n",
              "        [-1.61628962e-01,  6.86041117e-02, -7.93519616e-02, ...,\n",
              "          2.12800384e-01,  1.07716382e-01, -3.89832258e-02],\n",
              "        [-2.05989480e-01, -4.35201526e-02, -1.02497935e-01, ...,\n",
              "          1.42340302e-01, -1.04183972e-01, -9.27844048e-02]], dtype=float32)>,\n",
              " <tf.Variable 'dense_2/bias:0' shape=(32,) dtype=float32, numpy=\n",
              " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       dtype=float32)>,\n",
              " <tf.Variable 'dense_3/kernel:0' shape=(32, 10) dtype=float32, numpy=\n",
              " array([[ 1.58548355e-05, -1.52755514e-01, -1.93527207e-01,\n",
              "         -1.09702945e-01, -3.41708094e-01,  2.93607920e-01,\n",
              "         -1.48751736e-02, -1.96302533e-01,  2.22184092e-01,\n",
              "         -1.87874734e-01],\n",
              "        [-2.43583962e-01,  6.27197027e-02,  3.31693500e-01,\n",
              "         -2.67800868e-01, -6.18956089e-02,  3.50176066e-01,\n",
              "         -2.61830747e-01,  8.87440145e-02, -3.13131034e-01,\n",
              "         -2.28926510e-01],\n",
              "        [-5.25746942e-02,  2.50104755e-01, -3.39284390e-01,\n",
              "         -9.13900435e-02, -3.24579448e-01,  2.03775674e-01,\n",
              "          1.61506236e-02, -1.11946613e-01,  2.34786958e-01,\n",
              "          8.29686224e-02],\n",
              "        [-2.01509401e-01, -1.91950947e-01,  3.11533600e-01,\n",
              "         -3.65119010e-01,  3.33223552e-01,  3.20496112e-01,\n",
              "         -1.59742028e-01,  3.10845286e-01, -3.67388546e-01,\n",
              "          2.39706546e-01],\n",
              "        [ 1.68921381e-01,  2.09799021e-01,  3.08633149e-02,\n",
              "         -1.54721797e-01, -2.58240223e-01,  6.09652102e-02,\n",
              "          2.36147970e-01, -2.06069335e-01, -2.63620377e-01,\n",
              "         -6.49703145e-02],\n",
              "        [-2.72269517e-01, -2.75021851e-01,  1.54300362e-01,\n",
              "          3.70514661e-01,  1.32028908e-01, -3.57246399e-03,\n",
              "          2.31243938e-01,  9.66763794e-02, -2.31720567e-01,\n",
              "          1.01681024e-01],\n",
              "        [-1.77063152e-01, -3.23521078e-01, -6.61616921e-02,\n",
              "         -1.37357056e-01,  2.13318914e-01, -1.51063725e-01,\n",
              "         -3.50922048e-01,  2.41602272e-01,  2.27487594e-01,\n",
              "         -1.13383830e-01],\n",
              "        [ 9.38646495e-02,  2.25929767e-01, -3.13508004e-01,\n",
              "         -1.89252928e-01,  1.48873955e-01, -2.40906134e-01,\n",
              "         -5.90384603e-02,  3.55520040e-01,  4.77988422e-02,\n",
              "         -1.73970640e-01],\n",
              "        [ 2.22022980e-01,  2.18164593e-01,  1.21860921e-01,\n",
              "          2.41225362e-02,  1.60936922e-01,  1.65043622e-01,\n",
              "          1.46200269e-01, -6.28548861e-02, -1.45592913e-01,\n",
              "          2.65309125e-01],\n",
              "        [-8.04558098e-02, -2.25810274e-01, -2.65817642e-01,\n",
              "         -3.06375146e-01, -6.35147095e-02,  1.12257868e-01,\n",
              "          1.96156710e-01,  3.27837497e-01,  7.33207762e-02,\n",
              "         -1.68607250e-01],\n",
              "        [ 1.95934266e-01, -2.54286855e-01,  1.14290535e-01,\n",
              "         -3.31187040e-01,  2.31219202e-01, -2.17739701e-01,\n",
              "         -3.50729465e-01, -2.61629820e-02,  3.62061858e-02,\n",
              "          3.22896928e-01],\n",
              "        [-3.34767997e-02,  3.03806156e-01,  2.18954682e-02,\n",
              "         -5.00429571e-02, -1.62441298e-01, -1.30452096e-01,\n",
              "         -4.97280061e-02,  2.97714561e-01, -3.66304934e-01,\n",
              "          2.38779157e-01],\n",
              "        [-4.05724645e-02, -2.59882390e-01, -1.83010131e-01,\n",
              "         -3.63764793e-01,  1.53518766e-01,  3.33229750e-01,\n",
              "         -1.74175739e-01,  5.31447530e-02, -1.41681165e-01,\n",
              "         -3.09834063e-01],\n",
              "        [-6.80564344e-02,  1.96808606e-01,  5.47758341e-02,\n",
              "          8.56712461e-02,  2.57482916e-01, -9.77746844e-02,\n",
              "          3.32680255e-01, -2.49968335e-01,  3.52327526e-02,\n",
              "          1.16543382e-01],\n",
              "        [ 9.99805629e-02,  3.67089361e-01, -2.21675858e-01,\n",
              "         -3.53353679e-01,  5.14932573e-02,  2.36498743e-01,\n",
              "          1.71102583e-02, -3.73212308e-01,  3.14192683e-01,\n",
              "         -2.11197078e-01],\n",
              "        [-7.13458359e-02, -2.29283571e-02,  2.51774758e-01,\n",
              "         -3.67220104e-01, -3.33554864e-01, -4.55198884e-02,\n",
              "         -3.44309062e-01, -9.92258787e-02,  3.35397631e-01,\n",
              "          6.93917274e-02],\n",
              "        [ 8.15566480e-02, -6.12058043e-02, -6.33725524e-03,\n",
              "          1.17788613e-02,  3.41778010e-01,  3.09737712e-01,\n",
              "         -2.03520820e-01, -4.69350219e-02, -2.27996081e-01,\n",
              "          1.18029177e-01],\n",
              "        [-9.90106761e-02, -7.96349645e-02,  3.41388851e-01,\n",
              "         -4.99810576e-02,  2.28540450e-01,  2.38549680e-01,\n",
              "          3.71977657e-01,  1.64040476e-01, -2.50689268e-01,\n",
              "         -3.48363191e-01],\n",
              "        [ 1.56066805e-01,  2.74696440e-01,  1.23407871e-01,\n",
              "         -3.08800101e-01, -1.14852220e-01, -1.69657245e-01,\n",
              "         -2.27601290e-01, -2.63392121e-01, -2.65721023e-01,\n",
              "          3.71132225e-01],\n",
              "        [-2.32181549e-02, -2.35532120e-01,  2.76052505e-01,\n",
              "         -5.63066900e-02, -2.56766081e-01,  1.05848908e-03,\n",
              "          5.97636998e-02, -6.07863963e-02, -5.52996695e-02,\n",
              "         -1.54944360e-02],\n",
              "        [ 3.59164208e-01, -5.29347956e-02, -2.14234725e-01,\n",
              "         -8.83108377e-02, -1.09006107e-01, -1.34255886e-01,\n",
              "          1.35865301e-01,  3.07273567e-02, -2.74101436e-01,\n",
              "          4.23029959e-02],\n",
              "        [-2.07845926e-01, -2.72755682e-01,  5.24776578e-02,\n",
              "          2.69247919e-01,  1.82192177e-01, -2.12906808e-01,\n",
              "         -8.63516331e-03,  1.79357201e-01,  7.91874528e-02,\n",
              "          1.48479730e-01],\n",
              "        [ 3.40347886e-02, -2.03563809e-01,  1.12679005e-02,\n",
              "          1.40257984e-01, -1.07474923e-02,  3.20291519e-03,\n",
              "          3.74997407e-01, -1.04956478e-01, -1.82567120e-01,\n",
              "         -2.47522652e-01],\n",
              "        [-4.71895039e-02, -6.72902763e-02, -2.01415226e-01,\n",
              "         -1.45903528e-01,  3.35455090e-01,  2.12159246e-01,\n",
              "          2.40528017e-01,  1.23854071e-01,  3.34199280e-01,\n",
              "          3.32923532e-02],\n",
              "        [ 3.97446752e-03, -2.08659649e-01,  1.95483774e-01,\n",
              "          2.55418986e-01,  2.07138270e-01,  2.36818999e-01,\n",
              "         -1.32094026e-02, -1.49946034e-01, -2.35622495e-01,\n",
              "          2.46473521e-01],\n",
              "        [-1.70605153e-01,  4.95523214e-04,  1.47091478e-01,\n",
              "          6.65082633e-02,  1.44847482e-01,  2.49750704e-01,\n",
              "          1.06710643e-01,  1.94663197e-01,  3.37813050e-01,\n",
              "         -3.53520811e-02],\n",
              "        [-4.68506813e-02,  1.52009279e-01, -2.68673956e-01,\n",
              "          1.33318812e-01,  2.09226102e-01, -1.29592404e-01,\n",
              "          3.09265107e-01, -1.50333792e-01,  2.38118380e-01,\n",
              "          5.71832955e-02],\n",
              "        [ 1.13858551e-01, -2.34053984e-01, -3.22547853e-02,\n",
              "          1.39678687e-01,  3.43796998e-01, -1.85991362e-01,\n",
              "          2.81860560e-01, -2.28852704e-01,  1.42080814e-01,\n",
              "         -2.90234149e-01],\n",
              "        [-2.59836972e-01, -2.07459241e-01,  6.59272075e-02,\n",
              "         -1.52806520e-01,  3.94001603e-02,  1.11343473e-01,\n",
              "         -2.22317562e-01,  2.06849903e-01, -1.67051792e-01,\n",
              "         -1.50845915e-01],\n",
              "        [ 1.29008919e-01, -2.50254869e-02, -1.32368803e-01,\n",
              "          2.53504485e-01, -1.57895967e-01, -4.29106355e-02,\n",
              "         -1.56305268e-01,  2.13418454e-01,  1.36320204e-01,\n",
              "          6.28139675e-02],\n",
              "        [-1.35060236e-01, -2.60581136e-01, -3.75977010e-01,\n",
              "         -5.98400235e-02,  1.64761096e-01,  9.81793702e-02,\n",
              "         -6.22160733e-02, -7.19103217e-02,  1.45683199e-01,\n",
              "          1.98753446e-01],\n",
              "        [ 2.35953838e-01,  3.76663059e-01,  2.33644933e-01,\n",
              "         -6.02036715e-04,  1.03798240e-01,  7.10918307e-02,\n",
              "         -2.88765639e-01,  1.93439156e-01, -2.60225445e-01,\n",
              "         -7.16120601e-02]], dtype=float32)>,\n",
              " <tf.Variable 'dense_3/bias:0' shape=(10,) dtype=float32, numpy=array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.weights"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "88615553",
      "metadata": {
        "id": "88615553"
      },
      "source": [
        "Summary a model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b181ace0",
      "metadata": {
        "id": "b181ace0",
        "outputId": "c29e02b4-752a-47dd-8d31-987bdc837140"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 64)                256       \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 64)                4160      \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 32)                2080      \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 10)                330       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 6,826\n",
            "Trainable params: 6,826\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "858d1056",
      "metadata": {
        "id": "858d1056"
      },
      "source": [
        "Naming Each layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d6f5994a",
      "metadata": {
        "id": "d6f5994a",
        "outputId": "dd39ef82-41eb-4f02-d714-6e2b61f4e66b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"test_model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " first_layer (Dense)         (None, None, 64)          256       \n",
            "                                                                 \n",
            " last_layer (Dense)          (None, None, 64)          4160      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 4,416\n",
            "Trainable params: 4,416\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model = keras.Sequential(name='test_model')\n",
        "model.add(keras.layers.Dense(64, activation='relu', input_shape=(None, 3), name='first_layer'))\n",
        "model.add(keras.layers.Dense(64, activation='softmax', name='last_layer'))\n",
        "\n",
        "# summary\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "71a97961",
      "metadata": {
        "id": "71a97961"
      },
      "source": [
        "#### The Functional API"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c3c4db02",
      "metadata": {
        "id": "c3c4db02"
      },
      "source": [
        "Common ways to build a Model.\n",
        "\\\n",
        "Simple Functional model with two Dense layers\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f19568db-5be0-4994-b0bf-9bcbd980d35b",
      "metadata": {
        "id": "f19568db-5be0-4994-b0bf-9bcbd980d35b"
      },
      "outputs": [],
      "source": [
        "inputs = keras.Input(shape=(3,), name='inputs')\n",
        "features = keras.layers.Dense(units=64, activation='relu')(inputs)\n",
        "outputs = keras.layers.Dense(units=10, activation='sigmoid')(features)\n",
        "\n",
        "model = keras.Model(inputs, outputs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "04248b0c-5425-42bf-a73a-542d12881372",
      "metadata": {
        "id": "04248b0c-5425-42bf-a73a-542d12881372",
        "outputId": "1bfc31b1-e7a5-4e91-b312-32f411395902"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " inputs (InputLayer)         [(None, 3)]               0         \n",
            "                                                                 \n",
            " dense_4 (Dense)             (None, 64)                256       \n",
            "                                                                 \n",
            " dense_5 (Dense)             (None, 10)                650       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 906\n",
            "Trainable params: 906\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e9e97910-c938-414b-bc3d-0c1d840a805c",
      "metadata": {
        "id": "e9e97910-c938-414b-bc3d-0c1d840a805c"
      },
      "source": [
        "Multi-inputs, Multi-outputs Model\n",
        "\\\n",
        "Building a system to rank customer support ticket by priority and route them to the appropriate department"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9bc0995b-3095-4e39-8962-c77571941615",
      "metadata": {
        "id": "9bc0995b-3095-4e39-8962-c77571941615"
      },
      "outputs": [],
      "source": [
        "vocab_size = 10000\n",
        "num_tags = 100\n",
        "num_departments = 4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c112c9fe-b77c-47fc-ac47-a94d48000995",
      "metadata": {
        "id": "c112c9fe-b77c-47fc-ac47-a94d48000995"
      },
      "outputs": [],
      "source": [
        "# inputs \n",
        "\n",
        "# title of the ticket\n",
        "title = keras.Input(shape=(vocab_size,), name='title')\n",
        "\n",
        "# text body of the ticket\n",
        "text_body = keras.Input(shape=(vocab_size,), name='text_body')\n",
        "\n",
        "# tags added by the user\n",
        "tags = keras.Input(shape=(num_tags, ), name='tags')\n",
        "\n",
        "# features\n",
        "features = keras.layers.Concatenate()([title, text_body, tags])\n",
        "features = keras.layers.Dense(units=64, activation='relu')(features)\n",
        "\n",
        "# outputs\n",
        "priority = keras.layers.Dense(units=1, activation='sigmoid', name='priority')(features)\n",
        "department = keras.layers.Dense(units=num_departments, activation='softmax', name='department')(features)\n",
        "\n",
        "# create the model by specifying its inputs and outputs\n",
        "model = keras.Model(inputs= [title, text_body, tags], outputs= [priority, department])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5859a047-16be-4404-a1ee-272021b89bb6",
      "metadata": {
        "id": "5859a047-16be-4404-a1ee-272021b89bb6",
        "outputId": "51a94879-af9c-4063-83d2-d8ec239dd680"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " title (InputLayer)             [(None, 10000)]      0           []                               \n",
            "                                                                                                  \n",
            " text_body (InputLayer)         [(None, 10000)]      0           []                               \n",
            "                                                                                                  \n",
            " tags (InputLayer)              [(None, 100)]        0           []                               \n",
            "                                                                                                  \n",
            " concatenate_2 (Concatenate)    (None, 20100)        0           ['title[0][0]',                  \n",
            "                                                                  'text_body[0][0]',              \n",
            "                                                                  'tags[0][0]']                   \n",
            "                                                                                                  \n",
            " dense_8 (Dense)                (None, 64)           1286464     ['concatenate_2[0][0]']          \n",
            "                                                                                                  \n",
            " priority (Dense)               (None, 1)            65          ['dense_8[0][0]']                \n",
            "                                                                                                  \n",
            " department (Dense)             (None, 4)            260         ['dense_8[0][0]']                \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 1,286,789\n",
            "Trainable params: 1,286,789\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8fe0aa46-ea30-4121-b0b7-679d07032e15",
      "metadata": {
        "id": "8fe0aa46-ea30-4121-b0b7-679d07032e15"
      },
      "source": [
        "Training a Multi-input, Multi-output Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d9b67e77-c233-4794-8e6e-995bc4129388",
      "metadata": {
        "id": "d9b67e77-c233-4794-8e6e-995bc4129388"
      },
      "outputs": [],
      "source": [
        "num_samples = 1280\n",
        "\n",
        "# dummies dataset\n",
        "title_data = np.random.randint(0, 2, size=(num_samples, vocab_size))\n",
        "text_body_data = np.random.randint(0, 2, size=(num_samples, vocab_size))\n",
        "tags_data = np.random.randint(0, 2, size=(num_samples, num_tags))\n",
        "\n",
        "priority_data = np.random.random(size=(num_samples, 1))\n",
        "department_data = np.random.randint(0, 2, size=(num_samples, num_departments))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "738e99ee-b109-461f-8634-0cce225e1452",
      "metadata": {
        "id": "738e99ee-b109-461f-8634-0cce225e1452",
        "outputId": "4b26d0f1-9fe1-403d-8558-23a886130c66"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:34:40.483487: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n",
            "2022-03-22 19:34:40.804034: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "40/40 [==============================] - 2s 11ms/step - loss: 281.1333 - priority_loss: 0.3256 - department_loss: 280.8076 - priority_mean_absolute_error: 0.4914 - department_accuracy: 0.2508\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x16000e3a0>"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# train model\n",
        "model.compile(\n",
        "    optimizer = keras.optimizers.Adam(learning_rate=1e-2),\n",
        "    loss = ['mean_squared_error', 'categorical_crossentropy'],\n",
        "    metrics = [['mean_absolute_error'], ['accuracy']]\n",
        ")\n",
        "\n",
        "model.fit(\n",
        "    [title_data, text_body_data, tags_data],\n",
        "    [priority_data, department_data],\n",
        "    epochs = 1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "18cfea09-1367-4787-bc58-a37db66e90fa",
      "metadata": {
        "id": "18cfea09-1367-4787-bc58-a37db66e90fa",
        "outputId": "8599b286-7b44-4f1c-9c24-0f5d7b591006"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "13/40 [========>.....................] - ETA: 0s - loss: 599.3840 - priority_loss: 0.3391 - department_loss: 599.0449 - priority_mean_absolute_error: 0.5012 - department_accuracy: 0.1202"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:34:42.614768: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "40/40 [==============================] - 1s 9ms/step - loss: 585.4515 - priority_loss: 0.3281 - department_loss: 585.1232 - priority_mean_absolute_error: 0.4936 - department_accuracy: 0.1211\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[585.4515380859375,\n",
              " 0.3281307816505432,\n",
              " 585.1232299804688,\n",
              " 0.4935564696788788,\n",
              " 0.12109375]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# predict\n",
        "\n",
        "model.evaluate([title_data, text_body_data, tags_data], [priority_data, department_data])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f604cd8e-4df4-465d-b5c4-960d0ee991f2",
      "metadata": {
        "id": "f604cd8e-4df4-465d-b5c4-960d0ee991f2"
      },
      "source": [
        "Retrieving the inputs or outputs of a layers in a Functional model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1b0a593c-4ef4-40ec-95cd-eda60a68f621",
      "metadata": {
        "id": "1b0a593c-4ef4-40ec-95cd-eda60a68f621",
        "outputId": "c1670e04-beb4-4b5c-f00b-9f15f57f37c5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<keras.layers.merge.Concatenate at 0x162405c10>,\n",
              " <keras.layers.core.dense.Dense at 0x1624a6f70>,\n",
              " <keras.layers.core.dense.Dense at 0x16cb16370>,\n",
              " <keras.layers.core.dense.Dense at 0x16cb16550>]"
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d6fc85d7-9d6c-44ee-8dca-0ece5c68526e",
      "metadata": {
        "id": "d6fc85d7-9d6c-44ee-8dca-0ece5c68526e",
        "outputId": "b6b752d9-0db4-48e3-b216-d9bdc35aa628"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[<KerasTensor: shape=(None, 10000) dtype=float32 (created by layer 'title')>,\n",
              " <KerasTensor: shape=(None, 10000) dtype=float32 (created by layer 'text_body')>,\n",
              " <KerasTensor: shape=(None, 100) dtype=float32 (created by layer 'tags')>]"
            ]
          },
          "execution_count": 39,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# retrive input \n",
        "model.layers[3].input"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4d659e17-6a76-4ba6-ba16-b3aa34541641",
      "metadata": {
        "id": "4d659e17-6a76-4ba6-ba16-b3aa34541641"
      },
      "outputs": [],
      "source": [
        "# build a new model using exist inputs\n",
        "inputs = model.layers[3].input\n",
        "features = model.layers[4].output\n",
        "\n",
        "difficulty = keras.layers.Dense(units=3, activation='softmax', name='difficulty')(features)\n",
        "\n",
        "new_model = keras.Model(inputs = inputs, outputs = [priority, department, difficulty])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1cf3a60a-fb2f-4537-83f9-2b19d5f4df71",
      "metadata": {
        "id": "1cf3a60a-fb2f-4537-83f9-2b19d5f4df71",
        "outputId": "259ab973-6714-4005-e127-561aab573138"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjQAAAFgCAIAAAAjHnQoAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deVwT1/o/8BOSgJTFIiCKWlE2RVywilBApbjicrUCLmxiXVGLehW1Xq9o1aLeWq3aWlEL/EoVsIKCRcUNF8QFFW1FdgQEWRTZIQmZ3x9zv2kuS0QImQE+7z94JTOTM8+cOZmHmTlnwqEoigAAALCJEtMBAAAANITkBAAArIPkBAAArIPkBAAArMOTfnP37t39+/czFQrI3bp166ytrdtYCFoFyB1aJjTWoFX8z5lTbm7umTNnFB7S+yUkJCQkJDAdRQdz5syZ3NzctpfD2lbBBmiZrdDpWyZaRSs0bhW8xguFh4crKp6WcnZ2JqwMjM04HI4cS0PlNwktsxU6fctEq2iFxq0C95wAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1kJwAAIB1mvjJjA9VWlpqY2OzceNGT0/PBrNEItG5c+d++umnGTNm+Pj4tH1dMjx9+vTp06eSt71793ZwcGjXNd6/fz81NVXylsfjzZs3r13XCISQ58+f//zzz/Hx8Q8ePGh1IS9fvjx27FhISEh2drb8QmsaWiY0hlbxXq1MTnV1dSoqKv8tgsfT1tZWV1dvPDcvLy8/P//q1atTpkxpe6yyDRs2TFlZ2crKqqysLDg4ePz48e20IsnWWVpa1tXVOTg4CIXCc+fOTZgwoZ3W2FFIt4r2KyErK+vixYsCgaAtK8rMzLxx40ZeXl5bCmkhtEzGtb1lyh1axXu18rLeli1bxGIx/VpDQ+PWrVtz5sxpPNfAwGDBggVtj7KFBg0aZGRkxOFwXF1duVxuO61Fetvt7Oz69Omjra09c+bMjz76qJ3W2FFI10z7lTBt2rSRI0e2ZS2EEHt7exsbmzYW0nJomcxqe8tsD2gVsrUmOT179uzo0aMtnMvjyeHKYct169aNy+UqKbXXvbTG266srKysrNxOq+tAZLcK+ZbA5/PbsiI5FtJyaJlMaXvLbD9oFTJ8cOa4c+fOggULqqqqTp8+zefznZ2da2trw8PD9fT0Jk2a1Hhu4x/fLS8vDw0NTU5OHjhw4MKFC6WvB8pdenp6YGDgjh07MjIywsLCevbsuXDhQvqolJGRERUVtWbNmtu3b8fExJiYmLi7uyspKYWGhorFYj6f7+TkRAg5c+aMUChUVVWdNWtW461rSQxpaWl//PHHu3fvLC0tp06dSgg5d+5cdXU1IYTD4dCXff/66y/6AvSkSZO0tbWbrKKMjIzAwEA/P7+YmJjnz5+vXbtWwYdXGZqsmcZbcfXq1aKiIvojjo6Oqamp6enphJBJkya9ePGi5XXL4XDodnXx4sW4uDgLCwsXFxfJ3Lq6uri4uLi4OH19/SlTphgaGkpmCYXCiIiIx48fjx8/XvLv5Ht3h1yr6r/QMhWjyZppvOG06urq//f//l9RUdHgwYMdHBw0NTWVlJTolkbvi379+ikpKS1durSdokWr+B+UlNDQ0AZTGrt165arqyshJDo6+tKlS8nJybNmzSKE7Nmzp/FciqLKysoIIfv27aM/npqaOmPGjEuXLj158sTc3NzQ0LC0tFT2GimKcnJycnJyeu9iFEXZ2NjweDz6dWBgoJ6eHiHk/PnzX3zxxbRp0wghW7dupSjq0KFD6urqvXv3DgkJGTp0qKqqKiFkzpw5FEWVl5fb2NhoamrSheTn5w8dOrRXr15Nbh1FUSYmJr17924untWrV9vZ2ZWUlFy+fJnD4fj7+1MUlZyc3Lt3b7od0IvV19c7ODgcPnxYLBY3WUVBQUG9evUihAQGBlpYWBBC7ty5I7sqCCGhoaEtqTTZWtEqqGZ29Nu3b728vAghbm5uFEXl5uaqqqpGR0eLxeIm67Y5Hh4eBgYGX3/9tZWVVZ8+fSQFUhRVU1Mzfvz406dPl5aWHjp0SEND4/fff6dnvXv3zsHBwc/P782bN0FBQcrKylwul3rf7pAdCVpmh2uZTW44RVHFxcWGhoZBQUECgcDX15cQYmBgYGtrS1GUr69vSEhIVVXVqVOn1NXV3xsYWoVcWsUHJyeKorZv304IkXxvX716JUlOjec2SE4TJ06MiIigX8fExEhqX7bW7WyKouhGdu7cOfqtvb29iYkJ/Xru3Llqamq//vorRVH5+fnW1taEEHoXrlq1SrKzKYpavHgxvbMbbx31vp3dvXv3nTt30q/NzMysrKzo1yEhIZLVURQlEAhGjRolEomo5qtoy5Yt9M6mKOrFixfvPW4q8hBANaqZ5rZCIBDY2tpqaGjk5OT4+PicPXu2uRJk8PDw0NDQuH//PkVRtbW19K3dy5cvUxS1YMECLy8vyZJOTk6qqqq5ubkURXl7e8+aNUsya/r06XRyomTuDtnQMqmO1jKb2/A1a9ZoamoKhUKKonJzcwkhX3/9NUVRAoFAW1s7JSWFXszHx+e9a0SroOTRKuRwrbPl1+UKCgpiY2Pj4+M3b968efPmCxcujBo1ij5hbCdqamqEEEdHR/qtubm5pIOWmpqapqYm/Z9F7969v/32W0JIbGwsIaTBJeC2XBG+cOHCihUrCCH0kbSmpoaePnfuXCMjo//85z/024iIiFmzZnG5XBlVRP+7NH/+fEKIqalp4+ul7CFjK/h8fnBwMCFkxowZ3bp1mz17dutWoaOjM3r0aEKIiooKfZnl8uXL1dXV4eHh9H9qtBUrVtTU1Pzyyy9FRUUBAQGTJk2SzBo2bJjkdXO7o3WxtQRaJlOa2/C0tDTJRby+ffsaGRndvn2bEMLn8zU0NCZMmEAfeeljbjtBq5Amh94KLa+LtLQ0Qoivr6+Ojk7b19sSDWJTU1MTiUSSt9L1RR/p6P+Y5MjGxiYiIuLs2bOTJ082MDCgzzIJIVwud+PGjUuWLLl//76lpeWJEyeCgoKIzCpi+XdemuwdPWDAgN27d69evXrdunVyWd2kSZN4PF5+fn58fLxQKJTug2NsbEwISU1NTUpKEgqF9KUGmnR9Nrc72g9aJlOa23BbW9sLFy7cv3/f2tq6rq4uPz9/+vTp9KzDhw+7u7s7OjpaW1sHBgbq6uq2U2xoFdIU+oQIupfIo0ePpCdWVFQoMobmKCsrq6iofPLJJ/IqULLbTp48GRAQ4Obm1mCkhYeHR58+fXbt2pWSkvLxxx/Tx002V1HLyd4KsVh8+/ZtBweHr776Si7fru7du6uqqhobG9fX1xNC4uPjJbPo74yJiQm99oKCguYKaXJ3sAFapnw1t+Fr1651cnLy9fW9cuXK+vXrP/vssx07dtCzpk2blp6evmbNmsTExFGjRiUnJzMU+9+6QqtofXKiDwQfNNfU1JTL5W7btk0ygrK4uJi+msmI2tpayev4+Pi6ujpLS0tCiKamZl1dnWQWRVENNqfBW/qCqTSxWBwQEJCYmLhv376VK1d269at8ZLKysrr16+ne+AsX76cnsi2KvpQdM3I3oqdO3d6enqGhITw+XxPT88GtSe7XTXp9evXFRUVY8eOtbCwUFFRuXPnjmRWcXExIcTOzm7QoEGEEPrKjIT02JcmdwdT0DLljq4ZGRvO4XD09fW///57sVi8cuXK2NhYDQ0NQkhVVVVAQECPHj2+//77GzduVFZWnjp1ipFN6GqtojXJiT6rTUxMvHXrVm1tbWVlJSGkqqrqvXO1tLSWL1+ekJAwbty43377LTAw0NXVlb4oKS8VFRUikYheKSHk7du3hBDJlVORSCQUCiU7sqysLCcnh3598eLFUaNG0UOJ+/fvX1dXFxsbS1FUaGhofHx8WVlZWVlZfX19g60jhBQUFJSUlEg3jrq6uq+++srAwIAe5hYZGSkSia5cuZKUlFRaWpqWlpaVlUUvuWTJEm1t7aysLHt7e3qKjCoSCoWEkDdv3sixuuRIumZUVVWb24qLFy++efNm6tSpenp6e/fuvX79+t69exuXIP09bFJ1dbXkbuW+ffsWLlz4+eef9+zZc/Xq1VlZWdevX6dnRUZGOjs7jxs3zszMbMqUKdHR0YGBgYQQgUDw5MkTiqJyc3MlV04a7w45QstkinTN0PcRm9zwvXv3xsXF5ebm8vn8srKy58+f0w1DLBZv27aNrlJra2tjY2M5XtZDq5BFundEC3u/ZGZm6unpaWlpHT9+PCcnh76BZmZmFhMT02Bufn6+ZG5kZCRFUVVVVR4eHvSqNTU1Jd08ZGtJ75ekpKRVq1bRF21dXV0vX74cGRlpYGBACPHx8cnMzDx9+vSAAQMIIRs2bCgsLFy0aJGamtrMmTOPHDmydOlSW1vbrKwsuqiqqipzc3NCiJ6eXlBQ0NKlS7W0tNavX19SUiK9dQkJCfT9SUJI3759R48ebWlpOWzYMA0NDQ6Hk5eXR1EUPRZBT0/v6NGjO3fuVFJSWr9+vXTYvr6++/fvl57SZBWdOXPG1NSUEOLs7JyUlNSSSiOK7RMlXTNU81uhoaGxaNEiup/Pjz/+SAjhcrmbNm2qrq5uUIIMd+7csbKyMjU13b59u5eX17p166qrq+lZ9fX169at09XVpR/26OLiUlNTQ896/fq1nZ0dIcTExGTmzJlubm7q6uqrVq2i9xSt8e6QDS2zw7XM5jY8KipKcsZA++STTy5fvlxeXq6qqjp06NAffvjBz8/Py8tLIBDIXiNahVxaRWuSE0VRAoFAcjj40LkURRUXFycmJspeRlrLu2a23KJFi/T19evq6h4/fpyZmdlgrlgsfvr0aVVVFUVRqamp0qG+d+saKCoqkrTmt2/fNpjr6OjYeCL14VXUmIIPAVRTNfOhW/FBdVtQUPDgwYPKysrGs6qrqx89eiRJS9LS09NTUlLEYnFmZmZZWVmDuc3tjuagZbYC4y2zyQ0PDw8/depUSUlJSkrKo0ePrl+/fujQIXqwdlVVVXl5eWJiYkVFRUtWh1bRCo1bRSt76/H5fBnDfWXPJYTo6OgorMOebMrKyiNGjGg8ncPhDB06lH5N9/iSeO/WNSB9EUBLS0t6Vnx8fL9+/RpMpLGnilqucc186FZ8UN326tWruW4Lqqqq0h3KpUkeGEH/WypNxu5QPLRMOWpQM403PD09feXKla9evaIfY03PMjMzu3//PofDoa93tf2Jjm3XpVqFQh98xyrV1dWS+2QKdv/+/XXr1g0ZMuT58+fR0dGMxAASbNsdaJmKl5ubW1RU5O7uvmLFCvq/lpSUlOPHj+/atYvp0P6rC7aKrpichEJhQEBAXFxcRUXF1q1bly1b1rdvXwXHkJaWpqqqeuDAge7duyt41eyXm5tLP+WoSZ6enu7u7vJdI0t2B1omU+zt7S9duhQdHb18+fKsrCxjY+PJkycfO3ZMU1OT6dC6bqvoismJz+d7e3t7e3szFYClpWVhYSFTa2e/vn37Xrhwobm5cn/OPXt2B1omgyZNmkQ/QISiKFYNK+6yraIrJidgOQ6Hw7afhoOug1WZqStT6BMiAAAAWgLJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWKeJB786OzsrPg7ZEhISCCsDkwuKorKysj755BO5P29bjjpr5bdR526ZHYJ8K18kEuXk5AwYMKAtj39Fq5ALrp+fn+RNeXl5WVkZc8E0q2/fvor/CROFKS8vv3fvXnp6en19fffu3blcrlyKNTMzmzJlSr9+/doeHjtbBRtIt8zk5OSSkhLpHxKFJrGzZQoEgpSUlHv37r1+/VpfX79bt26tLqpzH6/aSeNWwaF/vB2YVV5e/tNPP+3bt6+2tvbLL7/csGEDGneH4+LiQggJCwtjOhD4MIWFhT/99NOBAwcoilqxYsWGDRskv9QODMI9J1bQ1NTcuHHjy5cvd+3adfbsWUNDQw8PjxcvXjAdF0BnlpWV5ePjY2BgcPTo0TVr1uTk5Pj7+yMzsQSSE4uoqan5+PhkZGQEBAQ8ePBgyJAhM2bMePDgAdNxAXQ2T58+9fDwMDExiYqK8vf3z87O9vPz61K/TM9+SE6so6ys7OHh8ddff0VGRhYWFlpaWtra2kZFRTEdF0BncPv27RkzZowYMSIpKenEiROpqak+Pj5tucME7QTJiaWUlJRmzJhx//79W7duaWlpzZw5k05RuEcI0Dq3b9+eMGGCnZ1daWnpuXPnnjx54uHhweYusl0ckhPb0Tnp9u3bWlpa//jHP4YPHx4cHCwSiZiOC6BjEIvFUVFRo0ePtrOzq62tvXLlCn3y1JbO4qAASE4dg42NTVRUVFJS0ogRI7788ksTE5ODBw/W1NQwHRcAewkEguDgYDMzs1mzZvXq1evBgwe3b992cHBgOi5oESSnjmTo0KHBwcGpqakzZszYvHmzgYGBn5/fu3fvmI4LgF0qKysPHjxoaGi4ZMkSS0vL58+fR0VFjRo1ium44AMgOXU8AwYMOHjwYHZ29ooVKw4ePNi/f38fH5/Xr18zHRcA80pKSvz8/Pr37/+vf/3riy++yMjICA4ONjU1ZTou+GBITh1Vz549/fz8Xr58uWPHjrCwMCMjIx8fn9zcXKbjAmBGYWGhn5+foaHhkSNHVq9e/fLly4MHD2Iwe8eF5NSxaWpq+vj4pKen79q1KyIigh69m5yczHRcAIqTmZkpGUu7du3ajIwMPz+/Hj16MB0XtAmSU2cgGb17/Pjxhw8fmpub093QmY4LoH1JxtJGR0dLxtJqamoyHRfIAZJT58Hn8z08PP7888/IyMiioqIxY8Zg9C50VpKxtE+fPj158iTG0nY+SE6dDT169969e5LRu59++ml4eDhG70LncOXKFRsbG8lY2sePH3t4eMjrWf7AHkhOnRZ92pSYmGhoaDh37txhw4Zh9C50XPRY2lGjRk2cOPGjjz66c+cOxtJ2bkhOndzIkSPDwsKePn1qYWHx5ZdfGhsbY/QudCz0WNrBgwfPmjWrd+/eDx8+jI2N/eyzz5iOC9oXklOXYG5uHhwcnJaWNnPmTIzehY6CHks7cODAJUuWjBkzJjk5OSoq6tNPP2U6LlAEJKcuxMDAQHr07ieffOLj41NQUMB0XAANSY+lnTNnTmZmZnBwsImJCdNxgeIgOXU59OjdnJycb775Jjw8fMCAAcuWLcvJyWE6LgBCCHn9+vWmTZsMDAykx9L26dOH6bhA0ZCcuigNDQ0fH5+srKwffvjh4sWLRkZGGL0LzMrIyPDx8RkwYEBQUNDWrVtfvnyJsbRdGZJTl6aiorJ06dL09HTp0bv37t1jOi7oWpKSkjw8PExNTS9cuODv75+VlbVx48aPPvqI6biASUhO8D+jd4uLi62srDB6FxSD7g5uYWFBj6VNSUnBWFqgITnBf9GjdxMSEiSjd0eOHBkcHFxfX890aNDZUBQVFRX12WefYSwtNAfJCRqiT5sePXpkbm6+aNEi/PYuyJFkLO0//vEPbW3t+Ph4jKWFJiE5QdMsLCyCg4OTkpJGjhyJ0bvQdtJjafX19R8+fBgVFWVtbc10XMBSSE4gy5AhQ4KDg9PT0+nRu/3798foXfhQDcbSvnjxIioqauTIkUzHBayG5ATv179//4MHD758+dLb2xujd6Hl6LG0n3zyifRYWmNjY6bjgg4AyQlaSldXt8HoXQ8Pj/T0dKbjAjbKycnx8fHp37//kSNHvvrqq5ycHIylhQ+C5AQfRjJ699ixY3fv3jUzM/Pw8Hj+/DnTcQFb0GNpTUxMIiMjd+/eTY+l1dLSYjou6GCQnKA1VFRU6Jx0/PjxxMREevRuQkIC03EBk548eSIZS7tnzx76BwAxlhZaB8kJWk8yevfcuXMlJSXW1tYYvds10d3BR44c+ezZM8lYWhUVFabjgg4MyQnaisPhzJgx4+7duxi929XQY2mtra0xlhbkDskJ5KbB6N1BgwYdO3ZMKBQyHRfIn1gsDg8PHzp06D/+8Q8dHZ27d+/SJ09MxwWdB5ITyBk9evfFixeff/75qlWr6NG71dXVTMcF8lFXVxccHDxo0KB58+aZmZk9e/YsKirKysqK6bigs+FQFMV0DNBpvXz5cv/+/cePH1dTU/P29vbx8elMvbZCQkJOnDghFovptykpKYQQU1NT+q2SktKXX37p6urKWHzyVlFRcfLkyb17975588bFxWXr1q0YsQTtB8kJ2l1xcfGRI0d++OEHkUjk5eW1ceNGfX19poOSg6SkpBEjRshY4MmTJ8OHD1dYPO2ns+5BYDMkJ1CQTvl/96BBg+gTpsaMjIzS0tIUHI/cde5zX2Az3HMCBaFH72ZmZh47diwhIWHQoEEuLi5//fWXjI/s3r27sLBQYRG2gru7O5/Pbzydz+d7eXkpPp6WKyws3L17t4wF0tPTly1bZmxsfO7cud27d2dnZ2MsLSgUBaBw9fX1YWFhQ4YM4XA406dPv3v3buNlMjMzuVyukZFRQUGB4iNsoYyMjOZ+6yEtLY3p6JpVUFBgZGTE5XIzMzMbz3306JG7uztd+T///LNAIFB8hAA4cwIGKCkpOTs7P3v2rMHoXUrqIvO+ffuUlJRevnxpa2ubn5/PYLQyDBw40MLCokF+4nA4n376qZGREVNRyZafn29ra/vy5UslJaV9+/ZJz5KMpf3zzz9Pnjz54sWLpUuXNnlqCNDumM6OANStW7emT5/O4XBGjBgRFBQkEolev36trKxMN1E+n29gYJCbm8t0mE07cOAAj8eT/k7xeLyDBw8yHVfTCgoKjI2NJfmGz+fn5+fX19efP3+e7g5uY2Nz/vx5psMEoNAhAtjiwYMH/v7+kZGRpqamxsbGMTExkgG8fD6/Z8+et2/fNjAwYDTGJhQVFfXu3VvSoZwQoqSk9OrVq169ejEYVZNyc3Pt7Ozy8/OlK9bR0TE1NTUlJWXWrFmbNm0aPXo0s0EC0JCcgF2Sk5N37tx55swZgUAgPZ3H49H5acCAAUzF1hx7e/tbt27Rj2vicrljx469du0a00E1lJOTQ2cmkUgkPZ3P5zs5OW3dunXw4MFMxQbQGO45AbsMHjx4yJAh0iciNJFIVFxcbGNjk5mZyUhgMri7u8t4ywY5OTm2trYFBQUNMhMhhKKooUOHIjMB2+DMCdiltra2b9++b968aXIun8/X0dG5deuWoaGhggOToby8XEdHh75Wxufzi4qKPv74Y6aD+hvdqaSwsLC55xxqaWnl5eXhty2AVXDmBOwSEBBQWlra3FyhUFhSUmJra8uqX+DV1NScOnUqj8fj8XiOjo6sykzZ2dmyMxMhpLy8/OTJk4qMCuC9kJyARYRCId25WVlZWUmp6cYpFArfvHlja2vLqucvuLm51dfX19fXs+phemlpaVZWVjIyk5KSkrKyMkVRe/fuxfPjgVVwWa+ruHv3bm5uLtNRvEdNTc3jx49LS0vfvHlTWlpaXFz85s2bsrIyyU9DcblcLpcrFotFIpGmpub27dtZ8pA3oVD45ZdfUhR14sQJSSd4ZuXn52/btq28vJzD4dCVJrmTx+VyNTU1tbS09PT0evTooa2traWlZWFhoaqqymzM79WvXz9ra2umowBFQHLqKpydnc+cOcN0FABt4uTkFB4eznQUoAi89y8CnUWn/GKLxeLmLgAq2MWLFzkczuTJk6UnhoWFzZ07V/H/ArKnWuTI2dmZ6RBAcZCcoGNjzyF4woQJTIfwN/ZUC0DrIDkByEeDhxgBQFvg3ysAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdPKoS4IMVFBRcvXo1NzfXxcXF0NBQjiW/evUqPDw8LS1NW1t7/Pjx1tbWOTk59fX1ZmZmclwLAPvhzAnYpa6ujuUlHzt2zMnJydjYeNOmTfLNTN9++62VlVVdXd3SpUt9fHyEQuHUqVPt7OxSU1PluJb3Yv8ugK4AyQnYZcuWLZJfE2dbyRRFzZo1KzQ09OrVq2PGjOFwOPKKjRBy4MCBbdu2nT59euPGjcOHD9fW1p48efLFixcHDhxYWloqxxW9F5t3AXQdSE7AIs+ePTt69ChrS/7Pf/6TkJAQEhLSrVs3uUQlcerUqbVr137zzTc2NjbS07t163bo0KG3b9/Kd3UysHwXQNeBe07QUG1tbXh4+F9//WVtbT1p0iRVVVV6el1dXVxcXFxcnL6+/pQpUyRXtNLT0wMDA3fs2JGRkREWFtazZ8+FCxfy+XzZpaWlpf3xxx/v3r2ztLScOnUqIeTOnTsLFiyoqqo6ffo0n8+nf5O7vLw8NDQ0OTl54MCBCxcuVFdXf+8a21KyDI8ePdqyZcuuXbt69eol1/omhJCdO3cSQhwdHRvPGj16tORX3rv4LoCuhYKuwcnJycnJ6b2LZWdn29nZBQQE5ObmOjg4DBw4sKamhqKompqa8ePHnz59urS09NChQxoaGr///jtFUYGBgXp6eoSQ8+fPf/HFF9OmTSOEbN26VXZpq1evtrOzKykpuXz5MofD8ff3pyjq1q1brq6uhJDo6OhLly5RFJWamjpjxoxLly49efLE3Nzc0NCwtLRU9hrbUrLsmnFzc+PxeOHh4Z6enuPGjVu3bt27d+/eW5+hoaHv/ZalpaURQng8Xl1dnYzFsAta2Iahc0By6ipa+MWeOHHi0qVL6dfR0dEcDufs2bMURS1YsMDLy0u6NFVV1dzcXIqifH19CSHnzp2jZ9nb25uYmMgurXv37jt37qSnm5mZWVlZ0a+3b99OCBGLxZKPR0RE0K9jYmIkR0AZa2xjyTKYmJjo6+uHhoZWVFScP39eVVV1+PDhQqFQ9qdakpxiY2MJIYMGDZK9GHYBklOXgst68Lf09PTY2Ng//h+dhHkAACAASURBVPiDfjtt2rTXr1/37Nmzuro6PDz8u+++kyy5YsWKM2fO/PLLL1u3blVTUyNSl6TMzc3v3bsnozRCyIULFwYPHkwIuX//PkVRNTU1jYMpKCiIjY0dMWIEXVplZeWoUaOqq6sJIc2tse0lN+fdu3dpaWmurq4uLi6EkBkzZnh7e3/33XenTp1yd3dvSd3KwOO9/2uIXQBdDZIT/C05OZkQIn3pnz6QxcfHC4VC6WOosbExIYTu4qyk9D/datTU1EQikYzSCCE2NjYRERFnz56dPHmygYHBq1evGgdDX+zy9fXV0dFpMKu5Nba95ObQV5ykl7e1tf3uu++ePHnS9uREV2ZaWlpNTY3khlAD2AXQ1aC3HvyNPjJeunRJemJxcXF9fT0hJD4+XjKRPqaYmJi0ojRCiK+v78mTJwMCAtzc3FRUVJr8uLKyMiHk0aNH0hMrKipkb0I7lWxgYKChoZGfny+ZYm1tTf7vBKKN+vTpM2TIkPr6+j///LPJBSiKwi6ArgbJCf5mZmampKQUFRVFHwoJIRkZGQ8fPrSwsFBRUblz545kSfoAZ2dn14rSEhMT9+3bt3LlSkmHbOr/eqPR6OVNTU25XO62bdsEAoFkpSEhITLW2H4lczicsWPHPn78WDIlNzeXEDJ27FgZn2o5+n6Mr6+vJCSJ4uLiX375BbsAuhokJ/ibvr6+h4fH06dPnZ2dr127duTIka1bt06ZMqVnz56rV6/Oysq6fv06vWRkZKSzs/O4ceMIIfQoHMmtBZFIJBQK6+rqmivto48+oksQiURXrlxJSkoqLS1NS0vLysrS1dUlhCQmJt66dUtVVXX58uUJCQnjxo377bffAgMDXV1d58+fL2ONbS9ZhkOHDr1+/VpyAL1w4cLEiRMnTJggl5qfM2fOzp074+LilixZUlVVJZmek5Pj7+/v7u6OXQBdDlM9MUDBWtjTqaysbPbs2XTbMDAwePDgAT29vr5+3bp1urq6Gzdu9PT0dHFxoXskR0ZGGhgYEEJ8fHwyMzNPnz49YMAAQsiGDRsKCwubK83d3V1JSUlPT+/o0aM7d+5UUlJav349RVGZmZl6enpaWlrHjx+nKKqqqsrDw4P+uKamJt25S/Ya21Lye0VFRQ0ePHjPnj0+Pj6urq5VVVXv/UhLeutJnDlzxtzcXFVVddy4cStXrlyyZImfn5+kfzl2AXrrdSkc6n9PuqGzooc9hoeHt2Th/Pz84uJiMzMzybBKWk1NzYsXLwYPHvxBj0hosrTi4uKPP/6YnlJaWqqlpUVPFwqFIpFIul9ASUlJTk7O4MGDm+ss0ED7lUwIEQgE6enp/fv3b+HdprCwsLlz537Qt6yqqurZs2dGRkZNdhboyrvgg9owdHRITl0FvtiMaEVyguagDXcpuOcEAACsg3FOAIQQkpub6+Xl1dxcT0/Pto9nAoCWQ3ICIISQvn37Xrhwobm5LXmIAwDIEb5yAIQQwuFwmhsxCgCKh3tOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOkhOAADAOnjwaxeSl5cXFhbGdBRdy927dwkhqHa5yMvL69u3L9NRgIIgOXUhCQkJc+fOZTqKrgjVLi9OTk5MhwAKgp9pB5APFxcXgpMkADnBPScAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdJCcAAGAdHtMBAHRU9+7dS0pKkrzNzMwkhBw7dkwyZdiwYVZWVgxEBtDxITkBtFJRUdGyZcu4XK6SkhIhhKIoQsiqVasIIWKxuL6+/vz58wyHCNBhcehvFAB8KKFQqKOjU15e3uRcDQ2NkpISZWVlBUcF0DngnhNAK/H5/Hnz5jWZfvh8/vz585GZAFoNyQmg9ebPny8QCBpPFwqFCxYsUHw8AJ0GLusBtJ5YLNbX1y8sLGwwXVdX9/Xr1/S9KABoBXx5AFpPSUnJzc2tweU7ZWVlT09PZCaAtsD3B6BNGl/ZEwgE8+fPZyoegM4Bl/UA2srIyCgjI0Pytn///tnZ2cyFA9AZ4MwJoK3c3Nz4fD79WllZ2cvLi9l4ADoBnDkBtFV6erqxsbHkbUpKiomJCYPxAHQCOHMCaCsjI6Nhw4ZxOBwOhzNs2DBkJoC2Q3ICkAMPDw8ul8vlcj08PJiOBaAzwGU9ADnIz8/v168fRVE5OTl9+/ZlOhyADg/JCdpRWFjY3LlzmY4C2kVoaKiLiwvTUUCnhaeSQ7sLDQ1lOgRFuHLlCofDcXBwkLHM999/TwhZu3atooJqL/ifA9obkhO0uy7y/zWdlrS1tWUsEx4eTjpFhSA5QXtDcgKQD9lpCQA+CHrrAQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6+Cp5MAuL1++PHbsWEhISHZ2NtOx/Nfly5eFQuG0adOkJ1ZVVQUFBWVmZnbv3t3FxcXU1FReq7tz505WVpbkLY/H6969e48ePYYOHfrRRx/Jay0ALIczJ2CXzMzMGzdu5OXlMR0IIYRcuXJl8uTJkydPfvjwofT04uLi4cOHa2pqfvPNN5MmTZo9e3ZkZKS8VvrZZ5/p6up6eHisXr06LS2ttrb28ePH/v7+2trajo6OL168kNeKANgMZ07ALvb29jY2Nvfu3WM6EEIIsbW1NTIyGjBgQIPpe/bsMTY2dnNzI4SMGTPG3d19w4YNs2bNkstKORzO5MmTtbS0dHV1t2/fLpl+9epVNzc3CwuLGzdujBkzRi7rAmAtnDkB6/D5fKZD+K9u3br16dOn8fS8vLyCggKKoui3ampq3bp1k++qlZWVG0xxcHA4ceJEbW3tnDlz6urq5Ls6ALbBmROwglAojIiIePz48fjx48VisfSs8vLy0NDQ5OTkgQMHLly4UF1dnRCSnp4eGBi4Y8eOjIyMsLCwnj17Lly4UJLVbt++HRMT069fPyUlpaVLl8oo5724XG7jiZ9//nloaOi///3vb775RiQShYSErFmzpk3b3zKOjo4ODg5Xr14NDw+nz9uYrRyAdkQBtJvQ0NCWtLF37945ODj4+fm9efMmKChIWVmZy+XSs1JTU2fMmHHp0qUnT56Ym5sbGhqWlpYGBgbq6ekRQs6fP//FF1/QXRW2bt1Kf8TX1zckJKSqqurUqVPq6uoyymnJJtCZcvv27dITBQKBvb09IcTT03PhwoXHjh1rYYU4OTk5OTm1ZMlevXqZmpo2nv6vf/2LELJo0SKK0cohhISGhrZwqwFaAckJ2lELk5O3t/esWbMkb6dPny5JThMnToyIiKBfx8TESI6zvr6+hJBz587Rs+zt7U1MTCiKEggE2traKSkp9HQfHx/Z5bxXk8mJoqjq6mo6P40cObKwsLAlRVHySE7BwcGEkIkTJ1KMVg6SE7Q3XNYDhhUVFQUEBBw8eFAyZdiwYfQhsqCgIDY2dsSIEXT/iMrKylGjRlVXVxNC1NTUCCGOjo70R8zNzell+Hy+hobGhAkTfv7556lTp27ZskV2Oa12//59fX39DRs27Nu3b8yYMTdv3uzXr19bCmyhqqoqQoiuri6bKweg7ZCcgGFJSUlCobBXr16SKRwOh36RlpZGCPH19dXR0WnwKSWl/+nLo6amJhKJ6NeHDx92d3d3dHS0trYODAzU1dWVUU7rJCQkeHp6Pn36VFNTs3///qtWrfL29o6KipJL4bKlpKQQQszMzFhbOQBygd56wLCKigpCSEFBQeNZdI+1R48eNV5ehmnTpqWnp69ZsyYxMXHUqFHJycmtK0eGI0eOWFpaampqEkJWrlzp6+t76dKlkpKSVhfYQgKBIDo6msfjzZ49m7WVAyAXSE7AsEGDBhFC6Ot4EvSdHlNTUy6Xu23bNoFAQE8vLi4OCQmRUVpVVVVAQECPHj2+//77GzduVFZWnjp1qhXlSFAUJfkrUVhYKN2Lb9myZUKhsKioqCUFtsW+ffvo1GJmZsaGygFoP0hOwDAzM7MpU6ZER0cHBgYSQgQCwZMnTyiKys3N1dDQWL58eUJCwrhx43777bfAwEBXV9f58+cTQt6+fUsIqampoQsRiURCobCurk4sFm/btq22tpYQYm1tbWxsrKurq6Wl1Vw570Ufsuk7PRKLFy+Ojo6WrP3x48fDhw+ns6xcCIXC4uJi6Sl1dXVr167dvn375s2bd+7cSQiRsVEKqxyAdsRwhwzo1FrYW+/169d2dnaEEBMTk5kzZ7q5uamrq69atSovL6+qqsrDw4Nuq5qamnSnssjISAMDA0KIj49PZmbm6dOn6Yc4bNiwISMjQ1VVdejQoT/88IOfn5+Xl5dAIKAoqsly3is+Pt7b25sQYmRkdOTIEaFQSE8XiUSbN28ePnz44cOHt2zZMm/evMzMzJYU2JLeejdv3pwzZw4hhMfjWVhYzJ49e86cOdOnT1++fHliYqL0kgxWDkFvPWhnHOp/r1cAyFFYWNjcuXNb2MYyMjLq6+uNjY2zs7O1tbXpOzq0kpKSnJycwYMHq6qqyi6Eoqiampr6+vq0tDQTE5MGg0lbXk5L1NXVpaen9+zZU1dXt4UfcXZ2JoSEh4e3fe0SjFQOh8MJDQ11cXFpU+gAzUNvPWALQ0ND+kXjZ9np6Oi0sC8Zh8OhH909cuTIxnNbXk5LqKioDBkyRF6ltRo7KwegjXDPCQAAWAdnTtBF5ebmenl5NTfX09PT3d1dkfEAgDQkJ+ii+vbte+HChebm8nj4agAwCd9A6KI4HI6KigrTUQBA03DPCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAfJCQAAWAcPfoV2x+FwmA6BXVAhAO+Fn2mHdpSXlxcfH890FO8XExMTFBT09ddfDxs2jMEwnj59unv3bk9Pz6lTpzIYRgt99tlnffv2ZToK6LSQnKCrCwwM/PLLL3fv3r1x40amYyEHDhxYt27dTz/9tGzZMqZjAWASLutBl3bmzJnFixf/+9//ZkNmIoSsWbPm3bt33t7e6urqrq6uTIcDwBgkJ+i6IiMj58+fv3r16m3btjEdy9/8/Pxqa2s9PT35fL6LiwvT4QAwA8kJuqjY2Nh58+YtW7Zs//79TMfSkL+/v1AodHd3V1NTmzZtGtPhADAA95ygK7pz587kyZOdnJxOnjyppMTGARUURS1btuzXX3+9cOGCvb090+EAKBqSE3Q5CQkJkyZNmjRp0unTp3k89l48qK+vd3d3P3fu3MWLF+3s7JgOB0ChkJyga0lKSvr8889tbW3PnDnD5/OZDuc96uvr58+ff/ny5StXrowaNYrpcAAUB8kJupCUlJRx48YNHTo0KiqqW7duTIfTIgKBYPbs2ffv379x48aQIUOYDgdAQZCcoKtIT08fN26cgYHBpUuX1NXVmQ7nA9TU1Dg6Oj5//jwuLm7QoEFMhwOgCEhO0CXk5uaOHTtWT08vNjZWQ0OD6XA+WFVV1ZQpU7Kzs2/evDlgwACmwwFod0hO0PkVFhaOGzdORUXl+vXrPXr0YDqcViorK3NwcHj37t3Nmzf19fWZDgegfSE5QSdXXFw8fvx4kUgUFxfXq1cvpsNpk860LQCysXGEB4C8lJWVTZ06ta6u7vr1653gaK6rq3vt2jUOhzN58uS3b98yHQ5AO0Jygk6rqqpq+vTphYWFsbGxneY6GH3brLy83NHRsaKigulwANoLkhN0TjU1NdOnT09NTY2Nje1kPQj69esXGxubm5s7ZcqUyspKpsMBaBdITtAJCQQCJyenP//889q1a52y77WRkdG1a9cyMjJmz55dW1vLdDgA8ofkBJ1NfX29m5vbnTt3YmJiOvGoVVNT00uXLj169Gju3LlCoZDpcADkDMkJOhX6eXQXLlyIiorq9M/7GT58+IULF65fvz5//nyRSMR0OADyhOQEnQdFUStWrIiMjIyOju4iT0q1srKKiYm5ePHi4sWLxWIx0+EAyA17H8kM8KHWr18fFBR09uzZLvUbEzY2NhERETNmzFBXVz906BCHw2E6IgA5QHKCTmLTpk0HDx787bffuuCv802cOPH06dPOzs58Pv/7779nOhwAOeD6+fkxHQNAW/n5+X377bfBwcHz589nOhZmDBo0yMzMzNfXl6Ko8ePHMx0OQFshOUGHd+DAga+//vqnn37y8vJiOhYmmZmZffLJJ//85z+7detma2vLdDgAbYLLetCx/fjjj+vWrdu7d++yZcuYjoV5CxcurKysXL16NZ/PX7duHdPhALQekhN0YEFBQatXr961a9f69euZjoUtVq1aJRQK//nPf2poaCxZsoTpcABaCckJOqrff/998eLFW7Zs2bx5M9OxsMvatWvfvn27YsUKdXX1LnsTDjo6JCfokM6fPz9//nxvb+8dO3YwHQsbffPNNwKBwN3dncfjOTs7Mx0OwAdDcoKO58qVK3PnznV3dz9w4ADTsbCXv79/RUWFm5ubmpqao6Mj0+EAfBj82CB0MPHx8ZMnT549e3ZgYKCSEh5xIgtFUUuWLPntt9/++OMP9C+HjgXJCTqSe/fuTZw4ceLEiaGhoTwezvvfj34MblRU1MWLF9G/HDoQJCfoMJ4+fWpvbz969Ohz586pqKgwHU6HIRQK58yZc/PmzatXr3766adMhwPQIkhO0DGkpqaOGzduyJAh0dHR3bp1YzqcDkYgEMyaNevhw4c3btwwMzNjOhyA90Nygg4gIyNj7Nix/fv3v3z5srq6OtPhdEjV1dVTp05NSUmJi4szNTVlOhyA90ByArbLy8sbO3Zs9+7dr127pqWlxXQ4HVh5efmECRNev3598+ZNAwMDpsMBkAWdnYBFtm3bFhkZKT2lqKho4sSJ6urqV65cQWZqI01NzcuXL+vo6EycODE/P196VkxMDEaMAavgzAnYorKyUl9fv6am5vTp03PmzCGElJSUjB8/XiAQxMXF9e7dm+kAO4ni4uLx48fX19fHxcXp6ekRQn7//fd58+apqqrm5+fjqimwBM6cgC2OHz9eXV0tEolcXFx+/fXXsrKyqVOnVlRUxMbGIjPJka6u7uXLl4VC4eTJk9++ffvrr7+6uLjU19dXV1cfP36c6egA/gtnTsAKIpGof//+BQUFdIPkcDgWFhZFRUU3b94cMGAA09F1QllZWWPHju3du/fDhw8lB4FevXrl5OTw+XxmYwMgOHMClggLC5NkJkIIRVGPHj1yc3NDZmonAwYMWLRo0YMHD6T/PS0qKjpz5gyDUQFI4MwJWGHYsGHPnz+vr69vMH3jxo3+/v6MhNS57dmzZ9OmTQ0mKikpDRo06M8//+RwOIxEBSCBMydg3rVr1549e9Y4MxFC9u7di1/EkLuvv/66yVoVi8XPnz+/fv264kMCaADJCZi3Z8+e5h6UR1GUv7//vn37FBxSJ7Zv375vv/22uUsmPB5vz549Cg4JoDFc1gOG/fnnn8OGDWuyHSopKenp6e3YscPT0xN36eVFKBQGBQVt27atsLBQLBY3rnkOh/P06VNzc3NGwgOg4cwJGLZv377Gp01cLldPT2///v2ZmZmLFy9GZpIjPp+/ePHi7OzskydP9unTR0lJqcEdJh6Ph1NVYBzOnIBJ+fn5/fv3F4lEkik8Hk9TU9PX1/err75SVVVlMLauQCAQBAYGbt269c2bN9L3/LhcbnZ2dt++fRmMDbo4nDkBkw4ePCj5t53H43388cc7d+7My8vbuHEjMpMCKCsrL126NDc398cff9TR0ZGcwiopKR0+fJjZ2KCLw5kTMKaiokJfX7+yspLL5aqrq2/ZssXb21tNTY3puLqoqqqqH3/8cffu3ZWVlSKRSF1dPT8/X0NDg+m4oItCclKE/fv33717l+koWCctLS0pKYnP55uamhoZGbHnl23XrVtnbW3dxkLu3r27f/9+ucSjSCKRKD09PSUlRSgUDh8+3NjYmOmIOjBra+t169YxHUVHhct6inD37t2EhASmo2hfCQkJH7SNFEVlZ2ebmZk5OjoOGjSIPZnpzJkzubm5bS8nNze3Iz5tgcfjDRo0yNHR0czMLDs7W/H/vObl5XXEemssISEB/5K2BVuOCJ2elZVVeHg401G0I2dnZ0JIy7extLSUEMLCX8GQ78MROvROZ2QfhYWFzZ07t0PXG43+RkCrITkBM1iYlqAB7CNgEC7rAQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6yA5AQAA6+Cp5B1VaWmpjY3Nxo0bPT09FfnZtrhz505WVpbkLY/H6969e48ePYYOHfrRRx8pMhJ2evny5bFjx0JCQrKzs5mOhdXQkLoCnDl1VDweT1tbW11dXS6fraurk19ozfrss890dXU9PDxWr16dlpZWW1v7+PFjf39/bW1tR0fHFy9eKCAGNsvMzLxx40ZeXp5iVqeYnd4eq0ND6gpw5tRRaWho3Lp1S16f3bJly969e5WU2vefFQ6HM3nyZC0tLV1d3e3bt0umX7161c3NzcLC4saNG2PGjGnXGNjM3t7exsbm3r17ilmdYnZ6e6wODakrwJkTkGfPnh09elRhq1NWVm4wxcHB4cSJE7W1tXPmzFHwv/Nsw+fzFbMiBe/09lgdGlLnhjMnFsnIyIiKilqzZs3t27djYmJMTEzc3d3p/zQzMjICAwP9/PxiYmKeP3++du3a+vr68PBwPT29SZMm0R+vq6uLi4uLi4vT19efMmWKoaGhpFgZn71z586CBQuqqqpOnz7N5/OVlZWrq6sJIRwOZ968eYSQv/766+nTp4SQSZMmaWtrt9O2Ozo6Ojg4XL16NTw83M3NjRBSXl4eGhqanJw8cODAhQsX0hch09PTAwMDd+zYkZGRERYW1rNnz4ULF0oO6HS99evXT0lJaenSpfTEJsthFaFQGBER8fjx4/Hjx4vFYulZTQYvo50QQtLS0v744493795ZWlpOnTqVntigDVhaWnp6ekp2urOzc3Z29i+//LJp06bCwsLAwEA9Pb358+d//PHHGRkZ4eHhysrKXl5e0j+M+0F7p0Eba9ffL+/KDamzoaD9OTk5OTk5yV7m0KFD6urqvXv3DgkJGTp0qKqqKiFkzpw5FEUFBQX16tWLEBIYGGhhYUEIOXHixKxZswghe/bsoT9eU1Mzfvz406dPl5aWHjp0SEND4/fff2/JZ2/duuXq6koIiY6OvnTpUnJycu/evQkhaWlpdMn19fUODg6HDx8Wi8Vt3EZar169TE1NG0//17/+RQhZtGgRRVGpqakzZsy4dOnSkydPzM3NDQ0NS0tL6YMmIeT8+fNffPHFtGnTCCFbt26lP+7r6xsSElJVVXXq1Cl1dXV6YpPlyA6PEBIaGtqSDZEtNDS0Jd+vd+/eOTg4+Pn5vXnzJigoSFlZmcvlygheRjuhKGr16tV2dnYlJSWXL1/mcDj+/v5UU21gz5490jv91KlTffv2JYSEh4e7u7u7urpyudwvvvgiLi5u3rx5rq6uPB7P0dFREvOH7p0GbUwu9UaxviG1/BsBTUJyUoQWNtO5c+eqqan9+uuvFEXl5+dbW1sTQugv85YtW+iDC0VRL168EIvFr169kk5OCxYs8PLykl6jqqpqbm5uSz5LX7WX5J6QkBDJeimKEggEo0aNEolEctlGqvljSnBwMCFk4sSJFEVNnDgxIiKCnh4TEyM5dvj6+hJCzp07R8+yt7c3MTGhg9TW1k5JSaGn+/j40C+aK0cGBScnb2/vWbNmSd5Onz5dkpyaC15GO+nevfvOnTvpj5iZmVlZWdGvG7eBBjudfiupWG9vb0JIcHAw/ZY+3JeVlckOrLm9QzVqY22vN4r1DQnJqY1wWY9F1NTUNDU16f8xe/fu/e23344fPz42NnbSpEn0P8jz588nhJiamhJCpK8qVFdXh4eHf/fdd5IpK1asOHPmzC+//LJ161bZn21s7ty527Zt+89//kNfMIyIiJg1axaXy22HLf4fVVVVhBBdXd2CgoLY2NgRI0bQXQMqKytHjRpFX2xUU1MjhDg6OtIfMTc3p5fh8/kaGhoTJkz4+eefp06dSh+LZZTDEkVFRQEBAQcPHpRMGTZsGH3sk10JzbWTCxcuDB48mBBy//59iqJqamroYhu3gQboJjF27Fj67fDhwwkhtra29NtBgwYRQl69eqWpqdmKvaNgXbAhdUpITuzC4XAkr0ePHk0Iyc3NbTCdJt3rKT4+XigU8nh/701jY2NCSGpq6ns/2xiXy924ceOSJUvu379vaWl54sSJoKCg1m3OB0lJSSGEmJmZpaWlEUJ8fX11dHQaLNMgcjU1NZFIRL8+fPiwu7u7o6OjtbV1YGCgrq6ujHJYIikpSSgU0tfcaJKdJTv45tqJjY1NRETE2bNnJ0+ebGBgQJ8ik6baQAN0xUoWU1FRkZ5Ldz0QCoWyA5OxdxSpCzakTgm99dhLWVlZRUXlk08+ee+S9fX1hJD4+HjJFPpbZGJi0rpVe3h49OnTZ9euXSkpKR9//LH00bOdCASC6OhoHo83e/Zs+lD4RJygewAABXFJREFU6NEj6QUqKipklzBt2rT09PQ1a9YkJiaOGjUqOTm5deUoEh1MQUFB41ktD166nfj6+p48eTIgIMDNza1BgpEXltdq12xInRKSE7vU1tZKXsfHx9fV1VlaWr73UxYWFioqKnfu3JFMKS4uJoTY2dm1fNV0hqMpKyuvX7+e7hK2fPnylhfSavv27aOPCGZmZqamplwud9u2bQKBgJ5bXFxM3wlrTlVVVUBAQI8ePb7//vsbN25UVlaeOnWqFeUoGH25jL6OJ0F32JMdfJPtJDExcd++fStXruzWrRs9i6Io2QFI7/QWakuttmJ1H6prNqROCcmJXcrKynJycujXFy9eHDVq1Jw5c8j/XVF58+aNZMnKykryf5fXe/bsuXr16qysrOvXr9NzIyMjnZ2dx40b997PEkJ0dXUJIYmJibdu3ZIc9ZYsWaKtrZ2VlWVvby/HDRQKhXTilKirq1u7du327ds3b968c+dOQoiWltby5csTEhLGjRv322+/BQYGurq60vdL3r59SwiR3EoRiURCobCurk4sFm/bto0O3tra2tjYWFdXV0Y5LGFmZjZlypTo6OjAwEBCiEAgePLkCUVRubm5GhoaMoJvsp3QT+6JjIwUiURXrlxJSkoqLS1NS0vLyspq3AYa7PQGTYJeXrqeJXNbsXcar67tVYeG1Pkx3CGja2hhv51FixapqanNnDnzyJEjS5cutbW1zcrKoijqzJkz9E1sZ2fnpKQkiqJycnJWrFhBCDEzM4uJiaEoqr6+ft26dbq6uvQT81xcXGpqalr42czMTD09PS0trePHj0vH4+vru3//fjlu482bN+lcy+PxLCwsZs+ePWfOnOnTpy9fvjwxMVF6yaqqKg8PD7qJampq0h2lIiMjDQwMCCE+Pj6ZmZmnT58eMGAAIWTDhg0ZGRmqqqpDhw794Ycf/Pz8vLy8BAJBc+XIRhTbW+/169f0Ca6JicnMmTPd3NzU1dVXrVqVl5fXXPDNtROKougBT3p6ekePHt25c6eSktL69esbtwHqf3f65cuXzc3NCSHLly9PSUn5448/6PtYbm5uT58+vXbtGt0zwsnJ6fnz59SH753CwsLm2ljr6q1DNCT01msjJCdFaHly0tfXr6ure/z4cWZmZitWVF1d/ejRIzotfRCBQFBdXd1goqOj49u3b1tYQnt8FYuLixMTExsH1phYLK6qqiovL09MTKyoqGh1OZTCkxMtPT09JSVFLBZnZmZKemzTGgcvu50UFRXRx1OKomTvviZ3est9UK22fHUfVG8txEhDQnJqI/TWYx1lZeURI0a07rOqqqr0KMsPxefzGzw4Jz4+vl+/ftIPBVA8HR2dFvaP4nA49EWtkSNHtqUcpkge50H/Cy+tueCbayf0BTSa7N3XeKd/kA+t1Tauri26TkPqTJCcWKS6ulpy0Z8p9+/fX7du3ZAhQ54/fx4dHc1sMNAkNrQTgPaGDhGsIBQKf/zxx7i4uIqKiq1btyrsRxOalJaWlpmZeeDAge7duzMYBjTGqnYC0K5w5sQKfD7f29ubfmYMsywtLQsLC5mOAprGnnYC0N5w5gQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyD5AQAAKyDp5IrSEJCgrOzM9NRtKOEhARCSOfexlZAhXwo+ndAOkG9JSQkWFlZMR1FB4bkpAjW1tZMh9DuOs330MnJqV+/fm0vp1+/fk5OTm0vp6vp27dv56g3KyurrvDFbz8ciqKYjgEAAOB/4J4TAACwDpITAACwDpITAACwDpITAACwzv8HbwHpuywcvAgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "keras.utils.plot_model(model, 'tickets_classifier.png')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f2fbb05c-353a-47db-b818-0eb222ecc620",
      "metadata": {
        "id": "f2fbb05c-353a-47db-b818-0eb222ecc620",
        "outputId": "5e25bf59-e493-45db-f658-a016b8e0ddab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_2\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " title (InputLayer)             [(None, 10000)]      0           []                               \n",
            "                                                                                                  \n",
            " text_body (InputLayer)         [(None, 10000)]      0           []                               \n",
            "                                                                                                  \n",
            " tags (InputLayer)              [(None, 100)]        0           []                               \n",
            "                                                                                                  \n",
            " concatenate_2 (Concatenate)    (None, 20100)        0           ['title[0][0]',                  \n",
            "                                                                  'text_body[0][0]',              \n",
            "                                                                  'tags[0][0]']                   \n",
            "                                                                                                  \n",
            " dense_8 (Dense)                (None, 64)           1286464     ['concatenate_2[0][0]']          \n",
            "                                                                                                  \n",
            " priority (Dense)               (None, 1)            65          ['dense_8[0][0]']                \n",
            "                                                                                                  \n",
            " department (Dense)             (None, 4)            260         ['dense_8[0][0]']                \n",
            "                                                                                                  \n",
            " difficulty (Dense)             (None, 3)            195         ['dense_8[0][0]']                \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 1,286,984\n",
            "Trainable params: 1,286,984\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "new_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bc0f1ba6-8644-4c1d-bce1-5fbb7fb886e5",
      "metadata": {
        "id": "bc0f1ba6-8644-4c1d-bce1-5fbb7fb886e5"
      },
      "source": [
        "Subclassing the Model class"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "68d20471-10f8-4a50-9e37-52a360e2b857",
      "metadata": {
        "id": "68d20471-10f8-4a50-9e37-52a360e2b857"
      },
      "outputs": [],
      "source": [
        "class CustomTicketModel(keras.Model):\n",
        "    def __init__(self, num_departments):\n",
        "        super().__init__()\n",
        "        self.concat_layer = keras.layers.Concatenate()\n",
        "        self.mixing_layer = keras.layers.Dense(units=64, activation='relu')\n",
        "        self.priority_layer = keras.layers.Dense(units=1, activation='sigmoid')\n",
        "        self.department_classifier = keras.layers.Dense(units=num_departments, activation='softmax')\n",
        "        \n",
        "    def call(self, inputs):\n",
        "        title = inputs['title']\n",
        "        text_body = inputs['text_body']\n",
        "        tags = inputs['tags']\n",
        "        \n",
        "        features = self.concat_layer([title, text_body, tags])\n",
        "        features = self.mixing_layer(features)\n",
        "        \n",
        "        priority = self.priority_layer(features)\n",
        "        department = self.department_classifier(features)\n",
        "        \n",
        "        return priority, department"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2065d3fb-9b13-4a5b-aaa8-6739be75ca8e",
      "metadata": {
        "id": "2065d3fb-9b13-4a5b-aaa8-6739be75ca8e"
      },
      "outputs": [],
      "source": [
        "model = CustomTicketModel(num_departments)\n",
        "\n",
        "priority, department = model({'title' : title_data, 'text_body': text_body_data, 'tags': tags_data})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "98036d96-aa95-4566-ad08-8988a6c6839f",
      "metadata": {
        "id": "98036d96-aa95-4566-ad08-8988a6c6839f",
        "outputId": "cc6f1853-a313-4366-afcf-38ed68a5fef1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " 1/40 [..............................] - ETA: 18s - loss: 4.2929 - output_1_loss: 0.1116 - output_2_loss: 4.1813 - output_1_mean_absolute_error: 0.2610 - output_2_accuracy: 0.4062"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:54:03.023016: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "40/40 [==============================] - 1s 13ms/step - loss: 28.8591 - output_1_loss: 0.3172 - output_2_loss: 28.5418 - output_1_mean_absolute_error: 0.4820 - output_2_accuracy: 0.2953\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x2deb2ce20>"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# compile and train are the same as Functional and Sequential Models\n",
        "model.compile(\n",
        "    optimizer = 'rmsprop',\n",
        "    loss = ['mean_squared_error', 'categorical_crossentropy'],\n",
        "    metrics = [['mean_absolute_error'], ['accuracy']],\n",
        ")\n",
        "\n",
        "model.fit(\n",
        "    {'title': title_data, 'text_body': text_body_data, 'tags': tags_data},\n",
        "    [priority_data, department_data],\n",
        "    epochs=1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8450f3d5-f2eb-4577-be18-cbc9ec7789dc",
      "metadata": {
        "id": "8450f3d5-f2eb-4577-be18-cbc9ec7789dc",
        "outputId": "86f4b286-cf0c-4c92-d523-06c618c00878"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "13/40 [========>.....................] - ETA: 0s - loss: 31.3204 - output_1_loss: 0.3391 - output_2_loss: 30.9813 - output_1_mean_absolute_error: 0.5012 - output_2_accuracy: 0.2452"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:54:03.910078: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "40/40 [==============================] - 1s 9ms/step - loss: 30.0347 - output_1_loss: 0.3281 - output_2_loss: 29.7065 - output_1_mean_absolute_error: 0.4936 - output_2_accuracy: 0.2508\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 19:54:04.489501: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        }
      ],
      "source": [
        "model.evaluate(\n",
        "    {'title': title_data, 'text_body': text_body_data, 'tags': tags_data},\n",
        "    [priority_data, department_data],\n",
        ")\n",
        "\n",
        "priority_pred, department_pred = model.predict( {'title': title_data, 'text_body': text_body_data, 'tags': tags_data},)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dedb4bb4-2099-4d2a-b468-6a2468f74524",
      "metadata": {
        "id": "dedb4bb4-2099-4d2a-b468-6a2468f74524",
        "outputId": "e8dafb16-65c9-4bcd-e112-f437086e0429"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[0.],\n",
              "       [0.],\n",
              "       [0.],\n",
              "       ...,\n",
              "       [0.],\n",
              "       [0.],\n",
              "       [0.]], dtype=float32)"
            ]
          },
          "execution_count": 35,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "priority_pred"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b2c088e1-beee-4554-af36-fe9158b44290",
      "metadata": {
        "id": "b2c088e1-beee-4554-af36-fe9158b44290"
      },
      "source": [
        "## Built-in trraining and evaluation loops\n",
        "\n",
        "**Early Stopping**: Stop the training when the metrics is no longer improving.\n",
        "**Model checkpoointing** Saving the current state of the model at different points during the training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3b564bba-f2d3-4e5a-8fb0-5b6c7c725ede",
      "metadata": {
        "id": "3b564bba-f2d3-4e5a-8fb0-5b6c7c725ede"
      },
      "outputs": [],
      "source": [
        "# custom callback function\n",
        "callbacks_list = [\n",
        "    keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2,),\n",
        "    keras.callbacks.ModelCheckpoint(filepath='checkpoint_path.keras', monitor='val_loss', save_best_only=True)\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "570ba745-a1fe-4c44-8546-79a683530f78",
      "metadata": {
        "id": "570ba745-a1fe-4c44-8546-79a683530f78"
      },
      "outputs": [],
      "source": [
        "# training mnist dataset\n",
        "from tensorflow.keras.datasets import mnist"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2754d538-ecce-4bf3-868b-43a6572b98ff",
      "metadata": {
        "id": "2754d538-ecce-4bf3-868b-43a6572b98ff"
      },
      "outputs": [],
      "source": [
        "# preapare dataset\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "# scale dataset\n",
        "x_train = x_train.reshape((60000, 28 * 28)).astype('float32') / 255\n",
        "x_test = x_test.reshape((10000, 28 * 28)).astype('float32') / 255\n",
        "\n",
        "# train and validation dataset split\n",
        "x_train, x_val = x_train[10000:], x_train[:10000]\n",
        "y_train, y_val = y_train[10000:], y_train[:10000]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "17aba2ef-0c5b-4b4e-9a2c-d23c9848ceb1",
      "metadata": {
        "id": "17aba2ef-0c5b-4b4e-9a2c-d23c9848ceb1"
      },
      "outputs": [],
      "source": [
        "# build a model\n",
        "# functional model (fully connected neural networks)\n",
        "\n",
        "def mnist_model():\n",
        "    inputs = keras.Input(shape=(28 * 28,))\n",
        "    features = keras.layers.Dense(units=64, activation='relu')(inputs)\n",
        "    features = keras.layers.Dense(units=64, activation='relu')(features)\n",
        "    outputs = keras.layers.Dense(units=10, activation='softmax')(features)\n",
        "    \n",
        "    model = keras.Model(inputs, outputs)\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "da9ee425-6f25-4a58-9b84-c382d78e6cfc",
      "metadata": {
        "id": "da9ee425-6f25-4a58-9b84-c382d78e6cfc",
        "outputId": "ff259b8c-6694-48d9-b7f3-d26182c09ebc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"model_23\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_20 (InputLayer)       [(None, 784)]             0         \n",
            "                                                                 \n",
            " dense_79 (Dense)            (None, 64)                50240     \n",
            "                                                                 \n",
            " dense_80 (Dense)            (None, 64)                4160      \n",
            "                                                                 \n",
            " dense_81 (Dense)            (None, 10)                650       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 55,050\n",
            "Trainable params: 55,050\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "m = mnist_model()\n",
        "m.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1751647f-a65b-4084-ac47-87e32ff79e62",
      "metadata": {
        "id": "1751647f-a65b-4084-ac47-87e32ff79e62",
        "outputId": "6c7fed0d-ab05-4274-aebe-a55868786f54"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            " 1/98 [..............................] - ETA: 29s - loss: 2.3506 - accuracy: 0.0820"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 20:50:43.704937: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "98/98 [==============================] - 1s 9ms/step - loss: 0.6365 - accuracy: 0.8362 - val_loss: 0.3161 - val_accuracy: 0.9092\n",
            "Epoch 2/20\n",
            " 1/98 [..............................] - ETA: 0s - loss: 0.2735 - accuracy: 0.9258"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 20:50:44.607943: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "98/98 [==============================] - 1s 7ms/step - loss: 0.2716 - accuracy: 0.9212 - val_loss: 0.2503 - val_accuracy: 0.9278\n",
            "Epoch 3/20\n",
            "98/98 [==============================] - 1s 7ms/step - loss: 0.2129 - accuracy: 0.9387 - val_loss: 0.2033 - val_accuracy: 0.9381\n",
            "Epoch 4/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.1798 - accuracy: 0.9473 - val_loss: 0.1708 - val_accuracy: 0.9498\n",
            "Epoch 5/20\n",
            "98/98 [==============================] - 1s 9ms/step - loss: 0.1568 - accuracy: 0.9545 - val_loss: 0.1543 - val_accuracy: 0.9530\n",
            "Epoch 6/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.1379 - accuracy: 0.9590 - val_loss: 0.1416 - val_accuracy: 0.9578\n",
            "Epoch 7/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.1221 - accuracy: 0.9637 - val_loss: 0.1387 - val_accuracy: 0.9581\n",
            "Epoch 8/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.1099 - accuracy: 0.9680 - val_loss: 0.1276 - val_accuracy: 0.9621\n",
            "Epoch 9/20\n",
            "98/98 [==============================] - 1s 7ms/step - loss: 0.0992 - accuracy: 0.9709 - val_loss: 0.1277 - val_accuracy: 0.9601\n",
            "Epoch 10/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.0901 - accuracy: 0.9735 - val_loss: 0.1093 - val_accuracy: 0.9670\n",
            "Epoch 11/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.0830 - accuracy: 0.9748 - val_loss: 0.1134 - val_accuracy: 0.9646\n",
            "Epoch 12/20\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.0753 - accuracy: 0.9773 - val_loss: 0.1092 - val_accuracy: 0.9663\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x367e87a60>"
            ]
          },
          "execution_count": 112,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# compile and train\n",
        "m.compile(\n",
        "    optimizer='rmsprop',\n",
        "    loss='sparse_categorical_crossentropy',\n",
        "    metrics= ['accuracy']\n",
        ")\n",
        "\n",
        "m.fit(x_train, y_train, callbacks=callbacks_list, epochs=20, batch_size=512, validation_data=(x_val, y_val))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d73b36a9-64a1-4588-9926-3d9e9d72b39b",
      "metadata": {
        "id": "d73b36a9-64a1-4588-9926-3d9e9d72b39b"
      },
      "outputs": [],
      "source": [
        "load_model = keras.models.load_model('checkpoint_path.keras')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4fff02b8-936b-4c27-ae84-88a071a2b5e9",
      "metadata": {
        "id": "4fff02b8-936b-4c27-ae84-88a071a2b5e9",
        "outputId": "d6fd0b91-c1eb-4465-8f35-9214840cd935"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " 23/313 [=>............................] - ETA: 1s - loss: 0.0933 - accuracy: 0.9660"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2022-03-22 20:52:19.137094: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "313/313 [==============================] - 2s 5ms/step - loss: 0.1095 - accuracy: 0.9664\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[0.10946598649024963, 0.9664000272750854]"
            ]
          },
          "execution_count": 116,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "load_model.evaluate(x_test, y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ca325504-c192-4992-b93e-7164c9eb587b",
      "metadata": {
        "id": "ca325504-c192-4992-b93e-7164c9eb587b"
      },
      "outputs": [],
      "source": [
        "prediction = x_test[0:10]\n",
        "test_pred = load_model.predict(prediction)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "99bd8aae-9f2c-430d-9e51-7bf52bb42cb6",
      "metadata": {
        "id": "99bd8aae-9f2c-430d-9e51-7bf52bb42cb6",
        "outputId": "dae31a24-2703-467b-ef9d-5025dd633109"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "7 7\n",
            "2 2\n",
            "1 1\n",
            "0 0\n",
            "4 4\n",
            "1 1\n",
            "4 4\n",
            "9 9\n",
            "6 5\n",
            "9 9\n"
          ]
        }
      ],
      "source": [
        "for i in range(len(test_pred)):\n",
        "    print(np.argmax(test_pred[i]), y_test[i])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "332d30ed-6a75-40ac-95f7-9ff5fc443ae6",
      "metadata": {
        "id": "332d30ed-6a75-40ac-95f7-9ff5fc443ae6",
        "outputId": "4afda0b2-d77b-447f-f440-ad2947e85ccb"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x367de4190>"
            ]
          },
          "execution_count": 141,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAN2UlEQVR4nO3dX6xV9ZnG8ecRwRDoBUokREBakIs6iTCiMY5OmJg2jiaKNwaNxsk00AtNRCeZwc5FMaNGx+nMZQ1YLGOKTRPtSOrEoqSOjlHiUVBBrTLmYCHIkSGxlhg6wDsXZ2FO8azfOu7/nvf7SU723uvda6/XHR7X2uu39/o5IgRg8juj3w0A6A3CDiRB2IEkCDuQBGEHkjizlxuzzal/oMsiwuMtb2vPbvtq27+1vdf2unZeC0B3udVxdttTJL0v6TuS9kt6TdJNEfFOYR327ECXdWPPfqmkvRHxYUT8UdLPJV3fxusB6KJ2wn6epN+Neby/WvYnbK+xPWR7qI1tAWhT10/QRcQGSRskDuOBfmpnz35A0vwxj+dVywAMoHbC/pqkC2x/0/Y0Saskbe1MWwA6reXD+Ig4bvsOSb+WNEXSpojY07HOAHRUy0NvLW2Mz+xA13XlSzUAvj4IO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiip1M2A2PNmjWrWF+wYEHXtr1v375i/a677irWd+/eXay///77xfqbb75ZrHcDe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJxdrTl2muvLdavu+662tqKFSuK6y5evLiVliakaRz8/PPPL9bPOuustrY/ZcqUttZvRVthtz0s6TNJJyQdj4jlnWgKQOd1Ys/+VxFxuAOvA6CL+MwOJNFu2EPSNtuv214z3hNsr7E9ZHuozW0BaEO7h/FXRMQB2+dKes72exHx4tgnRMQGSRskyXa0uT0ALWprzx4RB6rbEUm/lHRpJ5oC0Hkth932DNvfOHVf0ncllX/3B6BvHNHakbXtb2l0by6NfhzYEhH3N6zDYXyPLVq0qFi//fbbi/XVq1cX69OnTy/WbRfrWXVznD0ixn3TW/7MHhEfSrqo5Y4A9BRDb0AShB1IgrADSRB2IAnCDiTBT1wnuXnz5hXrd955Z4866b333nuvtrZnz54edjIY2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs/fA7Nmzi/Wmse6XX365WH/22Wdra8eOHSuu++mnnxbrR48eLdZnzJhRrG/btq221jTt8Y4dO4r1nTt3Fuuff/55ba3pv2syYs8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0m0fCnpljY2SS8l3TTW/NJLLxXrF11UvkjvDTfcUKxv3bq1WC9ZuHBhsT48PFysL1iwoFjfv39/be3kyZPFddGauktJs2cHkiDsQBKEHUiCsANJEHYgCcIOJEHYgST4PfsETZs2rba2ZcuW4rpN4+gPPPBAsf78888X6+1oGkdv8tFHH3WmEXRd457d9ibbI7Z3j1l2tu3nbH9Q3c7qbpsA2jWRw/ifSrr6tGXrJG2PiAskba8eAxhgjWGPiBclHTlt8fWSNlf3N0ta2dm2AHRaq5/Z50TEwer+x5Lm1D3R9hpJa1rcDoAOafsEXURE6QcuEbFB0gZp8v4QBvg6aHXo7ZDtuZJU3Y50riUA3dBq2LdKuq26f5ukpzvTDoBuafw9u+0nJK2QNFvSIUk/lPQfkn4haYGkfZJujIjTT+KN91oDexg/c+bMYv2ee+6pra1bVx6MOHz4cLG+ZMmSYr3p2u7AWHW/Z2/8zB4RN9WUrmqrIwA9xddlgSQIO5AEYQeSIOxAEoQdSIKfuFZWrlxZrJeG15p+5nnllVcW6wytoRfYswNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzVy6//PKW1925c2exXpq2GOgV9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kETjpaQ7urEBvpT0yEh5notzzjmntnbs2LHiug899FCx/vTT5cvu79q1q1gHxqq7lDR7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2StP7cPLkya5tu+m1H3nkkWL91Vdfra0tWLCguO7evXuL9T179hTrTS688MLa2iuvvFJcl+sAtKblcXbbm2yP2N49Ztl62wds76r+rulkswA6byKH8T+VdPU4y/8tIpZWf//Z2bYAdFpj2CPiRUlHetALgC5q5wTdHbbfqg7zZ9U9yfYa20O2h9rYFoA2tRr2H0taJGmppIOSflT3xIjYEBHLI2J5i9sC0AEthT0iDkXEiYg4KWmjpEs72xaATmsp7Lbnjnl4g6Tddc8FMBgax9ltPyFphaTZkg5J+mH1eKmkkDQs6fsRcbBxYwM8zv7www8X63fffXePOsnjk08+KdZfeOGFYn3VqlUd7GbyqBtnb5wkIiJuGmfxT9ruCEBP8XVZIAnCDiRB2IEkCDuQBGEHkuAnrpUpU6YU68uWLautbdmypbjumWeWBz3mz59frJ9xRs7/Jzf921y/fn2xft9993Wwm68PLiUNJEfYgSQIO5AEYQeSIOxAEoQdSIKwA0k0/uotixMnThTrQ0P1V9VasmRJW9u+6qqrivWpU6cW66Xx5ksuuaSVlgaCPe5w8RcuvvjiHnUyObBnB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGcfANu3b29r/aVLl9bWmsbZjx8/Xqw/9thjxfrGjRuL9bVr19bWbr755uK66Cz27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPsk8C2bdtqa/fff39x3aZr2q9evbpYX7x4cbG+YsWKYr0d+/fv79prT0aNe3bb823/xvY7tvfYvrNafrbt52x/UN3O6n67AFo1kcP445L+LiK+LekySbfb/rakdZK2R8QFkrZXjwEMqMawR8TBiHijuv+ZpHclnSfpekmbq6dtlrSySz0C6ICv9Jnd9kJJyyTtkDQnIg5WpY8lzalZZ42kNW30CKADJnw23vZMSU9KWhsRvx9bi9EZ+MadhS8iNkTE8ohY3lanANoyobDbnqrRoP8sIp6qFh+yPbeqz5U00p0WAXRC45TNHr2e72ZJRyJi7ZjlD0v634h40PY6SWdHxN83vNbATtn8dTZ9+vTa2qZNm4rr3njjjZ1uZ8KaLt/9zDPPFOu33HJLsX706NGv3NNkUDdl80Q+s/+FpFslvW17V7XsB5IelPQL29+TtE9S//7VAGjUGPaI+G9JdVfrL89uAGBg8HVZIAnCDiRB2IEkCDuQBGEHkmgcZ+/oxhhn77k5c8b9FvMXHn300WJ9+fLyFx/PPffcYn14eLi29vjjjxfXLU1FjXp14+zs2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZUXTrrbcW65dddlmxfu+999bWRka43kk3MM4OJEfYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzg5MMoyzA8kRdiAJwg4kQdiBJAg7kARhB5Ig7EASjWG3Pd/2b2y/Y3uP7Tur5ettH7C9q/q7pvvtAmhV45dqbM+VNDci3rD9DUmvS1qp0fnY/xAR/zLhjfGlGqDr6r5UM5H52Q9KOljd/8z2u5LO62x7ALrtK31mt71Q0jJJO6pFd9h+y/Ym27Nq1llje8j2UHutAmjHhL8bb3umpP+SdH9EPGV7jqTDkkLSP2n0UP9vG16Dw3igy+oO4ycUdttTJf1K0q8j4l/HqS+U9KuI+LOG1yHsQJe1/EMY25b0E0nvjg16deLulBsk7W63SQDdM5Gz8VdIeknS25JOVot/IOkmSUs1ehg/LOn71cm80muxZwe6rK3D+E4h7ED38Xt2IDnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEo0XnOyww5L2jXk8u1o2iAa1t0HtS6K3VnWyt/PrCj39PfuXNm4PRcTyvjVQMKi9DWpfEr21qle9cRgPJEHYgST6HfYNfd5+yaD2Nqh9SfTWqp701tfP7AB6p997dgA9QtiBJPoSdttX2/6t7b221/Wjhzq2h22/XU1D3df56ao59EZs7x6z7Gzbz9n+oLodd469PvU2ENN4F6YZ7+t71+/pz3v+md32FEnvS/qOpP2SXpN0U0S809NGatgelrQ8Ivr+BQzbfynpD5L+/dTUWrb/WdKRiHiw+h/lrIj4hwHpbb2+4jTeXeqtbprxv1Ef37tOTn/ein7s2S+VtDciPoyIP0r6uaTr+9DHwIuIFyUdOW3x9ZI2V/c3a/QfS8/V9DYQIuJgRLxR3f9M0qlpxvv63hX66ol+hP08Sb8b83i/Bmu+95C0zfbrttf0u5lxzBkzzdbHkub0s5lxNE7j3UunTTM+MO9dK9Oft4sTdF92RUT8uaS/lnR7dbg6kGL0M9ggjZ3+WNIijc4BeFDSj/rZTDXN+JOS1kbE78fW+vnejdNXT963foT9gKT5Yx7Pq5YNhIg4UN2OSPqlRj92DJJDp2bQrW5H+tzPFyLiUESciIiTkjaqj+9dNc34k5J+FhFPVYv7/t6N11ev3rd+hP01SRfY/qbtaZJWSdrahz6+xPaM6sSJbM+Q9F0N3lTUWyXdVt2/TdLTfezlTwzKNN5104yrz+9d36c/j4ie/0m6RqNn5P9H0j/2o4eavr4l6c3qb0+/e5P0hEYP6/5Po+c2vifpHEnbJX0g6XlJZw9Qb49rdGrvtzQarLl96u0KjR6ivyVpV/V3Tb/fu0JfPXnf+LoskAQn6IAkCDuQBGEHkiDsQBKEHUiCsANJEHYgif8HJSx00XCqlmAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "t = x_test[8].reshape(28, 28)\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.imshow(t, cmap='gray')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ca384709-2b6c-496f-97cc-0508e98154dd",
      "metadata": {
        "id": "ca384709-2b6c-496f-97cc-0508e98154dd"
      },
      "source": [
        "Tensorboard\n",
        "- Visually monitor metrics during training \n",
        "- Visualize your model architecture\n",
        "- Visualize histograms of activations and gradients\n",
        "- Explore embeddings in 3D"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d6bd1832-2320-420a-a7df-8138a21974d0",
      "metadata": {
        "id": "d6bd1832-2320-420a-a7df-8138a21974d0"
      },
      "outputs": [],
      "source": [
        "tensorboard = keras.callbacks.TensorBoard(\n",
        "    log_dir = 'full_path_to_your_log_dir',\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7d08b234-3bbf-48c8-b760-9dae5303ef83",
      "metadata": {
        "id": "7d08b234-3bbf-48c8-b760-9dae5303ef83",
        "outputId": "a8ba1806-c4d2-4618-d566-767dab6f685e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "98/98 [==============================] - 1s 9ms/step - loss: 0.0168 - accuracy: 0.9957 - val_loss: 0.1293 - val_accuracy: 0.9644\n",
            "Epoch 2/5\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.0158 - accuracy: 0.9959 - val_loss: 0.1085 - val_accuracy: 0.9736\n",
            "Epoch 3/5\n",
            "98/98 [==============================] - 1s 7ms/step - loss: 0.0144 - accuracy: 0.9966 - val_loss: 0.1061 - val_accuracy: 0.9747\n",
            "Epoch 4/5\n",
            "98/98 [==============================] - 1s 8ms/step - loss: 0.0134 - accuracy: 0.9967 - val_loss: 0.1074 - val_accuracy: 0.9737\n",
            "Epoch 5/5\n",
            "98/98 [==============================] - 1s 7ms/step - loss: 0.0127 - accuracy: 0.9967 - val_loss: 0.1129 - val_accuracy: 0.9736\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x368c196a0>"
            ]
          },
          "execution_count": 154,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "m.fit(x_train, y_train, callbacks=[tensorboard], epochs=5, batch_size=512, validation_data=(x_val, y_val))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8494922e-83b4-4fbb-983b-9f2924858761",
      "metadata": {
        "id": "8494922e-83b4-4fbb-983b-9f2924858761",
        "outputId": "d3f63bd2-4cef-4934-f28a-eb8b3126b9d6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tensorboard in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (2.8.0)\n",
            "Requirement already satisfied: numpy>=1.12.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (1.21.5)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (2.27.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (60.7.1)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (0.6.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (1.0.0)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (3.19.4)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (1.43.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (3.3.6)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (2.0.2)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (2.6.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (0.37.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (1.8.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from tensorboard) (0.4.6)\n",
            "Requirement already satisfied: six in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from absl-py>=0.4->tensorboard) (1.15.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard) (4.8)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard) (5.0.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from markdown>=2.6.8->tensorboard) (4.10.1)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard) (3.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard) (1.26.8)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard) (2.0.11)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from requests<3,>=2.21.0->tensorboard) (2021.10.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard) (3.7.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /Users/theaveasso/miniforge3/envs/one4all/lib/python3.8/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard) (3.2.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e4454db6-fa0c-4084-af84-c8066309fe35",
      "metadata": {
        "id": "e4454db6-fa0c-4084-af84-c8066309fe35",
        "outputId": "2bf4cd3f-22c9-46ff-a0db-e26d36ed1639"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "      <iframe id=\"tensorboard-frame-2879c088eda8b5bd\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
              "      </iframe>\n",
              "      <script>\n",
              "        (function() {\n",
              "          const frame = document.getElementById(\"tensorboard-frame-2879c088eda8b5bd\");\n",
              "          const url = new URL(\"/\", window.location);\n",
              "          const port = 6006;\n",
              "          if (port) {\n",
              "            url.port = port;\n",
              "          }\n",
              "          frame.src = url;\n",
              "        })();\n",
              "      </script>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "%load_ext tensorboard\n",
        "%tensorboard --logdir /full_path_to_your_log_dir"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0b3ca4de-8c94-4227-9ac1-79e0d529948c",
      "metadata": {
        "id": "0b3ca4de-8c94-4227-9ac1-79e0d529948c"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "interpreter": {
      "hash": "af938e745b869fc0bb119cb255e80df8e6694c91036d995a3efb80f9282657b8"
    },
    "kernelspec": {
      "display_name": "one4all",
      "language": "python",
      "name": "one4all"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "colab": {
      "name": "chapter07_working_with_keras_a_deep_dive.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}